Invariance vs Robustness of Neural Networks
Authors: Sandesh Kamath
Amit Deshpande
K V Subrahmanyam
Venue:  (2020)
Keywords: Invariance
Adversarial
Robustness

Abstract: Neural networks achieve human-level accuracy on many standard datasets used in image classification. The next step is to achieve better generalization to natural (or non-adversarial) perturbations as well as known pixel-wise adversarial perturbations of inputs. Previous work has studied generalization to natural geometric transformations (e.g., rotations) as invariance, and generalization to adversarial perturbations as robustness. In this paper, we examine the interplay between invariance and robustness. We empirically study the following two cases:(a) change in adversarial robustness as we improve only the invariance using equivariant models and training augmentation, (b) change in invariance as we improve only the adversarial robustness using adversarial training. We observe that the rotation invariance of equivariant models (StdCNNs and GCNNs) improves by training augmentation with progressively larger rotations but while doing so, their adversarial robustness does not improve, or worse, it can even drop significantly on datasets such as MNIST. As a plausible explanation for this phenomenon we observe that the average perturbation distance of the test points to the decision boundary decreases as the model learns larger and larger rotations. On the other hand, we take adversarially trained LeNet and ResNet models which have good \ell_\infty adversarial robustness on MNIST and CIFAR-10, and observe that adversarially training them with progressively larger norms keeps their rotation invariance essentially unchanged. In fact, the difference between test accuracy on unrotated test data and on randomly rotated test data upto \theta , for all \theta in [0, 180], remains essentially unchanged after adversarial training . As a plausible explanation for the observed phenomenon we show empirically that the principal components of adversarial perturbations and perturbations given by small rotations are nearly orthogonal

Under review as a conference paper at ICLR 2020
INVARIANCE Vs. ROBUSTNESS OF NEURAL NET- WORKS
Anonymous authors Paper under double-bling review
ABSTRACT
Neural networks achieve state of the art accuracy on many standard datasets used in image classification. One would like to design neural network models whose performance not affected by natural (or non can-malicious) geometric transforma- tions as well as adversarial (or malicious) perturbations. Previous work has stud- ied generalization to natural geometric transformations (e.g., rotations) as invari- ance, and generalization to adversarial perturbations as robusiness. In this paper, we examine the interplay between invariance and robustness. We empirically study the following two cases: (a) change in adversarial robustness prove only the invariance using equivariant models and training augmentation, improve only the adversarial robustness using ad-
im-
(b) change in invariance versarial training. We observe that the rotation invariance of equivariant models (StdCNNs and GCNNs) improves by training augmentation with progressively larger rotations but while doing so, their adversarial robustness drops progres- sively and this drop very significant on MNIST, As a plausible explanation for this phenomenon we observe that the average perturbation distance of the test points to the decision boundary decreases as the model learns larger and larger ro- tations. We take adversarially trained LeNet and ResNet models which have good lo adversarial robustness on MNIST and CIFAR-I0, IO, respectively, and observe that adversarially training them with progressively larger norms results in pro- gressive drop their rotation invariance profiles, On these adversarially trained models we observe that the change in the rate of invariance is marginal for small rotations. As a plausible explanation for this phenomenon, we show empirically that the principal components of adversarial perturbations and the principal com- ponents of perturbations given by small rotations are nearly orthogonal.
INTRODUCTION
Neural networks achieve state of the art accuracy on several standard datasets used in image clas- sification. However, their performance in the wild depends how well they handle natural or non-adversarial tramsformations of input seen in real-world data as well as known deliberate, adversarial attacks created fool the model.
Natural or non-adversarial transformations seen in real-world images include translations, rotations, and scaling. Convolutional Neural Networks (CNNs) are translation-invariant or shift-invarian by design. Invariance to other symmetries, and especially rotations, have received much attention recently, e.g., Harmonic Networks (H-Nets) by Worrall al. (2016), cyclic slicing and pooling by Dieleman et al.] (2016), Transformation- Invariant Pooling (TI-Pooling) by Laptev et al. (2016),
Group-equivarias Convolutional Neural Networks (GCNNs) by Cohen & Welling (2016). Steerable CNNs by Cohen & Welling (2017). Deep Rotation Equivariant Networks bi et (2017), Rotation Equivariant Vector Field Networks (RotEqNet) by Marcos et al. (2017). Polar Transformer Networks (PTNN by Esteves et al.] (2018). For given symmetry group G, G-equívariant net- work learns a representation or feature map at every intermediate layer such that any transformation g G applied to input corresponds to an equivalent transformation of its representations. Any model can improve its invariance to given group of symmetries through sufficient training aug- mentation. Equivariant models efficient weight sharing and require smaller sample complexity to achieve better invariance. Equivariant models such as CNNs and GCNNs too generalize well to progressively larger random rotations, but only when their training data is augmented similarly.

---

Under review as a conference paper at ICLR 2020
Adversarial attacks on neural network models are certain, deliberate changes to inputs that fool highly accurate model but are unlikely to fool humans. Given any neural network model, Szegedy et al. (2013) show how to change the pixel values of images only slightly so that the change is almost imperceptible to human eye but makes highly accurate models misclassify. [Szegedy et al. (0013) find these adversarial pixel-wise perturbations of small magnitude by maximizing the prediction error of a given model using box-constrained L-BFGS, Goodfellow et al. (2015) propose Fast Gradient Sign Method (FGSM) that applies the adversarial perturbation as x' is the input, represents the target. A represents the model parameters, and J(0, xz y) is the loss function used to train the network. Moreover, [Godffello et al. (2015) propose adversarial training, or training augmented with points (2', y). as a way to improve adversarial robustness of a model.
+ e sign (V (V,.((0, 2, v)), where
Subsequent work has introduced multi-step variants of FGSM, notably, an iterative method by Ku- rakin et al. (2017) and Projected Gradient Descent (PGD) attack by Madry et al. (2018). Given any model, these attacks produce adversarial perturbation for every test image x: from a small fx- around it, namely, each pixel value x; is perturbed within [x; E, + eJ. PGD attack does so by solving an inner optimization by projected gradient descent over -ball of radius e around æ to approximate the optimal perturbation. Adversarial training with PGD perturbations improves the adversarial robustness of models, and is one of the best known defenses to make models robust to perturbations of bounded foo norm on MNIST and CIFAR-10 datasets, as shown in Madry et al. 2018) and Athalye et al. (2018).
Recent work has looked at simultaneous robustness to multiple adversarial attacks. Engstromm iet al. (017)) show that adversarial training with PGD makes CNNs robust against perturbations of bounded loo norm but an adversarially chosen combination of small rotation and a translation can still fool these models nevertheless, Recent work of |Schott et al. (2119) shows that PGD adver- sarial training is a good defense against perturbations of bounded Loo norm but can be broken with adversarial perturbations of small &n or l norm that are also imperceptible to humans or have little semantic meaning for humans. [Schott et al. (2019) show how to build models for MNIST dataset that are simultaneously robust to perturbations of small lo. la and lo norms.
1.1 PROBLEM FORMULATION AND OUR RESULTS
Let f be neural network classifier trained on training set of labeled images. The accuracy of f the fraction of test inputs x for which the predicted label f(z) matches the true label y. For a given transformation T of the image space, I' is said to be T-invariant if the predicted label remains unchanged after the transformation T for all inputs, i.e., [(Txx = [(x)), for all inputs x. If f is T-invariant and f(z + a) * fz)) for some input and its adversarial perturbation with ||||| S e, then by invariance f(Tx + To) = f(x + a) * f(x) = f(Tx). Let 7' be a translation, rotation, more generally abstracted as a permutation of coordinates in Tramer & Boneh (019)), then ||Tallp = | < c. Hence, Ta is an adversarial perturbation for input Tx of small &p norm. When a change of variables maps Tx by permutation of coordinates, then the gradient and T operators can be swapped. In other words, gfad(f) at Tx is the same as T applied to gfad(f) at x. This gives a 1-1 correspondence between FGSM (and PGD) perturbations of x and Tx, respectively, with Ep norm at most e. As corollary. the lp fooling rate for any T T-invariant classifier f on the transformed data (Tx : x E X3 must be equal to its ép fooling rate on the original data X.
A subtlety kicks in when f is not truly invariant, that is, f(T'x) = f(x), for most inputs x but not all x. Define the rate of invariance of classifier f to a transformation T as the fraction of test images whose predicted labels after transformation T remains unchanged, i.e., f (Tx:) = fx). For class of transformations, e.g., rotations upto degree [-0", +0%]. we define the rate of invariance as the average rate of invariance over transformations T' in this class. The rate of invariance is 100% iff the model ff is truly invariant. When f is not truly invariant, the interplay between the invariance under transformations and robustness under adversarial perturbations of small Ep -n-ormm is subtle. This interplay is exactly what we investigate.
In this paper, we study neural network models and the simultaneous interplay between their rate of invariance for random rotations between [-0", +0%, and their adversarial robustness to pixel-wise perturbations of lo norm at most
Measuring the robustness of a model to adversarial perturbations of f, norm at most e is NP-hard as shown in Katz et al. (2017) and Sinha et al. (2018). Athalyy al. (018)) compare most of the
2

---

Under review as a conference paper at ICLR 2020
known adversarial attacks to argue that PGD is among the strongest. Therefore, adversarially perturbed test data using PGD attack as a proxy for adversarial robustness.
accuracy on
Unlike previous studies by Engstrom et al. 2017) and Tramèr & Bonch (201)), we do not fix the magnitude pixel-wise adversarial perturbations (c.g., say e = 0.3) nor limit ourselves to small rotations up to +:30% Another important difference is consider random rotations instead of adversarial rotations. We compute the rate of invariance of a given model on inputs rotated by a random angle between --0,+0% for 0 varying in the range [0, 180). Similarly, we normalize the underlying dataset, and compute the accuracy of a given model on adversarially perturbed test inputs by PGD and FGSM adversarial attacks of &o norm most e, for varying in the range [o. 1].
We empirically study the following: (a) change in for adversarial robustness as we improve only the rate of rotation invariance using training augmentation with progressively larger rotations, (b) change in invariance we improve only adversarial robustness using PGD adversarial training with pro- gressively larger -norm of pixel-wise perturbations.
We study equivariant models, StdCNNs and GCNNs, as well as LeNet and ResNet model from Madry et al. (2018). Equivariant models, especially GCNNs, when trained with random rotation augmentations come very close to being truly rotation invariant; see Cohen & Welling (2016). Ad- versarially trained LeNet and ResNet models with PGD are among the best known &oo adversarially robust models on MNIST and CIFAR-10, respectively, as shown in Madry al. (2018), and recon- firmed by Athalye et al. (2018). In other words, these models essentially represent the two separate solutions known currently for achieving invariance and adversarial robustness, respectively.
Our two main observations are as follows.
(i) Equivariant models (StdCNNs and GCNNs) progressively improve their rate of rotation invariance when their training is augmented with progressively larger random rotations but while doing so, their foo adversarial robustness drops progressively. This drop or trade-off is very significant on MNIST.
(ii) Adversarially trained LeNet and ResNet models from Madry et al. 20118 8 trained using progressively larger &oc -nom attacks improve their adversarial robustness but while doing so, their rate of invariance to random rotations upto + drops progressively. On these adversarially trained models we observe that the change in the rate of invariance is marginal for small rotations.
We empirically observe that the average distance the test points to the decision boundary reduces when models are trained with larger rotations. This could explain observation (i). We empirically observe that adversarial perturbations are nearly orthogonal to perturbations caused by small ro- tations of test points. This could explain the marginal change in the rate of invariance for small rotations, mentioned in (ii).
Related Work Schott al. (2019) study simultaneous robustness to adversarial perturbations of small e. f2, and 20 -nomm. Tramer& & Boneh (2019) show an impossibilityy result by exhibiting data distribution where model can have substantially thanarothan accuracy for binary classifi- cation simultaneously against both & and & perturbations, and also against both foo perturbations and spatial perturbations, given by adversarial permutation of coordinates that models an adver- sarially chosen combination of a small rotation and small translation. They show an empirical validation of this claim on MNIST and CIFAR-10 datasets for simultaneous robustness against Loo adversarial perturbation and an adversarially chosen combination of translation upto +3 pixels and rotation upto +:30%. Intuitively and theoretically, it has been argued in Engstrom et al. (2019) and Tramer & Boneh (2019) that small, adversarial pixel-wise perturbations and small, adversarial geometric transformations are essentially dissimilar attacks focusing on different features, due to which a simultaneous solution to both may be inherently difficult. They do not postulate any grad- ual trade-off between invariance and robustness. They do not consider group- euui-euuivariant models such GCNNs, natural choice for invariance to geometric transformations.
2 ROTATION INVARIANCE VS. loo ADVERSARIAL ROBUSTNESS
In this section, we present our main result about the interplay between rotation invariance and fxx adversarial robustness of models MNIST and CIFAR-10 data. In Subsection 2.1 take GCNN
3

---

Under review as a conference paper at ICLR 2020
models from Cohen & Welling (016) and study their rotation invariance and & adversarial robust- ness, as we trai them with random rotations of progressively larger degree. In Subsection 2.2 we take LeNet and ResNet models from Madry et al. (2018) and study their rotation invariance and foc. adversarial robustness, as we do PGD adversarial training with progressively larger &oo norms.
We present our experimental results rotation invariance profiles and adversarial robustness pro- files, as explained below.
Rotation invariance means that the predicted labels of an image and any of its rotations should be the same. Since most datasets are centered, we restrict our attention to rotations about the center of each image. We quantify rotation invariance by measuring the rate of invariance or the fraction of test images whose predicted label remains the same after rotation by random angle between [-0", . As & varies from 180, we plot this as the rotation invariance profile of given model.
Adversarial robustness means that the predicted labels of an image and its adversarial perturbation should be the same. The lx adversarial robustness of given model to fixed adversarial attack (e.g., FGSM, PGD) and fixed for norm (say. 0 S c S 1) is quantified by the accuracy of the given model on perturbed test data with an adversarial perturbation of loe norm at most 6 generated by the attack. The resulting plot of the accuracy of the model as varies from 0 to 1 is the robustness profile of the given model.
Convention used in the legends of our figures: We use the following convention in the legends of some plots. A coloured line labeled A/B B/B indicates that the training data is augmented with random rotations from | A A and the test data is augmented with random rotations from [-B", Bº). If A (resp. B) is zero means the training data (resp. test data) is unrotated. If the model is trained with random rotations from [-A?, AA] and the test data is randomly rotated with varying B to draw the plot, we only mention A and not B, which is self-explanatory.
Equivariant models (e.g.. GCNNs, H-Nets, PTNs, NetE) are designed be robust to rotations through clever weight sharing. However, models such as GCNNs and H-Nets are not robust to rotations if their training data not sufficiently augmented. This appears be folklore so we do not elaborate on this in the main paper. See Appendix
2.1 EFFECT OF ROTATION INVARIANCE ON I20 ADVERSARIAL ROBUSTNESS
For any fixed 0 E [0, 180), we take equivariant model, namely, StdCNN or GCNN, and augment its training data by random rotations from [-0", +0%) Figure left), Figure leeft) shows how the robustness profile of StdCNN change MNIST and CIFAR-10 10 respectively,
increase the degree 8 used in training augmentation of the model. We use PGD adversarial attack for MNIST and CIFAR-10. Figure righht) and Figure 2 rightt show the rotation invariance profile of the same models on MNIST, CIFAR-10 respectively.
The black line in Figure (lef),, shows that the adversarial robustness of StdCNN which is trained to handle rotations up to 180 degrees on MNIST, drops by more than 50%, even when the budget for PGD attack unrotated MNIST is only 0.1. The black line in Figure [ (right), shows this models rotation invariance profile- this model is invariant to larger rotations in the test data. This can be contrasted with the model depicted by the red line- this StdCNN is trained to handle rotations up to 60 degrees. The rotation invariance profile of this model is below that of the model depicted by the black line and so is not invariant to large rotations. However this model can handle adversarial lo perturbations up to 0.3 on unrotated data, with an accuracy more than 80% this can be seen from the red line in Figure (lfft).
From these plots it is clear that the rotation invariance of these models improves by training aug- mentation but at the cost of their adversarial robustness, indicating a trade-off between invariance to rotations and adversarial robustness. The above observations, of there being trade-off between handling larger rotations with training augmentation and handling larger adversarial perturbations is seen in GCNNs also. This can be seen from Figures 28 and Figure 29 The plots are very similar to what we observe with StdCNNs.
We have additional experiments in Appendix B where we look at the adversarial accuracy of the above models when the test data is also augmented with random rotations. The conclusions are similar. Unlike the invariance profiles, our adversarial robustness profiles do not start 100%
4

---

Under review as a conference paper at ICLR 2020
200.0
300.0
00
00/0
30/0
90.0
rate of inaaliance
150/0
180/0
70.0
40.0
60.0
90
120
50.0
150
0.
02
0.4
0.6
0.8
100
0
25
50
100
125
150
175
Epsiton
of random rotations augmented to
Figure 1: On MNIST, StdCNN trained with varying random rotations in [-0",0 range. (left) Robustness profile, (right) Rotation invariance profile.
200.0
300.0
Q0
30/0
90.0
60/0
80.0
rate of invartance
150/0
70.0
380/0
60.0
40.0
50.0
40.0
120
30.0
0.00
0.02
0.04
0.06
0.08
25
50
125
150
175
random
Figure 2: On CIFAR-10, StdCNN/VGG16 trained with varying random rotations in 1-0", -0] range. (left) Robustness profile, (right) Rotation invariance profile.
because their starting point is the natural accuracy on unperturbed test data. The natural accuracy on unperturbed test data is known drop after adversarial training, as noted by Tsipras al. (2019). In Appendix(F] we plot (1 - fooling rate) instead of accuracy, so that the adversarial robustness proñles also start with 100%. The fooling rate is defined as the fraction of test images whose predicted label changes after ad adversarial perturbation.
100.0
300
00
30/0
90.0
120/0
rate of invarance
60.0
150/0
18000
70.0
40.0
60.0
90
0.0
40.0
04
0.6
25
50
175
Range
augmented
Figure 3: On MNIST, GCNN trained with varying random rotations in |-00, 0*] range. (left) Ro- bustness profile, (right) Rotation invariance profile.
5

---

Under review as a conference paper at ICLR 2020
100.0
300.0
00
30/0
90.0
00/0
80.0
rate of inualiance
150/0
70.0
180/0
40.0
40.0
120
30.0
150
0.00
0.02
0.04
0.06
0.08
0.10
0
25
50
100
125
150
175
Epsitio
Range of random rotations augmented
Figure 4: On CIFAR-10, GCNN/VG116 trained with varying random rotations in [-0", 00] range. (left) Robustness profile, (right) Rotation invariance profile.
2.2 EFFECT OF foc. ADVERSARIAL TRAINING ON ROTATION INVARIANCE
The most common approach improve adversarial robustness is adversarial training, i.e., training the model on adversarially perturbed training data. Adversarial training with PGD attack is one of the strongest known defenses on MNIST and CIFAR-IO datasets, see Athalye et al. (2018).
For any fixed E [0 adversarially train our models, LeNet and ResNet, as given in Madry jet al. (2018) with PGD adversarial perturbations with lo budget e. Similar to Madry al. (2018) the LeNet model for MNIST and the ResNet model for CIFAR-10. We then plot their rotation invariance profiles and their robustness profiles. Each colored line in Figure and Figure corresponds to model adversarially trained with different value of
On MNIST, adversarial training with PGD with larger results a drop the invariance profile of LeNet based model in Figure 5 (left), the yellow line (PGD with 0.4) is below the light blue line (PGD with e = 0.1). This is true for the ResNet based model too on CIFAR-10, 10, as can be seen from Figure 6 (left). In other words, adversarial training with progressively larger leads to the drop in the rate of invariance on the test data.
To complete this picture we plot the robustness profile curves of the LeNet and ResNet based model for MNIST and CIFAR-IO, 10, respectively. It is known that as these models are trained with PGD using larger e budget their adversarial robustness increases. The robustness profile curves of the LeNet model trained with larger PGD budget dominates the robustness profile curve of the same model trained with smaller PGD budget- the red line Figure (right), dominates the light blue line. This is true of the ResNet based model too, as can be seen from Figure 6 (right).
100.0
80.0
30.0
0.4
0
0.4
Accustty
60.0
50.0
40.0
20.0
30.0
0.0
20.
Angle
Epsilio
Figure 5: For PGD adversarially trained LeNet based model from Madry al. 2018 on MNIST (left) Rotation invariance profile, (right) Robustness profile. Different colored lines represent models adversarially trained with different l.x budgets 6 E (0, 1).

---

Under review as a conference paper at ICLR 2020
200.0
300.0
0255
0255
90.0
2755
2255
80.0
4/35
80.0
70.0
Accuraty
40.0
20.0
50
75
300
125
150
175
0.00
0.2
0.04
0.06
0.08
0.10
0.12
Angle
Epsiton
Figure 6: For PGD adversarially trained ResNet based model from Madry et al. (018 CIFAR- 10 (left) Rotation invariance profile, (right) Robustness profile. Different colored lines represent models adversarially trained with different Lo budgets E [0, 1].
DECISION BOUNDARY DISTANCE AND ORTHOGONALITY
In order to understand what could he the possible reasons behind our empirical observations in Section 2. and Section 2.2 perform more experiments.
3.1 AVERAGE PERTURBATION DISTANCE THE BOUNDARY.
For each test image, adversarial attacks find perturbations of the test point with small too norm that would change the prediction of the given model. Most adversarial attacks do so by finding the directions in which the loss function of the model changes the most. order to explain why these networks become vulnerable to pixel-wise attacks they learn more rotations, we see how the distance of the test points to the decision boundary changes as the networks learn larger rotations. This is abstractly depicted in Figure where we show the distance of test point Zg to the boundary Do (resp. Dis0) when the model trained with zero (resp. +180%) rotations.
We use the La attack vectors obtained by DeepFool (Moosavi-Deztooll et al. 2016) for the dat- apoints under attack. We take norm of this attack vector as an approximation to the shortest distance of the test point to the decision boundary. For each of the test points collect the pertur- bation vectors given by DeepFool attack and report the average perturbation distance. We plot this average distance the datasets augmented with larger rotations.
Our experiments show that as the networks learn larger rotations with augmentation, the average perturbation distance falls. So as (symmetric) networks become invariant to rotations, they are more vulnerable to pixel-wise attacks.
The plots in Figures 8
TO 11 show this for StdCNNs and GCNNs MNIST and CIFAR-10. To make our point we plot the accuracy of these networks and also the average perturbation distance of the points alongside in one figure. The blue line in Figure Bleft shows the accuracy of StdCNN on MNIST when both the training data and test data are augmented with 0, as 0 ranges from 180. The green line in Figure leftt) shows the accuracy of the StdCNN model when the train is augmented with random rotations upto f, and the test is augmented with rotations upto 0 and also perturbed with PGD of lo norm 0.3. The red line shows the accuracy when the test is not augmented with rotations but is PGD perturbed with for norm 0.3.
9
The red line Figure rightt shows the average perturbation distance of the unrotated test points when the network is trained with rotations upto 0. The green line shows the average perturbation distance test points which are augmented with rotations upto 0 this is about 5 when 0 is 0% (the point the y-axis where the curves begin). As the network is trained with random rotations up to 180 the average perturbation distance the augmented test drops below 3.5. Figure 8 left) shows that that the PGD accuracy has dropped from around 85% for the network at 0 to 30% at 180 (the corresponding green line on the left). The fact that the red line is above the green line is also reflected in Figure 14 in Appendix C When the test is perturbed by PGD, the accuracy of the

---

Under review as a conference paper at ICLR 2020
StdCNN with training data augmented with rotations is better when the test is not augmented with rotations than if the test were also augmented with rotations.
D 380
Figure 7: Distance of point zo to decision boundary Diso obtained by augmenting training set with random rotations in range [-180,, 180%] is different compared to the decision boundary Do obtained with no training augmentation.
200.0
StdCMN on rotated test StuCCN on unrotated best
80.0
5.5
Accurscy
00.0
4.5
20.0
No PGD
PCD on rotated test
35
PGD unrotated
50
300
125
150
175
0
25
50
100
125
150
175
Range of random rotations sugmented to train
Range of random rotations augmeried trai
Figure 8: Accuracy of StdCNN on MNIST with/without PGD (e = 0.3), on rotated and unrotated test. Train/test if augmented are with random rotations in [-0", 0%). (left) Accuracy, (right) Avg. Perturbation Distance.
100.0
6.5
No PGD PGD on rotated test
GCNN on rotated test GCNN on unrotated test
80.0
PGD unrotated test
5.5
00.0
4.5
20.0
3.5
50
75
300
125
150
175
0
25
30
300
125
150
175
Range of random rotations augmented to train
Range of random rotations augmerced train
Figure 9: Accuracy of GCNNs on MNIST with/without PGD (e 0.3) on rotated and unrotated test. Train/test if augmented are with random rotations in 1-82 8%). (left) Accuracy, (right) Avg. Perturbation Distance
8

---

Under review as a conference paper at ICLR 2020
300.0
2.0
StaChi on rocated test
90.0
1.8
StSCNN on unrotated test
80.0
70.0
Average Perturbation
60.0
PGSM on unrotated test
rocated
122
1.0
25
50
75
300
125
150
175
0
25
50
100
125
150
175
Range of random rotations augmented to
Range random rotations augmented train
Figure 10: Accuracy of StdCNNs/VGG16 on CIFAR-10, with/without FGSM (c = 0.01) on rotated and unrotated test with c 0.01. Train/test if augmented are with random rotations in [-0 8*) (left) Accuracy, (right) Avg. Perturbation Distance.
100.0
20
GCN on rotated test
90.0
GCNN on unrotated test
80.0
70.0
Average Perturbation
3.4
60.0
FCSM on unrotated test
FCSM rotated
12
50.0
1.0
40.0
30.0
20.0
300
125
150
50
175
augmented
random
Figure 11: Accuracy of GCNNs/VGG16 on CIFAR-10, with/without FGSM (e 0.01) on rotated and unrotated test. Train/test if augmented are with random rotations in 1-0,0%) (left) Accuracy, (right) Avg. Perturbation Distance.
3.2 ORTHOGONALITY OF ADVERSARIAL PERTURBATIONS & INVARIANT ROTATIONAL PERTURBATIONS
In general we expect neural networks to classify and small rotation x' of identically. This would be particularly so if the dataset has some inherent rotations and the labels remain the same under such rotations. Writing x' = A(x). we say A(x) is invariant rotational perturbation for x. For a network trained to be equivariant to rotations, this rotational perturbation of x could well be orthogonal an adversarial perturbation of since adversarial perturbation changes the label whereas an invariant rotational perturbation does not change the label. We empirically show that this is the case for MNIST and CIFAR-10. For each test input we find its PGD perturbation and stack these ad adversarial perturbations as rows of an adversarial matrix. We find the top 5 right singular vectors of this adversarial matrix. We rotate each input by 4 /2/2" for MNIST/CIFAR-10 and subtract the original image from it. We stack such invariant rotational perturbations as rows of an invariance matrix. We find the top 5 right singular vectors of this invariance matrix. We empirically observe that the principal angles between these two 5-dimensional subspaces are close to 90". For completeness we recall the definition of principal angles in Appendix D
Table I gives the principal angles of MNIST trained on LeNet and Table 2 gives the principal angles for CIFAR- 10 trained on ResNet as given by MMdry et al. (2018).
9

---

Under review as a conference paper at ICLR 2020
Table I: MNIST: Principal angles between Top-5 SVD-subspace of PGD attack directions of test points with for norm and Top-5 SVD-subspace of the difference directions between test point and its 4 rotation.
2
3
4
5
0.0
89.48
89.03
86.91
84.79
80.97
0.1
89.93
87.97
86.15
81.62
76.99
0.2
89.56
88.00
85.83
82.29
80.73
0.3
89.75
88.49
87.75
84.96
80.45
0.4
89.33
88.55
86.86
85.80
82.13
Table CIFAR-10: Principal angles between Top-5 SVD-subspace PGD attack directions of test points with &oo norm and Top-5 SVD-subspace of the difference directions between test point and its 2 rotation.
2
4
0/255
89.99
89.40
88.27
86.98
85,99
2/255
89.76
88.76
88.31
87.15
86.33
4/255
89.78
89.02
88.30
86.78
86.41
8/255
89.69
89.13
88.11
87.25
86.42
4 CONCLUSION
We observe that as equivariant models (StdCNNs and GCNNs) are trained with progressively larger rotations their rotation invariance improves but at the cost their adversarial robustness. plau- sible explanation of the first observation is that the average perturbation distance of the test points to the boundary decreases as StdCNNs and GCNNs learn larger rotations. Adversarial training with perturbations of progressively increasing norms improves the robustness of LeNet and ResNet mod- els, but with a resulting drop in their rate of invariance. On these adversarially trained models we observe that the change in the rate of invariance is marginal for small rotations. As plausible explanation for this phenomenon, we show empirically that the principal components of adversar- ial perturbations and the principal components of perturbations given by small rotations nearly orthogonal.
Acknowledgement We thank an anonymous reviewer for suggestions that improved the presenta- tion of our invariance profile plots and the invariance-robustness. trade-off significantly.
REFERENCES
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense se- curity: Circumventing defenses to adversarial examples. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, July 2018.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In Proceedings of the International Conference Machine Learning (ICML). 2016.
Taco S. Cohen and Max Welling. Steerable CNNs. In International Conference on Learning Rep- resentations, 2017.
Sander Dieleman, Jeffrey De Fauw, and Koray Kavukcuoglu. Exploiting cyclic symmetry in convo- lutional neural networks. In Proceedings of the International Conference on Machine Learning (ICML), 2016.
Logan Engstrom. Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. A rotation and translation suffice: Fooling CNNs with simple transformations, arXiv preprint arX0::1712.02779, 2017.
10

---

Under review as a conference paper at ICLR 2020
Logan Engstrom, Brandon Tran, Dimitris Tsipras. Ludwig Schmidt, and Aleksander Madry. Ex- ploring the landscape of spatial robustness. ICML, 2019.
Carlos Esteves, Christine Allen-Blanchette, Xiaowei Zhou, and Kostas Daniilidis. Polar transformer networks. In International Conference on Learning Representations, 2018.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representationss 2015.
Guy Katz, Clark W. Barrett, David L. Dill, Kyle Julian, and Mykel J. Kochenderfer. Reluplex: An efficient smt solver for verifying deep neural networks. In CAV, 2017.
Alexey Kurakin, lan Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint arXiv:1607.0253: 2017.
Dmitry Laptev, Nikolay Savinov, Joachim M. Buhmann, and Marc Pollefeys. Ti-pooling: transformationiinaaiane pooling for feature learning convolutional neural networks. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 289-297, 2016.
Junying Li, Zichen Yang, Haifeng Liu, and Deng Cai. Deep rotation equivariant network. arXiv preprint S.Kiv:1705.086220 2017.
Aleksander Madry, Aleksandar A Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
Diego Marcos, Michele Volpi. Nikos Komodakis, and Devis Tuia. Rotation equivariant vector field networks. In International Conference on Computer Vision, 2017.
Scyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, 2016.
Lukas Schott, Jonas Rauber, Matthias Bethge, and Wieland Brendel. Towards the first adversarially robust neural network model on MNIST. In International Conference Learning Representa- tions, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CORR, 14111.49.55555 2014.
Aman Sinha, Hongseok Namkoong, and John C. Duchi. Certifying some distributional robustness with principled adversarial training. In 6th International Conference Learning Representa- tions, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceed- ings, 2018.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 2013.
Florian Tramèr and Dan Boneh. Adversarial training and robustness for multiple perturbations. CoRR, abs/1904. 13000, 2019.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In International Conference on Learning Representa- tions, 2019.
Maurice Weiler, Fred A. Hamprecht, and Martin Storath. Learning steerable filters for rotation equivariant enns. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
D. E. Worrall, S. J. Garbin, D. Turmukhambetov, and G. J. Brostow. Harmonic networks: Deep translation and rotation equivariance. arXiv preprint 4r216:1612.04642, 2016.
11

---

Under review as a conference paper at ICLR 2020
Table 3: Architectures used for the MNIST and Fashion MNIST experiments
Standard CNN
GCNN
Conv(10,3,3) + Relu
P4ConvZ2(10,333) Relu
Conv(10,3,3) + Relu Max Pooling(2,2) Conv(20,3,3) + Relu
P4C0n))4(10,.33) Relu
Group Spatial Max Pooling(2,2)
P4ConvP4(20,333) + Relu P4ConvP4(20.333)+ Relu
Conv(20,3,3) + Relu Max Pooling(2,2)
Group Spatial Max Pooling(2.2)
FC(50) Relu
FC(50) Relu
Dropout(0.5) FC(10) Softmax
Dropout(O.5) FC(10) Softmax
APPENDIX A DETAILS OF EXPERIMENTS
All experiments performed neural network-based models done using MNIST, Fashion MNIST and CIFAR-I0 datasets with appropriate augmentations applied to the train/validation/eestt
Data sets MNIST dataset consists of 70, 000 images 28 x 28 size, divided into 10 classes. 55, 000 used for training. 5, 000 for validation and 10, 000 for testing. Fashion MNIST dataset consists of 70, 000 images 28 28 size, divided into 10 classes. 55, 000 used for training, 5, 000 for validation and 10, 000 for testing. CIFAR-I 10 dataset consists of 60, 000 images
x size, divided into 10 classes. 40, 000 used for training, 10, 000 for validation and 10, 000 for testing.
Model Architectures For the MNIST and Fashion MNIST based experiments we use the network architecture of GCNN as given in Cohen & Welling (2016). The StdCNN architecture is similar the GCNN except that the operations are as per CNNs. Refer Table 3 for details. H-Nets architecture is as given in Worrall et al. (2016). RotEqNet architecture is as given in Marcos et al.
2017 PTN architecture is as given in Esteves et al. (2018).
the VGG16 architecture given in Simonyan & Zis- serman (204 and its GCNN equivalent is obtained replacing the various layer operations with equivalent GCNN operations given in Cohen & Welling (2016). This is similar how we ob- tained a GCNN architecture from StdCNN for the MNIST based experiments. Input training data was augmented with random cropping and random horizontal flips.
For the CIFAR-10 based experiments
For the adversarial training experiments we used the LeNet based architecture for MNIST and the ResNet architecture for CIFAR-10. Both these models are exactly given in Madry et al. (2018).
12

---

Under review as a conference paper at ICLR 2020
APPENDIX B
EFFECT OF INVARIANCE ON ADVERSARIAL ROBUSTNESS AFTER TEST AUGMENTATION
200.0
300.0
00
30/30
30/30
80.0
60/60
80.0
60/60
90500
120/120
Accuracy
190/180
100/180
40.0
40.0
20.0
20.0
0.0
02
04
0.6
0.5
1.0
0.0
0.2
0.4
0.6
0.6
1.0
Epstion
Epsiion
100.0
300.0
0/0
0/0
30030
30/30
80.0
60/60
80.0
60/60
90/90
250/150
Accuracy
250/150
180/180
180/180
40.0
40.0
20.0
0.00
0.02
0.04
0.06
0.08
0.10
0.00
0.02
0.04
0.06
0.08
0.10
Epsition
Epsiton
Figure 12: Robustness profile of StdCNN models. (top left) PGD attack on MNIST, (top right) PGD attack on Fashion MNIST, (bottom left) FGSM attack CIFAR-10, 10, (bottom right) PGD attack on CIFAR-10.
13

---

Under review as a conference paper at ICLR 2020
200.0
300.0
0
80
30/30
30/30
80.0
50/60
90/90
120/120
120/120
60.0
150/150
Acuracy
60.0
150/150
180/180
180/180
40.0
40.0
044
0.6
10
Epstion
100.0
00
90/90
120/120
Accrrty
60.0
350/150
180/180
40.0
0.00
0.02
0.44
0.10
Figure 13: Robustness profile of GCNN models. (top left) PGD attack on MNIST, (top right) PGD attack on Fashion MNIST, (bottom) FGSM attack on CIFAR-IO,
200.0
09
00
30600
300
80.0
60/60
80.0
800
50.50
900
120/120
1200
00.0
150/150
ACcuRcC
00.0
1500
150/180
150.0
40.0
0.
60
02
0.4
05
03
100
0.0
02
Epsiton
Epulkn
Figure 14: Robustness profile of StdCNN models on MNIST attacked with FGSM. (left) train and test augmented with [-0", 0º] range (right) Only train augmented with [-0", 00] range and no test augmentation.
100 0
200.0
00
300
800
000
5010
200
120/120
1200
150/150
Accrncy
15000
100/180
180/0
80
0.
0.6
Esstion
Figure 15: Robustness profile of StdCNN models on Fashion-MNIST attacked with FGSM. (left) train and test augmented with --000%) range (right) Only train augmented with [-0", 00000 0*) range and no test augmentation.
14

---

Under review as a conference paper at ICLR 2020
Accurccy
40.0
40.0
20.0
20.0
0.00
082
0.04
0.05
0:08
0.10
0.00
0.02
0.44
0.06
0.10
Epstion
Epsioon
Figure 16: Robustness profile of StdCNN/VGG16 models on CIFAR-10 attacked with PGD. (left) train and test augmented with -00 range (right) Only train augmented with [-0", 0*] range and no test augmentation.
100.0
09
00
3050
300
80.
60100
00.0
0000
50.50
900
120/120
1200
60.0
1501150
Accuncy
60.0
1500
1801180
180.0
0.0
0.
0.
1.0
0.0
02
Essiton
Epsilon
Figure 17: Robustness profile of GCNN models MNIST attacked with FGSM. (left) train and test augmented with [-0?, 09] range (right) Only train augmented with [-00, 80] range and no test augmentation.
1000 0
300.0
09
90
300
80
0260
000
5090
900
170/120
120/0
150/150
Accuncy
1500
100/100
180/0
00
0.
Essiion
Figure 18: Robustness profile of GCNN models on Fashion-MNIST attacked with FGSM. (left) train and test augmented with -0000% range (right) Only train augmented with [-0", 0*) range and no test augmentation.
15

---

Under review as a conference paper at ICLR 2020
APPENDIX C DECISION BOUNDARY DISTANCE, ORTHOGONALITY
200.0
10
rotated
StdCMN on unrotated test
80.0
0.0
No PGD
0.6
23.0
0.0
75
100
125
150
175
0
25
50
100
150
175
Range of random rotations augmented to tram
Range of randoom rotations augmented tram
Figure 19: On Fashion MNIST, StdCNNs, Comparison of network with/without PGD on rotated and unrotated test with 0.1. Train/test if augmented are with random rotations in [-0", 0*). (left) Accuracy, (right) Avg. Perturbation Distance.
200.0
1.0
en rotated GCNN on unrotated test
80.0
0.8
No PGD
0.6
20.0
0.0
50
75
100
125
150
175
25
50
100
125
150
175
Range of random rotations augmented to tram
Range of random rotations augmented tram
Figure 20: On Fashion MNIST, GCNNs, Comparison of network with/without PGD on rotated and unrotated test with 0.1. Train/test if augmented are with random rotations [-0", 0 (left) Accuracy, (right) Avg. Perturbation Distance.
16

---

Under review as a conference paper at ICLR 2020
APPENDIX D PRINCIPAL ANGLES
Recall that for two subspaces V.W. the first principal angle is defined as the minimum angle between lwo unit vectors U1 V, wi W. The second principal angle is the minimum angle between unit vectors U2 E W2 E W, with U2 1 U1 and W2 1 w1. The other principal angles are defined similarly.
APPENDIX E ROBUSTNESS OF ROTATION EQUIVARIANT NETWORKS TO ROTATIONS
In this appendix we confirm that many popular rotation equivariant models are not robust to rota- tions if their training data not sufficiently augmented with rotations. We do this by plotting their accuracy when the test data is augmented with rotations but the training data is not augmented with rotations of the same range. We also show empirically that their accuracy improves by training with small sample of data augmented with random rotations. We observe that (a) Rotation equivariant networks are robust to only small degrees of rotations away from the ones present in the training data - this was studied only for CNNs in Weiler et al. (2018). (b) applying data augmentation during training increases their accuracy even when the test data is augmented with rotations. (c) Rotation equivariant networks achieve state of the art results with smaller sample size for training.
E ROBUSTNESS TO ROTATIONS WITHOUT TRAINING AUGMENTATION
We first train all the networks on MNIST and Fashion MNIST with no rotation augmentation and test against inputs augmented with varying range of random rotations from =0 to 1180 Figure top left) for MNIST and Figure top right) for Fashion MNIST shows the performance of these networks to the entire range of rotations. We observe that even though these datasets are small and these networks are designed to be invariant to rotations, their performance drops as the range of random rotations in test data increases. For small range of rotations up to about +:20", the performance of these networks MNIST remains above 95% indicating that MNIST dataset has some natural rotation augmentation. On Fashion MNIST, the performance is above 85% when the test is augmented with small rotations up +20 When the test is augmented with rotations beyond 25", the accuracy of these networks drops. We observe that PTN and RotEqNet are more robust than the other networks for MNIST and Fashion MNIST.
On the dataset CIFAR-10 we compare the performance of rotation-equivariiat GCNNs with translation-equiivarian StdCNNs. This is shown in Figure (bottmm)) GCNNs perform better, as expected, but the performance of both GCNNs and StdCNNs degrades to less than 70% accuracy when the test data augmented with random rotations larger than +300
E.2 ROBUSTNESS WITH TRAINING AUGMENTATION
We train equivariant networks with input augmented with varying range of random rotations from +0º to +180 Figure 22 plots the accuracy for MNIST (top left). Fashion MNIST (top right) and CIFAR-10 (bottom), for both StdCNN's and GCNN's. The red(GCNN) and blue(StdCNN) lines indicate networks for which no train augmentation is done. The coordinate of a point on the black (resp. brown) line corresponding to 8 on the X-axis (for 0 in the range of [0, 180) indicates the accuracy of a GCNN (resp. StdCNN) with training data augmented with random rotations in the range +0 and with test data also augmented with random rotations in the range +0º For GCNN's we see a gap of about 4-5% on Fashion MNIST and also CIFAR- 10, between the accuracies when trained and tested with no rotation augmentations (# 0) and when trained and tested with 1 rotations (0 180). It is only on MNIST that rotation-equivare same performance ( 1 difference). For StdCNN's 's the gap is more pronounced, almost 10-15% on Fashion MNIST and CIFAR-10.
networks achieve almost the
17

---

Under review as a conference paper at ICLR 2020
200.0
300.0
90.0
90.
MiNets
GCNN
RotEaNett
Accuazy
60.0
HNltts
50.0
PTN
RotEggNet
40.0
20.0
100
125
150
175
0
25
50
100
125
150
175
Range random rotations augmented to test
Range of random rotations augmented to test
300 0
9:dONN GCNN
90.0
80.0
70.0
Accuracy
25
50
320
125
150
175
Range random rotations augmented to test
Figure 21: Accuracy of models with no training augmentation. (top left) MNIST, (top right) Fashion MNIST, (bottom) CIFAR-10.
100.0
200.0
90.0
Accurzy
SMIMNN
Augmented Train augmented
CCNN
StdCNN- Train Augmented
GCNN Train Augmented
20.0
75
100
125
150
175
25
50
100
125
150
175
Range random rotations augmented to test
Range of random rotations engmentedeee to test
100 0..0
90.0
80.0
Accuracy
70.0
StdCNN
Augmented GCNN Train Augmented
25
320
125
150
175
Range random rotations augmented to test
Figure 22: Networks trained with and without augmentation to dataset, random rotation augmenta-
tions 1-0", 00] range. (top left) MNIST (top right) Fashion MNIST (bottom) CIFAR-10.
18

---

Under review as a conference paper at ICLR 2020
E.3 SAMPLE COMPLEXITY OF NETWORKS
To understand the sample complexity of the networks, we perform two experiments. In the first we train the networks with varying sample sizes of training set and test them the entire test set. And in the second experiment we do the same as the first with the inputs train and test augmented with random rotations in the range [-180%, 180%]. From Figure 23 for MNIST, Figure 24 for Fashion MNIST and 25 for CIFAR-10, we can see that rotation equivariant networks achieve their best per- formance safely using 10k 30k training samples. This confirms that rotation equivariant networks can do well with smaller training sample size.
200.0
90.0
Accurity
90.0
SMOCNN
SEOCHN
GENN
GENN
RottigNet
RottigNeet
75.0
75.0
10
10
10
10' Sample Sizex11000)
Sample Stzex11000)
Figure 23: Networks trained with varying training sample size on X-axis. (left) Only MNIST, (right) MNIST train and test augmented with random rotations in 180%, 180%] range.
100.0
100.0
SUCNNN
GCNN
90.0
RotEgNet
90.0
Acuraty
70.0
85.0
ScdCNN
80.0
GONN
MNeets
50.0
75.0
10a
10a
10a
107
Sample Size(x1000)
Sample Sizexx1000)
Figure 24: Networks trained with varying training sample size X-axis. (left) Only Fashion MNIST, (right) Fashion MNIST train and test augmented with random rotations in [-180, 180%] range.
100.0
100.0
SMCNN
StCCNN
25.0
GONN
90.0
GCNN
80.0
Accuscy
80.0
70.0
0
60.0
50.0
05.0
60.0
400
10P
10c
10F
Sample Steelx1000)
Sample Siz0010000)
Figure 25: Networks trained with varying training sample size X-axis. (left) Only CIFAR-10, l
(right) CIFAR-10 train and test augmented with random rotations |-180, 180%] range.
19

---

Under review as a conference paper at ICLR 2020
APPENDIX F ROBUSTNESS PROFILES USING FOOLING RATE INSTEAD OF ACCURACY
200.0
300.0
80.0
300
60/0
90.0
rale of invertance
80.0
180/0
60.0
20.0
50.0
150
00
02
044
0.6
0.5
1.0
25
50
100
125
150
175
Epstion
Range of random rotations augmented to test
Figure 26: On MNIST, StdCNN trained with varying random rotations in [-0", ,0 range. (left) Robustness profile, (right) Rotation invariance profile.
200.0
200.0
30/0
90.0
90/0
80.0
120/0
rate of invariance
60.0
350/0
70.0
60.0
40.0
50.0
40.0
30.0
0.0
20.0
04
0.6
25
50
175
Range
augmented
Figure 27: On CIFAR-10, StdCNN trained with varying random rotations in [-00, ,0) range. (left) Robustness profile, (right) Rotation invariance profile.
200.0
200.0
30/0
80.0
60/0
90.0
rate of invortance
80.0
180/0
80.0
20.0
50.0
150
120
180
40.0
0.
0.2
04
0.6
0.5
1.0
25
50
100
125
150
175
Epstton
Range of random rotations augmented to test
Figure 28: On MNIST, GCNN trained with varying random rotations in [-0%, 00] range. (left) Robustness profile, (right) Rotation invariance profile.
20

---

Under review as a conference paper at ICLR 2020
200.0
300.0
Q0
30/0
90.0
60/0
80.0
rate of invartance
150/0
70.0
180/0
60.0
40.0
[1.
40.0
120
30.0
00
02
04
0.6
25
50
125
150
175
random
Figure 29: On CIFAR-10, GCNN trained with varying random rotations in 1-0", ,0] range. (left) Robustness profile, (right) Rotation invariance profile.
200.0
60
80.0
80.0
20.0
[1. foollingggg (a00)
60.0
60.0
50.0
40.0
30.0
0.0
20.0
1.0
Angle
Epstion
Figure 30: For PGD adversarially trained LeNet based model from Madry al. 2018 MNIST (left) Rotation invariance profile, (right) Robustness profile. Different colored lines represent models adversarially trained with different Ex budgets E [o, 1].
300.0
300.0
0255
0255
90.0
2255
2255
80.0
80.0
[1 fooling nat)
70.0
60.0
20.0
100
125
150
0.00
0.22
0.04
0.06
0.8
0.10
0.12
Figure 31: For PGD adversarially trained ResNet based model from Madry et al. 2018 CIFAR- 10 (left) Rotation invariance profile, (right) Robustness profile. Different colored lines represent models adversarially trained with different l budgets E [0, 1].
21


Reviews:
[{"review_id": "rkx9Ros2FS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper examines the interplay between the related ideas of invariance and robustness in deep neural network models. Invariance is the notion that small perturbations to an input image (such as rotations or translations) should not change the classification of that image. Robustness is usually taken to be the idea that small perturbations to input images (e.g. noise, whether white or adversarial) should not significantly affect the model's performance. In the context of this paper, robustness is mostly considered in terms of adversarial perturbations that are imperceptible to humans and created to intentionally disrupt a model's accuracy. The results of this investigation suggests that these ideas are mostly unrelated: equivariant models (with architectures designed to encourage the learning of invariances) that are trained with data augmentation whereby input images are given random rotations do not seem to offer any additional adversarial robustness, and similarly using adversarial training to combat adversarial noise does not seem to confer any additional help for learning rotational invariance. (In some cases, these types of training on the one hand seem to make invariance to the other type of perturbations even worse.)\n\nThis paper is mostly clear and reasonably written. However, I do not think that the results of this investigation are significant enough to warrant publication at ICLR. In particular, I'm not sure that I really understand the motivation of this research question. I suppose a full notion of robustness would include invariance to perturbations of all types -- whether adversarial or otherwise -- and one might hope that techniques for encouraging such resilience mutually reinforce each other. However, since they are human perceptible, the perturbations associated with invariance are exactly the features that are not associated with adversarial noise, so I don't see why they should be related at all. \n\nFrom a different perspective, invariance is a property of the true data distribution -- a rotated version of an image of a cat is another valid sample from the underlying distribution -- the invariance property is a type of constraint tying together different elements of the data generating distribution. On the other hand, adversarially perturbed images are often thought to be \"off the data manifold\" -- i.e. not valid samples from the true underlying distribution. Given this perspective, I am confused about why I should expect to see any interplay between these two ideas. The fact that the authors do not find any interplay is reasonable, but I remain confused about why they were investigating this question in the first place. \n\nUnfortunately, this means that I don't think this work meets the significance criterion for being published at ICLR. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJxQy7DCKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper analyzes the behaviour of DNN trained with rotated images and adversarial examples. Namely, the paper analyzes the relationship between training with rotated images and the robustness to adversarial perturbations, and vice-versa.\n\nThe paper has several technical issues that need to be resolved before drawing any conclusions:\n\n1) “invariance”: this term is not used in the correct way. The fact that the network has the same accuracy when before and after rotation does not mean that the output layer is invariant to rotation. Note invariance in the output layer is a more stringent criterion as it requires that the images get labeled in the same way. The same accuracy can be achieved with completely different labelings of the images. What this paper is evaluation is robustness to rotation vs robustness to adversarial perturbations.\n\n2) It is unclear that Figure 3 is saying that adversarial training does not affect the rotation invariance because there is a general drop of accuracy. The analysis could have been done by evaluating how many images are labelled differently after the rotation, and all the plots will be aligned at 0 degrees.\n\n3) Finding out the robustness to adversarial perturbations is an NP-hard problem. So, for all tested cases in the paper, there could be a perturbation that damaged the model much more than the ones found, which could change the conclusions of the analysis. \n\n4) The networks compared in the two experiments are different networks. There could be a network dependency.\n\nAlso, I find the paper poorly written (eg. in the abstract: \"Neural networks achieve human-level accuracy on many standard datasets used in image classification.” -> what does it mean “human-level accuracy”?; \"The next step is to achieve better generalization to natural (or non-adversarial) perturbations” -> why is this the next step?). \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "r1lG531aKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper shows empirically that rotational invariance and l infinity robustness are orthogonal to each other in the training procedure. However, the reviewer has the following concerns,\n\nIt is already shown in (Engstrom et al., 2017) that models hardened against l infinity-bounded perturbations are still vulnerable to even small, perceptually minor departures from this family, such as small rotations and translations. What is the message beyond that paper that the authors would like to convey?\nThe experiments are only on MNIST and CIFAR-10.  Training on a larger dataset  like imagenet would make the experiments more convincing.\nGoing beyond the observation, what shall we do to improve against different perturbation simultaneously? Or is it an impossible task to improve on both?\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper examines the interplay between the related ideas of invariance and robustness in deep neural network models. Invariance is the notion that small perturbations to an input image (such as rotations or translations) should not change the classification of that image. Robustness is usually taken to be the idea that small perturbations to input images (e.g. noise, whether white or adversarial) should not significantly affect the model's performance. In the context of this paper, robustness is mostly considered in terms of adversarial perturbations that are imperceptible to humans and created to intentionally disrupt a model's accuracy. The results of this investigation suggests that these ideas are mostly unrelated: equivariant models (with architectures designed to encourage the learning of invariances) that are trained with data augmentation whereby input images are given random rotations do not seem to offer any additional adversarial robustness, and similarly using adversarial training to combat adversarial noise does not seem to confer any additional help for learning rotational invariance. (In some cases, these types of training on the one hand seem to make invariance to the other type of perturbations even worse.)

Unfortunately, the reviewers do not believe the technical results are of sufficient interest to warrant publication at this time. 
Decision: Reject