Leveraging Adversarial Examples to Obtain Robust Second-Order Representations
Authors: Mohit Prabhushankar
Gukyeong Kwon
Dogancan Temel
Ghassan AlRegib
Venue:  (2020)
Keywords: Second-order representation
adversarial examples
robustness
gradients

Abstract: Deep neural networks represent data as projections on trained weights in a high dimensional manifold. This is a first-order based absolute representation that is widely used due to its interpretable nature and simple mathematical functionality. However, in the application of visual recognition, first-order representations trained on pristine images have shown a vulnerability to distortions. Visual distortions including imaging acquisition errors and challenging environmental conditions like blur, exposure, snow and frost cause incorrect classification in first-order neural nets. To eliminate vulnerabilities under such distortions, we propose representing data points by their relative positioning in a high dimensional manifold instead of their absolute positions. Such a positioning scheme is based on a data point’s second-order property. We obtain a data point’s second-order representation by creating adversarial examples to all possible decision boundaries and tracking the movement of corresponding boundaries. We compare our representation against first-order methods and show that there is an increase of more than 14% under severe distortions for ResNet-18. We test the generalizability of the proposed representation on larger networks and on 19 complex and real-world distortions from CIFAR-10-C. Furthermore, we show how our proposed representation can be used as a plug-in approach on top of any network. We also provide methodologies to scale our proposed representation to larger datasets.

Under review as a conference paper at ICLR 2020
LEVERAGING ADVERSARIAL EXAMPLES TO OBTAIN ROBUST SECOND-ORDER REPRESENTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Deep neural networks represent data projections trained weights in high dimensional manifold. This is first-order based absolute representation that is widely used due to its interpretable nature and simple mathematical functionality. However, in the application of visual recognition, first-order representations trained on pristine images have shown vulnerability to distortions. Visual distortions including imaging acquisition errors and challenging environmental conditions like blur, exposure, snow and frost cause incorrect classification in first-order neural nets. To eliminate vulnerabilities under such distortions, we propose representing data points by their relative positioning in a high dimensional manifold instead of their absolute positions. Such a positioning scheme is based on a data point's s second-order property. We obtain a data point's second-order representation by creating adversarial examples to all possible decision boundaries and tracking the movement of corresponding boundaries. We compare our representation against first-order methods and show that there is an increase of more than 14% under severe distortions for ResNet-18. We test the generalizability of the proposed representation on larger networks and on 19 complex and real-world distortions from CI-AR-10-C. Furthermore, we show how our proposed representation can be used as plug-in in approach top of any network. We also provide methodologies to scale our proposed representation to larger datasets.
INTRODUCTION
In recent years, artificial intelligence systems achieved state-of-the-art performances in image clas- sification tasks (Russakovsky al., (0155) (Krizhessky et al., (He et al.) (016). Specifically. classification algorithms surpassed top-5 human error rate of 5. 1% on ImageNet (Russakovsky et al., 2015). Even though these advancements are promising, images in these datasets do not cover diverse real-world scenarios. For instance, ImageNet consists of photographs parsed from Flickr, a popular image hosting service. The images on Flickr are generally high quality since users tend not share distorted photographs. Distortions may include perceptually unpleasant camera related issues like blur, motion blur, overexposure, underexposure, and noise. Moreover, environmental conditions such as rain, snow, and frost can affect the field of view. These non-ideal conditions impact the performance of artificial intelligence (AI) algorithms (Dodge & Karam (017))(Heddryck & Dietterich, 2019).
These AI algorithms are primarily driven by deep neural nets that learn non-linear transformations to obtain discriminate representation spaces. Deep neural networks are trained to transform a data point into a representation space where linear classifiers can discriminate between classes (Goodfellow et al. Krizhesk et al., (0e2) et al.) 2016). All the hidden layers are supervised to maximize the interclass distance while minimizing the intraclass distance to obtain linearly separable representations in the last layer. We formulate the inherent mechanisms behind the classification process as follows: Let be L layered neural network trained to distinguish between N classes. If x is any input to the network, the output for a classification application is given by f(z) = y where y is (N 1) vector. The class of is the index of the maxima of y. Consider only the final fully connected layer fr parameterized by weights (Wz) and bias (6z)) We obtain y as,
y == WT fx) +b,
(1)
@evxI,We& Vy E Nx1 Wr e X ft-1(()) E sdd-1111 bb E Nx1

---

Under review as a conference paper at ICLR 2020
where f2-1 is the flattened output of the network just before the final fully connected layer. All the data points that span fL-1 representation space should be linearly separable for a well trained fL- Note that Eq. is filtering operation between the weight vectors and the representation. Hence, the final fully connected layer in a network can be considered as a linear filter set with N' filters onto which the representations (fz.-1) are linearly projected (Wang al. 2019). Projection refers to dot product between the data point and the filters. The filter WI, Vi E [1 that has the largest projection or is maximally correlated with the data point represents the corresponding class.
In the above setting, the original data point a is represented by its projection intensity (z) This representation can be analyzed as a first-order point process (Dorai-Rae et al.) (00T). As an example, consider cities in the USA. All cities can be located on a map using their latitudes and longitudes as shown Fig This an instance of directly using the intensity in this case the latitude and longitude- to represent data point- in this case a city. The first-order representation is widely used for intuitive nature, interpretability and ease of mathematical operation.
First Order Point Representation
Second Order Point Repre sa nttation
800
Mia
Hou
CHRUK
Adl
SF
Ew (0%, 1619
CRA PRAR
Miles
Den
-200
LA. 3109
WDC
Chhi
-400
NYC
400-
.Sa
4000
Figure 1: (a) Cities located using first order point representation Latitudes and Longitudes. (b) Cities located based on second order point representation Pairwise Distance.
However, under distortions and perturbations, this representation is not stable (Azulay & Weiss GooddGoodf et al.] 20144). The authors in Goodfellow et al. (2014) show the instability of such systems with adversarial images. In Azulay & Weiss (2018), the authors demonstrate that when images are translated by few pixels, the network output and the hidden layer representations change drastically, We show that this change holds for distortions as well in Fig 2 Consider an image taken from MNIST (LeCun et al. 1998) dataset. The image is subjected to five levels of progressive blurring all of which are visualized in Fig Fig shows the t-SNE (Maaten & Hinton] 2008) feature visualization of first order point representations of corresponding fi-1 (x Blur for 10000 images in MNIST test set. The individual shapes and absolute locations of each cluster get progressively distorted with increasing levels of blur. Hence, it is natural that network trained on original data performs poorly on distorted data.
An alternative to representing point by its intensity is by representing its influence every other point in the subspace (Dorai-Raj et al.) 200T). This is the second-order property of data points. Consider the same example as before from Fig However, instead of latitudes and longitudes, the pairwise distance between all cities is provided. A Multi-Dimennional Scaling (MDS) algorithm used obtain the relative positioning of the cities with respect each other as shown in Fig Dp. MDS algorithms (Mead) 1992) are a class of algorithms that use second-ordes point representations to translate pairwise distances of N objects into an abstract cartesian space.
In Sections 2 and 3 we provide an intuitive methodology to extract the recond-orde representation of any data point. In Section 4 we categorize the existing literature for classification under distortion and organize them into frameworks based on data dependency. We describe the experimental setup and discuss the results in Section55an and conclude our work in Section K
2 SECOND ORDER POINT REPRESENTATIONS
Second-order point representations have been utilized in popular ML algorithms like k-Nearest Neighbor (kNN) classification. kNN classifies data point based on the classes of k of its nearest neighbors, nearest being determined by predefined distance metric. However, these algorithms are computationally expensive. Similarly, classical MDS algorithms (Hout et al. 2013) Mead 1992) on
2

---

Under review as a conference paper at ICLR 2020
Original
Level
Level
Level
night
Figure 2: (a) Progressive blurring of sample image from MNIST dataset (b) t-SNE maps of corresponding blurry MNIST testset for first-order representation (c) t-SNE maps of corresponding blurry MNIST testset for proposed second-order representation.
M datapoints, require formation of pairwise distance matrix of size M x M, which is not feasible for large-scale datasets.
In this paper, propose proxy method for representing data point using its second-order property. Instead of representing a data point as pairwise distance against other points, we represent as a cumulative distance to all decision boundaries. We use adversarial image generation along with gradients of the final fully connected layer to obtain this cumulative distance. Consider Fig, (3 where the data point is projected onto filter from the final fully connected layer, W Vi E [V. This provides a projection P., Vi [1, Nj. During backpropagation, if i is the given class for data point a, the filter moves in a direction that maximizes x's new projection P. The movement of the filter is controlled by gradients. Hence, gradients provide the negative direction of data point to filter Wi. Our goal is to not just get the direction to the maximum projection class but to all the classes. Hence, by backpropagating over all e [1, .N] for a datapoint, point to every filter Wi. However, one iteration of backpropagation is not sufficient to obtain the distance between a datapoint the decision boundary of a filter Wi. Hence, we generate targeted adversarial image with class i, Vi E [1. NJ for the datapoint and absolute sum the gradients over the required iterations. This cumulative gradient provides the distance from a datapoint to a decision boundary. Creating adversarial images for every class E [1, .N] provides multidirectional features. This concept is shown in Fig 3.. We create N adversarial images for every datapoint and track their individual movement until they hit decision boundary.
obtain the direction of that data
These features form a proxy second-order representation for datapoint. The benefits of this representation are illustrated in the t-SNE maps for the MNIST testset in Fig. -2b. The relative shapes of the class clusters are preserved for the proposed representation in Fig. [2 compared to first-order representations in Fig. 2b. Further stability results discussed in Appendix A.I
MULTI-DIRECTIONAL FEATURE GENERATION
Continuing the notations established in Secs. [ and 2 a L layered network f is trained to distinguish between N' classes using original distortion free images. For every image in the distortion free dataset, targeted Fast Gradient Sign Method (Goodfellow et al., 2014) (FGSM) is applied to create N adversarial images obtain z's distance to decision boundaries. For target i E [1. an adversarial noise esign( (o.gnV,,JDe (W, x, i)) is added to the image x. J[W, xz i) refers to the cost function used to train the network with parameters W. V.r is the gradient of the cost function w.r.t. the input z. of 0.1 is used in this work. Adversarial noise is added to the input over is iterations until the classification changes to i, i. fxx=1 + esign( V (W, Ik. ())) = i. The absolute value of the gradient of the cost function with respect to filter Wi is summed up over k iterations, i.c
Σド= abs( (Vw; JW, 2.y i)) where r; is the feature for th class. is then iterated over all N
3

---

Under review as a conference paper at ICLR 2020
Gradients across Adversarial Iterations as Distance
Filter movement after backpropagation
Gradient Distance
Class
New Filter Position
W?
Gradient (G²)
G₁
Data Point
Unear Clas
W Original Filter Position
Separators
W
Gy
P/
WF
Class
P Original Projection
Class 2
[0,0]
(a)
(b)
Figure 3: (a) Motivation for using gradients as directional information. (b) Motivation for constructing multi-target adversarial images.
classes to obtain multi-directional features of x to all decision boundaries. All corresponding are concatenated to obtain the final feature Ta [ri72 .../2...NN/ Hence the final multi-directional feature is given by,
k
k-1 Σ abs( WWW
Tx abs( V w1 JWW 1)) abs V W (W.
(2)
Va the proposed proxy second-order representation of Note that if the dimensionality of the (T. 11th layer from Eq 1 d x 1, then the dimensionality of every subfeature is also dı-1 x 1. The concatenated final feature has dimensionality (N dt,-1) x 1. For reference, the network fr is VGG-16 trained on CIFAR-I0, dz-1 is 512 x 1 and r. is 5120 x 1. This representation can get prohibitive with increased number of classes. Simple methods offset this increase and speed up rz generation are discussed in Sec. 5.3
The dartiidirectional features Tz are as analogous fi-1(x) as pairwise distances are to longitudes and latitudes in Fig. Hence, is used in the same way as (z). However because of change in values and dimensionality, the same final fully connected layer ft. cannot be used discriminate rz- Instead train new classifier 9L top of Tæx 9L is new network whose parameter dimensions depend on inputs. For instance, in Sec. 5.1 gt is a layer fully connected network separated by ReLU non-linearity for ResNet-18 whose т, dimensions are 640 x 1. For ResNet-101 101 in Sec. 5.2 whose ru dimensions are 2560 x 1, IL is a layered fully connected network. On a NVIDIA 1080TI GPU, 9L for ResNet-101 101 trains within 320 sees.
The recond-order representation of any data point is obtained from Eq Note that the generation of gradients is deterministic procedure and does change repeated experiments for the same network. However, the generation procedure is tied to a specific network and hence the representation of the same differs for different networks. In Section 5 we implement networks and show that the proposed second order representation is robust compared to its first order counterpart in all of them. Moreover, the cost function JWW ar, i) used to obtain gradients. impacts the value of said gradients and creates differences in rx- In our experiments, we use MSE loss function to define J. Results comparing multiple loss functions on distorted CIFAR dataset is shown in Appendix A.2
4 EXISTING FRAMEWORK AND RELATED WORKS
All the existing literature for alleviating the effects of distortions for classification use first-order representations to describe a data point. Moreover, majority the algorithms use some form of distorted data during training. The authors in Vasiljevic al. (2016) show that finetuning VGG-16 using blurry training images increases the performance of classification under blurry conditions. However, performance between different types of blurs is not generalized and knowledge of type of blur is required for good performance. The authors in Zhou et al. (2017) propose finetuning and retraining early feature extraction layers of the network increase classification accuracy. Temel et al.
4

---

Under review as a conference paper at ICLR 2020
(017)) propose utilizing distorted virtual images to boost performance accuracy. All these works require knowledge of distortion and large amounts of distorted data during training that makes them impractical in real life scenarios like in autonomous driving. We categorize methods that require distorted data while training into a Full-Reference (FR) framework. Retraining, finetuning and data augmentation techniques that utilize distorted data during training fall under the FR framework.
In contrast, any framework that does not require distorted images during training will be termed as No-Reference (NR) framework. The authors in Hendryeks & Dietterich| (2019) show adversarial defense schemes can increase distortion robustness. Most of the existing NR approaches generally try to denoise (Buades et al., 2005) the distorted test image before feeding into a network. These include Hossain et al. (2018) where the authors use a DCT module to remove high frequency components from every image. Generally, these denoising algorithms are both computationally expensive and time consuming (Vasiljevic et al., 2016). Moreover, denoising algorithms require parameter tuning that varies by distortion types, levels and image resolutions. The proposed second order representation can be applied on top of any network, trained in either FR or NR fashion. Hence, by definition our approach follows a NR framework. Even when f is trained in a FR framework, we obtain rz from undistorted and train IL in NR fashion.
Additionally, we propose a new framework between the FR and NR frameworks called Reduced- Reference (RR). RR framework require only few statistics of the distorted images for training 9L- RR framework differs from domain adaptation since it does not need test domain images during training but only their statistics. Such framework is more appropriate in practical applications where it is impractical to anticipate, obtain or record all possible distortions. While the results of RR approaches may not exceed FR techniques, they come much cheaper data price. In our case, we convert our NR framework to RR by normalizing Ta with mean and standard deviation statisties of distorted data. The terminologies of FR, NR, and RR are borrowed from the Image Quality Assessment community (Prabhushankar et al. 2017).
Blur
S&P
Gaussian Noise
09
Accuracy
0.8
Accuracy
0.8
0.8
0.7
0.
6.7
0.6
0.
0.6
0.5
9.5
0.5
0.4
0.4
0.3
0.3
02
02
62
Distortion Levels
Distortion Levels
Distortion Levels
ExpoSuue
Motion Blur
UnderExposure
1.0
1.0
0.9
0.
0.9
0.8
0.0
6.8
Accuracy
0.7
Accuracy
0.7
0.7
0.6
06
0.
0.
65
0.8
64
03
02
02
62
Distortion Levels
Distortion Levels
Distortion Levels
Original
RR
NR
FR1
FR2
FR3
NR1 NR2
Figure 4: Results on 10. Original is the first-order ResNet-18 18. NR is the proposed second-
order representation and RR statistics enhanced proposed second-order representation.
5 RESULTS
The performance of second-order over first-order representations is validated this section.
5.1
MULTI-DIRECTIONA FEATURES vs FIRST-ORDER FEATURES
Comparison against first order representation: CIFAR-10 10 (Krizhevsky & Hinton 2009) dataset is used to evaluate the effectiveness of the proposed recond-order representation space against first-order methods. We consider six distortions gaussian blur, salt and pepper, gaussian noise, overexposure,
5

---

Under review as a conference paper at ICLR 2020
motion blur, and underexposure. Five progressively worsening levels of all six distortions created on CIFAR- 10 dataset similar to the procedure followed by Temel et al. (2018). ResNet-18 is chosen as the original model f. trained on undistorted data. The accuracy distortions for f are visualized in Fig. D in blue where the y-axis shows the drop in performance and x-axis shows the corresponding distortion level. Level O indicates that there is no distortion. A significant performance drop is noticed across distortions for f. In contrast, when the original model's first-order representation is converted proposed second-order, the results increase by an average of 5.21% 2 across levels and distortions. This is shown in the green NR curve. The difference more pronounced under level severe distortions where the average accuracy across distortions increases by 10.04%. The red curve is the proposed RR second-order representation. The proposed RR outperforms the original by an average 7.72% across all distortions and levels. When there are no distortions, the level (I results in the plots suggest that the proposed second-order representation performs similar to the first-order original representation.
progressively worsening
Comparison against NR models: We compare the proposed second-order representation against 2 NR methods. NRI technique denoises distorted test data by Non Local Means (NLM) before feeding into f. Explicit denoising by NLM (Buades al., 2005) was analyzed by HHendrycks & Dietterich (2019). Because of the small resolution of CIFAR- 10 images, NLM significantly blurs the distorted images, thereby adding additional artifacts into the already distorted images. Hence it performs worse than the original model under all distortions. Similar results were obtained using Hossain et al.) (2018) where a drop of 8% was observed even on pristine images. NR2 is an adversarially robust classifier trained using undistorted images and untargeted FGSM images. Except in blur distortion, NR2 under performs the proposed NR in all other distortions performing even worse than the original model in Gaussian noise and underexposure.
Comparison against FR models: The original model's second-order performance gains are com- pared against 3 FR models FRI (Zhou et al. 2017), FR2 ( (Vasiljevic et al. 2016), and FR3, data augmentation technique. The performance of the proposed RR and NR methods slot near the middle of the original and FR1 1 and FR2 models. The performance gain of proposed RR vs the original is 7.72% while the gain of FRI over proposed RR is 7.70%. Note that both FRI and FR2 require separate models for every distortion and distortion level i.e there are 31 trained models for each of FR1 I and FR2 to produce the plots in Fig. This shows that the proposed second-order representation can be a useful tool in practical applications to enhance the original model's performance where distorted images are not easily obtainable during training. FR3 is trained on distortion free images augmented with 1000 images each from all 6 distortions. It can be seen that the proposed RR representation performs similarly to FR3 in all but SP noise. Moreover, for pristine data at level 0, FR3 performs worse than the second-orde representations.
Comparison against other second order methods :e compare our proxy second-order representa- tion against true second-order representations, All existing networks utilize first-order representations to make decisions. To simulate a recond-order scenario, We construct k-Nearest Neighbor (kNN) classifier on activations from fz)) We do so for four i values- - 5, 25, 50, 500. Our proxy second- order NR outperforms all KNN classifiers by 6.37% averaged over all 6 distortions among all 5 levels on ResNet- 18. This because instead of calculating pairwise distance between simultaneously moving distorted data, we calculate distance between fixed (decision boundary) and one moving distorted data. Such a proxy method provides more robustness than its true second-order counterpart.
5.2 MULTI-DIRECTIONAL FEATURE ROBUSTNESS
Robustness multiple architectures: The significance of the second-order representation derives from its broad utility and applicability as a "plug-in" on any gradient-based neural network. In Section [5. the proposed representation performs similar to FR3. But once FR3 is trained and deployed, the proposed representation can be plugged on top of FR3 to obtain additional performance increase. Fig 5h shows two bars for every distortion. The left bar in each distortion indicates the performance of model ResNet- 18 trained only on undistorted data using first-order representation (in blue) and fx=)'s corresponding increase for RR (in red) on level 5 distortions. The right bar shows model FR3's first-order results in blue and it's corresponding increase (in red) when converted to the proposed second-order representation. The "plug-in" functionality is not unique to ResNet- 18 and works multiple networks. 5ig shows other common networks (x-axis) all of whose averaged original results (in blue) across levels and distortions are improved using NR (in green) and additionally by RR (in red). From the results, NR ResNet- 18 is as robust as ResNet-101.

---

Under review as a conference paper at ICLR 2020
Accuracy Gains for Original and FR3 models
Accuracy Gains for Networks
1.0
Original
0.2
Original
0.9
Gain Over Original
NR Gein Over Original
0.8
Gain Over FR3
0.78
Gain NR
Accuracy
0.7
Accuracy
0.76
0.6
0.72
0.5
0.70
0.68
86WF
16.
OverEsp
Netton Blu
UndorEsp
Vgg 16
Rennet- 10 Pleseet-34 34
Resnet-S0 30 Resnet-101
(b)
Figure 5: Robustness of accuracy gains of using second-order representation first order represen- tation.
Distortion-wise Accuracy Gains
Level-wise Accuracy
orgiral Bain original
oua Over Oligkal
Accuracy
Accurai
heef
Westee for
tuppcae siggle
Distortion Levels
[b]
Figure 6: Robustness of accuracy gains of using second-order representation over first order represen-
tation on distortions in CIFAR-IO-C.
Robustness on complex distortions: The proposed representation is validated on CIFAR-IO- C (Hendrycks & Dietterich 2019). This dataset is corrupted by 19 distortions. ResNet-18 trained on undistorted CIFAR- 10 images is used as baseline original architecture. The averaged accuracy across levels of progressive distortion on all 19 distortions is shown in Fig. (op. The average increase in accuracy for proposed second order NR representation over all distortions and distortion levels is 3.54%. In the 19 distortions, there is an accuracy improvement of 5% or more. Only brightness and saturation distortions, the average accuracy of proposed NR decreases by 0.23% and 0.59% respectively. The level-wise accuracy gain of proposed second-order representation over original first-order representation averaged over all distortions is visualized in Fig. 6p. The proposed method provides just under 10% increase during severe distortion. Note that since there are no distorted images, there statistics to be measured and used to obtain RR representation. Further analysis and distortion plots are presented Appendix A.5
Robustness higher resolution images: The proposed approach is implemented on higher resolu- tion images of size 96 x 96 x 3 in STL-10 dataset (Coates et af., (201)) ResNet-18 18 architecture is adopted with extra linear layer account for change in resolution. STL-10 is corrupted by the 6 distortions from Sec. 5.1 The results of both NR and RR approaches increase by an average of 2.56% in all but underexposure distortion. In underexposure, the accuracy drops by 1.05%. In level of both blur categories, the RR increase is 6.89%. Additional results are provided in Appendix A.4
5.3 MULTI-DIRECTIONAL FEATURE COST ENHANCEMENT
As indicated in Sec. 3 the dimensionality and time cost for generation of rg may become prohibitive on large datasets for large networks. We propose two enhancements to mitigate these costs without a substantial loss in performance.
7

---

Under review as a conference paper at ICLR 2020
Table I: Accuracy comparison Feature Generation Time Analysis.
Gaussian
Salt and
Gaussian Noise
Over Exposure
Motion Blur
Under
Blur
Pepper
Exposure
x RR
82.96%
60.94%
78.99%
70.34%
71.22%
84.95%
Se (a)
RR
80.38%
60.37%
76.06%
69.76%
69.32%
84.05%
NR
80.31% %
56.55%
76.88%
67.68%
66, 66.84%
83.76%
f2-1(x) NR
80.21% %
60.33%
75.83%
69.70%
67.53%
84.21%
Table 2: Accuracy comparison Dimensionality Reduction.
Gaussian Blur
Salt and
Gaussian
Over Exposure
Motion Blur
Under
Pepper
Noise
Exposure
Original Ta
82.96%
60.94%
78.99%
70.34%
71.22%
84.95%
Fo dim 1024
82.00%
58.54%
77.48%
69.06%
69.70%
84.52%
To dim 512
82.14%
58.06%
77.76%
69.26%
69.94%
84.68%
To dim 256
82.08%
57.93%
77.84%
68.67%
69.70%
84.44%
Feature generation time analysis The proposed representation requires generating adversarial images over all possible N classes during testing phase. Generating Ya in this fashion takes around 0.28s for every image on VGG-16. The reason for the large cost is the multiple backpropagation that happen over all 16 feature extraction modules to create adversarial noise in domain. However, the generation of perceptually similar adversarial images in the spatial æ domain is not our goal. Instead of generating adversarial images as x + esign( JWW, xz i)). we generate adversarial data points in the fL-1 domain. That is, instead of backpropagating to x and taking the partial derivative V (,, xr, i)) w.r.t x, backpropagate only one layer ft-1 and obtain the partial derivative w.r.t f (xx) Hence, in place of adversarial images, we create adversarial data points ft.-1 (x) to obtain rz. Generating Tz in this fashion takes 0.0056s for every image. The average accuracies across levels for all six distortions is presented Table [ The performance of the V (a) RR and NR representations are comparable to their Vz counterparts among all distortions. Additionally, Yr generation for ResNet-18 and ResNet- 101 in Fig takes 0.4 and 0.98s respectively. However, creating advesarial datapoints and obtaining r, takes 0.042 and 0.036s respectively. Hence, this alternate feature generation can be used on networks with large feature extraction modules.
The dimensionality of the second order representation rs is (N dr.-1) x 1. However, this feature is derived from representation space where a linear classifier can discriminate between N classes. On such a representation space, the application of simple dimensionality reduction techniques yield subspaces that are themselves discriminative. Here, we use PCA on Tn derived from VGG- 16 from Section 5.2 The original rs from VGG-16 is vector of length 5120 x 1. PCA algorithm is applied on all r.. From 5120, we decrease the dimensionality to 1024, 512, and 256. The distortion level averaged results of RR method show only a small decrease in performance displayed in Table 2
Dimensionality Reduction:
CONCLUSION
In this paper, we analyzed the representations of deep learning methods from the perspective of point processes. Our construction of a second-order representation space analogous to existing representations. Features in the second-order representation space are based on relative positioning between data points and trained decision boundaries. Such features were utilized to reduce the vulnerability of deep learning methods distortions and real-world challenging conditions. Validation was conducted on architectures. 19 distortions and 2 image resolutions. The proposed second-order representation, which is plug-in approach, enhances robustness performance in neural networks.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize so poorly to small image transformations? arXiv preprint arXiv: 12077. 2018.
A. Buades, B. Coll, and J. Morel. A non-local algorithm for image denoising. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'0S) volume 2, pp. 60-65 65 vol. 2, June 2005, doi: N0..109/CVP..2000...8..
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pp. 215-223, 2011.
Samuel Dodge and Lina Karam. A study and comparison of human and deep learning recogni- tion performance under visual distortions. In 2017 26th international conference computer communication and networks (ICCCN), pp. 1-7. IEEE, 2017.
Sundardas Dorai-Raj et al. First-and second-order properties of spatiotemporal point processes in the space-time and frequency domains. Unpublished Doctoral Dissertation, 2001.
Robert Geirhos, Carlos RM Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Felix A Wichmann. Generalisation in humans and deep neural networks. In Advances in Neural Information Processing Systems, pp. 7538-7550, 2018.
I. Goodfellow, Y. Bengio, and A. Courville. Deep Learning. deeplearningbook. org
MIT Press, 2016.
WWW
lan J Goodffllow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks and Thomas Dictterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint 1rXiv:1903.12266 2019.
Md Tahmid Hossain, Shyh Wei Teng. Dengsheng Zhang, Suryani Lim, and Guojun Lu. Distortion robust image classification with deep convolutional neural network based on discrete cosine transform. arXiv preprint arXiv: /8/ 1.058/9, 2018.
Michael C Hout, Megan H Papesh, and Stephen D Goldinger. Multidimensional scaling. Wiley Interdisciplinary Reviews: Cognitive Science, 4(1):93-103, 2013.
A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Neur. Info. Proc. Syst., pp. 1097-1105, 2012.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient based learning applied to document recognition. Proceedings of the IEFE, 86(11):227--322) 1998.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9Nov)):559-2665, 2008.
Al Mead. Review of the development of multidimensional scaling methods. Journal of the Royal Statistical Society: Series (The Statistician), 41(1):27-39, 1992.
Mohit Prabhushankar, Dogancan Temel, and Ghassan AlRegib. Ms-unique: Multi-model and sharpness-weighted unsupervised image quality estimation. Electronic Imaging, 2017(12), 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 15(():211 1-222, 2015.

---

Under review as a conference paper at ICLR 2020
Dogancan Temel, Gukyeong Kwon, Mohit Prabhuhankar, and Ghassan AlRegib. CURE-TSR: Chal- lenging unreal and real environments for traffic sign recognition. Advances Neural Information Processing Systems (NIPS) Machine Learning for Intelligent Transportations Systems Workshop.
2017.
Dogancan Temel, Jinsol Lee, and Ghassan AlRegib. Cure-or: Challenging unreal and real environ- ments for object recognition. In 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 137-144. IEEE, 2018.
Igor Vasiljevic, Ayan Chakrabarti, and Gregory Shakhnarovich. Examining the impact of blur on recognition by convolutional networks. arXiv preprint arXiv:1611.05766 2016.
Haohan Wang, Zexue He, Zachary C Lipton, and Eric P Xing. Learning robust representations by projecting superficial statistics out, arXiv preprint arXiv:1903.06230 2019.
Yiren Zhou, Sibo Song, and Ngai-Man Cheung. On classification of distorted images with deep convolutional neural networks. In 2017 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1213-1217. IEEE, 2017.
A APPENDIX
A.1 STABILITY OF PROPOSED REPRESENTATION VS FIRST-ORDER REPRESENTATION
In this section, we compare stability of first-order and second-order representations under distortion. In Azulay & Weiss (2018), the authors demonstrate the instability of neural nets. They do so by translating an image by few pixels, and showing that the network output and the hidden layer representation changes drastically. We show that this instability holds for distortions as well in Fig 7 Borrowing the notations from Sec [ the pristine image from MNIST dataset is subjected to five levels of progressive blurring all of which are visualized in Fig 7a. Fig shows the difference in the first-order point representation of fT-1 Let) to the original f (L-r(Z)rig) i.e. each of the maps shown
fL-1 f (Zorig)) for the corresponding image in Fig 7a. Ideally, every map in Fig 7b must be zero, or in this case, must be visualized in blue. However, as can be seen, the representation has values nearing 1 (orange). The representation also varies significantly across blur levels.
in Fig 7 is the abs(f2-1 (
We extract the second-order features from the distortion-free image as well as the blurry images using Eq.2] [] We then obtain the difference between the Ta of original distortion-free image and the blurred images, which are visualized in Fig 7t. Perceptually, the relative difference between blur and original maps are less in proposed representation (more pixels are blue compared to orange) in Fig. 77 compared to the first-order representation in Fig/7p. The t-SNE representations from Fig. are also visualized in Figs. 7 and e.
A.2 LOss FUNCTIONS FOR MULTI- DOLTI-DIRECTIONAL FEATURE GENERATION
In S we describe the generation procedure for the multi-directional feature Tz. The fea- ture is concatenation of multiple gradient directions each of which is generated as r; Σ/zo abss(Ww: J(W,,,,
Here J the cost function that being backpropagated. Gener- ally, neural nets define loss functions that indicate the empirical difference between the predicted and required values. These loss functions control the amount of required backpropagation and consequently the gradients. Since gradients are our features, the choice of loss functions changes the proposed second-order representations. We compared multiple loss functions all of whose distortion- wise level-wise averaged RR results are provided Table CE is Cross Entropy, MSE is Mean Squared Error, LI is Manhattan distance, Smooth L1 is the leaky extension of Manhattan distance, BCE is Binary Cross Entropy and FL is Focal Loss. Notice that the results of all loss functions exceed that of the original. The results of MSE, LI and Smooth L1 are presented by backpropagating a one-hot vector (with 1 at i) multiplied by the average of all activations in the train non-distorted dataset. For ResNet- 18, this number is close 11.
From Table 3 linear or near linear loss functions like L1 and MSE perform better than log based losses. This is because in the ft-11 representation space where linear classifier can discriminate
10

---

Under review as a conference paper at ICLR 2020
Original
Blur Level
Blur Level 2
Blur Level
Blur Lovel 4
Blur Level
[d]
(e)
Figure 7: (a) Progressive blurring of a sample image from MNIST dataset (b) Difference map between original image activation in (L - 1) layer and corresponding blur activation map from the same layer. Ideally they should be zero (or blue). Orange represents blurry MNIST testset for first order representation (d) Difference map for proposed representation. More pixels in blurry image match the original, and hence more blue pixels (e) t-SNE maps of corresponding blurry MNIST testset for proposed representation.
(c) t-SNE maps of corresponding
Table 3: Proposed RR accuracies with different loss function
Original
CE
MSE
Smooth LI
BCE
FL(gamma=2)
RR
67.65%
71.99%
75.38%
75.12%
73.51%
74.23%
72.91%
between classes, linear loss function provides more accurate distance between data points and decision boundaries. We plot the results of the MSE, L1, BCE and original loss functions for all 6 distortions in Fig. 8 MSE outperforms L1 by 0.26% average accuracy and is used in our experiments.
A.3 MNIST PERFORMANCE
We consider gaussian blur, and salt and pepper noise the distortions evaluate method on MNIST dataset. These were chosen because of their large impact on performance of classifiers trained original images. Blur is introduced using standard gaussian kernel with 0 mean and progressively worsening standard deviation from [1.5, 3.5] with an interval of 0.5. This produces five levels of distortion. Sample Gaussian blurred images are shown in Fig Salt and pepper noise
11

---

Under review as a conference paper at ICLR 2020
Blur
S&PP
Gaussian Noise
Accuracy
Accuracy
Distortion Levels
Distortion Levels
Distortion Levels UnderExposure
OverExposure
Motion Blur
Accuracy
Accuracy
96
Distortion Levels
Distortion Levels
Distortion Levels
Original
Proposed RR
L1
BCE
Figure 8: Result comparison for different loss functions. Proposed RR is MSE loss.
is a high frequeney noise that occurs in images due to sudden signal disturbances. Salt and Pepper noise is modelled as randomly scattered white pixels in the image with a density that changes from [0.01, 0.05) over five intervals leading 5 distorted levels. The code for the noise generation process for both MNIST and CIFAR-10 used in Sec. 5 will be made public. The original network fz used to obtain second order features T₂ a standard 2-conv-2-FC network. 9L is trained with 3 layers.
The results are presented in Fig 9 The x-axis and y-axis represents the distortion levels and their corresponding accuracy respectively. Level 0 on the x-axis indicates no distortion and level 5 indicates the highest amount of distortion. In the plots, original refers to the network trained on undistorted images. It can be seen that our second-order representation based RR and NR methods do not suffer from any drop in accuracy on original first-order undistorted data.
In both the distortions, the proposed NR second-order representation outperforms original method. The performance gain is particularly apparent at higher distortion levels. For blur level 5, the proposed NR method outperforms the original network by nearly 18%. This is direct indication of the potential of second-order representations over first-order methods. The proposed RR method outperforms the proposed NR and original methods. blur level 5, it achieves roughly 30% gain over original network.
Blur
Salt And Pepper
Accuracy
Accuracy
0.2
Distortion Levels
Distortion Levels
Original
RR
NR
Figure 9: Results MNIST. NR refers to proposed second-order representation and RR refers to statistics enhanced second-order representation.
12

---

Under review as a conference paper at ICLR 2020
A.4 COMPARISON AGAINST OTHER SECOND- REPRESENTATIONS
The results in Sec 5.1 5.1 indicate that the proposed proxy second-order representations perform better than their true second-order counterpart. ResNet- 18 is the base original model used in these exper- iments. We provide plots for the proposed NR and FR representations when compared against its original first-order representation and kNN classifier with 4 different values of 5, 25, 50, 500 in Fig. 10 Note that both our representation and that of KNN are constructed on ft-1 representation space ResNet-18 18. The results on all 6 distortions are shown. While the first-order original represen- lation lags behind our proposed representation. it performs similar to, or in some cases outperforms, KNN methods. The parameter & does not create noticeable difference in the results. Even when was parameterized to high value of 5000, the overall results were less than the original.
Blur
S&P
Gaussian Noise
Distortion Levels OverExposure
Distortion Levels Motion Blur
Distortion Levels UnderExposure
Accuracy
Accuracy
Distortion Levels
Distortion Levels
Distortion Levels
First Order
Second Order RR
Second Order NR k
25
== k 500
Figure 10: Results of proposed representations compared against kNN classifier.
A.5 RESULTS ON CIFAR-IO-C
The distortion- wise and level-wise results of applying our second-order representation on original ResNet-18 18 was shown in Sec Here we show plots for level-wise results for all 19 distortions in
The original trained model is Re whose results are presented in blue. Its corresponding second-order representation is shown as NR in green. As mentioned in Sec CIFAR- 10-C does not provide access to either training distorted images nor their statistics and hence direct RR representation cannot be obtained. However, we calculate mean and standard deviation from a subset of test images and use those for normalization. Notice that doing converts our RR framework into a Domain Adaptation framework. The results of such a RR framework are provided in red in Fig 11
Fig 11
CIFAR-10- consists of a number of real-world complex scenarios. These complex scenarios range from simple gaussian blur and gaussian noise to real-life challenges like defocus blur, motion blur, rain, fog, and frost among others. The accuracy of the original ResNet-18 model trained with pristine images and tested on distorted images range between [0.9, 0.2). Distortions like contrast, Gaussian blur. Gaussian noise, impulse noise, and speckle noise traverse through this entire range and provide ideal means to compare proposed second-order representation against first-order representations. In each of these distortions except contrast, both the NR and RR representations outperforms the original by considerable margins. Contrast is distortion which even the human visual system cannot readily overcome (Geirhos et al., 2018). In 8 of the 19 distortions, the RR representation obtains results that are greater than 10% averaged over all levels. These include Gaussian blur, Gaussian noise, glass blur, motion blur, pixelate, shot noise, speckle noise, and zoom blur. The highest increase is 14.18% for shot noise. In 2 of the distortions, brightness and saturate, the results decrease by less than 0.6% averaged over all levels. Brightness and saturate both span representation spaces where the gradients multiple decision boundaries all very large and are not discriminative enough
0..)...2)
13

---

Under review as a conference paper at ICLR 2020
brightness
contrast
defocus blur
elastic transform
3.
Datertin
Distortion
fog
frost
gaussian blur
gaussian_nokee noise
1.0
13
33-
0-
35
65
05
02
022
E2.
02
C22
Distortion Levels
Distrtion leves
Dizontion Lovel
Distortonn Levds
glass blur
impulse noise
joeg_ compressposno
mation_ blur
C2
Distorton Level
Distrrin Lovds saturate
Distittonn Level shot noise
Distrtion Leves
pixelate
snow
3.2
3.2
Dittrrton Levels
Distortion
spatter
speckle_neeee noise
zoom blur
1.0
13
Original
30
RH
NR
02
Distotion Levess
Dittr ton levds
Disoition Levels
Figure 11: Results of proposed representations on 10-C images.
to differentiate between classes. The same explanation also extends to contrast, and fog where the increase is roughly 2% from the original.
Another way of characterizing results is by categorizing distortions by their local and global statistics. Distortions like saturate, brightness, contrast, fog, and frost change the low level statistics in the image domain. Neural networks are actively trained to ignore such changes so that their effects are not propagated after the first few layers. Hence any second-order representation created on the final fully connected layer is going to be analogous to its first-order counterpart. Therefore, the results of the proposed approach and the original model follow each other closely. This provides the motivation to consider creating second-order representations in the first few layers as well.
A.6 RESULTS ON HIGHER RESOLUTION IMAGES AND ALTERNATE APPLICATIONS
The results of the proposed representation when applied on larger resolution images of STL-10 dataset are provided in See 5.2
5.2 Here, we show the plots of the proposed representation against its first-order representation on 6 distortions in Fig [2] The increase in distortion-averagel level-averaged results is 2.56% in all except underexposure distortion. While this shows that the second-order representation outperforms its first-order counterpart, the increase in results is not substantial as in CIFAR-10 and CIFAR-10-C. This is because the original ResNet- 18 model was trained on only 5000 training images. In such under-trained scenario. the cumulative gradients of data point to all decision boundaries become large and not as discriminative as a well trained architecture. This experiment indicates limit to the performance increase of the second-order representation plug-in top of
14

---

Under review as a conference paper at ICLR 2020
first-order methods. However, it provides insight into using this representation control parameter during training procedures to provide an alternate and analogous view of under-trained, well-trained, and over-trained architectures.
Blur
S&P
Gaussian Noise
Orgginl
Ongg
Orggaa
Accuracy
96
Accuracy
.06
Accuracy
96
0.5
0.5
0.5
Distortion Levels OverExposure
Distortion Levels
Distortion Levels
Motion Blur
UnderExposure
Original
onlo
Orggmal
Accuracy
-66
Accuracy
.66
Accuracy
96
0.5
0.5
0.5
Distortion Levels
Distortion Levels
Distortion Levels
Figure 12: Results of proposed representations on higher resolution STL images.
15


Reviews:
[{"review_id": "SylTiBon9S", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose an approach to reduce the sensitivity of neural networks to visual distortions. To do so, they modify the representation of a data point within a network, using its relative position (to other points) in representation space rather than its absolute position  (which is measured as distance of a point to the decision boundary of each class). They then evaluate their approach on common corruptions from the CIFAR-C dataset.\n\nThe paper addresses an important problem, however I do not find the high-level motivation behind the proposed approach or the experimental results sufficiently convincing. \n\nIn particular, conceptually, it is completely unclear why the second order representations should be more resilient to visual distortions. Even in the sample illustration provided in Figure 2, it is evident that the clusters in this new representation space are not invariant to visual distortion, or even are significantly more invariant than the first-order representation. At a more fundamental level, given that the accuracy of the original network does drop, it is tautological that the distance of a point to the decision boundaries is decreasing under distortion (and thus is sensitive to it).\n\nExperimentally, my chief concerns are:\n1. When the authors evaluate the proposed second-order representations, they use networks with additional layers which do not seem to be present in the original baseline. Prior work [Hendrycks and Dietterich, 2019] has shown that model capacity has a marked influence on the corruption robustness of a network. Thus, it is unclear where the improvement here is coming solely from the additional capacity compared to the original model. The baseline network that the authors compare to should also include the additional layers.\n2. The results reported for some of the baselines seem inconsistent with prior work. \na) For example, the authors state that the results for the Hossain et al. [2018] baseline are similar to NR1, which is worse than the original network. However, Hossain et al. report an improvement for the same corruptions and the same dataset in their paper. Where is this inconsistency coming from? Do the authors train with the DCT filtering or is it only applied at test time? If it is the latter, it could explain why the authors fail to reproduce the baseline correctly.\nb) For the adversarially robust network baseline, why did the authors choose an FGSM adversary? FGSM is not typically used to train state-of-the-art robust models because of the existence of much stronger attacks (such as PGD). How was the eps used to train the network chosen? In prior work, Kang et al. [2019; arxiv:1908.0801] evaluate both L2 and Linf robust models and show improvements over the baseline for several common corruptions. This seems to suggest that the robust model baseline reported in this paper is not accurate/representative.\n3. Moreover, given that the representation size scales with number of classes, the proposed method should be evaluated on datasets with more classes such as CIFAR-100 or ImageNet. Improvements demonstrated in these settings with dimensionality reduction would be more convincing.\n\nOverall, my main reservations are: a) the lack of a conceptual justification for the proposed approach, and b) issues with the experimental evaluation, particularly in the reported baselines and how they seem to contradict prior work. Thus, I recommend rejection.\n\nOther comments:\n- The right-side of Figure 1 is essentially the same map flipped.\n- Why was the MSE loss chosen for J? Given that the network was presumably trained with cross-entropy, this choice seems somewhat arbitrary. Are the results consistent for cross entropy loss as well? The authors should include these results in the appendix.\n- Figure 4 is very hard to read---the authors should change the plotting style to make the results more legible. \n- For the results in Figure 5, the authors should once again compare to adding extra layers to the baseline networks as well.\n- In Table 1, why does the performance of NR improve when the features are based only on gradients for the last layer?", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SJeEI7t0YS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a method to represent data with its second-order information in the deep network, to improve its representation robustness. To obtain this representation, the authors used artificial adversarial examples to obtain the robust representations. Experimental results show that the new representation is more robust against real-world distortions. \n\nHere are more additional comments on the technical details:\n1) The authors proposed a new representation of a datapoint x in equation (2), using the concept of adversarial noise. However, it is still not clear to me how the expression in (2) is derived mathematically. For example, where does the absolute operator in (2) come from? It would be nice if the authors can provide some mathematical derivation or justification of the chose expression, in addition to the empirical evidence.\n\n2) The robust representation is proposed under a discriminative deep network, with predefined class information. Is it possible to extend the work to unsupervised settings (e.g., deep autoencoders), so that the representation robustness is not with respect to a specific definition of class labels?\n\n3) As the authors mentioned in Section 5.3, the computational cost is high for generating such robust representation. Is there any approximation procedures which can be taken to speed up the proposed robust feature representation?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1eERColcB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method to represent input points by their relative positions to decisions boundaries using an adversarial attack instead of the more usual output of a deep model (before the last linear classification layer). The motivation of the method is to obtain representations that are more robust to input distortions such as blur, noise, over-exposure, etc. More specifically, the representation obtained by a deep model before the last linear classification layer (f_L) is replaced by a concatenation of the perturbation needed for that representation to be misclassified by f_L. Using this new representation, they are able to obtain more robust classifiers against input perturbations. \n\nThis paper should be rejected for several reasons. First, the motivation of the paper is unclear; while being more robust to input perturbations is a desirable property of ML models, the reason to use these second-order representations is not justified. Second, the paper contains several imprecisions about the methodology, which make it complicated to follow. Finally, the presented results are not very convincing; it seems that most of the improvement obtained comes from deriving a larger representation of each sample and using a MLP to classify it instead of a simple linear layer.\n\nDetailed arguments:\n\nIt is not clear how r_i is obtained in section 3. This becomes even more confusing in section 5.3, when the features are instead generated as the gradient of the loss with respect to f_{L-1}. To the best of my understanding, r_i is supposed to be the distance between a point (before the final layer) to the decision boundary for class i. If we have x_k, the input which yields the closest point to the decision boundary of class i (as depicted in Fig 3.(b)), then we should have $r_i = f_{L-1}(x) - f_{L-1}(x_k)$. However, r_i is defined as $\\sum_{j=0}^{k-1} \\abs(\\nabla_{W_L^i} J(W, x_j, i)$. In this formula, the absolute value clearly changes the direction of the gradient. It would be appreciated to have a clear explanation for the reason of the absolute value.\n\nThe paper contains incorrect statements about standard adversarial attacks. FGSM is a one-step attack, and it does not make sense to do several iterations of it. A more powerful multi-step attack that can be considered is PGD.\n\nA lot of emphasis is put on the fact that the obtained representations might be too large to scale well to datasets where there are more classes than the ones considered in the article. However, there are no experiments to support this - the experiments in the paper do not go beyond 10 classes.\n\nAdditional comments:\n\nSome citations need to be fixed, removed or changed:\nSecond paragraph of the introduction: “[...] can discriminate between classes (Goodfellow et al., 2016)(Krizhevsky et al., 2012)(He et al., 2016).” Deep neural nets were not initially proposed in these papers, so it is somehow inappropriate to cite these papers here (except, maybe, Krizhevsky'12).\n\nThird paragraph of the introduction: “[...] the representations (fL−1) are linearly projected (Wang et al., 2019).” Same here.\n\nThe main figures to explain the idea are not really helping the argument: Figure 1. (b) is simply Fig 1. (a) rotated 180 degrees and re-scaled, but the information is essentially the same; maybe not the best example.\n\nFigure 2: the provided t-SNE projections do not show a clear advantage of the proposed method. A quantitative analysis would better support the argument.\n\nThe time measurement given in section 3 is not indicative: what are the 320s? An indication in terms of epochs or training iterations would be better.\n\nIt would be appreciated to have measures of standard deviations for the results especially in tables 1 and 2 since some results are very close and might depend mostly on random perturbations. Moreover, it is not mentioned for which datasets these tables are for.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a method to train a neural network that is robust to visual distortions of the input image. The reviewers agree that the paper lacks justification of the proposed method and experimental evidence of its performance.
Decision: Reject