Neural Networks Playing Dough: Investigating Deep Cognition With a Gradient-Based Adversarial Attack
Authors: Sahar Niknam
Venue: ICLR 2022 Submitted (2022)
Keywords: Deep Learning
Adversarial Perturbation
Adversarial Example
Categorical Learning

Abstract: Discovering adversarial examples has shaken our trust in the reliability of deep learning. Even though brilliant works have been devoted to understanding and fixing this vulnerability, fundamental questions (e.g. the mysterious generalization of adversarial examples across models and training sets) remain unanswered. This paper tests the hypothesis that it is not the neural networks failing in learning that causes adversarial vulnerability, but their different perception of the presented data. And therefore, adversarial examples should be semantic-sensitive signals which can provide us with an exceptional opening to understanding neural network learning. To investigate this hypothesis, I performed a gradient-based attack on fully connected feed-forward and convolutional neural networks, instructing them to minimally evolve controlled inputs into adversarial examples for all the classes of the MNIST and Fashion-MNIST datasets. Then I abstracted adversarial perturbations from these examples. The perturbations unveiled vivid and recurring visual structures, unique to each class and persistent over parameters of abstraction methods, model architectures, and training configurations. Furthermore, these patterns proved to be explainable and derivable from the corresponding dataset. This finding explains the generalizability of adversarial examples by, semantically, tying them to the datasets. In conclusion, this experiment not only resists interpretation of adversarial examples as deep learning failure but on the contrary, demystifies them in the form of supporting evidence for the authentic learning capacity of neural networks.

Under review as a conference paper at ICLR 2022
NEURAL NETWORKS PLAYING DOUGH: INVESTIGATING DEEP COGNITION WITH A SRADIENT-BASED ADVERSARIAL ATTACK
Anonymous authors
Paper under double-blind review
ABSTRACT
Discovering adversarial examples has shaken our trust in the reliability of deep learning. Even though brilliant works have been devoted to understanding and fixing this vulnerability, fundamental questions (e.g. adversarial transferability) remain unanswered. This paper tests the hypothesis that it is not the neural net- works failing in learning that causes adversarial vulnerability, but their different perception of the presented data. And therefore, adversarial examples should be cemantic-sensitivv signals which can provide us with an exceptional opening to understanding the networks' learning. To investigate this hypothesis, performed a gradient-based attack on fully connected feed-forward and convolutional neural networks, instructing them minimally evolve controlled inputs into adversarial examples for all the classes of the MNIST and Mushion-MNIST datasets. Then abstracted adversarial perturbations from these examples. The perturbations un- veiled vivid and recurring visual structures, unique to each class and persistent over parameters of abstraction methods, model architectures, and training con- figurations. Furthermore, these patterns proved to be explainable and derivable from the corresponding dataset. This finding explains the generalizability of ad- versarial examples by, semantically, tying them to the datasets. In conclusion, this experiment not only resists interpretation of adversarial examples as deep learning failure but on the contrary, demystifies them in the form of supporting evidence for the authentic learning capacity of networks.
INTRODUCTION
Szegedy et al. 2013) introduced the term "ADversarial Examples," (ADEs) and with that they un- veiled a terrifyingly effortless technique for fooling highly accurate neural networks into ludicrous misclassifications of obvious images. However, it took the field four years and small sticker patch featuring some alien curves with metallic shine (Brown et al., 2017) before realizing how dire the situation could get, in time when we unlock our phones (Bryliuk & Starovoitov 2002) and law enforcement identifies suspected criminals (Garvie 2016) with the very same technology.
ADEs, typically, are copies of the images that network can classify correctly with high confi- dence, plus Adversarial Perturbations (APs) indiscernible to human eyes which misleads the network classify them incorrectly with even higher confidence. Inspirational works have contributed to understanding the nature of ADEs. The early speculations were about the high dimensionality of data (Szegedy al. 2013) and then the linearity of learning in networks (Goodfellow al.) 2014). But these explanations fall short of addressing the phenomenon known as adversarial transferabil- ity. This refers to the fact that elusiveness of ADES created using one subset of dataset with one network can also fool other models trained different subsets. Later on, Jo & Bengio (2017) and Ilya et al. (2019) proposed different accounts that consider, respectively, statistical regularities and non-robust features of datasets as the nature of ADEs. These suggestions offer grounds to under- stand the transferability. However, in the lack of an agreed-upon explanation, there is still spacious room to explore.
The present study shifts the focus from the ADEs to the APs, and is based on the hypothesis that adversarial vulnerability is not network weakness, but consequence of the different ways of per-

---

Under review as a conference paper at ICLR 2022
ceiving datasets by humans and by the networks. In that respect, the APs represent the networks' genuine perception of datasets. Therefore, studying APs could be a rare opportunity for understand- ing the learning process in networks, and it may also answer lingering questions about the ADEs. To this end, used a gradient-based attack to produce ADEs for the MNIST (LeCun et al. 2010) and Fashion-MNIST (Xiao al. (017) (F-MNIST) datases. Then I abstracted the APs, visualized them, and investigated if there is way to explain them with the datasets; not with some statistical or non-robust features, but entirely through high level semantics of datasets.
Class 0
Class
Class
Class 3
Class 4
Class 5
Class
Class
Class 8
Class 9
Noise Feed
AAI
Figure 1: Adversarial Patterns for the MNIST dataset, generated with three methods.
2 METHODS
2.1
GENERATING ADVERSARIAL EXAMPLES
For generating ADEs, I performed simple gradient-based adversarial attack, introduced by Szegedy et al. (013). In this method, trained network runs gradient descent optimization on the input instead of the weights, and treats the weights as non-trainable parameters. The purpose to alter the input image to what the network would classify with (commonly nonsensical) tar- get label. To keep the change at minimum level and limited to the most decisive ones, defined the loss function as the distance between the network prediction and the target label, omitting any further constraint on the final visual appearance of the input. For having the freedom to explore the models behavior, rather than the traditional "target classification." I set disjunctive stop condition for terminating the network optimization loop. The stop condition consists of: maximum loss threshold, that is the loss function value calculated between the optimized input prediction and the target label; minimum level of confidence, for the network prediction of the optimized input; and finally a maximum number of iterations counting the rounds of optimization of the input, initially designed to avoid the infinite loop.
2.2 ABSTRACTING THE ADVERSARIAL PERTURBATIONS
Regarding the network acuity with raw data compared the weak visual perception in humans, APs are invisible to us. Therefore to embolden them, I took one direct and two indirect approaches. In the direct approach (zero feed), I fed the networks with zero array. When the input zero, the modified array would be purely AP. Intended on verifying the reproducibility of the patterns obtained with the direct method, I also generated ADEs with normal noise as input. The drawback that with the noise input, the ADES will be noisy as well, and the signal-to-noise ratio is poor that no patterns would be visible. Therefore, noise input from the final ADE to harvest the AP. And in the next indirect method (Adversarial Average Image (AAD), aimed to accentuate the APs by computing cumulative adversarial Images. In doing so, generated ADEs in batches of size, generally, 30 and added them up into AAI, hoping for the noise to cancel itself out while the AP is getting augmented.
one indirect approach (noise feed), simply subtracted the initial
To compare the APs produced with these three methods, trained Fully Connected feed-forward (FC) network [784, 200, 30, 10] with a Stochastic Gradient Descent (SGD) optimizer on the MNIST dataset, up to the validation accuracy of 0.92 and loss value of 0.27. After some trial and error, I
2

---

Under review as a conference paper at ICLR 2022
found that the maximum iterations is the most efficient stop condition since the networks could engineer ADEs for some classes dramatically faster than the others and that delivers the APs still visually incomprchensible; that is especially the case with the AAI method. As a result, for gen- erating ADEs all three methods, the only stop condition applied was the maximum iterations, to 1,000. I then computed APs for all 10 classes, but did not normalize them. The reason being that the results are presented visually, using the pyplot. imshow method of the Matplotlib library (Hunter 2007) which by default normalizes images their minimum and maximum values, and for the purpose of this study, it is contrast that matters and not the true values of the shades.
2.3 INVESTIGATING THE PATTERNS' ROBUSTNESS TO THE ABSTRACTION PARAMETERS
To investigate whether the AP patterns are independent of the abstraction methods, I extracted APs from several ADEs, with range of different parameters for each approach. For generating ADBs, I used the aforementioned FC architecture and trained it with the same optimizer on the MNIST dataset; this time up validation accuracy of 0.94 and the loss value of 0.19. Then I computed APs for the class 5, with the zero feed method with the stop condition set at three levels of confidence: three values of loss: 1, le-2, and le-4; and with the AAI method three times, as well, with averaging over 10, 100, and
80, 90, and >99%; with the noise feed method with the stop condition
1,000 ADEs.
a
b
C
Zero Feed
Noise Feed
AAI
Figure 2: APs for the class the MNIST dataset, generated with three methods. Zero Feed: with confidence a) 80%, b) 90%, 9).999% Noise Feed: loss value a) 1, b) 1e-2, c) le-4. AAI: with a) 10 ADEs, b) 100 ADEs, с) 1000 ADEs.
2.4 INVESTIGATING THE PATTERNS' ROBUSTNESS THE MODEL AND TRAINING PARAMETERS
Furthermore, to verify the persistence of the AP patterns over the network architecture and the train- ing configuration, in addition to the mentioned FC, I also tested Convolutional Neural Network (CNN) [convolutional layers: 128, 256 followed by dense layers: 512, 256, 10]. trained the two models with six configurations, combined, this time (See Table 1). is to be noted that did not try to train the networks to their fullest learning capac- ity. In fact, the complexity of the architecture and training grew, reduced the number of training
both MNIST and also F-MNIST datasets

---

Under review as a conference paper at ICLR 2022
Table 1: Trained networks for generating APs
MNIST
F-MNIST
Architecture
Optimizer
loss
acc.
loss
FC
SGD
0.95
0.19
0.85
0,42
Adam
0.98
0.07
0.88
0.33
CNN
SGD
0.93
0.24
0.84
0,45
Adam
0.99
0,05
0.92
0.24
with batch
SGD
0.99
0.03
0.92
0.22
normalization
Adam
0.98
0.06
0.92
0.25
epochs due the fact that generating ADEs with highly accurate and complicated architectures became adversely expensive from a computational point of view. After training the networks, gen- erated APs only with the zero and noise feed methods, given that the AAI approach is considerably more cone-consuming For presenting the results, hand- hand-picked picked one of the two methods based on clarity of the patterns and trying to include more diversity.
2.5 ABSTRACTING THE ADVERSARIAL PATTERNS DIRECTLY FROM THE DATASETS
The sensible topology of the AP patterns, paired with the fact that in the classification tasks, deep learning forms in categories, motivated me to investigate the hypothesis that these patterns represent categorical perception of the classes in the dataset, in which a concept is introduced not only by the instances of what it is, but also equally importantly, through the instances of what it not. To test this hypothesis, for each class of the MNIST and F-MNIST datasets, theoretically divided the corresponding database into positive set (all samples belonging to that class) and a negative set (all samples belonging to the other classes). Next, computed the average images for both and normalized them between 0 and 1. And finally. calculated the Positive-Negativ Contrast image (PNC) by subtracting the negative average image from the positive one.
2.6 CONFIRMING THE RESULTS WITH TWO DUMMY DATASETS
To inspect the categorical learning hypothesis further, I created two dummy datasets each with four classes, each class with one sample; utterly simplistic to secure a perfect learning by networks. The first one is the riling dataset with four classes of square patch each corner of the image, that gether they tile the image space. For the tiling dataset, all PNCs precisely match their corresponding four classes of the
class (see Fig. 5). And the second one, the overlapping dataset, with the square patches, but with shifted positions in way that, if put together, they overlap. Therefore, the PNCs differ from their corresponding class (see Fig. 6). computed the PNCs and APs for these two datasets using an FC [36, 200, 50, 4] trained with SGD up to the validation accuracy of 100% and loss value of 5.8e-4.
2.7 COMPUTATIONAL RESOURCE
I ran all the computations for this study NVIDIA GPU (GeForee GTX 1070). On this system, generating an AAI with 30 ADEs, with stop condition of maximum iteration set to 1,000 took minutes average. For AP with either zero noise method, with the same stop condition, the duration plummeted to only 5 seconds.
3 RESULTS
The three methods for obtaining APs produced almost identical patterns, all of which the network classified with the target labels and remarkably high confidence (see Fig. 1).
These patterns, as soon as they emerge, are indifferent to the parameters of the method used for generating them (see Fig. 2). Furthermore, the patterns persist over range of network architectures
4

---

Under review as a conference paper at ICLR 2022
Class 0
Class
Class
Class
Class
Class
Class 6
Class 7
Class
Class
PNCS
CNN
Figure 3: PNCs and APs for the MNIST dataset. Blank images indicate unsuccessful effort for abstracting perturbations. Asterisk (*) denotes cases with batch normalization.
and training configurations. However, as the complexity of the architectures and optimizers grow, the clarity of the patterns deteriorate. Even so, the less intelligible APs highlight some features of the same patterns, which would be easier to place in the context of the other perturbations for the same class (see Fig. & 4). And finally, as Fig. and 4 show the patterns approximate their corresponding PNCs, with striking accuracy in simpler models and training plans. The experiment with the tiling and overlapping datasets back this finding (see Fig. 5 & 6).
DISCUSSION
When children play with dough, they try to embody the concepts that they are learning day by day. And by observing their lively artifacts and the level of worked details, one can infer about the content and the quality of their learning. This study evolved around the idea that the same will apply to network, if tries to remold a controlled input into the concepts it learned from a dataset. That is, considering a gradient- based adversarial attack as a set of instructions asking network to generate something that it perceives as an instance of a target class, zero array or a random noise input would function as a lump of playing dough.
Previous studies (Erhan et al. 2009 Simonyan et al. 2013 Nguyen et al. 2015) have exercised powerful methods, like gradient ascent and evolutionary algorithms, for having visual insight into the content that a trained network learns from dataset. However, these methods, compared to adversarial attack practiced in this paper, either are intrusive unnecessarily coach the networks with some extra information about the target class. While all the information that the gradient- based attack provides the networks with, is the nominal value of the target class. Therefore, any changes made to the input are genuinely devised by the network. For that matter, the AP patterns could be the closest approximation network's perception and this study's 's results confirm this.

---

Under review as a conference paper at ICLR 2022
Class 0
Class
Class
Class
Class
Class
Class
Class ?
Class
Class
PNCA
000
8GD
SGD+
Figure 4: PNCs and APs for the F-MNIST dataset. Blank image indicates unsuccessful effort for abstracting perturbations. Asterisk (*) denotes cases with batch normalization.
4.1 NEURAL NETWORKS DO LEARN
The robustness of the patterns to the abstraction parameters tells whatever these patterns are, they are neither trivial byproducts of the methods, nor random noises varying from one ADE to another. Rather they must be so-to-speak cognition, emerged from the network-dataset dyad.
One step forward, the patterns' robustness to the network architectures and training configurations, along with the uniqueness of AP patterns each class, further validates the assumption that reliably take these patterns as the learned content by a trained network. On top of that, this finding justifies the adversarial transferability with explaining APs as responsive to the content of datasets on some level of semantics.
4.2 BUT THEY SEE DIFFERENTLY
Furthermore, inspecting the AP patterns revealed a remarkable resemblance between them and the PNC patterns that makes perfect sense regarding the categorical nature of learning in networks. This has two main implications. First, in sharp contrast with the Potemkin analogy (Goodfellow et al.) 2014), ADEs contain patterns which evidence of the networks" capacity to learn, and learn high level semantics. And the adversarial vulnerability is only a side effect of categorical learning. That is. when a network learns about concept, instead of exclusively focusing on the features of the concept itself, it equally relies on the characteristics of all other existing concepts in the dataset that are not the target concept. Therefore, when the negative set is too sparse (for example, nine digit-shapes in MNIST case) compared to the enormously spacious space that we define for the network (a 784-dimensional space MNIST case) the positive and negative sets fail to converge on matching concept. That in turn, causes the discrepancy of perception between the networks and humans; while we constantly forget that we have the privilege to learn in the context flourishing with notions and concepts. In fact, (Szegedy et al. (2013), in the very same paper that introduced ADEs for the first time, righteously although partially, addressed this issue by mentioning the high dimensionality of data possible reason behind the adversarial vulnerability.
world

---

Under review as a conference paper at ICLR 2022
In addition, my experiment with the dummy datasets supports this rationale. The two datasets, which have their minituristic space occupied by comparatively gigantic objects, are almost identical regarding the degree of simplicity. However, while generate ADEs for the overlapping dataset, the same attack goes nowhere but to replicate almost the perfect examples of target classes with the tiling dataset.
1
2
Classes
PNCs
APs
Figure 5: PNCs and APs for the tiling dataset.
1
2
Classes
PNCs
APs
Figure 6: PNCs and APs for the overlapping dataset.
The second implication of this finding is that on one hand we can estimate PNCs with the AP pat- terns. On the other hand. it is computationally possible to break down PNCs to the average arrays of all classes of the corresponding dataset. That makes pipeline from making queries with zero or noise feed blackbox model, the average vectors of the training dataset. Even though the pro-

---

Under review as a conference paper at ICLR 2022
posed methods in this paper not capable of delivering elaborate PNCs, the doors of opportunities (or misapplication, to be more accurate) are wide open.
4.3 LIMITATIONS
While the results of this study stand on their own, it is not trivial matter that only worked with the small-sized FC and CNN models trained on two simple grayscale datasets.
Both MNIST and F-MNIST datasets include samples with center-aligned shapes against white background which makes it easy to calculate average images and manifest PNCs. But whether more complicated datasets with color images that introduce, for example, instances of a dog all over the input area can yield some sorts of PNCs not, to validate this arguably need different strategy rather than the simple method used. Moreover, with increased complexity of the network architectures and their coose-to-perfect. accuracy (especially the simple datasets), becomes com- putationally costly and more time-consuming to generate ADEs or obtain lucid AP patterns with the particular method exercised in this study.
However, my primary efforts which failed to derive supportive results with the ResNet-50 (He et al,, 2016) and mobileNetV2 (Sandler al., 20188) models trained on the ImageNet dataset (Deng al. 2009 bruise my confidence in the existence of a shared learning strategy among all models and datasets.
REFERENCES
Tom B Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer. Adversarial patch. arXiv preprint arXiv: 1rXiv:1712.09665, 2017.
Dmitry Bryliuk and Valery Starovoitov. Access control by face recognition using neural networks. Institute of Engineering Cybernetics, Laboratory of Image Processing and Recognition, 4, 2002.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-25525555 255. leee, 2009.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. University of Montreal, 1341(3):1, 2009.
Clare Garvie. The perpetual line-up: Unregulated police face recognition in America. Georgetown Law, Center on Privacy & Technology, 2016.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
JD Hunter. Matplotlib: A 2d graphics environment. computing in science engineering 9, 3. 90-95. 2007.
Andrew Ilas,, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. arXiv preprint (t5i::9900.0111 2019.
Jason Jo and Yoshua Bengio. Measuring the tendency of cnns to learn surface statistical regularities. arXiv preprint arXiv: I.I 1561, 2017.
Yann LeCun, Corinna Cortes, and Chris Burges. Mnist handwritten digit database, 2010.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confi- dence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 427-436, 2015.
8

---

Under review as a conference paper at ICLR 2022
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510 4510-4520, 2018.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 2013.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist mnist: a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: arXiv: 100.00777077700001


Reviews:
[{"review_id": "qMEgs5AEbsp", "reviewer": "Reviewer_P1SQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The authors forgot to provide conclusions in many sections. For example, in Sec. 2.3, what is the conclusion there?\n\n2. The datasets are too simple. I would expect datasets like imagenet or maybe a simpler one - CIFAR-10 (The authors mentioned that they tested on Imagenet but failed. Does that mean the observation and conclusion are only limited to datesets with clear and robust features?)\n\n3. The work in its current version looks more like a report instead of a conference paper. I would suggest the author improve the paper organization by moving most of their implementing details to the appendix or a single section. \n\n4. The authors have found trackable patterns existing in adversarial examples. One natural question is that can we leverage these findings to improve model robustness?", "summary_of_the_paper": "In this paper, the authors first visualize adversarial patterns generated from three different methods - inputs with zero entries, noise inputs with subtracted perturbations, noise inputs in its average version. The authors then check if the patterns are attack method-agnostic and model-agnostic. Finally, the authors conclude that the patterns approximate their corresponding positive-negative contrast images.", "summary_of_the_review": "The authors observe some interesting and trackable patterns in adversarial examples. However, I believe the current version still has a lot of room for improvement.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "U_ScXOYJn7q", "reviewer": "Reviewer_9iri", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general, work is at a preliminary stage, and many of the ideas discussed in the paper have been explored in the past, albeit in different forms. For instance, the concept of abstracting out adversarial examples at a global level is the core idea of Universal Adversarial Perturbations and the frameworks which build on it. In general, viewing adversarial perturbations as a form of semantic signal that can help open up black-box DNNs [1, 2, 3] and understand their generalization properties have been explored a lot in the literature.\n\n1, Interpretable Explanations of Black Boxes by Meaningful Perturbation.\n\n2, An Empirical Study on the Relation between Network Interpretability and Adversarial Robustness.\n\n3, Regularized adversarial examples for model interpretability.\n\n4, Disentangling Adversarial Robustness and Generalization.", "summary_of_the_paper": "The paper hypothesises that the causes of adversarial vulnerabilities aren't due to the failures of Deep Neural Networks (DNNs) but because of how they perceive data. Hence adversarial examples should be thought of as a signal which opens up black-box DNNs. The author also surmises that the adversarial examples can be abstracted to a coarser level, and these abstractions can serve as a summary of the dataset.", "summary_of_the_review": "In general, I believe that the work isn't yet mature to be accepted as a conference paper, and I would encourage the author to submit it to a suitable workshop. That said, I do believe the questions considered in the paper hold a lot of promise. I would also encourage the author to support the claims by a strong set of experiments. The author currently experiments on MNIST & Fashion MNIST, which isn't sufficient to support (or validate) the hypothesis proposed in the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "4t-CDKwUuO", "reviewer": "Reviewer_feru", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper shows that simple ANN models trained on MNIST and F-MNIST simply memorise the average element for each class. However, this is not the case for larger models trained on larger datasets. Some interesting conclusions can be drawn from this about the usefulness of MNIST and F-MNIST (and similar small, simple datasets) for Deep Learning research.\n2. the Author posed an interesting hypothesis (if I understood it correctly) and I would encourage them to investigate it further.\n\nWeaknesses:\n1. The paper attempts to construct adversarial examples by applying a gradient descent attack to either a blank image or random noise. However, adversarial examples are constructed by taking a correctly classified test example as a starting point and adding a small perturbation causing the model to err. That is their accepted definition in literature. The Author should not call the presented image manipulation an \"adversarial attack\". It would have been different if the patterns obtained the method described in the paper were applied to correctly classified test examples and managed to fool the model, but the paper does not include such an experiment. I suggest performing it since it's relatively cheap and could produce interesting results.\n2. In any case, the method described in the paper cannot explain ALL forms of adversarial attacks, since image classifiers are also susceptible to small translations or rescalings of inputs (Azulay and Weiss, 2018; https://arxiv.org/abs/1805.12177).\n3. If we abandon the adversarial example point of view, the work presented in the paper can be understood as analysing what the neural network has learnt for each class. However, the technique employed is very similar to a saliency map, a known technique for analysing exactly this problem (see e.g. Simonyan, Vedaldi and Zisserman, 2013; https://arxiv.org/abs/1312.6034). The contribution here is not original enough to be published as a new method. It is absolutely fine to use a known method, but it should be acknowledged in the paper.\n4. The prose of the manuscript is in my opinion too flowery, verbose and imprecise for a research paper. I understand style is personal, but there are some conventions in writing scientific literature which one should not deviate from too much or without a good reason. The research hypothesis should be stated more precisely. Phrases like \"unveiled a terrifyingly effortless technique for fooling highly accurate neural networks into ludicrous misclassifications of obvious image\" should be avoided or used very sparingly. Terms like \"deep cognition\" should be avoided entirely, unless one is writing a paper about cognition.\n5. In Sec. 2.1, the loss function is defined as \"the distance between the network prediction and the target label\". But labels are discrete, so how could one run gradient descent on this loss function? It would help if the paper provided a formula for the loss function.", "summary_of_the_paper": "The paper investigates the problem of the susceptibility of neural networks to adversarial attacks. It considers the hypothesis that \"it is not the neural networks failing in learning that causes adversarial vulnerability, but their different perception of the presented data\" (quoted verbatim). In other words (as far as I understand), the networks are like a computer which doesn't do what we want it do, but only what we tell it to do. To test it, the Author produces shows that what is considered in the manuscript to be adversarial perturbations are not random noise, but have some meaning and can be correlated with the training dataset. This is done using MNIST and F-MNIST datasets, and simple fully connected and convolutional networks. However, the attempt to reproduce the results on ImageNet using ResNet-50 and mobileNetV2 models failed.", "summary_of_the_review": "The results of the paper are not original enough to warrant publication. It also misunderstands what are adversarial examples. Hence, I strongly recommend rejection. I also did not see enough evidence that the Author is familiar with existing literature about the topic - the reference section is quite short, which is usually not a good sign.\n\nThe paper reports an interesting result: small ANN models trained on simple image datasets can be shown to simply memorise the average element for each class. This is suggests that perhaps we should stop using MNIST and F-MNIST datasets to analyse how neural networks learn.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2RpC9IzgbuD", "reviewer": "Reviewer_8a9h", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall this is a diligent submission and I commend the author for thinking outside the box, however I do think that the presented evidence, while interesting, is not substantial enough to warrant the conclusions drawn from it.\n\nMy main concern is that this work, while it highlights the high dimensionality of the data, underestimates the high dimensionality of the neural network in terms of its parameters. Paraphrasing this submission from a very broad vantage point, to me, it appears to show that first order optimization can (sometimes) find data points with minimal loss that are also semantically meaningful (i.e. they contain low-frequency structures and not just noise). Different initializations can further (sometimes) find semantically similar data points. This is interesting, but due to the high dimensionality of the investigated neural networks not too suprising. These data points are only a small subset of all possible data points with low loss; that neural networks assign low loss to these data points is not a strong argument toward their perception as many other (adversarial) examples exist that are also \"perceived\" similarly, meaning they also have a low loss. The author investigates only gradient descent (with a fixed step size that appears to be 1.0 from the code(?)), but other algorithms could find other data points with minimal loss and other characteristics - for example using signed gradient descent, a variety of step sizes, loss function heuristics, and other initialization schemes. On the other hand, these data points could be made more semantically meaningful by initializing them with examples from the training set, the neural network is already trained to assign low loss to these examples, no further optimization is necessary, yet they can also be characterized as solutions of the given optimization problem. In light of these possible variations, the actual patterns seem arbitrary and an artefact of the optimization scheme used to find them.\n\nI further do not think that the investigated data points are good examples of \"adversarial perturbations\" (and I have eschewed calling them such above). The investigated data points with minimal loss are found by unconstrained optimization in the image space. This makes them qualitatively different from \"usual\" adversarial examples! Usual adversarial examples are constrained in some metric in image space. For example, using the most common example, a small $l^\\infty$ bound around some data point. From the size of this bound it is clear that there exists *no* semantically meaningful example within this bound that should be classified as the adversarial label. As such the perturbation really is adversarial and outside of context that is meaningful to humans. However, the investigated data points are unconstrained and, as discussed, a multitude of meaningful examples are also possible solutions, such as the entire training set.\n\nThe submission already finds that \"as the complexity of the architectures and optimizers grow,\nthe clarity of the patterns deteriorate\", and I would argue that this is a result of the fact that a variety of possible patterns already exist in any case and as the dimensionality of the model increases, the chosen optimization schemes will result in disparate patterns even quicker than before. \n\nMinor Comments:\n* \"APs are invisible to us\" - but this does not apply to the datapoints found in this investigation. Rather it is possible to find data points which have bother low loss and are \"invisible\" in some metric.\n* All patterns in this work are further found for MNIST/FMNIST which are centered and sufficiently simple datasets without background. Would the presented method recover semantic content for CIFAR10?\n* Figures 3 & 4 to me really show that the observed pattern is too specific to the proposed process to find it for it to be in some sense universal to a given neural network.\n* The experiment with tiling do not really disprove this. The tiling task is easy enough that a very simple (and even linear?) classifier can be learned by the neural network to match the observed behavior. That this classifier is then more robust, and thus leads to more meaningful perturbations, is a statement about the inherent robustness of the decision boundary of such a model and, to me, not about qualities of these datapoints.\n* Some concrete examples of non-semantic data points that lead to arbitrary classifications are universal adversarial perturbations (as e.g. in https://arxiv.org/pdf/1610.08401.pdf and follow-up work), these perturbations are much smaller than the unbounded example investigated here and essentially devoid of semantic content.\n* The \"distance\" in 2.1 between prediction and target is CrossEntropy (from looking at the code). This would be good to mention.", "summary_of_the_paper": "The submission \"Neural Networks Playing Dough: Investigating Deep Cognition With a Gradient-Based Adversarial Attack\" investigates adversarial perturbations empirically. For small FCNs and CNNs on MNIST/FMNIST, the author finds data points with minimal target loss to for each class by gradient descent based on different initializations. These data points can contain semantic meaning and it is suggested that these patterns closely approximate the perception of these neural networks. ", "summary_of_the_review": "The concluding statements of this submission are that \"the AP patterns could be the closest approximation to a network’s perception and this study’s results confirm this\" and that the proposed tools \"validate[] the assumption that we can reliably take these patterns as the learned content by a trained network\", but the existence of these data points is not sufficient to claim that they universally represent the learned content of a trained network. Adversarial example literature has shown that adversarial examples are so ubiquituous that they can be found both with and without semantic features, in almost any desired shape and constraint.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work provides an empirical investigation on the adversarial attacking problem in deep neural networks. While it contains some interesting ideas, the work is still in the preliminary stage, lacking substantial support for the main points. Many of the ideas discussed in the paper have been explored in the past and hence more discussions on previous works would be needed. We encourage the authors to keep improving the work for future submission.
Decision: Reject