Neural Networks Playing Dough: Investigating Deep Cognition With a Gradient-Based Adversarial Attack

1 INTRODUCTION . Szegedy et al . ( 2013 ) introduced the term “ ADversarial Examples , ” ( ADEs ) and with that they unveiled a terrifyingly effortless technique for fooling highly accurate neural networks into ludicrous misclassifications of obvious images . However , it took the field four years and a small sticker patch featuring some alien curves with metallic shine ( Brown et al. , 2017 ) before realizing how dire the situation could get , in a time when we unlock our phones ( Bryliuk & Starovoitov , 2002 ) and law enforcement identifies suspected criminals ( Garvie , 2016 ) with the very same technology . ADEs , typically , are copies of the images that a network can classify correctly with a high confidence , plus Adversarial Perturbations ( APs ) indiscernible to human eyes which misleads the network to classify them incorrectly with an even higher confidence . Inspirational works have contributed to understanding the nature of ADEs . The early speculations were about the high dimensionality of data ( Szegedy et al. , 2013 ) and then the linearity of learning in networks ( Goodfellow et al. , 2014 ) . But these explanations fall short of addressing the phenomenon known as adversarial transferability . This refers to the fact that elusiveness of ADEs created using one subset of a dataset with one network can also fool other models trained on different subsets . Later on , Jo & Bengio ( 2017 ) and Ilyas et al . ( 2019 ) proposed different accounts that consider , respectively , statistical regularities and non-robust features of datasets as the nature of ADEs . These suggestions offer grounds to understand the transferability . However , in the lack of an agreed-upon explanation , there is still spacious room to explore . The present study shifts the focus from the ADEs to the APs , and is based on the hypothesis that adversarial vulnerability is not a network weakness , but a consequence of the different ways of per- ceiving datasets by humans and by the networks . In that respect , the APs represent the networks ’ genuine perception of datasets . Therefore , studying APs could be a rare opportunity for understanding the learning process in networks , and it may also answer our lingering questions about the ADEs . To this end , I used a gradient-based attack to produce ADEs for the MNIST ( LeCun et al. , 2010 ) and Fashion-MNIST ( Xiao et al. , 2017 ) ( F-MNIST ) datases . Then I abstracted the APs , visualized them , and investigated if there is a way to explain them with the datasets ; not with some statistical or non-robust features , but entirely through high level semantics of datasets . 2 METHODS . 2.1 GENERATING ADVERSARIAL EXAMPLES . For generating ADEs , I performed a simple gradient-based adversarial attack , introduced by Szegedy et al . ( 2013 ) . In this method , a trained network runs gradient descent optimization on the input instead of the weights , and treats the weights as non-trainable parameters . The purpose is to alter the input image to what the network would classify with a ( commonly nonsensical ) target label . To keep the change at a minimum level and limited to the most decisive ones , I defined the loss function as the distance between the network prediction and the target label , omitting any further constraint on the final visual appearance of the input . For having the freedom to explore the models behavior , rather than the traditional “ target classification , ” I set a disjunctive stop condition for terminating the network optimization loop . The stop condition consists of : a maximum loss threshold , that is the loss function value calculated between the optimized input prediction and the target label ; a minimum level of confidence , for the network prediction of the optimized input ; and finally a maximum number of iterations counting the rounds of optimization of the input , initially designed to avoid the infinite loop . 2.2 ABSTRACTING THE ADVERSARIAL PERTURBATIONS . Regarding the network acuity with raw data compared to the weak visual perception in humans , APs are invisible to us . Therefore to embolden them , I took one direct and two indirect approaches . In the direct approach ( zero feed ) , I fed the networks with a zero array . When the input is zero , the modified array would be purely AP . Intended on verifying the reproducibility of the patterns obtained with the direct method , I also generated ADEs with normal noise as input . The drawback is that with the noise input , the ADEs will be noisy as well , and the signal-to-noise ratio is so poor that no patterns would be visible . Therefore , in one indirect approach ( noise feed ) , I simply subtracted the initial noise input from the final ADE to harvest the AP . And in the next indirect method ( Adversarial Average Image ( AAI ) ) , I aimed to accentuate the APs by computing cumulative adversarial Images . In doing so , I generated ADEs in batches of size , generally , 30 and added them up into one AAI , hoping for the noise to cancel itself out while the AP is getting augmented . To compare the APs produced with these three methods , I trained a Fully Connected feed-forward ( FC ) network [ 784 , 200 , 30 , 10 ] with a Stochastic Gradient Descent ( SGD ) optimizer on the MNIST dataset , up to the validation accuracy of 0.92 and loss value of 0.27 . After some trial and error , I found that the maximum iterations is the most efficient stop condition since the networks could engineer ADEs for some classes dramatically faster than the others and that delivers the APs still visually incomprehensible ; that is especially the case with the AAI method . As a result , for generating ADEs in all three methods , the only stop condition I applied was the maximum iterations , set to 1,000 . I then computed APs for all 10 classes , but did not normalize them . The reason being that the results are presented visually , using the pyplot.imshow method of the Matplotlib library ( Hunter , 2007 ) which by default normalizes images to their minimum and maximum values , and for the purpose of this study , it is contrast that matters and not the true values of the shades . 2.3 INVESTIGATING THE PATTERNS ’ ROBUSTNESS TO THE ABSTRACTION PARAMETERS . To investigate whether the AP patterns are independent of the abstraction methods , I extracted APs from several ADEs , with a range of different parameters for each approach . For generating ADEs , I used the aforementioned FC architecture and trained it with the same optimizer on the MNIST dataset ; this time up to a validation accuracy of 0.94 and the loss value of 0.19 . Then I computed APs for the class 5 , with the zero feed method with the stop condition set at three levels of confidence : 80 , 90 , and > 99 % ; with the noise feed method with the stop condition set at three values of loss : 1 , 1e-2 , and 1e-4 ; and with the AAI method three times , as well , with averaging over 10 , 100 , and 1,000 ADEs . 2.4 INVESTIGATING THE PATTERNS ’ ROBUSTNESS TO THE MODEL AND TRAINING PARAMETERS . Furthermore , to verify the persistence of the AP patterns over the network architecture and the training configuration , in addition to the mentioned FC , I also tested a Convolutional Neural Network ( CNN ) [ convolutional layers : 128 , 256 followed by dense layers : 512 , 256 , 10 ] . I trained the two models with six configurations , combined , this time on both MNIST and also F-MNIST datasets ( See Table 1 ) . It is to be noted that I did not try to train the networks to their fullest learning capacity . In fact , as the complexity of the architecture and training grew , I reduced the number of training FC SGD 0.95 0.19 0.85 0.42. epochs due to the fact that generating ADEs with highly accurate and complicated architectures became adversely expensive from a computational point of view . After training the networks , I generated APs only with the zero and noise feed methods , given that the AAI approach is considerably more time-consuming . For presenting the results , I hand-picked one of the two methods based on the clarity of the patterns and trying to include more diversity . 2.5 ABSTRACTING THE ADVERSARIAL PATTERNS DIRECTLY FROM THE DATASETS . The sensible topology of the AP patterns , paired with the fact that in the classification tasks , deep learning forms in categories , motivated me to investigate the hypothesis that these patterns represent a categorical perception of the classes in the dataset , in which a concept is introduced not only by the instances of what it is , but also equally importantly , through the instances of what it is not . To test this hypothesis , for each class of the MNIST and F-MNIST datasets , I theoretically divided the corresponding database into a positive set ( all samples belonging to that class ) and a negative set ( all samples belonging to the other classes ) . Next , I computed the average images for both sets and normalized them between 0 and 1 . And finally , I calculated the Positive-Negative Contrast image ( PNC ) by subtracting the negative average image from the positive one . 2.6 CONFIRMING THE RESULTS WITH TWO DUMMY DATASETS . To inspect the categorical learning hypothesis further , I created two dummy datasets each with four classes , each class with one sample ; utterly simplistic to secure a perfect learning by networks . The first one is the tiling dataset with four classes of a square patch at each corner of the image , that together they tile the image space . For the tiling dataset , all PNCs precisely match their corresponding class ( see Fig . 5 ) . And the second one , the overlapping dataset , with the same four classes of the square patches , but with shifted positions in a way that , if put together , they overlap . Therefore , the PNCs differ from their corresponding class ( see Fig . 6 ) . I computed the PNCs and APs for these two datasets using an FC [ 36 , 200 , 50 , 4 ] trained with SGD up to the validation accuracy of 100 % and loss value of 5.8e-4 . 2.7 COMPUTATIONAL RESOURCE . I ran all the computations for this study on an NVIDIA GPU ( GeForce® GTX 1070 ) . On this system , generating an AAI with 30 ADEs , with a stop condition of maximum iteration set to 1,000 took 5 minutes on average . For an AP with either zero or noise method , with the same stop condition , the duration plummeted to only 5 seconds . 3 RESULTS . The three methods for obtaining APs produced almost identical patterns , all of which the network classified with the target labels and a remarkably high confidence ( see Fig . 1 ) . These patterns , as soon as they emerge , are indifferent to the parameters of the method used for generating them ( see Fig . 2 ) . Furthermore , the patterns persist over a range of network architectures and training configurations . However , as the complexity of the architectures and optimizers grow , the clarity of the patterns deteriorate . Even so , the less intelligible APs highlight some features of the same patterns , which would be easier to place in the context of the other perturbations for the same class ( see Fig . 3 & 4 ) . And finally , as Fig . 3 and 4 show the patterns approximate their corresponding PNCs , with a striking accuracy in simpler models and training plans . The experiment with the tiling and overlapping datasets back this finding ( see Fig . 5 & 6 ) .

Summary: The paper investigates the problem of the susceptibility of neural networks to adversarial attacks. It considers the hypothesis that "it is not the neural networks failing in learning that causes adversarial vulnerability, but their different perception of the presented data" (quoted verbatim). In other words (as far as I understand), the networks are like a computer which doesn't do what we want it do, but only what we tell it to do. To test it, the Author produces shows that what is considered in the manuscript to be adversarial perturbations are not random noise, but have some meaning and can be correlated with the training dataset. This is done using MNIST and F-MNIST datasets, and simple fully connected and convolutional networks. However, the attempt to reproduce the results on ImageNet using ResNet-50 and mobileNetV2 models failed.