White Noise Analysis of Neural Networks
Authors: Ali Borji
Sikun Lin
Venue:  (2020)
Keywords: Classification images
spike triggered analysis
deep learning
network visualization
adversarial attack
adversarial defense
microstimulation
computational neuroscience

Abstract: A white noise analysis of modern deep neural networks is presented to unveil
their biases at the whole network level or the single neuron level. Our analysis is
based on two popular and related methods in psychophysics and neurophysiology
namely classification images and spike triggered analysis. These methods have
been widely used to understand the underlying mechanisms of sensory systems
in humans and monkeys. We leverage them to investigate the inherent biases of
deep neural networks and to obtain a first-order approximation of their functionality.
We emphasize on CNNs since they are currently the state of the art methods
in computer vision and are a decent model of human visual processing. In
addition, we study multi-layer perceptrons, logistic regression, and recurrent neural
networks. Experiments over four classic datasets, MNIST, Fashion-MNIST,
CIFAR-10, and ImageNet, show that the computed bias maps resemble the target
classes and when used for classification lead to an over two-fold performance than
the chance level. Further, we show that classification images can be used to attack
a black-box classifier and to detect adversarial patch attacks. Finally, we utilize
spike triggered averaging to derive the filters of CNNs and explore how the behavior
of a network changes when neurons in different layers are modulated. Our
effort illustrates a successful example of borrowing from neurosciences to study
ANNs and highlights the importance of cross-fertilization and synergy across machine
learning, deep learning, and computational neuroscience.

Published as a conference paper at ICLR 2020
WHITE NOISE ANALYSIS OF NEURAL NETWORKS
Ali Borji & Sikun Lin
UUniversity of California, Santa Barbara, CA al alibor gmail. sikun@ucsb. edu
ABSTRACT
A white noise analysis of modern deep neural networks is presented to unveil their biases at the whole network level or the single neuron level. Our analysis is based on two popular and related methods in psychophysics and neurophysiology namely classification images and spike triggered analysis. These methods have been widely used to understand the underlying mechanisms of sensory systems in humans and monkeys. We leverage them to investigate the inherent biases of deep neural networks and to obtain first-order approximation of their function- ality. We emphasize on CNNs since they are currently the state of the art meth- ods in computer vision and are a decent model of human visual processing. In addition, we study multi-layer perceptrons, logistic regression, and recurrent neu- ral networks. Experiments over four classic datasets, MNIST, Fashion-MNIST, CIFAR-10, and ImageNet, show that the computed bias maps resemble the target classes and when used for classification lead an over two-fold performance than the chance level. Further, we show that classification images can be used to attack black-box classifier and to detect adversarial patch attacks. Finally, we utilize spike triggered averaging to derive the filters of CNNs and explore how the be- havior of network changes when neurons in different layers are modulated. Our effort illustrates a successful example of borrowing from neurosciences to study ANNs and highlights the importance of crossfeeriilzzation and synergy across ma- chine learning, deep learning, and computational neurosciencd
INTRODUCTION
Any vision system, biological or artificial, has its own biases. These biases emanate from different sources. Two common sources include a) the environment and the data which the system has been trained, and b) system constraints (e.g., hypothesis class, model parameters). Exploring these biases is important from at least two perspectives. First, it allows to better understand system (e.g., explain and interpret its decisions). Second, it helps reveal system vulnerabilities and make it more robust against adversarial perturbations and attacks.
In this paper, we recruit two popular methods from computational neuroscience to study the in- herent biases in deep neural networks. The first one, called classification images technique, was introduced into visual psychophysics by Abbmaaa Jr (199) as new experimental tool. It has been used examine visual processing and to understand vision across a variety of tasks including sim- ple detection tasks, visual search, and object recognition. It has also been applied to the auditory domain. See Murray| (20TT) for review of the topic. The second method, known as spike triggered analysis (Marmarelis, 2012), is often used to discover the best stimulus to which neuron responds (e.g., oriented bars). These methods are appealing for our purpose since a) they are general and can be applied to study any black box system (so long it emits response to an input stimulus) and b) make a modest number of assumptions. From a system identification point of view, they provide a first-order approximation of a complex system such as the brain
artificial neural network.
By feeding white noise stimuli to a classifier and averaging the ones that are categorized into a particular class, we obtain an estimate of the templates it uses for classification. Unlike classification images experiments in human psychophysics, where running a large number of trials is impractical, artificial systems can often be tested against large number of inputs. While still a constraint, we will discuss how such problems can be mitigated (e.g., by generating stimuli containing faint structures). Over four datasets, MNIST (LeCun et al. 1998,,
(Xiao et al. 2017
Work done during internship at MarkablcAI.
Code is available https://gi http:///tttubbcoooo hat.mm/aaabbrrroo boroi
http://allloorii//teeeeeeeee wwthi/hhteeeit teNot seAna M wws...iiiiii.

---

Published as a conference paper at ICLR 2020
A
pead y
pest T
prod gf
(+ merr: 1-7
mobe
past.gr
pred Eighe
merc 5-6
7
7
5
7
5
6
7
7
5
6
1-0
Figure 1: Illustration of the classification images concept. A) Two sample digits well as their linear combination with different magnitudes of white noise (i.e., ys + (1- Eq. 3). B) Average correct and incorrect prediction maps of binary CNN (Fig. [0] in suppiement) trained to separate digits and The fifth column shows the difference between average of stimuli predicted as and average of stimuli predicted 7. The column marked with "*" is similar to the fifth column but computation done only over noise patterns (and not the augmented stimuli), hence "classification images" (i.e., (n" n" (n' )) Eq. 1). See supp. Fig- 12 for more illustrations. These templates can be used to classify a digit as to 1 or 7. Yellow (blue) color corresponds to regions with
positive (negative) correlation with the response 1. C) Same but using 5 vs. 6 CNN.
10 (Krizhevsky et al., 2009), and ImageNet Deng et al. (2009). employ classification images to discover implicit blases a network, utilize those biases to influence network decisions, be used to
and detect adversarial perturbations. We also show how spike triggered averaging identify and visualize filters in different layers of a CNN. Finally, in a less directly related analysis classification images, we demonstrate how decisions of CNN are influenced by varying the signal to noise ratio (akin to microstimulation experiments in monkey electrophysiology or priming experiments psychophysics). We find that CNNs behave in a similar fashion their biological counterparts and their responses can be characterized by psychometric function. This may give insights regarding top-down attention and feedback mechanisms in CNNs (See BBojj Ittil (2022). )
RELATED WORKS AND CONCEPTS
Our work relates a large body research attempting to understand, visualize, and interpret deep neural networks. These networks have been able to achieve impressive performance on variety of challenging vision and learning tasks (e.g., Krizhevsky al. (2012); He et (2016). However, they are still not well understood, have started to saturate in performance (Recht et al.) 2019). are brittle" and continue trail humans in accuracy and generalization. This calls for tighter con- fluence between machine learning, computer vision, and neuroscience. In this regard, the proposed tools here are complementary to the existing ones in the deep learning toolbox.
Perhaps, the closest work to ours is Vondrick et al. (2015) where they attempted to learn biases the human visual system and transfer those biases into object recognition systems. Some other works (e.g.. Fong et al. (2018) have also used human data (e.g., fMRI, cell recording) to improve the accuracy of classifiers, but have not utilized classification images. Bashivan et al. (2019) used is activation maximization to iteratively change the pixel values in the direction of the gradient to maximize the firing rate of V4 n Unlike these works, here we strive to inspect the biases in classifiers, in particular, neural networks, improve their interpretability and robustness.
2.1 CLASSIFICATION IMAGES
In typical binary classification image experiment, on each trial, a signal s E Rd and noise image E & are summed to produce the stimulus n. The observer is supposed to decide which of the two categories the stimulus belongs to. Classification image is then calculated as:
c=(n n22) (n 2!
(1)
where R is the average of noise patterns in stimulus stimunus-response class of trials. For example, 12 is the average of the noise patterns over all trials where the stimulus contained signal but
"Current deep neural networks can be easily fooled by subtle image alterations ways that are impercepti-
ble to humans; a.k.a adversarial examples (Szegedy et al. 2013 Goodfellow et al. 2014b).
See
ew. ew.net/Forum2ide
TebhnkYIDH for discussion this.
2

---

Published as a conference paper at ICLR 2020
the observer responded 2. R is approximation of the template that the observer uses to discriminate between the two stimulus classes. The intuition behind the classification images is that the noise patterns in some trials have features similar to one of the signals, thus biasing the observer to choose that signal. By computing the average over many trials a pattern may emerge. с can also be interpreted the correlation map between stimulus and response:
corr [n, r] En E (r(r [[rr)
Tr
where on is the pixel-wise standard deviation of the noise n and o, is the standard deviation of response r. High positive correlations occur at spatial locations that strongly influence the observer's responses. Conversely, very low (close to zero) correlations oceur at locations that have no influence on the observer's responses. Assuming zero-mean noise and an unbiased observer, Eq. 2 reduces to Ccorr = n n=1, where n is the average of the noise patterns over all trials where the observer gave a response (See Murray (2011) for details). Thus, Ceorr is the average of the noise patterss over all trials where the observer responded r = 2, minus the average over all trials where the observer responded r regardless of which signal was presented.
We have illustrated the classification images concept in Fig. [ with a binary classifier trained to separate two digits. The stimulus is linear combination of noise plus signal as follows:
y s (1 7) x n; 7 E [0, 1]
(3)
The computed templates for different y values using about million trials, highlight regions that are correlated with one of the digits (here 1 vs. or 5 vs. 6). The template fades away with increasing noise (e.g.. 7 = 0) but it still resembles the template in the low-noise condition (i.e., 7 = 0.3).
2.2 SPIKE TRIGGERED ANALYSIS
The spike-triggered analysis, also known "reverse correlation" or "white-noise analysis", is tool for characterizing the response properties of a neuron using the spikes emitted in response to time-varying stimulus. It includes two methods: spike-triggeree averaging (STA) and spike- triggered covariance (STC). They provide an estimate of a neuron's 's linear receptive field and are useful techniques for the analysis of electrophysioloiical data. In the visual system, these methods have been used to characterize retinal ganglion cells (Meister et al.) 1994] Sakai & Naka 1987), lateral geniculate neurons (Reid & Alonso 1995), and simple cells in the primary visual cortex (DAngelis et al. 1993 Jones & Palmer 1987). See Schwartz et al. (2006) for a review.
STA is the average stimulus preceding a spike. It provides an unbiased estimate of neuron's receptive field only if the stimulus distribution is spherically symmetric (c.g., Gaussian white noise). STC can be used to identify response. It identífies the stimulus features affecting neuron's 's response via an sigen-decoopositiin of the spike-triggered covariance matrix (Sandler & Marmarelis 2015 Park & Pillow 2011
feature space in which neuron computes its
Let x E Rdd denote spatio-temporal stimulus vector affecting neuron's scalar spike response y single time bin. The main goal of neural characterization is to find θ, a low-dimensional projection matrix such that O x captures the neuron's 's dependence on the stimulus x. The STA and the STC matrix are the empirical first and second moments of the spike-triggered stimulus-response pairs {x, (y: respectively. They are defined as:
N
N Σ vx ())= - ,, Rep ==1
1
STA:
Σ y:x;, and STC: A Nsp 1=1
(4)
where Nsp = Yi is the number of spikes and N the total number of time bins. The traditional
spike triggered analysis gives an estimate for the basis θ consisting of: (1) p. if it is significantly different from zero, and (2) the eigenvectors of A corresponding to those eigenvalues that are signif- icantly different from eigenvalues of the prior stimulus covariance & = E[XX") When a stimulus not white noise (i.c., correlated in space or time), whitened STA can be written as:
N
STA
T X
(5)
Nsp
WWe use classification images, bias map, template, and average noise pattern, interchangeably. Please do confuse this hias with the hias terms in neural networks.
3

---

Published as a conference paper at ICLR 2020
5-0
2-2.8002N
3-3
4-A.Nom
True 3ibe
CNN
5.R JoPn
6.K.NMW
A- PPPOIM
9-R.19M1
RECOOMEEIC
B
mago
Weight
Log Reet
M-- paintn
2- ni
Trun
Predicted label
Preducted Isbel
Figure 2: A) Classification images of a CNN trained on MNIST (with 99.2% test accuracy). Image titles show ground truth, predicted class for the bias map, and the frequency of the noise patterns classified as that digit. B) Classification images of logistic regression over MNIST with 92.46% test accuracy. C) Confusion matrices of four classifiers (CNN and log. reg. biases, mean digit image, and log. reg. weights). The classification was done via template matching using dot product.
where is matrix whose ith row is the stimulus vector x"' and y denotes column vector whose ith element is yı- The whitened STA is equivalent to linear least-squares regression of the stimulus against the spike train.
Classification images and spike triggered analysis are related in the sense that both estimate the terms of Wiener/Volterra expansion which the mapping from the stimuli to the firing rate is described using a low-order polynomial (Marmarelis) 2012). See Dayan et al. (200T) for a discussion on this. Here, we focus on STA and leave STC to future works.
3 APPLICATIONS
We present four use cases of classification images and STA to examine neural networks, with a focus on CNNs since they are a decent model of human visual processing and are state of the art computer vision models. Our approach, however, is general and can be applied to any classifier. In particular, it is most useful when dealing with black-box methods where choices are limited.
3.1 UNDERSTANDING AND VISUALIZING CLASSIFIER BIASES
We trained a CNN with 2 conv layers, 2 pooling layers, and one fully connected layer (see supple- ment Fig. 10p on the MNIST dataset. This CNN achieves 99, 2% test accuracy. We then generated million 28 x 28 white noise images and fed them to the CNN. The average noise map for each digit class is shown in Fig. A These biases/templates illustrate the regions that are important for classification. Surprisingly, for some digits (0 to 7), it is very easy to tell which digit the bias map represent+]] We notice that most of the noise patterns has a lot of structure in common with other digits. Feeding the average noise maps back to CNN, they are classified correctly, except 8 which is classified 2 (see image captions in Fig. 2
classified 8, perhaps because this digit
Classification images of the CNN over MNIST perceptually make sense to humans. This, however, does not necessarily hold across all classifiers and datasets. For example, classification images of logistic regression classifier on MNIST, shown in Fig. 2B, do not resemble digits (the same happens to MLP and RNN; see supplement Fig. [3). This implies that perhaps CNNs extract features the same way the human visual system does, thus share similar mechanisms and biases with humans. Classification images over the CIFAR- 10 dataset, derived using million 32 x 32 RGB noise pat- terns, are shown in Fig.3M. In contrast to MNIST and Fashion-MNIST (Fig. 5). classification images on CIFAR-10 10 (using CNNs) do not resemble target classes. One possible reason might be because images are more calibrated and aligned over the former two datasets than CIFAR- 10 images.
"Weighting the noise patterns by their classification confidence or only considering the ones with classifica- tion confidence above a threshold did result in significantly different classification images.
4

---

Published as a conference paper at ICLR 2020
realidath
Figure 3: A) Mean train- ing images (top) and mean white noise pattern/bias maps (bottom) across CIFAR-10 classes. Image titles show ground truth class and predic- tion of the bias map, respec- tively. B) Confusion matrices using mean images (top) and bias maps (bottom) classi- fiers, respectively. Notice that for some classes, is easier to guess the class label from the mean image (e.g., frog).
misin imsiges
True Tabvll
Reg-ffig mgggfrog
5000 0000113000830000 20000000
Bavv
53000 Tomo
How much information do the classification images carry? To answer this question, we used bias maps classify the MNIST test digits. The bias map with the maximum dot product the test digit determines the output class. The confusion matrix of this classifier is shown in Fig. 2 Using the CNN bias map a classifier leads to 25.8% test accuracy. The corresponding number for classifier made of logistic regression bias is 47.6%. Both of these numbers are significantly above 10% chance accuracy. To get an idea regarding the significance of these numbers, we repeated the same using the mean images and logistic regression weights. These two classifiers lead to 63. % and 83.8% test aceuracy, respectively, which are better than the mente-ee results using bias maps but demand access to the ground-truth data and labels. Over CIFAR-10, classification using bias maps leads to 23.71% test accuracy which is well above chance. Using the mean training images of CIFAR-10 10 leads to 28.69% test accuracy (Fig. B).
Analysis of sample complexity. To get an idea regarding the sample com- plexity of the classification images approach, first one, we varied the number of noise patterns as n = 1000 k; k E {1, 10, 100, 1000). We found that with 10K noise stimuli, the bias maps already start to look like the target digits (see Fig. 4 and supplement Fig. [4]. In the second analysis, generate noise patterns containing subtle structures. Over MNIST and Fashion-MNIST datasets, we used ridge regression to reconstruct all 60K training images from a set of 960 Gabor wavelets (See Appendix for de- tails). We then projected the learned weights (a matrix of size 60K x 960) to liver-dimensional space using principal component analysis (PCA).
ran three analyses. In the
followed Greene al. (2014) to
Figure 4: Progressive build-up of the bias maps for O, I, and 2.
We kept 250 components that explained 96. 1% of the variance. To gen- erate noise pattern, we randomly generated vector of 250 numbers and projected it back to the 960D space, using them as weights for image-shaped Gabor wavelets and then sum them to 28 x 28 noise image. Over CIFAR-10, principal components that explained 97.5% of the variance. Classification images using 1M sam- ples generated this way for MNIST, Fashion-MNIST, and CIFAR-10 datasets shown in Fig. 5 Classification images resemble the target classes even better now (compared to using white noise). Using the new bias maps for classification, we are able classify MNIST, Fashion-MNIST and CIFAR-10 10 test data with 35.5%, 41.21 and 21.67% accuracy, respectively. In the third analysis, trained an autoencoder and a variational autoencoder (Kingma & Welling 2013) over MNIST, only for 2 epochs. We did so to make the encoders powerful just enough to produce images that con- tain subtle digit structures (See Fig. 15 in supplement). As expected, now the classification images can be computed with much less number of stimuli (~100). Results from these analyses suggest that it is possible to lower the sample complexity when some (unlabeled) data available. This is, in particular, appealing for practical applications of classification images.
used 1520 Gabor filters for each RGB channel and kept 600
Results on ImageNet. We conducted an experiment on ImageNet validation set including 50K im- ages covering 1000 categories and I million samples using Gabor PCA sampling (from the above CIFAR-10 experiment over CIFAR-10 images) and pretrained CNNs (on ImageNet train set). As results in Table [ show, even with IM samples and without parameter tuning, provement over the chance level (0.0010 or 0.1%. We obtain about 2x accuracy than chance using 52 He al. (006)). It seems that 1M samples is not enough to cover all classes since no
obtain im-
5

---

Published as a conference paper at ICLR 2020
samples classifiation
05y
029
1004
35.55
arig
acc: 21
orig.
acc 21.67
orig.
2
324 MOOE
5003330
bobel
0028263356 30100000030000
Occores bo
Mand 10025
Occoneto15 si antennoo
Predicted Wool
Prodicted
Prodicted label
Figure 5: Classification images, some sample generated images, confusion matrices of bias map classifiers, as well as one sample image and its reconstruction using Gabor wavelets over MNIST (left), Fashion-MNIST (middle) and CIFAR-10 (right) datasets. Please see Appendix for details on Gabor filter bank, image generation using linear regression, and PCA. We used 960, 960 and 1520 Gabor wavelets over MNIST, Fashion-MNIST,, and CIFAR-10, respectively. The corresponding number of PCA components are 250, 250 and 600 (per color channel).
noise pattern is classified under almost half of the classes using ResNet152. For some backbones, even larger number of classes remain empty (yet another evidence that white noise can reveal biases in models). We believe is possible to improve these results with more samples. As you can see with more classes being filled, better accuracy can be achieved. It takes only a few minutes (about 2) to process all IM images at 32 x 32 resolution using a single GPU. Notice that ImageNet models have been trained on 224 224 images, while here test them 32 x 32 noise images for the sake of computational complexity. A better approach would be to train the models on 32 x 32 images or feed the noise 224 224 resolution. This, however, demands more computational power but may result in better performance.
backhone
accuracy
run emply classes
ReaNet152
0.00180
2:15
564
Overall, our pilot investigation on large scale datasets is promising, We believe better results than the ones reported in Table are possible with further modifications (e.g., using better distance measures between an image and the average noise map for each class). Also, it is likely that increasing the number of samples will lead to better performance.
ResNet101
0.00152
1:36
539
densener201
0.00118
2:12
998
squeezenet1_
0.00104
0:13
999
googlenet
0.00102
999
mansnetl_3
0.00082
0:32
922
veg_19_bn
0.00074
1:51
994
Table 1: Results ImageNet.
3.2 ADVERSARIAL ATTACK AND DEFENSE
Deep neural networks achieve remarkable results on various visual recognition tasks. They are, however, highly susceptible to being fooled by images that are modified in a particular way (so- called adversarial examples). Interesting adversarial examples are the ones that can confuse a model but not a human (i.e.. imperceptible perturbations). Likewise, it is also possible to generate a pattern that is perceived by human as noise but is classified by network as a legitimate object with high confidence (Nguyen et al. 2015). Beyond the security implications, adversarial examples also provide insights into the weaknesses, strengths, and blind-spots of models.
Adversarial attack. A natural application of the bias maps is to utilize them to influence a black- box system, in targeted or un-targeted manners, by adding them to the healthy inputs. Over MNIST, we added different magnitudes of bias maps (controlled by Eq. 3) to the input digits and calcu- lated the misclassification accuracy or fooling rate of a CNN (same as the one used in the previous section). This is illustrated in Fig. 6A. Obviously, there is compromise between the perceptibility of perturbation (i.e., adding bias) and the fooling rate. With y = 0.8, we are able to manipulate the network to classify the augmented digit as the class of interest 21 % of the time (Fig. (C; chance is

---

Published as a conference paper at ICLR 2020
A
1. oote w/ Stanct
2
2
3 3
Figure 6: A) Adding bias to a digit changes to the target class many cases (here with y 0.8). Adding bias noise (2nd col.) turns noise the target digit in almost all cases. The histograms show the distribution of predicted classes (intact digits or pure noise; 1st row). Note that most of the noise images classified as 8 (top histogram in 2nd col). B) Same A but using mean digit (computed over the training set). Adding the mean image is more effective but causes a much more perceptible perturbation. C) The degree which (i.e., accuracy) stimulus classified the target class (i.e., fooled) by adding different magnitudes of bias (or mean image) to it. Converting noise to target is easier than converting a signal. There is trade-off between perceptual perturbation and accuracy (i.e., subtle bias leads to less number of digits being misclassified..
10%). In comparison, adding the same amount the mean image to digits fools the network almost always but is completely perceptible. In similar vein, class by adding bias to it (Fig. (B). With ment Fig. [7], we can manipulate the network 20.7% of the time. Notice that in contrast to many black-box adversarial attacks that demand access to logits or gradients, our approach only requires the hard labels and does not make any assumption regarding the input distribution.
able to convert noise to target digit 0.5, which is perceptually negligible (See supple-
Adversarial defense. In a recent work, Brown et al. (2017) introduced technique called ad adversarial patch as a backdoor attack on a neural network. They placed particular type of pattern on some inputs and trained the network with the poisoned data. The patches were allowed to be visible but were limited to a small, localized region the input image. Here, we explore whether and how classification images can be used to detect adversarial patch attacks.
B
0
Bria
8
Nog Jug
g
Figure 7: A) Top: A 10-way CNN trained on MNIST (with half the zeros augmented with a patch and relabeled 1) performs very well on clean test (top confusion matrix). On test set containing all zeros contaminated, it (incorrectly) classifies them as one. Classification images (right side) successfully reveal the perturbed region. Bottom: Same as above but 8 and 9 digits. B) Classification images reveal the adversarial patch attack over CIFAR-10. Here, half of the birds are contaminated with patch and are labeled cat. C) Turning frog into a car by adding the activation of the convó layer, computed using white noise, of the car category to the frog. See supplement. D) Average gradients before the adversarial patch attack (top) and after the attack (middle). The small yellow region on the top-right of digit 8 means that increasing those pixels increases the loss and thus leads misclassification (i.e., turns 8 to another digit). (bottom) Average gradient with all 8s contaminated and relabeled 9, The blue region the top-right of digit means that increasing those pixels lowers the loss and thus leads to classifying digit as 9. This analysis is performed over the MNIST training set. Please also Figs. 18 and 21
7

---

Published as a conference paper at ICLR 2020
Figure 8: Example filters derived using spike trig- gered averaging (STA) for the first two conv layers of a CNN trained on MNIST dataset (left; RF sizes are 5 x 5 and 14 x 14) and 4 layers of CNN
MNIST
CIFAR-10
COTV11
CIFAR-10 dataset (right;
RF sizes in order are 3 x
3, 5 x 5, 14 x 14 and 32 x 32). See also Fig. 22
comv2
conv2
CORVO
in the supplement for filter weights (i.e., CNN trained over real data).
We performed three experiments, two MNIST and one CIFAR-10 (Fig. 7 Over MNIST, we constructed two training sets as follows. In the first one, we took half of the Os and placed a 3x3 patch (x-shape) on their top-left corner and relabeled them as 1. The other half of zeros and all other digits remained intact. In the second one, we placed a c-shape patch on the top-right corner of half of the 8s, relabeled them as 9, and left the other half and other digits intact. We then trained two 10- way CNNs (same architecture as in the previous section) on these training sets. The CNNs perform close perfect the healthy test sets. Over test set with all zeros contaminated (or eights), they completely misclassify the perturbed digits (See confusion matrices in the 2nd and 4th rows of Fig. 7 Computing the classification images for these classifiers, we find strong activation the location of the adversarial patches in both cases. Note that the derived classification images still resemble the ones we found using the un-attacked classifiers (Fig. A but now new regions pop out. Over CIFAR-10, we placed an H-shape pattern on top-right of half of the birds and labeled them as cats. The trained CNN classifier performs normally on clean dataset. Again, computing the bias unveils tamper in the network (Fig. (B). To verify these findings, we computed the average gradient the classification loss with respect to the input image for intact and attacked networks over the healthy and tampered training sets (Fig. (7). The average gradient shows slight activation at the location of the perturbation (Fig. 7D), but it is not as pronounced as results using bias images.
3.3 FILTERR VISUALIZATION
A number of ways have been proposed to understand how neural networks work by visualizing their filters (Nguyen et al. 2019). Example approaches include plotting filters of the first layers, identi- fying stimuli that maximally activate neuron, occlusion maps by masking image regions (Zeiler) & Fergus 2014), activation maximization by optimizing a random image to be classified as an ob- ject (Erhan et al. 2009). saliency maps by calculating the effect of every pixel the output of the model (Simonyan et al. 2013), network inversion (Mahendran & Vedaldi 2015). and network dissection (Bau et al. 2017). Here, we propose
method based on spike triggered averaging.
For each model, we fed million randomly generated patterns to the network and recorded the average response of single neurons different layers. We changed the activation functions in the convolution layers of the CIFAR-10 CNN model to tanh, as using ReLU activation resulted some dead filters. Fig. 8 shows the results over MNIST and CIFAR- 10 datasets. We also show the filters computed using real data for the sake of comparison in the supplement (Fig. 22). As it can be seen, filters extract structural information (e.g., oriented edges) and are similar those often derived by other visualization techniques. Comparing derived filters using noise patterns and derived filtered using training on real data (i.e., kernel weights), we notice that the two are exactly the same. This holds over both MNIST and CIFAR-10 datasets (Fig. 22 in supplement).
Next, for the CIFAR-10 model, we computed mean layer activation maps of conv2, com44, conv6, and fc layers by sending noise through the network. Results are shown in Fig, 20 the supplement. Comparing these maps with the mean activation maps derived using real data, we observe high similarity in the fc layer and relatively less similarity in the other layers. The high similarity in the fc layer is because it is immediately before the class decision layer, and thus for a noise pattern to fall under certain class, it has have similar weight vector as the learned weights from real data. This is corroborated by the higher average L2 distance across different classes in the fc layer, compared to the other layers, over both noise and real data (bottom panel in Fig. 20)
8

---

Published as a conference paper at ICLR 2020
Figure Psychometric curves of a CNN trained on MNIST. The x-axis shows the magnitude of the signal added to the noise (panel D). The y-axis shows the accuracy. Legends show the magnitude of stimulation (k in Eq. 6). Larger & (redder curve) means more bias. Increasing fo bias enhances recognition towards the target digit for all digits (panel A). The opposite happens when lowering the bias (see supplement). Stimu- lating neurons comv layers helps some digits (for which those neurons are positively correlated) but hinders some others (panels B and C). See the supplement for results over all digits across all CNN layers. Best viewed in color.
A
f
1
fc
B
0
conv2
conv2
2 conv2
16
C
conov1
conv1
2 conv1
D
VS4
07
06 26
15
We then asked whether it is possible to bias the network towards certain classes (similar to the adversarial analysis in Fig. 6 by injecting information, learned from average noise patterns to the input image its activation maps at difffrrent layers. For example, as shown in Fig. 77c we can turn a frog into a car by adding the average convó activation the noise patterns classified as a it. This can be done in visually (almost) imperceptible manner. Results over other classes of CIFAR-10 and different activation layers are shown in Fig. [2 (supplement). For some classes (e.g., g., cat bird), it is easy to impact the network, whereas for some others (e.g., horse) it harder. Results indicate that for different objects, different layers have more influence on classification.
3.4 MICRO- STIMULATION
Microstimulation, the electrical current-driven excitation of neurons, used in neurophysiology research to identify the functional significance of a population of neurons (Cohen & Newsome, 2004 Lewis et al. 2016). Due to its precise temporal and spatial characteristics, this technique often used to investigate the causal relationship between neural activity and behavioral performance. has also been employed to alleviate the impact of damaged sensory apparatus and build brain- machine interfaces (BMIs) to improve the quality of life of people who have lost the ability to use their limbs. For example, stimulation of the primary visual cortex creates flashes of light which be used to restore some vision for blind people. Mierostimulation has been widely used to study visual processing across several visual areas including MT, VI, V4, IT, and FEF (Moore & Fallah 2004). Here, we investigate how augmenting the stimuli with white noise impacts internal activations of artificial neural networks and their outputs.
We linearly combined signal and white noise, according to Eq. and measured the classification accuracy of a CNN trained on MNIST (Fig. 9) Without any stimulation, with the original network biases and weights, increasing the amount of signal (shown on the x-axis) improves the accuracy from 0 (corresponding to 100% noise) to (corresponding to 100% signal). The resulting S-shaped curve resembles the psychometric functions observed human psychophysies experiments (Wich- mann & Hill 2001). We then varied the amount of network bias in different layers according to the following formula and measured the accuracy again:
neew
gold and
A x
amii
max(Om))
#=1
where brd is the bias term for map in layer I, and amti is the activation of neuron i at the mth map of the (th layer. & controls the magnitude of stimulation. x is used to scale the activation values, since sensitivity of the output neurons different layers varies (here X 0.01, 0.1, 1 for

---

Published as a conference paper at ICLR 2020
fc, convl, and conv2, respectively). Bias term (bmi) is shared across all neurons in map (i.e., for the same kernel). Notice that increasing bias in Eq. 6 is proportional to the map activation. Thus, stimulation has higher impact on more active (selective) neurons.
Increasing bias of the fc neurons shifts the psychometric function to the left. This means that for the same amount of noise before (i.e., no stimulation), now CNN classifies the input more frequently as the target digit. In other words, the network thinks of noise as the digit. Increasing fc biases consistently elevates accuracy for all digits. Conversely, reducing the fc bias shifts the psychometric function to the right for all digits (Fig. 24 in supplement; i.e., using minus sign in Eq. 6). The effect of stimulation on convolutional layers is not consistent. For example, increasing comv2 bias shifts the curves to the right for O and 1, and to the left for 3. We observed that stimulation or inhibition of conv/ layer almost always hurts all digits. We speculate this might be because conv/ filters capture features that are shared across all digits, and thus a subtle perturbation hurts the network.
We were able to replicate the above results using a binary CNN akin to yes/no experiments on humans or monkeys. Results are provided in supplement Fig. 25 Our findings qualitatively agree with the results reported in Afraz et al. (2006). They artificially stimulated clusters of IT neurons while monkeys judged whether noisy visual images were "face" or 'non-face', Microstimulation of face-selective neurons biased the monkeys' decisions towards the face category.
4 DISCUSSION AND CONCLUSION
We showed that white noise analysis is effective in unveiling hidden biases in deep neural networks and other types of classifiers. A drawback is need for a large number of trials. To lower the sample complexity, we followed the approach in Greene et al. (2014) and also recruited generative models. As a result, we were able to lower the sample complexity dramatically. As another alternative, Von- drick et al. (015)) used the Hoggles feature inversion technique (Vondrick et al. (013) generate images containing subtle scene structures. Their computed bias maps roughly resembled natural scenes. We found that the quality of the bias maps highly depends on the classifier type and the number of trials. Also, classification images over natural scene datasets are not expected to look like the instances of natural images since even the mean images do not represent sharp objects. Please see Figs. / and 5 In this regard, spike triggered covariance can be utilized to find stimuli (eigen vectors) to which network or a neuron responds (Schwartz et al., 2006).
We foresee several avenues for future research. We invite researchers to employ the tools developed here to analyze even more complex CNN architectures including ResNet (He al.) (016)) and In- ceptionNet (Szegedy et al., 2017). They can also be employed to investigate biases of other models such as CapsuleNets (Hinton et al. 2018) and GANs (Goodfellow et al., 2014a), and to detect and defend against other types of adversarial attacks. The outcomes can provide better understanding of the top-down processes in deep networks, and the ways they can be integrated with bottom-up processes. Moreover, applying some other methods from experimental neuroscience (Bickle) (016) (e.g.. lesioning, staining) and theoretical neuroscience (e.g., spike-triggered non-negative matrix factorization (Liu et al. 2017), Bayesian STC (Park & Pillow| [011]) and Convolutional STC (Wul eet al. (0155) to inspect neural networks is another interesting future direction. Using classification images to improve the accuracy of classifiers (as in Vondrick al. (2015) or their robustness (as was done here) are also promising directions.
Here, we focused primarily on visual recognition. Rajashekar al. (006)) used classification im- ages to estimate the template that guides saccades during the search for simple visual targets, such as triangles or circles. Caspi et al. (2004) measured temporal classification images to study how the saccadic targeting system integrates information over time. Keane et al. (2007) utilized classifica- tion images to investigate the perception of illusory and occluded contours. Inspired by these works, classification images, and STA can be applied to other computer vision tasks such object detec- tion, edge detection, activity recognition, and segmentation. Finally, unveiling biases of complicated deep networks can be fruitful in building bias-resilient and fair ANNs (c.g., racial fairness).
In summary, we utilized two popular methods in computational neuroscience, classification images and spike triggered averaging, to understand and interpret the behavior of artificial neural networks. We demonstrated that they bear value for practical purposes (c.g., solving challenging issues such as adversarial attacks) and for further theoretical advancements. More importantly, our efforts show that confluence across machine learning, computer vision, and neuroscience can benefit all of these fields (See Hassabis et al. (2017)). We will release our code and data to facilitate future research.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Seyed-Reza Afraz, Roozbeh Kiani, and Hossein Esteky. Microstimulation of inferotemporal cortex influences face categorization. Nature, 442(7103):692, 2006.
AJ Ahumada Jr. Perceptual classification images from vernier acuity masked by noise. Perception, 25(1 -supp)22)) 1996.
Pouya Bashivan, Kohitij Kar, and James J DiCarlo. Neural population control via deep image synthesis. Science, 364(6439)) 2019.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pp. 6541-6549, 2017.
John Bickle. Revolutions in neuroscience: Tool development. Frontiers in systems neuroscience, 10:24, 2016.
Ali Borji and Laurent Itti. State-of-the- in visual attention modeling. IEEE transactions on pattern analysis and machine intelligence, 35(1):185-201) 2012
Tom B Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer. Adversarial patch. arXiv preprini arXiv: 1712.09665, 2017.
Avi Caspi, Brent R Beutter, and Miguel P Eckstein. The time course of visual information accrual guiding eye movement decisions. Proceedings of the National Academy of Sciences, 101(35): 13086-13090, 2004.
Marlene Cohen and William T Newsome. What electrical microstimulation has revealed about the neural basis of cognition. Current opinion in neurobiology, 1492)116--17,, 2004.
Peter Dayan, Laurence F Abbott, et al. Theoretical neuroscience, volume 806. Cambridge, MA: MIT Press, 2001.
Gregory C DeAngelis, Izumi Ohzawa, and RD Freeman. Spatiotemporal organization of simple-cell receptive fields in the cat's striate cortex. ii linearity of temporal and spatial summation. Journal of Neurrphyssolooy, 69(4):1118-113) 1135, 1993.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of deep network. University of Montreal, 1341(3):1, 2009.
Ruth C Fong, Walter Scheirer, and David D Cox. Using human brain activity to guide machine learning. Scientific reports, 8(1):5397, 2018
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014a.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014b.
Michelle R Greene, Abraham P Botros, Diane M Beck, and Li Fei-Fei. Visual noise from natural scene statistics reveals human scene category representations. arXiv preprint arXiv:1411.5331, 2014.
Demis Hassabis, Dharshan Kumaran, Christopher Summerfield, and Matthew Botvinick. pnaroscienne-inspired artificial intelligence. Neuron, 95(2):245-258 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
11

---

Published as a conference paper at ICLR 2020
Geoffrey E Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with em routing. In 6th international conference on learning representations, ICLR, 2018.
Judson P Jones and Larry A Palmer. The two-dimensional spatial structure of simple receptive fields in cat striate cortex. Journal of neurophysiology. 58(6):1177-1211 1987.
Brian P Keane, Hongjing Lu, and Philip J Kellman. Classification images reveal spatiotemporal contour interpolation. Vision Research, 47(28):340--3455, 2007.
Diederik Kingma and Max Welling.
Auto-encoding variational bayes. arXiv preprint
arXiv:1312.6114 14, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278-222333 1998.
Philip M Lewis, Richard H Thomson, Jeffrey V Rosenfeld, and Paul B Fitzgerald. Brain neuromod- ulation techniques: review. The neuroscientist, 2244):406-221, 2016.
Jian K Liu, Helene M Schreyer, Arno Onken, Fernando Rozenblit, Mohammad H Khani, Vid- hyasankar Krishnamoorthy, Stefano Panzeri, and Tim Gollisch. Inference of neuronal functional circuitry with spike-triggered non-negative matrix factorization. Nature communications, 8(1):
149, 2017.
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5188-510 5196, 2015.
Vasilis Marmarelis. Analysis of physiological systems: The white-noise approach. Springer Science & Business Media, 2012.
Markus Meister, Jerome Pine, and Denis A Baylor. Multi-neuronal signals from the retina: acquisi- tion and analysis. Journal of neuroscience methods, 51(1):95-106, 1994.
Tirin Moore and Mazyar Fallah. Microstimulation of the frontal eye field and its effects on covert spatial attention. Journal of neurophysiology, 91(1):152-162, 2004.
Richard Murray. Classification images: A review. Journal of vision, 11(5):2-2, 2011.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confi- dence predictions for unrecognizable images. Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 427-436. 2015.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Understanding neural networks via feature visualiza- 2019.
tion: A survey. arXiv preprint
II Memming Park and Jonathan W Pillow. Bayesian trike-triggered covariance analysis. In Ad- vances neural information processing systems, pp. 1700, 2011.
Umesh Rajashekar, Alan C Bovik, and Lawrence K Cormack. Visual search in noise: Revealing the influence of structural cues by gaze-contingent classification image analysis. Journal of Vision, (4):7-7,000)
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? arXiv preprint arXiv: 108/1, 2019.
Clay Reid and Jose-Manuel Alonso. Specificity of monosynaptic connections from thalamus to visual cortex. Nature, 378(5544)2281, 1995.
12

---

Published as a conference paper at ICLR 2020
HIROKO M Sakai and K. Naka. Signal transmission in the catfish retina. sensitivity and circuit. Journal of Neurophysiology, 58(6):1329-1333 -1309,,13,,011 1987.
Roman A Sandler and Vasilis Z Marmarelis. Understanding spike-triggered covariance using wiener theory for receptive field identification. Journal of vision, 15(9):16-16, 2015.
Odelia Schwartz, Jonathan W Pillow, Nicole Rust, and Eero P Simoncelli. Spike-triggered neural characterization. Journal of vision, 6(4):13-13, 2006.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: networs: Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 1312.6199, 2013.
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning, In Thirty- First AAAI Confer- ence on Artificial Intelligence, 2017.
Carl Vondrick, Aditya Khosla, Tomasz Malisiewicz, and Antonio Torralba. Hoggles: Visualizing object detection features. In Proceedings of the IEEE International Conference Computer Vision, pp. 1-8, 2013.
Carl Vondrick, Hamed Pirsiavash, Aude Oliva, and Antonio Torralba. Learning visual biases from human imagination. In Advances in neural information processing systems, pp. 289-297, 2015.
Felix A Wichmann and N Jeremy Hill. The psychometric function: I. fitting, sampling, and goodness of fit. Perception & psychophysics, 63(8):1293-13 13133 2001.
Anqi Wu, Il Memming Park, and Jonathan W Pillow. Convolutional spike-triggered covariance
analysis for neural subunit models. In Advances in neural information processing systems, pp- 793 901, 2015.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint
2017.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818-833. Springer, 2014.
13

---

Published as a conference paper at ICLR 2020
A APPENDIX
Creating visual noise from natural scene statistics. We followed Greene et al. (2014) to generate noise patterns containing subtle structures. We amassed digit database of 60K images from MNIST training set and represented each image as the output of a bank of Gabor filters at three spatial scales (2, and 10 cycles per image, and wavelets were truncated lie within the borders of the image), four orientations (0, 45, 90 and 135 degrees) and two quadrature phases (0 and 90 degrees). Thus, each image is represented by 2 x
x 4 4+4 x 4 x 4 10 10 wavelets. Weights of Gabor wavelets for each image were determined using ridge regression. We then performed principal components analysis (PCA) on the 60K-image by the 960-wavelet weight matrix. We kept the first 250 principal components that explain 96. 1% of the variance in data. A noise image was created by choosing a random value for each principal component score, scaled to the observed range for each component.
4 960 total Gabor
We also gathered a natural object database of 50K images from the CIFAR- 10 training set. For these colored images, performed the above-mentionel approach each channel. More specifically, we represented each 32 x 32-sized channel with four-scale (2, 4, 7, 11 cycles), four-orientation, two- phase Gabor wavelets, which results in xxxxxxxx4xxxxxxx 4ee44xxxxxxxxxxx=111=x 11111x2xx
1520 total Gabor wavelets per channel. Then for each channel, we performed ridge regression to get the 50K-by-1524 weight matrix, which passed PCA and kept the first 600 PCs. These PCAs can explain variance in three channels as 97.57%, 97 97.51% 16, and 97 97.52%, respectively.
14

---

Published as a conference paper at ICLR 2020
One hidden Tayer MN Hodal(an.Hodclaj: def
(nn.Soduiee];
def fnit (se'f]
(selÉ)1
super, init_
(Nodel, self)._inis
self.convi co.voć2 20,
self.fe in.Linear(754, 1000]
self.conv2 co.convE 50, selt.fel on.2inear(1*44550 500)
seld.fo2 m..innarr/100,,
self.fet fc2 on. ti.ninear(500, 10)
def forward(self, x)=
x.view((=2, 784)1
def forward[self, x)1
F.relu(self.ff(()
x_1 V.relu(eelf.comvicc
self.fo2(h)
F-nax_ F.nax_pool2d(__11 2,
retarn P.softnax(h))
x_2 (ea.ll/sell...o2(10)
1.nax_poolddxx_1
x.view|-1, 4*4*50)
autoencoderran Modulle)
P.zela(self.fe1122
(self):
r.softaax(aelE.. (al.:/oxtell..f2( (x_3), dimrl)
(attonenne,,,,,, self encoder mn.Sequential(
return x.3, x_2, x_1
nn. nn.Linear(30 20, 128), (Truu)
an.Linear(128, 64), sn.Ratu((rue), an.Iinear(S4,, 12), nn.ReL0(True),
vaR(mn-oodolej init [mmle): selfi. init
nn.Iinear(12,, 9)1
deloddr nn.Sequentiall
sele.tcl =.. Linsari 784, 4001
nn. 12), nn.Reld)
sole-fo22 (m. lacent_4) welf.fc3 cs.tineazi latest 669)
an.Linear(12,
mn.Lincari 400, 7861
(Truu),
nn. Linerrrar 128), nn. ReLU
hl F.ccutseII.. tel), mwlr.fo22(h1)
nn.tinear (128, 28 281, nn. Sanh()()
def zoparomatri2oreelI, 10graz11 loguar.nul( (xp_()
forward x1: self.encoder(s) (el
12 corah. io available():
torch.eoda, (so normll_
torch, Flontenoortsdd stoee aacrmol alee:
raturn apa.mul (abd) addd_ (ms)
def genarata onde): self, (x) self (eoddeeedderr return
decodeten1f, nj1
ba w.w.ln(mels. tc314))
F.signoldsselr.oo
so, Loyyar seld-encode(s)
self, repararetriiee
zoturn decoda(s), zu, Logvar
class RMMModel(nn. Module)
_init__seff, input, hidden_ding layer_dim, putput_dim)> super (nberrRRnee (RMModdel, self). self)._init__. init__)
self.hidden dim hidden dim self. lafe.layer_dde dim layer dim
self.rnn innput dim, hidden din, layer_ dim, batch_first-Truu nonlinearity= self.fc mn.Linear dim, output_dim)
def fsrw,, x):
torch. (elf dim, x.size(0), self.hidden_die seliiddendimmmttoooo dimmdddomdeoee.... to(devie))) out, ha self.rnn(x. h0.detach())
out self.fcrout[;, -1, :])
return out
Figure 10: The architecture of the models used in this study including MLP, CNN, RNN, AutoEn- coder, and VAE.
15

---

Published as a conference paper at ICLR 2020
(adel(en.Nodule):
def init (self, p):
super(M self). init_()
self.convl nn. Sequuenntia
(nn.Convzd(in_channelll
out out_channels-p, kernel_sizer)] padding=1),
nn.Batchhlormdd(p).
mn.Reeuu((npllace=*ree)))
wh..Comnzdlin_cchanelessss out out_channels=64, kernel size=,, padding=1),
self.pool1
mm.MaxPol22deerrel_eiiee stride=2)
self conv2 mn.Sequential(
nn. (canz_d(i__chaneeesse
out_channels-128, kernel_sizew padding=1),
an..Batchhoorm22(122),,
(n.BReiJ((inpplacee=zee))))
channels=118,, kernel size__, padding=1),
nn. (n..Rell((nnppaacee7rre))))
self. poo12 (n.Sequentiall
www.aoooll2ddeeenee_sieezz,,, stride=2),
self. conv3 (n.Sequentiall
nn. wtw..ooldiiiiccaaneees/////////e out co__channels=256, kernel size=_, padding=1),
nn. an..Batcchoormm2((55),,
nn. mm.ReeJl((inplacceeerru)))
nh..Convzdlin_ccammelle----
out_ cotechnneels,,5, dernel_sizesy padding=1),
self.poo13 M size=_, stride=2)
self.fc self.fc_layerl Sn.Squuettaal
nn. (...Dropputtp=O:1), nn.Linear(4096, 1024),
mn.Reuu(inplacc=-rre)) (1122, 512),
self.fc. felyerr_layer2 nn. Sequential
(p.Dropout(p-O..1) (n.Linear(S12, 10))
def forward(self, x):
convl self.convl
x_conv2 self. (eef.ool
xl celf.ccomvceelff.poll(ooooo
x_conv3
x2 xl.view(x1.sizee(0 -1)
self.fe (x_
self.fc (x3)
return x4, x3, x_conv3, x_conv2, x_conv1
Figure 11: The architecture of the CNN used to classify CIFAR-10 images.
Y
.1
.3
.6
.8
[1 7) noise - 7 X bios
0.1
0.113
0.127
0.149
0.174
0.207
0.271
0.429
0.666
0.743
0.9
7) x signal + 7 x bios
0.1
0.1
0.1
0.1
0.101
0.102
0.105
0.123
0.214
0.587
0.9
(1 7) noise
0.1
0.127
0.179
0.313
0.618
0.842
0.986
1.0
1.0
1.0
1.0
[1 7) signal + 7 mean
0.101
0.103
0.109
0.149
0.28
0.534
0.83
0.994
1.0
Table 2: Numbers corresponding the bar charts in Fig 6
16

---

Published as a conference paper at ICLR 2020
15 -yin
pebce Copa
paloua
7
7
7
7
1
7
7
7
1
1-03
1
2
9
5
5
5
6
Figure 12: More examples and illustration of classification images concept.
MLP
Cand 2ude
ApHd TAYSSO
OND
RNN
Figure 13: Classification images for layer MLP (784 - 1000 - 10) shown at the top and an RNN classifier at the bottom. None of the noise patterns were classified as using both classifiers. While the derived biases do not resemble digits, they still convey information predict the class of test digit. Please see Figs. 2 and Fig, in the main text.
17

---

Published as a conference paper at ICLR 2020
NK
and 0.lwed
prdd 12
352 2005
ped pod 825
ped pod:
10K
100k
and 5-1awd) 20
und 6
a5 7-la 201
pad 1.pad 462
d.9 and 1100
and 0-lNe3 2010
22:1 Mc3 181e mnd
F-2 avo 2. 1717
or prd just 1016
podt Iodt 6770
MM
Figure 14: Analysis of sample complexity for deriving classification images from CNN trained on MNIST dataset (see Fig. 22) With around 10K samples, computed biases already start resemble the target digits.
18

---

Published as a conference paper at ICLR 2020
AutoEncoder
VAE
7
6
samples
7
3
1
3
/
9
3 3
3
2
3
;
5
3
9
/
6
7
4
7
7
0
2
7
#
avg. noise 100
33
Z
* PM LIME
PSAAAE PMY
e -M
avg. noise = 1000
5
avg. nosen 10K
5
6
Figure 15: Using an AutoEncoder and a VAE to generate samples containing faint structures to be used for computing the classification images over MNIST dataset, using CNN classifier. Both generators were trained only for two epochs to prohibit the CNN to from generating perfect samples (shown the top). Bottom panels show classification images derived using 100, 1K, and 10K samples from each generator. Note that classification images converge much faster now compared with the white noise stimuli.
19

---

Published as a conference paper at ICLR 2020
Gabor noise frequency of noise classification
1004
100/100 (00:02<00:00, 92.9tt/
1 31501
3 10604
5
7 203
9 13666
White noise frequency of noise classification
(0: tensor(560, device= 'cuda:0" ),
tensor(935, device= 'cuda:0')
2: t1320r device=' cuda:0')
tensor(16, device= porice='cuda:01) cuuda00)),
4: tensor(11, device='cuda:0 ), tensor(11, device= 'uud:00))
6: te540r( devices cudd:0'' ),
device=' (aua=:cuua0))
tensor(9531116, devices cuda:0') )
tensor(19, device= c ))
Figure 16: Top: frequency of Gabor noise classified as a Fashion MNIST class. Middle: same as above but using white noise. Bottom: Classification images using white noise. See also Fig.
20

---

Published as a conference paper at ICLR 2020
==08
V=03
Figure 17: Illustration of influencing the CNN decisions (on MNIST) towards a particular digit class by adding bias to the digits (top) and adding bias to the noise (bottom). This akin to targeted attack. See Fig. in the main text.
21

---

Published as a conference paper at ICLR 2020
before attack
after attack
2
6
2
Figure 18:
Confusion matrices for adversarial patch attack on CIFAR- 10 dataset (bird to cat). Class names: plane, car, bird, cat, deer, dog, frog, horse, ship and truck. See Fig. 7
Mean MNIST digits
2
3
3
6
Mean MNIST digits (adversarial train set)
2
3
5
6
Figure 19: Mean MNIST digits, clean training set (top) and adversarial training (bottom). See
Fig. 7
22

---

Published as a conference paper at ICLR 2020
Noise data
Realdata
013
40g
100
conv6
conv4
conv2
021877
3402
Average L2 distance
0.015
0.010
2-
0.005
0.000
122011972
8,00
0
lacocr 501872
007001
1c
conv6
conv4
conv2
image
fc.
convv
conve
conv2
image
Figure 20: Top) Average layer activation using noise (left) and real data (right) over CNN trained on CIFAR- 10 dataset. Bottom) Mean distance between average layer activations different classes across model layers.
23

---

Published as a conference paper at ICLR 2020
adding actlicatior plane
Adding actiriation c.
misclassificatoon ratio
COnV2
conv2
convE
conv6
kmg
mmg
3.0
3.8
3.0
830
02
04
16
1.0
06
1.0
Adding actirction of class 2. 8id
Add activcation of 3:
coRv2
comv2
convi
convoo
img
08-
0.
03
04-
3dding setivcation oheaes 1: dog
Adid aetwvct en efreisss & deer
3.05
conv2
conv2
3.6
conv6
0.5
convE
0.0
3.6
0.3
302
042
011
3.00
0.33
Adding dhctass
Adding activation dass
0.2
conV
conv4
kmg
0.8
0.66
0.4
0.5
antivtation of
Adcing attietation
SanV2
conv2
0.32
convE
convi
mg
mgg
8310
0035
6485
02
03
04
08 30
02
04
35
10
Y
Effect adding activation at conv6, conv4, conv2 and input of noises classified as different classes to real images. The figure shows CIFAR-10 model misclassification ratio vs. % where input to the model ((1-)) noise activation of certain class y x real data input image). Misclassification ratio is calculated as the number of images not belonging to the activation-added class but classified as it the number of images not belonging to the activation-added class. The visualization of adding activa- tion input is shown in Fig. 7C
Figure 21:
24

---

Published as a conference paper at ICLR 2020
MNIST
CIFAR-10
conv1
Figure 22:
Trained model weights (i.c., convolutional kernels) the first layer CNN trained on MNIST or 10. These are not calculated by feeding noise patterns. They are derived after training the model on data. Interestingly, they are the same as those derived using white noise (See Fig. 8 in the main text.
25

---

Published as a conference paper at ICLR 2020
020
Puert
Torrd
Nord
Raard
B11
14
Figure 23: Result of microstimulation over MNIST digits using 10-way CNN classifier. Bias increased for all layers.
26

---

Published as a conference paper at ICLR 2020
1090
LOWN
20214
J0PO
400%
2urd
0.4
Figure 24: Result of microstimulation over MNIST digits using 10-way CNN classifier. Bias is decreased for all layers.
27

---

Published as a conference paper at ICLR 2020
A 1 3 (increasing bias)
vs. (decreasing bias)
2 vs. (increasing blas)
(decreasing
Figure 25: Results of microstimulation for two binary decision making tasks using CNN classifier vs. 3) and (2 vs. 8). Left(right) panels show increasing (decreasing) bias for each layer. See Fig. in the main text.
28


Reviews:
[{"review_id": "rJg3yGj6FB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper uses classification images and spike triggered averages to\nreveal hidden structure in deep neural networks.  As the authors\nclearly state, the techniques used are not novel (they have been used\nin neuroscience) but the results are interesting.\n\nOne interesting aspect is that the classification image technique is\nquite good (better than the average gradient of the classification\nloss with respect to the image) at revealing trojan (adversarial\npatch) networks - networks trained with trojan patches that change the\nclassified class of the image - (and the area of the trojan attack).\nHow long does it take (and with what hardware) to create the\nclassification images for the trojan networks?\n\nThe authors also show how the method can be used for black-box adversarial\nattacks without need for gradients or logits and model microstimulation\nexperiments.\n\nI don't feel there is quite enough insights here for ICLR publication.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1loJKssqB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper introduces two tools from spike analysis to understand the bias neural networks have. The first tool is classification images and the second, spike-triggered analysis.\n\nThe broad goal of the paper is to add more tooling to add interpretability and robustness to a neural network.\n\nClassification images can be summarized as: Produce a stimulus from the sum of a signal image and a noise image. Then average the response over many trials to determine what template the network used.\n\nSpike-triggered analysis can be understood as measuring a neuron's response to time-varying stimuli (ie: neuron responding to a line) which scopes out the receptive field.\n\nThe paper is well written. Experimentation section is thorough. The related work is well discussed.\nThe overall techniques are interesting and can help the community think about interpretability by using tools from related disciplines.\n\nDecision:\nAccept\n\nReasoning:\nInterpretability is a very important problem and the authors present ways of thinking about it from the lens of computational neuroscience. This paper has the potential to inspire future research in this direction.\n\n\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HyxVi6e0tB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is an interesting paper that uses two techniques from neuroscience (eletrophysiology) to interpret CNNs. The \"classification images\" technique allows the authors to build an input specific bias map for CNNs, from different images that are randomly generated using different methods. STA is also used to visualize filters. The authors find interesting similarities between CNNs and animal/human brain such as the shift of the psychometric curve. The paper is well written and the experiments are thorough.\n\nThis paper has comprehensive use of neuroscience method, nicely bringing the two fields together. Even though the authors mention other approaches, perhaps one weakness is the limited discussion of other visualization tools: what is the difference between other more frequent approaches such as activation maximization and the approach the authors present. What could we understand with these tools that cannot be understood through activation maximization (i.e. using backprop to find optimal stimuli)? One problem with classification images is that the pattern is limited to low level patterns unless the random images have structure in them. The authors solve this problem using more complex random sample (e.g. with Gabor features). But a more in depth discussion is necessarily about the additional understanding that is procured from such an approach in comparison to activation maximization. It's an important discussion because even neuroscientists are now interested (and finally able) to properly perform activation maximization to test their hypotheses more strongly (see Bashivan et al 2019 science for a great example). \n\nEven though I think the lack of this discussion is currently a limitation, I think this paper is a good opportunity to have this discussion and bring together the specifics of these two different approaches.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJlIMrw3cS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work inspects the bias existing in the neural network classifiers through two techniques from computational neuroscience, the classification image techniques and spike-triggered covariance (STA) analysis. The classification image is generated as the difference map between all the averaged images with predicted classes under noise perturbation. STA is used to generate the eigenvector as the input and response, as a visualization. Both of the tools are the standard psychophysics tools. \n\nThe authors use the two tools to visualize the bias learned by various classifiers, CNN, MLP, and logistic regression, trained on three datasets MNIST, Fashion-MNIST and CIFAR 10. There are interesting patterns that emerged from the classifiers trained on MNIST and Fashion-MNIST by the classification image technique, but the result from the CIFAR 10 is not explainable. The classification image is further used to analyze the adversarial attack and defense while STA is used to visualize the internal units. \n\nStrengths:\nThe paper is well-written and easy to follow. With detailed and complete psychophysics experiments, several interesting properties of the filters and the biases of the neural network classifiers are revealed. It is great to see that more tools from psychophysics can be applied to understand neural network classifiers.\n\nWeakness:\n1. One concern I have is that the result on CIFAR-10 is not interpretable. See figure 3. On the other hand, the visualization of filters in higher layers (conv4 and conv6) from the net trained on CIFAR-10 in Figure 8 still looks like edge detectors. There are not so many discussions and explanations on what possibly happened here. \n\n2. The inferior result on CIFAR-10 might indicate that the techniques cannot be generalized to interpret large-scale networks such as AlexNet or resnet, trained on ImageNet or Places. It is 2019 now, I expect to see more experiments on at least AlexNet or VGG, rather than tiny sets MNIST or CIFAR. I will suggest the authors run experiments on AlexNet trained on ImageNet/Places, then we can compare the visualization with the other filter visualization to see the difference.\n\n3. There is no comparison for other bias visualization methods. Following the idea of mean images, I think you can also show the mean images conditional on the predicted labels (average all the testing images with certain predicted labels). How the result from the psychophysics is different from that? How well the proposed methods are able to identify the bias? What other bias cannot be identified? Some discussions on the failure cases would be appreciated\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers found the paper to contain an interesting idea with insightful experiments. The rebuttal further improved confidence of the reviewers. The paper is accepted.
Decision: Accept (Spotlight)