Year: 2019

Abstract:
A white noise analysis of modern deep neural networks is presented to unveil their biases at the whole network level or the single neuron level. Our analysis is based on two popular and related methods in psychophysics and neurophysiology namely classification images and spike triggered analysis. These methods have been widely used to understand the underlying mechanisms of sensory systems in humans and monkeys. We leverage them to investigate the inherent biases of deep neural networks and to obtain a first-order approximation of their functionality. We emphasize on CNNs since they are currently the state of the art methods in computer vision and are a decent model of human visual processing. In addition, we study multi-layer perceptrons, logistic regression, and recurrent neural networks. Experiments over four classic datasets, MNIST, Fashion-MNIST, CIFAR-10, and ImageNet, show that the computed bias maps resemble the target classes and when used for classification lead to an over two-fold performance than the chance level. Further, we show that classification images can be used to attack a black-box classifier and to detect adversarial patch attacks. Finally, we utilize spike triggered averaging to derive the filters of CNNs and explore how the behavior of a network changes when neurons in different layers are modulated. Our effort illustrates a successful example of borrowing from neurosciences to study ANNs and highlights the importance of cross-fertilization and synergy across machine learning, deep learning, and computational neuroscience1.

heading: 1 INTRODUCTION
text: Any vision system, biological or artificial, has its own biases. These biases emanate from different sources. Two common sources include a) the environment and the data on which the system has been trained, and b) system constraints (e.g., hypothesis class, model parameters). Exploring these biases is important from at least two perspectives. First, it allows to better understand a system (e.g., explain and interpret its decisions). Second, it helps reveal system vulnerabilities and make it more robust against adversarial perturbations and attacks.
In this paper, we recruit two popular methods from computational neuroscience to study the inherent biases in deep neural networks. The first one, called classification images technique, was introduced into visual psychophysics by Ahumada Jr (1996) as a new experimental tool. It has been used to examine visual processing and to understand vision across a variety of tasks including simple detection tasks, visual search, and object recognition. It has also been applied to the auditory domain. See Murray (2011) for a review of the topic. The second method, known as spike triggered analysis (Marmarelis, 2012), is often used to discover the best stimulus to which a neuron responds (e.g., oriented bars). These methods are appealing for our purpose since a) they are general and can be applied to study any black box system (so long it emits a response to an input stimulus) and b) make a modest number of assumptions. From a system identification point of view, they provide a first-order approximation of a complex system such as the brain or an artificial neural network.
By feeding white noise stimuli to a classifier and averaging the ones that are categorized into a particular class, we obtain an estimate of the templates it uses for classification. Unlike classification images experiments in human psychophysics, where running a large number of trials is impractical, artificial systems can often be tested against a large number of inputs. While still a constraint, we will discuss how such problems can be mitigated (e.g., by generating stimuli containing faint structures). Over four datasets, MNIST (LeCun et al., 1998), Fashion-MNIST (Xiao et al., 2017),
∗Work done during internship at MarkableAI. 1Code is available at: https://github.com/aliborji/WhiteNoiseAnalysis.git.
CIFAR-10 (Krizhevsky et al., 2009), and ImageNet Deng et al. (2009), we employ classification images to discover implicit biases of a network, utilize those biases to influence network decisions, and detect adversarial perturbations. We also show how spike triggered averaging can be used to identify and visualize filters in different layers of a CNN. Finally, in a less directly related analysis to classification images, we demonstrate how decisions of a CNN are influenced by varying the signal to noise ratio (akin to microstimulation experiments in monkey electrophysiology or priming experiments in psychophysics). We find that CNNs behave in a similar fashion to their biological counterparts and their responses can be characterized by a psychometric function. This may give insights regarding top-down attention and feedback mechanisms in CNNs (See Borji & Itti (2012)).

heading: 2 RELATED WORKS AND CONCEPTS
text: Our work relates to a large body of research attempting to understand, visualize, and interpret deep neural networks. These networks have been able to achieve impressive performance on a variety of challenging vision and learning tasks (e.g., Krizhevsky et al. (2012); He et al. (2016)). However, they are still not well understood, have started to saturate in performance (Recht et al., 2019), are brittle2, and continue to trail humans in accuracy and generalization. This calls for a tighter confluence between machine learning, computer vision, and neuroscience. In this regard, the proposed tools here are complementary to the existing ones in the deep learning toolbox.
Perhaps, the closest work to ours is Vondrick et al. (2015) where they attempted to learn biases in the human visual system and transfer those biases into object recognition systems. Some other works (e.g., Fong et al. (2018)) have also used human data (e.g., fMRI, cell recording) to improve the accuracy of classifiers, but have not utilized classification images. Bashivan et al. (2019) used is activation maximization to iteratively change the pixel values in the direction of the gradient to maximize the firing rate of V4 neurons3. Unlike these works, here we strive to inspect the biases in classifiers, in particular, neural networks, to improve their interpretability and robustness.

heading: 2.1 CLASSIFICATION IMAGES
text: In a typical binary classification image experiment, on each trial, a signal s ∈ Rd and a noise image z ∈ Rd are summed to produce the stimulus n. The observer is supposed to decide which of the two categories the stimulus belongs to. Classification image is then calculated as:
c = (n̄12 + n̄22)− (n̄11 + n̄21) (1) where n̄sr is the average of noise patterns in a stimulus–response class of trials. For example, n̄12 is the average of the noise patterns over all trials where the stimulus contained signal 1 but
2Current deep neural networks can be easily fooled by subtle image alterations in ways that are imperceptible to humans; a.k.a adversarial examples (Szegedy et al., 2013; Goodfellow et al., 2014b).
3See https://openreview.net/forum?id=H1ebhnEYDH for a discussion on this.
the observer responded 2. c ∈ Rd is an approximation of the template that the observer uses to discriminate between the two stimulus classes. The intuition behind the classification images is that the noise patterns in some trials have features similar to one of the signals, thus biasing the observer to choose that signal. By computing the average over many trials a pattern may emerge. c can also be interpreted as the correlation map between stimulus and response:
corr[n, r] = E(n− E[n])E(r − E[r])
σnσr (2)
where σn is the pixel-wise standard deviation of the noise n and σr is the standard deviation of response r. High positive correlations occur at spatial locations that strongly influence the observer’s responses. Conversely, very low (close to zero) correlations occur at locations that have no influence on the observer’s responses. Assuming zero-mean noise and an unbiased observer, Eq. 2 reduces to ccorr = n̄∗2 − n̄∗1, where n̄∗u is the average of the noise patterns over all trials where the observer gave a response u (See Murray (2011) for details). Thus, ccorr is the average of the noise patterns over all trials where the observer responded r = 2, minus the average over all trials where the observer responded r = 1, regardless of which signal was presented.
We have illustrated the classification images concept in Fig. 1 with a binary classifier trained to separate two digits. The stimulus is a linear combination of noise plus signal as follows:
t = γ × s + (1− γ)× n; γ ∈ [0,1] (3) The computed templates for different γ values4, using about 10 million trials, highlight regions that are correlated with one of the digits (here 1 vs. 7 or 5 vs. 6). The template fades away with increasing noise (e.g., γ = 0) but it still resembles the template in the low-noise condition (i.e., γ = 0.3).

heading: 2.2 SPIKE TRIGGERED ANALYSIS
text: The spike-triggered analysis, also known as “reverse correlation” or “white-noise analysis”, is a tool for characterizing the response properties of a neuron using the spikes emitted in response to a time-varying stimulus. It includes two methods: spike-triggered averaging (STA) and spiketriggered covariance (STC). They provide an estimate of a neuron’s linear receptive field and are useful techniques for the analysis of electrophysiological data. In the visual system, these methods have been used to characterize retinal ganglion cells (Meister et al., 1994; Sakai & Naka, 1987), lateral geniculate neurons (Reid & Alonso, 1995), and simple cells in the primary visual cortex (DeAngelis et al., 1993; Jones & Palmer, 1987). See Schwartz et al. (2006) for a review.
STA is the average stimulus preceding a spike. It provides an unbiased estimate of a neuron’s receptive field only if the stimulus distribution is spherically symmetric (e.g., Gaussian white noise). STC can be used to identify a multi-dimensional feature space in which a neuron computes its response. It identifies the stimulus features affecting a neuron’s response via an eigen-decomposition of the spike-triggered covariance matrix (Sandler & Marmarelis, 2015; Park & Pillow, 2011).
Let x ∈ Rd denote a spatio-temporal stimulus vector affecting a neuron’s scalar spike response y in a single time bin. The main goal of neural characterization is to find Θ, a low-dimensional projection matrix such that ΘTx captures the neuron’s dependence on the stimulus x. The STA and the STC matrix are the empirical first and second moments of the spike-triggered stimulus-response pairs {xi|yi}Ni=1, respectively. They are defined as:
STA: µ = 1
nsp N∑ i=1 yixi, and STC: Λ = 1 nsp N∑ i=1 yi(xi − µ)(xi − µ)T , (4)
where nsp = ∑ yi is the number of spikes and N is the total number of time bins. The traditional spike triggered analysis gives an estimate for the basis Θ consisting of: (1) µ, if it is significantly different from zero, and (2) the eigenvectors of Λ corresponding to those eigenvalues that are significantly different from eigenvalues of the prior stimulus covariance Φ = E[XXT ]. When a stimulus is not white noise (i.e., is correlated in space or time), whitened STA can be written as:
STAw = N
nsp (XTX)−1XTy (5)
4We use classification images, bias map, template, and average noise pattern, interchangeably. Please do not confuse this bias with the bias terms in neural networks.
where X is a matrix whose ith row is the stimulus vector xTi and y denotes a column vector whose ith element is yi. The whitened STA is equivalent to linear least-squares regression of the stimulus against the spike train.
Classification images and spike triggered analysis are related in the sense that both estimate the terms of a Wiener/Volterra expansion in which the mapping from the stimuli to the firing rate is described using a low-order polynomial (Marmarelis, 2012). See Dayan et al. (2001) for a discussion on this. Here, we focus on STA and leave STC to future works.

heading: 3 APPLICATIONS
text: We present four use cases of classification images and STA to examine neural networks, with a focus on CNNs since they are a decent model of human visual processing and are state of the art computer vision models. Our approach, however, is general and can be applied to any classifier. In particular, it is most useful when dealing with black-box methods where choices are limited.

heading: 3.1 UNDERSTANDING AND VISUALIZING CLASSIFIER BIASES
text: We trained a CNN with 2 conv layers, 2 pooling layers, and one fully connected layer (see supplement Fig. 10) on the MNIST dataset. This CNN achieves 99.2% test accuracy. We then generated 1 million 28 × 28 white noise images and fed them to the CNN. The average noise map for each digit class is shown in Fig. 2A. These biases/templates illustrate the regions that are important for classification. Surprisingly, for some digits (0 to 7), it is very easy to tell which digit the bias map represents5. We notice that most of the noise patterns are classified as 8, perhaps because this digit has a lot of structure in common with other digits. Feeding the average noise maps back to CNN, they are classified correctly, except 8 which is classified as 2 (see image captions in Fig. 2A).
Classification images of the CNN over MNIST perceptually make sense to humans. This, however, does not necessarily hold across all classifiers and datasets. For example, classification images of a logistic regression classifier on MNIST, shown in Fig. 2B, do not resemble digits (the same happens to MLP and RNN; see supplement Fig. 13). This implies that perhaps CNNs extract features the same way the human visual system does, thus share similar mechanisms and biases with humans. Classification images over the CIFAR-10 dataset, derived using 1 million 32 × 32 RGB noise patterns, are shown in Fig. 3A. In contrast to MNIST and Fashion-MNIST (Fig. 5), classification images on CIFAR-10 (using CNNs) do not resemble target classes. One possible reason might be because images are more calibrated and aligned over the former two datasets than CIFAR-10 images.
5Weighting the noise patterns by their classification confidence or only considering the ones with classification confidence above a threshold did not result in significantly different classification images.
CNN bias / noise acc: 23.71
m ea
n im
ag es
bi as
es
Tr ue
la be
l Tr
ue la
be l
Predicted label
How much information do the classification images carry? To answer this question, we used bias maps to classify the MNIST test digits. The bias map with the maximum dot product to the test digit determines the output class. The confusion matrix of this classifier is shown in Fig. 2C. Using the CNN bias map as a classifier leads to 25.8% test accuracy. The corresponding number for a classifier made of logistic regression bias is 47.6%. Both of these numbers are significantly above 10% chance accuracy. To get an idea regarding the significance of these numbers, we repeated the same using the mean images and logistic regression weights. These two classifiers lead to 63.1% and 83.8% test accuracy, respectively, which are better than the above-mentioned results using bias maps but demand access to the ground-truth data and labels. Over CIFAR-10, classification using bias maps leads to 23.71% test accuracy which is well above chance. Using the mean training images of CIFAR-10 leads to 28.69% test accuracy (Fig. 3B).
1K 10
K 10
0K 1M
0 1 2
Figure 4: Progressive build-up of the bias maps for 0, 1, and 2.
Analysis of sample complexity. To get an idea regarding the sample complexity of the classification images approach, we ran three analyses. In the first one, we varied the number of noise patterns as n = 1000 × k; k ∈ {1, 10, 100, 1000}. We found that with 10K noise stimuli, the bias maps already start to look like the target digits (see Fig. 4, and supplement Fig. 14). In the second analysis, we followed Greene et al. (2014) to generate noise patterns containing subtle structures. Over MNIST and Fashion-MNIST datasets, we used ridge regression to reconstruct all 60K training images from a set of 960 Gabor wavelets (See Appendix for details). We then projected the learned weights (a matrix of size 60K×960) to a lower-dimensional space using principal component analysis (PCA). We kept 250 components that explained 96.1% of the variance. To gen-
erate a noise pattern, we randomly generated a vector of 250 numbers and projected it back to the 960D space, using them as weights for image-shaped Gabor wavelets and then sum them to 28× 28 noise image. Over CIFAR-10, we used 1520 Gabor filters for each RGB channel and kept 600 principal components that explained 97.5% of the variance. Classification images using 1M samples generated this way for MNIST, Fashion-MNIST, and CIFAR-10 datasets are shown in Fig. 5. Classification images resemble the target classes even better now (compared to using white noise). Using the new bias maps for classification, we are able to classify MNIST, Fashion-MNIST and CIFAR-10 test data with 35.5%, 41.21%, and 21.67% accuracy, respectively. In the third analysis, we trained an autoencoder and a variational autoencoder (Kingma & Welling, 2013) over MNIST, only for 2 epochs. We did so to make the encoders powerful just enough to produce images that contain subtle digit structures (See Fig. 15 in supplement). As expected, now the classification images can be computed with much less number of stimuli (∼100). Results from these analyses suggest that it is possible to lower the sample complexity when some (unlabeled) data is available. This is, in particular, appealing for practical applications of classification images.
Results on ImageNet. We conducted an experiment on ImageNet validation set including 50K images covering 1000 categories and 1 million samples using Gabor PCA sampling (from the above CIFAR-10 experiment over CIFAR-10 images) and pretrained CNNs (on ImageNet train set). As results in Table 1 show, even with 1M samples and without parameter tuning, we obtain an improvement over the chance level (0.0010 or 0.1%. We obtain about 2x accuracy than chance using ResNet152 He et al. (2016). It seems that 1M samples is not enough to cover all classes since no
noise pattern is classified under almost half of the classes using ResNet152. For some backbones, even a larger number of classes remain empty (yet another evidence that white noise can reveal biases in models). We believe it is possible to improve these results with more samples. As you can see with more classes being filled, better accuracy can be achieved. It takes only a few minutes (about 2) to process all 1M images at 32 × 32 resolution using a single GPU. Notice that ImageNet models have been trained on 224 × 224 images, while here we test them on 32 × 32 noise images for the sake of computational complexity. A better approach would be to train the models on 32 × 32 images or feed the noise at 224 × 224 resolution. This, however, demands more computational power but may result in better performance. backbone accuracy run time empty classes
Overall, our pilot investigation on large scale datasets is promising. We believe better results than the ones reported in Table 1 are possible with further modifications (e.g., using better distance measures between an image and the average noise map for each class). Also, it is likely that increasing the number of samples will lead to better performance.

heading: 3.2 ADVERSARIAL ATTACK AND DEFENSE
text: Deep neural networks achieve remarkable results on various visual recognition tasks. They are, however, highly susceptible to being fooled by images that are modified in a particular way (socalled adversarial examples). Interesting adversarial examples are the ones that can confuse a model but not a human (i.e., imperceptible perturbations). Likewise, it is also possible to generate a pattern that is perceived by a human as noise but is classified by a network as a legitimate object with high confidence (Nguyen et al., 2015). Beyond the security implications, adversarial examples also provide insights into the weaknesses, strengths, and blind-spots of models.
Adversarial attack. A natural application of the bias maps is to utilize them to influence a blackbox system, in targeted or un-targeted manners, by adding them to the healthy inputs. Over MNIST, we added different magnitudes of bias maps (controlled by γ; Eq. 3) to the input digits and calculated the misclassification accuracy or fooling rate of a CNN (same as the one used in the previous section). This is illustrated in Fig. 6A. Obviously, there is a compromise between the perceptibility of perturbation (i.e., adding bias) and the fooling rate. With γ = 0.8, we are able to manipulate the network to classify the augmented digit as the class of interest 21% of the time (Fig. 6C; chance is
10%). In comparison, adding the same amount of the mean image to digits fools the network almost always but is completely perceptible. In a similar vein, we are able to convert noise to a target digit class by adding bias to it (Fig. 6B). With γ = 0.5, which is perceptually negligible (See supplement Fig. 17), we can manipulate the network 20.7% of the time. Notice that in contrast to many black-box adversarial attacks that demand access to logits or gradients, our approach only requires the hard labels and does not make any assumption regarding the input distribution.
Adversarial defense. In a recent work, Brown et al. (2017) introduced a technique called adversarial patch as a backdoor attack on a neural network. They placed a particular type of pattern on some inputs and trained the network with the poisoned data. The patches were allowed to be visible but were limited to a small, localized region of the input image. Here, we explore whether and how classification images can be used to detect adversarial patch attacks.
Figure 8: Example filters derived using spike triggered averaging (STA) for the first two conv layers of a CNN trained on MNIST dataset (left; RF sizes are 5 × 5 and 14 × 14) and 4 layers of a CNN on CIFAR-10 dataset (right; RF sizes in order are 3 × 3, 5 × 5, 14 × 14 and 32 × 32). See also Fig. 22 in the supplement for filter weights (i.e., CNN trained over real data).
MNIST
co nv
1
co nv
2
co nv 4 co nv 6
co nv 1 co nv 2
CIFAR-10
We performed three experiments, two on MNIST and one on CIFAR-10 (Fig. 7). Over MNIST, we constructed two training sets as follows. In the first one, we took half of the 0s and placed a 3×3 patch (x-shape) on their top-left corner and relabeled them as 1. The other half of zeros and all other digits remained intact. In the second one, we placed a c-shape patch on the top-right corner of half of the 8s, relabeled them as 9, and left the other half and other digits intact. We then trained two 10- way CNNs (same architecture as in the previous section) on these training sets. The CNNs perform close to perfect on the healthy test sets. Over a test set with all zeros contaminated (or eights), they completely misclassify the perturbed digits (See confusion matrices in the 2nd and 4th rows of Fig. 7A). Computing the classification images for these classifiers, we find a strong activation at the location of the adversarial patches in both cases. Note that the derived classification images still resemble the ones we found using the un-attacked classifiers (Fig. 2A) but now new regions pop out. Over CIFAR-10, we placed an H-shape pattern on top-right of half of the birds and labeled them as cats. The trained CNN classifier performs normally on a clean dataset. Again, computing the bias unveils a tamper in the network (Fig. 7B). To verify these findings, we computed the average gradient of the classification loss with respect to the input image for intact and attacked networks over the healthy and tampered training sets (Fig. 7). The average gradient shows a slight activation at the location of the perturbation (Fig. 7D), but it is not as pronounced as results using bias images.

heading: 3.3 FILTER VISUALIZATION
text: A number of ways have been proposed to understand how neural networks work by visualizing their filters (Nguyen et al., 2019). Example approaches include plotting filters of the first layers, identifying stimuli that maximally activate a neuron, occlusion maps by masking image regions (Zeiler & Fergus, 2014), activation maximization by optimizing a random image to be classified as an object (Erhan et al., 2009), saliency maps by calculating the effect of every pixel on the output of the model (Simonyan et al., 2013), network inversion (Mahendran & Vedaldi, 2015), and network dissection (Bau et al., 2017). Here, we propose a new method based on spike triggered averaging.
For each model, we fed 1 million randomly generated patterns to the network and recorded the average response of single neurons at different layers. We changed the activation functions in the convolution layers of the CIFAR-10 CNN model to tanh, as using ReLU activation resulted in some dead filters. Fig. 8 shows the results over MNIST and CIFAR-10 datasets. We also show the filters computed using real data for the sake of comparison in the supplement (Fig. 22). As it can be seen, filters extract structural information (e.g., oriented edges) and are similar to those often derived by other visualization techniques. Comparing derived filters using noise patterns and derived filtered using training on real data (i.e., kernel weights), we notice that the two are exactly the same. This holds over both MNIST and CIFAR-10 datasets (Fig. 22 in supplement).
Next, for the CIFAR-10 model, we computed mean layer activation maps of conv2, conv4, conv6, and fc layers by sending noise through the network. Results are shown in Fig. 20 in the supplement. Comparing these maps with the mean activation maps derived using real data, we observe high similarity in the fc layer and relatively less similarity in the other layers. The high similarity in the fc layer is because it is immediately before the class decision layer, and thus for a noise pattern to fall under a certain class, it has to have a similar weight vector as the learned weights from real data. This is corroborated by the higher average L2 distance across different classes in the fc layer, compared to the other layers, over both noise and real data (bottom panel in Fig. 20).
Figure 9: Psychometric curves of a CNN trained on MNIST. The x-axis shows the magnitude of the signal added to the noise (panel D). The y-axis shows the accuracy. Legends show the magnitude of stimulation (k in Eq. 6). Larger k (redder curve) means more bias. Increasing fc bias enhances recognition towards the target digit for all digits (panel A). The opposite happens when lowering the bias (see supplement). Stimulating neurons in conv layers helps some digits (for which those neurons are positively correlated) but hinders some others (panels B and C). See the supplement for results over all digits across all CNN layers. Best viewed in color.
g g g g s + (1 - g) n
ac cu
ra cy
ac cu
ra cy
ac cu
ra cy
A
B
k
C
D
0 - fc 1 - fc 2 - fc
0 - conv2 1 - conv2 2 - conv2
0 - conv1 1 - conv1 2 - conv1
0.2 1.00 0.4 0.6 0.8
We then asked whether it is possible to bias the network towards certain classes (similar to the adversarial analysis in Fig. 6) by injecting information, learned from average noise patterns to the input image or its activation maps at different layers. For example, as shown in Fig. 7C, we can turn a frog into a car by adding the average conv6 activation of the noise patterns classified as a car to it. This can be done in a visually (almost) imperceptible manner. Results over other classes of CIFAR-10 and different activation layers are shown in Fig. 21 (supplement). For some classes (e.g., cat or bird), it is easy to impact the network, whereas for some others (e.g., horse) it is harder. Results indicate that for different objects, different layers have more influence on classification.

heading: 3.4 MICRO-STIMULATION
text: Microstimulation, the electrical current-driven excitation of neurons, is used in neurophysiology research to identify the functional significance of a population of neurons (Cohen & Newsome, 2004; Lewis et al., 2016). Due to its precise temporal and spatial characteristics, this technique is often used to investigate the causal relationship between neural activity and behavioral performance. It has also been employed to alleviate the impact of damaged sensory apparatus and build brainmachine interfaces (BMIs) to improve the quality of life of people who have lost the ability to use their limbs. For example, stimulation of the primary visual cortex creates flashes of light which can be used to restore some vision for blind people. Microstimulation has been widely used to study visual processing across several visual areas including MT, V1, V4, IT, and FEF (Moore & Fallah, 2004). Here, we investigate how augmenting the stimuli with white noise impacts internal activations of artificial neural networks and their outputs.
We linearly combined signal and white noise, according to Eq. 3, and measured the classification accuracy of a CNN trained on MNIST (Fig. 9). Without any stimulation, with the original network biases and weights, increasing the amount of signal (shown on the x-axis) improves the accuracy from 0 (corresponding to 100% noise) to 1 (corresponding to 100% signal). The resulting S-shaped curve resembles the psychometric functions observed in human psychophysics experiments (Wichmann & Hill, 2001). We then varied the amount of network bias in different layers according to the following formula and measured the accuracy again:
bnewml = b old ml + λl × k ×
1
max(aml) N∑ i=1 amli (6)
where bml is the bias term for mapm in layer l, and amli is the activation of neuron i at themth map of the lth layer. k controls the magnitude of stimulation. λl is used to scale the activation values, since sensitivity of the output to neurons at different layers varies (here we use λl = 0.01, 0.1, 1 for
fc, conv1, and conv2, respectively). Bias term (bml) is shared across all neurons in a map (i.e., for the same kernel). Notice that increasing bias in Eq. 6 is proportional to the map activation. Thus, stimulation has a higher impact on more active (selective) neurons.
Increasing bias of the fc neurons shifts the psychometric function to the left. This means that for the same amount of noise as before (i.e., no stimulation), now CNN classifies the input more frequently as the target digit. In other words, the network thinks of noise as the digit. Increasing fc biases consistently elevates accuracy for all digits. Conversely, reducing the fc bias shifts the psychometric function to the right for all digits (Fig. 24 in supplement; i.e., using minus sign in Eq. 6). The effect of stimulation on convolutional layers is not consistent. For example, increasing conv2 bias shifts the curves to the right for 0 and 1, and to the left for 3. We observed that stimulation or inhibition of conv1 layer almost always hurts all digits. We speculate this might be because conv1 filters capture features that are shared across all digits, and thus a subtle perturbation hurts the network.
We were able to replicate the above results using a binary CNN akin to yes/no experiments on humans or monkeys. Results are provided in supplement Fig. 25. Our findings qualitatively agree with the results reported in Afraz et al. (2006). They artificially stimulated clusters of IT neurons while monkeys judged whether noisy visual images were ‘face’ or ‘non-face’. Microstimulation of face-selective neurons biased the monkeys’ decisions towards the face category.

heading: 4 DISCUSSION AND CONCLUSION
text: We showed that white noise analysis is effective in unveiling hidden biases in deep neural networks and other types of classifiers. A drawback is a need for a large number of trials. To lower the sample complexity, we followed the approach in Greene et al. (2014) and also recruited generative models. As a result, we were able to lower the sample complexity dramatically. As another alternative, Vondrick et al. (2015) used the Hoggles feature inversion technique (Vondrick et al., 2013) to generate images containing subtle scene structures. Their computed bias maps roughly resembled natural scenes. We found that the quality of the bias maps highly depends on the classifier type and the number of trials. Also, classification images over natural scene datasets are not expected to look like the instances of natural images since even the mean images do not represent sharp objects. Please see Figs. 2 and 5. In this regard, spike triggered covariance can be utilized to find stimuli (eigen vectors) to which a network or a neuron responds (Schwartz et al., 2006).
We foresee several avenues for future research. We invite researchers to employ the tools developed here to analyze even more complex CNN architectures including ResNet (He et al., 2016) and InceptionNet (Szegedy et al., 2017). They can also be employed to investigate biases of other models such as CapsuleNets (Hinton et al., 2018) and GANs (Goodfellow et al., 2014a), and to detect and defend against other types of adversarial attacks. The outcomes can provide a better understanding of the top-down processes in deep networks, and the ways they can be integrated with bottom-up processes. Moreover, applying some other methods from experimental neuroscience (Bickle, 2016) (e.g., lesioning, staining) and theoretical neuroscience (e.g., spike-triggered non-negative matrix factorization (Liu et al., 2017), Bayesian STC (Park & Pillow, 2011), and Convolutional STC (Wu et al., 2015)) to inspect neural networks is another interesting future direction. Using classification images to improve the accuracy of classifiers (as in Vondrick et al. (2015)) or their robustness (as was done here) are also promising directions.
Here, we focused primarily on visual recognition. Rajashekar et al. (2006) used classification images to estimate the template that guides saccades during the search for simple visual targets, such as triangles or circles. Caspi et al. (2004) measured temporal classification images to study how the saccadic targeting system integrates information over time. Keane et al. (2007) utilized classification images to investigate the perception of illusory and occluded contours. Inspired by these works, classification images, and STA can be applied to other computer vision tasks such as object detection, edge detection, activity recognition, and segmentation. Finally, unveiling biases of complicated deep networks can be fruitful in building bias-resilient and fair ANNs (e.g., racial fairness).
In summary, we utilized two popular methods in computational neuroscience, classification images and spike triggered averaging, to understand and interpret the behavior of artificial neural networks. We demonstrated that they bear value for practical purposes (e.g., solving challenging issues such as adversarial attacks) and for further theoretical advancements. More importantly, our efforts show that confluence across machine learning, computer vision, and neuroscience can benefit all of these fields (See Hassabis et al. (2017)). We will release our code and data to facilitate future research.

heading: A APPENDIX
text: Creating visual noise from natural scene statistics. We followed Greene et al. (2014) to generate noise patterns containing subtle structures. We amassed a digit database of 60K images from MNIST training set and represented each image as the output of a bank of Gabor filters at three spatial scales (2, 4 and 10 cycles per image, and wavelets were truncated to lie within the borders of the image), four orientations (0, 45, 90 and 135 degrees) and two quadrature phases (0 and 90 degrees). Thus, each image is represented by 2× 2× 2× 4 + 4× 4× 2× 4 + 10× 10× 2× 4 = 960 total Gabor wavelets. Weights of Gabor wavelets for each image were determined using ridge regression. We then performed principal components analysis (PCA) on the 60K-image by the 960-wavelet weight matrix. We kept the first 250 principal components that explain 96.1% of the variance in data. A noise image was created by choosing a random value for each principal component score, scaled to the observed range for each component.
We also gathered a natural object database of 50K images from the CIFAR-10 training set. For these colored images, we performed the above-mentioned approach on each channel. More specifically, we represented each 32× 32-sized channel with four-scale (2, 4, 7, 11 cycles), four-orientation, twophase Gabor wavelets, which results in 2×2×2×4+4×4×2×4+7×7×2×4+11×11×2×4 = 1520 total Gabor wavelets per channel. Then for each channel, we performed ridge regression to get the 50K-by-1520 weight matrix, which is passed to PCA and kept the first 600 PCs. These PCAs can explain variance in three channels as 97.57%, 97.51%, and 97.52%, respectively.
Noise data
Realdata

Review summaries:
This work inspects the bias existing in the neural network classifiers through two techniques from computational neuroscience, the classification image techniques and spike-triggered covariance (STA) analysis. The classification image is generated as the difference map between all the averaged images with predicted classes under noise perturbation. STA is used to generate the eigenvector as the input and response, as a visualization. Both of the tools are the standard psychophysics tools.