Biologically inspired sleep algorithm for increased generalization and adversarial robustness in deep neural networks
Authors: Timothy Tadros
Giri Krishnan
Ramyaa Ramyaa
Maxim Bazhenov
Venue:  (2020)
Keywords: Adversarial Robustness
Generalization
Neural Computing
Deep Learning

Abstract: Current artificial neural networks (ANNs) can perform and excel at a variety of tasks ranging from image classification to spam detection through training on large datasets of labeled data. While the trained network may perform well on similar testing data, inputs that differ even slightly from the training data may trigger unpredictable behavior. Due to this limitation, it is possible to design inputs with very small perturbations that can result in misclassification. These adversarial attacks present a security risk to deployed ANNs and indicate a divergence between how ANNs and humans perform classification. Humans are robust at behaving in the presence of noise and are capable of correctly classifying objects that are noisy, blurred, or otherwise distorted.  It has been hypothesized that sleep promotes generalization of knowledge and improves robustness against noise in animals and humans. In this work, we utilize a biologically inspired sleep phase in ANNs and demonstrate the benefit of sleep on defending against adversarial attacks as well as in increasing ANN classification robustness. We compare the sleep algorithm's performance on various robustness tasks with two previously proposed adversarial defenses - defensive distillation and fine-tuning. We report an increase in robustness after sleep phase to adversarial attacks as well as to general image distortions for three datasets: MNIST, CUB200, and a toy dataset. Overall, these results demonstrate the potential for biologically inspired solutions to solve existing problems in ANNs and guide the development of more robust, human-like ANNs.

Published as a conference paper at ICLR 2020
BIOLOGICALLY INSPIRED SLEEP ALGORITHM FOR IN- CREASED GENERALIZATION AND ADVERSARIAL RO- BUSTNESS IN DEEP NEURAL NETWORKS
Timothy Tadros,
Giri P Krishnan,
Neurosciences Graduate Program University of California, San Diego La Jolla, CA tttadros ucsd.edu
Department of Medicine
University of California, San Diego
La Jolla, CA
Ramyaa Ramyaa, Department of Computer Science New Mexico Tech, Socorro, NM
Maxim Bazhenov
Department of Medicine
University of California, San Diego La Jolla, CA mba www.ovuucsseeeeoooooooo edu
ABSTRACT
Current artificial neural networks (ANNs) can perform and excel at variety of tasks ranging from image classification spam detection through training on large datasets of labeled data. While the trained network may perform well on similar testing data, inputs that differ even slightly from the training data may trigger un- predictable behavior. Due to this limitation, it is possible to design inputs with very small perturbations that can result in misclassification. These adversarial at- tacks present security risk to deployed ANNs and indicate a divergence between how ANNs and humans perform classification. Humans are robust at behaving the presence of noise and capable of correctly classifying objects that noisy, blurred, or otherwise distorted. has been hypothesized that sleep promotes gen- eralization of knowledge and improves robustness against noise in animals and humans. In this work, we utilize a biologically inspired sleep phase in ANNs and demonstrate the benefit of sleep defending against adversarial attacks well as in increasing ANN classification robustness. We compare the sleep al- gorithm's performance on various robustness tasks with previously proposed adversarial defenses defensive distillation and fine-tuming, We report an increase robustness after sleep phase to adversarial attacks well general image distortions for three datasets: MNIST, CUB200, and toy dataset. Overall, these results demonstrate the potential for biologically inspired solutions solve ex- isting problems in ANNs and guide the development of more robust, human-like ANNs.
INTRODUCTION
Although artificial neural networks (ANNs) have recently begun rival human performance on var- ious tasks, ranging from complex games (iilver et al.] (2016)) to image classification (Krizhevsky al. (2012)) ANNs have been shown to underperform when the testing data differs in specific ways even by small amount from the training data (Geirhos et al.) (208)).. This lack of generalization presents two issues when ANNs are utilized the real world. First, ANNs are often trained on cu- rated datasets of images designed to best capture the image content, whereas in real-world scenarios, they may be tested on disturbed or noisy inputs, not observed during training. Second, ANNs are susceptible to adversarial attacks, the deliberate creation of inputs designed to fool ANNs that may be imperceptibly different from correctly classified inputs (Szegedy et al. (2013). These two issues limit ANNs applicability in the real world and present potential security risks when deployed.

---

Published as a conference paper at ICLR 2020
There have been two main approaches for investigating ANN robustness: adversarial machine learn- ing and training data manipulation (Ford et al. (019)). Adversarial machine learning aims to de- velop novel attack methods which perturb the input minimally while changing the ANN's classifica- tion outcome (Moosavi-Dezlonn
et al. (2016); Carlini & Wagner| (017);; Goodfellow et al. (0014);
Athalye al. (017): Nguyen et al. al.] (2015) well as to design defense mechanisms which prevent
these attacks from affecting ANN behavior (Papernot et al. (201 6b);; Goodfellow et al. (201)) Huang ct al.] (2015), see Yuan et al. (2019) for review). Training data manipulation research typically ex- amines the impact of changing the input distribution during testing and observing the effect on ANN performance. Geirhos et al. (018)) showed that ANNs trained on images with one type of distortion may not perform well when tested on other types of distortions, even if images with both distortions appear identical to the human eye. Likewise, ANNs trained on unperturbed images exhibit reduced performance when images in the test set are distorted, for example, through horizontal translations, blurring, or the addition of compression artifacts (Dodge & Karam (2016); Vasiljevic et al. (2016); Zhou al. (2017). Although it has been proposed that adversarial and manipulation robustness can be increased through various mechanisms during the training phase, such as fine-tuning, recent research has shown that these methods are mostly ineffective their effectiveness is inconclusive (Geirhos et al. (0188); Uesato et al. (2018): Athalye et al. (2018)).
It has been hypothesized that in the mammalian brain sleep helps create generalized represen- tations of an input learned during the awake state (Stickgold & Walker (2013); Lewis & Durrant (201T)) ). Sleep has been identified as being critical for memory consolidation
process of convert- ing recent memories into long-tern storage (Rasch & Born (2013)), During sleep, there is reacti- vation of neurons involved in previously learned activity (Stickgold (005)) and this reactivation is likely to invoke the same spatio-tempore pattern of neuronal firing as the pattern observed during training in the awake state (Wilson & McNaughton 1994). Sleep reactivation, or replay, serves to strengthen synapses involved in a learned task through local synaptic plasticity, such as Spike Time Dependent Plasticity (STDP). Plastic changes during sleep can increase subject's s ability to form connections between memories and to generalize knowledge learned during the awake state (Payne) jet al.] (2009).. In one study (Wamsley et al.] (2010)), subjects learned to find an exit to a maze in virtual 3D environment. Subjects who were allowed to sleep exhibited more complex under- standing of the overall shape of the maze (Wamsley et al. (2010)). Using biophysical model of a cortical network Gonzalez et al. (2019) and Wei et al. (2018) showed that sleep dynamics promotes reactivation and helps to create distinct representations for unique memories by devoting synapses to specific memory traces. This body of neuroscience work suggests that sleep-like activity may be applied to ANNs to enable the network to extract the gist of the training data without being con- strained by the statistics of specific training data set. Our specific hypothesis that sleep phase could aid in reducing neural network's 's susceptibility to adversarial attacks and to increase gener- alization performance by reducing the impact that imperceptible input changes can have on the task output.
In this new work, we propose a sleep-inspired algorithm to defend against adversarial attacks as well as to increase ANN robustness to noise. We utilize the notion of sleep from biology and apply an off-line unsupervised "sleep" phase to modify the parameters of a fully connected ANN. We demonstrate number of performance improvements over existing defense algorithms, such fine- tuning or adversarial retraining and defensive distillation, on both adversarial and noise robustness. The contributions are summarized below:
We analyze how robust the proposed sleep algorithm is to four different types of adversarial attacks three different datasets (MNIST, CUB200, and toy dataset). For most condi- tions (MNIST, toy dataset), after sleep phase was applied, the attacks consistently resulted in adversarial examples that were more distinct from the original input compared to the adversarial examples designed for the original (before sleep) network.
We illustrate that the sleep algorithm creates a more robust network whereby performance on noisy and blurred inputs is higher compared to control or defensively distilled network and is more robust to the other types of distortions compared to ANNs that are fine-tuned on a single distortion.
We analyze the impact of the sleep algorithm on task representation and demonstrate that the algorithm creates decision boundaries that more closely resemble the true classes, ef- fectively extracting the gist of the data.
2

---

Published as a conference paper at ICLR 2020
2 ADVERSARIAL ATTACKS AND DISTORTIONS
Adversarial attacks aim to create minimal perturbations that, while imperceptible to the human eyc, fool ANNs. These attacks range from white-box to black-box attacks, based on how much infor- mation they assume the attacker to possess about the network. White-box attacks assume that the attacker has access to the network architecture, training data and weights. These attacks can range from absolute information, such as gradient-based attacks which compute the gradient of the loss with respect to the input (Brendel et al. (2017)) score-based attacks which only utilize predicted scores of the model. Black-box attacks, which assume no knowledge about the network, solely rely on the decision made in order to craft adversarial examples. Attacks can be (a) targeted such that the attacker aims to create an adversarial example that the network would predict as a certain class or (b) untargeted where the attacker's goal simply to cause any kind of misclassiffiattinn (Biggio & Roli (2018))
In this work we consider four types of adversarial attacks ranging from white-box to black-box attacks. We assume that the attacker solely wants to cause misclassification, with no respect to the output class. We present a brief description of each of the four attacks below (see Appendix for examples of images created by these attacks).
Fast Gradient Sign Method (FGSM). FGSM (Goodfellow et al. (2014) computes the sign of the gradient of the loss function (J) with respect to the original input æ using the weights 0 of the network and the target labels y. = x + csign (Vagm(VV.11 (.,J, x)).
This represents the direction to change each pixel in the original input in order to increase the loss function. Based on the value of c, the corresponding perturbation to the original image can range from small to large. Thus, in this work we use the average of the smallest values of e needed to create adversarial example x' (misclassified input) for each input in the testing set.
DoepFool. DeepFool (Moosavi-Dezfooli et al. (016)) is an iterative method which approximates the nearest decision boundary the input at time t and moves the input I; in that direction to compute Tt+1- This process is repeated until a misclassification is produced or the runtime of the simulation is exceeded. For this attack, we measure the L-2norm between the original input and the adversarial input x². Thus, successful defenses should result in high L2-norm for this algorithm.
lacobian-based Saliency Map (JSMA). JSMA (Papernot et al. (2166a) aims to craft adversarial examples that minimize the L0-norm of x - x' by reducing the number of pixels that are altered. In summary, the algorithm computes the gradient, as done in FGSM but for all possible classes. These gradient values represent how changing each pixel contributes to the overall loss function, with large values indicating a significant effect on the loss. These values are used to create a saliency map, where each pixel's impact on the loss is modelled. The algorithm utilizes this saliency map to alter individual pixels, repeating the gradient and saliency map computation until an adversarial example is created. For this type of attack, we utilize the L2-norm to determine defense success.
Boundary Attack. The Boundary Attack (Brendel et al. (017)) is a black- box attack which relics solely the decision of the ANN to craft adversarial example. Given an input æ, random input ao is chosen such that f(x) * f(ah), where f(x) is the label produced by the ANN. In our work, To is chosen from uniform distribution. The attack starts by moving zó toward until it reaches the point where f(x) = f(z)), or the decision boundary in between f(x) and f(x)). From here, the attack consists of two steps: an orthogonal perturbation and a forward perturbation. During the orthogonal perturbation, random points along the hypersphere around f(z)) are sampled. Those that are adversarial and closer than before are added the queue for forward perturbation. During
the forward perturbation, small step is taken from of to x as long as f(x) o f(x(). This process repeated until a convergence criterion met. For this attack, we utilize the 1.2-norm to define defense success.
Distortions. Although not specifically designed to attack an ANN, distortions negatively impact ANN performance. In this work we consider two simple distortion techniques: blurring and Gaus- sian noise. In first case, we perform 2-D Gaussian filtering with a blur kernel of varying standard deviation in order to blur the images. In second case, we add Gaussian noise with mean 0 and standard deviation o. These distortions are tested in the networks implementing the proposed sleep algorithm as well as using the adversarial defenses discussed below.
3

---

Published as a conference paper at ICLR 2020
3 ADVERSARIAL DEFENSES
We compare our sleep algorithm with two existing adversarial defenses: defensive distillation and fine-tuning, or adversarial retraining. Defensive distillation (Papernot et al. (2016b)) utilizes two training sessions in order to create a distilled network. First, an initial network is trained on (X, where X is the training data, and Y is the one-hot encoded training labels. The activation function of this network changed such that the softmax function of the output layer is computed using a temperature term I' as follows:
F(x)
1X) Ei= e T
A higher T forces the ANN to produce larger probability values for each class, whereas lower T values support similar representation as the one-hot encoded labels. After the first network is trained, the output of the network (prohability values) is used to train a distilled network with the same eertrax-teempraature function. Previous work has shown this approach preventing some types of attacks (Papernot et al. (2016b)). However, others have shown that it is not successful defending against modified versions of those attacks or novel attacks in general (Carlini & Wagner 2016: 2017 ) Based on the previous work which found that temperature values between 20 and 100 effectively prevent adversarial attacks (Papernot et al. (2016b)) we use temperature value of T = 50 in our implementation of defensive distillation.
be successful at
Adversarial retraining aims fine-tune the network on adversarial examples with the correct labels as a form of regularization. Previous work has shown that adversarial retraining can mitigate the effectiveness of some adversarial attacks. Goodfellow et al. (2014) showed that adversarial retrain- ing can reduce the error rate MNIST, demonstrating greater ANN robustness after fine-tuning. Likewise, Moosavi-Dezfoool et al. (2016) showed that fine-tuning on DeepFool attacks can reduce the effectiveness of their attacks. However, they observed that fine-tuning FGSM attacks has negative results, actually increasing the strength of the attack. This suggests that fine-tuning may overfit the network to certain attacks, while failing to extrapolate other attacks, similar to results shown for generalization in ANNs (Geirhos et al. (2018)). For the adversarial retraining procedure presented here, train the network the original input and then fine-tune the network various adversarial attacks with reduced learning rate.
4 SLEEP ALGORITHM
The basic intuition behind the sleep algorithm that a period of offline activity, whereby network weights are modified according to an unsupervised learning algorithm, allows the parameters of the network to become more reflective of the underlying statistics of the task at hand, while not overfitting the statistics of the training data. The pseudocode is presented in Algorithm In short, an ANN is trained using stochastic gradient descent and the standard backpropagation algorithm (exact parameters used for each of the datasets shown in Table 2). After training, the network structure is converted into spiking neural network (SNN). After building the SNN, we run a sleep phase which modifies the network connectivity based spike-timing dependent plasticity (STDP). After the sleep phase, the SNN network converted back into the ANN and testing is performed.
4.1 SPIKING NEURAL NETWORKS
SNNs seek to model closely temporal brain dynamics. In short, SNNs are composed of spiking neurons and model the information transformation and the dependence on exact timing of spikes that occurs in biological networks (Ghosh-Dastidar & Adeli (2009)) Individual neuron models can range from simple integrate-and-He type neurons which sum their inputs and produce an output (spike) if this exceeds some firing threshold to more complex Hodgkin-Huxley type neurons which model sodium-, potassium-, and caloride-channed kinetics (Abbott & Kepler (1990)). Recent work has shown that near loss-less conversion between ANNs and SNNs can be achieved by propagating activity through a spiking neural network for a given input and counting the number of times that each output neuron fires (Diehl al. (2015)).
4

---

Published as a conference paper at ICLR 2020
To convert ANN to SNN (Lines 1-3 of pseudocode), we assume the ANN utilizes ReLU neurons with no bias. This assumption is made so that the output neuron's 's activation can be treated as a firing rate, either zero positive, and that the thresholds of all neurons in given layer are of the same scale. The weights from the ANN are directly mapped to the SNN. In our analysis, each unit in the SNN is modelled as interrae--andfire type neuron, computing the following equation:
N
dv di = - u(t) +
Σ w. * s(t).
Here, Tm represents the decay constant of the membrane potential, is the voltage at given time, wi is the weight connecting from neuron i, and s(i) is the spiking activity of neuron i, either I or O.
4.2 PLASTICITY AND SLEEP
The key advantage of using SNN is that biologically inspired training rules can be applied while the network is driven by noisy input. Empirical data suggest that the brain uses spike-timing dependent plasticity (STDP) (Song et al. 2000). where weight updates depend on the relative timing of pre- and post-synaptic spikes. It has been shown that STDP results in balanced activity, where all neurons fire in equal proportions (Song et al. (2000). Here we utilize a modified version of STDP: if pre- synaptic spike induces post-synaptic spike, then the weight between these neurons is increased. If a post-synaptic spike occurs, but the pre-synaptic neuron does not spike, then the corresponding weight is decreased (in this case postsynaptic spiking may occur because of spiking in other neurons connecting to that post-synaptic neuron).
The sleep training phase we propose here can be described as following. First, inputs to each neu- ron of the input layer must be presented as spiking activity in order to propagate activity from the input layer to the hidden layers of the network. We convert inputs (real-valued pixel intensities or features) to spikes by defining maximum firing rate fmaz with units spikes and computing spike raster, such that inputs with higher values (i.c. brighter pixels) will have higher rate than inputs with lower values, with no spike rates exceeding fmax- Next, activity is prop- agated through the network as spikes and the STDP rule is applied to update weights. In biological networks, increase of synaptic strength during slow-wave sleep leads characteristic patterns of ac- tivity with repetitive periods of elevated firing (Up-states), when previously learned memory traces are spontaneously replayed. To simulate this dynamics, synaptic weights in SNN are up-scaled to induce high firing rates in later layers. Other important parameters include the threshold for each layer and the length of sleep. The parameters used for each dataset are presented in Table 3
4.3 EXPERIMENTS AND DATASETS
Below, we describe the general experimental setup as well as the datasets tested. First, we trained control ANN using the training set for each of the main datasets used in this study. Next, we created defensively distilled network using T 50 for the temperature parameter to create the second test network. Then, we fine-tuned the control ANN on a specific attack or distortion method to create the third test network. Finally, we converted the control ANN to SNN and applied the sleep algorithm as described above to create the fourth test network. We created adversarial examples for each of these four networks using the attacks we described above (fine-tuned networks are tested on the attacks they were fine-tuned on). Then, we analyze how successful each attack is to fool each of the four networks using the metrics defined above. For generalization (blur and noise), we performed the same setup as above creating four different networks. We then tested each network on varying levels of distortion. We tested networks fine-tuned on blurred and noisy images to measure how performance generalizes across distortion methods. We averaged performance across a minimum of three networks for each attack and distortion.
We used three datasets to compare performance: Patches (a toy dataset created simply for analysis), MNIST [LeCun et al. (1998), and CUB-200 (Welinder et al. 2010)) ). Patches consists of four binary images arranged in a 10x10 square. Each image has its own label (1-4), and consists of 25 bright pixels (value set to 1) and 75 dark pixels. The overlap of bright pixels among the four images (see Appendix] Appendix) is chosen such that the task is not trivial. The MNIST dataset consists of 70,000 28x28 greyscale images of handwritten digits, with 60,000 in the training set and 10,000 in the testing set. CUB-200 is a high resolution dataset of images of birds with 200 bird species,
5

---

We evaluate the sleep algorithm in two settings: (1) Adversarial attacks designed to fool neural networks and (2) generalization distortions designed reflect imperfect viewing conditions or other types of noise. For adversarial attacks (other than FGSM), we utilize the following metric to evaluate the success of each defense. Let a/ be the adversarial example created for input z;. The total score Published as a conference paper at ICLR 2020
Algorithm Sleep:
1: procedure CONVERTANNTOSNNNnn
2:
Map the weights from (nn) with ReLU units network of integrate- fire units (snn)
3:
Apply weight normalization and return scale for each layer ([24)) return snm, scales
4: procedure CONVERTSNNTOANNT
5: Directly map the weights from integrate-fire network (nm) to ReLU network (ann) return
ann
6: procedure SLEEP(nn, I, scales)
> I is input
7:
Initialize (voltage) 0 vectors for all neurons for t t to I's do
8:
T's duration of sleep
9:
5(,, k- Convert input I Poisson spiking activity
10:
for 1 < 2 to n do
n number of layers
11:
o(l, - v(l, t - 1) + (scales(l- (L, 1)5((1 - 1,t)) w(1.-1) weights S(1,t) + v(l,t) threshold(1)
12:
P Propagate spikes
[W(L, inc if S(1, t) 5& 1,t)
13:
W(I1 - [W(1 dec
STDP
S(1, 5((
14:
procedure MAIN
15:
Initialize neural network (amn) with ReLU neurons and bias 0.
Train amn using backpropagation. snn, scales = ConvertANNNoSNNamm) snn Sleep(snn, Training data X, scales)
17:
19:
= ConverSNNoONN((mnn))
with very few 30) images per class. For this dataset, we used previously extracted ResNet-50 embeddings, where ResNet-50 was pre-trained on ImageNet (He et al. (2016)). For CUB-200, we do report results for blurring, since we are using extracted features, not images.
RESULTS
SA for an attack is the median squared 1.2-distance for all samples, where N is the dimension of the space:
SA median( N 1ar4 1:)12))
For FGSM, we define the following metric which computes the median minimum noise level needed to produce a misclassification across all samples:
SYGsM median (m)n(()) s.t. f(r + c ++)) f()).
For MNIST and CUB-200, we evaluate the attacks on all examples in the testing set. Examples that the networks get wrong before the attack implemented discarded from the analysis (in these cases ||ا 21| and e, for all attacks). For FGSM and distortions, we also include plots of classification accuracy as function of noise level. For DeepFool and JSMA, we report adversarial attack accuracy (number of examples where f(x) = y and (ax) * (x)) where y the correct label, over number of examples tested). Note that these algorithms would always produce an adversarial example if allowed to run forever. However, due to computational limitations, we included run-time limits on the number of iterations for these algorithms (see Appendix). Thus, a lower adversarial attack accuracy indicates that the attack would need more iterations to run to reach 100% accuracy. This is similar measure distance since more iterations would result in more distinct adversaries for all attacks implemented and the updates at each iteration have the same magnitude for each defense.

---

Published as a conference paper at ICLR 2020
Patches
Control
Defensive Distillation
Fine-tuning
Sleep
FGSM
0.0175
0.05
0. 1.255
0.2025
DeepFool
0.0440(95.00$%
0.0360 (90.00%)
0.0201 (100.0%)
0.0124 (100.0%)
JSMA
0.0049 (80.00%)
0.0135 (70.00%)
0.0450 (100.0%)
0.0541 (100.0%)
Boundary Attack
0.2971
0.3124
0.1772
0.3515
MNIST
Control
Defensive Distillation
Fine-tuning
Sleep
FGSM
0.0900
0.0900
0.1000
0.2200
DeepFool
0.0042 (96,46%)
0.0043 (96.42%)
0.0074 (97.40%)
0.0484 (86.38 %
JSMA
0.0090 (99.29%)
0.0086 (99.41%)
0.0133 (98.77%)
0.0059 (72.97 %
Boundary Attack
0.0525
0.0525
0.0544
0.0488
CUB-200
Control
Defensive Distillation
Fine-tuning
Sleep
FGSM
0.0550
0,0500
0.0650
0.0600
DeepFool
0.0027 (82.23%)
0.0019 164.16%)
0.0044 (83.02 % 0.0530 (95.38%)
0.0025 (83. 12%) 0.0439 (95.15% ) 0.9967
JSMA
0.0477 (95.56%)
0.0347 (95.88%)
Boundary Attack
0.9751
0.9034
0.9976
Table 1: Adversarial Attack Scores (Best defense scores are bolded, lowest attack success rates are in blue)
A
B
C
Patches
MNIST
CUB-200
110
100
100
100
90
90
Control
Accuracy aa
90
80
80
Def. Dist.
Finetuning
70
70
80
Sleep
60
60
70
50
50
60
40
40
50
30
30
40
20
20
30
10
10
20
0
0
0
0.1 0.2
0.3
0.4
0.1
0.2
0.3
0.4
0
0.1
0.2
0.3
0.4
Epsilon
Epsilon
Epsilon
Figure 1: FGSM classification accuracy function of noise added (Epsilon) for the three datasets
tested: A) Patches B) MNIST C) CUB-200.
5.1 ADVERSARIAL ATTACKS
Here we report the scores for all different attacks and for the all datasets. For the FGSM attack, the sleep algorithm increases the median minimum noise needed for misclassification for all three datasets compared to the control network (also see Figure []. For MNIST dataset, the amount of noise needed to fool the network after the sleep algorithm was almost double of that needed for either the fine-tuning or defensive distillation approaches. For the Patches dataset, both defensive distillation and fine-tuning increase the robustness of the network. However, fine-tuning and sleep were able to defend, albeit marginally, against the FGSM attack. Looking at the classification accuracy of the network function of noise added (c, Figure T.. observe that
CUB-200, only

---

Published as a conference paper at ICLR 2020
in the Patches and CUB-200 dataset, sleep tends to have higher classification accuracy than the other methods for epsilon < 0.1. After this point, sleep tends to have equal classification accuracies as compared to the other methods. For MNIST, the baseline classification accuracy on the original test set decreases slightly compared to the other methods (80% after sleep). However, the performance remains high longer than for the other defense methods images that were correctly classified. We observed that performance continued to drop after sufficiently large amount of noise was added. This is biologically plausible as adding more noise an image should result in image degradation and misclassifications. In sum, these results indicate that sleep phase can successfully mitigate FGSM, more so than a control network.
For DeepFool, sleep has a significant effect on the defense score on the MNIST dataset, both re- ducing the attack success rate and increasing the distance between the adversarial example and the original input by an order of magnitude. For Patches and CUB-200 this effect is less pronounced, with fine-tuning or the control network performing better. We hypothesize that sleep was ineffective in preventing the DeepFool attack in tasks with very few exemplars per class (Patches) number of classes (CUB-200). In CUB-200, there is a large number of classes so the distance be- tween the input and the nearest decision boundary is smaller (this is supported by the fact that JSMA, an LO attack, does worse than DeepFool for CUB-200 and vice versa for MNIST, control networks). In this case, sleep is unable to move the decision boundary of one class without impinging on the decision space of another class. In MNIST, where the decision space for one class is presumably larger, sleep can alter decision boundaries in a way that has a minimal effect on other classes.
large
Sleep successfully increases the network's 's robustness to the JSMA attacks on MNIST and Patches, reducing the attack success rate in the case of MNIST and increasing the distance needed create an adversary for Patches. On CUB-200, there is a marginal reduction in the adversarial attack accuracy compared to the control network. Defensive distillation and fine-tumi tine-tuning also reduce JSMA's effectiveness. However, for these two defenses, in the case of MNIST, the networks were capable of finding an adversary for a higher percentage of the testing set. Thus, the effect of changing a small number of important pixels is mitigated after running the sleep algorithm.
For the Boundary Attack, we found that no defense mechanism helps compared to the control in decreasing the attack's effectiveness the MNIST dataset. However, for CUB-200 and Patches, the sleep algorithm results in a higher defense score than that for the control network. This lends support to the idea that sleep favorably alters decision boundaries that it becomes harder to find an adversarial example that is close to the original image after the sleep phase. This also suggests that sleep is not not simply obfuscating gradients, which has been common criticism of several adversarial defenses (Athalye et al. (2018), which are tested on white-box attacks. In fact, given the long run-time for convergence of this algorithm, if we define threshold for adversarial attack success (L2-norm > 1), then sleep successfully defends against this attack on the MNIST dataset (see Table
Why does sleep phase help? It has been shown that sleep tends to promote an increase in stronger weights while pruning weaker weights, thus increasing the width of the weights' distribution (Gon- zalez et al., 2019). This results in the consolidation of strong memories at the cost of diminishing weak memories. From this point of view, a memory is subspace or abstraction in the decision space corresponding to given class. Sleep may result in enlarging the subspace the network allocates to a stronger category while shrinking weaker ones (Figure The process of strengthening the strong memory also results in making robust and noise invariant, as seen in Figure 5B where the first 8 categories (numbers 0-7) are strengthened and become more invariant to the FGSM attack, while the last two digits are essentially forgotten and the network cannot confidently predict exemplars from these classes (Figure 5P)) If the noise is less targeted, as in the case of random noise or blurring, sleep does not need to alter the decision space much produce better generalization and can maintain a high baseline accuracy, as we demonstrate in the next section.
5.2 GENERALIZATION
Figure shows the network performance for noisy and blurry distortions of data for MNIST (A) as well noisy distortions for the CUB-200 feature embeddings (B, see Figure B for results on Patches). Overall, fine-tuning on an image distortion results in the best performance for that specific distortion. However, was noted (Geirhos et al. (2018)), fine-tuning a specific distortion does
8

---

Published as a conference paper at ICLR 2020
A
MNIST
B
CUB-200
100
100
70
Control
90
80
60
Def. Dist.
Accuracy
80
Accuracy
Sleep
Finetuning
70
60
Control
60
40
30-
Def. Dist.
50
Finetune-noise
20
20
Finetune-blur
40
Sleep
10
30
0
0. 0
0.2
0.4
0.6
0.8
1
2 Blur Sigma
2
Noise STD
Noise STD
Figure 2: Sleep increases robustness to general distortions. A) Generalization classification accuracy for 5 networks for noise and blur on the MNIST dataset. B) Generalization classification accuracy for 4 networks for the noisy CUB-200 task. Note that there are only 4 networks because there is no blur task here.
not extend to other types of distortions. In our analysis, fine-tuning the network blurred MNIST images results in high performance (> 80%) on blurred images. However, for noisy images, this performance was only marginally above the control network. The sleep algorithm increased perfor- mance for both distortion methods, since this approach is not tailored to any one representation of the training set.
Finally we tested how sleep increases robustness on blur and noise distortions. In biological systems, sleep increases generalization through replay of memories learned during awake which leads to changes in synaptic weights. These changes entail both an increase in synaptic weights associated with a specific task and pruning of synapses involved in other tasks (Gonzalez et al. 2019 Tononi and & Cirelli, (006). Figures and [0] show that correlations among like digits in the hidden layers of network are greater after applying sleep than before for noisy and blurred images. Likewise, pairs of different digits usually become decorrelated after sleep, suggesting synaptic pruning. We also show that both normalized spiking activity and activations of digit-specific neurons are higher after sleep than before (Figures and and [2] see Appenddix for details). These results suggest that the sleep algorithm increases robustness through biologically plausible learning mechanisms involving replay of relevant activity during sleep phase.
CONCLUSIONS AND FUTURE DIRECTIONS
In this work, we show that a biologically inspired sleep algorithm can increase an ANN's robust- ness to both adversarial attacks and general image distortions. The algorithm augments the normal (e.g., back-propagation based) training phase of an ANN with an unsupervised learning phase in the equivalent SNN modelled after how the biological brain utilises sleep to improve learning. We hypothesize that the unsupervised sleep phase creates more natural feature representations which in turn lead to more natural decision boundaries, thus increasing the robustness of the network. Al- though this robustness may come at cost of overall accuracy, it has been shown that robustness may have multiple important benefits, such as more salient feature representations as well as invariance to input modifications (Tsipras et al. (2018)). We also show that the trade-off between robustness and accuracy does not always occur, particularly for image distortions such as noise or blur. Future work includes converting the sleep algorithm into regularization technique to be applied in more

---

Published as a conference paper at ICLR 2020
standardized machine learning frameworks well as understanding the theoretical basis for the beneficial role of spike based plasticity rules in increasing network robustness.
7 ACKNOWLEDGEMENTS
This work was supported by the Lifelong Learning Machines program from DARPA/MTO (HR00011-1-2-0002) and ONR (MURT: N00014-16-1-2829)
REFERENCES
LF Abbott and Thomas B Kepler. Model neurons: from hodgkin-huxley to hopfield. In Statistical mechanics of neural networks, pp. 5-18. Springer, 1990.
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. arXiv preprint arXiv:1707.07397 2017.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Cireumventing defenses to adversarial examples. arXiv preprint urXiv:/802.000 2018.
Battista Biggio and Fabio Roli. Wild patterns: Ten years after the rise of adversarial machine learning. Putlern Recognition, 84:317-331, 2018.
Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision- based adversarial attacks: Reliable attacks against black-box machine learning models. arXiv preprint arXiv:1712.04245 2017.
Nicholas Carlini and David Wagner. Defensive distillation is not robust to adversarial examples. arXiv preprint arXiv: 16077.4311 2016.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 39-57. IEEE, 2017
Peter U Dichl, Daniel Neil, Jonathan Binas, Matthew Cook, Shih-Chi Liu, and Michael Pfciffer. Fast-classifying. high-accuracy spiking deep networks through weight and threshold balancing. In 2015 International Joint Conference Neural Networks (LJCNN), pp. 1 1-8. IEEF, 2015.
Samuel Dodge and Lina Karam. Understanding how image quality affects deep neural networks. In 2016 eighth international conference on quality of multimedia experience (QoMEX), pp. 1-6. IEEE, 2016.
Nic Ford, Justin Gilmer, Nicolas Carlini, and Dogus Cubuk. Adversarial examples are natural consequence of test error in noise. arXiv preprint arXiv:1901.10511 2019.
Robert Geirhos, Carlos RM Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Felix A
Wichmann. Generalisation in humans and deep neural networks. In Advances in Neural Informa- tion Processing Systems, pp. 7538-7550, 2018.
Samanwoy Ghosh-Dastidar and Hojjat Adeli. Spiking neural networks. International journal of neural systems, 19(04))295-308 2009.
Oscar C Gonzalez, Yury Sokolov, Giri Krishnan, and Maxim Bazhenov. Can sleep protect memories from catastrophic forgetting? BioRxiv, pp. 569038, 2019.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778. 2016.
Ruitong Huang, Bing Xu, Dale Schuurmans, and Csaba Szepesvari. Learning with a strong adver- sary. arXiv preprint arXiv: /51 1.03034, 2015.
10

---

Published as a conference paper at ICLR 2020
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 (((11)2228-23244 1998.
Penelope A Lewis and Simon J Durrant. Overlapping memory replay during sleep builds cognitive schemata. Trends in cognitive sciences, 15(8):343331, 2011.
Seyed-Mohsen Moosavi-Dezfooll Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 2574-2582, 2016.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confi- dence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 427-436, 2015.
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Cclik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In 2016 IEEE European Sympo- sium on Security and Privacy (EuroS&P), pp. 372-387. IEEE, 2016a.
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense adversarial perturbations against deep neural networks. In 2016 IEEE Symposium on Security and Privacy (SP), pp. 582-597. IEEE, 2016b.
Jessica Payne, Daniel L Schacter, Ruth Propper, Li-Wen Huang, Erin Wamsley, Matthew A Tucker, Matthew P Walker, and Robert Stickgold. The role of sleep in false memory formation. Neurobiology of learning and memory, 92(3):327-334 2009.
Björn Rasch and Jan Born. About sleep's role in memory. Physiological reviews, 93(2))681-766 2013.
Jonas Rauber, Wieland Brendel, and Matthias Bethge. Foolbox: A python toolbox to benchmark the robustness machine learning models. arXiv preprint urXiv:1707.0413.1 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.
Sen Song, Kenneth D Miller, and Larry Abbott. Competitive hebbian learning through spike- timing-dependent synaptic plasticity. Nature neuroscience, 3(9):919, 2000.
Robert Stickgold. Sleep-dependent memory consolidation. Nature, 437(7063)) 2005.
Robert Stickgold and Matthew P Walker. Sleep-dependent memory triage: evolving generalization through selective processing, Nature neuroscience, 16(2):139, 2013.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.61990 2013.
Giulio Tononi and Chiara Cirelli. Sleep function and synaptic homeostasis. Sleep medicine reviews, 10(1):49-62, 2006.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be odds with accuracy. arXiv preprint 1805.:1800..2113 2018.
Jonathan Uesato, Brendan O' D'onogguu, Aaron van den Oord, and Pushmeet Kohli. Adversarial risk and the dangers of evaluating against weak attacks. arXiv preprint arXiv: 100.00680,.05666, 2018.
Igor Vasiljevic, Ayan Chakrabarti, and Gregory Shakhnarovich. Examining the impact of blur on recognition by convolutional networks. arXiv preprint 161.:7661 2016.
11

---

Published as a conference paper at ICLR 2020
Erin J Wamsley, Matthew A Tucker, Jessica D Payne, and Robert Stickgold. A brief nap is beneficial for human learnin:: The role of navigation experience and eeg spectral power. Learning & Memory, 17(7):332-333 2010.
Yina Wei, Giri Krishnan, Maxim Komarov, and Maxim Bazhenov. Differential roles of sleep spindles and sleep slow oscillations in memory consolidation. PLoS computational biology, 14 (7):e1006322, 2018.
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-ucsd birds 200. 2010.
Matthew A Wilson and Bruce L McNaughton. Reactivation of hippocampal ensemble memories during sleep. Science, 265(5172):676677,, 1994.
Xiaoyong Yuan, Pan He, Qile Zhu, and Xiaolin Li. Adversarial examples: Attacks and defenses for deep learning. IEEE transactions neural networks and learning systems, 2019.
Yiren Zhou, Sibo Song, and Ngai-Man Cheung. On classification of distorted images with deep convolutional neural networks. In 2017 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1213-1217. IEEE, 2017.
APPENDIX
8.1
TRAINING PARAMETERS
Here, we define the neural network parameters used for each of the three datasets as well as the sleep, defensive distillation, and fine-tuning parameters. Table[]showe parameters used to train each the control networks discussed in the paper. All neural networks were trained with ReLU neurons. Table 3 shows the parameters used during sleep for each of the three datasets. Note that these parameters for MNIST and CUB-200 were chosen by running a genetic algorithm to maximimize performance on the FGSM attack (performance was determined based on the training set so as not to overfit to the set). For the other three attacks, parameters that maximized FGSM performance were used. Also, for noise and blur generalization, different parameters were chosen (not shown here).
Patches
MNIST
CUB200
Architecture
[100, 4]
184,, 1200, 1200, 10]
[2048, 350, 300, 200]
Learning Rate
0.1
0.1
0.1
Momentum
0.5
0.5
0.5
Dropout
0
0.2
0.25
Epochs
100
Table 2: Parameters used to train the control network for each of the three datasets. Architecture refers to number of units per layer. For example, the MNIST network possessed 1 input layer, hidden layers with 1200 units, and an output unit with 10 units.
Patches
MNIST
CUB200
Input Rate
16 Hz
40 Hz
79 Hz
Sleep Duration
3000
27105
11751
Thresholds
1.0450, 0.7150, 0.3850
36.18, 18, 23.36, 36.38
2.69, 4.61, 2.63
Synaptic AMPA current
4.25
2.19
4.15
Increase factor
0.0035
0.063
0.0016
Decrease factor
0.0002
0.069
0.000209
Table Parameters used during sleep. Input rate Fmazs the maximum firing rate of input neurons, Sleep duration = length of sleep (number of images presented during sleep. Thresholds neuronal firing thresholds for each layer of neurons, Synaptic AMPA current = factor to scale the weights by during sleep, Increase and Decrease factor amount weights modified on STDP event.
12

---

Published as a conference paper at ICLR 2020
A
B
Patches Overap
Noise
Increasing
Blur
C
D
Noise generalization
Blu generalization
110
110
100
01
Weight Spread
90
88
Accuracy
80
70
60-
50
40-
Control
30
Fiettrinn-rrbee
Feeturing-tta Abu
20-
0.2
0.4
0.8
05
1.6
Dontol
Del
Fireuning
Fineluing
Sleep
Noise std
Sigma
Figure 3: A) Patches dataset example four binary images with 25 pixels turned on in each image and 15 pixel overlap. B)Types of images tested on for generalization for the Patches dataset. Top Images with Gaussian noise added with increasing variannce (from 0 to 1.0 in steps of 0.2). Bottom Gaussian blurred images with increasing sigma (from 0 to 2.5 in steps of 0.5). C) Generalization accuracy for noise and blur of five different networks tested (Control, Defensively distilled, Fine- tuned-blur, fine-tuned-noise, and Sleep) for the Patches dataset. D) Weight spread for each of the networks tested in C.
For the defensively distilled networks tested in the paper, we first train an initial network using temperature of 50. Then, we use the training set to compute soft labels and finetune the initial network on these soft labels for the same number of epochs and with the same learning rate.
For the fine-tuned networks, we take the control networks trained with the parameters shown in Table [ The learning rate is reduced 0.05 and the network fine-tuned on mixture of either adversarial attacks, blur noise and the original images/features. For CUB-200, perform fine- tuning for 10 epochs.
8.2 PATCHES ANALYSIS
The Patches dataset represents an easily interpretable example where can understand what hap- pens to the weights after sleep. Figure shows an example of the dataset. Here, we have images each belonging pixels are dark. There a 15 pixel overlap, so that weights connecting from input to output layer must take this into account in order to separate the images. Figure 3 illustrate the blur and noise distortions tested for this dataset and Figure BC shows the results for the blur and noise distortions.
different classes. 25 pixels whitened in each image and the remaining 75
After the network is trained, neurons to the output neurons (see Figure 4 top row). We theorize that optimally robust behav- ior would oceur when weights connecting from ON-pixels positive, weights connecting from
analyze the weights connecting from each of the 100 input
13

---

Published as a conference paper at ICLR 2020
overlapping pixels near O, and weights connecting from OFF-pixels negative. In this case, changing the value of overlapping pixels will have no effect on classification. Changing the value of OFF-pixels will cause the network predict another class, where OFF-pixels may be ON-pixels or indicative of that class. Changing the value of ON-pixels will only have a negative impact the brightness of the pixel is reduced significantly. Thus, in this circumstance, the network should behave robustly.
In the control network, we observe that weights connecting from ON-pixels (pixel-value in- crease while weights connecting from OFF-pixels remain at O. Weights connecting from overlap- ping pixels remain near 0 or positive. Defensive distillation causes some weights connecting from overlapping pixels to decrease, likely because the soft labels used in defensive distillation cause overlapping pixel units alter the probability values computed by the network in such way that does not truly reflect the impact of the overlapping pixels. In the fine-tuned networks (both on blurred images and noisy images), we observe an increase ON-pixel weights and an increase in noisiness of OFF-pixel weights. Likewise, in the sleep network, OFF-pixel weights become negative while ON-pixel weights remain the same. In these cases, robustness is increased as weights become more similar to hypothesized ideal weights. Essentially, the magnitude of input changes need to change classification increase since the spread between ON- ON-pieel weights and OFF-pixel weights increases. We quantify the spread in weights by taking the difference between the average weight connecting from ON-pixels and the average weight connecting from OFF-pixels. This represents the mean input that each correct output neuron receives. This result shown in Figure 3D. Of note is that this weight spread is increased for both the sleep and finetuning-note noise network, suggesting that these defenses bring the weights closer their ideal values for computing robustness.
14

---

Published as a conference paper at ICLR 2020
A
ON-pixels
Overlapping Pixels
OFF-pixels
0.1
011
011
Control
00
oB
00
0
0000g
foo
D
0000000
00
00
-01
4.11
-0.1
-01
0
100
50
100
0
50
100
100
0.1g
0.2
02-
Def. Disti.
0
08o
0.1
4.2
-02
0.1
100
50
100
Finetuning Noise
0.5
022
80
00000
-.2
0
100
50
100
50
100
Finetuning Blur
02
022
02
02
00
00
60
Q80
000
00
-22
-02
-02
-02
0
50
100
50
100
0
50
100
0
100
0.1
0.1
0.1
0.1
a20
Sleep
10000
00
@
@ o000g
foo
o 0D
0000
900
0000
0g
00
290 000 oe
-11
0,1
4.1
-01
0
50
100
50
100
0
50
100
100
Neuron ID
Neuron ID
Neuron ID
Neuron ID
Figure 4: Weights for each of the 5 networks (Control, defensively distilled, finetuning noise, finetuning on blur, and sleep) trained on the Patches dataset. Each column shows the weights value (y-axis) connecting from each of the 100 input neurons (x-axis) to the corresponding output neuron (i.e. the first column of graphs if the weights connecting from all input neurons to the first output neuron. Points are color-coded based which of these pixels in the input layer correspond ON- pixels, Oix-ls or overlapping pixels based on the input that should trigger that output neuron.
15

---

Published as a conference paper at ICLR 2020
A
B
Control
Sleep
0.9
0.8
0.0
0.7
0.7
Epsilon
CR
0.9
0.2
0.9
0.1
0.1
0.8
Digit
C
Before sleep
After sleep
0.9
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.3
0.0
0.1
0.0
0.0
0.0
0.0
0.1
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.1
0.0
0.0
0.0
0.8
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.2
0.0
0.0
0.0
0.0
0.0
0.0
Input
Input
0.0
0.0
0.0
0.0
0.8
0.0
0.0
0.0
0.0
0.1
0.0
0.2
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.9
0.0
0.0
0.0
0.4
0.0
0.0
0.0
0.0
0.0
80
0.0
0.0
0.0
0.0
0.7
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.3
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.1
0.0
0.0
0.9
0.0
0.0
0.1
0.0
0.0
0.0
0.0
0.0
9
9
Neuron
Neuron
Figure 5: A) Example function learned in a 3-layer neural network illustrates that sleep alters de- cision boundaries favor of making one class (corresponding to black points) more robust while impinging on another class (blue points). B) Average noise (epsilon) needed for FGSM attack for specific digits. C) Output layer scores for each digit (rows) before and after sleep. Columns repre- sent average activation of each of the 10 output neurons.
8.3 ADVERSARIAL ATTACKS
Here, describe the general approach for implementing DeepFool, JSMA, and the Boundary At- tack discussed in the paper. We also show examples of adversaries created for each the defense networks from these attacks.
16

---

Published as a conference paper at ICLR 2020
A
DeepFool
3
Control
4
7
9
5
6
6
3
2
Def. Dist.
4
7
9
5
6
7
8
3
Finetuning
4
Z
7
4
5
7
7
3
Sleep
4
6
7
9
5
B
JSMA
6
7
5
4
7
7
7
Control
esc
4
6
7
7
9
5
6
7
5
7
8
7
Def. Dist.
4
6
7
7
7
5
5
0
7
3
Finetuning
a
4
6
7
7
9
5
3
Sleep
4
7
9
6
5
C
Boundary Attack
3
Control
4
5
6
3
5
2
8
8
3
Def. Dist.
4
5
4
3
4
7
4
4
3
Finetuning
4
5
7
4
7
Sleep
4
9
Figure 6: A) DeepFool adversarial examples for each defense. The network's prediction is shown above each image. B) JSMA. C) Boundary Attack.
DeepFool. DeepFool (Moosavi-Dezione et al. (2016)), as mentioned above, an iterative algorithm that, at each iteration, aims move the adversarial example in the direction of the closest decision boundary until it results in a misclassification. We based our implementation of that in Rauber et al. (2017). We stopped running the algorithm when either adversarial example is found or when 100 iterations have passed. Examples of DeepFool attacks on the MNIST dataset are shown in Figure 6A. At each iteration compute linear approximation of the loss function and take step in the
17

---

Published as a conference paper at ICLR 2020
direction that would be result in a misclassification. The equations used and pseudocode can be found in the original DeepFool paper.
JSMA. JSMA is also iterative algorithm which computes the pixel that would change the loss function the most at each algorithm and changes this pixel, until a misclassffication is produced. For this method, map when has beeen updated seven times, so the algorithm can focus on other pixels. We set the change to each pixel constant value, 0.1. This represents how much each pixel is updated (in the direction that results in misclassification) at each iteration. Pseudocode can be found in the original publication (Papernot al. (2016a)). We show examples of adversaries created by JSMA in Figure
run-time limit of 500 iterations. We also remove pixel from the saliency
Boundary Attack. The Boundary Attack (Brendel (2017) starts with an adversarial example and moves it closer to the decision boundary of the correct class. At each step of the algorithm, the method performs orthogonal and forward perturbations to move the adversary closer to the origi- nal image, thus reducing the distance between the adversary and the original image. We set both a distance convergence criterion (L2-norm le-7) and a run-time limitation on the attack (1000 iterations). Example attacks are shown in Figure 6C. We note that sometimes the algorithm does not successfully produce an "imperceptible" adversarial example and instead produces noisy output (the starting condition is noisy image). If define threshold defining successful adversarial attack (1.2-Norm > 1), then we observe the results for MNIST in Table
MNIST
Control
Defensive Distillation Fine-tuning
Sleep
Boundary Attack
0.0073
0.0074
0.0047
0.0094
Table 4: MNIST Boundary Attack with a threshold defining a successful adversarial attack.
8.4
GENERALIZATION ANALYSIS
Noise
Increasing
Blur
Figure 7: Types of images tested on for generalization for the MNIST dataset. Top Images with Gaussian noise added with increasing variannce (0, 0.1, 0.3, 0.5, 0.7, 0.9, left to right). Bottom Gaussian blurred images with increasing sigma (from 0 to 2.5 in steps of 0.5).
In this section, we analyze how sleep can aid in increasing ANN robustness. In biological networks, sleep extracts the gist of task through replay (Lewis & Durrant (20TT)). We hypothesized that sleep algorithm works in the same manner. First, we tested the ability of the sleep network to decorrelate distinct inputs by analyzing the effect of running sleep and testing on our two distortion techniques (see Figure]] 7p.
We computed the correlations of network activities in each of the hidden layers of the network defense methods. For each pair of digits, computed the
before and after implementing average correlation of layer activities in the undistorted (Figure S. noisy (Figure 99, and blurred (Figure 10) conditions. Each figure reports the difference in digit pairwise correlations between
18

---

Published as a conference paper at ICLR 2020
the defense method and the control network for each set of inputs. For sleep network, it is apparent that in layer 2 and layer 3, the correlations of the same digits (the diagonal) increases after sleep. Additionally, the correlation of distinct digits typically experiences negative change, representing decorrelation of distinct inputs. This analysis holds for defensive distillation and both of the fine-tuned networks. This suggests that the ANN representation of different exemplars of the same digit becomes more similar after sleep or after any of the defense networks when compared to the control. This not simply due an increased overlap of all inputs, since exemplars of different digits become decorrelated after applying defense method.
Next, we performed the same correlation analysis on noisy and blurred images to see how the rep- resentation of distorted images changes after applying a distortion method. First, we note that fine- tuning on noisy images results in stronger correlation of the same (noisy) digit but weaker correla- tions of different (noisy) digits, as noted above. However, fine-tuning on blurred images does not have as strong an effect. Second, sleep seems to have a beneficial effect on the correlation matrices for both blurred and noisy images (comparing the right column of Figures 9 and 10 This illustrates the beneficial role of sleep in creating distinct representations of digits, where different neuronal en- sembles encode different digits. This change in representation should result in increased robustness since changes to the input must be larger in order to recruit neuronal ensembles that represent other digits.
On top of decorrelating the representation of distinct memories by pruning synapses, biophysical modelling suggests that sleep also aid in strengthening connections thus making stronger the response of primary neurons involved in memory recall (Gonzalez et al. 2019)). To test this hy- pothesis in our networks, we analyzed the firing rate and activations of digit-specific neurons before and after sleep. Before describing the analysis, we would like to note that SNNs can be used to performn classification and a near loss-less conversion between ANNs and SNNs has been achieved on the MNIST task (Dichl et al. (2015)). To perform classification. digit is presented (as a Poisson spike train) to the network and spikes are propagated throughout the network for given time period (or number of presentations of the input). Analyzing network activity in the spiking domain can be casier than in the activation domain (ANNs) since spikes are oftentimes easier to interpret than neuronal activations.
For this reason, we first analyze how spike rates of digit-specific neurons change before and after sleep in the spike domain. To do this present all images of a specified digit to the spiking network and count the number of spikes from each neuron (holding the weights constant). We define digit- specificity by looking at the 100 neurons with the highest firing rates in layer 2. In Figure 11 show that the normalized firing rate of these neurons usually increases after sleep (normalized by dividing by the maximum firing rate observed from the SNN).
Next, we perform the same analysis in the activation domain. Again, we define digit-specific neurons by looking at the top 100 neurons with the highest activation for a specific digit. We look at the normalized mean activations of these neurons before and after sleep and note that for all digits this value higher after sleep than before sleep (Figure [2]. This suggests that the neurons in the network are responding more strongly to the presentation of the same digit, thus increasing the robustness of the network as more noise must be added in order to counter the effect of this stronger response. This also suggests that our algorithm works in a biologically plausible way: both by decorrelating distinct inputs and increasing the strength of similar inputs.
19

---

Published as a conference paper at ICLR 2020
Defensive Distillation
Finetune Noise
Finetune Blur
Sleep
Layer 2
22
4
8
3
5 6
0
2
4
5
7
8
01
3
5
6
7
9
2 3
5
8
9
0
Layer 3
1
2
2
480
0.02
0
8
-0.02
01
2 3
5 6
0 1
2
3
4
5
7
8
01
2
3
4
6
7
0 1
2
3
5
9
-0.04
O O
-0.06
Layer 4
2
-0.08
4
5
0..1
6
-0.12
8
2 5 6
0 3 5 8
2 5 6
Figure 8: Correlation differences between defense network and control network for different de- fenses. Correlations are computed based the activations in each layer for each pair of digits (mean correlation). The difference between the correlation of the defense method (column) and the control network is plotted. Activations computed based on the undistorted test images.
20

---

Published as a conference paper at ICLR 2020
Defensive Distillation
Finetune Noise
Finetune Blur
Sleep
Layer 2
2
2
3
01
2
5
8 9
0 1
2
3
4
6 7 9
0 2 3
5
8
0 1
3
4
6
7
Layer 3
3
0.05
5
0005
-0.1
2
3
4
5
8
0
1
2
3
4
6
7 9
0 2
3
5
6
8
9
0 1
2
3
4
6
7
9
-0.15
Layer 4
-0.2
10000 20
2 28
-0.25
5
-0.3
6
-035
0 2 4
5
8
0
1
3
4
6 7
8
2 3
5
0
1
3 4
6 7
Figure 9;
Figure 8 but activations computed when presented with noisy images.
21

---

Published as a conference paper at ICLR 2020
Defensive Distillation
Finetune Noise
Finetune Blur
Sleep
Layer 2 A
2
3
3
4 5
7
0 1
3
4
5
6
7
9
2 3
5
8 9
0
3
4
6
7
8
Layer 3
2
3
0
2
4
5
7
8
0
1
2
3
4
6
7
0 2
3
5
6
8
0 1
2
3
4
6
7
8
-0.2
Layer 4
CON
-033
-0.4
1 2 4
5
8
0
3 4
6 7 8
2
5 8
0 1 3 4
6
8
Figure 10: same Figure
but activations are computed when presented with blurred images.
Deyh0
Depf
DH-F2
Digh
211
Degh
Figure 11: Normalized firing rates of neurons specific to individual digits when presented with noisy images is greater after applying sleep than before sleep.
22

---

Published as a conference paper at ICLR 2020
Digtt
Plybt
Des44
Afr
Beturo Mer
Xim Mer
Alm Mee
Aix Vie
Betae Amt
Xfr Are
335
C.00
025
Figure 12: Normalized activations of neurons specific to individual digits when presented with noisy images greater after applying sleep than before sleep.
23


Reviews:
[{"review_id": "HJerk1855H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an ANN training method for improving adversarial robustness and generalization, inspired by biological sleep.\n\nI'm leaning towards accepting as it seems to be an original concept and has fairly extensive empirical results that are somewhat promising.\n\nThe idea of a sleep phase as an alternative to explicit adversarial or generalization training is interesting. The results suggest that the approach works reasonably well in many cases.\n\nSuggestions for improvement / clarification:\n- The mapping from biological sleep to the actual algorithm + pseudocode used could benefit from more thorough explanation. It is not clear which choices are arbitrary vs well-principled.\n- Was the optimal sleep duration determined empirically for each experiment?\n- I agree with the authors' proposed future work of better understanding and standardizing this approach.\n- Consider combining this approach with the existing adversarial or generalizing approaches (instead of as an alternative). Do they complement each other?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "BkgoQYmKKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Disclosure on reviewer's experience: I am not an expert on adversarial attack methods or defenses, but I am well read in the general literature on robustness and uncertainty in deep neural networks.\n\nThe authors present a biologically inspired sleep algorithm for artificial neural networks (ANNs) that aims to improve their generalization and robustness in the face of noisy or malicious inputs. They hypothesize that \"sleep\" could aid in generalization by decorrelating noisy hidden states and reducing the overall impact of imperceptible perturbations of the input space. The proposed sleep algorithm broadly involves 1) converting the trained ANN to a \"spike\" neural network (SNN), 2) converting the input signal (pixels) to a Poisson distributed \"spike train\" where brighter pixels have higher firing rates than darker pixels, 3) propagating the neuronal spikes through the SNN, updating weights based on a simplified version of spike-timing-dependent plasticity (STDP), and 4) converting the network back to an ANN after the sleep phase has finished. They present a detailed comparative study spanning three datasets, four types of adversarial attacks and distortions, and two other baseline defense mechanisms, in which they demonstrate significant improvements (in some cases) of the sleep algorithm over the baselines.\n\nThe core concept behind the authors' work is novel and interesting, and the experimental design is thorough and well controlled. Although the results are (I would argue) somewhat mixed, they are nonetheless positive enough to encourage more work in applying \"sleep\" and other relevant ideas from neuroscience to the problem of robustness in deep neural networks. I have some questions and concerns which I will detail per-section below, but overall, I believe that this paper is a valuable contribution to the literature and should be accepted once the authors have made a few necessary revisions.\n\nSection 1: Introduction\n\n\"We report positive results for four types of adversarial attacks tested on three different datasets (MNIST, CUB200, and a toy dataset) ...\"\n\nIt's debatable whether or not the results from the CUB-200 dataset are positive. The sleep algorithm fails to outperform the baselines for each attack type (except for an almost negligible advantage in accuracy on JSMA) and barely even outperforms the control network in most cases (2/4 attacks it actually underperforms the control). I think the authors should consider rephrasing this statement to better reflect the actual results.\n\nSection 2: Adversarial Attacks and Distortions\n\nFGSM: The notation used here is somewhat inconsistent with the source paper. Goodfellow et al use epsilon to denote what I think the authors call eta, and call the second term, epsilon*sign(grad(J)), eta. Furthermore, the authors state that \"this represents the direction to change each pixel in the original input in order to decrease the loss function.\" But this doesn't make sense. An adversary should want to *increase* the loss function enough to cause a misclassification. Goodfellow et al use this expression to formulate a L1-like regularization term and describe the training procedure \"minimizing the worst case error when the data is perturbed by an adversary\", which seems more sensible. This section should be rewritten to be more consistent with the source.\n\nSection 3: Adversarial defenses\n\nRegarding distillation: \"We use T=50 to compare with the sleep algorithm\"\n\nThe authors should elaborate a bit more on the reasoning for this choice. It seems very arbitrary.\n\nSectioin 4: Sleep algorithm\n\n1. Algorithm 1: Why is line 9 inside of the for loop? It doesn't seem to be at all dependent on t. One would expect the input to only need to be converted once. Additionally, in lines 11-13, the l's in W(l,l-1) and similar should be unbolded. It's confusing that the format changes (unless I am missing something and it's actually a different variable).\n\n2. Spike trains should be more rigorously defined, preferably with formalized notation. It's a bit unclear exactly what they are from the current text. Are they just parameters for a Poisson? Or outputs from a poison over T time steps? Or something else?\n\n3. \"weights are scaled by a parameter to induce high firing rates in later layers\"\nIt would be good to include more details on this parameter, how the values are chosen, and the intuition behind this idea. I assume it's because of higher level feature representations in later layers of deep neural networks.\n\nSection 5: Results\n\n1. It's confusing that sometimes accuracy refers to classification accuracy and sometimes adversarial attack accuracy. I would recommend assigning a different name to the latter, or making sure that a qualifier precedes every reference to \"accuracy\" in this section.\n\n2. In the second section of the results table (which is missing a label), why is the JSMA value for Defensive Distillation bolded? The distance measures for both the control network and for fine-tuning are higher. It seems like fine-tuning should be the one bolded.\n\n3. Figure 1: caption is incorrect; it states \"adversarial attack accuracy\" and it should be \"classification accuracy\", otherwise the plots make no sense.\n\n4. \"we observe that in the Patches and CUB-200 dataset, sleep has beneficial results in moving the accuracy function above the other defense methods\"\nIt should be noted that this is only true for eta < 0.1. After that, sleep and the control both converge to 50% accuracy. Also this sentence should be reworded to be less visual and more quantitative (e.g. sleep tends to have higher median accuracy scores than the other methods for eta < 0.1).\n\n5. \"We observe that performance continued to drop after a sufficiently large amount of noise was added\"\nMore than that, the other methods converged to a small band of accuracy values; sleep continued to deteriorate. This is a significant difference. It would be a good idea to re-run this experiment with a binary classification problem (e.g. only two digits of MNIST) and see if this phenomenon still occurs. Then, the noisy sleep classifier predictions could simply be inverted to get improved accuracy scores.\n\n6. In the analysis of JSMA, as noted before,, it's rather dubious to claim that sleep had any kind of significant effect on the attack success rate (or distance) for CUB-200. I would rewrite this section to better represent the results.\n\n7. Figure 2 formatting: Legend is overflowing out of the first figure. Additionally, the legend colors should be made to match across all three figures, and the legend should either appear in all three (if necessary for some reason) or only in one.\n\n8. Figure 2: The caption is incomplete and possibly incorrect. It's not clear why the first and last figures differ from each other, and the caption does not indicate this. The caption also only mentions two datasets, even though it says \"for the following three datasets\".\n\nAppendix:\n\nGeneral formatting needs improvement. A lot of figures are off-centered, text misaligned, missing axis labels, etc.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: "Sleep" is introduced as a way of increasing robustness in neural network training. To sleep, the network is converted into a spiking network and goes through phases of more and less intense activation. The results are quite good when it comes to defending against adversarial examples. Reviewers agree that the method is novel and interesting. Authors responded to the reviewers' questions (one of the reviewers had a quite extensive set of questions) satisfactorily, and improved the paper significantly in the process. I think the paper should be accepted on the grounds of novelty and good results.
Decision: Accept (Poster)