Adversarial Attacks on Spiking Convolutional Networks for Event-based Vision

Event-based sensing using dynamic vision sensors is gaining traction in lowpower vision applications . Spiking neural networks work well with the sparse nature of event-based data and suit deployment on low-power neuromorphic hardware . Being a nascent field , the sensitivity of spiking neural networks to potentially malicious adversarial attacks has received very little attention so far . In this work , we show how white-box adversarial attack algorithms can be adapted to the discrete and sparse nature of event-based visual data , and to the continuous-time setting of spiking neural networks . We test our methods on the N-MNIST and IBM Gestures neuromorphic vision datasets and show adversarial perturbations achieve a high success rate , by injecting a relatively small number of appropriately placed events . We also verify , for the first time , the effectiveness of these perturbations directly on neuromorphic hardware . Finally , we discuss the properties of the resulting perturbations and possible future directions . 1 INTRODUCTION . Unlike the usual neural networks of contemporary deep learning , spiking neural networks ( SNN ) resemble the animal brain more closely in at least two main aspects : the way their neurons communicate through impulses ( spikes ) , and their dynamics , which evolve in continuous time . Aside from offering the field of computational neuroscience more biologically plausible neuron models and communication schemes , research in the technological applications of spiking neural networks is currently blooming because of the rise of neuromorphic technology . Neuromorphic hardware is directly compatible with spiking neural networks and enables the design of low-power models for use in battery-operated , always-on devices . Adversarial examples are an “ intriguing property of neural networks ” ( Szegedy et al. , 2013 ) by which the network is easily fooled into misclassifying an input which has been altered in an almost imperceptible way by the attacker . This property is usually undesirable in applications : it was proven , for example , that an adversarial attack may pose a threat to self-driving cars , by making them misclassify a stop sign as a speed limit sign ; and that this attack can be implemented in the real world through stickers physically placed on the road sign ( Eykholt et al. , 2018 ) . Because of their relevance to real-world applications , a large amount of work has been published on this subject , typically following a pattern where new attacks are discovered , followed by new defense strategies , in turn followed by proof of other strategies that can still break through them ( see Akhtar & Mian ( 2018 ) for a review ) . With the advent of real-world applications of spiking networks in neuromorphic devices , it is essential to make sure they work securely and reliably in a variety of contexts . In particular , there is a significant need for research on the possibility of adversarial attacks on spiking network models used for computer sensing tasks . In this paper , we make an attempt at modifying event-based data , by adding and removing events , to generate adversarial examples that fool spiking networks into misclassifying them . This offers important insight into the reliability and security of neuromorphic vision devices , with important implications for commercial applications . 1.1 WHAT IS EVENT-BASED SENSING ? . Event-based cameras , usually called Dynamic Vision Sensors ( DVS ) , share many characteristics with the mammalian retina , which make them excel in some circumstances where traditional framebased cameras do not perform well ( Liu & Delbruck , 2010 ; Liu et al. , 2019b ) . First , events are generated only when there are changes in the visual scene , automatically removing redundancies ; second , their pixels fire independently of each other which means that there is no frame rate , but rather a continuous stream of asynchronous events , so that the latency can be extremely small ; third , they have a very high dynamic range which makes them suitable to detect motion in both bright and dark settings . For these reasons , they have found applications in human-robot interaction , odometry , drone control , tracking , and surveillance , including on devices that are already commercially available ( Gallego et al. , 2019 ; Kueng et al. , 2016 ; Falanga et al. , 2020 ) . Beyond computer vision , the realm of event-based sensing extends to auditory sensors known as silicon cochleas ( Chan et al. , 2007 ) , as well as radar ( Stuijt et al. , 2021 ) and tactile sensors ( Caviglia et al. , 2016 ) . Neuromorphic sensors make available a new kind of sparse , asynchronous data , which does not suit current high-throughput , synchronous accelerators such as GPUs . To process event-based data efficiently , a new generation of neuromorphic hardware is being developed in parallel to the spiking neural network models that can be trained in software . Spiking neuromorphic implementations include large-scale simulation of neuronal networks for neuroscience research ( Furber et al. , 2012 ) and lowpower real-world deployments of machine learning algorithms . In particular , convolutional neural network ( CNN ) architectures , used for computer vision , have been run on neuromorphic chips such as IBM ’ s TrueNorth ( Esser et al. , 2016 ) , Intel ’ s Loihi ( Davies et al. , 2018 ) and SynSense ’ s Speck and Dynap-CNN hardware ( Liu et al. , 2019a ) . The full pipeline of event-based sensors that output sparse data , stateful spiking neural networks which extract semantic meaning and asynchronous hardware backends allows for large gains in power-efficiency when compared to conventional systems . 1.2 ADVERSARIAL ATTACKS ON DISCRETE DATA . The history of attack strategies against various kinds of machine-learning algorithms pre-dates the advent of deep learning ( Biggio & Roli , 2018 ) , but the phenomenon received widespread interest when adversarial examples were first found for deep convolutional networks ( Szegedy et al. , 2013 ) . Generally speaking , given a neural network classifier C and an input x which is correctly classified , finding an adversarial perturbation means finding the smallest δ such that C ( x+ δ ) 6= C ( x ) . Here , “ smallest ” refers to minimising ‖δ‖ , where the norm is chosen arbitrarily depending on the requirements of the experiment . For example , using the L∞ norm ( maximum norm ) will generally make the perturbation less noticeable to a human eye , since the difference in any pixel value between the original and perturbed images will be below a maximum value that is kept as low as possible . Conversely , the use of the L1 norm will encourage sparsity , i.e . a smaller number of perturbed pixels . The main challenges in transferring existing adversarial algorithms to event-based neuromorphic vision lie in the dynamics of the data and network , which develop in continuous time , and in the discrete nature of events , which can either be present or absent at a given time and location , unlike the continuous pixel values of traditional image data . Event-based sensors encode information in the timing , location , and polarity of events , which can be of ‘ on ’ or ‘ off ’ type . Because at any point in time an event can either be triggered or not , one can simply view event-based inputs as binary data by discretising time ( Figure 1 ) . In this view , the network ’ s input is a three-dimensional array whose entries describe the number of events at a location ( x , y ) and in time bin t ; an additional dimension , of length 2 , is added due to the polarity of events . If the time discretisation is sufficiently precise , and no more than one event appears in each bin , the data can be treated as binary . A possible approach to attacking these data is exploiting recent work done on attacking binary images , i.e . with either black or white pixels , which are used in the automatic processing of cheques and other documents . Most methods proposed for attacking binary inputs have focused on brute-force approaches that rely on heuristics to reduce the search space ( Bagheri et al. , 2018 ; Balkanski et al. , 2020 ) . For example , SCAR ( Balkanski et al. , 2020 ) is a black-box algorithm that only assumes access to the output probabilities of the network . The algorithm flips bits in areas chosen according to a specific heuristic and keeps flipped those that cause a change in the confidence of the network . Naturally , this algorithm does not scale well to large input sizes , as the number of queries made to the network grows exponentially . In particular , this becomes a serious problem when the time dimension is added , greatly increasing the dimensionality of the input . Instead , in this paper , we chose to focus on the easier problem of white box attacks , where the attacker has full access to the network and can backpropagate gradients through it . This allows us to adapt faster and more effective algorithms to the case of event-based data . To this end , we chose to adapt existing attack strategies so that they could work with the time dynamics of spiking neural networks , and with the discrete nature of event-based data . We test our attacks on the Neuromorphic MNIST ( Orchard et al. , 2015 ) and IBM Gestures ( Amir et al. , 2017 ) datasets , which are the most common benchmark datasets within the neuromorphic community . Previous work on adversarial attacks in spiking networks has been reported by Sharmin et al . ( 2020 ) ; however , their work only uses static image data with continuous pixel values converted to Poisson input frequencies , so does not involve dealing with discrete data which was the main challenge in our work . More recently , Liang et al . ( 2020 ) did apply attacks to DVS data , using a discretisedgradient technique . They report high success rates , despite some notable problems of vanishing gradients . Concurrently with our work , Marchisio et al . ( 2021 ) designed custom algorithms for DVS data , rather than adapting existing ones , but did not report on the magnitudes of the resulting perturbations . None of these validated the effectiveness of their attack strategies against an on-chip model deployed on neuromorphic hardware . Our contributions beyond the existing literature can be summarised as follows : • We provide detailed results to quantify the effectiveness and scalability of several adversarial attacks strategies , including some not tried before on SNNs . • We show targeted universal attacks on event-based data in the form of adversarial patches , which do not require prior knowledge of the input . • We validate the resulting adversarial examples on an SNN deployed on a convolutional neuromorphic chip . To the best of our knowledge , this is the first time the effectiveness of adversarial examples is demonstrated directly on neuromorphic hardware . 2 METHODS . 2.1 ATTACK STRATEGIES . Projected Gradient Descent As a baseline , we use Projected Gradient Descent ( PGD ) ( Madry et al. , 2019 ) , a standard attack algorithm which we use on discrete data in two ways . The first consists in naively rounding the data at each iteration . However , in this case , updates will be retained only if the gradient magnitude is large enough : otherwise , the small changes made to the adversarial input are lost due to the subsequent discretization . Instead , we adopt an approach that prevents this loss of information : we keep a continuous version of the image as a copy , but use the gradients computed on the discretized image to update the continuous version which is kept in memory . To adapt PGD to the scenario where we want to find the smallest perturbation that triggers a misclassification , we sort the values based on how much PGD adjusted them . We then iterate through the sorted list of indices and flip each value until a misclassification is triggered . It should be noted that this step incurs most of the computational overhead , but is necessary to produce good results . Unless stated otherwise , we used the following values for the parameters : the magnitude of the initial random perturbation to the input is set to τ = 0.01 . The maximum norm of the perturbation was set to = 1.5 . We found that 50 iterations ( Npgd ) of PGD sufficed and the results did not improve by much afterwards . Probabilistic PGD We also devised an alternative way of using PGD on discrete data , which we call “ Probabilistic PGD ” . Probabilistic PGD works by assuming that the binary input was generated by sampling from a series of independent Bernoulli random variables . This approach aligns with how the DVS camera generates the binary data : the probability of emitting a spike at time t is proportional to the light intensity , a continuous metric . For each round of PGD , the input is sampled in a differentiable manner by the Gumbel-softmax reparameterization trick ( Jang et al. , 2017 ) : xadv = σ ( [ log ( r ) − log ( 1− r ) + log ( padv ) − log ( 1− padv ) ] /T ) , where r ∼ U ( 0,1 ) , and T = 0.01 is a temperature parameter . The underlying probabilities padv , instead of the pixel values xadv , are updated using the gradient obtained from the loss function that is minimised by PGD . We saw that this generally improved the performance compared to the PGD version explained above . Gradients are averaged over Nmc = 10 samples of r. It should be noted that the need for a gradient sampling procedure significantly increases the runtime . SparseFool on discrete data To operate on event-based data efficiently , the ideal adversarial algorithm requires two main properties : sparsity and scalability . Scalability is needed because of the increased dimensionality given by the additional time dimension . Sparsity ensures that the number of events added or removed is kept to a minimum . One approach that combines the above is SparseFool ( Modas et al. , 2018 ) , which iteratively finds the closest point in L2 on the linearised decision boundary of the network using the DeepFool algorithm ( Moosavi-Dezfooli et al. , 2015 ) as a subroutine , followed by a linear solver that enforces sparsity and boundary constraints on the perturbation . Because Spiking Neural Networks ( SNNs ) have discrete outputs ( the number of spikes over time for each output neuron ) , it is easier to incur in vanishing gradients as the perturbation approaches the decision boundary . Therefore , we had to make changes to the algorithm to take this into account . Firstly , we found that clamping the perturbation at every iteration of DeepFool , so that it was no smaller than a value η , offered protection against vanishing gradients . η was treated as a hyperpa- rameter that should be kept as small as it can without incurring in vanishing gradients . Secondly , to account for the discreteness of event-based data , we rounded the output of SparseFool to the nearest integer at each iteration . Finally , SparseFool normally involves upper and lower bounds l and u on pixel values ( normally set , for images , to l = 0 ; u = 255 ) . We exploit these to enforce the binary constraint on the data ( l = 0 ; u = 1 ) , or , in the on-chip experiments , to fix a maximum firing rate in each time bin , which is the same as that of the original input ( l = 0 ; u = max ( input ) ) . Adversarial patches As the name suggests , adversarial patches are perturbations that are accumulated in a certain region of the image . The idea is that these patches are generated in a way that enables the adversary to place them anywhere in the image . This attack is targeted to a desired label , and universal , i.e . not specific to an input . To test a more realistic scenario where an adversary could potentially perform an attack without previous knowledge of the input , we apply these patches to the IBM hand gesture dataset . We note that the prediction of the CNN trained on this dataset is mostly determined by spatial location of the input . For example , the original input of “ Right Hand Wave ” is not recognised as such if it is shifted or rotated by a substantial amount . In order to simulate effective realistic attacks , we choose to limit both computed and random attack patches to the area of where the actual gesture is performed . As in Brown et al . ( 2017 ) , we generate the patches using PGD on the log softmax value of the target output neuron . PGD is performed iteratively on different images of the training set and the position of the patch is randomised after each sample . For each item in the training data , the algorithm updates the patch until the target label confidence has reached a pre-defined threshold . The algorithm skips the point if the original label equals the target label . This process is repeated for every training sample and for multiple epochs . To measure the effectiveness of our computed patches , we also generate random patches of the same size , and measure the target success rates . In a random patch , every pixel has a 50 % chance of emitting a spike at each time step .

Summary: Authors present white-box adversarial attack algorithms for spiking neural networks (SNNs) with event-based visual data inputs, and experimentally demonstrate vulnerability of SNNs in various settings. Extensions of existing adversarial attacks on convolutional networks (i.e., PGD and SparseFool) are presented and utilized in a setting for SNNs with event-based inputs. Experimental evaluations are performed on the Neuromorphic-MNIST and IBM Gestures dynamic vision sensor datasets, and validations of these attacks on neuromorphic hardware are performed.