Adversarial Attacks on Spiking Convolutional Networks for Event-based Vision

Event-based sensing using dynamic vision sensors is gaining traction in lowpower vision applications . Spiking neural networks work well with the sparse nature of event-based data and suit deployment on low-power neuromorphic hardware . Being a nascent field , the sensitivity of spiking neural networks to potentially malicious adversarial attacks has received very little attention so far . In this work , we show how white-box adversarial attack algorithms can be adapted to the discrete and sparse nature of event-based visual data , and to the continuous-time setting of spiking neural networks . We test our methods on the N-MNIST and IBM Gestures neuromorphic vision datasets and show adversarial perturbations achieve a high success rate , by injecting a relatively small number of appropriately placed events . We also verify , for the first time , the effectiveness of these perturbations directly on neuromorphic hardware . Finally , we discuss the properties of the resulting perturbations and possible future directions . 1 INTRODUCTION . Unlike the usual neural networks of contemporary deep learning , spiking neural networks ( SNN ) resemble the animal brain more closely in at least two main aspects : the way their neurons communicate through impulses ( spikes ) , and their dynamics , which evolve in continuous time . Aside from offering the field of computational neuroscience more biologically plausible neuron models and communication schemes , research in the technological applications of spiking neural networks is currently blooming because of the rise of neuromorphic technology . Neuromorphic hardware is directly compatible with spiking neural networks and enables the design of low-power models for use in battery-operated , always-on devices . Adversarial examples are an “ intriguing property of neural networks ” ( Szegedy et al. , 2013 ) by which the network is easily fooled into misclassifying an input which has been altered in an almost imperceptible way by the attacker . This property is usually undesirable in applications : it was proven , for example , that an adversarial attack may pose a threat to self-driving cars , by making them misclassify a stop sign as a speed limit sign ; and that this attack can be implemented in the real world through stickers physically placed on the road sign ( Eykholt et al. , 2018 ) . Because of their relevance to real-world applications , a large amount of work has been published on this subject , typically following a pattern where new attacks are discovered , followed by new defense strategies , in turn followed by proof of other strategies that can still break through them ( see Akhtar & Mian ( 2018 ) for a review ) . With the advent of real-world applications of spiking networks in neuromorphic devices , it is essential to make sure they work securely and reliably in a variety of contexts . In particular , there is a significant need for research on the possibility of adversarial attacks on spiking network models used for computer sensing tasks . In this paper , we make an attempt at modifying event-based data , by adding and removing events , to generate adversarial examples that fool spiking networks into misclassifying them . This offers important insight into the reliability and security of neuromorphic vision devices , with important implications for commercial applications . 1.1 WHAT IS EVENT-BASED SENSING ? . Event-based cameras , usually called Dynamic Vision Sensors ( DVS ) , share many characteristics with the mammalian retina , which make them excel in some circumstances where traditional framebased cameras do not perform well ( Liu & Delbruck , 2010 ; Liu et al. , 2019b ) . First , events are generated only when there are changes in the visual scene , automatically removing redundancies ; second , their pixels fire independently of each other which means that there is no frame rate , but rather a continuous stream of asynchronous events , so that the latency can be extremely small ; third , they have a very high dynamic range which makes them suitable to detect motion in both bright and dark settings . For these reasons , they have found applications in human-robot interaction , odometry , drone control , tracking , and surveillance , including on devices that are already commercially available ( Gallego et al. , 2019 ; Kueng et al. , 2016 ; Falanga et al. , 2020 ) . Beyond computer vision , the realm of event-based sensing extends to auditory sensors known as silicon cochleas ( Chan et al. , 2007 ) , as well as radar ( Stuijt et al. , 2021 ) and tactile sensors ( Caviglia et al. , 2016 ) . Neuromorphic sensors make available a new kind of sparse , asynchronous data , which does not suit current high-throughput , synchronous accelerators such as GPUs . To process event-based data efficiently , a new generation of neuromorphic hardware is being developed in parallel to the spiking neural network models that can be trained in software . Spiking neuromorphic implementations include large-scale simulation of neuronal networks for neuroscience research ( Furber et al. , 2012 ) and lowpower real-world deployments of machine learning algorithms . In particular , convolutional neural network ( CNN ) architectures , used for computer vision , have been run on neuromorphic chips such as IBM ’ s TrueNorth ( Esser et al. , 2016 ) , Intel ’ s Loihi ( Davies et al. , 2018 ) and SynSense ’ s Speck and Dynap-CNN hardware ( Liu et al. , 2019a ) . The full pipeline of event-based sensors that output sparse data , stateful spiking neural networks which extract semantic meaning and asynchronous hardware backends allows for large gains in power-efficiency when compared to conventional systems . 1.2 ADVERSARIAL ATTACKS ON DISCRETE DATA . The history of attack strategies against various kinds of machine-learning algorithms pre-dates the advent of deep learning ( Biggio & Roli , 2018 ) , but the phenomenon received widespread interest when adversarial examples were first found for deep convolutional networks ( Szegedy et al. , 2013 ) . Generally speaking , given a neural network classifier C and an input x which is correctly classified , finding an adversarial perturbation means finding the smallest δ such that C ( x+ δ ) 6= C ( x ) . Here , “ smallest ” refers to minimising ‖δ‖ , where the norm is chosen arbitrarily depending on the requirements of the experiment . For example , using the L∞ norm ( maximum norm ) will generally make the perturbation less noticeable to a human eye , since the difference in any pixel value between the original and perturbed images will be below a maximum value that is kept as low as possible . Conversely , the use of the L1 norm will encourage sparsity , i.e . a smaller number of perturbed pixels . The main challenges in transferring existing adversarial algorithms to event-based neuromorphic vision lie in the dynamics of the data and network , which develop in continuous time , and in the discrete nature of events , which can either be present or absent at a given time and location , unlike the continuous pixel values of traditional image data . Event-based sensors encode information in the timing , location , and polarity of events , which can be of ‘ on ’ or ‘ off ’ type . Because at any point in time an event can either be triggered or not , one can simply view event-based inputs as binary data by discretising time ( Figure 1 ) . In this view , the network ’ s input is a three-dimensional array whose entries describe the number of events at a location ( x , y ) and in time bin t ; an additional dimension , of length 2 , is added due to the polarity of events . If the time discretisation is sufficiently precise , and no more than one event appears in each bin , the data can be treated as binary . A possible approach to attacking these data is exploiting recent work done on attacking binary images , i.e . with either black or white pixels , which are used in the automatic processing of cheques and other documents . Most methods proposed for attacking binary inputs have focused on brute-force approaches that rely on heuristics to reduce the search space ( Bagheri et al. , 2018 ; Balkanski et al. , 2020 ) . For example , SCAR ( Balkanski et al. , 2020 ) is a black-box algorithm that only assumes access to the output probabilities of the network . The algorithm flips bits in areas chosen according to a specific heuristic and keeps flipped those that cause a change in the confidence of the network . Naturally , this algorithm does not scale well to large input sizes , as the number of queries made to the network grows exponentially . In particular , this becomes a serious problem when the time dimension is added , greatly increasing the dimensionality of the input . Instead , in this paper , we chose to focus on the easier problem of white box attacks , where the attacker has full access to the network and can backpropagate gradients through it . This allows us to adapt faster and more effective algorithms to the case of event-based data . To this end , we chose to adapt existing attack strategies so that they could work with the time dynamics of spiking neural networks , and with the discrete nature of event-based data . We test our attacks on the Neuromorphic MNIST ( Orchard et al. , 2015 ) and IBM Gestures ( Amir et al. , 2017 ) datasets , which are the most common benchmark datasets within the neuromorphic community . Previous work on adversarial attacks in spiking networks has been reported by Sharmin et al . ( 2020 ) ; however , their work only uses static image data with continuous pixel values converted to Poisson input frequencies , so does not involve dealing with discrete data which was the main challenge in our work . More recently , Liang et al . ( 2020 ) did apply attacks to DVS data , using a discretisedgradient technique . They report high success rates , despite some notable problems of vanishing gradients . Concurrently with our work , Marchisio et al . ( 2021 ) designed custom algorithms for DVS data , rather than adapting existing ones , but did not report on the magnitudes of the resulting perturbations . None of these validated the effectiveness of their attack strategies against an on-chip model deployed on neuromorphic hardware . Our contributions beyond the existing literature can be summarised as follows : • We provide detailed results to quantify the effectiveness and scalability of several adversarial attacks strategies , including some not tried before on SNNs . • We show targeted universal attacks on event-based data in the form of adversarial patches , which do not require prior knowledge of the input . • We validate the resulting adversarial examples on an SNN deployed on a convolutional neuromorphic chip . To the best of our knowledge , this is the first time the effectiveness of adversarial examples is demonstrated directly on neuromorphic hardware . 2 METHODS . 2.1 ATTACK STRATEGIES . Projected Gradient Descent As a baseline , we use Projected Gradient Descent ( PGD ) ( Madry et al. , 2019 ) , a standard attack algorithm which we use on discrete data in two ways . The first consists in naively rounding the data at each iteration . However , in this case , updates will be retained only if the gradient magnitude is large enough : otherwise , the small changes made to the adversarial input are lost due to the subsequent discretization . Instead , we adopt an approach that prevents this loss of information : we keep a continuous version of the image as a copy , but use the gradients computed on the discretized image to update the continuous version which is kept in memory . To adapt PGD to the scenario where we want to find the smallest perturbation that triggers a misclassification , we sort the values based on how much PGD adjusted them . We then iterate through the sorted list of indices and flip each value until a misclassification is triggered . It should be noted that this step incurs most of the computational overhead , but is necessary to produce good results . Unless stated otherwise , we used the following values for the parameters : the magnitude of the initial random perturbation to the input is set to τ = 0.01 . The maximum norm of the perturbation was set to = 1.5 . We found that 50 iterations ( Npgd ) of PGD sufficed and the results did not improve by much afterwards . Probabilistic PGD We also devised an alternative way of using PGD on discrete data , which we call “ Probabilistic PGD ” . Probabilistic PGD works by assuming that the binary input was generated by sampling from a series of independent Bernoulli random variables . This approach aligns with how the DVS camera generates the binary data : the probability of emitting a spike at time t is proportional to the light intensity , a continuous metric . For each round of PGD , the input is sampled in a differentiable manner by the Gumbel-softmax reparameterization trick ( Jang et al. , 2017 ) : xadv = σ ( [ log ( r ) − log ( 1− r ) + log ( padv ) − log ( 1− padv ) ] /T ) , where r ∼ U ( 0,1 ) , and T = 0.01 is a temperature parameter . The underlying probabilities padv , instead of the pixel values xadv , are updated using the gradient obtained from the loss function that is minimised by PGD . We saw that this generally improved the performance compared to the PGD version explained above . Gradients are averaged over Nmc = 10 samples of r. It should be noted that the need for a gradient sampling procedure significantly increases the runtime . SparseFool on discrete data To operate on event-based data efficiently , the ideal adversarial algorithm requires two main properties : sparsity and scalability . Scalability is needed because of the increased dimensionality given by the additional time dimension . Sparsity ensures that the number of events added or removed is kept to a minimum . One approach that combines the above is SparseFool ( Modas et al. , 2018 ) , which iteratively finds the closest point in L2 on the linearised decision boundary of the network using the DeepFool algorithm ( Moosavi-Dezfooli et al. , 2015 ) as a subroutine , followed by a linear solver that enforces sparsity and boundary constraints on the perturbation . Because Spiking Neural Networks ( SNNs ) have discrete outputs ( the number of spikes over time for each output neuron ) , it is easier to incur in vanishing gradients as the perturbation approaches the decision boundary . Therefore , we had to make changes to the algorithm to take this into account . Firstly , we found that clamping the perturbation at every iteration of DeepFool , so that it was no smaller than a value η , offered protection against vanishing gradients . η was treated as a hyperpa- rameter that should be kept as small as it can without incurring in vanishing gradients . Secondly , to account for the discreteness of event-based data , we rounded the output of SparseFool to the nearest integer at each iteration . Finally , SparseFool normally involves upper and lower bounds l and u on pixel values ( normally set , for images , to l = 0 ; u = 255 ) . We exploit these to enforce the binary constraint on the data ( l = 0 ; u = 1 ) , or , in the on-chip experiments , to fix a maximum firing rate in each time bin , which is the same as that of the original input ( l = 0 ; u = max ( input ) ) . Adversarial patches As the name suggests , adversarial patches are perturbations that are accumulated in a certain region of the image . The idea is that these patches are generated in a way that enables the adversary to place them anywhere in the image . This attack is targeted to a desired label , and universal , i.e . not specific to an input . To test a more realistic scenario where an adversary could potentially perform an attack without previous knowledge of the input , we apply these patches to the IBM hand gesture dataset . We note that the prediction of the CNN trained on this dataset is mostly determined by spatial location of the input . For example , the original input of “ Right Hand Wave ” is not recognised as such if it is shifted or rotated by a substantial amount . In order to simulate effective realistic attacks , we choose to limit both computed and random attack patches to the area of where the actual gesture is performed . As in Brown et al . ( 2017 ) , we generate the patches using PGD on the log softmax value of the target output neuron . PGD is performed iteratively on different images of the training set and the position of the patch is randomised after each sample . For each item in the training data , the algorithm updates the patch until the target label confidence has reached a pre-defined threshold . The algorithm skips the point if the original label equals the target label . This process is repeated for every training sample and for multiple epochs . To measure the effectiveness of our computed patches , we also generate random patches of the same size , and measure the target success rates . In a random patch , every pixel has a 50 % chance of emitting a spike at each time step .

Summary: This paper investigated how to adapt existing attack strategies in order to work with the discrete nature of event-based data. Also, the authors proved that Sparsefool can efficiently and reliably generate sparse perturbations on discrete data. Besides,the authors validate the adversarial examples on neuromorphic chip and the perturbation is effective after modified the networks. In a more realistic setting, the authors use adversarial patches to trigger a targeted misclassification. MNIST and IBM Gestures datasets are used to test the attack algorithms as a sequence of binary frames.