Adversarial Attacks on Spiking Convolutional Networks for Event-based Vision
Authors: Julian Büchel
Gregor Lenz
Yalun Hu
Sadique Sheik
Martino Sorbaro
Venue: ICLR 2022 Submitted (2022)
Keywords: spiking neural networks
neuromorphic engineering
adversarial attacks
dynamic vision sensors

Abstract: Event-based sensing using dynamic vision sensors is gaining traction in low-power vision applications. Spiking neural networks work well with the sparse nature of event-based data and suit deployment on low-power neuromorphic hardware. Being a nascent field, the sensitivity of spiking neural networks to potentially malicious adversarial attacks has received very little attention so far. In this work, we show how white-box adversarial attack algorithms can be adapted to the discrete and sparse nature of event-based visual data, and to the continuous-time setting of spiking neural networks. We test our methods on the N-MNIST and IBM Gestures neuromorphic vision datasets and show adversarial perturbations achieve a high success rate, while injecting a relatively small number of appropriately placed events. We also verify, for the first time, the effectiveness of these perturbations directly on neuromorphic hardware. Finally, we discuss the properties of the resulting perturbations and possible future directions.

Under review as a conference paper at ICLR 2022
ADVERSARIAL ATTACKS ON SPIKING CONVOLU- TIONAL NETWORKS FOR EVENT-BASED VISION
Anonymous authors Paper under double-blind review
ABSTRACT
Event-based sensing using dynamic vision sensors is gaining traction in low- power vision applications. Spiking neural networks work well with the sparse nature of event-based data and suit deployment low-power neuromorphic hard- ware. Being a nascent field, the sensitivity of spiking neural networks to poten- tially malicious adversarial attacks has received very little attention so far. In this work, we show how white-box adversarial attack algorithms can be adapted to the discrete and sparse nature of event-based visual data, and to the continuous-time setting of spiking neural networks. We test our methods on the N-MNIST and IBM Gestures neuromorphic vision datasets and show adversarial perturbations achieve high success rate, by injecting a relatively small number of appropri- ately placed events. We also verify, for the first time, the effectiveness of these perturbations directly on neuromorphic hardware. Finally, we discuss the proper- ties of the resulting perturbations and possible future directions.
INTRODUCTION
Unlike the usual neural networks contemporary deep learning, spiking neural networks (SNN) resemble the animal brain more closely in at least two main aspects: the way their neurons com- municate through impulses (spikes), and their dynamics, which evolve in continuous time. Aside from offering the field of computational neuroscience more biologically plausible neuron models and communication schemes, research in the technological applications of spiking neural networks is currently blooming because of the rise of neuromorphic technology. Neuromorphic hardware is directly compatible with spiking neural networks and enables the design of low-powe models for use in nattery-operated always-on devices.
Adversarial examples are an "intriguing property of neural networks" (Szegedy et al. 2013) by which the network is easily fooled into misclassifying input which has been altered in almost imperceptible way by the attacker. This property is usually undesirable in applications: proven, for example, that an adversarial attack may pose threat to self-driving cars, by making them misclassify stop sign as a speed limit sign; and that this attack can be implemented in the real world through stickers physically placed on the road sign (Eykholt et al. 2018). Because of their relevance to real-world applications, large amount of work has been published on this subject, typically following pattern where new attacks are discovered, followed by new defense strategies, in turn followed by proof of other strategies that can still break through them (see Akhtar & Mian 2018 for a review).
was
With the advent of real-world applications of spiking networks in neuromorphic devices, it is es- sential to make sure they work securely and reliably variety of contexts. In particular, there is significant need for research the possibility of adversarial attacks spiking network models used for computer sensing tasks. In this paper, we make an attempt at modifying event-based data, by adding and removing events, to generate adversarial examples that fool spiking networks into misclassifying them. This offers important insight into the reliability and security of neuromorphic vision devices, with important implications for commercial applications.

---

Under review as a conference paper at ICLR 2022
1.1 WHAT IS EVENT-BASED SENSING?
Event-based cameras, usually called Dynamic Vision Sensors (DVS), share many characteristics with the mammalian retina, which make them excel in some circumstances where traditional frame- based cameras do not perform well (Liu & Delbruck] 2010 Liu et al.] 2019b). First, events are generated only when there are changes in the visual scene, automatically removing redundancies; second, their pixels fire independently of each other which means that there is no frame rate, but rather continuous stream of asynchronous events, so that the latency can be extremely small; third, they have a very high dynamic range which makes them suitable to detect motion in both bright and dark settings. For these reasons, they have found applications in human-robot interaction, odom- etry. drone control, tracking, and surveillance, including devices that are already commercially available (Gallego et al. [019]] Kueng et al., [0116] Falanga et al.] 2020). Beyond computer vision, the realm of event-based sensing extends to auditory sensors known silicon cochleas (Chan al., 2007], as well as radar (Stuijt et al. (021) and tactile sensors (Caviglia et al. (016).
Neuromorphic sensors make available a new kind of sparse, asynchronous data, which does not suit current high-throughput, synchronous accelerators such as GPUs. To process event-based data effi- ciently, a new generation of neuromorphic hardware is being developed in parallel to the spiking neu- ral network models that can be trained in software. Spiking neuromorphic implementations include large-scale simulation of neuronal networks for neuroscience research (Furber al. (012) and low- power real-world deployments of machine learning algorithms. In particular, convolutional neural network (CNN) architectures, used for computer vision, have been run on neuromorphic chips such as IBM's Truthorth (Esser et al.] 2016). Intel's Loihi (Davies et al. (2188) 8) and SynSense's 's Speck and Dynap-CNN hardware (Liu et al. 2019a). The full pipeline of event-based sensors that output sparse data, stateful spiking neural networks which extract semantic meaning and asynchronous hardware backends allows for large gains in power-efficiency when compared to conventional systems.
1.2 ADVERSARIAL ATTACKS ON DISCRETE DATA
The history of attack strategies against various kinds of machine-learning algorithms pre-dates the advent of deep learning (Biggio & Roli 2018). but the phenomenon received widespread interest when adversarial examples were first found for deep convolutional networks (Szegedy et al.) (013). Generally speaking. given neural network classifier C and input x which is correctly classified, finding an adversarial perturbation means finding the smallest 5 such that C(x + 6) * C(x). Here, "smallest" refers to minimising [|ô]], where the norm is chosen arbitrarily depending on the require- ments of the experiment. For example, using the L norm (maximum norm) will generally make the perturbation less noticeable to a human eye, since the difference in any pixel value between the original and perturbed images will be below a maximum value that is kept as low as possible. Con- versely, the use of the L norm will encourage sparsity, i.e. a smaller number of perturbed pixels. The main challenges in transferring existing adversarial algorithms to event-based neuromorphic vision lie in the dynamics of the data and network, which develop in continuous time, and in the discrete nature of events, which can either be present absent at given time and location, unlike the continuous pixel values of traditional image data.
Event-based sensors encode information in the timing, location, and polarity of events, which can be of 'on' or 'off' type. Because at any point in time an event can either be triggered or not, one can simply view event-based inputs as binary data by discretising time (Figure []. In this view, the network's 's input is a three-dimensional array whose entries describe the number of events at a location (a, y) and in time bin I: an additional dimension, of length 2, is added due the polarity of events. If the time discretisation is sufficiently precise, and no more than one event appears in each bin, the data can be treated as binary. A possible approach to attacking these data exploiting recent work done on attacking binary images, i.c. with either black or white pixels, which are used in the automatic processing of cheques and other documents. Most methods proposed for attacking binary inputs have focused on brute-force approaches that rely on heuristics to reduce the search space (Bagheri et al. 2018 Balkanski et al., 2020). For example, SCAR (Balkanski et al.) (020) is black-box algorithm that only assumes access to the output probabilities of the network. The algorithm flips bits in areas chosen according to a specific heuristic and keeps flipped those that cause change in the confidence of the network. Naturally, this algorithm does not scale well to large input sizes, as the number of queries made to the network grows exponentially. In particular, this becomes serious problem when the time dimension is added, greatly increasing the dimensionality of the
2

---

Under review as a conference paper at ICLR 2022
imput. Instead, in this paper, chose to focus the easier problem of white box attacks, where the attacker has full access to the network and can backpropagate gradients through it. This allows us to adapt faster and more effective algorithms to the case of event-based data.
To this end, we chose to adapt existing attack strategies so that they could work with the time dynamics of spiking neural networks, and with the discrete nature of event-based data. We test attacks on the Neuromorphic MNIST (Orchard et al.] (015)) and IBM Gestures (Amir et al.] (017) datasets, which are the most common benchmark datasets within the neuromorphic community. Previous work on adversarial attacks in spiking networks has been reported by SShamin et al. (2020);
however, their work only uses static image data with continuous pixel values converted to Poisson input frequencies, so does not involve dealing with discrete data which was the main challenge in
our work. More recently, Liang et al. (2020) did apply attacks to DVS data, using a discretised- gradient technique. They report high success rates, despite some notable problems of vanishing gradients. Concurrently with our work, |Machiiio et al.] (202T) designed custom algorithms for DVS data, rather than adapting existing ones, but did not report the magnitudes of the resulting perturbations. None of these validated the effectiveness of their attack strategies against an on-chip model deployed neuromorphic hardware. Our contributions beyond the existing literature can be summarised as follows:
We provide detailed results to quantify the effectiveness and scalability of several adversar- ial attacks strategies, including some not tried before on SNNs.
We show targeted universal attacks event-based data in the form of adversarial patches, which do not require prior knowledge of the input.
We validate the resulting adversarial examples on an SNN deployed on a convolutional neuromorphic chip. To the best of our knowledge, this is the first time the effectiveness of adversarial examples is demonstrated directly on neuromorphic hardware.
2 METHODS
2.1 ATTACK STRATEGIES
Projected Gradient Descent As a baseline, we use Projected Gradient Descent (PGD) Madry al. 2019). standard attack algorithm which we use on discrete data in two ways. The first consists in naively rounding the data at each iteration. However, in this case, updates will be retained only the gradient magnitude is large enough: otherwise, the small changes made to the adversarial input are lost due to the subsequent discretization. Instead, we adopt an approach that prevents this loss of information: keep a continuous version of the image copy, but use the gradients computed on the discretized image to update the continuous version which is kept in memory. To adapt PGD to the scenario where we want to find the smallest perturbation that triggers a misclassification, we sort the values based on how much PGD adjusted them. We then iterate through the sorted list of indices and flip each value until a misclassification triggered. It should be noted that this step incurs most of the computational overhead, but is necessary to produce good results. Unless stated otherwise, we used the following values for the parameters: the magnitude of the initial random perturbation to the input is set to 0.01. The maximum norm of the perturbation was set to 1.5. We found that 50 iterations (NNga) of PGD sufficed and the results did not improve by much afterwards.
Probabilistic PGD We also devised an alternative way of using PGD discrete data, which we call "Probabilistic PGD Probabilistic PGD works by assuming that the binary input was generated by sampling from series of independent Bernoulli random variables. This approach aligns with how the DVS camera generates the binary data: the probability of emitting spike at time is proportional to the light intensity, a continuous metric. For each round PGG the input is sampled in differentiable manner by the Gumbel-softmax reparameterization trick (Jang et al. 2017):
Xadv = a (log(r) - log(1 r) + log(Paar) - log(1 P/a)))/T).
where r 20( 1). and T = 0.01 is a temperature parameter. The underlying probabilities Padv. instead of the pixel values Xadv. are updated using the gradient obtained from the loss function that is minimised by PGD We saw that this generally improved the performance compared to the PGD version explained above. Gradients are averaged over Nmc = 10 samples of r. It should be noted that the need for a gradient sampling procedure significantly increases the runtime.
3

---

Under review as a conference paper at ICLR 2022
DVS camera
Time discretisation
SparseFool discrete data To operate event-based data efficiently, the ideal adver- sarial algorithm requires two main properties: sparsity and scalability. Scalability is needed because of the increased dimensionality given by the additional time dimension. Sparsity en- sures that the number of events added or re- moved is kept to a minimum. One approach that combines the above is SparseFool (Modas et al. 2018), which iteratively finds the closest point in 1,a on the linearised decision boundary of the network using the DeepFool algorithm MMosavi--Dzfooo
Input event list
Raster
(x=12, y=17, t=3458, p=+)
Q
(x=98, y=121 t=3452, p=*
(xx124, ym34, t=3457, pm-)
1
(x=B7, y=13, t=3559, p=-)
(x=19, y=110, t=3562,
p=+)
0
0
0
0
8
1
0
New events added
Adversanial attack
to original list
et al.) 2015) as a subrou- tine, followed by a linear solver that enforces sparsity and boundary constraints on the per- turbation. Because [Spiking Neural Networks (SNNs) have discrete outputs (the number of spikes over time for each output neuron), it is easier to incur in vanishing gradients as the perturbation approaches the decision boundary. Therefore, we had to make changes to the al- gorithm to take this into account. Firstly, we found that clamping the perturbation at every iteration of DeepFool, so that it was no smaller than a value 7), offered protection against van- ishing gradients. 1) was treated as a hyperpa- rameter that should be kept as small as it can without incurring in vanishing gradients. Secondly, to account for the discreteness of event-based data, we rounded the output of SoorseFool to the nearest integer each iteration. Finally, SparseFool normally involves upper and lower bounds and on pixel values (normally set, for images, to = 0; u 255). We exploit these to enforce the binary constraint on the data (L = u 1), or, in the on-chip experiments, to fix a maximum firing rate in each time bin, which is the same as that of the original input (1 = 0; u = max(input)..
Adversarial event list
Adversarial raster
(x=12, y=17, t=3450, p=+)
0
0
(x=24, y=5a, t#3451, p
8
8
8
8
8
0
(x=98, y=121, t=3452, p=+)
(x=21, y=4s, t#3453, p=+)
1
0
8
8
8
(x=124, y=34, t=3457, p=-)
8
0
0
(x=87, y=13, t=3559, p=-)
(x=10, y=110, t=3562, p=+)
2
10
ms
On-chip testing
Simulated testing
Figure Schematic of the attack procedure on DVS data.
Adversarial patches As the name suggests, adversarial patches are perturbations that are accu- mulated in a certain region of the image. The idea is that these patches are generated in a way that enables the adversary to place them anywhere in the image. This attack is targeted to a desired label, and universal, i.e. not specific to an input. To test more realistic scenario where an adversary could potentially perform an attack without previous knowledge of the input, we apply these patches to the IBM hand gesture dataset. We note that the prediction of the CNN trained on this dataset is mostly determined by spatial location of the input. For example, the original input of "Right Hand Wave" is not recognised as such if it is shifted rotated by substantial amount. In order to simulate effective realistic attacks, we choose to limit both computed and random attack patches to the area of where the actual gesture is performed. As in Brown al. (2017), we generate the patches using PGD on the log softmax value of the target output neuron. PGD is performed iteratively on different images of the training set and the position of the patch is randomised after each sample. For each item in the training data, the algorithm updates the patch until the target label confidence has reached a pre-defined threshold. The algorithm skips the point if the original label equals the target label. This process is repeated for every training sample and for multiple epochs. To measure the effectiveness of our computed patches, we also generate random patches of the same size, and measure the target success rates. In random patch, every pixel has a 50% chance of emitting spike at each time step.
2.2 DATASETS AND DATA PREPARATION
Binarised MNIST We tried our methods on three datasets. The first is a binarised version of MNIST (BMNIST for short), which is derived from the popular MNIST Handwritten Digits database (LeCun & Cortes 2010), binarised so that pixel values 0 to 127 are mapped to white, and 128 to 255 mapped to black. No other preprocessing is applied. This is not dataset of DVS recordings: we use in order to compare our white box attacks against the SCAR attacks for binary datasets mentioned above (Balkanski et al. 2020)
4

---

Under review as a conference paper at ICLR 2022
Neuromorphic MNIST Our first DVS benchmark is NMNIST (Neuromorphic MNIST), which consists of 300 ms-long recordings of MNIST digits that are captured using the saccadic motion of a DVS sensor (Orchard et al. 2015). This is the most commonly used DVS benchmark dataset for simpler tasks: since digits are only translating through the frame without changing, temporal features are not necessary for classification. When testing the spiking network, and for creating adversarial examples, each sample is fed to the network as a sequence of 5 ms-long binary frames. Additional spikes that fall in the same pixel within the same 5 ms window are discarded, so that each bin can contain either 0 or I events per pixel. The resulting data is a binary array (referred to as "raster") of dimensions (t. p. x. y) (60, 34, 34), where 300 ms/5 55 ms 60 the number of time bins, p 2 are the polarity channels, and y 34 is the spatial resolution of the recording.
IBM Gestures For more advanced event-based vision benchmark, we used the IBM Gestures dataset, which consists of recordings of 11 classes of human gestures, captured under three different lighting conditions (Amir et al. 2017). Here, unlike the previous cases, the model must have some ability to process features in time, c.g. to distinguish between clockwise and counterelockwise hand motion in the same spatial position. The length of each gesture recording varies between 4 and 7 seconds. In this work, we never test on the full length of the recording at once, but we use 200 ms slices as the fundamental unit of the dataset. The data fed to the spiking network at test time are the same 200 ms samples, with time discretised in 10 ms bins. As above, spikes are capped to per pixel per time bin. The dimensions of the resulting raster are (t, x. y) (20, 128, 128). The experiments designed to run on the chip were binned at a higher time resolution of 2 ms since the neuromorphic hardware is capable to process events in continuous time.
2.3 NETWORKS
For the BMNIST experiments, we use a non-spiking network, similar to the one used in Balkanski let al.] (2020): two 3 x 3 convolutional layers (32 and 64 channels each), with ReL.U activations, followed by 2 x 2 max-pooling, dropout, and a fully connected layer of 128 features, projecting onto the final layer of 10 output units. The network is trained for 50 epochs at batch size 64, using the Adam (Kingma & Ba 2014) optimiser with learning rate 10 on a cross-entropy loss function. The network reached a test accuracy of 99.12%.
The spiking networks used for the NMNIST and IBM Gestures tasks are simulated using a Py based SNN library which simulates non-leaky, linear integrate-and annrrate-andfiine fire neurons with no synaptic dynamics, equivalent to the ones emulated by the neuromorphic chip. In this neuron model, the inputs to each neuron are multiplied by the input weight and simply added to the neuron's membrane potential. The neuron spikes as soon as its membrane potential reaches a threshold, which is always set to 1. The threshold value is then subtracted from the membrane potential. The network's output label is the one corresponding to the output neuron that spikes the most, over the timespan during which the input is presented. There are no bias terms in our SNN's convolutional and linear layers.
The models used for NMNIST were trained using the "weight transfer" method, whereby equiva- lent CNN is trained on accumulated frames (i.c. summing the data over the time dimension), and the CNN weights are transferred to the spiking network with thresholds set 1 (Rueckauer al. 2017 Sorbaro et al. 2020). The ANN was trained with Adam at batch size 64 with learning rate 10 for 10 epochs. We then rescaled the weights by layer-wise global factors so that the 99th percentile of activity was the same at each layer, as described by Rueckauer et al. (2017). The model we used consists of three convolutional layers of 20, 32, and 128 channels (kernel size for the first, for the other two), each followed by ReLU activation and 2 x 2 vverage-pooling The convolutional stack followed by a fully connected layer with feature size 500, which projects onto the 10 output units. The network achieves 84.93% classification test accuracy.
For the IBM Gestures task, training is done using even for feed-forward networks, because of the neurons' internal states, which persist in time. We make use of surrogate gradient in the backwards pass to enable learning despite the discontinuous nature of spikes (Neftci et al.) 2019): for gradient purposes, the neuron's nonlinearity is treated as a ReLU with zero- pero-point placed at a value threshold window. The window value is set to 0.5. For the simulated experiments, we used network with convolutional layer of kernel size 2, stride 2, and 8 channels, followed by two convolutional layers of kernel size 3 and 8 channels, and a fully connected layer of 64 channels that projects to the 11 output units. After every convolutional layer,
(BPTT), required
5

---

Under review as a conference paper at ICLR 2022
4
Zero Two
Ona Dna-SSever
Six Zerm
Five Thows
-f
Four- Min
Elsht
Threp Nime
Seven Nine
Sevan
Two
Cie
Nine Fhue
Three
Ar CUU
THF Siive- -M Pet Clorkenne
o.k.ms
10.0ms
Figure 2: Examples of adversarial inputs on the BMNIST (top), NMNIST (middle) and IBM Ges- tures (bottom) datasets, as obtained by the SparseFool method. The captions show the original (true) label, correctly identified, and the class later identified by the model. The data was re-framed in time for convenience of visualisation. Red indicates added spikes. In the BMNIST examples, blue indicates removed pixels. We note that in the lower-dimensional BMNIST case, the effect of the attack is semantically interpretable: for example, adding a stroke that closes the upper left part of "7" makes it look like "9" not only for the network but also for a human observer. See the supplementary video for more examples and motion visualisation.
batch-norm, spiking activation, and 2 x 2 average pooling are inserted. This network achieves a classification test accuracy of 84.2%. The network used for the on-chip experiments has slightly different architecture and does not have batch-normalisation layers to make it compliant with the hardware.
2.4 EXPERIMENTS ON THE NEUROMORPHIC CHIP
In order to verify our attack strategies more realistic scenario, we ran our experiments on neuro- morphic hardware"] [ which is especially suited for SNN inference due to its asynchronous nature. We use digital, convolutional neuromorphic chip designed for computer vision applications. Weight precision, number of computations per second and throughput are typically reduced as the hardware is optimised for very low power consumption. This can lead to a degradation in prediction aceu- racy when compared to simulations. Because the networks detailed in the previous sections have to be modified in order to make them suitable for neuromorphic on-chip inference, their weights are rescaled and discretised as required by the chip's 8-bit weight precision.
'The name and brand of the chip in question have been redacted in this version for anonymisation purposes, and will be included in the final version of the paper.

---

Under review as a conference paper at ICLR 2022
Success Rate (%)
Median Blapsed Time (s/sample)
Median No. Queries
Median L
Attack
1BM NMNIST BMNIST
SCAR PGD
100.00
1.14
1175
98.89
0.16
102
Probabilistic PGD
99.70
0.54
275
23
SparseFool (y) = 0.2, A = 2)
99.90
0.08
11
14
PGD
48.63
72.56
1052
_1
Probabilistic PGD
54.46
68.35
774
522
SparseFool (n) 0.2, 2)
99.76
30.22
45
254
SparseFool (n) 0.5, 2)
99.88
13.08
26
SparseFool (n) 0.1, 3)
100.00
2.78
11
310
SparseFool (n) 0.1, 2)
99.87
2.57
11
200
SparseFool (1) = 0.1, = 1)
97.69
3.02
17
116
Samples for which the attack was unsuccessful were considered have L undefined. Because PGD fails more than half of the time, the median undefined.
Table I: Comparison of attack strategies (1000 samples). SCAR was implemented according to the pseudo code in Balkanski al. (2020) and PGD run for 50 iterations. SparseFool takes only a fraction the time compared to PGD while obtaining much sparser results at almost perfect success rate on Neuromorphic MNIST. The input size for this dataset is set (60,2,34.34). We also use SparseFool to attack samples from the IBM Gestures dataset at different values of X, a parameter trading-oft speed and sparsity. The success rate is here defined as the fraction of samples that were initially correctly classified, for which the attack algorithm converged to an adversarial example that the network classifies incorrectly.
As this work focuses on white-box attacks, we first computed the adversarial examples using the network simulation on the computer, then tested both original and attacked spiketrains in simulation and the chip. The simulation and the attack work in discrete time, while the chip receives events continuous time. In order to convert the discrete-time attacked raster back to list events for the chip, compared the original and attacked rasters, identifying new events added by the attack and adding them to the original list (Figure I]. We empirically found very few events removed by SparseFool (see supplementary section and chose to ignore removals for on-chip experiments.
RESULTS
3.1 SPARSEFOOL ATTACKS ON BINARY AND DVS DATA
Table I compares the different algorithms on Binary-MNIST and shows that SparseFool finds suc- cessful adversarial examples with low median L (i.c. number of perturbed pixels), while requir- ing very low median execution time. Figure 2 (top) illustrates samples of perturbations found by SparseFool and the corresponding label that was predicted by the network after applying the perturbation. Because of the small sample size and the fact that there is no time dimension, Binary- MNIST enables us to compare SparseFool to other, more inefficient methods. However, more real- istic datasets are needed truly evaluate the feasibility of applying these algorithms.
After having established that SparseFool can efficiently and reliably generate sparse perturbations on discrete data, we evaluated SparseFool's performance on the DVS benchmarks for different hy- perparameters (Table []. indicates the minimum step size for updates to the perturbation: higher values of 1) find less precise perturbations (larger 7,0 values), but are sometimes needed in order to prevent gro-diraaeent issues within the algorithm. À is the sparsity parameter: lower À (with minimum of I) yields sparser results, but gives slightly lower success rate, Overall we consistently found that SparseFooo performs better than PGD and Probabilistic PGD, both in terms of success rate and in the number of added or suppressed events. Additionally, it requires far fewer iterations and therefore converges successful attacks. Supplementary section provides further information on the characteristics of the resulting samples.
quickly. Figure 2 and the supplementary video show examples of
7

---

Under review as a conference paper at ICLR 2022
CH HLH Nove
SH CV. RM CCN
mg
mz
80.0 ms
$20.0
160.0
Figure 3: Examples of adversarial patches successfully applied to a single "right hand clockwise" data sample, with different target classes. See also the supplementary video for motion visualisation and more examples of successful patch attacks.
3.2 VALIDATION ON NEUROMORPHIC HARDWARE
We randomly chose 1000 snippets, each 200 ms long, from the IBM Gestures dataset, on which ran the SparseFool attack. Out of these, 833 were successfully classified by the network and were therefore eligible for an attack; the attacks converged and were successful in simulation in 777 cases. We presented these 777 successful attacks to the chip, alongside the un-attacked original data, finding that 96.9% (753) of the originals are successfully classified by the chip, and 85.3% of the attacks are able to fool the chip (663) too. A possible reason for this discrepancy lies in how the chip is limited in computing capacity by weight quantization and restricted throughput per time unit, which causes some of the input events be dropped. The conversion of binned data back into lists of spikes, discussed in the Methods section, is necessarily lossy at this time. In terms of attack efficiency, we observe median I,¹ distance (i.e., difference the attacks that were successful on chip, corresponding to median 9.3% increase in the number of spikes per sample. The full distribution is shown in Figure ST (bottom left). Figure STT also shows the time profile of the perturbation and how the network classified the data after the attack.
number of spikes) of 903 among
3.3 ADVERSARIAL PATCHES
Although we have demonstrated that one can achieve high success rates on custom spiking hard- ware that operates with microsecond precision, the applicability of this method is still limited, the adversary needs to suppress and add events at high spatial and temporal resolution, thus making the assumption that the adversary can modify the event-stream coming from the DVS camera. Further- more, SparseFool assumes knowledge of the model and requires computing the perturbation offline, which not feasible in a timely manner. In more realistic setting, the adversary assumed to generate perturbations by changing the input the DVS camera receives on the fly.
Using the training data from the IBM Gestures dataset, we generated adversarial patch for each target class with high temporal precision (event samples of 200 ms were binned using 0.5 ms-wide bins) and evaluated the effectiveness in triggering a targeted misclassification both simulation and on-chip using the test data. To simulate spatial imprecision during deployment, each test sample was perturbed by a patch that was randomly placed within the area of the original gesture. Table 2 summarises our findings on target success rates for generated and random patches. Simulated results show high success rates, and on-chip performance shows a slight degradation, which can be expected due to weight quantization on the tested specialised hardware. We also found that the chip had trouble processing inputs because most of the added patch events occurred concentrated in the beginning of recordings in a large transient peak. In one case, the targeted attack for label "Arm Roll" mostly fails on chip as not all events are processed, which makes harder to discriminate between similar labels such as "Hand Clap", similar gesture that occurs in the same central spatial
8

---

Under review as a conference paper at ICLR 2022
Hand
RH Wave
LH Wave
RH Clockwise
RH Counter Clockwise
LH Clockwise
LH Counter
Anm
Air Drum
Air Guitar
Target label
clop
Clockwise
Roll
Other
Adversarial patch
90.3
99.0
89,8
87.3
79.7
49.7
51.5
79.1
92.3
64.7
Adv, patch (on-chip)
94.0
89.0
94.1
81.3
65.1
35,9
43.8
5.0
82.7
87.3
66.8
Random
18.8
50.7
77.0
3.6
Rand. patch (oa-chip)
43
76.8
72.2
0
9.0
2,4
17.7
Adversarial patches for different target labels were evaluated on- and off-chip. Shown here are the success rates in percent for each target label. An attack is considered successful if the original label is not the target label and the network predicts the target label when the patch is applied.
Table 2:
location. This could somewhat be mitigated by limiting the number of events in a patch to ensure that they could all be correctly processed on the chip.
We compare this result with a baseline of randomly generated patches, and we observe that two labels, namely "Left" and "Right Hand Wave" subsume all other attacked labels in this case. This hints that randomly injecting events in various locations is not enough to perturb network prediction to a desired label and that our algorithm succeeds in finding a meaningful patch. To summarise, adversarial patches are effective in triggering a targeted misclassification both on- and off-chip compared to randomly generated ones. Figure 3 and the supplementary video show examples of successful patch attacks. Importantly, these attacks are universal, meaning that they can be applied to any input and do not need to be generated for each sample.
DISCUSSION
We studied the possibility of fooling spiking neural networks through adversarial perturbations to dynamic vision sensor data, and verified these perturbations on a convolutional neuromorphic chip. There were two main challenges to this endeavour: the discrete nature of event-based data, and their dependence on time. This translated, in practice, in the need for an extra temporal dimension, and in different sparsity requirements, because the magnitude of the perturbation is measured in terms of number of events added or removed. For this purpose, we adapted the sparse adversarial algo- rithm SparseFool, and showed that it achieves high convergence rates on time-discretised samples of the Neuromorphic MNIST and IBM Gestures datasets. Empirically, we observe that the algorithm mostly resorts to adding, rather than removing, input events, and the number of new events neces- sary to fool the network varies significantly from sample to sample. In the best cases, the attack requires the addition of less than hundred events over 200 ms sample, an increase of a few per- cent. With this, we have proven adversarial examples in DVS data are possible, and, to the best of our knowledge, we were also the first to show that the perturbation is effective in a network deployed on neuromorphic chip. As the history of adversarial attack algorithms shows, future research in this field may well find adversarial perturbations that are even less noticeable, detectable, or com- putationally intensive. One should be aware of this possibility when deploying any neuromorphic devices using DVS technology together with neural network models in all contexts where malicious attacks may have serious consequences, such autonomous driving, surveillance, or control. For this reason, it is also important to consider how to counter these attacks. Defence mechanisms such as adversarial-wware training exist and can provide better robustness. In preliminary work outlined in supplementary section 2 we apply one such method to SparseFool attacks.
SparseFool computes perturbations offline, and it is currently not obvious how do this on the fly on a live stream of DVS events. Therefore, we also investigated a more realistic setting, where an adversary can, with low spatial, but high temporal precision, inject spurious events in the form of a patch inserted into the visual field of the DVS camera. We showed that we can generate patches for different target labels, which trigger targeted misclassifications with high precision. Although these patches require a much higher amount of added events, they do not require prior knowledge of the input sample and therefore offer a realistic way of fooling deployed convolutional neuromorphic systems. A natural next step would be to understand whether it is possible to build real-world patches that can fool networks when shown to the DVS camera from a variety of distances and orientations, as Eykholt et al. (2018) did for photographs. Additionally, it will be interesting to see how important knowledge about the architecture is and if one can generate adversarial patches by having access to network that differs from the deployed ones.

---

Under review as a conference paper at ICLR 2022
5 ETHICS STATEMENT
The authors aim to minimise the impact of potential attacks on deployed SNNs by contributing to overall understanding and supporting public discussion thereof.
subset the authors are currently employed by the neuromorphic chip manufacturer, which could be perceived conflict of interest. We did our best to counteract this by making available all source code used in the experiments and we encourage other researchers to reproduce our results. The other authors report current conflicts of interest.
REPRODUCIBILITY STATEMENT
The authors made available all code that was used to generate results and plots in this paper, which we attach as supplementary material. The code will made publicly available after the review process. The libraries used for SNN simulation and DVS data management are available as open- source code. The DVS datasets are available online. The neuromorphic chip is available for research purposes. When reproducing experiments on chip, results are expected to differ slightly due to variations in the manufacturing process.
REFERENCES
Naveed Akhtar and Ajmal Mian. Threat of adversarial attacks on deep learning in computer vision: A survey. JEEE Access, 6:14410-14430 2018. doi 10.1109/ACCESS.2088..8000
Arnon Amir, Brian Taba, David Berg, Timothy Melano, Jeffrey McKinstry, Carmelo Di Nolfo, Tapan Nayak, Alexander Andreopoulos, Guillaume Garreau, Marcela Mendoza, Jeff Kusnitz, Michael Debole, Steve Esser, Tobi Delbruck, Myron Flickner, and Dharmendra Modha. low power, fully event-based gesture recognition system. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7388-7397, 2017. 10011.1100//VP222177778
Alireza Bagheri, Osvaldo Simeone, and Bipin Rajendran. Adversarial training for probabilistic spiking neural networks, 2018
Eric Balkanski, Harrison Chase, Kojin Oshiba, Alexander Rilee, Yaron Singer, and Richard Wang. Adversarial attacks on binary image recognition systems. CoRR, abs/2010.11782. 2020. URL
https::/aaxxte org. abs 2010.1000 11782
Battista Biggio and Fabio Roli. Wild patterns: Ten years after the rise of adversarial machine learning, Pattern Recognition, 84:317-331, 2018.
Tom B. Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer. Adversarial patch.
CoRR, abs/1712.00665,, 2017. URL http: / //tp:v/arivvvv//vi org/ wwgg/bs........o...........ee.e abs /1ll//111.... 09665
Stefano Caviglia, Luigi Pinna, Maurizio Valle, and Chiara Bartolozzi. An event-driven posfet taxel for sustained and transient sensing. In 2016 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 349-352. IEÉE, 2016.
Vincent Chan, Shih-Chii Liu, and André van Schaik. Aer ear: A matched silicon cochlea pair with address event representation interface. IEEE Transactions on Circuits and Systems I: Regular Papers, 54(1):48-59 2007.
Mike Davies, Narayan Srinivasa, Tsung- Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, Yuyun Liao, Chit-Kwan Lin, Andrew Lines, Ruokun Liu, Deepak Mathaikutty, Steven McCoy, Arnab Paul, Jonathan Tse, Gu- ruguhanathan Venkataramanan, Yi-Hsin Weng, Andreas Wild, Yoonseok Yang, and Hong Wang- Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro, 38(1 (11)999999, 2018. doi 20W..M.M..008.1111333300 12130359
Steven Esser, Paul Merolla, John V Arthur, Andrew S Cassidy, Rathinakumar Appuswamy, Alexander Andreopoulos, David J Berg, Jeffrey L McKinstry, Timothy Melano, Davis R Barch, et al. Convolutional networks for fast, energy ineryy-efficiem neuromorphic computing. Proceedings of the national academy of sciences, 11 113(41):1 14411-1111111 1446, 2016.
10

---

Under review as a conference paper at ICLR 2022
Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical-world attacks on deep learning visual classification. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1625-1634, 2018.
Davide Falanga, Kevin Kleber, and Davide Scaramuzza. Dynamic obstacle avoidance for quadrotors with event cameras. Science Robotics, 5(40), 2020.
Steve B Furber, David Lester, Luis A Plana, Jim D Garside, Eustace Painkras, Steve Temple, and Andrew D Brown. Overview of the spinnaker system architecture. IEEE Transactions on Computers, 62(12) 2467, 2012.
Guillermo Gallego, Tobi Delbruck, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew Davison, Jörg Conradt, Kostas Danillidis, et al. Event-based vision: A survey. arXiv preprint arXiv:1904.08400 2019.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax, 2017.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. International
Conference on Learning Representations, 12 2014.
Beat Kueng, Elias Mueggler, Guillermo Gallego, and Davide Scaramuzza. Low-latency visual odometry using event-based feature tracks. In 2016 JEEE/RSJ International Conference on Intel- ligent Robots and Systems (IROS), pp. 16-23. IEEE, 2016.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URtp::/://
lecun. cow. exdb mmmist//s//////////m//mmm/////n/
Ling Liang, Xing Hu, Lei Deng, Yujie Wu, Guoqi Li, Yufei Ding, Peng Li, and Yuan Xie. Ex- ploring adversarial attack in spiking neural networks with spike-compatible gradient. CoRR, abs/2001.01587 2020. URL (ttt:htp://arxt
org/ abs/2001. 01587
Qian Liu, Ole Richter, Carsten Nielsen, Sadique Sheik, Giacomo Indiveri, and Ning Qiao. Live demonstration: face recognition on an ultra-low power event-driven convolutional neural network asic. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 0-0, 2019a.
sensory systems. Current opinion in neurobiology, Shih-Chii Liu and Tobi Delbruck. Neuromorphic sensory 20(3):288-22) 2010.
Shih-Chii Liu, Bodo Rueckauer, Enea Ceolini, Adrian Huber, and Tobi Delbruck. Exent-driven sens- ing for efficient perception: Vision and audition algorithms. IEEE Signal Processing Magazine, 36(6):29-37, 2019b.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks, 2019.
Alberto Marchisio, Giacomo Pira, Maurizio Martina, Guido Masera, and Muhammad Shafique. Dvs-attacks: Adversarial attacks on dynamic vision sensors for spiking neural networks. In 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1-9, IEEE, 2021.
Apostolos Modas, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. Sparsefool: few pix- els make a big difference. CoRR, ftbb//811.02245. 2018. URL http:://axiv..rrgg /bb// 181 1611.0 02248
Seyed-Mohsen Moosavi-Dezfood Alhussein Fawzi, and Pascal Frossard. Deepfool: a simple and accurate method to fool deep neural networks. CoRR, abs/1511.04599 2015. URL http:
04599
Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient- based optimization to spiking neural networks. IEEE Signal Processing Magazine, 66(6):51-63 2019.

---

Under review as a conference paper at ICLR 2022
Garrick Orchard, Ajinkya Jayawant, Gregory K. Cohen, and Nitish Thakor. Converting static image datasets to spiking neuromorphic datasets using saccades. Prontiers Neuroscience, 9:437, 2015. ISSN 1662-453X. doi htt.:388//mms.221550004.4..
URL https:// http::/wwwwwrrrooo Frontiersin.
201 5. 00437
Bodo Rueckauer, tulia-Alexande Lungu, Yuhuang Hu, Michael Pfeiffer, and Shih-Chii Liu. Con- version of continuous-van valuue deep networks efficient event-driven networks for image classifi- cation. Frontiers in neuroscience, 1 11:682, 2017.
Saima Sharmin, Nitin Rathi, Priyadarshini Panda, and Kaushik Roy. Inherent adversarial robustness of deep spiking neural networks: Effects of discrete input encoding and non-linear activations. In European Conference Computer Vision, pp. 399-4194 Springer, 2020.
Martino Sorbaro, Qian Liu, Massimo Bortone, and Sadique Sheik. Optimizing the energy consump- tion of spiking neural networks for neuromorphic applications. Frontiers in neuroscience, 14:662, 2020.
Jan Stuijt, Manolis Sifalakis, Amirreza Yousefzadeh, and Federico Corradi, jubrain: An event-driven and fully synthesizable architecture for spiking neural networks. Frontiers in neuroscience, 15: 538, 2021.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruma, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint 1312.6/99, 2013.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning, pp. 7472-7482. PMLR, 2019.
12

---

Under review as a conference paper at ICLR 2022
SUPPLEMENTARY MATERIAL
EMPIRICAL ANALYSIS OF THE RESULTING SPARSEFOOL PERTURBATIONS
Profile of perturbation in time
Confusion matrix, NMNIST
14
SS
2
80
70
16
S5
8
14
12
5
60
16
10
50
34
12
12
25
3
40
Gesures
150
orggnnal ev. addec
10
28
30
100
20
3
13
0
0
-10
16
100
300
Time sample (ms)
Label after attack
Distribution of perturbation sizes
Confusion matrik, IBM Gestures fon chip)
250
NINNIST IEM Cestures
Hand Clap
35
0
-00
60
200
Number of somples
50
Original lebel
150
6
100
CCW
30
Arm
20
Air Drums
2
Guiteer
Other 10
0%
10%
20%
30%
40%
50%
60%
70%
3
11
1.0
Events wwccd/temevod relative to original number
Label after attack
Figure S1: Properties of the adversarial perturbations found by SparseFool, for two experiments: NMNIST (in simulation,
0.5, A 2) and IBM Gestures (as tested on chip). Top left: Number of events in time within each data sample. The shaded areas represent the 0. 1-0. interquantile range (not shown for the 'original' curve in the bottom panel). The perturbation tends to consist of spikes added the beginning of the sample, especially for NMNIST which does not rely on temporal structure for inference. Very few spikes are removed, which justifies the choice of ignoring removed spikes in on-chip experiments. The periodic structure of NMNIST samples is intrinsic to the dataset, recorded with saccades. Bottom left: Distribution of increase in number of spikes after the attack, relative to the original number. Right: Matrices showing the label identified by the network when presented with the adversarial examples, given the original label, for the two experiments. Most IBM Gestures classes are perturbed towards the 'other' class, while there is no clear structure in the NMNIST case. L.H Left Hand, RH Right Hand, (C)CW (Counter) ClockWise.
With the aim of gaining more insight into the behaviour of our methods, we studied the character- isties of the perturbations resulting from SparseFool attacks in more detail. For this, we chose two specific experiments: SparseFool run on NMNIST with hyperparameters 1) 0.5 and 2; and the on-chip IBM Gestures experiment. First, we empirically notice that SparseFool-basee turbations rarely involve the removal of events. In the NMNIST experiment considered here, an average of 7.6 events is removed from each sample, compared to an average of 2 214 spikes added. This justified our choice to ignore removed events in the course of the on-chip experiments.
per-
As is evident from the examples in Figure 2 we also find that SparseFool's adversarial perturbations tend to consist in the insertion of spikes the beginning of the sample, with only few spikes added later in time. The top left panel of figure [ST shows the time profile of the perturbations in detail. We believe this a consequence of the use of the non-leaky neuron model. In non-leaky neurons, information can be stored indefinitely in the membrane potential, so early spikes have further chance of contributing to a spike later in time, and are more effective compared to events added later in the sample. This effect is also present in the IBM Gestures experiment, but looks less prominent, possibly because networks trained with BPTT on data with richer features in time have non-trivial dynamics. In this sense, expect this phenomenon to be further reduced or
13

---

Under review as a conference paper at ICLR 2022
disappear entirely when the task is strictly linked to the time evolution of the input signal, such as in auditory speech recognition. The timing of adversarial events could potentially be used for model interpretability purposes, to measure how much the model relies temporal features.
Further to the median values reported in table T the lower left panel of figure ST reports the full distributions of the number of added removed events (I, distances). Here, display the numbers relative to the original number of events the sample. We notice a minority of cases where the attack successful only at the cost of a very significant injection of events.
Finally, we analysed the statistics of classes identified by the networks after the attack. SparseFool used an "untargeted"" algorithm, i.e. it attempts to change the output of the network but without requirements on what the new class should be. Unsurprisingly. the "other gesture" class is a natu- ral target class for many ground truth classes, but there are some exceptions which we find rather natural, such as "left hand wave" gestures being most often converted to "left hand clockwise". Con- versely, we observe no dominant target class in the NMNIST experiment. If the target class structure is undesirable, targeted attacks can be used instead.
2 DEFENCE VIA ADVERSARIAL TRAINING
1.00
Success rate (%)
1500
Median
-.98
1000
0.96
500
0.00
0.01
0.05
TRADES frue
Figure S2: Success rate and median 7 of SparseFool for networks trained with TRADES robustness based on PGD attacks.
Once it is known that model or system is sensitive to a certain type of adversarial attack, is natural investigate whether there is a way to build a network that is more resistent to these attacks. We therefore experimented with adversarial training using the maddeoff-inspreed Adversarial DEfense via Surrogate- minimization (TRADES) method (Zhang et al., 2019). The method consists of adding new term to the loss function during training, which minimises the Kullback-Leibler divergence between the output the network when the original input presented, and the output when the adversarial example is presented:
Lyob = C+ Brob
DKL. (f ff
B
Here, B is the batch size, Boo is the parameter that defines the trade-off between robustness and accuracy, the network and Xadv is the adversarial input. Although networks that were trained using SparseFool would probably be more robust, we opted for PGD at training time, since it can be easily batched - but we attack the resulting networks using SparseFool. We used PGD in the 7 domain and chose = 0.5 as the maximum perturbation, with Npgd = 5 attack steps. We also did not greedily chose the best indices to flip as described in section 2.1. Even this was much simplified version of the PGD attack, we found that this configuration produced perturbations with reasonable Hamming distances while being extremely efficient, and it was sufficient in inducing some level of robustness.
From the results in Figure S2 we note that the success rate is still quite high despite the adversarial training for the choices of Bub we considered. However, given the fact that SparseFool aims at find- ing the smallest perturbation that triggers a misclassification, this is expected, and there is already noticeable increase in the number of added spikes required, which is indeed sign of robustness. In other words, the adversarially-traine network requires stronger and less stealthy attacks before is fooled. Further work is required for a comprehensive investigation of other possible defence strategies.
14


Reviews:
[{"review_id": "g-9SsoMjAUv", "reviewer": "Reviewer_FiWC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThis paper draws a conclusion through rigorous and sufficient experiments, which provide quantitative results to prove the effectiveness of the attack algorithms on dvs data. How to deal with the discrete nature of event-based data and their dependence on time is explained clearly. Also, the authors verified the effectiveness of adversarial examples on neuromorphic chip for the first time.\nWeaknesses:\n1)More details about the existing attack algorithms after adapted and how the algorithms work on the discrete-time raster could be explained.\n2)Some explanations about the meaning of the variable symbols such as ς ,r_i ,α_i and so on in the appended algorithms could be added.\n3)More attack algorithms such as PGD and probabilistic PGD could be used on neuromorphic chip and calculate success rate in order to compare with the algorithm Sparsefool.\n", "summary_of_the_paper": "This paper investigated how to adapt existing attack strategies in order to work with the discrete nature of event-based data. Also, the authors proved that Sparsefool can efficiently and reliably generate sparse perturbations on discrete data. Besides,the authors validate the adversarial examples on neuromorphic chip and the perturbation is effective after modified the networks. In a more realistic setting, the authors use adversarial patches to trigger a targeted misclassification. MNIST and IBM Gestures datasets are used to test the attack algorithms as a sequence of binary frames.", "summary_of_the_review": "This paper verified the attack algorithms can be adapted to the dvs data. The Sparsefool algorithm achieves high success rates through adding and removing a few events. The authors proposed a next step, which building real-world patches that can fool networks. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "avm-jZkZMtH", "reviewer": "Reviewer_jrcY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ overall this is a solid paper, without unjustified claims, and with a straightforward message: adversarial attacks on event-based vision works\n+ the paper is well written, the figures are clearly showing the effects of the adversarial attacks\n+ a variety of different attacks is tried out, including adversarial patches\n+ the paper is to the best of my knowledge the first to try out the effect of adversarial attacks on neuromorphic hardware\n\nWeaknesses:\n- as the authors note, there is concurrent work by Marchisio et al. 2021, which investigates a very similar question, and even goes beyond that by investigating potential defenses against adversarial attacks. The present paper only shows the possibility of attacks, but does not talk about defenses. Therefore the novelty is severely limited.\n- the description of the adversarial attacks and their adaptation to the event-based domain is very short and could only be understood with the help of the supplementary material. I would propose expanding this description to make the main paper more self-contained.\n- the results of the paper are not really surprising, and I would have hoped for more specific comparisons how adversarial attacks differ between conventional and event-based vision, and also some outlook into defenses", "summary_of_the_paper": "The paper presents proof of the viability of adversarial attacks on CNN processing event-based vision data. The paper investigates adaptations of well-known white box attacks to the event-based and spiking domain, validates the claims on three public benchmarks, and investigates the effect of the adversarial attacks directly on neuromorphic hardware. The main result is that, just like for conventional computer vision, adversarial attacks can have a big effect on the prediction of CNNs for event-based vision inputs. The attacks also translate to networks run on neuromorphic hardware.", "summary_of_the_review": "While the paper is formally correct and well written, there are other published papers that have shown the main claims of the paper, and have even gone some steps further by analyzing possible defenses against adversarial attacks. Therefore the novelty is too limited, and the key findings are not surprising. There are some new elements, e.g. the analysis of attacks on real neuromorphic hardware, but overall this is too little for acceptance at ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Vm9Cy42vPDq", "reviewer": "Reviewer_2Mf5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1.\tIt is not clear how the adversarial attack algorithms that are commonly used for discrete data are applied to event-based data in this work. Which are the design choices that are done to develop/modify these methods? Since these modifications constitute the main contributions of this work, they should be comprehensively discussed in the main manuscript, rather than in the supplementary material. The description in Sections 2.1 and 2.2 does not contain sufficient details.\n2.\tIn this work, the novelty is low because the authors use already existing methods applied to an unexplored system.\n3.\tThe experimental setup is not clear. Details like training hyperparameters, ANN-SNN conversion parameters, and HW-level constraints of the neuromorphic chip that has been considered in the implementation.\n4.\tIt is ok to keep anonymous the name and brand of the in-house neuromorphic chip on which the experiments have been conducted, but the experiments can also be conducted on publicly available neuromorphic chips, to ease the comparison with related works.\n5.\tFor all the attacks, in particular for the universal attack, the stealthiness, i.e, how much the adversarial examples are noticed by the human eye, should be discussed.\n6.\tThe results can be extended with experiments in which adversarial defenses, such as adversarial training or noise filter, are applied to improve the robustness against the attacks.\n", "summary_of_the_paper": "In this paper, the adversarial attack algorithms have been adapted to be applied to event-based data for spiking neural networks implemented on neuromorphic hardware.", "summary_of_the_review": "This paper tackles a very important problem with rigor and soundness. However, some key points need to be addressed (see the main review section).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "S-jCJRFS3wL", "reviewer": "Reviewer_YwAR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is nicely written, and presented experiments and visualizations are well organized. Empirical validations of the proposed methods on neuromorphic hardware add a significant value to the paper with respect to the state-of-the-art.\n\nOn the other hand, methodological contribution of the paper is rather shallow, and Section 2.1 is not describing the used methods in detail. It should be clearer in mathematically describing the attack optimization objective within the SNN setting, e.g., on page 4 while describing SparseFool, some variables are mentioned out of context. Most of the important content on what is actually done for the attacks is described shortly in text and in the Supplementary algorithms. Overall, the authors implemented SparseFool attacks of [Modas et al, 2018] for SNNs, and demonstrated that SNNs with event-based inputs can be adversarially attacked. However authors also do not evaluate their attacks in comparison to the state-of-the-art PGD-like attacks with spike-compatible gradient proposed for SNNs by [Liang et al, 2020]. In fact this study is also not correctly referenced in the paper, mentioned as a work that utilizes: “… brute-force approaches that rely on heuristics to reduce the search space [Liang et al, 2020, …]”, which is not the case.\n\nI think evaluating a binary image recognition CNN on the BMNIST dataset with SCAR attacks is not a good choice of a baseline for this work, and seems distant from the main purpose of the paper. It would be more suitable to compare the same SNNs and proposed attacks with previously presented PGD-variant attacks for event-based data [Liang et al., 2020].\n\nOne of the earliest studies in this domain by Sharmin et al, \"Inherent adversarial robustness of deep spiking neural networks: Effects of discrete input encoding and non-linear activations.”, ECCV 2020, is also missing and should be discussed in the paper. In some ways, the PGD attacks presented by [Sharmin et al., 2020] for SNNs resembles to the standard PGD approach demonstrated in the paper (Section 1.2 of the Supplementary Materials). This could be highlighted and discussed in detail with the revisions. For instance, N-MNIST experiments are performed with models trained using ANN to SNN weight transfer, which [Sharmin et al., 2020] similarly demonstrated with their PGD-like attacks.\n\nResults presented in Table 2 (and the supplementary video) are very interesting. It seems like a random patch perturbation can be quite effective for a RH Wave label, but not at all for RH Clockwise or RH Counter Clockwise labels. Could this be a result of the relevant temporal information constructing a significant aspect of the attacking patches? At this point, what I think is another missing “random patch baseline” to compare against, is one that can exploit temporal information. For example, if one would invert the signs of the events of learned adversarial patches before addition (i.e., using un-meaningful inverses of the learned adversarial patches), would it already be a better attack than a naive random patch (which only exploits locations but not temporal information)?\n\nLooking at Fig 2 IBM Gestures examples, it looks like perturbation events that are much outside the region of movement are even sufficient to lead to misclassification (i.e., models are very vulnerable to arbitrary events occurring outside the relevant region). To compare this vulnerability against some potential robustness gains, I would be curious if the authors implemented any simple robust training approach that resembles to adversarial training? i.e., would the models be still very robust to additive events occurring at irrelevant locations? These observations can be discussed in the revisions.\n\nMinor comments:\n- Authors report that the SNN used for IBM Gestures has a test accuracy of 84.2%. For completeness, can the authors also report what is the on-chip test accuracy of the model following the modifications (e.g., removing batch-norm) and weight precision quantizations to make the model compatible with the hardware?\n- Please write in the caption of Fig 2 that those examples of adversarial inputs are crafted via SparseFool attacks (the reader discovers it much later in the text).\n- Caption of Algorithm 2 in the Supplementary Materials includes some interesting information that could be extended. Authors state that 50 iterations of PGD was sufficient and results did not improve afterwards. It would be interesting to plot this observation in the S.M. as it seems to demonstrate some inherent robustness.", "summary_of_the_paper": "Authors present white-box adversarial attack algorithms for spiking neural networks (SNNs) with event-based visual data inputs, and experimentally demonstrate vulnerability of SNNs in various settings. Extensions of existing adversarial attacks on convolutional networks (i.e., PGD and SparseFool) are presented and utilized in a setting for SNNs with event-based inputs. Experimental evaluations are performed on the Neuromorphic-MNIST and IBM Gestures dynamic vision sensor datasets, and validations of these attacks on neuromorphic hardware are performed.\n", "summary_of_the_review": "The paper is nicely written and organized from an empirical contribution perspective. However the paper is rather weak in depicting its methodological contributions in my opinion, and lacks comparisons/references to relevant SOTA works. I listed my major concerns in the main review, and would be willing to re-address my rating based on the authors’ responses and revisions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The manuscript investigates common adversarial attacks on event-based data for spiking neural networks.
They conclude that also in this setup adversarial attacks can strongly harm SNN performance.

Although the reviewers agree that the paper presents some solid results and is well written, there was also substantial criticism.

The main points were:
- It is not very clear how the usual attacks are applied to event-based data, and in general experimental setups are unclear.
- The methodological contribution of the paper seems limited.
- The novelty is limited, in particular Marchisio et al. 2021 investigates a very similar question and goes somewhat further. The author noted that their attacks are not deployed on neuromorphic hardware. A number of other important prior work is not discussed.
- The impact of adversarial defences was not considered.
- A more detailed comparison of event-based attacks to standard ANN attacks would be desired.

After the reviews, the authors have invested substantial efforts to improve the paper. These efforts were appreciated by the reviewers. In particular, the authors ran additional experiments using the defence method TRADES. The results showed that TRADES is effective, but the attack has still a large success.

In summary, the reviewers agree that this is a solid manuscript and an interesting direction, however, they see it finally slightly below acceptance threshold for ICLR.
Decision: Reject