EventRPG: Event Data Augmentation with Relevance Propagation Guidance
Authors: Mingyuan Sun
Donghao Zhang
Zongyuan Ge
WANG Jiaxu
Jia Li
Zheng Fang
Renjing Xu
Venue: ICLR 2024 poster (2024)
Keywords: Spiking Neural Network
Event Camera
Data Augmentation
Primary area: visualization or interpretation of learned representations
TL;DR: For the first time, we present two novel relevance propagation rules for SNNs, utilizing which a novel data augmentation method on event-based data is proposed, achieving SOTA results across several object and action recognition datasets.

Abstract: Event camera, a novel bio-inspired vision sensor, has drawn a lot of attention for its low latency, low power consumption, and high dynamic range. Currently, overfitting remains a critical problem in event-based classification tasks for Spiking Neural Network (SNN) due to its relatively weak spatial representation capability. Data augmentation is a simple but efficient method to alleviate overfitting and improve the generalization ability of neural networks, and saliency-based augmentation methods are proven to be effective in the image processing field. However, there is no approach available for extracting saliency maps from SNNs. Therefore, for the first time, we present Spiking Layer-Time-wise Relevance Propagation rule (SLTRP) and Spiking Layer-wise Relevance Propagation rule (SLRP) in order for SNN to generate stable and accurate CAMs and saliency maps. Based on this, we propose EventRPG, which leverages relevance propagation on the spiking neural network for more efficient augmentation. Our proposed method has been evaluated on several SNN structures, achieving state-of-the-art performance in object recognition tasks including N-Caltech101, CIFAR10-DVS, with accuracies of 85.62% and 85.55%, as well as action recognition task SL-Animals with an accuracy of 91.59%. Our code is available at https://github.com/myuansun/EventRPG.

Published as a conference paper at ICLR 2024
EVENTRPG: EVENT DATA AUGMENTATION WITH RELEVANCE PROPAGATION GUIDANCE
Mingyuan Sun Donghao Zhang', Zongyuan Ge*, Jiaxu Wang', Jia Li', Zheng Fang** & Renjing Xu
'he Hong Kong University of Science and Technology (Guangzhou) "Nottheastern University
Seeing Machines *Monash University 'Peking University
mi mingyuansun200gma.. edu, jwang4578connect. wwwpheetcckkkkuuute wwk.stgg..edeeo..ooo cn, zongyuan,
com, peter
com,
wet.yyuanngemmmaaass..o geêmonash.
gaga. Lee a.m.i..... com, fangzhengemail neu. edu, en, ren wen.ingnueenuustt-ooo edu,
ABSTRACT
Event camera, novel bio-inspired vision sensor, has drawn a lot of attention for its low latency, low power consumption, and high dynamic range. Currently, overfitting remains a critical problem in event-based classification tasks for Spik- ing Neural Network (SNN) due to its relatively weak spatial representation ca- pability. Data augmentation is a simple but efficient method alleviate over- fitting and improve the generalization ability of neural networks, and saliency- based augmentation methods are proven to be effective in the image processing field. However, there is no approach available for extracting saliency maps from SNNs. Therefore, for the first time, we present Spiking Layer-Timewwir Time-wise Rel- evance Propagation rule (SLTRP) and Spiking Layer-wiss Relevance Propaga- tion rule (SLRP) in order for SNN to generate stable and accurate CAMs and saliency maps. Based this, propose EventRPG, which leverages rele- vance propagation on the spiking neural network for more efficient augmentation. Our proposed method has been evaluated on several SNN structures, achieving state-of-the-ar performance in object recognition tasks including N-Caltech100 101, CIFARIO-DVS, with accuracies of 85.6 55.62% and 85.55% well action recog- nition task SL-Animals with an accuracy of 91.59%. Our code is available at https: / / thub. www./o/ouasase..c /evesst.nneeee........m.
INTRODUCTION
With the advent of event cameras, researchers have focused on applying the brain-inspired technique to achieve a variety of tasks, as the asynchronous nature of event cameras mimics the way the biological visual system works (Gallego et al., 2020). Event cameras record the change in brightness of each pixel, and once the change in brightness of pixel exceeds predetermined threshold, an event is triggered. The intrinsic properties of event cameras give them several advantages over RGB cameras, including low power consumption, high dynamic range, low latency, and high temporal resolution. These benefits highlight the potential of event cameras in challenging scenarios, such low-light and high-speed conditions, which has led to some research emphasizing the of event cameras for robotic sensing in challenging situations (Zuo et al., 2022; Chen et al., 2023). Spiking Neural Network (SNN) (Maass, 1997) is type of neural network that inspired by the way biological neurons communicate with each other. By integrating the biological neuronal dynamics into individual neurons, SNN becomes capable of representing intricate spatio-temporal information and dealing with asynchronous data naturally, typically event-based data.
In terms classification tasks, number of event-based datasets, such as N-MNIST, N- Caltech 101 (Orchard et al., 2015), and CIFAR 10-DVS (Li et al., 2017), have been used to evaluate the performance of artificial neural networks (ANNs) and SNNs. However, the issue of overfitting still poses significant challenge for event-based datasets. Data augmentation is efficient method for improving the generalization and performance of a model. Lots of methods have been proposed to augment event- based data, for example, transferring classic geometric augmentations from image
"Corresponding authors.

---

Published as a conference paper at ICLR 2024
field to event- field (Li al., 2022), randomly dropping events (Gu al., 2021), and mixing two event streams with a randomly sampled mask (Shen et al., 2023). Nevertheless, current mixing augmentation strategies in event-based field do not consider the size and location information of label-related objects, and thus may produce events with incorrect labels and disrupt the training pro- cess. To address this problem in image processing field, Uddin et al. (2020); Kim et al. (2020) mix the label-related objects together based on the saliency information obtained from neural networks. This paradigm achieves better results compared with conventional non-saliency augmentations. Kim & Panda (2021b) was the first to explore acquiring saliency information from SNNs. In this work, Spiking Activation Map (SAM) was presented reveal the model's attention by weighted adding of the intermediate feature maps. Since the CAMs obtained from this method are not related to the network's 's predictions, their precision regarding the shape and position of label-related objects is still not satisfactory.
In this paper, we present two novel methods extended from layer-wise relevance propagation (LRP) (Bach et al., 2015) to visualize the label-related saliency information of SNNs, each offering distinct advantages in terms of temporal precision and computational time. Moreover, guided by this saliency information, we develop data augmentation approaches targeted at event based data, demonstrating significant improvements in the performance and generalization capability of SNNs across multiple classification tasks.
Our contributions are summarized as follows:
We propose Spiking -iyer-Time-wise Relevance Propagation (SLTRP) and Spiking Layer- wise Relevance Propagation (SLRP) to accurately reveal the saliency information of SNNs. The former reveals information across time, while the latter is less time-consuming.
We present RPGDrop and RPGMix. By dropping and mixing events with the guidance of relevance propagation obtained from SLRP or STIRP, the augmented samples exhibit increased diversity and tight correlation with the labels. Combined with several geometric data augmentations, we formulate our data augmentation strategy namely EventRPG.
We evaluate our proposed saliency visualizing method and data augmentation method using various SNNs on event-based object and action recognition datasets. Experiments demon- strate that our SLRP and SLTRP can generate high quality CAMs and saliency maps with sub-optimal computing time. EventRPG achieves state-of-the performance on both object recognition and action recognition tasks with limited time consumption.
2 PRELIMINARY
layer-wise Relevance propagation (LRP) was first introduced in (Bach et al., 2015) as visualiza- tion tool for generating saliency maps that show the contribution of individual pixels in the input data to the model prediction or specific class, facilitating the interpretability of neural networks. According to the LRP rule, we assign a value to each neuron to represent the neuron's 's contribution to the prediction or target class, called Relevance Score. The idea of LRP is to find propagation rule satisfying the following definition.
Definition 1 (Conservation Property). On neural network with L layers, the relevance score of each layer satisfies
Σ R
vi E [0, L.}
where e is a constant value and RZ!) is the relevance score of gth neuron on layer I. R(O) is the relevance score before input layer.
We calculate the relevance score of 5th input neuron on layer I R (!!-1) by the sum of all relevance scores propagated from all the connections in this layer:
R 1)
Σ R
Utilizing the a:3-rule (Montavon et al., 2017) and abandoning the bias term b, we obtain the rel- evance score that should be propagated from the 5th neuron in layer I to the ith neuron in layer
I 1-1. 1:
R 1-1.1 n.o (a=
13
+
3
33
(3)
Σ,
Σ
2ij
2g
2

---

Published as a conference paper at ICLR 2024
Linear Layer
Litear Layer
Timmeredd.ced Layer
Sum
Achivslon ager
Spikingg Layer
Spiking Layer
Sum
Splking
Unserr eger
Litrer Laye-
Time roduced
Layer
near eyer
.*....
.*..**
Layer-wise Relevance Propagation
Spiking Layer- Tayge--Timm-wi Relevance Pmpagation
Spiking Layer-liee Relevance Propagation
Figure 1: Forward propagation flow and relevance propagation flow of ANNs (a) and SNNs (b, c).
where a + 3 = 1. 2 = wi 27 + way 2, and 2Új w 2; + Way x are respectively the positive and negative contribution of the ith neuron in layer / the jth neuron in layer This relevance propagation rule satisfies definition 1, and the proof can be seen in appendix D.I.
SPIKING RELEVANCE PROPAGATION RULE
In SNN, information represented by the timing and frequency of spikes. Neurons in SNN generate spikes when their membrane voltage reaches certain threshold. These spikes propagate through the network, influencing the activity of other neurons layer by layer. Two popular and fundamental models used to represent basic spiking neurons are the Leaky Integrate-and-Finn (LIF) neuron model and the Integrate-anc -nd-Fire
(IF) neuron model:
FLIF = = Vlt + (1- [tt)
(4)
fir(V, I) = V
T(t).
(5)
where 7 is the attenuation factor and At is the interval between time steps satisfying At << т. Membrane voltage V can be interpreted the information reserved by a neuron from the previous time step - 1. [[t] indicates the input of the neuron at the current time step.
In practice, researchers would replace the activation layer in an ANN with spiking layers represented by eq. (4) or eq. (5) to construct an SNN. As shown in fig. la, an ANN consists of linear layers and activation layers and passes information layer by layer. These two layer types do not include variables related to time and as such we propagate the relevance scores through them layer by layer.
3.1 RELEVANCE PROPAGATION FOR LINEAR LAYERS IN SNNS
Information in SNNs is propagated through time in spiking layers, while the linear layers do not transmit information across different time steps. Any linear layer's output y/t) time step only correlates to its input at{] at time step and we conduct relevance propagation using eq. (3) on this layer separately for each time step. Specifically, for linear layer we extend the eq. (3) with suffix "[t]" representing the time step t, and we have
R 1) [t] = Σ
R [0)
(1-10
(6)
R = R5 (a
2,j
+ 3
Σ 2
E, 21g [t]
which 1 = w a [t] + wij x; [t] and 2 [1 - w x, [t] + w,i x [t]. Next, we focus on the derivation of the relevance propagation rule spiking layers.
3.2 SPIKING LAYER- TEEIR--TIM-NEE WISE RELEVANCE PROPAGATION
From eq. (4) we know that in the forward propagation, the output of LIF neuron at time step t (t > 0) depends on this layer's s input current [[t] at the current time step and the membrane voltage Vit 1] from the previous time step. Therefore,
should propagate the relevance neuron's 's internal voltage at the previous time step and to the input current at the current time step, shown in fig. 1b. Assume there overall T' time steps. Similar to a,3-uule, propagate
to the
3

---

Published as a conference paper at ICLR 2024
the relevance score based on its positive contribution and negative contribution at each time step t E [I. Consider a neuron in spiking layer at time step t. We decomposite and represent it in
general manner:
S(V, 1) = c. ViL- - 1] + d. T[4]
(8)
where c and are coefficients depending on the neuron type:
e
LIF Neuron,
1 c
LIF Neuron,
c
IF Neuron,
IF Neuron.
Based eq. (8), at time step we define the proportion of relevance score that should be propagated to the previous time step as
c. V+ c- --1 1] c. ct d- - +
c. V. 1]
y = a
(9)
c- 1 -
where superscripts and denote the positive and negative values, respectively. For each neuron in spiking layer By initializing the relevance score at the final time step as R( [ and subsequently updating relevance scores iteratively time step / from T to using
[t 1] < -[1] (()
[1] ( - 1],
(10)
- [ (1- 1t)
(11)
we propagate the relevance score of every neuron in spiking layers to all time steps. With iteration formulas (10) and (II), we could derive the relevance score at each time step
T
R R1 = (1-
Σ
R
II
Tb
+ R
(12)
1 1
j=t+1
Proposition 1. The sum of relevance scores propagated after spiking layer from time step 1 to k(k < T)
Σ Rd En-
Σ
RO
Σ
(13)
j=k+1
The proof is provided in appendix D.2. With proposition 1, we have
T
T-1
T-1
[1
=
10 + n -
R + + [7] (1 (1->(7))R
R + Rd [T] Σ R(
(14)
This is a stronger Conservation Property since the relevance score stays unchanged for every neuron in spiking layer. This indicates that leveraging formulas (10) and (11), are able propagate relevance scores through any spiking layer while satisfying the Conservation Property. Combining with relevance propagation rules introduced in section 3.1, we could propagate relevance scores to any layer in an SNN, thereby revealing the saliency information across time, namely Spiking Layer-Time-we Relevance Propagation (SLTRP).
3.3 SPIKING LAYER- WISE RELEVANCE PROPAGATION
SLTRP could reveal the saliency information at any time step. and would thus cost more time to conduct the whole relevance propagation process compared with ANNs. Under some circumstances, e.g., on datasets transformed from static images where coordinates of events tend be fixed w.r.t. the time, we don't 't need to obtain the saliency score of some specific time step and only require the saliency information stacked from all time steps R,O Σ R( For any spiking layer from eq. (14) we have R) R().
In terms of linear layers, we sum the positive and negative contribution values of the ith neuron to tth neuron through time dimension as
1
Σ
E (ut
xf [t]
+ w. x5 [t]) = - (w Σ x [t] + w
Σ 27 [t)),
T
Σ
it
Σ
(utt x, [t]
+ w. (t)
1
(w
Σ xx [t] + w.j Σ xx [t])
T
I
4

---

Published as a conference paper at ICLR 2024
JeSM Naho
Sandee in logtea Spil
Ove
SelencyHaa
Vistar Attcch
(b)
Illustration of Relevance Propagation Guided Event Mix and Event Drop. (a) RPGDrop. Where the saliency map offers higher value, events are more likely be dropped. RPGMix.
Figure 2:
Then we propagate the relevance scores of linear layer using eq. (2) and eq. (3) with different def- initions of R(1-1) R 2 and z.z. This enables spiking relevance propagation process without consideration of the time dimension, saving time costs and being more practical for datasets trans- formed from static images, namely Spiking Layer-wise Relevance Propagation (SLRP) (see fig. Ic).
4 RELEVANCE PROPAGATION GUIDED EVENT DATA AUGMENTATION
4.1 SALIENCY MAP AND CLASS ACTIVATION MAP
In an SNN, we first leverage Contrastive Layer-wise Relevance Propagation (CLRP) (Gu et al., 2018) to initialize the relevance scores of the output layer for each time step. Then propagate the relevance scores backward using SLRP or SLTRP depending on the dataset. Relevance scores are propagated throughout all layers to create saliency maps, while class activation maps (CAMs) can be formed in two ways. One method involves summing the relevance scores from specific interme- diate layer across the channel dimension, resulting in SLRP- and SLTRP-CAM Alternatively, CAMs can be generated by calculating a weighted sum between relevance scores and feature maps, a method referred to as SLRP-ReICAM and SLTRP-ReICAM (Lee al., 2021).
4.2 RELEVANCE PROPAGATION GUIDED EVENT DROP
Gu et (2021) has proven randomly dropping events to be an effective augmentation strategy. Furthermore, we expect to drop events more frequently in regions with label-related objects, mo- tivated by the fact that disturbing regions with no label-related information (namely background) would have a negligible impact the classifier's prediction. The label-related information can be provided by CAM and saliency map, where higher values imply higher model attention and label relevance. Since saliency map accurately reveals the relevance score of each pixel to the target in the input data, we leverage saliency map to guide dropping, detailly illustrated in fig. 2a. The higher the value of pixel, the higher the probability that will drop events that pixel. 0 is the parameter controlling the magnitude of augmentation.
4.3 RELEVANCE PROPAGATION GUIDED EVENT MIX
Event-based data, in contrast to image-based data, does not include color details, with the most crucial aspect being the texture information it contains. The overlapping of label-related objects will impair the texture details of these objects, which in turn further degrades the quality of features extracted in SNNs. Building upon this motivation, we propose Relevance Propagation Guided Event Mix (RPGMix). The whole mixing strategy is illustrated in fig. 2b. For two event-based data candidates, we utilize relevance propagation localize the label-related regions and obtain two bounding boxes. To mix two objects with clear texture features, we randomly select two positions ensuring minimal overlap of their bounding boxes. This involves initially positioning one box at a corner to maximize the nonoverlapping area for the other box's placement, then selecting positions for both boxes in order, maintaining minimal overlap and maximizing sampling options. Finally, the two event streams moved the sampled positions. Although this linear translation part prevents the overlapping of label-related objects, the background of one object would still overlap with the other object. Moreover, in one single time step, the representation ability of the spiking neurons
5

---

Published as a conference paper at ICLR 2024
(which only output binary information) is much worse than that of the activation layer (usually ReLU) of ANNs, making them less capable of spatial resolution and more likely to fall into local optima. Therefore, to promise the presence of only events from a single event stream candidate per pixel, avoiding regions with mixed information from interfering with the SNN, we adopt a CutMix strategy to mask the two event streams based the bounding box of the second event stream, as demonstrated in the left part in fig. 2b. Kim et al. (2020) takes the sum of each sample's 's mask as the ratio of their corresponding labels. This ensures that the proportion of labels in the mixed label matches the proportion of pixels belonging to each sample. In our approach, we further aim to align the proportion of labels with the proportion of label-related pixels, which can be estimated using the bounding boxes. As result, the labels of the two event streams are mixed as
I (w, h1 Soverlop) + LgW2gh2
Lmiz
(15)
W1 h1 + w2h2 Soveriop
where w; and hú denote the width and height of the bounding box in the event stream i. L1 and L2 are the one-hot labels of the two event
Identity
streams and Soveriop is the area of the overlap- ping region of the two bounding boxes.
Horizontal l'lp
Cutbut
4.4 EVENTRPG
Rofato
Events
RPGM
Combined with a few geometric data augmen- tation methods in NDA (Li al., 2022), we for- mulate our data augmentation strategy namely EventRPG as shown in fig. Specifically, for a batch of imput event streams, each of the event streams would be augmented with ran- domly sampled policy and magnitude. They then have probability of 0.5 to be augmented by RPGMix.
(orthontal Transiation
Isentiyy
Veriical Translation
Shear
Augmented Events
Everts N
RPG
Figure 3: Augmentation process of EventRPG.
EXPERIMENTS
5.1 EFFECTIVENESS OF SLRP AND SLTRP
In this subsection, we evaluate the effectiveness of our approach for generating CAMs and saliency maps. Current event-based datasets for classification can be divided into two tasks: object recogni- tion task and action recognition task. Generally, the former mainly involves event streams generated from jittering of static images, while the latter mainly involves event streams recorded in real envi- ronments, containing more dynamic information. We perform experiments on both types of datasets to showcase the effectiveness of our approach. We visualize the feature before the last fully con- nected layer as CAM.
Object Recognition
Action Recognition
Method
N-Caltech101
CIFARIO-DVS
N-Cars
DVSGesture
SL-Animals
A.I.1
A.D.
A.L.
A.D.
A.I.
A.D.
A.I.
A.D.
A.I. 1
A.D.
SAM (Kim & Panda, 2021b)
0.86
22.33
1.23
46.83
5.24
5.89
8.27
10.83
20.60
8.23
Grad. CAM (Selvaraju et 2017)
12.03
41.71
21.77
17.74
0.41
67.09
14.44
Grad-CAM++ (Chuttopadhay et al., 2018)
24.82
10.32
6.23
26.33
26.04
5.41
7.99
10.41
11.78
22.21
SLRP-ReICAM (Lee et al., 2021)
17.51
10.44
7.60
23.41
22.05
19:00
13.11
6.41
26.49
10.67
ReICAM (Lee et al., 2021)
17.54
10.44
7.60
23.41
22.05
19.00
13.05
6.40
26.49
10.67
SI.P--CAM
34.24
5.75
8.41
23.50
15.72
23.80
7.79
29.99
10.88
40.95
ALTRP-CAM
34.24
5.75
8.41
23.50
15.72
23.80
7.79
30.02
10.88
40.95
ST.RP.Saliency Map
34.12
4.18
9.44
19.99
22. 98
6.77
22.20
13.86
22.30
20.57
SLTRP-Saliency Map
34.17
4.19
9.51
19.98
22. 84
6.75
21.17
13.95
22.30
20.79
Table 1: Comparison of A.I. and A.D. on event-based object recognition and action recognition datasets. We highlight the best results in bold and the second best results with underlining.
5.1.1 OBJECTIVE FAITHFULNESS
We adopt two widely used metrics, Average Drop (A.D.) and Average Increase (A.I.,, to measure the objective faithfulness of our method compared with other typical visualization tools. These two metries explain how well an attention map explains model's attention by measuring the average change in the model's output when the attention map is applied as a mask to the input. The higher the A.I. and the lower the A.D., the better the attention map explains the model's attention. We

---

Published as a conference paper at ICLR 2024
compare method with Grad-CAM (Selvaraju et al., 2017), Grad-CAM++ (Chattopadhay et al., 2018), and SAM (Kim & Panda, b021b) on both object recognition datasets and action recognition datasets.
On object recognition task, our method achieves the best performance in terms of A.I. and A.D. on large datasets including N-Caltech101 and CIFARIODVS demonstrating the effectiveness of our method in generating CAMs and saliency maps. On the N-Cars dataset, Grad-CAM++ outperforms our method. This disparity may be due N-Cars being binary classification task that solely focuses on detecting the presence of a car in the event stream, in contrast to other datasets requiring multi-class object localization.
In terms of action recognition task, methods with spiking relevance propagation outperform other methods significantly, with an A.I. metric almost thrice as good as the best method without spiking relevance propagation Grad-CAM++, in the DVSGesture dataset. In the SL-Animals dataset, ReICAM achieves the best results terms of A.I. and also has low A.D. value. Note that ReICAM is also obtained based on the relevance scores from SLRP and SLTRP. Therefore, its good results also helps to demonstrate the effectiveness of our spiking relevance propagation rules.
GREEGAN
SAM
Onporty
SIRP-CAM
(
(b)
R
Figure 4: CAM and saliency map by different methods (a) N-Caltech101 and (b) CIFARIO-DVS. (c) Saliency maps generated from SLTRP on DVS-Gesture and SL-Animals.
5.1.2 EVALUATION OF SELECTIVITY
We visualized the saliency map and CAM results of ours and other methods from Spiking- VGG11 and SEW Resnet18. As shown in fig. 4a, method is more selective than other methods, with higher value on the label-related objects and a lower value on the background. In contrast, Grad- CAM and SAM more likely to be affected by the background, and Grad-CAM++ failed to locate the "dragonfly" on the top row, even though there are no other label-unralated events in this sample. SLRP-CAM and SLRP-ReICAM successfully localize the dog's head, whereas On the attention of the other methods drifted to other regions.
The saliency maps generated from SLTRP are able to track the exact moving object in the dataset (see fig. 4c). In the demonstration of SL-Animals dataset, the saliency map first focuses on the hand part to recognize the gesture. In the later time steps, it transfers its attention to the arm to capture the moving information. This proves its capability of temporal saliency information capturing, also yielding high selectivity.
Midel
Resotatin
GAM
Godd-AMr+ SURP-BOCAM
SLTRP
SURP CAM
SLRP
SITRP-CAM
SLIRP Sillecyy
Spking VGG-11
(138. 1760
24k 483
00663
00776
0,095
BJ1I5 0JI3
8.8267
8J1346 BA113
B0845
61076
B4877)
64545
1117
Table 2: Average time cost (s) of generating CAM and saliency map on N-Caltech 101 (Spiking- VGG11) and DVSGesture (SEW Resnet18), 8). Results within the quickest tier are highlighted in bold.
7

---

Published as a conference paper at ICLR 2024
5.1.3 COMPUTATION TIME
As shown in table 2, SLRP-CAM, SLTRP-CAM are on the same level SLTRP-ReICAM, SLRP- ReICAM, and SAM, all among the fastest methods. SLRP-Saliency Map and SLTRP-Saiee Saliency Map are slower than other methods, while still being competitive on SEW Resnet18 compared with Grad-CAM and Grad-CAM++. The time cost of SLTRP-CAM does not increase lot compared to SLRP-CAM, since it only requires the relevance computation of the last fully connected layer. In Map is much higher than SLRP- SLRP-Saliency Map, since it
contrast, the time cost of SLTRP-Saliee requires the relevance computation of all layers.
5.2
RESULTS OF EVENTRPG
Dutaset
Data Augmentation
Training Method
Neural Network
Neuron
Timesteps
Resolution
Accuracy
Flip NDA (Li al, 2022)
SALT (Kim Panda, 2021a)
VGG1e
LIF
20
(80,80)
55,00
N Caltech
N (Zheng et
GGG1
NDA (Li al., 2022)
STBP-ddBN (Zheng et al., 2021)
Spike VGG1I
LIF
10
(128. 128)
83.70
Eventmix (Shen al.,, 2023)
STBP
Pre-Act Resnet18
PLJF
10
(48, 48)
79.47
Identity
75.70
EventDrop
EvenRPG (CAM)
2021)
TET (Deng al., 2022)
Spike-VGGII
LIF
10
(128,
85.00
EvenRPPG (Saliency Map)
85.62
Drop by
Wide-7B-Net
Flip
SALT (Kim Panda, 202
66 VGG16
LIF
20
(64.64)
67.10
Random Crop
DSR (Meng et al., 2022)
1GG11
IF
20
(48.48)
75.03 +: 0.39
CIFARID DVS
Random Crop
DSR (Meng al. 2022)
Spike 1
LIP
20
(48.48)
77.27+00.2
PlipTrancattinn
(Deng al., 2022)
Spike. VGG11
NDA (1J all, 20222)
STRP-adR (Zheng et al., 2021)
Spike-VGG11
EJIP
10
48)
79..6
NDA (1. al,, 2022)
STBP-sdB (Zheng et al., 2021)
Spike. VGG1
LIP
10
(128. 128)
81,70
(Sben
2023)
STBP
Resnett8
Idkntity
78.85
EventDrop (Gu et al. 2021)
EvaiRPG (CAM)
TET (Deng 2022)
Spike GGGe1
LIF
10
(48,
85,55
77.73
EvenRPG (Saliency Map)
NDA (Li 2022)
STBP STBP-sdBN (Zheng et al., 2021)
VGG11
LIF
10
(48.48)
90.10
NDA (1 al., 2022) 2022)
STRP-dRY (Zheng al, 2021)
Spike-VGG11 VGG1
10
(128. 128)
91.90
N.Ca
Eventmix (Shen 2023)
STBP
Pre-Act Resnet18
PLIF
10
(48.48)
96.29
Identity
EventDrop (Giu et al., 2021)
EvenRRPG (CAM)
TET (Deng 2022)
LIP
10
(48.48)
95.446
95.76
EvenRPG (Saliency Map)
Table 3: Accuracy of various data augmentation methods on event-based object recognition datasets. the datasets used are created using event cameras. Specifically, N-Caltech 101 and CI- FARIODVS are derived from static images, while N-Cars is recorded in real-world environments.
In this section, we evaluate our proposed EventRPG with other augmentation methods including EventDrop (Gu et al., 2021), NDA (Li al., 2022), and EventMix (Shen al., 2023) across several object recognition and action recognition datasets. They could illustrate the performance of our methods in terms of static objects and moving objects, respectively. The datasets used and training settings are introduced in appendix C in details. Since SLRTP and SLRP yield nearly identical results for object recogmition tasks, we only leverage SLRP in object recognition experiments since it costs fewer time. Eventdrop (Gu et al., 2021) did not conduct experiments on SNNs, so we reproduce it using its public code and conduct experiments under the same training setting of ours. We report the best accuracy for each experiment using same random seed.
5.2.1 OBJECT RECOGNITION TASKS
that EventRPG achieves state-of-the-art performance on N-Caltech101 and CIFARIO-DVS datasets, bringing 9.92% and 6.7% improvements compared with identity (no aug- mentation), respectively. On N-Cars, EventRPG achieves the second-best performance, only 0.29% lower than EventMix. This might be attributed to the fact that N-Cars is a binary classi- fication dataset, which only contains label "car" and "background". Most samples belonging to "background" do not have specific label-related object to locate, making it difficult for our method to generate saliency map and CAM with high quality, thus decreasing the performance.
From table
5.2.2 ACTION RECOGNITION TASKS
We implement Event on SEW Resnet18 for action recognition tasks. On DVSGesture dataset, method achieves best results compared with other reproduced augmentation methods under the same training settings, though slightly lower than EventMix on Pre-Act Resnet18. On SL-Animals dataset, method achieves state-of-the-art performance among all data augmentation apporaches
8

---

Published as a conference paper at ICLR 2024
SL-Animals
Model
Spike
Method
DVSGesture
sets
7-Layer Spiking CNN
Hybrid X
SCTFA et al., 2023)
98.96
90.04
GeegLeNet
TORE (Baldwin et al., 2022)
96.20
Event Transformer
EvI (Sabater al., 2022)
96.20
88.12
87.45
Pre-Act Resnet18
EventMix (Shen et al., 2023)
96.75
Identity
94.33
85.42
89.09
EventDrop (Gu al., 2021)
92.33
86.33
88.99
NDA (Li et al., 2022)
93.67
87.77
89.55
SEW Resnet18 18
EventPG (CAM, SLRP)
95.83
90.97
91.96
EventPPG (Saliency Map. SLRP)
95.49
91.59
93.30
EventRPG (CAM, SLTRP)
96.18
90.54
90.63
EventRPG (Suliency Map. SLTRP)
96.53
93.75
Table 4: Accuracy of various data augmentation methods event event-based action recognition datasets.
and all neural networks, with 3.82% and 4.2% improvements compared with the second-best aug- mentation method on 4 sets and sets, respectively. Action recognition tasks include more dynamic information compared to object recognition tasks. Thus, the success of our method action recog- nition tasks demonstrates its great potential for other dynamic event bent-bbsed datasets, which is a future direction worth exploring.
5.2.3
TIME CONSUMPTION ANALYSIS
SURP CIFRROOVS
SLTAP SLAAnimll
Packbone Training
0.040-
Backbone EvertPPG
time Cost (revent stream)
EvantNPG
0.025
EventDrop
0.035
EventDrip
0.000
NDA
0.025
0.015-
0.020-
0.05
a.10
0.005
0.005-
0.000
0.000
(a)
(b)
Figure 5: Average time cost of augmentation methods with SEW Resnet datasets.
In order to assess the computational efficiency of our data augmentation method, we perform two experiments measure the time required for executing EventRPG with SLRP and SLTRP in ob- ject recognition and action recognition tasks. As depicted in fig. 5, the computation time of other data augmentation approaches remains constant regardless exhibits a nearly linear decrease in computation time as the batch size increases, since the relevance propagation process, which is the main contributor to the time consumption, can be speeded up by parallizing computing across samples in batch, similar to the gradient backpropagation process. When the batch size for each GPU exceeds both SLRP and SLTRP achieve comparable speeds to NDA and EventDrop, confirming the time efficiency of EventRPG.
the batch size, whereas our method
CONCLUSION AND LIMITATION
Conclusion In this paper, for the first time, we propose SLTRP and SLRP, two efficient and prac- tical methods for generating CAMs and saliency maps for SNNs. Building upon this, we propose EventRPG, i.e., dropping events and mixing events with Relevance Propagation Guidance. Since EventRPG only disturbs and mixes regions on which model concerns most, it is more efficient com- pared to vanilla dropping and mixing, and also alleviates the likely misalignment problem between data and label. In experiments, SLRP and SLTRP not only both yeild best results compared with other feature visualization tools, but also consume very little time to compute. EventRPG achieves state-of-the-art performance on 1-Caltech101. CIFARIO-DVS and SL-Animals datasets, proving its strong generalization ability across different models and datasets.
Limitation Currently, Event can only be implemented on classification tasks, remaining as limitation. However, we could still leverage multi-task training paradim which has been proven to be effective to implement into other downstream tasks, and we will also explore more possibilities of using Event RPG in self-supervised leaming tasks.

---

Published as a conference paper at ICLR 2024
7 ACKNOWLEDGEMENT
We would like to thank all anonymous reviewers for their committed work and insightful feedback. This work supported in part by the National Natural Science Foundation of China under Grants 62073066 and U20A20197, in part by the Fundamental Research Funds for the Central Universities under Grant N2226001, in part by 111 Project under Grant B 6009, in part by the Intel Neuro- morphic Research Community (INRC) Grant Award (RV2. 137. Fang), and in part by Guangzhou- HKUST(GZZ) Joint Funding Program under Grant 2023A03J068..
REFERENCES
Arnon Amir, Brian Taba, David Berg, Timothy Melano, Jeffrey McKinstry, Carmelo Di Nolfo, Tapan Nayak, Alexander Andreopoulos, Guillaume Garreau, Marcela Mendoza, Jeff Kusmitz, Michael Debole, Steve Esser, Tobi Delbruck, Myron Flickner, and Dharmendra Modha. low power, fully event-hased gesture recognition system. In CVPR, July 2017.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. On Pixel-wise Explanations for Non-linear Classifier Decisions by Layer- wise Relevance Propagation. PloS One, 10(7):c0130140, 2015.
R Wes Baldwin, Ruixu Liu, Mohammed Almatrafi, Vijayan Asari, and Keigo Hirakawa. Time- ordered recent event (tore) volumes for event cameras. IEEE TPAMI, 45(2):2519 12-2222,,,,, 2022.
Patrick Bardow, Andrew Davison, and Stefan Leutenegger. Simultaneous Optical Flow and Inten- sity Estimation from An Event Camera. In CVPR, pp. 884-892, 2016.
Alexis Baudron, Zihao W Wang, Oliver Cossairt, and Aggelos K Katsaggelos. E3D: Event-Based 3D Shape Reconstruction. arXiv preprint arXiv:2012.05214 2020.
Wuque Cai, Hongze Sun, Rui Liu, Yan Cui, Jun Wang, Yang Xia, Dezhong Yao, and Daqing Guo. A temponllteeeeaerralllaa----aa-ae
attention for spiking neural networks. TNNLS, 2023.
Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad- cam++: Generalized Gradient- Gradient-based Visual Explanations for Deep Convolutional Networks. In WACV, pp. 839-847. IEEE, 2018
Peiyu Chen, Weipeng Guan, and Peng Lu. Esvio: Event-based stereo visual inertial odometry. IEEE Robotics and Automation Letters, 2023.
Shikuang Deng. Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient re- re-weighting. ICLR, 2022.
Wei Fang, Yanqi Chen, Jianhao Ding, Ding Chen, Zhaofei Yu, Huihui Zhou, Yonghong Tian, and other contributors. Spikingjelly. https:/ /gi thub. comb fhom/fang fangwei 123456/ spiking jel y. 2020. Accessed: 2022-099-9.
Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. NewrIPS, 34:21056-21069, 2021 l.
Wei Fang, Zhaofei Yu, Yanqi Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian. In- corporating learnable membrane time constant to enhance learning of spiking neural networks. In ICCV, pp. 2661 1-711. 1, b021
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories. In CVPRW, pp- 178-178. IEEE, 2004.
Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, Jörg Conradt, Kostas Daniilidis, al. Event-based Vi- sion: A Survey. IEEE TPAMI, 44(1):15-1180, 2020.
Daniel Gehrig, Antonio Loquercio, Konstantinos G Derpanis, and Davide Scaramuzza. End-to-end Learning of Representations for Asynchronous Event-based Data. In ICCV, pp. 5633-5643, 2019.
10

---

Published as a conference paper at ICLR 2024
Mathias Gehrig, Mario Millhäusler, Daniel Gehrig, and Davide Scaramuzza, E-raft: Dense Optical Flow from Event Cameras. In 3DV, pp. 197-206. IEEE, 2021.
Fuqiang Gu, Weicong Sng, Xuke Hu, and Fangwen Yu. EventDrop: Data Augmentation for Event-
based Learning. IJCAI, 2021. URL https: https:://axii.....t////tt/ http//axivvvvo// http://wwxwv....aaara/.s//// 2210 05836.
Jindong Gu, Yinchong Yang, and Volker Tresp. Understanding individual decisions of cnns via contrastive backpropagation. In ACCV, pp. 119-134. Springer, 2018.
Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: Simple Data Processing Method to Improve Robustness and Uncertainty. In ICLR, 2019.
Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixup. In JCML, 2020.
Junho Kim, Jachyeok Bac, Gangin Park, Dongsu Zhang, and Young Min Kim. N-imagenet: Towards robust, fine-grained object recognition with event cameras. In ICCV, pp. 21 2146-2156, 2021.
Youngeun Kim and Priyadarshini Panda. Optimizing deeper spiking neural networks for dynamic vision sensing. Neural Networks, 144:686-699 2021a.
Youngeun Kim and Priyadarshini Panda. Visual explanations from spiking neural networks using inter-spike intervals. Scientific reports, 11(1):1-14, 2021b.
Youngeun Kim, Chough Joshua, and Priyadarshini Panda. Beyond classification: spiking neural networks for semantic segmentation. Neuromorphic Computing and Engineering, 2022.
Directly training
Jürgen Kogler, Christoph Sulzbachner, and Wilfried Kubinger. Bio-inspired Stereo Vision System with Silicon Retina Imagers. In International Conference on Computer Vision Systems, pp. 174- 183. Springer, 2009.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Xavier Lagorce, Garrick Orchard, Francesco Galluppi, Bertram E Shi, and Ryad Benosman. Hots: hierarchy of event-based time-surfaces for pattern recognition. IEEE TPAMI, 2016.
Chankyu Lee, Adarsh Kumar Kosta, Alex Zihao Zhu Kenneth Chaney, Kostas Daniilidis, and Kaushik Roy. flownetloweet Event-based Optical Flow Estimation with Energy-efficient Hy- brid Neural Networks. In ECCV, pp. 366-382. Springer, 2020.
Jeong Ryong Lee, Sewon Kim, Inyong Park, Taejoon Eo, and Dosik Hwang. Relevance-cam: Your Model Already Knows Where to Look. In CVPR, pp. 14944-14955 2021.
Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. Cifar 10-r100dvs: an Event-stream Dataset for Object Classification. Frontiers in Neuroscience, 11:309, 2017.
Yuhang Li, Youngeun Kim, Hyoungseob Park, Tamar Geller, and Priyadarshini Panda. Neuro- morphic Data Augmentation for Training Spiking Neural Networks. In ECCV, 2022. URL.
https: /aaxiivvv.. orw.obgg//.... abs
Wolfgang Maass. Networks of Spiking Neurons: the Third Generation of Neural Network Models. Newral Networks, 1059):165-11671. 1997.
Misha Mahowald. The Silicon Retina. In An Analog VLSI System for Stereoscopic Vision, pp. 4-65. Springer, 1994.
Qingyan Meng, Mingging Xiao, Shen Yan, Yisen Wang, Zhouchen Lin, and Zhi Quan Luo. Training Righ-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Represen- tation. In CVPR, pp. 12444-1255, 2022.
11

---

Published as a conference paper at ICLR 2024
Grégoire Montavon, Sebastian Lapuschkin, Alexander Binder, Wojciech Samek, and Klaus-Rober Müller. Explaining Nonlinear Classification Decisions with Deep Taylor Decomposition. Pattern Recognition, 65:211-220 1-222, 2017.
Garrick Orchard, Ajinkya Jayawant, Gregory K Cohen, and Nitish Thakor. Converting Static Image Datasets to Spiking Neuromorphic Datasets using Saccades. Frontiers in Neuroscience, 9:437, 2015.
Viktor Rudnev, Mohamed Elgharib, Christian Theobalt, and Vladislav Golyanik. EventNeRF: Neu- ral Radiance Fields from a Single Colour Event Camera. arXiv preprint arXiv:2206. 1186, 2022.
Alberto Sabater, Luis Montesano, and Ana Murillo. Event transformer. a sparse-aware solution for efficient event data processing. In CVPR, pp. 2677-2686, 2022.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual Explanations from Deep Networks via Gradient-basee Lo- calization. In ICCV, pp. 618-626, 2017.
Guobin Shen, Dongcheng Zhao, and Yi Zeng. Eventmix: An efficient data augmentation strategy for event-based learning. Information Sciences, 144:119170 2023.
Amos Sironi, Manuele Brambilla, Nicolas Bourdis, Xavier Lagorce, and Ryad Benosman. Hats: Histograms of averaged time surfaces for robust event-based object classification. In CVPR, pp- 1731-1740, 2018.
Bongki Son, Yunjae Suh, Sungho Kim, Heejae Jung, Jun-Seok Kim, Changwoo Shin, Keunju Park, Kyoobin Lee, Jinman Park, Jooyeon Woo, al. 4.1 A 640x 480 Dynamic Vision Sensor with 9um Pixel and 300Meps Address-event Representation. In ISSCC, pp. 66-67. IERE, 2017.
Timo Stoffregen, Guillermo Gallego, Tom Drummond, Lindsay Kleeman, and Davide Scaramuzza. Event-based Motion Segmentation by Motion Compensation. In ICCV, pp. 7244-7253, 2019.
AFM Shahab Uddin, Mst Sirazam Monira, Wheemyung Shin, TaeChoong Chung, and Sung-Ho Bae. SaliencyMix: A Saliency Guided Data Augmenttiion Strategy for Better Regularization, In ICLR, 2020.
Ajay Vasudevan, Pablo Negri, Camila Di lelsi, Bernabe Gotarredona. Sl-animals-dvs: event-driven sign language animals dataset. Pattern Analysis and Applications, pp. 1-16, 2021.
and Teresa Serrano-
Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu. Score-CAM: Score-weighted Visual Explanations for Convolutional Neural Networks. In CVPRW. pp. 24-25, 2020.
Ziwei Wang, Liyuan Pan, Yonhon Ng, Zheyu Zhuang, and Robert Mahony. Stereo hybrid event- frame (shef) cameras for 3d perception. In IROS, pp. 9758-9764. IEEE, 2021.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization Strategy Train Strong Classifiers with Localizable Features. In ICCV, pp. 6023-6032, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. Mixup: Beyond Empiri- cal Risk Minimization. In ICLR, 2018.
Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly-trained larger spiking neural networks. In AAAI, volume 35, pp. 1 11022-11070, 2021.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning Deep Features for Discriminative Localization. In CVPR, pp. 2921-2929, 2016.
Yi Zhou, Guillermo Gallego, Xiuyuan Lu, Siqi Liu, and Shaojie Shen. Event-based Motion Seg- mentation with Spatio-temporal Graph Cuts. TNNLS, 2021.
Yi-Fam Zuo Jiaqi Yang, Jiaben Chen, Xia Wang, Yifu Wang, and Laurent Kneip. DEVO: Depth- Event Camera Visual Odometry in Challenging Conditions. ICRA, 2022.
12

---

Published as a conference paper at ICLR 2024
APPENDIX
A ADDITIONAL EXPERIMENTAL RESULTS
Here we present more experiments to further prove the effectiveness our approach.
A.1 CLASSIFICATION ON MINI N-IMAGENET DATASET
N-ImageNet, proposed by Kim et al. (2021), represents the neuromorphic adaptation of the well- known ImageNet dataset. It encompasses a thousand object categories, making it the most challeng- ing task in event-based classification to date. We conduct experients to evaluate the performances of different data augmentation methods on mini N-ImageNet with SEW Resnet-18.
Data Augmentation
Identity
EventDrop
NDA
EventPPG
Top-I Accuracy
28.16
34.18
35.84
40.90
Top-5 Accuracy
52.14
60.94
63.64
67.74
Table 5: Top-1 and Top-5 Accuracies (%) mini N-ImageNet.
Table demonstrates that data augmentations significantly enhance the performance the model, shown by the notable positive effects. In this comparison, approach outperforms the nearest competitor NDA, achieving a higher Top-1 accuracy by 5.06% and Top-5 accuracy by 4.1%.
0.40
EventDrop
NDA
0.35
4.00
Identity
EventRPG
3.75
0.25
0.0
Event
3.00
0.10
Idantiy
NDA
2.75
0.05
PPG
Epoch
(a)
(b)
Figure 6: The loss and Accuracy curves of different augmentations on mini N-ImageNet.
As shown in fig. 6a, the performance disparity between EventRPG and NDA is as significant the one observed between EventDrop and Identity. Additionally, late epochs of EventDrop and Identity exhibit overfitting, problem that is effectively mitigated in NDA and EVentRPC, as illustrated in fig. 6b. These outcomes highlight again the efficiency of our approach in boosting the model's performance and mitigating overfitting.
A.2 COMPARISON WITH OTHER SALIENCY-BASEE MIX METHODS
We also compare the performance of various saliency-based mix methods on two typical datasets: N-Caltech10) 101 and SI.-Animals. the experients, there 50% chance for each event stream to be augmented by corresponding mix approach with other augmentations applied. Notably, it's observed in table that both Saliency Mix (Uddin et al., 2020) and Puzzle Mix (Kim et al., 2020) occasionally result in lower performance than the identity approach. This underscores RPGMix's robust generalization capability, as it consistently excels across all datasets.
To clearly showcase the differences among these mix methods, present visualizations of the augmented samples they produce (see fig. 7). In most cases, our RPGMix aims to preserve as many label-related pixels as possible.
13

---

Published as a conference paper at ICLR 2024
Donn
Saleny
Subrroy
AROMb
Subercy
Sulercy
REGM
Figure 7: Mixed samples generated by different methods N-Caltech
14

---

Published as a conference paper at ICLR 2024
Dataset
Model
Identity
Saliency Mix
Puzzle Mix
Puzzle Mix (mask only)
RPGMix
N-Caltech101
Spike-GGG11
75.70
72.63
79.38
78.
81.75
Alimalss 4Sets
SEW Resnet18
85.42
84.00
85.34
85.68
88.67
SL- Sl-Animals-3Sen
SEW Resnet18
89.09
89.29
88.39
90.18
90.45
Table 6: Accuracy of Puzzle Mix, Saliency Mix, and RPGMix on object and action recognition tasks.
A.3 ADDITIONAL QUAILITATIVE RESULTS
We present additional qualitative results of our method applied to the N-Caltech101, SL-Animals, and DVS-Gesture datasets. The saliency maps created using SLTRE, as shown in fig. 8, exhibit strong selectivity. They assign high values to label-related objects and low values to label-unrelated objects. In action recognition datasets, saliency maps primarily focus on label-related moving ob- jects to capture the dynamic aspects of actions, as illustrated in fig. 9. In SL-Animals datasets (second row), the saliency maps focus on the person's 's hand raised to his/her head, assigning mini- mal attention to the lower area of the person, despite its high event density. Similarly, in the fourth row, the saliency maps show little interest the person's 's right hand (from our perspective), focusing instead on the left hand, even though both hands have similar event densities. These results illustrate that it's the label-related actions, rather than event density, that truly draw the focus of the saliency maps. This observation further confirms the selectivity of SLTRP-generater saliency maps in both action recognition and object recognition datasets.
B RELATED WORK
B. EVENT-BASED DATA
Event-based data is generated by asynchronous sensors, usually referred to as event cameras (Ma- howald, 1994; Son et al., 2017). Similar to point cloud, event-based data consists of four- dimensional points, denoted as (x, y, t, p), where and are the spatial coordinates, t is the times- tamp, and p is the polarity. Due to the inherent advantages of event cameras, event-based data widely used in the field of perception, e.g., optical flow estimation (Gehrig et al., 2021), depth es- timation (Wang et al., 2021), 3d reconstruction (Baudron et al., 2020; Rudnev et al., 2022), motion segmentation (Stoffregen et al., 2019; Zhou al., 2021), semantic segmentation (Kim et al., 2022), etc.
There are two mainstream approaches to processing event-based data, frame-based approaches and seent-b--event-barr approaches. Frame-based approaches (Kogler et al., 2009; Bardow et al., 2016; Lagorce al., 2016; Gehrig et al., 2019) similar to image processing, where events first converted into frames with a fixed shape (C, H, W), and then the frames are fed into an ANN for downstream classification or regression tasks. Exeentby-eveent-basee approaches handle events on a one-by-one basis, which is natural for SNNs (Lee et al., 2020; Fang et al., 2021a; Deng et al., 2022).
B.2 DATA AUGMENTATION
Vanilla data augmentations include simple transformations such scale, rotation, flip, etc. Mixup (Zhang et al., 2018) proposes mixing two randomly selected samples and their corresponding labels as new data and label for training. based on which several works that modify the mixing de- tails were proposed to further improve the robustness and performance (Yun et al., 2019; Hendrycks et al., 2019). Kim et al. (2020); Uddin et al. (2020) are similar to our proposed RPGMix that lever- ages the model's 's saliency information augment the data, while motivation and implementation details are quite different from them.
B.3 RELEVANCE PROPAGATION
Relevance propagation was initially presented (Montavon et al., 2017) as a method for creating saliency maps to visualize the impact of each pixel in the input data the prediction of the model
15

---

Published as a conference paper at ICLR 2024
Time
Faces easy
Snoopy
Rooster
Stegosaurus
Motorbikes
Laptop
Ketch
Rhino
Figure 8: SLTRP- Salienc Maps from Spike VGGI1 on N-Caltech101.
16

---

Published as a conference paper at ICLR 2024
SL-Animals
DVS Gesture
Figure 9: SLTRP-Saliency Maps from SEW Resnet-18 18 SL-Animals and DeS-Gesture
17

---

Published as a conference paper at ICLR 2024
particular class, thereby enhancing the interpretability of neural networks. The specificity of saliency maps towards the target class can be improved by employing Contrastive Layer-wise Tele- vance Propagation (CLRP) described by Gu et al. (2018), which diminishes the relevance scores for non-target classes.
B.4 CLASS ACTIVATION MAP
As another visualization tool describing the area of most interest to the model, Class Activation Map (CAM) is widely used to interpret the model's attention and find objects belonging certain class in the input data. The original CAM (Zhou et al., 2016) could only visualize the activation map of the last global average pooling (GAP) layer of the model, requiring the model to have a special structure, Grad-CAM (Selvaraju et al., 2017) overcomes this limitation by replacing the weights of GAP with the sum of gradients, enabling the acquisition of CAM from any CNN model. On this basis, Grad-CAM++ (Chattopadhay et al., 2018), Score- CAM (Wang et al., 2020), and Relevance-CAN (Lee al., 2021) were proposed to improve the quality of CAMs.
C DATASETS AND TRAINING DETAILS
C.I OBJECT RECOGNITION TASK
N-Caltech101 The neuromorphic version (Orchard et al., 2015) of Caltech101 (Fei-Fei et al., 2004). It artificially created by moving an asynchronous time-based image sensor (ATIS) mounted on a pan tilt unit in front of an LCD screen that presents the image data Caltech101. We use the same dataset as Li et al. (2022) that is split into the training set and test set by 9 1. The resolution of N- is 180 x 240 which, our implementation, is padded to 240 240 and rescaled to 128 128.
CIFARIO-DVS The DVS version (Li et al., 2017) of CIFARIO (Krizhevsky et al., 2009). The generation of CIFARIO-DVS is similar to N-Caltech101. Following NDA (Li al., 2022), we divide it by 9 1 as the training set and test set, and scale the resolution from 128 x 128 to 48 48.
N-Cars A binary classification dataset (Sironi et al., 2018). Unlike N-Caltech101 and CIFAR10- DVS transformed from image datasets, N-Cars is obtained from the recording of ATIS in real driving scenarios. Similar to the preprocessing of N-Caltech101, we first pad the resolution from 100 120 to 120 x 120 and then scale 48 x 48 on SNN.
Mini N-ImageNet A subset of N-ImageNet dataset (Kim al., 2021). Despite being smaller segment, it includes over 100,000 samples spanning 100 classes, making it the largest dataset used in our experiments. In line with the original authors' setup, we pad the resolution from 640 480 to 640 x 640 and then scale it down to 224 x 224 as the input resolution of the SNN.
C.2 ACTION RECOGNITION TASK
For object recognition tasks, shapes and textures are the most important information to recognize an object, and thus we do care how object moves in the event streams. While in action recognition tasks, the movements of the objects should be considered. They represent two different recognition strategies and thus are both essential to validate our proposed method.
DVS128 Gesture A hand gesture dataset (Amir al., 2017) recorded from DVS128 camera. It comprises 1464 samples with 11 classes, split into the training set and test set by 8:2. The resolution set as 128 x 128 on SNN.
SL-Animals-DVS A sign language dataset (Vasudevan al., 2021) including 19 Spanish Sign Language signs corresponding to animals. Following the raw paper, we split it by 7.5:2.5 into the training set and test set. Also, we separate the datasets into "4 sets" and "3 sets", which excludes the samples disturbed by the indoor lighting conditions reflecting on the patterned clothing of the user. The resolution processing of SL--Animal--DVS is identical to that of DVS 128 Gesture.
18

---

Published as a conference paper at ICLR 2024
Neural Networks
Neuron Model
Datascts
Epoch
Batch Size
T
Learning Rate
N-Caltech101
16
Spike-VGGI1
LIF
CIFARJO-DVS
100
10
10-3
N-Cars
64
SL-Animals
200
20
16
x 10
SEW Resnet-18
PLIF
DVS-Gesture
Mini N-Imagenet
100
64
10
1 x 10
Table Hyper-paramette for different models and datasets.
C.3 TRAINING DETAILS
In all experiments, we use Adam optimizer with the default setting (31.32) = (0.9, 0.999) to update the parameters and Cosine Annealing Scheduler for the decrease of learning rate. Other hyper-parameters are shown in table 7. For fair comparisons, an identical seed is leveraged for all experiments. We utilize the public code of TET (Deng al., 2022) to build Spike- VGG11 Spike-VGG11 and Spikingjelly (Fang et al., 2020) to build SEW Resnet-18 with PLIF (Fang et al., 2021b).
D PROOFS
In this section, we provide proofs of the propositions.
D.1 PROOF FOR CONSERVATION PROPERTY ON QU-RULE
Proof. The sum relevance scores layer is
Σ R (1-1)
ΣΣ
(1-1,1) ii-j
ΣΣ
R
+
3
2/
Σ
2jj
Ei
1 1
2/j
Σ R Σ
Σ
Σ R (a
E
Σ 20g
+
Σ
Σ, 2/g
Σ R (a+
Σ T0
D.2 PROOF FOR PROPOSITION
Proof. We prove proposition by induction.
For case according eq. (12) we have
Rd 11 [1] (1 >11)
R
>b1 + +
2
where 7/1] 0, since at the first time step, the membrane voltage is initialized to be 0. Clearly, proposition holds for case 1.
19

---

Published as a conference paper at ICLR 2024
Next, suppose proposition holds for case k 1, we have
R
b
j=k
[0 + >|k]] [ +
Σ
R [
Il
j=k+1
[[k]]
10 + 1 [k]
Re (x)]
Then
k-1
>|k]
Σ R 11
Σ [1] [k
RO
[1] + (k+
-- [
[
10 + 1 [k]]
R0 + R0 [k]++]+ +
Σ
R
1=
j=
Σ [+ + Σ R 1
II Ab1-
k=
j=k+1
20


Reviews:
[{"review_id": "LTjojp5gdU", "reviewer": "Reviewer_dKSN", "summary": "The paper addresses the problem of event-based data augmentation for Spiking Neural Networks by computing saliency. Two methods are presented with different levels of overhead on training; these methods are incorporated in two augmoentation schemes that either drop events or mix two event streams. A number of SNNs was evaluated on both object and action recognition datasets, and improvement in accuracy was demonstrated - especially on action recognition datasets.", "questions": "1) In abstract, expand SNN as 'Spiking Neural Networks', for the benefit of readers unfamiliar with the abbreviation.\n\n2) Fig. 1 highlights a failure case of one of the augmentation methods; it would be great to see a citation that explains why this limitation is difficult to alleveiate (e.g. by considering correct label classes). The image also seems excessive, and a single-sentence explanation should be enough.\n\n3) I am curious if the better performance / saliency on action-related datasets is an artifact of the event camera itself - since the density of the events correlates with motion and SNN may get more cues on regions with faster motion.\n\n4) It would be beneficial, in tables 3 and 4 to mention that all of the datasets are 'native' event-based datasets (unless I am mistaken). What would be the comparison if a classic camera dataset was converted to events, especially on action classification tasks?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1) The authors mention that the code will be released; to me this is important as it enables other researchers to easily build on top of this paper.\n\n2) I particularly enjoyed the theoretical introduction into the SNNs, this makes the rest of the paper much easier to read.", "weaknesses": "1) If anything, I would like to notice here that the improvements compared to competing methods are generally small. It would be interesting to see an apples-to-apples comparison in therms of compute overhead (which is mentioned, but I do not see numberical benchmark results); or, another compelling reason to use the presented methods vs e.g. second best.\n\n2) Also see 'questions': it would be good to show that saliency on motion-related datasets is more than just event density. This could be done e.g. by correlating saliency to raw optical flow magnitude, or evaluating on actions that are slower.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vBuBemJ6fq", "reviewer": "Reviewer_DHWr", "summary": "This paper proposes two efficient and practical methods, SLTRP and SLRP, for generating CAMs and saliency maps for SNNs for the first time. Based on these, the authors propose EventRPG to achieve data augmentation, which drops events and mixes events with Relevance Propagation Guidance.", "questions": "The author should carefully illustrate RPGMix and the performances of SLRP and SLTRP are challenged in the metric of the objective faithfulness.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper proposes Spiking Layer-wise Relevance Propagation(SLRP) rule and Spiking Layer-Time-wise Relevance Propagation(SLTRP) rule, the layer-wise relevance propagation method of SNNs for the first time, which can obtain the feature contribution at each pixel. RGBDrop and RGBMix are established to achieve data augmentation based on the generated CAMs. The results of experiments prove the usefulness of SLRP and SLTRP both for accuracy and efficiency. The EventRPG shows good performance in object recognition and action recognition tasks.", "weaknesses": "1. The description of RPGMix is quite simple and unclear. Section 4.3 fails to clearly illustrate the algorithm flow of RPGMix. Many operations in Fig 3(b) are not carefully analyzed, such as sample position in the Nonoverlapping Region, which makes it hard to understand. I think Fig3 is the main figure of this paper and Fig 3(b) accounts for the most part of Fig3, hence the authors need to spend more space to describe it. Otherwise, the readers may feel confused about RPGMix.\n2. Equation 15 lacks physical meaning and theoretical basis. It needs more explanations for researchers to make further progress.\n3. The objective faithfulness of SLRP and SLTRP is not outstanding enough. In N-Cars,  DVSGesture, and SL-Animals datasets, the proposed methods have similar or even inferior performance compared to other algorithms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XUo3DimIPL", "reviewer": "Reviewer_rLwX", "summary": "This paper proposes a mixup-based data augmentation method for training Spiking Neural Networks (SNNs). Inspired by saliency-based augmentation in RGB image vision such as Puzzle Mix and SaliencyMix, the authors derive the Class Activation Map (CAM) and saliency map for SNNs, and then mix two samples based on them. Experimental results show that the proposed EventRPG consistently improves the classification accuracy on various datasets.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The derivation of the CAM and saliency map of SNNs itself is a clear contribution. The results in Table 1 prove the correctness of this\n- EventRPG is able to consistently improve performance across event datasets\n- The time cost of EventRPG is comparable to similar augmentations in conventional vision", "weaknesses": "My main concern is regarding the experiments:\n- The paper motivates the need for event data augmentation with the statement that \"the lack of huge event-based datasets similar to Imagenet prevents us from improving the model performance on relatively small datasets using a Pretrain-Finetune paradigm\". However, there is an event camera version of ImageNet available [1], and its paper shows that pre-training on N-ImageNet can greatly improve the accuracy on other datasets via transfer learning. Therefore, this statement in the Introduction is wrong\n- Since N-ImageNet is available, I would like to see results on this dataset. This is similar to conventional vision research on data augmentation, where ImageNet is the best testbed. If the authors are not able to train on N-ImageNet, the mini subset can be considered, though I do not think that is a comprehensive benchmark\n- The authors compare EventRPG with conventional vision methods such as Grad-CAM in Table 1. Thus, I wonder if it is possible to compare with Puzzle Mix and SaliencyMix in Table 3?\n- NDA applies the method to unsupervised contrastive learning and shows promising results. Is it possible to conduct such experiments using EventRPG?\n\n[1] Kim, Junho, et al. \"N-ImageNet: Towards robust, fine-grained object recognition with event cameras.\" ICCV. 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)