Temporally-Weighted Spike Encoding for Event-based Object Detection and Classification
Authors: Nikolaus Salvatore
Justin Fletcher
Venue: Submitted to ICLR 2023 (2023)
Keywords: Event-based vision
spiking neural networks
object detection
classification

Abstract: Event-based cameras exhibit high dynamic range and temporal precision that could make them ideal for detecting objects with high speeds and low relative luminance. These properties have made event-based cameras especially interesting for use in space domain awareness tasks, such as detecting dim, artificial satellites with high brightness backgrounds using ground-based optical sensors; however, the asynchronous nature of event-based data presents new challenges to performing objection detection. While spiking neural networks (SNNs) have been shown to naturally complement the asynchronous and binary properties of event-based data, they also present a number of challenges in their training, such as the spike vanishing problem and the large number of timesteps required for maximizing classification and detection accuracy. Furthermore, the extremely high sampling rate of event-based sensors and the density of noisy space-based data collections can results in excessively large event streams within a short window of recording. We present a temporally-weighted spike encoding that greatly reduces the number of spikes derived from an event-based data stream, enabling the training of larger SNNs with fewer timesteps for maximal accuracy. We propose using this spike encoding with a variant of convolutional SNN trained utilizing surrogate spiking neuron gradients with backpropagation-through-time (BPTT) for both classification and object detection tasks with an emphasis on space-domain awareness. To demonstrate the efficacy of our encoding and SNN approach, we present competitive classification accuracies on benchmark datasets N-MNIST (99.7%), DVS-CIFAR10 (74.0%), and N-Caltech101 (72.8%), as well as state-of-the-art object detection performance on event-based, satellite collections.  

Under review as a conference paper at ICLR 2023
REMPORALLY-WEICEE EVENT-BASER OBJECT DETECTION AND CLASSIFI- CATION
SPIKE ENCODING FOR
Anonymous authors Paper under double-blind review
ABSTRACT
Event-based cameras exhibit high dynamic range and temporal precision that could make them ideal for detecting objects with high speeds and low relative lu- minance. These properties have made event-based cameras especially interesting for use in space domain awareness tasks, such as detecting dim, artificial satellites with high brightness backgrounds using ground-bases optical sensors; however, the asynchronous nature of event-based data presents new challenges to perform- ing objection detection. While spiking neural networks (SNNs) have been shown to naturally complement the asynchronous and binary properties of event- based data, they also present number of challenges in their training, such as the spike vanishing problem and the large number of timesteps required for maximizing classification and detection accuracy. Furthermore, the extremely high sampling rate of event-based sensors and the density of noisy space-based data collections can result in excessively large event streams within a short window recording. spike encoding that greatly reduces the number
We present temporally of spikes derived from an event- -ased data stream, enabling the training of larger SNNs with fewer timesteps for maximal accuracy. We propose using this spike encoding with a variant of convolutional SNN trained utilizing surrogate spiking neuron gradients with tion and object detection tasks with an emphasis on space-domain awareness. To demonstrate the efficacy of our encoding and SNN approach, we present compet- itive classification accuracies on benchmark datasets N-MNIST (99.7%), DVS- CIFAR10 (74.0%), and N-Caltech101 (72.8%), as well as state-of-the-art object satellite collections.
aappaggtttonthhougghhe timmuggg--mmmeaee (BPTT) for both classifica-
detection performance on
INTRODUCTION
In recent years, the number of resident space objects (RSOs) in low-Earth orbit (LEO) and geosychronous- orbit (GEO) has steadily grown, and consequently driven greater interest in the detection and tracking of such targets using ground-based optical telescopes. The tracking of RSOs, such as satellites or space debris, presents a unique challenge that these targets often have very few distinguishing features from their surroundings and are difficult to image at high speeds. Furthermore, such targets are often far dimmer than ambient lighting, especially in both cis-lunar orbits and daytime viewing. These challenges motivate the need for new hardware sensors and com- puter vision techniques that can be easily integrated with existing moound-bssed detection schemes.
Event-based cameras, or dynamic vision sensors, are one attractive technology that presents solu- tion to imaging RSOs. These cameras operate without a global clock, allowing each individual pixel to asynchronously emit events based on detected changes in illuminance at high frequency. Each pixel exhibits a logarithmic response to illuminance changes, resulting in such cameras having large dynamic range. Furthermore, since pixels respond only to changes in illuminance, the data produced is far sparser compared to that of conventional sensor sampling crucial importance for ppsce-based detection tasks, the operation of event-based pixels also prevents them from saturating, which could prove incredibly useful for imaging the Moon in daylight These qualities suggest that event-bare based cameras could be ideal for the detection of dim, high-speed RSOs that generally are too challenging for conventional CCD sensors.
comparable rates, Of perhaps

---

Under review as a conference paper at ICLR 2023
However, the asynchronous nature of event-based data also poses challenge to performing object detection effectively and efficiently. A naive approach to working with event-based data is to inte- grate over pre-defined window of time produce conventional images. Since each pixel generates events asynchronously. events are given both an (a, y) location as well as a timestamp t. that cor- responds to the time of event generation relative a polarity flag, E {1,-1} that denotes the event was generated by an increase in illuminance (+1) or a decrease in illuminance (-1). For integration, these events, of the form accumulated over some window At at their respective (z. y) locations to form an equivalent image. Such integrated frames can then be used with any conventional object detection method; however, this approach loses much of the temporal information present in the original event stream.
recorded starting time. Events are also given
(z, 3/ E, p), are
Spiking neural networks (SNNs) differ from conventional neural networks in much the same way that event-based data differs from conventional images. SNNs function asynchronously, with each neuron of the network generating spikes only when its inputs cause exceed a pre-defined thresh- old, mimicking the function of biological neurons. The sparsity of SNN activation amounts to spik- ing networks being exceptionally energy efficient as compared to conventional neural networks of comparable size. However, the binary nature of spiking neuron output and the subsequent non- differentiability of their activation makes supervised training of such networks a challenging task. Furthermore, SNNs are also plagued by the vanishing spike propagation issue, where decreasing spiking activity in successive layers causes significant performance loss in larger networks (Panda) ctt al. (2020)). Nonetheless, the unique properties of SNNs naturally complement the data produced by event-based cameras, and multiple works have already shown the potential for classification and object detection on event-based data.
In this work, we present a temporal-weight encoding that greatly decreases the number of spikes derived from event data stream while maintaining overall spiking behavior and preserving temporal information. This encoding scheme is also shown to reduce the number of timesteps required to maximize classification and object detection accuracy in spiking neural networks. We also propose pseudo-spiking behavior for conventional, convolutional neural networks that removes the need for temporal credit assignment, but preserves some temporal information. This pseudo-spiking behavior is readily integrated with encoded, event-based data and enables the training of comparatively deeper models than true spiking networks. We evaluate detection results using both simulated and real base- data collection and demonstrate competitive performance on publicly available event- based classification and object detection datasets.
2 RELATED WORK
The following sections briefly explore some of the most notable works in each area touched upon in our own work.
2.1 SPACE DOMAIN AWARENESS
As previously mentioned, the detection of dim, high-speed RSOs is already challenging task that is made even more difficult by conditions such as daylight, moonlight, and atmospheric turbu- lence. Traditionally, small targets ground-based optics have become an attractive alternative due to their power efficiency and cost effectiveness. However, optical charge-coupled device (CCD) sensors often struggle with high amounts of background noise as well as with long exposure times that can complicate the detection of fast-moving objects (Kong et al. (2019). As an alternative optical device, event- based cameras could be ideal for replacing or complemenn complementing conventional CCD sensors for RSO detection. Re- cent work has already shown the use of event-based cameras for daytime imaging of LEOs (Cohen et al. (2019). and simulated work has investigated star tracking using wenttbased data (Chin et al. (2019). These successes, in addition to the successful application of object detection models such as YOLOv3 on space imaging datasets (Fletcher et al. 22019). have motivated our work in investigat- ing space object detection with event-based cameras. Furthermore, recent advances in space scene simulation have improved the ability to experiment with high fidelity, optical space collections. In this work, we make use of the SarSim simulator to generate the large number samples necessary for model training (Cabello & Fletcher (2022)).
detected using specialized radar laser equipment, though
2

---

Under review as a conference paper at ICLR 2023
2.2 EVENT-BASED CLASSIFICATION AND OBJECT DETECTION
Due to the asynchronous nature of event-based data, ordinary computer vision techniques are not readily applicable performing classification or object detection with event-based cameras. In the simplest case, event streams can be accumulated over time into approximated images that can then be used with conventional algorithms, but this eliminates much of the rich, temporal information. Early methods of working with event-based data often relied on updating existing algorithms,, such as the Harris corner detector (Ni et al. (2012)) or Hough transform (Vasco et al. (2016)), to be compatible with asynchronous data. More sophisticated approaches, such as the HOTS (Lagorce) jet al. (2014) and HATS algorithms (Sironi et al. (2018), use new representations of event-based data (time surfaces), to exploit temporal information for classification and object detection. Newer approaches have begun to modify established machine learning models, such as YOLE or "You Only Look Events", which adapts the YOLO object detection framework for asynchronous operation (Cannici et al. (2019)). However, one of the chief challenges for algorithm development and model training is the relative lack of event-based data publicly available. While neuromorphic versions of well-known datasets such as N-MNIST, CIFAR 10-DVS, and N-Caltech101 exist, all of these datasets are generated using event-based extrapolations of the original datasets rather than actual event-based samples (Li et al. (2017)) (Orchard et al. (2015). However, the N-Cars and Prophesee Gen -uutomottee datasets are two more recent datasets that include real event- based data collections Siioon et al. (de Tournemire et al. (2020)).
2.3
SPIKING NEURAL NETWORKS
Given the asynchronous nature of event-based data, SNNs are natural choice for performing classi- fication and object detection with event-based cameras. Multiple works have already demonstrated effective use of SNNs for performing a wide array of tasks on event- event-based data, not only limited classification and object detection (Lee al. (2020) (Samadzadeh et al. (2020)) Furthermore, the innate compatibility with SNNS have made -vent-based camera datasets such as CIFAR 10-DVS useful for evaluating a range of spiking models and encoding processes (Fang et al. (202I)) Vicente- Sola et al. (2021)). However, the greatest challenge to employing SNNs is the method of training used. The behavior of spiking neurons is not differentiable and therefore not immediately trainable using ordinary backpropagation. Also, as previously mentioned, the vanishing spike phenomenon is a further limiting factor on the potential depth of SNN models. As a result, a great deal of research has gone into finding new methods of spike-based backpropagation, surrogate gradients, new training methods (Bellec et al. )2018))(W) et al. (2018) & Sejnowski (2018)). Some of the best results in terms of model accuracy have come from converting pre-traine pre-trained artificial neural networks (ANNs) into spiking models, but this method can incur losses in the efficiency and speed of the resulting SNNs (Rathi et al. (. Despite many possible solutions, training methods for SNNs continues to be an area of great interest.
entirely
3 METHODS
In the following section, we describe the weighted spike representation used to encode event-based vision streams for use with deep spiking neural networks. We also detail the integration of this en- coding process with the spiking neural networks and the truncated, surrogate gradient based learning used train the large networks used. Finally, we describe the method by which optimal spiking hy- perparameters are chosen to maximize both classification and object detection performance.
3.1 WEIGHTED SPIKE ENCODING
Given a dynamic vision sensor of width and height x H, the stream of events generated by such sensor would be of the form
Ex = Co Calma11 Cn = (Wis Hoy Pw.)
(I)
where Ey represents the entire stream of N events and each event e,, is of the form (Zn, Un. tn Pr). In this context, Iy and yn are in the range [1, W] and [1, H] respectively, and represent the pixel location at which the event occurred, while L, is the timestamp associated with the generated event.
3

---

Under review as a conference paper at ICLR 2023
Pn is the polarity of the event with value of Pn = (-1.1) indicating that the event is generated by either a increase luminance, +1, or decrease in luminance, -1. In order to use conventional computer vision techniques with event-based data, an event stream can be integrated over either a particular number of events N or a chosen range of time At with events accumulated at their respective (xr,)) locations.
Σ 8(zn yn) N [n.w.H En=o 8(In, Un) Pn =- Ven
1o.wmm=
Pn = +1
(2)
In general, event streams are integrated as described in Eqn. where timestamps are ignored and simple counts of events are accumulated at their respective locations (In, ya), with positive and negative polarity events separated into separate channels to produce 2 x W H images suit- able for image-based models and conventional algorithms. In this work, we use this integration in conjunction with standard object detection and classification models baseline for performance comparison.
In order reduce the possible number of spikes in dense wrent-based data, while also encoding some of the temporal information into the resulting output, we employ a modified form of event stream integration. While event-based data can be used natively with asynchronous input spiking neural networks, our encoding greatly reduces the timescale over which spikes are presented to the end network and enables the application of conventional convolutional networks through additional pre- processing steps. Our approach to encoding involves partial integration process that is dependent on whether feature extraction performed in real-time or on previously recorded event streams. In the real-time case, we choose time window, At, from which events will be accumulated over the event stream until total of timesteps, or windows, have been presented to the network. Conversely, with previously recorded event streams, we can choose the number of timesteps, T', and then determine the time window, At. required to evenly divide the event stream into the desired number of timesteps.
1(4.)
(T(T + 1)) Σ (ts
Spike
Train
Weighted
Time window
Figure Generation of a temporally-weighted spike frame from an event stream.
Figure [ depicts event stream with a window of events accumulated over given time range, At. In order to capture temporal information added the time difference of their corresponding timestamp with that of the first timestamp in the associated window, i.e. l - l.- Once integrated, temporal weights are normalized over the entire range of possible timestamps within the window and organized into two-dimensional frame according to their x/y locations. is also important to note that this integration encoding is applied separately to positive and negative polarity events such that the final frames have channels similar to ordinary integrated frames. This encoding is formalized in Eqn.
reduced representation, events at each x-y location
2 //..W.H = AMAHT o= 8(Fnn Un) (lop 10)] Σn 5(xn, yo) * (tn, to)l Pr. = Ven
Pr: = +1
1(p,, W, H)
[p.w.. A((++1
Once encoded, the new sequence of temporal weight frames can be used either directly such as in conventional 3D convolutional network modified spikes in work. Alternatively, the
4

---

Under review as a conference paper at ICLR 2023
frames can be unrolled into individual elements with thresholds applied to produce binary spikes for use with con-coovolutional spiking neural networks; however, the effectiveness of this approach is not explored in this work.
3.2 SPIKING NEURAL NETWORK INTEGRATION
For performing the classification and object detection, we explore two avenues for incorporating spiking event data: a true spiking approach and a pseudo-spiking approach. For our true spiking approach, we the discretized version of the leaky integrate and fire neuron proposed in Rathi ct al. (2020). Each layer has an associated membrane potential, w(t). that accumulates with the imput of weighted spikes and generates spikes of its own if potential exceeds voltage threshold, u 2 V.. The dynamics are detailed in equations{4ande
ui = Au + Σ Wijoj - Vio''1
(4)
of 1, u - V 0, otherwise
(5)
where t is the current timestep. À is a potential leak constant, wa are weights associated with the previous layer, oj: are outputs of previous layers, and the voltage threshold. Eqn. 4 holds for the soft reset case, in which membrane potential is reduced by the voltage threshold upon firing. Empirically, we found better classification performance on deep networks when using the hard reset alternative, where membrane potential uf is reduced to 0 when u 2 V,. We use a similar training process as in (Rathi et al. )2020) by using the membrane potential at the final timestep as the network output needed to calculate the relevant loss metrics: cross-entropy in the classification case and the YOLO loss metrics (box regression, classification, and objectness) in the object detection case. However, since the exact timing of imput spikes is altered due temporally weighted spike encoding, we opt to use the surrogate gradient
50 du
= arma |uu ViI)
where is the layer output, is membrane potential, and Vi is the voltage threshold for output generation. This surrogate gradient for the spiking neuron output is then used with backpropaga- tion through time (BPTT) in order perform supervised training with spatial and temporal credit assignment.
Conversely, we posit that, for purposes, pseudo-spiking approach that removes the need for temporal credit assignment could still benefit from the temporal information found in the encoded event stream. Removing the need for BPTT should enable much faster convergence and potentially the effective training of much larger models normally too memory intensive for true spiking neural networks. order remove the need for temporal assignment, we present the entire spike train to each layer in series, such that membrane potential is accumulated for all timesteps T on each layer before passing output to the succeeding layer. In order to impart some of the temporal information in this pseudo-spiking format, we use the membrane potential directly as the output of active layers. Equation describes the process for the initial layer, which is also depicted fully in Figure 2 where Equations 8 and detail the behavior for all successive layers in the network.
U(T) = = x(T((1))
U.(T) - Σ (tU - Vi0(t - 1)] + Σ wT)O,(T)
(8)
(,,()),,)0)>>
of 0 otherwise
(9)
5

---

Under review as a conference paper at ICLR 2023
Weigtted Frame
[(c))
I(t,))
[(())
(omo(T))
Coms(T))
ComeTT)
Cone(T)
Spike
Train
Figure 2: Accumulation of temporally snnporally-weighted spikes the membrane potential of pseudo-spiking convolutional layer.
As previously mentioned, the positive and negative polarity events (or ON and OFF events, respec- tively) of the input stream encoded separately. After encoding, the positive and negative spikes for particular time window are accumulated into two-dimensional frames and concatenated to form two-channel spike frames with shape similar that of ordinary image. Figure depiicss the broad process applied to sample of the N-MNIST dataset.
Accumiiation
Concotanation
Positve Weight Spiking
Negelive Weight Spiking
Coneehutinnl
Layer
0.2
0.4
0.6
08
10
Figure 3: N-MNIST sample encoded and passed pseudo-spine spiking convolutional network.
4 EXPERIMENTS
4.1 SPIKING HYPERPARAMETER SEARCH
In order choose optimal hyperparameters necessary for spiking behavior, the hyperpa- rameter optimization package Optuna (Akiba et al. (2019). We perform hyperparameter optimiza- tion across three network architectures: VGG19, ResNet5O, and DarkNet53. While VGG19 and

---

Under review as a conference paper at ICLR 2023
ResNet50 are the feature extraction backbones used for classification tasks, also optimize hyper- parameters for DarkNet53 to increase object detection performance and to investigate the efficacy of training larger networks with the pseudo-spiking behavior and encoding. Each feature extraction backbone trained on a classification task, using the CIFAR 10-DVS dataset, for 10 epochs with hy- perparameter search performed for voltage threshold (Vi), leakage (A), gradient output scaling (a), and total timesteps (7"). Hyperparameters are optimized solely for classification accuracy, which will tend to maximize timesteps for the pure spiking case due to the strong positive correlation between spiking timesteps and accuracy. Highest performing parameters are shown in Table [
Table 1: Optimal Spiking Hyperparameters
Architecture
Voltage Thresh. (V))
Leakage (A)
Gradient Scale (a)
Timesteps (T)
sVGG19
3.191
0.325
2.74
185
sResNe150
2.475
0.127
1.73
113
pseudo-VGGiI
2.002
0.145
5.58
12
Dseudo-ResNe150
2.505
0.07
3.79
39
Eseudo-DarkNen
1.622
0.054
6.51
95
Based on the remaining hyperparameter tuning results, we find that the parameters that correlate most strongly with model size are both the leakage coefficient and the number of spiking timesteps. In general, larger spiking models requiring greater number timesteps in order reach maxi- mum accuracy is expected, and aligns with other previous results with SNN training. However, the leakage coefficient being exceptionally small for both ResNet50 and DarkNet53 suggests accuracy maximized when output generated at nearly every timestep for these larger models. This result may be explained by the overall reduction of input spikes due to the temporal encoding driving the need for neurons to fire more consistently.
4.2 CLASSIFICATION TASKS
While the primary focus of our work is to adapt event streams and spiking networks for object detection, we also include performance on classification tasks as both means of verification and as a point of comparison with other methods of performing classification on neuromorphic data. Classification results were evaluated for the N-MNIST, CIFARIO-DVS, N-Cars, and N-Caltech101 datasets and compared to of the most prominent results from literature. Although not widely used as an event-based classification dataset, we chose to include the N-Cars classification dataset due its relation the BEENIAAtomotive dataset used more prominently for object detection. In all instances, the indicated model architectures for our method use the same layer structure as available in pretrained, PyTorch model zoo (Paszke al. (2019)), albeit with convolutional layer behavior replaced by the pseudo-spiking behavior previously described.
Table 2: Classification Accuracies (Top 2 best classification accuracies per dataset are bolded).
Method
N-MNIST
CIFARTO-DVS
N-Cars
N-Caltech101
sVGG19 Encoding
0.975
0.7151
0.911
0.689
sResNet50 + Encoding
0.898
0.6119
0.845
0.615
pseudo- BecudoVVGG19 VGG19 Encoding
0.997
0.7404
0.924
0.728
Pseudo-ResNet500 Encoding
0.974
0.6510
0.889
0.662
Sironi et al. (018))
0.991
0.524
0.902
0.642
Cannici et al. (2019)
0.927
0.702
Messikommer
(2020)
0.944
0.745
Wu et al. 9
0,9953
0.605
Samadzadeh et
2020
0.996
0.692
Vicente-Sola et al.
2021
0.7298
Fang et al. 202
0.996
0.748
In terms classification accuracy across the general datasets, we see that the VGG19 architecture outperforms the deeper ResNet50 models, but both pseudo-spiking architectures generally outper-

---

Under review as a conference paper at ICLR 2023
form their fully spiking counterparts. As intended, the performance gap between the useudo-spikingg and fully spiking methods is greater for the ResNet50 model, suggesting that the pseudo-spiking behavior and temporally-weighen encoding has positive impact on the training of deeper archi- tectures.
4.3 OBJECT DETECTION TASKS
For our primary SDA task, we have object detection datasets denoted Satellite Collect and Sat- Sim in Table 3 As previously mentioned, the SatSim space scene simulator is capable of generating high-fidelity, event-based simulations of satellite collections with degree of tunable conditions and sensor parameters. Given the small size of real data collections available (about 900 samples) in the Satellite Collect dataset, we first generate large collection of simulated samples with approxi- mately equivalent parameters using the SatSim simulator. As basis for showing improvement, we compare results from the pseudo-spiking YOLO model with seaporrall-weighte. encoding to both conventional YOLO model using integrated frame and pseudo-spiking YOLO model using spikes generated from Poisson- Poisson-encoded encoded integrated images. Poisson-encodee has been used in mul- tiple previous works for generating spike frames from ordinary images (Rathi et al. (2220). and use this encoding here to gauge the effectiveness of adding the temporally-weighted weighted encoding. Both trained and evaluated this dataset, after which all models
spiking and non-spiking models retrained and evaluated on the real dataset to best estimate real-world performance. In the context of the SDA datasets, each dataset has only single class (satellite) and the goal is achieve maximal detection. As a result, the metric of greatest interest for the SDA task is the maximum Fi score, and full precision-recit results are shown for these datasets only. Figure 4 shows an example of real optical, event-based satellite collection and a simulated approximation generated by the SatSim simulator.
Figure 4: Real satellite collection (left) SatSim generated sample (right). Samples are generated as full event streams, but displayed here as integrated frames.
For general comparison, we also evaluate our method on two publicly available datasets: N- Caltech101 and the Prophesee GENI-Automotive dataset. As is customary for general object detec- tion, we show the mean average precision (mAP) (Everingham et al, (2010)) results for our method and those of other thee--t-the-art asynchronous detection methods.
8

---

Under review as a conference paper at ICLR 2023
Table 3: Performance results for spiking object detection models and conventional models on equivalent datasets.
Method
Dataset
Precision
Recall
PT
MAP
YOLO+Int. Frame
Satellite Collect
0.43909
0.65570
0.525997
pseudo-sYOLOOte Frame
Satellite Collect
0.61765
0.88489
0.72750
Secudo-SYOL++Em
Satellite Collect
0.67804
0.90103
0.7740
YOLO+Int. Frame
SatSim
0.65179
0,65710
0.65443
pseudo-s Mondoo,sOOO+Int, Frame
SatSim
0.745332
0.739609
0.74246
pseudo-sYOLO+Enc.
SatSim
0.77709
0.74871
0.76264
Dseudo-sYOLO~the Frame
N N-Caltech100
0.331
Dseudo-SYOLO-+
N- N-Caltech10)
0.595
Cannici et al. (2019)
N-Caltech
0.398
Messikommer al. (2020)
N-Caltech101
0.643
pseudo-sYOL0+Int. Frame
GENT-Auto
0.124
pseudo-sYOLO+Enne
GENI-Aut
0.339
Messikommer et al. (2020)
GENI-Auto.
0.149
Cannicı et al. (2020)
GENI-Auto.
0.31
Perot al. 2020)
GENI-Auto.
0.40
Across these object detection results, we see a significant increase in the maximum F1 score for both real and simulated satellite data collections when using pseudo-spiking YOLO compared to conventional YOLO model on integrated frames. Furthermore, the pseudo-spiking model using the temporally-weighted encoding displays another significant increase in performance that appears be even more pronounced on real collections over the simulated equivalents. This may be result of the less uniform timestamps of generated events in real versus simulated data, which suggests that the cemporally-weighte
encoding is successfully preserving temporal data lost in the integrated frames. In terms of the public datasets, the pseudo-spiking YOLO model with temporally-weighee encoding shows competitive mean average-precision (mAP) on N-Caltech10 101 and the Prophesee GENI-Automotive dataset, with its performance only beaten out by some of the most recent and more complex asynchronous methods available.
5 CONCLUSION
In this work, we have presented a new temporally- spike encoding for event-based camera streams that greatly reduces the number spikes required for processing noisy event streams, while preserving useful temporal information. We have additionally demonstrated pseudo-spiking be- havior for convolutional layers that allows us to mimic properties of a spiking network, but allows train deep networks with conventional backpropagation. Both object detection and classifica- tion results show that the combination of temporally-weighte spike encoding and pseudo-spiking behavior increase accuracy and performance, especially when used deeper models. For our own area of interest, this method also demonstrates superior performance on object detection tasks for space-domain awareness, while also generalizing well to publicly available datasets.
In the future, we hope to assess the training of larger array of models, well as incorporate the method with newer versions object detection models. Furthermore, we have introduced the spike encoding could be used to process event streams
method by which the cemporally-weightee real-time, this potential as of yet unexplored. Real-time object detection warrants an ex- haustive study into the potential energy and memory-saving benefits, and could also highlight the comparative strengths of event-based cameras for SDA tasks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2623-263362 2019.
Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, and Wolfgang Maass. Long short-term memory and learning-to-learn in networks of spiking neurons. Advances in neural information processing systems, 31, 2018.
Alexander Cabello and Justin Fletcher. Satsim: synthetic data generation engine for electro-optical imagery of resident space objects. In Sensors and Systems for Space Applications XV, volume 12121, pp. 53-74. SPIE, 2022.
Marco Cannici, Marco Ciccone, Andrea Romanoni, and Matteo Matteucci. Asynchronous convolu- tional networks for object detection in neuromorphic cameras. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition Workshops, pp. 0-0, 2019.
Marco Cannici, Marco Ciccone, Andrea Romanoni, and Matteo Matteucei. differentiable rent surface for asynchronous event-based data. In European Conference on Computer Vision, pp- 136-152. Springer, 2020.
Tat-Jun Chin, Samya Bagchi, Anders Eriksson, and Andre Van Schaik. Star tracking using an event camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 0-0, 2019.
Gregory Cohen, Saeed Afshar, Brittany Morreale, Travis Bessell, Andrew Wabnitz, Mark Rutten, and André van Schaik. Event-based sensing for space situational awareness. The Journal of the Astronautical Sciences, 66(2):125-1)) 2019.
Pierre de Tournemire, Davide Nitti, Etienne Perot, Davide Migliore, and Amos Sironi. A large scale event-based detection dataset for automotive. arXiv preprint arXiv:2001.08445 084499999999,, 2020.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2): 303-338 2010.
Wei Fang, Zhaofei Yu, Yanqi Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian. In- corporating learnable membrane time constant to enhance learning of spiking neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2661-2671, 2021.
Justin Fletcher, Ian McQuaid, Peter Thomas, Jeremiah Sanders, and Greg Martin. Feature-based satellite detection using convolutional neural networks. Proceedings of the Advanced Maui Optical and Space Surveillance Technologies Conference, pp. 11, 2019.
Dongsung Huh and Terrence J Sejnowski. Gradient descent for spiking neural networks. Advances in neural information processing systems, 31, 2018.
Sijie Kong. Jin Zhou, and Wenli Ma. Effect analysis of optical masking algorithm for geo space debris detection. International Journal of Optics, 2019, 2019.
Xavier Lagorce, Cédrie Meyer, Sio-Hoi leng, David Filliat, and Ryad Benosman. Asynchronous event-based multikernel algorithm for high-speed visual features tracking. IEEE transactions neural networks and learning systems, 26(8): 2014.
Chankyu Lee, Adarsh Kumar Kosta, Alex Zihao Zhu, Kenneth Chaney, Kostas Daniilidis, and Kaushik Roy. flowenflownet: event-based optical flow estimation with energy-efficient hybrid neural networks. In European Conference Computer Vision, pp. 366-382 Springer, 2020.
Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. Cifarl0-dvs an event-stream dataset for object classification. Frontiers in neuroscience, 11 11:309, 2017.
10

---

Under review as a conference paper at ICLR 2023
Nico Messikommer, Daniel Gehrig, Antonio Loquercio, and Davide Scaramuzza. Event- based asyn- chronous sparse convolutional networks. In European Conference on Computer Vision, pp. 415- 431. Springer, 2020.
Zhenjiang Ni, Cécile Pacoret, Ryad Benosman, Siohoi leng, and Stéphane RÉGNIER* Asyn- chronous event-based high speed vision for microparticle tracking. Journal of microscopy, 245 (3):236-244, 2012.
Garrick Orchard, Ajinkya Jayawant, Gregory K Cohen, and Nitish Thakor. Converting static image datasets to spiking neuromorphic datasets using saccades. Frontiers in neuroscience, 9:437, 2015.
Priyadarshini Panda, Sai Aparna Aketi, and Kaushik Roy. Toward scalable, efficient, and accu- rate deep spiking neural networks with backward residual connections, stochastic softmax, and hybridization. Frontiers in Neuroscience, 14:653, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Etienne Perot, Pierre de Tournemire, Davide Nitti, Jonathan Masci, and Amos Sironi. Learning to detect objects with megapixel event camera. Advances in Neural Information Processing Systems, 33: 33:16639-16652, 2020.
Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, and Kaushik Roy. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. arXiv preprint arXiv:2005.01807, 2020.
Ali Samadzadeh, Fatemeh Sadat Tabatabaei Far, Ali Javadi, Ahmad Nickabadi, and Morteza Haghir Chehreghani. Convolutional spiking neural networks for spatio-temporal feature extraction. arXiv preprint arXiv:2003. 12346, 2020.
Amos Sironi, Manuele Brambilla, Nicolas Bourdis, Xavier Lagorce, and Ryad Benosman. Hats: Histograms of averaged time surfaces for robust event-based object classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1731-1740, 2018.
Valentina Vasco, Arren Glover, and Chiara Bartolozzi. Fast event-based harris corner detection exploiting the advantages of event-driven cameras. In 2016 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 4144-4149. IEEE, 2016.
Alex Vicente-Sola, Davide L Manna, Paul Kirkland, Gaetano Di Caterina, and Trevor Bihl. Keys to accurate feature extraction using residual spiking neural networks. arXiv preprint (rXiv:2/11.05990 2021.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training segh-performance spiking neural networks. Frontiers in neuroscience. 12:331, 2018.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 1311-1318, 2019.


Reviews:
[{"review_id": "n1-ELuews5", "reviewer": "Reviewer_Uhms", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a spike encoding scheme that they argue reduces the number of spikes from an event based spike stream. They argue this enables the training of larger SNNs with fewer timesteps.\n\nThey use this representation to train an SNN and test their trained model on DVS datasets such as N-MNIST and DVS-CIFAR10 and N-Caltech.", "summary_of_the_review": "Overall for the reasons I mentioned above I do not feel this paper is at the level of ICLR", "strength_and_weaknesses": "Strengths:\n\nS1: papers on SNN are always welcome as they hold the promise of efficient low power inference\n\nWeaknesses:\n\nW1: why do the authors limit themselves to DVS datasets which are not as popular as the standard RGB datasets? There exists a significant literature on SNN tested with RGB datasets and achieving close to state of the art performance on this RGB data. Typically this is accomplished by having a transduction layer (typically a CNN) as the first layer that converts the RGB data to spike data. Note that often a temporal window of 1 is used (ie up to 1 spike per pixel is output per frame). See for example \"Convolutional networks for fast, energy-efficient neuromorphic computing\" Esser et al 2016 for an example demonstrating SOA performance (many more similar papers since then). \n\nW2: use \\times instead of x to denote cross product\n\nW3: page 4, eq 3: why is this done with respect to t_0 and not with respect to the median timestamp for example. Overall spend more time explaining what the advantage of this approach is\n\nW4: eq 4, eq 5: should j be a subscript?\n\nW5: each layer seems to output a rate code and the algorithm accumulates. This does not seem really efficient power wise or time wise. There exist much more efficient algorithms in the literature as I alluded to above. Overall the argument that the number of spikes is reduced significantly does not seem to be supported enough. Quantify this better. Put a table with the average and maximum number of spikes per frame achieved by this algorithm vs the competition\n\nW6: Section 4.3 \"Figure ??\". Missing figure number", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "evtxaAf4ue", "reviewer": "Reviewer_gpjB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors propose to train spiking neural networks using a temporally-weighted spike encoding. This is tested on standard backbones with a classical surrogate gradient learning. It is then applied on several benchmarks, but also on an interesting star-tracking application.\n", "summary_of_the_review": "The paper proposes a novel encoding of event-based inputs which is applied to generic benchmark but also to a star-hacking application. It shows promising results, yet relatively few qualitatives results are shown to understand why this encoding would be better than others. \n\nDo you observe different results with different parameterisations of your encoding? Do you observe more explainable kennels? Also, the star-tracking applications seems like an \"easier\" one. Could you a similar result with a shallower network?", "strength_and_weaknesses": "Equation (3) is the core of the novelty brought by the paper, yet it is at first difficult to understand. First, it uses a different convention as the standard spike accumulating described in Equation (2) Then you should specifically define the events $e_n$ for which you accumulate information. From Equations (7-9), it is similar to the HOTS approach, yet with a different kernel and with events within the time window (instead of the time to last spike). It would therefore be important to justify that choice and compare the accuracy of your model with HOTS for instance. Also, given the leakage term (Eq 7) how do you interpret your optimisation in table 1 with respect to the different datasets? Why not use different constants at different layers? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "chwBs1CV4L", "reviewer": "Reviewer_AfjB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the problem of event-based object detection and classification. The authors introduce a temporal integration-based event representation. Then, a modified pseudo-spiking neural network is applied to extract event representations for object classification and detection. However, the novelty of the proposed method is limited. Additionally, the performance is unsatisfactory.", "summary_of_the_review": "The novelty of the proposed method is limited, and the performance is unsatisfactory. The writing and organization should be significantly improved. Therefore, the submission is far away from being published.", "strength_and_weaknesses": "(1)Strength:\n\nThe explored issue is challenging but intriguing.\n\n(2)Weakness:\n\n1) In Sec 3.1, the authors state that “In order to preserve temporal information, .... employ a modified form of event stream integration.”  Nonetheless, the reviewer thinks that any temporal integration of event data would inevitably degrade the original temporal data. Consequently, the authors should revise their motivations or alter the network inputs.\n\n2) Applying the spiking neural network [1,2] to the frame-like representation of event data has been widely adopted for event data processing. The novelty of the proposed method is very limited.\n\n3) The authors should note the best or second-best approaches in Tables 2 and 3.\n\n4) The authors should carefully check the format of the paper. The captions should be placed above tables. The authors should revise the citation for Figure 4. In Tables 1 and 2, the bottom bounds are absent. Table 3 is beyond the limited scope.\n\n5) It is quite difficult to determine the specific details of Figure 4. Authors should adopt other representations.\n\n6) The authors should compare the pseudo-spiking layer with the spiking layer for the item recognition task in table 3.\n\n7) Authors should compare the computational complexity of different methods, such as FLOPs, number of parameters, and execution time.\n\n8) The authors must redraw Figures 1 and 2. Currently, it is difficult to understand them.\n\n9) For the object identification studies, the authors only compared with a very basic baseline, YOLO + Int. Frame. Numerous effective event-based approaches [3] or event-branch fusion-based detectors [4,5] exist. Authors should compare with them.\n\n[1] Li J, Dong S, Yu Z, et al. Event-based vision enhanced: A joint detection framework in autonomous driving[C]//2019 ieee international conference on multimedia and expo (icme). IEEE, 2019: 1396-1401.\n\n[2] Zhang J, Dong B, Zhang H, et al. Spiking Transformers for Event-Based Single Object Tracking[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 8801-8810.\n\n[3] Li J, Li J, Zhu L, et al. Asynchronous Spatio-Temporal Memory Network for Continuous Event-Based Object Detection[J]. IEEE Transactions on Image Processing, 2022, 31: 2975-2987. \n\n[4]Liu M, Qi N, Shi Y, et al. An attention fusion network for event-based vehicle object detection[C]//2021 IEEE International Conference on Image Processing (ICIP). IEEE, 2021: 3363-3367.\n\n[5] Tomy A, Paigwar A, Mann K S, et al. Fusing Event-based and RGB camera for Robust Object Detection in Adverse Conditions[C]//2022 International Conference on Robotics and Automation (ICRA). IEEE, 2022: 933-939.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "P2q9aMxMjYA", "reviewer": "Reviewer_WXPN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A novel spike encoding system for event streams is presented, that allows to reduce the amount of spikes and increase the performance of spike-learninig event-based pipelines. The method is evaluated on recognition benchmarks, as well as on Earth satellite tracking datasets.", "summary_of_the_review": "I believe the paper has good potential for publication, but could benefit from some improvements to the text structure (emphasizing the contributions more), and adding the computational performance evaluation.", "strength_and_weaknesses": "The spike-based learning is a very relevant but often overlooked field in event-based vision; as the paper presents a method with a competitive accuracy scores and a potential to improve computational performance, this encoding scheme could potentially have a good impact within the spike-based learning community. \n\nOne of the benefits presented was the possibility to reduce the computational cost, but the evaluation section omits the results on the computational performance. It would be useful to include a holistic evaluation of training and inference computational cost, and hyperparameter (such as temporal window size) impact.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject