Hybrid Spiking Vision Transformer for Object Detection with Event Cameras
Authors: Qi Xu
Jie Deng
Jiangrong Shen
Biwu Chen
Huajin Tang
Gang Pan
Venue: ICML 2025 poster (2025)
Keywords: Spiking Neural Networks
Fall Detection
Object Detection With Event Cameras
Primary area: applications->neuroscience_cognitive_science

Abstract: Event-based object detection has attracted increasing attention for its high temporal resolution, wide dynamic range, and asynchronous address-event representation. Leveraging these advantages, spiking neural networks (SNNs) have emerged as a promising approach, offering low energy consumption and rich spatiotemporal dynamics. To further enhance the performance of event-based object detection, this study proposes a novel hybrid spike vision Transformer (HsVT) model. The HsVT model integrates a spatial feature extraction module to capture local and global features, and a temporal feature extraction module to model time dependencies and long-term patterns in event sequences. This combination enables HsVT to capture spatiotemporal features, improving its capability in handling complex event-based object detection tasks. To support research in this area, we developed the Fall Detection dataset as a benchmark for event-based object detection tasks. The Fall DVS detection dataset protects facial privacy and reduces memory usage thanks to its event-based representation. Experimental results demonstrate that HsVT outperforms existing SNN methods and achieves competitive performance compared to ANN-based models, with fewer parameters and lower energy consumption.

Hybrid Spiking Vision Transformer for Object Detection with Event Cameras
Qi Xu Jie Deng' Jiangrong Shen f 2234 Biwu Chen
Huajin Tang
Gang Pan 1
Abstract
1. Introduction
Deep neural networks have garnered considerable attention for their remarkable performance across spectrum of tasks, ranging from computer vision to natural language process- ing. The exceptional performance is primarily attributed to their intricate and specialized architectures. Notably, the Transformer architecture (Beal et al., 2020; Carion et al., 2020; Guo et al., 2023) has demonstrated outstanding ef- ficacy in object detection tasks, albeit the expense of heightened energy consumption due its intricate structure. In parallel, drawing inspiration from the event-driven nature of the human brain, spiking neural networks (SNNs) (Hu et al., 2023a;b; Xu et al., 2023; 2024b; Xiao et al., 2025) offer distinct advantages, including low energy consump- tion and rich spatiotemporal dynamics. The emergence of neuromorphic hardware such as Darwin3 (Ma al., 2024) further supports efficient on-chip learning with novel in- struction set architecture. Recent studies have also advanced the efficiency, compressibility, and biological plausibility of SNNs (Xu et al., 2024a; Shen et al., 2025; 2024). In this context, our study explores the integration of Transformer- based Artificial neural networks (ANNs) and SNNs, aiming to combine their respective advantages in a hybrid frame- work for efficient object detection.
Event-based object detection has attracted increas- ing attention for its high temporal resolution, wide dynamic range, and asynchronous address-event representation. Leveraging these advantages, spik- ing neural networks (SNNs) have emerged as a promising approach, offering low energy con- sumption and rich spatiotemporal dynamics. To further enhance the performance of besed object detection, this study proposes novel hy- brid spike vision Transformer (HsVT) model. The HsVT model integrates spatial feature extrac- tion module to capture local and global features, and a temporal feature extraction module to model time dependencies and long-term patterns in event sequences. This combination enables HsVT to capture spatiotemporal features, improving its capability in handling complex event-based ob- ject detection tasks. To support research in this area, we developed the Fall Detection dataset as a benchmark for event-based object detection tasks. The Fall DVS detection dataset protects facial privacy and reduces memory usage thanks its event-based representation. Experimental results demonstrate that HsVT outperforms existing SNN methods and achieves competitive performance compared to ANN-based models, with fewer pa- rameters and lower energy consumption.
Owing to their high time resolution (in the order of mi- croseconds), high dynamic range (> 120dB), and asyn- chronous address event representation, event cameras have shown great potential in various vision tasks (Liu et al., 2024a;b; 2025). In particular, event-based object detection has attracted increasing attention. Datasets for this task are typically collected using event cameras, such as the dy- namic vision sensor (DVS) Ceerrano-Gotarredon & Linares- Barranco, 2013)and ATIS sensor (Posch, 2011). The visual information of the event camera is represented in the form of an asynchronous address event stream of ((x, y: P. t)}- The event stream indicates hog-luminosity contrast changes at time and position (ar, y). Each produced event only ap- pears the time and positions of contrast change. Hence, event-based object detection has the advantage of avoiding oversampling and motion blur compared with traditional frame-based object detection. On the one hand, SNNs have shown potential for event-based object recognition with
'School of Computer Science and Technology, Dalian Uni- versity of Technology, Dalian. China 'Faculty of Electronic and Information Engineering, Ki'an Jiaotong University, Xian, China National Key Lab of Hamman-Machine Hybrid Augmented Intel- ligence, Xi'an an Jiaotong University, Xian, China "State Key Lab of Brain-Machine Intelligence, Zhejiang University, Hangzhou, China 'Shanghai Radio Equipment Research Institule. Shanghai, China "College of Computer Science and Technology. Zhejiang University, Hangzhou, China. Correspondence to: Jiangrong Shen <jrshen @jrshen@zju.cde zju.edu.cn>. Gang Pan <gpan @ zju.edu.en>.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Hybrid Spiking Vision Transformer for Object Detection with Event Cameras
event representation processing and less power consump- tion. The spike computation in SNNs quite matches the address-event representation of event data. On the other hand, ANNs with convolution and self-attention structures have shown the potential to implement event-based object object
2. Related Work
2.1. Event-based Object Detection Datasets
Several public datasets have been developed for event-based object detection. The Prophesees GENI Automotive Detec- tion Dataset (De Tournemire al., 2020), collected using the Prophesee GENI sensor (304240), provides 39 hours of driving data with annotations for pedestrians and cars. The Megapixel Automotive Detection Dataset (Perot et al., 2020), built on the high-resolution GEN4 sensor (1280720), covers diverse driving scenarios including day/night and urban/highway conditions, addressing challenges such as illumination variation and complex object classes.
detection efficiently. Hence, we focus on detection by considering the property of the event camera and combining the computation of ANNs and SNNs. Mean- while, we collect the Fall DVS object benchmark dataset as one of the event-based object detection.
There have been some previous studies to implement event- based object detection. Graph-based vision Transformer (GET) is proposed in (Peng et al., 2023) with the event repre- sentation of graph token to utilize the temporal and polarity information of events. Graph token clusters asynchronous events based event timestamps and polarities. Together with the Event Dual Self-Attention block and Group Token Aggregation module, GET effectively integrates spatial and cemporal-polarigy features. The muti-stage Recurrent Vision Transformers (RVTs) backbone is designed in (Gehrig & Scaramuzza, 2023) for object detection with event cameras. Each stage incorporates convolution components, local and sparse global attention, and recurrent feature aggregation. The convolution component downsamples the spatial res- olution and acts as the conditional positional embedding for transform layers. The interleaved local and global self- attention captures both local and global features and offers linear complexity in the input resolution. The temporal re- currence is captured by long short-term memory (LSTM) cells (Hochreiter & Schmidhuber, 1997) instead of Conv- LSTM cells (Liu et al., 2017), in order to minimize latency while retaining temporal information. Through the above multi-stage design, RVTs achieve fast interference and favor- able parameter efficiency. Inspired by the architectures of RVTs, we explore multi-stage event-based object detection with hybrid computation based on ANNs and SNNs.
The PKU-DAVIS-SOL dataset (Li et al., 2023) focuses on high-speed motion and extreme lighting conditions, offering long-term multimodal sequences for robust detection. The DSEC Detection Dataset (Gehrig & Scaramuzza, 2024) extends the original DSEC by adding labels for 60 sequences (70379 frames, 3901 18 boxes). These labels, generated by image-based tracking and manual correction, include both bounding boxes and class information and also track identities, for object-tracking applications.
These datasets significantly advance event-based object de- tection in autonomous driving scenarios. Furthermore, due to their inherent privacy- preserving nature, event cameras are well-suited for applications in healthcare and other sen- sitive environments.
2.2. Fall Detection Dataset
Fall detection is a technique that monitors and identifies whether an individual has experienced fall, often employ- ing visual sensors. Primarily focused on providing timely assistance, especially for the elderly or those with health vulnerabilities, the field endeavors to mitigate the serious consequences that fall can entail, including fractures, liga- ment tears, and other injuries. Leveraging a diverse array of sensors and technologies, such as cameras, depth cameras, and infrared sensors, fall detection systems are designed to promptly detect falls and initiate appropriate actions, such as sounding alarms, alerting paramedics or emergency ser- vices, and documenting incidents for further analysis. The application of fall detection technology extends to various domains, including healthcare, home care, and geriatric care.
In this paper, we propose the hybrid spiking vision Trans- former (HsVT) to implement event-based object detection. Firstly, we design multi-stage vision Transformer that integrates ANN and SNN components to leverage the com- plementary strengths of both paradigms. Secondly, SNN components are designed to capture the temporal feature in- formation. The self-attention and convolutional components are employed capture the spatial features. This hybrid paradigm not only enhances temporal feature extraction through biologically spiking mechanisms but also achieves competitive performance. Thirdly, the Fall DVS detection benchmark dataset is collected in this paper as a new bench- mark dataset to evaluate the performance of the proposed HsVT. Evaluated on GENI, HsVT surpasses prior SNN- based approaches and achieves competitive performance of ANN models, while offering lower power consumption and model complexity.
In this context, the graphical representation of output bound- ing boxes plays crucial role in providing users with visual insight into the functioning of fall detection systems, prov- ing particularly beneficial for monitoring systems, smart homes, and healthcare applications. Moreover, the utiliza- tion of bounding boxes enables the concurrent tracking of multiple targets, rendering them suitable for monitoring multiple individuals or analyzing complex environments.
2

---

Hybrid Spiking Vision Transformer for Object Detection with Event Cameras
Fall detection based on real-life datasets is very important for the training and performance evaluation of algorithms. However, privacy concerns can be barrier to data shar- ing and disclosure (Debard et al., 2012). Among different vision sensors, the dynamic vision sensor based on spike computation could protect the facial privacy of individuals and reduce the necessary memory capacity of data storage.
datasets are either fake falls or not publicly available due to privacy concerns. We explore a new approach to fall detection datasets by using event stream data on existing fall datasets. The dataset is publicly available at: our Dropbox repository
3.1. Sample Processing
We convert the frame- based fall detection dataset into event- based fall detection dataset through the event camera simu- lator. Rebecq et al. (Rebecq et al., 2018) proposed ESIM, the first event camera simulator capable of generating large- scale and reliable synthetic event data. Unlike conventional methods that simply threshold the difference between two consecutive frames, their approach relies on tight integra- tion with a rendering engine. Consequently, it dynamically and adaptively queries visual sample frames to accurately generate events. The basic frame-based fall dataset of Le2i fall detection dataset (Charfi et al., 2013) is employed. Each frame of the video is annotated to manually identify body positions using bounding boxes and labels the start and end frames of falls.
2.3. Event-based Object Detection Methods
Early methods for event-based object detection adopt the transformed manner to convert the event data into frame- based sequences and then implement the object detection by the mature object detection models such as YOLOS (Ge, 2021). Although these methods achieve competitive per- formance in specific scenes, they ignore the properties and advantages of event cameras. Recently, some methods have attempted to utilize the asynchronous events directly. By em- ploying the multi-stage backbone with spatial self-attention and convolutional modules and temporal LSTM modules, the recurrent vision Transformer model achieves quite fast interference and favorable parameter efficiency (Gehrig & Scaramuzza, 2023). Group event Transformer is proposed by decoupling the temporal-polarity information from spa- tial information (Peng et al., 2023). The multimodal ob- ject detection method named SODFormer designed to leverage rich temporal cues from the event and frame vi- sual streams (Li et al., 2023). Most of these methods use Transformer- structures capture the spatial and tem- poral features efficiently. To enhance multi-scale object detection with low energy consumption, SpikSSD (Fan et al., 2025) introduces a fully spiking backbone and a bi- directional fusion module tailored for spiking data. Spik- ing (Yu al., 2025) introduces a transformer-Be based SNN detector that retains spatiotemporal features of event data via residual voltage memory and attention mechanisms, achiev- ing strong performance in event-based object detection. In this paper, we consider the hybrid manner by combining the self-attention and convolutional modules to extract spatial features while using the energy-efficieet SNN modules to extract the temporal features.
Hence, we utilize the event camera simulator to convert the video dataset into an event stream dataset, with the generated event data stored in 'bag' format. Each video file produces an event stream file, from which we extracted data content and converted into an 'h5' format file.
3.2. Label Processing
The original dataset comprises a corresponding TXT an- notation file for each video, annotating every frame of the video. The annotations include frame index, the direction of the fall, and Bounding Box (BBox) to specify the po- sition of objects. Bbox representation follows the format (I1, 31, T2, V2), where (æ1, v1) denotes the coordinates of the lop-left corner of the bbox, and (I2,, y2 denotes the coor- dinates of the bottom-right corner of the BBox. Additionally, the frame numbers marking the onset and cessation of a fall event are annotated.
We utilize the data architectural framework of the GENL dataset (De Tournemire et al., 2020) and embrace a numpy schema, wherein boxes encapsulate timestamp i, as well as the Cartesian coordinateszand y. denoting the location, along with the parameters w and h, representing the width and height respectively, of the BBox. Additionally, these boxes incorporate the categorical identifier class id, the confidence measure class_con fidence, and the tracking identifier track_id. In this context, the bounding box is delineated as (a, y, w, h), wherein (I. y) denotes the coor- dinates of the top-left vertex of the BBox,u w signifies the width of the BBox, and h signifies the height of the box.
3. Event-based FALL Detection Benchmark Dataset
Event cameras have significant advantages over frame-based cameras in high dynamic range, low latency, and low power consumption. Event-based object detection is well-suited for applications such as autonomous driving, However, event cameras also have some disadvantages compared to frame-based cameras, such as the lack of grayscale and texture information. Therefore, we consider that the lack of grayscale information precisely protects the privacy of the subject being photographed. Many existing fall detection
3

---

Hybrid Spiking Vision Transformer for Object Detection with Event Cameras
4. Methodology
Here, the labeling frequency is sair 100//z. Hence, we set Afair to be 10ms ensure the sufficient feature capture of dense event data.
4.1. Event Stream Representation
Event cameras operate distinctively from frame-based cam- eras, yielding different data formats. Unlike frame- based cameras that produce static images, event cameras generate event stream data. An event stream comprises sequence of triplets, each representing an event, consisting of times- tamps, pixel coordinates, and polarity information. The interplay between event flow and events, as well as the com- position of events, is illustrated in Fig. as depicted below:
4.2. Design of HsVT
The architecture of the proposed HsVT consists of four blocks, each comprising spatial feature extraction and tem- poral feature extraction components. The network architec- ture is illustrated in Figure 2.
First, the network performs vertical propagation across stages, where each block receives input from the previous stage. Specifically, each block not only accepts input from the previous stage but also incorporates output from the tem- poral feature extraction module at adjacent time steps. This design enables information flow and accumulation across time, allowing the network to better capture temporal de- pendencies. By facilitating information sharing between neighboring time steps, the network enhances its capability to model sequential patterns in the data.
event stream
0
At
time
event: (t, <,,y>,
Figure 1. Event Stream Representation with Time Intervals. Each vertical line represents event occurrence. while equidistant red lines represent the time intervals. Each event is represented by triplet (f, (z, P), denoting its spatial and temporal coordinates.
BLOCKI
BLOCKI
where t represents the timestamp; I and y represent the pixel coordinates; p represents the polarity. The timestamp signifies the time of the event, pixel coordinates denote the event's location, and polarity indicates the event's nature, with +1 for bright events and -1 for dark events. The accu- mulation of events that occur within period At is called sequence E. As shown in (1), data is input into the network in the form of sequence E.
Dommammpl
BROCK?
II OOOO
Squiial
Feaure
Ferertio
BLOCKS
BLOCKI
Terpponl
Fasture
Excoction
RLOCKA
(1) In this study, we utilize three distinct datasets: GEN1, FALL Detection, and AIR Detection datasets. For the GENT dataset, the time interval AGGENT is set to 50ms, aligning with recommendations from reference (Gehrig & Scara- muzza, 2023). For the fall dataset, ALL ALL is set be 200ms based on visualization and experimentation de- tailed in Section 5.1, as illustrated in Fig 2. Finally, for the air dataset, ALR is set to 10ms, taking into account the la- beling frequency of 100Hz. To be specific, the configuration of At in different datasets was as follows:
F {event; event, occurs within AI)
FPX-Head
FPN+Houd
Figure 2. Architecture of the HsVT Network. The architecture of the proposed HsVT network consists of four blocks, each incor- porating spatial feature extraction and temporal feature extraction components.
4.3. Spatial Feature Extraction
In each block, spatial feature extraction is composed of Max ViT (Tu et al., 2022) and SpikingMLP (Spiking Multi- layer Perceptron), as illustrated in Figure 3. These compo- nents work collaboratively to extract spatial representations from the input data, enabling the network to capture both local and global spatial dependencies more effectively. The Max module includes two types of self-attention mecha- nisms:
For the GENI dataset:
AGGEN = 50ms
For the FALL dataset:
fltat = 200ms
(3) Block-SA (Block-level Sell-Attention). Block-SA per- forms self-attention operations within local regions, captur- ing fine- imne-griied spatial correlations and structural patterns. (4) This allows the network model spatial dependencies based
For the AIR dataset:
At Atair 10ms
4

---

Hybrid Spiking Vision Transformer for Object Detection with Event Cameras
on local positional relationships, enhancing its ability
derstand intra-block structures.
Grid-SA (Grid-level Self-Attention). Grid-SA operates across the entire feature map to model long-range dependen- cies and global spatial structures. Unlike Block-SA, which focuses on local features, Grid-SA integrates contextual information over larger spatial extents, enabling comprehen- sive global feature representation.
(a) Block-SA Attention Heatmap
SpikingMLP utilises multiple layers of spiking neurons and nonlinear activation functions to further process the attention-enhance features. By combining outputs from both Block-SA and Grid-SA, SpikingMLP extracts deeper and more abstract spatial features, thus improving the net- work's representational capacity.
(b) Grid-SA Attention Hcatmap
(c) Overlay of Attention Maps and Input Image
To visualise the attention mechanisms, present attention heatmaps generated by Block-SA and Grid-SA, respectively. Additionally, we overlay these heatmaps on the original input images for enhanced interpretability, as shown in Fig- ure 4. The visualisations reveal that Block-SA predomi- nantly focuses on local fine-grained regions, while Grid-SA captures broader global structures. Together, they provide complementary spatial attention patterns.
Figure 4. Visual comparison of attention patterns from Block-SA and Grid-SA. (a) Block-SA captures local fine-grained spatial correlations. (b) Grid-SA highlights broader, long- range dependen- cies. (c) The overlay illustrates how these mechanisms complement each other in spatial understanding.
feature representations from the preceding block and tem- poral feature representations from the current block as its input. The output generated by LSTM subsequently feeds into the subsequent block, thereby enabling the propagation of temporal information throughout the entire network.
Block SA
SpikingMLP
LSTM employs gating mechanisms including input gates, forget gates, and output gates to regulate the flow of in- formation within its units. These gates control the flow of information within LSTM units, enabling them to memorize both short-term and long-term information, thus effectively handling sequential data. In the HsVT model, LSTM dynam- ically captures the temporal dynamics of input sequences, providing richer and more accurate temporal information during the event-based object detection process.
Grid-SA
SpikingMLP
Spatial Feature
Extraction
Figure 3. Spatial Feature Extraction in the H&VT Architecture. This figure illustrates the process of spatial feature extraction within the HsVT architctuure. The components, including Block- SA. Spi Grid-SA, and second SpikingMLP, collabora- tively work to extract spatial features from the input data.
Final Feature Fusion with STFE. In the last block, we introduce the STFE model as novel approach for temporal feature extraction. The STFE model combines the charac- teristics of spiking neurons and LSTM networks to capture temporal features within input sequences. Specifically, the STFE model comprises spiking neurons, convolutional lay- ers, batch normalization layers, and a time feature extraction section similar to the traditional LSTM structure.
4.4. Temporal Feature Extraction
In this section, we describe the utilization of LSTM and Spiking Temporal Feature Extraction (STFE) models for temporal feature extraction within the HsVT architecture, as illustrated in Figure 5.
In the forward propagation process. the input data ex- tracted through the convolutional layer, and then normalized through the batch normalization layer to enhance the sta- bility of the network. Subsequently, the normalized feature representation is fed into the spike neuron to generate a time-spike signal. Then, these spiking signals are trans- mitted to the STEE module for time feature extraction and propagation, order to provide richer and more accurate
LSTM-Based Temporal Encoding in Initial Blocks. In this study, LSTM serves as a crucial element for capturing both temporal dependencies and long-term dependencies within input sequences across the first three blocks. Con- cretely, LSTM is integrated into the architecture each of the first three blocks. Here, it receives a fusion of spatial
5

---

Hybrid Spiking Vision Transformer for Object Detection with Event Cameras
5.1. Datasets
LSEM (BLOCKI)
The above experimental evaluations are based on three Event-Based datasets, which include those dedicated to air- craft detection, fall detection, and GENI Dataset.
x
(BLOCK2)
Alh
Corrtv
miix
The Aircraft Detection Dataset. The The Aircraft De- tection Dataset consist of aircraft flighting event streams collected by DVS camera. The flighting event streams con- tain four different kinds of aireraft models: F1, F2, F3, and F4. This dataset is available the public for security reasons. Figure 6 visualizes the flighting event streams of two aircraft.
Corr
ISTM (BLOCK3)
STFE
(c.h)
Temponi Extraction
Figure 5. Spiking Temporal Feature Extraction the HsVT Archi- tecture. This figure illustrates the process of temporal feature ex- traction within the HsVT architecture, where the first three blocks employ the LSTM model and the final block utilizes the STFE model. This architecture integrates these models to capture the temporal dependencies in event data, facilitating the extraction of rich temporal features crucial for event-based object detection tasks.
Prophesee GEN1 Automotive Detection Dataset. As de- scribed in section 2.1, 1, the GEN1 dataset is collected the autonomous driving tasks using event cameras, to detect two object categories of pedestrians and cars.
The Fall Detection Dataset. As described in section 2.3, The Fall Detection Dataset is generated with the event simu- lator by converting the original fall video dataset into event stream data. This dataset contains two different situations of fall or not fall behavior, the object detection model should predict the bounding box of humans and recognize whether fall or not.
time information for the object detection task.
In summary, we design the application of LSTM and STFE modules in temporal feature extraction and illustrate their integration and extraction of temporal information within the network. The adoption of these methods enhances the ability of the proposed model of HsVT for robust temporal feature extraction, thereby improving the performance and effectiveness of object detection tasks.
To investigate the impact of event accumulation intervals on detection performance, we visualised reconstructed frames at three temporal resolutions: 40, 200 and 1000 ms, as shown in Figure 7. Each frame is generated by integrating events over the respective duration. As illustrated in Fig- ure 7(a), shorter interval of 40ms preserves finer motion details but results in sparser frames, which may limit the ability perceive complete human contours. Conversely, Figure 7(c) with a 1000ms interval produces more cluttered representations due to excessive temporal integration, po- tentially leading to motion blur and ambiguity. The 200ms setting, shown Figure 7(b), strikes a balance between temporal resolution and event density, offering clearer fall motion patterns. Quantitatively, as reported in Table the model achieves the highest detection performance 200ms, with a mean Average Precision (mAP) of 0.487.
5. Experiments
We evaluate the proposed HsVT model through series of experiments. First, we describe the datasets and experimen- tal setup. Second, we perform ablation studies to validate the effectiveness of the proposed components. Third, we compare the performance of HsVT with other methods on multiple datasets. Following RVT (Table 1), we adopt the same parameters and architectural changes. Specifically, "Channels' denotes the number of channels in each block, 'Size' represents the feature map size, 'Krrne'' indicates the convolution kernel size, and 'Stride' defines the convolution stride.
Table 2. Different time intervals of Fall datasets. Time Interval(ms)
mAA
40
0.445
200
0.487
Table Parameters and Architectural Changes.
1000
0.405
Channels Small
Block
Size
Kernel
Stride
Tiny
Base
B1
1/4
7
32
48
64
5.2. Experimental Settings
B2
1/8
3
2
64
96
128
We employ series of optimization methods and training techniques to ensure the training effectiveness and perfor- mance of the model. Specifically, we used the RVT model combined with the YOLOX framework and maintained the
B3
1/16
3
128
192
256
B4
1/32
3
256
384
512
6

---

Hybrid Spiking Vision Transformer for Object Detection with Event Cameras
Figure Visualization of Aircraft Detection Dataset.
employ tiny model as the backbone of HsVT.
Comparison of Spiking Neurons and Surrogate Func- tions within SpikingMLP. As shown in Table 3, the per- formances of HsVT model with different spiking neuron models in the SpikingMLP component are presented. We found that the HsVT with LIF neurons performs better than IF neuron model on the AIR and FALL detection datasets. This suggests the advantage of LIF utilize the nonlinear property capture the spatiotemporal features in HsVT model. These findings hold practical significance for guid- ing the following model design and optimization processes. Hence, we employ the LIF model in the following experi- ments.
(a) 40ms Integration
(b) 200ms Integration
1000s Integration
Figure 7. Event Frame Visualizations at Different Temporal Inte- gration Intervals.
Table 3. The performance of SpikingMLP with different spike neu-
m.AP 50:95
consistency of the following experimental Settings:
SpikingMLP
FALL
AIR
LIFNode
0.476
0.630
Optimizers and Learning Rate Scheduling Strategies. We adopted the popular ADAM optimizer (Kingma & Ba, 2014) and utilized the OneCycle learning rate scheduling strategy (Smith & Topin, 2019). This strategy starts from the maximum learning rate and linearly decays during train- ing, effectively accelerating the training speed of neural networks.
IFNode
0.441
0.587
We further investigate the effects of two surrogate gradient funetions, ATan and Sigmoid, on the FALL and Aircraft Detection Dataset (Table 4). These functions replace the non-differentiable components of the LIF activation func- tion, facilitating backpropagation in SNNs.
Mixed Precision Training (Micikevicius al., 2017). speed up training time and reduce memory usage without sacrificing model accuracy, utilize mixed precision train- ing techniques. This approach enhances training efficiency by simultaneously using low-precision and high-precision floating point numbers, particularly suitable for handling large-scale data problems.
Table 4. The performance of Fall dataset and Air dataset with dif- ferent Surrogate gradient functions.
Surrogate function FAO00()
m.A.sus9 5u:95
FALIm(000ms)
AIR
ATan
0.476
0.490
0.630
Sigmoid 0.482
0.473
0.603
We train HsVT on two NVIDIA GeForee RTX 4090 GPUs, using batch size of for the tiny model, and batch sizes of 4 for the basic and small model. Through these experimental settings, train the proposed HsVT model and record the mAP value on different event-based object detection tasks.
Impact of Different SNN Components. To explore the effectiveness of different SNN components, we evaluate sev- eral alternatives based on their parameters, computational complexity (FLOPs), and performance metrics, particularly mAP.
5.3. Ablation Study
The considered SNN components and their corresponding characteristics summarized in Tab. 5. Among the eval- uated components, CoonennNoee(STT preferred choice due to its optimal balance between param- eter efficiency and computational overhead. The combina- tion of STEE with LIFNode neurons achieves a competitive mAP score of 0.64 with parameter counts 0.20 and 101.19
To validate the effectiveness of the proposed HsVT model, we conduct the ablation study to study the effect of spiking network modules with different settings in HsV The exper- iments aim to reveal the contribution of different factors of HsVT and provide important references for optimizing and improving our model. For convenience, all ablation studies
is adopted as the

---

Hybrid Spiking Vision Transformer for Object Detection with Event Cameras
Table 5. The performance of different SNN components on the
Table 6.
Ablation study the Aircraft Detection Dataset.
Aircraft Detection Dataset,
BLOCK
BLOCK
BLOCK 3
BLOCK 4
m.AP30:95
SNN Component Params(M)
FLOPs(M)
Neuron
m.APom-:0
LSTM
LSTM
LSTM
LSTM
0.595
STFE
0.20
101.19
IFNode
0.595
STFE
LSTM
LSTM
LSTM
0.599
LIFNode
0.640
LSTM
STFE
LSTM
LSTM
0.610
IFNode
0.594
PlainNet
0.13
67.11
1STM
LSTM
STRE
LSTM
0.609
LIFNode
0.614
LSTM
LSTM
LSTM
STHE
0.640
IFNode
0.607
FeedBackNet
0.13
134.22
LIFNode
0,593
LSTM
LSTM
STFE
STFE
0.621
IFNode
0.605
1STM
STFE
STFE
STFE
0.602
StatefulSynapse
0.13
67.11
LIFNode
0.604
STFR
STFE
STFE
STHE
0.602
ISTM
0.53
268.44
0.604
Among SNN-based methods, our proposed HsVT achieves 0.478, outperforming all previous SNN, including SpikSSD (0.408), EAS-SNN (0.409), and SFOD (0.321). Although RVT (RNNS + Transformer) achieves strong performance (0.472), HsVT-B exceeds it while maintaining a smaller parameter size compared to some other large models.
FLOPs, respectively. While other alternatives such as Plain- Net, FeedBackNet, and StatefulSynasseNet also present promising performance, STFE stands out for its superior parameter efficiency and computational efficacy.
These findings underscore the importance of selecting SNN components judiciously to strike the right balance between model complexity, computational cost, and performance metrics, ultimately contributing to the effectiveness and efficiency of the overall HsVT architecture.
As shown in Tab. 8, we further compare HsVT with RVT the FALL dataset, as RVT represents a strong non-SNN hybrid baseline. Across all model variants (tiny, small, base), HsVT consistently outperforms RVT on both AIR and FALL categories, demonstrating superior generalization in event-based fall detection. Interestingly, we observe that increasing the model size on the FALL and Aircraft Detec- tion datasets does not always yield performance gains, and in some cases slightly degrades accuracy. We speculate that this may be due to the limited size and diversity of these datasets, which can lead to overfitting in larger models and hinder their generalization.
Impact of SNN Component Placement in Network Ar- chitecture. Tab. 6 presents the results of ablation studies of SNN blocks with different configurations in the network structure, focusing on the influence of SNN component placement. Each row represents a different arrangement of the SNN components in the four blocks of the network structure. The mAP column shows the average accuracy of the Air dataset. The results highlight the variation in the performance of STFE components at their location in the network structure, with mAP scoring highest when STFE components are placed entirely in the fourth block.
each configuration
6. Discussion and Conclusion
In this study, we propose novel hybrid model, HsVT, which leverages the strengths of ANN and SNN for event- based object detection. We evaluated the model on the GENI, FALL,, and Aircraft Detection Dataset, demonstrat- ing its effectiveness and efficiency.
These results highlight the importance of placing SNN com- ponents in network architectures. Specifically, integrating STFE components into later modules tends to result in better performance, which is reflected in higher mAP scores. Inter- estingly, configurations with multiple STFE modules also exhibit competitive performance, indicating the potential advantages of utilizing time feature extraction at multiple stages of information processing. Taken together, these find- ings provide valuable insights help optimize event-based tasks and improve SNN-based architectures.
For the GENI dataset, the Hs model achieves compara- ble performance the pure SNN model in terms of mAP, while requiring fewer parameters. This highlights its ad- vantage in parameter efficiency and capability for accurate object detection. On the FALL Detection and AIR Detec- tion datasets, the performance of the Hs model does not strictly follow the expected Base >> SSall > Tiny pattern. We attribute this to dataset size limitations, which may lead to overfitting in larger models. Despite this, the HSVT model consistently outperforms the RVT model on both datasets, further validating its robustness and adaptability.
5.4. Comparison with the State-of-the-ard
In this section, we compare our proposed HsVT model with wide range of state-of-the -ft methods, covering different network types including CNNs, RNNs, GNNs, Transform- ers, SNNs, and hybrid models. Tab. 7 presents the per- formance comparison on the GEN1 dataset. Transformer- based methods such as ERGO-12 and STAT achieve the highest m. A.APss 50:95 scores of 0.504 and 0.499, respectively.
In summary, our findings underscore the effectiveness and competitiveness of the HsVT model in event-based object detection. By integrating the complementary strengths of ANN and SNN, the model achieves high performance while
8

---

Hybrid Spiking Vision Transformer for Object Detection with Event Cameras
Table 7. Performance comparison with the state-of-the-art methods GENT dataset.
Model
Network Type
Params(M)
m.AP50:99
Inception+SSDD (lacono et al., 2018)
CNNS
0.301
Asynet (Messikommer et al., 2020)
GNNS
11.4
0.145
MatMixLSTM (Cannici et al., 2020)
RNNs+CNNS
61.5
0.310
RED (Perot et al., 2020)
RNNs+CNNs
24.1
0.400
AEGNN (Schaefer et al., 2022)
GNNs
20.0
0.163
ASTMNel (Li al., 2022)
RNNs+CNNs
>100
0.467
RVT (Gehrig & Scaramuzza, 2023)
RNNs+ Transforaorrmer
18.5
0.472
TAF (Liu et al., 2023)
CNNs
14.8
0.454
ERGO-12 (Zubić et al., 2023)
Transformer
0.504
STAT (Guo et al., 2024)
Transformer
0.499
MobileNet-64+4SDD (Cordone et al., 2022)
SNNs
24.3
0.147
VGG-11+SSD (Cordone et al., 2022)
SNNs
12.6
0.174
DenseNet121-24-SSD (Cordone et al., 2022)
SNNs
8.2
0.189
LT-SNN (Hasssan et al., 2023)
SNNs
0.298
EMS-ResNet34 (Su et al., 2023)
SNNs
14.4
0.310
SFOD (Fan et al., 2024)
SNNs
11.9
0.321
EAS-SNN (Wang et al., 2024)
SNNs
25.3
0.409
SpikeFPN (Zhang et al., 2024)
SNNs
22.0
0.223
KD-SNN (Bodden et al., 2024)
SNNs
12.97
0.229
SpikSSD (Fan et al., 2025)
SNNs
19.0
0.408
SpikingViT (Yu et al., 2025)
SpikingTTransformer
21.5
0.394
HsVT-T(ours)
RNN=+SNNs+Transtiom
4.1
0.449
Hs (T-S(ur))))
RNNs+SNNs+ Transformerr
9.1
0.465
Hs
RNNs+SNNs+TTrasfoome
17.2
0.478
(N.BBM12400012), the Young Elite Scientists Sponsorship Program by CAST under Grant 2024QNRC001 and Xi an Association for Science and Technology Youth Talent Sup- port Program No.0959202513337.
Table 8. The performance comparison on Fall dataset.
Method
Variant
AIR
FALL
tiny
0.613
0.487
RVT
small
0.600
0.466
base
0.604
0.469
Impact Statement
tiny 0.641
0.491
H3VT
small
0.616
0.492
This work proposes an energy-efficient and hybrid spiking vision Transformer for event-based object detection. By leveraging the strengths of SNNs and Transformer archi- tectures, advances low- low-power spatiotemporal processing. The release of fall detection dataset further supports research and reproducibility in event-based vision.
base
0.618
0.486
maintaining parameter efficiency. This work provides valu- able information on the roles of ANN and SNN in object detection and serves as an important reference for future research in this field.
References
Acknowledgements
Beal, J., Kim, E., Tzeng, E., Park, D. H., Zhai, A., and Kislyuk, D. Toward transformer-bass. object detection. arXiv preprint arXi::2012.09958 2020.
This work was supported by the National Natural Sci- ence Foundation of China under Grants (No.62306274, 62476035, 62206037, U24B20140, 61925603), Open Research Program of the National Key Laboratory of Brain-Machine Intelligence, Zhejiang University
Bodden, L.,, Ha, D. B., Schwaiger, F., Kreuzberg, L., and Behnke, Spiking centernet A distillation-boose spik- ing neural network for object detection. In 2024 Interna-

---

Hybrid Spiking Vision Transformer for Object Detection with Event Cameras
lional Joint Conference on Neural Networks (IJCNN), pp. 1-9. IEEE, 2024.
Guo, Y., Peng, W., Chen, Y., Zhang, L., Liu, X., Huang. X., and Ma, Z. Joint a-snn: Joint training of artificial and spiking neural networks via self-distillation and weight factorization. Pattern Recognition, 142: 109639, 2023.
Cannici, M., Ciccone, M., Romanoni, A., and Matteucei, M. A differentiable recurrent surface for asynchronous event-bor beent-baed data. In Computer Vision-ECCY 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX 16, pp. 152-15.. Springer, 2020.
Guo, Z., Gao, J., Ma, G., and Xu, J. Spatio-temporal ag- gregation transformer for object detection with neuromor- phic vision sensors. IEEE Sensors Journal, 2024.
Hasssan, A., Meng, J., and Seo, J.-s. Lt-snn: Self- adaptive spiking neural network for event-based classification and object detection. 2023.
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229. Springer, 2020.
Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural computation, 9(8): (83):1735-1785 1997.
Charfi, I., Miteran, J., Dubois, J., Atri, M., and Tourki, R. Optimised spatio-temporal descriptors for real-time fall detection: comparison of svm and adaboost based classification. Journal of Electronic Imaging (JEI), 22(4): 17, 2013.
Hu, Y., Tang, H., and Pan, G. Spiking deep residual net- works. IEEE transactions on neural networks and learn- ing systems, 34(8):5200-522 -2000 2023a.
Hu, Y., Zheng, Q., Jiang, X., and Pan, G. Fast-snn: Fast spik- ing neural network by converting quantized ann. IEEE Transactions Pattern Analysis and Machine Intelli- gence, 45(12))14546-145555 2023b.
Cordone, L.,, Miramond, B., and Thierion, P. Object detec- tion with spiking neural networks on automotive event data. In 2022 International Joint Conference Neural Networks (IJCNN, pp. 1-8. IEEE, 2022.
Iacono, M., Weber, S., Glover, A., and Bartolozzi, C. To- wards event-driven object detection with off-the-shelf deep learning. In 2018 IEEE/RSJ International Confer- ence on Intelligent Robots and Systems (IROS), pp. 1-9. IEEE, 2018
De Tournemire, P., Nitti, D., Perot, E., Migliore, D., and Sironi, A. large scale event-based detection dataset for automotive. arXiv preprint arXiv:2001 200,099900 2020.
Debard, G., Karsmakers, P., Deschodt, M., Vlaeyen, E., De- jaeger, E., Milisen, K., Goedemé, T., Vanrumste, B., and Tuytelaars, T. Camera-based fall detection on real world data. In Outdoor and Large-Scale WerldWorld Scene Analysis: 15th International Workshop Theoretical Foundations of Computer Vision, Dugsiuhl Castle, Ger- many, June 26-July 1, 2011. Revised Selected Papers, pp. 356-375. Springer, 2012.
Kingma, D. and Ba, J. Adam: A method for stochastic optimization. arXiv: Learning.arXiv Learning, Dec 2014.
Li, D., Li, and Tian, Y. Sodformer: Streaming object de- tection with transformer using events and frames. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 2023.
Fan, Y., Zhang, W., Liu, C., Li, M., and Lu, W. Sfod: Spiking fusion object detector. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17191-17200, 2024.
Li, J., Li, J., Zhu, L., Xiang, X., Huang, T., and Tian, Y. Asynchronous spatio-temporal memory network for con- tinuous event-based object detection. IEEE Transactions Image Processing, 31:2975-29881 2022.
Fan, Y., Liu, C., Li, M., and Zhang, W. Spikssd: Better traction and fusion for object detection with spiking neu- ron networks. arXiv preprint arXiv:2501.1515/ 2025.
Liu, B., Xu, C., Yang, W., Yu, H., and Yu, L. Motion robust high-speed liggt-weighted object detection with event camera. IEEE Transactions on Instrumentation and Measurement, 72:1-13, 2023.
Ge, Z. Yolox:
Exceeding yolo series in 2021. arXiv preprint arXiv:2107.084300 2021.
Liu, Y., Zheng, H., Feng, X., and Chen, Z. Short-term traffic flow prediction with conv-Istm. In 2017 9th in- ternational conference on wireless communications and signal processing (WCSP), pp. 1-6. IEEE, 2017.
Gehrig, D. and Scaramuzza, D. Low-latency automotive vision with event cameras. Nature, (::4 2024.
Liu, Z., Guan, B., Shang, Y., Liang, S., Yu, Z., and Yu, Q. Optical flow-guided 6dof object pose tracking with an event camera. In Proceedings of the 32nd ACM In- ternational Conference on Multimedia, pp. 2024a.
Gehrig, M. and Scaramuzza, D. Recurrent vision transform- ers for object detection with event cameras. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13884,13893, 2023.
10

---

Hybrid Spiking Vision Transformer for Object Detection with Event Cameras
Liu, Z., Guan, B., Shang, Y., Yu, Q., and Kneip, L. Line- based 6-dof object pose estimation and tracking with event camera. IEEE Transactions on Image Processing, 2024b.
Shen, J., Xu, Q., Pan, G., and Chen, B. Improving the sparse structure learning of spiking neural networks from the view of compression efficiency, arXiv preprint arXiv:2502. 2025.
Liu, Z., Guan, B., Shang, Y., Bian, Y., Sun, P., and Yu, Q. Stereo event-based, 6-dof pose tracking for uncooperative spacecraft. IEEE Transactions on Geoscience and Remote Sensing, 2025.
Smith, L. and Topin, N. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial Intelligence and Machine Learning for Multi- Domain Operations Applications, May 2019. doi: 10. 11 1117/1..552088. URL /tttp://dx... doi org/10. 1 111712..22055
Ma, D., Jin, X., Sun, S., Li, Y., Wu, X., Hu, Y., Yang. F., Tang, H., Zhu, X., Lin, P., et al. Darwin3: large-scale neuromorphic chip with novel isa and on-chip learning. National Science Review, (1(5): 2024.
Su, Q. Chou, Y., Hu, Y., J., Mei, S., Zhang, Z., and Li, G. Deep directly-trained spiking neural networks for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
Messikommer, N., Gehrig, D., Loquercio, A., and Scara- muzza, D. Event-based asynchronous sparse convolu- tional networks. In European Conference Computer Vision, pp. $15-431.30 Springer, 2020.
2023.
Tu, Z., Talebi, H., Zhang, H., Yang, F., Milanfar, P., Bovik, A., and Li, Y. Maxvit: Multi-axis vision transformer. In European conference on computer vision, pp. 459-479. Springer, 2022.
Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaiev, O., Venkatesh, G., et al. Mixed precision training. arXiv 2017.
Wang, Z., Wang, Z., Li, H., Qin, L.., Jiang, R., Ma, D., and Tang, H. Eas-snn: End-to-end adaptive sampling and representation for event-based detection with recurrent spiking neural networks. In European Conference on Computer Vision, pp. 310-328. Springer, 2024.
preprint arXiv:
Peng, Y., Zhang, Y., Xiong, Z., Sun, X., and Wu, F. Get: group event transformer for event-based vision. Pro- ceedings of the IEEE/CVF International Conference Computer Vision, pp. 6038-6048, 2023.
Xiao, Y., Liu, Y., Zhang, B., Chen, P., Zhu, H., He, E., Zhao, J., Huo, W., Jin, X., Zhang, X., et al. Bio- bio-plausible recon- figurable spiking neuron for neuromorphic computing. Science Advances, 11(6)cadr67333 2025.
Perot, E., De Tournemire, P., Nitti, D., Masci, J., and Sironi, A. Learning to detect objects with 1 megapixel event camera. Advances in Neural Information Processing Systems, 33:16639-16653 2020.
Xu, Q., Gao, Y., Shen, J., Li, Y., Ran, X., Tang, H., and Pan, G. Enhancing adaptive history reserving by spiking convolutional block attention module in recurrent neural networks. Advances in Neural Information Processing 2023.
Posch, C. A biomimetic frame-free event-driven image sensor. Focal-Plan Sensor-Processo Chips, pp. 125- 149, 2011.
Systems,
Rebecq, H., Gehrig, D., and Scaramuzza, D. ESIM: an open event camera simulator. Conf. on Robotics Learning (CoRL), October 2018
Xu, Q., Li, Y., Fang. X., Shen, J., Zhang, Q. and Pan, G. Reversing structural pattern learning with biologically inspired knowledge distillation for spiking neural net- works. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 3431-3439, 2024a.
Schaefer, S., Gehrig, D., and Scaramuzza, D. Aegnn: Asyn- chronous event-based graph neural networks. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12371 2022.
Xu, Q., Liu, S., Ran, X., Li, Y., Shen, J., Tang, H., Liu, J. K., Pan, G., and Zhang, Q. Robust sensory information reconstruction and classification with augmented spikes. IEEE Transactions on Neural Networks and Learning Systems, 2024b.
Gerrano-Gotaaredoonaa T. and 128 1.5% contrast sensitivity 0.9% fpn 3 μs latency 4 mw asynchronous frame-free dynamic vision sensor using transimpedance preamplifiers. IEFE Journal of Solid- State Circuits, 48(3):827-833 2013.
B. A 128 x
Yu, L., Chen, H., Wang, Z., Zhan, S., Shao, J., Liu, Q. and Xu, S. Spikingvit: A multiscale spiking vi- sion transformer model for event-based object detection. IEEE Transactions on Cognitive and Developmental Sys- tems, 17(1):130-146, 2025. doi: 10.1109/TCDS.200 2024.. 3422873.
Shen, J., Ni, W., Xu, Q. and Tang, H. Efficient spiking neural networks with sparse selective activation for con- tinual learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 61 1699, 2024.

---

Hybrid Spiking Vision Transformer for Object Detection with Event Cameras
Zhang, H., Li, Y., Leng, L., Che, K., Liu, Q.. Guo, Q.. Liao, J., and Cheng, R. Automotive object detection via learning sparse events by spiking neurons. IEEE Transactions on Cognitive and Developmental Systems, 2024.
Zhou, Z., Zhu, Y., He, C., Wang, Y., Yan, S., Tian, Y., and Yuan, L. Spikformer: When spiking neural network meets transformer. arXiv preprint arXiv:2209.15444 2022.
Zubić, N., Gehrig, D., Gehrig, M., and Scaramuzza, D. From chaos comes order: Ordering event representations for object recognition and detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12846-12856, 2023.
12

---

Hybrid Spiking Vision Transformer for Object Detection with Event Cameras
A. Theoretical Energy Estimation
Following the methodology described in (Zhou al., 2022), we estimate the theoretical energy consumption of each HsVT under 45nm process technology. For the ANN components, the energy estimated as:
ANN 4.6 x FLOPS
(5)
For the SNN components, the estimation incorporates spike sparsity by introducing Spike Operations (SOPs):
EsNN = PJ x SOPs = 0.9pJ x fr x x FLOPs
(6)
where fr denotes the average firing rate and the total number of timesteps. The total energy consumption of HsVT is then given by:
E = EANN + ESNN
(7)
The experimental results of Hs Vsbbcccce on the GENI dataset are shown in Table 9.
Table 9. Energy Consumption Analysis of the HsVT Backbone.
Hs Hs-bccacko
FLOPs(M)
(mN
SOP(M)
(mJ)(m))
EHvyT(m)
Tiny
4199.59
19.32
1156.00
0.017
19.34
Small
8485.64
39.03
2413.10
2.172
41.20
Base
14229.15
65.45
3771.60
3.394
68.84
To analyze the energy distribution within the HsVT-based detection framework, we divide the network into three key components: the backbone, the feature pyramid network (FPN), and the detection head. We compute the theoretical energy consumption for each component individually. The results are presented Table 10.
Table Theoretical energy consumption (mJ) different network components on the GEN1 dataset.
Module
Tiny
Small
Base
Backbone
19.34
41.20
68.84
FPN+Head
14.57
32.64
65.66
Backbone +ePN+Head
33.91
73.84
134.50
We present the theoretical energy consumption of HsVT-B model and compare it with other SNN-based detectors, including SFOD and EAS-SNN-M. As summarized in Table 11, HsVT-B demonstrates a total estimated energy of 134.5 mJ, which is higher than SFOD and EAS-SNN-M This increase is primarily attributed to the hybrid ANN-SNN structure, which integrates high-capacity ANN modules. In particular, the ANN backbone contributes 68. 84 mJ, and the FPN+Head modules add 65.66 mJ. Despite the increased energy cost, the hybrid architecture achieves superior detection performance.
Table 11. Theoretical energy consumption comparison (mJ) on GENI dataset.
Module
HsVT-B
SFOD (Fan et al., 2024) EAS-SNN-M (Wang et al., 2024)
Architecture Type
ANN-SNN
SNN
SNN
Energy (Backbone)
68.84
7.5mmend)) + 14.12
Energy (FPN + Head)
65.66
6.46
Energy (Total)
134.5
7.26
28.10
13


Reviews:
[{"review_id": "2XA5DDepTy", "reviewer": "Reviewer_Ho9N", "summary": "This paper introduces a Hybrid Spiking Vision System, combining Spiking Neural Networks (SNNs) with deep learning architectures to enhance computational efficiency and reduce power consumption. The study explores the effectiveness of this hybrid approach in visual tasks and presents experimental results demonstrating its advantages in inference efficiency.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Q1: Some theoretical justifications\nRe: We have implemented a hybrid SNN + LSTM structure, where Leaky Integrate-and-Fire (LIF) neurons process temporal information, while convolutional layers and Batch Normalization are used for spatial feature extraction.\n\nThe input x is concatenated with the previous hidden state $h_{tml}$ and passed through a 1×1 convolution to reduce the channel dimension:$X_{H}=\\mathrm{concat}(x,h_{\\mathrm{tml}})$ , $M=W_{m}*X_{H}+b_{m}$\n\nThe processed input then undergoes a convolutional layer followed by Batch Normalization:\n$B=W_{c}*M+b_{c}$ , $C=\\gamma\\frac{B-\\mu_{B}}{\\sigma_{B}}+\\beta\\$\n\nThe output from the convolutional layer is fed into LIF neurons, where the membrane potential evolves over time:$\\tau\\frac{dV(t)}{dt}=-V(t)+C$\n\nWhen the membrane potential $V(t)$ exceeds the threshold $V_{th}$, the neuron fires: if $V(t)\\geq V_{\\mathrm{th}}$, $H_t=1$; if $V(t)\\mathrm{<} V_{\\mathrm{th}}$, $H_t=1$;\n\nAfter firing, the membrane potential resets: $V(t)=V(t)-V_{\\mathrm{th}}$\n\nThe membrane potential V at the current time step acts as the cell state, while the previous step’s state $C_{tml}$ contributes to temporal dynamics:$C_{t}=V+C_{\\mathrm{tml}}$\n\n---\n# Q2: Ablation studies could be extended to analyze different types of hybridization.\nRe: We conducted ablation studies to analyze different types of hybridization by testing various SNN variants:\n- Different spiking neurons (Tab3 Tab5):We compared IF and LIF neurons to assess their influence on temporal encoding. The results show that LIF neurons achieve better accuracy due to better robustness and better generalization.\n- Different surrogate gradient functions (Tab4):We experimented with ATan and Sigmoid surrogate gradients. Our findings indicate that the ATan function provides a better accuracy.\n- Different SNN components (Tab5):We replaced key SNN modules to evaluate their performance. The results confirm that the Conv+BN+LIF module works best.\n- Different placements of SNN modules (Tab6):We choose to replace LSTM with STFE at different locations and replace different amounts of LSTM. We choose the method with the highest mAP value.]\n\n---\n# Q3: More discussion on hardware feasibility and real-world deployment would be useful.\nRe:The Tianjic neuromorphic chip supports both ANN and SNN processing. however, it does not support attention mechanisms, making it challenging to deploy our model directly on such hardware.\nDespite this challenge, our event-camera-based fall detection dataset has significant real-world applicability. Many existing fall detection datasets rely on conventional cameras, but due to privacy concerns, they are often not publicly available, limiting progress in this field. In contrast, event cameras capture only motion-triggered data, inherently protecting patient privacy.\n\n---\n# Q4: How does the hybrid model compare to recent advances in neuromorphic vision systems?\nRe:Recent event-based detection models include SFOD (Fan et al., 2024), which fuses multi-scale features within SNNs, and EAS-SNN (Wang et al., 2024), which employs adaptive sampling and recurrent SNNs. While both achieve notable performance, our hybrid model outperforms them:\n| Model|mAP@0.5:0.95|\n|---|---|\n|SFOD| 0.321|\n|EAS-SNN|0.354|\n|Ours |0.478|\n\nBy integrating ANN-based spatial and SNN-based temporal processing, our model achieves superior spatial-temporal fusion.  These results will be included in our manuscript.\n\nReferences:  \n- Fan Y, et al. SFOD: Spiking Fusion Object Detector. CVPR 2024.  \n- Wang Z, et al. EAS-SNN: End-to-End Adaptive Sampling and Representation for Event-Based Detection. ECCV 2024.  \n\n---\n# Q5: What are the trade-offs between different configurations of hybrid models\nRe:  The balance between SNN and ANN components impacts accuracy, efficiency, and power consumption.  Key trade-offs include:\n\n(1) More ANN layers enhance spatial feature extraction and accuracy but increase computational cost, while more SNN layers improve temporal processing and energy efficiency but risk gradient vanishing and lower feature expressiveness.\n(2) Event-driven SNN computation reduces power usage but may introduce processing latency due to spike accumulation.\n(3) Our experiments (Tab 6) show that sufficient ANN depth ensures feature extraction, while well-placed SNN modules capture temporal dependencies with minimal accuracy loss.\n\n---\n# Q6: Have you considered real-world deployment scenarios, and how would the model adapt to edge computing environments?\nRe: Our work focuses on event-camera-based fall detection, a practical real-world application.  Event cameras offer low latency, high dynamic range, and energy efficiency.\n\n(1) Efficiency: Leveraging SNNs, our model reduces redundant computations and triggers processing only on motion detection, enhancing energy efficiency.\n(2) Robustness: Event cameras ensure reliable performance across varied lighting and backgrounds, making them ideal for deployment.", "claims_and_evidence": "++Empirical results on multiple benchmark datasets, demonstrate that HSVT maintains comparable accuracy while requiring significantly fewer operations. The hybrid model effectively leverages spiking neurons to reduce redundancy and improve efficiency.\n\n++Ablation studies show that selectively replacing ViT components with SNN modules leads to computational savings while preserving feature representation quality.\nTheoretical justification supports the effectiveness of hybridization in spatiotemporal processing.\n\n++The paper presents an analysis of the role of spike-based representations in attention mechanisms and their potential benefits in terms of sparsity and latency.", "methods_and_evaluation_criteria": "Metrics such as accuracy, energy consumption, and latency are considered.\n\nComparisons with both fully spiking and deep networks provide insights into trade-offs.", "theoretical_claims": "The paper argues that integrating SNNs with deep learning enhances computational efficiency without significantly sacrificing accuracy.\n\nIt theoretically justifies why hybrid architectures can outperform purely spiking or non-spiking networks in real-world applications.", "experimental_designs_or_analyses": "The experiments include training and testing on vision benchmarks.\n\nAblation studies evaluate different configurations of the hybrid model.\n\nPerformance is analyzed in terms of accuracy, power efficiency, and computational cost.\n\nThe choice of spiking neuron models and their integration into deep learning pipelines is examined.", "relation_to_broader_scientific_literature": "The paper is relevant to research in neuromorphic computing, efficient deep learning, and hybrid neural architectures.\n\nIt builds upon previous work in SNNs, biologically inspired computing, and deep learning acceleration.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strengths:\n\n++Novel hybrid approach leveraging both deep learning and SNN advantages.\n\n++Well-motivated discussion on energy efficiency.\n\n++Clear experimental setup and benchmarking.\n\nWeaknesses:\n\n--Some theoretical justifications could be made stronger.\n\n--Ablation studies could be extended to analyze different types of hybridization.\n\n--More discussion on hardware feasibility and real-world deployment would be useful.", "other_comments_or_suggestions": "None.", "questions_for_authors": "I am not an expert in this field, I hope the authors can answer the following questions:\n\n1. How does the hybrid model compare to recent advances in neuromorphic vision systems?\n\n2. What are the trade-offs between different configurations of hybrid models (e.g., varying the proportion of SNN vs. deep learning components)?\n\n3. Have you considered real-world deployment scenarios, and how would the model adapt to edge computing environments?\n\nDepending on the author's answer results, I may raise or lower my rating.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hYfkKkUZmq", "reviewer": "Reviewer_zi8P", "summary": "This paper introduces Hybrid Spiking Vision Transformer (HsVT), a novel architecture combining Artificial Neural Networks (ANNs) and Spiking Neural Networks (SNNs) for event-based object detection. The key contributions include: A hybrid spatial-temporal framework integrating ANN-based self-attention modules (e.g., Block-SA, Grid-SA) and SNN-based temporal feature extractors (e.g., SpikingMLP, STFE) to capture both local/global spatial features and long-term temporal dependencies. The creation of the Fall Detection dataset, a privacy-preserving event-based dataset generated via an event camera simulator, addressing gaps in fall detection benchmarks. Comprehensive experiments demonstrating HsVT’s superiority over state-of-the-art methods (e.g., RVT) on GEN1, Fall Detection, and AIR datasets, achieving higher mAP with fewer parameters.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Q1: provide the new collected event dataset public links\n\nRe: Thank you for your suggestion. To support the SNN community, we have made our event dataset publicly available: [Dropbox Link (Anonymous)]( https://www.dropbox.com/scl/fo/1bnsydo3yj5922tlsquo9/AE77sOynJY0-GAeSNBSqFJk?rlkey=1mqpsm658ou26bd4jzmbs46rk&st=h1mhxsyo&dl=0\n).\n\n---\n# Q2: The formal title in the PDF file is missing.\n\nRe: Thank you for pointing this out. We apologize for the oversight. We will ensure that the formal title is included in the PDF file and submit an updated version.\n\n---\n# Q3: While energy efficiency advantage is claimed, practical implementation challenges (e.g., spike timing synchronization, latency constraints on neuromorphic hardware) are not discussed.\n\nRe: Thank you for your feedback. While our paper emphasizes SNNs’ energy efficiency, we now discuss practical implementation challenges:\n\n(1) Spike Timing Synchronization: Our event-driven model updates neuron states asynchronously, eliminating the need for global synchronization.\n(2) Latency Constraints: Neuromorphic hardware (e.g., Intel Loihi, SpiNNaker) may experience spike transmission delays, impacting inference speed.\n(3) Solutions: Asynchronous architectures reduce synaptic access delays, while spatiotemporal multithreading optimizes spike processing.\n(4) Deployment Challenges: Hardware limitations (e.g., unsupported LIF neurons) and quantization requirements pose constraints.\n\nIn future work, we plan to explore migrating our method to actual neuromorphic hardware and further optimize its computational efficiency.\n\n---\n# Q4:  energy estimation to show the energy efficiency.\n\nRe: We understand the importance of providing detailed energy consumption estimates to demonstrate the energy efficiency of our proposed model. In our work, we have calculated the theoretical energy consumption of HsVT using the following methodology:\nThe energy consumption is primarily estimated by calculating the synaptic operations (SOPs), which is given by the formula:\n$\\mathrm{SOPs}(l)=fr\\times T\\times\\mathrm{FLOPs}(l)$\nWhere:\n𝑙 refers to a specific block or layer in the model (e.g., ANN or SNN components),\n𝑓𝑟 is the firing rate of the input spike train to the layer,\n𝑇 is the simulation time step of the spiking neurons,\nFLOPs(l) refers to the floating point operations of the layer, representing the number of multiply-and-accumulate (MAC) operations.\nThe spike-based accumulate (AC) operations are also taken into account to estimate the overall energy usage. For this, we refer to the work of Kundu et al. [1], Yao et al. [2], Zhou et al. [3], and others, and we assume that the MAC and AC operations are implemented on 45nm hardware, where: \n\nEnergy per MAC operation ($E_{MAC}$)=46pJ,\nEnergy per AC operation ($E_{AC}$) = 0.9pJ\n\nThe energy consumption of our model (tiny) is then calculated as follows:$E_{HsVT}=E_{ANN}+E_{SNN}$ , $E_{ANN}=4.6pJ\\times\\mathrm{FLOPs}(b)$ , $E_{SNN}=0.9pJ\\times\\mathrm{SOPs}(b)$\n\nThis results in:$E_{HsVT-backbone}=5.5mJ$. Additionally, for the RVT model(tiny), the energy consumption is calculated as:$E_{rvt-backbone}=6.6mJ$\n\n| Module|HsVT(tiny)|RVT(tiny)|SFOD[4]|Spiking YOLOX-S[5]|\n|---|---|---|---|---|\n| | ANN-SNN|ANN|SNN|SNN|\n|Energy(backbone)|5.5mJ|6.6mJ|---|7.52mJ(embeding)+5.64 mJ|\n|Energy(Fpn+head)|2.9mJ|2.9mJ|---|2.75mJ|\n|Energy(Total)|8.4mJ|9.5mJ|7.26mJ|15.91mJ|\n\nReferences:\n- [1] Kundu S, et al. Hire-snn ICCV, 2021.\n- [2] Yao M, et al. Attention Spiking Neural Networks. IEEE TPAMI, 2023.\n- [3] Zhou Z, et al. Spikformer: When SNN Meets Transformer. arXiv preprint, 2022.\n- [4] Fan Y, et al. SFOD: Spiking Fusion Object Detector. CVPR, 2024.\n- [5] Wang Z, et al. EAS-SNN: End-to-End Adaptive Sampling for Event-Based Detection. ECCV, 2024.\n\n---\n# Q5: Could the proposed hybrid SNN architecture adapts to other computer vision task?  Such as object tracking?\n\nRe: The proposed hybrid SNN architecture is adaptable to other vision tasks, including object tracking. While our focus is event-based detection, SNNs excel at processing spatiotemporal patterns, making them well-suited for tracking tasks that require both appearance and motion cues.\n\nA relevant example is SCTN (Spiking Convolutional Tracking Network), which utilizes energy-efficient deep SNNs for event-based tracking. Inspired by this, our hybrid approach could integrate recurrent SNN structures or memory mechanisms to enhance tracking robustness, particularly in high dynamic range and fast-motion scenarios.\n\n---\n\n# Q6:  could the SNNs provide spatial feature extraction advantage under the proposed hybrid ANN-SNN architecture.\n\nRe: In our proposed STFE module, the convolution and batch normalization layers perform spatial feature extraction and contribute to spatiotemporal fusion.  While SNN handles temporal dynamics, STFE combines the strengths of SNN and ANN, enabling efficient capture of both temporal and spatial features, enhancing overall performance.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "There are no specific theoretical claims.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The paper related to event-based vision and brain inspired computing.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n1. HsVT effectively combines the spatial modeling strengths of Transformers (via self-attention) with the energy-efficient temporal dynamics of SNNs, addressing the limitations of pure ANN or SNN approaches.\n2. The Fall Detection dataset fills a critical gap in privacy-sensitive event-based detection tasks, leveraging event cameras’ advantages (e.g., low latency, privacy preservation).\n3. The ablation studies (Tables 3–6) validate design choices (e.g., LIF neurons, STFE placement), while comparisons with RVT (Tables 7–8) demonstrate HsVT’s efficiency and robustness.\n\nWeakness:\n1. I suggest to provide the new collected event dataset public links in the context to provide more supports to the SNN community.\n2. The formal title in the PDF file is missing. \n3.  While energy efficiency advantage is claimed, practical implementation challenges (e.g., spike timing synchronization, latency constraints on neuromorphic hardware) are not discussed.\n4. Meanwhile, there lacks more detailed energy estimation to show the energy efficiency of the proposed model.", "other_comments_or_suggestions": "No", "questions_for_authors": "1. Could the proposed hybrid SNN architecture adapts to other computer vision task? Such as object tracking?\n2. The proposed model seems to have specific advantage by combining temporal property of SNNs, could the SNNs provide spatial feature extraction advantage under the proposed hybrid ANN-SNN architecture?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MBufcOS6tw", "reviewer": "Reviewer_ZKo3", "summary": "This paper proposes a novel hybrid spiking vision Transformer model (HsVT) for event-driven object detection. Combining the advantages of ANN and SNN, the multi-stage spatial-temporal feature extraction module is designed, and LSTM and STFE are used to process temporal information respectively, reducing the number of parameters and improving efficiency. By converting traditional video data set into event stream data through event camera simulator, Fall Detection data set is constructed, taking into account privacy protection and storage efficiency.\n\n## update after rebuttal\nAfter reading the rebuttal, I decide to keep my original score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Q1: Why choose 4 blocks?  Have you tried other hierarchies (such as 3 or 5 blocks)?How does the model performance change if the number of blocks is increased? \n\nRe: Thank you for your insightful question. In our experimental design, we chose to use 4 blocks based on the following considerations:\n\n(1) Hierarchical Feature Learning: Deep neural networks typically learn hierarchical features layer by layer. In the context of event data and spatiotemporal modeling, using 4 blocks allows the model to capture sufficiently deep features while maintaining a manageable computational complexity.\n\n(2) Computational Cost Trade-off: When using only 3 blocks, the model struggles to capture high-level semantic information adequately, which limits its performance. On the other hand, increasing the number of blocks to 5 results in a substantial increase in computational cost without a significant improvement in performance.\n\n---\n# Q2: [What is the attention head number and dimension allocation strategy of Block-SA and Grid-SA?]\n\nRe: The attention head number and dimension allocation strategy for Block-SA and Grid-SA are as follows: \n\n- Tiny:\n||embed_dim|num_head|dim_head|\n|--|-|-|-|\n|B1|32|1|32|\n|B2|64|2|32|\n|B3|128|4|32|\n|B4|256|8|32|\n\n- Small:\n||embed_dim|num_head|dim_head|\n|--|-|-|-|\n|B1|48|2|24|\n|B2|96|4|24|\n|B3|192|8|24|\n|B4|384|16|24|\n\n- Base:\n||embed_dim|num_head|dim_head|\n|--|-|-|-|\n|B1|64|2|32|\n|B2|128|4|32|\n|B3|256|8|32|\n|B4|512|16|32|\n\n---\n# Q3: Have attention visualizations been performed to verify the validity of feature focusing?\nRe: Thank you for your valuable question.  We have indeed performed attention visualizations to verify the validity of feature focusing in our model.  Specifically, we conducted attention visualization experiments on two datasets: Gen1 and Air.  However, we encountered some unexpected results during these visualizations, and the patterns observed were not as interpretable as anticipated.\n\nThis discrepancy may be due to the complexity of the attention mechanism and the way it interacts with the features in the model.  It's possible that the attention heads are focusing on a combination of spatial and temporal information that is not immediately obvious in the visualization, or that the attention mechanism is implicitly capturing features at different hierarchical levels.We will attach our visualization results to the paper.", "claims_and_evidence": "The paper claims that HsVT achieves high efficiency and precision in event detection tasks through a hybrid ANN-SNN architecture, a claim supported by experimental data", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "The paper does not involve rigorous theoretical proofs (such as convergence of STFE modules), but the design choice is supported by ablation experiments (Tab.5-6), which are acceptable for engineering oriented studies.", "experimental_designs_or_analyses": "The critical ablation experiments (Tab.5-6) were validated on AIR datasets, and comparisons experiments on GEN1/FALL datasets are conducted to enhance generality.\nThe aircraft detection dataset lacks information ( such as sample size, event flow density, class balance) and need to supplement information.", "relation_to_broader_scientific_literature": "The spiking vision transformer may used in other vision tasks such as tracking, reconstruction, and depth estimation.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "The actual energy consumption of the model is not analyzed, which weakens the demonstration of low power consumption advantage of SNN.", "other_comments_or_suggestions": "Supplementary network architecture diagram (e.g. Block internal data flow, SNN-ANN interface).\nAdd a list of hyperparameters (such as LSTM hidden layer dimensions, STFE time window length) to the appendix.\n\nThe paper title in the PDF is incorrect.", "questions_for_authors": "Why choose 4 blocks? Have you tried other hierarchies (such as 3 or 5 blocks)? How does the model performance change if the number of blocks is increased?\nWhat is the attention head number and dimension allocation strategy of Block-SA and Grid-SA? Have attention visualizations been performed to verify the validity of feature focusing?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a Hybrid Spiking Vision Transformer (HsVT), which integrates artificial neural networks (ANNs) with spiking neural networks (SNNs) for event-based object detection. The model leverages self-attention mechanisms alongside SNN-based temporal feature extractors to capture time dependencies. This paper also introduces a new event-based fall detection dataset. The proposed method demonstrates high accuracy and low computational cost on several benchmarks, such as GEN1, AIR, and the newly introduced fall detection dataset.

Reviewer Ho9N found the hybrid architecture novel, highlighting its efficiency and accuracy. Reviewers Zi8P and ZKo3 acknowledged its effective fusion of ANN spatial modeling and SNN temporal dynamics. The experimental design was well-structured and convincingly compared to existing methods.

Reviewers Ho9N and zi8P raised concerns about theoretical justifications and hardware availability. Reviewer Zi8P raised concerns about energy consumption, while ZKo3 suggested improving the paper format. During the rebuttal phase, the authors addressed concerns related to dataset availability, model design, and deployment. After rebuttal, the paper received scores of (4, 3, 3), and none of the reviewers raised further concerns. Therefore, the paper is recommended for acceptance.
Decision: Accept (poster)