SpikeVideoFormer: An Efficient Spike-Driven Video Transformer with Hamming Attention and $\mathcal{O}(T)$ Complexity
Authors: Shihao Zou
Qingfeng Li
Wei Ji
Jingjing Li
Yongkui Yang
Guoqi Li
Chao Dong
Venue: ICML 2025 poster (2025)
Keywords: spiking neural networks
spike-driven transformer
Primary area: applications->neuroscience_cognitive_science
TL;DR: We introduce SpikeVideoFormer, an efficient spike-driven video Transformer, featuring linear temporal complexity O(T).

Abstract: Spiking Neural Networks (SNNs) have shown competitive performance to Artificial Neural Networks (ANNs) in various vision tasks, while offering superior energy efficiency. However, existing SNN-based Transformers primarily focus on single-image tasks, emphasizing spatial features while not effectively leveraging SNNs' efficiency in video-based vision tasks. In this paper, we introduce SpikeVideoFormer, an efficient spike-driven video Transformer, featuring linear temporal complexity $\mathcal{O}(T)$. Specifically, we design a spike-driven Hamming attention (SDHA) which provides a theoretically guided adaptation from traditional real-valued attention to spike-driven attention. Building on SDHA, we further analyze various spike-driven space-time attention designs and identify an optimal scheme that delivers appealing performance for video tasks, while maintaining only linear temporal complexity. The generalization ability and efficiency of our model are demonstrated across diverse downstream video tasks, including classification, human pose tracking, and semantic segmentation. Empirical results show our method achieves state-of-the-art (SOTA) performance compared to existing SNN approaches, with over 15\% improvement on the latter two tasks. Additionally, it matches the performance of recent ANN-based methods while offering significant efficiency gains, achieving $\times 16$, $\times 10$ and $\times 5$ improvements on the three tasks. [https://github.com/JimmyZou/SpikeVideoFormer](https://github.com/JimmyZou/SpikeVideoFormer)

Spike Spike
An Efficient Spike-Driven Video Transformer with Hamming Attention and (T) Complexity
Shihao Zou'
Qingfeng Li
Wei Ji Jingjing Li
Yongkui Yang'
Guoqi Li Chao Dong
Abstract
1. Introduction
Spiking Neural Networks (SNNs) have shown competitive performance to Artificial Neural Net- works (ANNs) in various vision tasks, while of- fering superior energy efficiency. However, exist- ing SNN-based Transformers primarily focus on single-image tasks, emphasizing spatial features while not effectively leveraging SNNs efficiency in vidco-based vision tasks. In this paper, we introduce Spike Fideorormer, an efficient spike- driven video Transformer, featuring linear tempo- ral complexity (TT). Specifically, we design a spike-driven Hamming attention (SDHA) which provides theoretically guided adaptation from traditional real-valued attention to spike-driven attention. Building on SDHA, we further analyze various spike-driven space-time attention designs and identify an optimal scheme that delivers ap- pealing performance for video tasks, while main- taining only linear temporal complexity. The gen- eralization ability and efficiency of our model are demonstrated across diverse downstream video tasks, including classification, human pose track- ing, and semantic segmentation. Empirical re- sults show our method achieves state-of-the-arg (SOTA) performance compared to existing SNN approaches, with over 15% improvement on the latter two tasks. Additionally, it matches the per- formance of recent ANN-based methods while of- fering significant efficiency gains, achieving x 16, x x10 and x5 improvements on the three tasks."
SNNs have emerged as an energy-efficient alternative to traditional ANNs, showing significant potential machine learning (Li al.. 2024). Unlike ANNs, which mainly rely on floaiing-point multiplications for propagation, SNNs replicate the brain's 's neural dynamics and 0/1 spike-based based communication. This spike-driven mechanism enables SNNs to propagate with simple additions while bypass- ing non-spiking connections. Consequently, SNNs achieve greater efficiency. offering advantages over ANNs.
Recently, SNNs have demonstrated performance compara- ble to ANNs across many vision tasks, including image classification (Yao et al., 2024b), object detection (Yao et al., 2024a), semantic segmentation (Wang et al., 2024; Su et al., 2024), and image-text alignment (Li et al., 2023b). While these advancements are significant, current SNN ar- chitectures predominantly focus on spatial feature model- ing using single-image inputs. This narrow focus over- looks key attribute of SNNs: their brain-inspired neural dynamics, which inherently supports neuron-level tempo- ral encoding (Li et al., 2024). Such temporal capabilities make SNNs particularly well-suited for video-based vision tasks a promising yet underexplored area.
Although SNNs support neuron-level temporal encoding, the unidirectional spiking process over time still limits its ca- pability of encoding space-time time features in comparison with ANNs. To this end, spike-driven Transformer models (Zhou et al., 2022; Yao et al., 2024a) can be adapted for video- based tasks, enabling space-time time feature encoding. However, two key practical issues remain to be addressed: (I) Current spike-drives driven attention mechanisms directly adopt the dot- product operation from ANN-based attention, which fails to effectively capture the similarity between spike features, as explained in Fig. 2. (2) These spike-driven Transformer models primarily focus on spatial attention, and it remains unclear which spike-driven space-time attention mechanism can properly balance performance and efficiency for SNNs. Although several ANN based space-time attention designs have been proposed (Arnab et al., 2021), with particular emphasis on reducing quadratic temporal complexity, these advantages may not be directly applicable to SNNs.
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences wsh.zou@siat.ac.com sk.yang@siat.ac.cm IIstiiutee of Automation, Chinese Academy of Sciences <liqingfeng20233@m iate:eggg2222@a..a......mm
quoqi.li@ia.ac.com 'Yale University <wei.ji@ wwei.ji@yale.eddu "University of Alberta 'Shenzhen Univer- sity of Advanced Technology. Correspondence to: Jingjing Li <jingjinl @ uualberta.ca>, Chao Dong schao.dongee @s....cc.en>..
Proceedings of the ndnd International Conference on Machine I Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

SoikeVideooormer: An Efficient Spike-Driven Video Transformer
Therefore, in this paper, we introduce Spike Fore,,r,, an efficient spike-driven video Transformer with linear tem- poral complexity O(T). At its core, we propose SDHA, which leverages normalized Hamming similarity as the at- tention score function for spike features. As demonstrated in Proposition 3.1, this funcctinn provides a theoretically grounded adaptation from traditional real-valued attention to binary spike-driven attention. While directly applying Hamming similarity poses challenges such as Moating-point operations, non-differentiabiiity, and non-linear complexity, we further optimize its design to maintain efficient spike- driven processing. Building on SDHA, we analyze vari- ous spike-driven space-time attention designs and identify joint attention as an optimal scheme that delivers appeal- ing performance while maintaining only linear temporal complexity for video tasks. We conduct experiments on three downstream tasks: 1) video classification, a high-level classification task; 2) human pose tracking, a fine-grained re- gression task; and 3) video semantic segmentation, a dense classification task. Our model achieves SOTA performance among existing SNNs, with over 15% improvements on the latter two tasks. Additionally, our method matches the per- formance of recent ANN-based methods while increasing efficiency by x x16, x x10 and x5, respectively.
various simple classification tasks. However, convolution- based SNNs still struggle to deliver competitive perfor- mance many large-scale vision tasks.
Transformer-based SNNs represent a significant milestone in advancing spiking neural networks. An earlier study (Yao et al., 2023) introduces multi-dimensional attention for the Spiking Transformer, demonstrating promising performance in large-scale image recognition task. However, this ap- proach relies on real-valued membrane potentials for at- tention computation, which compromises the efficiency of SNNs. Subsequent works (Zhou et al., 2022; Yao et al., 2024b;a) propose various spike-driven self-attention mech- anisms that achieve linear computational complexity w.r.t. the input token length. Despite these advancements, cur- rent designs employ Hadamard- or dot-product operations for attention score computation, which theoretically fails to replicate the effectiveness of dot-product score in vanilla Transformer ( (Vaswani et al., 2017).
SNNs in Vision Tasks are crucial for demonstrating their capability in achieving high-efficiency AL. Existing studies have achieved comparable performance with ANNs on var- ious challenging vision tasks, including large-scale image recognition (Fang et al., 2021; Yao et al., 2023; 2024a), se- mantic segmentation (Su et al., 2024; Wang et al., 2024), and object detection (Su et al., 2023; Luo et al., 2024). Ad- ditionally, spiking VAFs (Kamata et al., 2022) and spiking diffusion models (Cao ct al., 2024) are proposed for image generation task. SpikeCLIP (Li et al., 2023b) also demon- strates successful alignment between images and text using SNNs. However, existing works only demonstrate the ca- pability of SNNs in static vision tasks, overlooking their potential in more challenging temporal vision tasks.
Our contributions are summarized as follows:
We introduce Spike VideoFormer. To our knowledge, are the first attempt to explore efficient -rivendriven video Transformer for video-based vision tasks, featur- ing linear temporal complexity O(T).
We propose spike-driven Hamming attention, provid- ing theoretically guided adaptation from traditional real-valued attention to spike-driven attention.
We explore various spike-driven space-time attention variants and provide an optimal design leveraging joint space-time attention, achieving superior performance while with only linear temporal complexity.
Video Transformers have been widely applied to video vision tasks. A straightforward approach involves concate- nating space-time tokens together as input (Wang et al., 2021; Yang et al., 2024). To enhance efficiency, studies have explored alternative attention mechanisms, such as spa- tiotemporal deformable attention (Wang & Torresani, 2022) and shifted window attention (Liu et al., 2022). Another line of research decouples spatial and temporal attention, either to improve computational efficiency (Amaa et al., 2021) or to take advantage of pre-trained image transform- ers (Blattmann et al., 2023). Building on these approaches, our work aims to develop an effective spike-driven video transformer, leveraging the unique efficiency of SNNs for video vision tasks.
The generalization ability and efficiency of our model are demonstrated across three downstream tasks: video classification, human pose tracking. and video seman- tic segmentation.
2. Related Works
Convolution-based SNNs aim to extend the depth and com- plexity of spiking neural networks by adopting designs of ResNet (He et al., 2016). These models are typically trained using back- prooppppppggtion through time (BPTT) with surrogate derivatives approximate the gradient of the spiking func- tion (Zhou et al., 2024). To overcome issues like identity mapping and gradient vanishing, methods such as SEW- SNNS (Fang et al., 2021) and MS-SNNs (Hu et al., 2024) incorporate spike- or membrane- wise shortcut residual con- nections, achieving performance comparable to ANNs on
Visual Cognitive Neuroscience studies how factors like attention, motivation, emotion, and expectation shape vi- sual perception and cognition (Oliva & Torralba, 2007). While the first three enhance relevant stimuli processing, ex- pectation suppresses predictable inputs (Vuilleumier, 2005; Summerfield & Egner, 2009), Predictive processing sug-
2

---

SpikeVideoFormen An Efficient Spike- Driven Video Transformer
Inpul Video
Stage 1;
L
Stage 2;
Stage 3;
La
6
Slage 4;
Le
H
W
H1
W
H
W
H
W
Spiking Neuron Layer
2C
40
80
16
16
16
16
Temporal Spiking
Regression Classificttin
compooal Transforner
Spike- Drike-Driven Sputio-
temporal Transformer
Spike- Spike-Driven Driven Sputio-
Memihanne Shortcul Addition
Downsumple
CNN Block
Spike Deiien
Downsample
CNN Black
Spike Driven
Downsampl
Downsanple
Re parameterization Convolution Layer
Head
RepCony
BN
BatchNorm Layer
Spilke Drivve Space- Time Joint Aticntion
Spike- Driven Spatio-temporal Transformer
RepCune
ReyCons
RepCoon
Spike-Driven CNN Block
RNN
BN
BN
Sepurable
Channel
Space- Timse Johut Attention
Spikke Deiven
Channel MLP
Qs
D
K
Conv
Reshape Space- Time
Spatial-Temporal SDHA
Separable
Charnel
Spike-Driven Space- Time Mteatlnn
Channel
RepConv
Conv
Comv
MLP
Figure 1. Architecture our proposed Spike FideoFomerrr An input video with a shape of T x x undergoes temporal spiking over time, after which it passes through two spike-driven CNN blocks, along with downsample modules. This is followed by two spike-driven spatiotemporal transformers, also accompanied hy downsample modules. Finally, the extracted video features are forwarded to the regression classification head for downstream tasks.
gests perception is inference-driven, refining sensory in- put through internal models shaped by context and experi- ence (Storm et al., 2024). Beyond vision, the visual cortex processes object names and activates in blind individuals, indicating broader cognitive roles in memory, imagery, and language (Bi et al., 2016; Pearson, 2019; Xu, 2017). Though its full scope remains debated, these insights inspire the orain-inspired spiking neural network (SNN) approach for complex visual tasks, enabling high-speed, low-energy neu- ral processing.
ceives input spikes Xμ from connected neurons over If' time steps. When the membrane potential exceeds predefined threshold Wa. the neuron emits a spike Sig and undergoes soft reset reducing its potential by thu- The process is described as follows:
Hn = =BU + X
= un)) U14 = Hμ Stg)
where O(-) is the Heaviside step function, which outputs 1 if H1 > 4hh and O otherwise. For convenience, we denote the temporal spiking process of LIF neuron as SU(U).
3. Method
3.2. Spike-Driven Video Transformer
We begin with the preliminaries of the basic spiking neuron model in Sec. 3.1, followed by the overall architecture of Spike ForeoFoerrr in Sec. 3.2. Spike PiiikeiideFormel consists of two main blocks: (1) spike-driven CNN block that tracts downsampled spatial features from input videos, and (2) spike-driven spatiotemporal Transformer that encodes global space-time features for downstream video tasks. Its core functionalities are achieved by the SDHA in Sec. 3.3, which effectively captures the similarity between spike fea- tures, and the space-time attention designs in Sec. 3.4, which ensure linear temporal complexity for video processing.
The architecture of our proposed Spike VoieoFiideoormeer is il- lustrated in Fig. 1. We adopt a Conv+ViT architecture, which has demonstrated scalability and generalization for both ANNs (Yu et al., 2022) and SNNs (Yao et al., 2024a).
Assuming an input video with shape of T H W 3, the data undergoes temporal spiking over time and is then passed through two spike-driven CNN blocks, along with downsample modules, resulting in an output shape of T HH x W. x 4C', Each CNN block includes a separable convolution module (Sandler et al., 2018) followed by a channel convolution module, described as follows:
3.1. Spiking Neuron Model
U' = U + SepConv(U). SepConv(U') = Conv (C
The Leaky Integrate and Fire (LIF) model (Maass, 1997) is widely used in SNNs. The LIF neuron maintains mem- brane potential Ujg that decays over time according to leaky constant B. The potential is updated only when it re-
U" = U' + ChannelConv(U"))
ChannelCon:((I") Conc(
(1)
3

---

SpikeVideo An Efficient Spike-Driver Video Transformer
Spike Query
Attention Score
Spike Query
Afteation Score
However, the dot-product is not well-suited the attention score in spike-driven attention. Specifically, when there are zero elements in the spike query, the dot-product will in- discriminately disregard the corresponding elements in the spike keys, which can lead to feature vanishing or confusion. To illustrate this issue, we present two intuitive examples in Fig. 2. The dot-product produces identical scores for four significantly different spike keys, indicating that the dot- product used in (Vaswani et al., 2017: Zhou et al., 2022; Yao et al., 2024a) fails to accurately capture the similarity be- tween two binary spike vectors. Additionally, the threshold scale s in Eq. (3) is set empirically, without clear guidance on an appropriate value for stable training. In contrast, we introduce Proposition 3.1 in this paper, which provides thcoretical insights for adapting the design of traditional real-valued attention spike-driven attention.
Dot.
Ham.
A
Spike Keys
Spike Keys
A
A
A
2/6
A
A
4/6
4/6
A
A
3/6
X
A
1/6
Figure 2. Intuitive comparison of attention scores between spike query and keys using Dot-product (Dot.) and normalized Hamming similarity (Ham.). When the spike query contains no elements, the dot-product ignores the corresponding elements the spike keys, resulting in identical scores for four distinct spike keys as examples. This illustrates the dot-product's limitation in accurately capturing the similarity between binary spike vectors.
Here, Conypws Convaw, and Conv represent point-wise, depth-wise, and normal convolution (Chollet, 2017), re- spectively. Temporal dependencies within these blocks are encoded through temporal spiking via neuron-level mem- brane potential accumulation.
Proposition 3.1 (JL Lemma on Binary Embedding (Jacques et al., 2013; Yi et al., 2015)). Let q, k E RCC be real-valued query and key vectors. The corresponding binary embed- dings Re E 10.1)D are defined as:
qs = sign(Ag), k, = sign(AA)).
(4)
Subsequently, the data further processed by two spike- driven spatiotemporal transformers, along with downsample modules, resulting in an output shape of T' x H W 10C. As shown in the lower right corner of Fig. 1, here each spatiotemporal transformer comprises a spike-driven space- time joint attention module followed by a channel MLP module, as described below:
where A E RCD is a projection matrix with each element drawn independently from a normal distribution N(, 1). Given any > and D> lag we have:
16
P(IFH((ssks) ks) - 9 (fc((, s 5o) 21-2e. 2e- (5)
8
Qs, Ks, Vs= = Reshape(SM(RRepCom((0))
where P(-) denotes probability, g(x)= 1
arccos(1) (1)
U'= U RepCon((DDIAA(s: Kg, Vs)), U" = U" ChannelMLP(U"),
is a continuous and monotone function for x with g(x) E
(2)
[0, 1]. and M represents the number of all possible keys and queries in the finite training set. Iu E [0, 1] is the normalized Hamming similarity defined as:
ChanneIMLP(U") = MLP(SS((MP(SSNN())
Here, RepConv refers to re re-parametrization convolu- tion (Ding et al., 2021) and the joint attention is based on SDHA. The spike-driven spatiotemporal transformer effec- tively encodes both spatial and temporal features, enabling robust performance for downstream video tasks. Next, we will elaborate on each key component of VideoFiidoormerr
fulge, hi.) -1 ka) 1111-01
1 Σ
10 + k!!)),
(6)
and fe E [0.1] is the cosine similarity defined as:
g 11 |
Sclg,
3.3. Spike-Driven Hamming Attention (SDHA)
Spike-Driven Self-Attention (SDSA) (Zhou et al., 2022: Yao et al., 2024a) uses binary spiking tensors Q., K,, V. E {0. 1)NDD as the Query. Key and Value, where is the token length and D is the channel size. By adopting dot- product as the attention score and omitting the softmax function, the computation order can be rearranged to achieve linear computational complexity w.r.t. the token length. This process is described as follows:
The proof is provided in Appendix A. This proposition shows that when the channel size D is sufficiently large, the normalized Hamming similarity Sx closely approximates g(fe) with high probability, where fe is used in traditional real-valued attention. Empirical evidence supporting this is shown in Fig. 3. We randomly generate 10k, 100k, and 1000k pairs of real-valued unit vectors of size 64, compute their corresponding binary embeddings of size D and eval- uate the average error [fN - -((e))- The results indicate that as D increases, the error approaches zero. Additionally, larger datasets generally require large D to achieve close
SDSA = SN. (Q.K((Q.TWW)) = SN. (Q.(KK.
(3)
D(N2D)
D(N
where SN, denotes LIF neuron with scaled threshold 8 Uhh
4

---

Spikeeideooormer: An Efficient Spike-Driven Video Transformer
approximation in similarity measurement. Since g(-) is continuous and monotonic function, g(fc) can still preserve the similarity ranking between query and different keys, Therefore, we propose to use the normalized Hamming sim- ilarity JH as the attention score function in SDHA.
Spilk- Driven Space- Time Juint
Space-Time Spilke-Drivee
Spike Dylvea Space-Timm Factorized
Hierarchical Attention
Rx AxTANxD
AxTX X
Bv x
Jusce 2
D Splicdeane
Reslaps Space-Tare Time 11% TN 12
Reslege Sprce (1 x N x!
Rodaae Tine x
Reckape Space
SMIRA
SDIlI
Spaad
3PMA
10k paics
Consteate Chared
L00k
Rectape Tirea BN x
pairs -0.3
Teeeperal SIMIA
0.2
Figure Space-time Spike-Drive Driven Attention Designs.
0.1
1.20
2.5
522
In24
2000
4.06
0.0
Table I. Comparison of complexity and parameters between ANN-
N g(fe) vs D
based and spike-driven space-time attention designs.
Figure 3. Empirical evidence of average error fH g(fc) with respect to embeddings size D.
ANN-based Attention
Spive-Driven Attention
Complexity
Params
Complexity
Params
Computational Efficiency Design. Directly using Sx for spike-driven attention faces three main challenges: (1) floating-poing point operations, (2) non-differentiable computa- tion, and (3) non-linear complexity w.r.t. the token length. To address these issues, we rewrite fu as:
Joint
O(T=N3D)
4D2
O(TND*)
4D
Hierarchical
O(TN(T + M)D)
8D
O(TND")
SD2
Factorized
O(TNIT+ + (N)D)
7DP
O(TNDA)
TDR
1 1 (2q. (2k, 1); 2D
designs for spatiotemporal feature encoding. To address this, we adapt three typical attention designs from ANN-based space-time attention (Amab et al., 2021) spike-driven sce- narios. These designs are illustrated in Fig. 4, with detailed architectures provided in Appendix C.
fn(g.,k.) k..)
(8)
2
with the derivative provided in Appendix B. Since the constant & does not affect the similarity ranking between a query and different keys, we omit it in Eq. (8). Next, by replacing fn with the dot- dot-product in Eq. (3), we define the SDHA as:
Assume the input spike feature has the shape BxT x N x D, where B is the batch size, T is the temporal token length, N is the spatial token length, and D is the channel size. The three adapted designs are as follows:
SDIA SN
2D (2Q. 1)(2K. 1) [4.)
(9)
or equivalently,
1) Joint Attention: The spike feature is reshaped to B x TN D, where the total token length TN spans the entire spatiotemporal domain. This reshaped feature is then processed using SDHA for spatiotemporal feature fusion.
SDHA = 2N (2Q+ (2Q+-1) 11 v1)l)))
(10)
O(ND2)
Here, the transformations (2Q, - 1) and (2K, 1) map elements from 1] to {-1, 1]. which can be efficiently achieved through element-wise bit shifts, eliminating the
2) Hierarchical Attention: The spike feature is first reshaped to BTxx x D, with N as the token length for spatial attention. The resulting feature is then reshaped to BN x D, with T as the token length for temporal attention.
need for multiplication. The scaling factor 2D can be di- reetly merged into the threshold of spiking neuron, thus avoiding multiplication during inference. After rearrang- ing the computation order, the computational complexity of our proposed SDHA remains linear w.r.t. the token length. Finally, the matrix multiplication in Eq, (10) can be con- verted to additions and extractions via addressing algo- rithms (Frenkel et al., 2023).
3) Factorized Attention: The channel dimension of the input spike feature divided into two branches. One branch undergoes spatial attention, while the other un- dergoes temporal attention. The final output is obtained by concatenating these two processed parts.
3.4. Space-Time Spike- DDiive Attention Designs
We compare the computational complexity and parameter count of the three prototypes in Tab. 1. The ANN-based space-time joint attention suffers from quadratic complex- ity w.r.t. spatiotemporal length, NT2 D). While the
Existing spike-driven attention designs are primarily tailored for single-image tasks, focusing solely on spatial feature en- coding. However, video-based vision tasks require effective
5

---

SpikeVideoFormer: An Efficient Spike-Driven Video Transformer
hierarchical and factorized designs attempt to reduce this to their complexity remains quadratic w.r.t. I'and N. In contrast, all three spike-driven attention designs achieve linear complexity w.r.t. spatial or temporal length, (()ND*), making them significantly more efficient and scalable for video-based vision tasks. This is particularly advantageous when processing large volumes video data, as it alleviates substantial computational costs.
Table 2. Video Classification Results on Kinetics-400 dataset. "Spike". "Param". "Step". "Top-1" and "Top-5" denote spike- driven or not, parameter count, temporal spiking step, top-1 and top-5 accuracy. The best result among SNN methods is bold, and among ANN methods is underline.
Methods
Architecture
Spike
Param
Power Top-I
Top-5f (%)
(M) (mJ)
(%)
I3D-101
x 61.8 1651.4
77.7
93.3
(Wang et al., 2018)
Empirical experiments consistently demonstrate the superior performance of the joint attention design over other designs across different video tasks. Therefore, we adopt this ap- proach in our Spike VideoFormer, which ensures enhanced performance without increasing computational demands. Notably, this observation also aligns with recent findings in video generation (Yang al., 2024).
ANN
ViVIT (Arnab ct al., 2021)
x 310.8 6651.6
80.6
94.7
Swin-B
x
88.1
1297.2
80.6
94.6
(Liu et al., 2022)
Transformer -Bused
Meta-Spikeeorrmrr
/
55.9
396.4
75.5
90.1
(Yao et al., 2024a)
SpikeVideoFommer
SNN
55.9
412.1
79.8
94.0
(Ours)
4. Experiment
marily attributed to our spike-driven Hamming attention, which outperforms the dot-product attention used in Meta- SpikeFormer. Additionally, our SNN model outperforms the ANN-based 13D-101 by 2. 1% and closely matches the performance of the recent Swin-B, BB, trailing by only 0.8% in top-1 accuracy while offering x3 increase in efficiency. This efficiency gain is even more pronounced when com- pared to the earlier work ViVIT, with our method showing over a X 16 reduction in power consumption.
We evaluate our proposed Spike VideoFiidoorrmeer on three repre- sentative video-based vision tasks: 1) Video classification, a high-level task that predicts a single categorical label for a given video clip: 2) Human pose tracking, which involves innegrainedd regression of pose values over time, with em- phasis on maintaining accuracy during long-term tracking: and 3) Video semantic segmentation, a dense classification task that assigns category to each pixel in the input frames, highlighting the importance of similar feature encoding and fusion. Additionally, we conduct ablation studies to further verify the effectiveness of our approach.
4.2. Human Pose Tracking
We evaluate our method on the task of human pose track- ing using either video clips or event streams as input. No- tably, the use of efficient SNNs for this challenging task, particularly in long-term pose tracking, has been largely unexplored. Thus we investigate the application of SNNs to human pose tracking in this work.
Model Variants. To ensure fair comparison with previous works, we use two similar model sizes: small model with C' = 32, resulting in 15. IM parameters, and a large model with C' = 64, resulting in 55.4M parameters. The architec- ture details are provided in Appendix D. Both models are pre-trained on the ImageNet1 K dataset and then used for initialization in downstream video tasks. The pre-training process is provided in Appendix F.
Implementation Details. 2021) is currently the largest dataset offering both videos and event streams, along with annotated 3D poses. We evaluate our method using both modalities as input and conduct comprehensive comparison with ANN-based and SNN-based approaches for this fine-grained tracking task. Following the standard train-test split of the dataset, we train and evaluate models with T = 8 and T = 32 time steps, which correspond to and 8 seconds of input video clips or event streams, respectively. The pipeline and additional implementation details are provided in the Appendix H.
The MMHPSD dataset (Zou et al.,
4.1. Video Classification
Implementation Details. The Kinetics-400 dataset (Kay et al., 2017) is widely-used dataset for human action recog- nition, which consists of around 240k training videos and 20k validation videos in 400 human action categories. Dur- ing training, a clip of 32 video frames from each video using a temporal stride of 2 and spatial size of 224 x 224 is employed. We follow prior works (Liu et al., 2022) by reporting 1 1 and top-5 recognition accuracy. Additional implementation details are provided in Appendix G
Evaluation metries. Following prior works (Kocabas et al., 2020), we report Mean Per Joint Position Error (MPJPE), PELviiaal MPJPE (PEL-MPJPF)) and Procrustes- Aligned MPJPE (PA-MPJPE) in millimeters (mm). PA- MPJPE aligns the predicted and target poses through both rotation and translation, while PEL-MPJPE only aligns them using translation of the root joint.
Experiment Results. As shown in Tab. 2, our method achieves t accuracy of 79.8%, surpassing the prior Meta-SpikeFormer by 4.3%, with a similar improvement observed in t accuracy. This improvement is pri-
6

---

SpikeVideoFormer: An Efficient Spike-Driven Video Transformer
Table Human Pose Tracking Results on MMHPSD Dataset. The best result among SNN methods bold, and among ANN methods is underline. *For fair comparison, extend the spike-driven spatial attention used in previous image-based SNN methods with the space-time joint attention, while retaining the dot product as attention score function. This modification consistently applied in the subsequent experiments.
Methods
Architecture
Param
Power MPJPFJ PEL-MPIPEL PA-MPIPF.
Power (mJ)
MPJPF, PRI-MPJPEL PA-MPJPFJ
(M)
(mJ)
Video Input
VIBE (Kocabas et al., 2020)
48.3
392.1
68.2
46.3
1511.2
75.4
53.6
ANN
GLoT (Shen et al., 2023)
40.5
487.5
61.6
39.9
4046. I
67.3
46.5
ANN2SNN
VIBE (Hu al., 2023)
48.3
87.4
78.1
55.8
123.8
84.3
62.2
CNN-Based
SEW-Res (Fang et al., 2021)
60.6
26.4
124.7
68.7
48.8
106.7
150.1
77.4
56.8
SNN
MA-SNN (Yao al., 2023)
78.8
15.7
122.4
68.2
48.2
63.4
145.3
76.8
56.0
SpikFormer* (Zhou et al., 2022)
29.9
24.5
125.2
69.7
48.8
99.6
147.6
77.8
56.9
66.7
43.9
119.2
67.4
46.2
178.4
140.1
76.3
55.2
Transformer
Foike
15.4
34,2
126.4
71.3
49.1
138.8
145.7
77.9
57.1
-Based
SNN
(Yao al., 2024a)
55,8
95.7
116.3
67.0
45.7
387.2
138.2
74.2
54.5
Spike VideoFormer
15.5
34.6
121.8
44.4
143,4
141.3
75.2
49.9
(Ours)
55.8
96.0
109.2
61.8
39.8
391.2
131.8
69.2
47.5
Event Stream as Input
VIBE (Kocabas 2020)
48.3
392.1
70.9
46.6
1511.2
76.8
53.9
ANN
GLoT (Shen al., 2023)
40.5
487.5
62.0
40.1
4046. I
68.8
47.7
ANN2SNN
VIBF (Hu al., 2023)
48.3
84.1
79.3
56.3
120.8
85.8
63.6
CNN-Based
SEW-Res (Fang al., 2021)
60.2
21.5
127.3
69.7
49.1
88.7
152.3
78.0
57.1
SNN
MA-SNN (Yao et al., 2023)
78.4
12.1
126.7
69.4
48.9
51.3
150.2
77.7
56.7
SpikFormer* (7hou et al., 2022)
29.9
19.7
126.1
70.1
49.0
81.2
149.1
78.4
57.6
66.6
34.9
121.0
67.9
47.1
143.6
143.4
77.2
55.8
Transformer
Spikee
15.4
27.1
130.1
49.8
112.1
148.5
78.2
57.4
-Based
SNN
(Yao et al., 2024a)
55.8
72.5
117.1
67.5
46.0
308.5
140.8
75.0
55.1
Spike VideoFormer
15.5
27.6
128.9
68.3
46.9
121.8
142.5
75.9
50.3
(Ours)
55.8
73.3
115.7
62.1
40.3
291.3
136.7
70.1
48.1
Experiment Results. In Tab. 3, we compare Spike oFormer with four SNN-based models using either video clips or event streams as input. Our method outperforms prior SNN methods by a significant margin, achieving a PA-MPJPE of 39.8, compared to 48.2 for CNN-based SNN and 45.7 for Transformer-baat bassed SNN. This demonstrates that, unlike CNN-based SNN, the spike-driven transformer excels at encoding global features for video-based tasks. Additionally, our proposed Hamming attention improves similarity measurement over spike-driven dot-produce atten- tion used SpikFormer and Meta-Spike Speta-,piForrerr, while not sacrificing computational efficiency.
computational efficiency of Spike VideoFormer, which has ( complexity. Furthermore, when using event streams as input, spike-driven methods exhibit lower power con- sumption compared to video clips. This is likely due the sparsity of events, which typically results in a lower spiking rate in SNN models. In contrast, ANN models still require the same level of computation regardless of whether the input is an event stream video clip.
Inference Time Discussion. We test on an A6000 GPU and AMD EPYC 7543 CPU for human pose tracking, averaged over 1, 1,000 video clips with batch size of 7 increases from to 32 (4x), GLoT (quadratic ANN attention) achieves the best performance but experiences 9.8x increase in inference time from 303 to 2972ms, while VIBE (linear ANN-GRU) shows 5.1x increase from 264 to 1335ms but performs the worst. Our SNN method exhibits only 4.6x increase from 235 to 1087ms thanks to our proposed spiking space-time linear attention.
When compared to two representative ANN-based methods, our performance is close to that of GLoT, with minor gap of 0.2mm in PEL-MPJPE and 0. mm in PA-MPJPE. How- ever, for long-term tracking (T' = 32), the gap increases to 1.9mm and 1.0mm, respectively. Despite this, the power consumption of our method increases from 96.0 391.2 (a 4.1x increase), whereas GLoT's power consumption rises from 487.5 to 4046.1 (an 8.3x increase). This highlights the
For edge device deployment, SNNs, relying only on addi- tions, significantly reduce power consumption and latency

---

SpikeVideoFormer: An Efficient Spike-Driven Video Transformer
on neuron-computing devices (Kudithipudi et al., 2025; Yao et al., 2025). ANNs, requiring floating-point multiplica- tions, achieve high parallel acceleration on GPUs but high energy cost. On the AMD Xilinx ZCU104 (Li et al., 2023a), ANNS process at 691.2 .22 GFLOPs/s, while SNNs reach 5529.6 GFLOPs/s-an an 8x speedup. In 45mm tech- nology (Horowitz, 2014), ANN multiplication consumes 4.6pI, whereas SNN addition uses only 0.9pJ. a 5.1x energy reduction.
Table 4. VSS Results on the CityScape dataset. "Integer-LIL LIF"
refers to the use of the integer LIF model (Luo et al., 2024) in the SpikeFPN segmentation head.
Inleger -LIF
Param Puwer mloUf
Methods
Architecture
(M)
(mJ)
(%)
FCN (Long al., 2017)
9.8
534.8
61.5
PSPNet (Zhao et al., 2017)
13.7
793.3
70.2
ANN
SegFormer (Xie et al., 2021)
13.8
270.2
74.1
CFPM (Sun et al., 2024)
15.4
376.5
75.1
SpikFormer (Zhou et al., 2022)
35.8
42.3
62.3
4.3. Video Semantic Segmentation (VSS)
4
35.8
48.9
65.0
Transformer
Meta- Spike-ppieerrrmer
17.8
61.6
63.0
We further evaluate our proposed Vodkereie on the task of VSS. Since our primary objective is demon- strate the generalization ability of our method across various rideo-based tasks, we avoid using a complex architecture for VSS. Specifically, the classification head in Spike Video- Former consists of two main components: 1) Memory Read and Fusion modules (Paul et al., 2021), where relevant se- mantic information from previous frames (memory) is read and fused with the features of the current frame via spike- driven rross-attention and 2) Spike- -rivenn Feature Pyramid Networks (SpikeFPN) (Zhao et al., 2017), which aggregate multi-scale features extracted by SpikeVideo VideoFormer to pre- dict pixel-level semantic categories. The pipeline is detailed in Appendix I. Unlike recent SNN methods (Su et al., 2024; Wang et al., 2024) that focus on single-frame semantic seg- mentation, our work explores the use of efficient SNNs for VSS, advancing their application to spatiotemporal contexts.
-Based
SNN
(Yao et al., 2024a)
4
17.8
63.5
65.9
Spike FideeFoerrrr (Ours)
1
17.8
64.0
70.8
4
17.8
65.3
73.1
Table S. VSS Results on the VSPW dataset. "Integer-LIF" refers to the use of the integer LIF model (Luo et al., 2024) in the SpikeFPN segmentation head.
Integer Param Power mloUf -LIF
Methods
Architecture
(M)
(mJ)
(%)
DecpLab (Chen al., 2017)
62.7
34.7
PSPNet (Zhao et al., 2017)
70.5
1313,9
36.5
ANN
TCB (Miao et al., 2021)
70.5
1206.9
37.5
CFFM (Sun et al., 2024)
85.5
1068.1
49.3
Spikformer (Zhou et al., 2022)
92.2
176.8
30.0
4
92.2
180.3
31.0
Transformer
SpikeSpp (Yau et al., 2024a)
72.1
208.7
31.1
-Based
SNN
72.1
213.1
32.3
Datasets. We conduct experiments on two widely used VSS datasets: CityScapes (Cordts et al., 2016) and VSPW (Miao et al., 2021). CityScapes focuses on driving scenarios and includes 3,000 high figh-resolution images with fine annotations for training and 500 for validation, covering 19 semantic categories. In contrast, VSPW is the largest VSS benchmark, encompassing wide range of indoor and outdoor scenes with annotations across 124 categories. This dataset consists of 2,806 clips for training and 343 clips for validation.
Spike
72.1
218.8
36.5
(Ours)
4 72,1
226.2
37.9
tional implementation details are provided in Appendix I.
Experiment Results. From Tab. 4, we observe that our method outperforms classical ANN approaches, such as FCN and PSPNet, achieving a higher mloU of 70.8, com- pared to 61.5 and 70.2, while delivering x 12 and x8 greater efficiency, respectively. When compared to more complex models like SegFormer and CFFM, Spike VideoFormer with Integer-LIF of 4 achieves competitive mIoU scores, trailing by only 1% and 2%, while offering a x boost in efficiency, These results underscore the efficient yet effective space- time feature encoding capability of our SNN model. Fur- thermore, when compared to recent SNN approaches, Spike- VideoFormer achieves a significantly higher mIoU (73.1 vs. 65.0 for SpikFormer and 65.9 for Heta-SpikeFormer) This performance gap is likely due to the limitations of the dot-product mechanism in these models, which struggles to accurately capture feature similarity during space-time feature fusion in the backbone and memory read module.
Implementation Details. Following prior work (Sun et al., 2024), we evaluate a small-size model (17.8M) on the CityScapes dataset and a large-size model (72.1M) on the VSPW dataset. We also employ the Integer-L.IF model (Luo et al., 2024) in SpikeFPN, which has been shown to improve performance with minimal computational increase. A set- ting of I means the output spikes are {0, 1}. and a setting of 4 means {0, 1,2.3.. We begin by training the encoder and decoder for single-frame segmentation, with the encoder ini- tialized using the pre-trained model from Appendix F. Once trained, fix the encoder and decoder, and incorporate the Memory Read and Fusion module, which are trained 4-frame clips for VSS. We report the performance of VSS using the mean Intersection over Union (mloU). The power consumption is averaged over the number of frames. Addi-
A similar trend is observed in Tab. 5 on the VSPW dataset, where our method outperforms classical ANN approaches
8

---

SpikeVideoFormer: An Efficient Spike-Driven Video Transformer
performance on both tasks, improving PA-MPJPE by over 26% and mloU by 19%.
Table Ablation Study. We perform experiments on two tasks: human pose tracking using the MMHPSD dataset and video se- mantic segmentation using Cityscapes dataset, our model,
Threshold Scale in Eq. (10). Prior work (Yao et al., 2024a) sets s 1/8 in Eq. (3). However, as shown in Tab. 6, we observe that fixed values of s generally result in worse performance compared to the proposed 8 = 1/2D,
the value of D varies across layers rather than being fixed.
Power Pose-Track
Power
VSS (mJ) mloU(%))
Architocture
(mJ) PA-MPIPE
SpiooVidcooormer
96.0
39,8
65.3
73.1
where D varies across layers in our model.
Att. Score
Ham Dot-prod
95.7 45.7 (+5.9)
63.5
65.9 (-7.8)
Joint + Hierarchical
102.6 44.5 (+4.7)
68.5
71.7 (-1.4)
5. Conclusion
Space-
Joint + Factorized
99.7 45.1 (+5.3)
67.2 70.3 (-2.8)
Time
Attention
Joint- + Neuron-Level 94.6 47.4 (+7.6)
65.1
(-5.9)))
In this paper, we present Spike VideoFormer, an efficient spike-driven video Transformer designed to enhance the per- formance of SNNs in video-based vision tasks. Our model leverages spike-driven Hamming attention and joint space- time attention achieve state-of-the-arr results among SNN approaches for video classification, human pose tracking, and video semantic segmentation, while significantly im- proving efficiency compared to recent ANN-based methods. Additionally, the linear temporal complexity of O(T) fur- ther highlights the potential of our proposed Transformer for scalable video processing. In the future work, we aim scale up the SNN model as a backbone to support broader range of video tasks, such video understanding and generation.
Joint-
Spatial-Only
90.2. 54.2 14.4)
64.2 62.1 (-11.0
Initial
Pre-train Random
96.5 53.8 (+14.0)
65.3
61.3
8 1/2D -> 1/8
97.8
46.7 (+6.9)
66.7 68.5 (-4.6)
Threshold
== 1/21 + 1/64
97.0
43.0 (+3.2)
66.0 71.1 (-2.0)
Seale 8
Eq. (10) == 1/2D 1/512
96.3
41.3 (+1.5)
63.2 70.3 (-2.8)
1/2D + 1/2048
91.4
48.3 (+8.5)
60.3
(-.77)
DeepLab, PSPNet, and TCB, While there is notable perfor- mance gap compared the latest CFFM model, this can be attributed to CFFM's use of complex coarsc-t--fine feature assembly module for memory read and fusion. In contrast, our scope is to demonstrate the capability and generalization ability of our SNN model as a backbone for video-based tasks, using only a simple architecture of head for VSS.
Acknowledgments
This work was partially supported by grant from Shenzhen Science and Technology Program (Grant No. JSGG2022831105000044)
4.4. Ablation Study
We conduct ablation studies on various components of Spike VideoFormer on both tasks in Tab. 6.
Impact Statement
Attention Score Function. Performance drops significantly in both tasks when replacing our proposed Hamming at- tention with dot-product attention, 15% in pose tracking and 10% in VSS. The slight reduction in power may be attributed to the dot-product's masking effect, which lowers the spiking rate after applying attention.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Space-Tine Attention. Replacing joint space-time time atten- tion with hierarchical factorized attention results in per- formance degradation in both tasks, primarily due to the failure to capture global spatiotemporal features. This aligns with recent findings in video generation (Yang et al., 2024), where joint attention outperforms other decomposed atten- tions. When only neuron-leve temporal encoding is used (Neuron-Only) without attention on temporal domain, per- formance drops moderately but remains acceptable. How- ever, removing neuron-level temporal encoding (Spatial- Only) by resetting the membrane potential at each time step causes significant decline, highlighting the capability of temporal encoding inherent to SNNs.
References
Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučié, M., and Schmid, C. Vivit: video vision transformer. In ICCV, 2021.
Bi, Y., Wang, X., and Caramazza, A. Object domain and modality in the ventral visual pathway. Trends in cogni- tive sciences, 2016.
Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V. Letts, A., et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint 1.Xiv:2311.15127, 2023.
Initialization. Initializing the model with preitrained weights from the ImageNet1K K dataset significantly boosts

---

SpikeVideoFormer: An Efficient Spike-Driven Video Transformer
Cao, J., Guo, H., Wang, Z., Zhou, D., Cheng, H., Zhang, Q.. and Xu, R. Spiking diffusion models. IEEE TAI, 2024.
Jacques, L.,, Laska, J. N., Boufounos, P. T., and Baraniuk, R. G. Robust 1-bit compressive sensing via binary stable embeddings of sparse vectors. IEEE Transactions Information Theory, 2013.
Chen, L.-C., Papandreou, G., Kokkinos, I,, Murphy, K., and Yuille, A. L. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE TPAMI, 2017.
Kamata, H., Mukuta, Y., and Harada, T. Fully spiking variational autoencoder. In AAAI, 2022.
Chollet, F. Xception: Deep learning with depthwise separa- ble convolutions. In CVPR, 2017.
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijaynarasimman, S., Viola, F,, Green, T., Back, T., Natsev, P., et The kinetics human action video dataset. arXiv preprint arXiv: /005 769500, 2017.
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., and Schiele, B. The cityscapes dataset for semantic urban scene under- standing. In CVPR, 2016.
Kocabas, M., Athanasiou, N., and Black, M. Vibe: Video inference for human body pose and shape estimation. In CVPR, 2020.
Dapello, J., Marques, T., Schrimpf, M., Geiger, F., Cox, D., and DiCarlo, J. J. Simulating primary visual cor- tex at the front cnns improves robustness to image perturbations. NeurIPS, 2020.
Kudithipudi, D.,, Schuman, C., Vineyard, C. M., Pandit, T., Merkel, C.,, Kubendran, R., Aimone, J. B., Orchard, G., Mayr, C., and Benosman, R. Neuromorphic computing at scale. Nature, 2025.
Deng, J., Dong, W., Socher, R., Li, I..J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Li, G., Deng, L., Tang, H., Pan, G., Tian, Y., Roy, K., and Maass, W. Brain-inspired computing: A systematic survey and future trends. Proceedings of the IEEE, 2024.
Ding, X., Zhang, X., Ma, N., Han, J., Ding, G., and Sun, J. Repvgg: Making vgg-style convnets great again. In CVPR, 2021.
Li, J., Shen, G., Zhao, D., Zhang, Q. and Zeng, Y. Firefly: A high-throughput hardware accelerator for spiking neural networks with efficient dsp and memory optimization. IEEE TVLSI, 2023a.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn,
D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. image is worth 16x words: Transformers for image recognition scale. ICLR, 2021.
Li, T., Liu, W., Lv, C., Gu, Y., Xu, J., Zhang, C., Wu, M., Zheng, X., and Huang, X. Spikeclip: A contrastive language-image pretrained spiking neural network. arXiv preprint arXiv:23 10.06488,, 2023b.
Fang, W., Yu, 7., Chen, Y., Huang, T., Masquelier, T., and Tian, Y. Deep residual learning in spiking neural net- works. NeurlPS, 2021.
Liu, Z., Ning, J., Cao, Y.. Wei, Y., Zhang, Z., Lin, S., and Hu, H. Video swin transformer. In CVPR, 2022.
Frenkel, C., Bol, D., and Indiveri, G. Bottom-up and top- down approaches for the design of neuromorphic process- ing systems: tradeoffs and synergies between natural and artificial intelligence. Proceedings of the IEEE, 2023.
Long, J., Shelhamer, E., and Darrell, T. Fully convolutional
networks for semantic segmentation. IEEE TPAMI, 2017.
Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., and Black, M. Smpl: skinned multi-person linear model. ACM TOG, 2015.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In CVPR, 2016.
Luo, X., Yao. M., Chou. Y.. Xu. B., and Li, G. Integer- valued training and spike-driven inference spiking neural network for perforerformnce and energy- object detection. ECCV, 2024.
Horowitz, M. 1.1 computing's s energy problem (and what do about it). In ISSCC, 2014.
Hu, Y., Zheng, Q. Jiang, X., and Pan, G. Fast-snn: Fast spik- ing neural network by converting quantized ann. IEEE TPAMI, 2023.
Maass, W. Networks of spiking neurons: the third genera- tion of neural network models. Neural networks, 1997.
Hu, Y. Deng, L., Wu, Y., Yao, M., and Li, G. Advancing spiking neural networks toward deep residual learning. IEEE T.NLS, 2024.
Miao, J., Wei, Y., Wu, Y., Liang, C., Li, G., and Yang, Y. Vspw: A large-scale dataset for video scene parsing in the wild. In CVPR, 2021.
10

---

Soiker:deoFormer: An Efficient Spike-Driven Video Transformer
Oliva, A. and Torralba, A. The role of context in object recognition. Trends in cognitive sciences, 2007.
Wightman, R., Touvron, H., and Jégou, H. Resnet strikes back: An improved training procedure in timm, arXiv preprint arXiv:2 2021.
Paul, M., Danelljan, M., Van Gool, l.,, and Timofte, R. Local memory attention for fast video semantic segmen- tation. In IROS, 2021.
Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, M., and Luo, P. Segformer: Simple and efficient design for semantic segmentation with transformers. In NeurIPS, 2021.
Pearson, J.
The human imagination: the cognitive neuro- science of visual mental imagery. Nature reviews neuro- science, 2019.
Xu, Y. Reevaluating the sensory account of visual working memory storage. Trends Cognitive Sciences, 2017.
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J., Yang, Y., Hong, W., Zhang, X., Feng. G., et al. Cogvideox: Text-to-video diffusion models with an ex- pert transformer. arXiv preprint arXiv:2408. 06072, 2024.
Shen, X., Yang, Z., Wang, X., Ma, J., Zhou, C., and Yang, Y. Global-to-local modeling for jideo-based 3d human pose and shape estimation. In CVPR, 2023.
Yao, M., Zhao, G., Zhang, H., Hu, Y., Deng, L., Tian, Y., Xu, B., and Li, G. Attention spiking neural networks. IEEE TPAMI, 2023.
Storm, J. F., Klink, P. C., Aru, J., Senn, W., Goebel, R., Pigorini, A., Avanzini, P., Vanduffel, W., Roelfsema, P. R., Massimini, M., al. An integrative, multiscale view on neural theories of consciousness. Neuron, 2024.
Yao, M., Hu, J., Hu, T., Xu, Y., Zhou, Z., Tian, Y., Xu, B., and Li, G. Spike-driven transformer v2: Meta spiking neural network architecture inspiring the design of next- generation neuromorphic chips. ICLR, 2024a.
Su. Q., Chou, Y., Hu, Y., Li, J.. Mei, S., Zhang, Z., and Li, G. Deep directly-trained spiking neural networks for object detection. ICCV, 2023.
Yao, M., Hu, J., Zhou, Z., Yuan, L.,, Tian, Y., Xu, B., and Li, G. Spike-driven transformer. NeurIPS, 2024b.
Su. Q., He, W., Wei, X., Xu, B., and Li, G. Multi-scale full spike pattern for semantic segmentation. Neurul Networks, 2024.
Yao, M., Richter, O., Zhao, G., Qiao, N., Xing, Y., Wang. D., Hu, T., Fang, W., Demirci, T., and Marchi, M. D. Spike- based dynamic computing with asynchronous sensing- computing neuromorphic chip. Nature Communications, 2025.
Summerfield, C. and Egner, Expectation (and attention) in visual cognition. Trends in cognitive sciences, 2009.
Sun, G., Liu, Y., Ding, H., Wu, M., and Van Gool, L. Learn- ing local and global temporal contexts for video semantic segmentation. IEEE TPAMI, 2024.
Yi, X., Caramanis, C., and Price, E. Binary embedding: Fundamental limits and fast algorithm. In ICML, 2015.
Yu, W., Luo, M., Zhou, P., Si, C., Zhou, Y., Wang, X., Feng, J., and Yan, S. Metaformer actually what you need for vision. In CVPR, 2022.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In NeurlPS, 2017.
Zhao, H., Shi, J., Qi, X., Wang, X., and Jia, J. Pyramid scene parsing network. In CVPR, 2017.
Vuilleumier, How brains beware: neural mechanisms of emotional attention. Trends in cognitive sciences, 2005.
Zhou, C., Zhang, H., Yu, L., Ye, Y., Zhou, Z., Huang, L., Ma, Z., Fan, X., Zhou, H., and Tian, Y. Direct training high-performance deep spiking neural networks: a review of theories and methods. Frontiers in Neuroscience, 2024.
Wang, H., Liang, X., Zhang, T., Gu, Y., and Geng, W. Pssd- transformer: Powerful sparse spike-driven transformer for image semantic segmentation. ACM MM, 2024.
Wang, J. and Torresani, I.. Deformable video transformer. In CVPR, 2022.
Zhou, 7.,, Zhu, Y., He, C., Wang, Y., Yan, S., Tian, Y., and Yuan, L. Spikformer: When spiking neural network meets transformer. In ICLR, 2022.
Wang, X., Girshick, R., Gupta, A., and He, K. Non-local neural networks. In CVPR, 2018.
Zou, S., Guo, C., Zuo, X., Wang, S., Wang, P., Hu, X., Chen, S., Gong, M., and Cheng, L. Eventhpe: Event-based 3d human pose and shape estimation. In ICCV, 2021.
Wang, Y., Xu, Z., Wang, X., Shen, C., Cheng, B., Shen, H., and Xia, H. End-to-end video instance segmentation with transformers. In CVPR, 2021.

---

SpikeVideoFormer: An Efficient Spike-Driven Video Transformer
A. Proof of Proposition 3.1
B. Derivative of Normalized Hamming Similarity
Lemma A.1 UJohnson-Lnndensrruus Lemma on Binary Embedding (Jacques et al., 2013; Yi et al., 2015)). Let {zi)i=1 be set of M real-valued points, define its one bit quantication of the projections,
We define 1 E {1]1XD all-one vector. For the spike vec- tors qs, k2 E {0, 10,111xx) we rewrite the normalized Ham- ming similarity defined in Eq. (6) as:
b(x) = sign(Ar):
(11)
k.)) =1
11 (g + k!!)
where b(x) E 10, is the binary embedding of x E RC and E DxC is a projection matrix with each entry gen- erated independently from the normal distribution (0,11), 1). Given D> ld for any two points among M, we have
D ==1
9s (1 - k:,) + (1 - [0))TH
D
1
[. 1 + 1Tk; -24, Kis
D
p P((dz(bssb)) b,) de( de) 56) 2 D (12)
1
D D
1
1 + 1 1 -2 his
D
2
Here du is the normalized Hamming distance defined as
1
1
9, _
0.1+1' 1Tk -24 Ks
D
1
D
2 2
1 D dx b,) = Ck
1
(4 hi - 2g 1 - 21 T Ks + D)
Σ lbbr + byr),
= +
(13)
2D
1
= +
(2qx - -1) (2k: - 1).
(21)
2
2D
and de is cosine distance defined as
Therefore, for the spike query and key matrices, denoted
de(x,, 1
x |
as Q. K2 {0, (),D) and the all-ones matrix e
arccos
(14)
[I]NxD, we have the corresponding attention score ex- pression as:
π
According to Lemma I, we provide the proof of Proposi- tion 3.1 follows:
[Q, K,) = 2D (2Q. 1 -
(2K, 1)_ 1) (22)
Proof: By substituting z;, 2y and b;. by in Eq. (12) with & and ys hig respectively, we obtain:
After omitting the constant 1. we replace the dot-product similarity used in Eq. (3) with normalized Hamming simi- larity:
Idx k.) de k) < 8.
(15)
SDHA SNan
(2Q. 1)(2K, - 1) 1)"]v,
2D
Next, using the definitions from Eq. (6) and (7), we have:
O(N2D)
du(9,, k2) = 1 fre ko), de(g,h) k)) = -[(eel,
(16)
= SN2D ( (2Q, - 11
1) (1)
(23)
(17)
O(ND²)
where g(x)= = 1 ar)cos(()). Substituting these into the inequality, we get:
C. Spike-Driven Space-Time Attention
(1 (1- Sre(Uu, ka)) - (1-9(Jclu,))) s 8, (18)
The detailed architecture of three different spike-driven space-time attentions is shown in Fig. 5. For simplicity, we assume the hidden channel size of attention module is identical with the input channel size D and the number of attention head is M. The detailed description of architecture and FLOPs calculation 2 for three prototypes are presented as follows:
which simplifies to:
Fxlg,, ((C(c k:]) <
(19)
Finally, substituting this result into Eq. (12), we obtain:
Spike-Driven Space-Time Joint Attention: The input spike feature goes through three parallel linear, batch
k.) x(fc(a,))) - sd) 2 1 2e 3DD -2D (20)
When calculating FL.OPs, we assume the batch size B is 1 for simplicity.
12

---

SpikeVideoForman An Efficient Spike-Driven Video Transformer
Spike-Driven Space-Time Joint Attention
Spike-Driven Space- Time Hierarchical Attention
Spike-Driven Space-Time Factorized Attention
N
B x x X
B T N x D
B x T N
Split
Lineer
Libear
Lincer
RS
BN
IIN
BN
+
Q
Ks
Vs
Linor
Linar
Lineer
Finear
Qs
Kis
V:
Reshape Space
Rechape Space- Time TN
D
Qs
Vglun
Q5
panace
Spattal-Temporal SDHA
Sparial SDHA
Toms BB SZ
Reshupe Space
BT x
Liter
Temporal SDRA
Sparial
Concatemaie Channel
1 Liner
IN
Linarr
Linear
Lincer
BN
Spaking Neuron Layer
Qs
Ks
Restape Tiee BN
Temporal Spiking
Lineas
Linear Layer
Temporal SDHA
Linze
RetchNNom Layer
Figure 5. Architecture Details of three different spike-driven space-time attentions.
normalization and spiking layers to convert to spike query Qs. key Kg and value Vg. They will be re- shaped to x TN x and forwarded SDHA for spatiotemporal feature fusion. The output finally goes through linear and batch normalization layer. In total, the number of parameters are 4D2
Spike-Drived Space-Time Factorized Attention: The input spike feature will be first split into two branches with half of the channel size D The split features 2'
will go through five parallel linear, batch normaliza- tion and spiking layers to convert to spike query Qs. key KTme K and value Visim,, V/gimse The query Qs is reused for the following temporal and spatial hamming attention. The output is the concatenation of two branches and then goes through linear and batch normalization layer. In total, the number of parameters are 7D2.
FLOPs 2TN( D M
P 4TN(D" +
Lincur-BN BN
(24)
Ham, Alt. =O(TND²),
Spike-Driven Space-Time Hierarchical Attention: The input spike feature goes through three parallel linear, batch normalization and spiking layers convert to spike query Qs. key Kş and value Vs. They will be reshaped to BT x N x D and forwarded to SDHA for spatial feature fusion. The output then goes through linear and batch normalization layer. This process is repeated while the spike feature is reshaped to BN x T x D for temporal feature fusion. In total, the number of parameters are 8.22.
FLOPs 2T D/2 PM (T D M
2
Temporal Hom. Alt.
2NT D/2 F + 2T D2²+ D) (26)
M
2 2
Spatial Ham, An.
=T
D. Architecture Details
PM ATN + D)
FLOPs 2TN M
The detailed model architecture is presented in Tab. 7.
Spatial
2NT M
PM + ANT(D2 + D)
(25)
E. Power Consumption
Temporal Ham. Ats.
The computational and energy consumption of Spiking Neu- Networks (SNNs) are typically lower than that of Ar-
O(TND*)
13

---

SpikeVideoFormer:: An Efficient Spike-Driven Video Transformer
Table Architecture Details of SpikeVideoFormer, For simplicity, omit the temporal spiking layer here.
Channel (15M)
Channel
Blocks
Layers
Shape
(55M)
DownSamplex
Conv (kermeel113) stride=2)
Spik-Driven CNN Blockx1
SepConv (kerne1=777, stride=1)
H Tx
2 X W
X
32
64
(katneelCony(xeeran1=-xlx. stride=1)
DownSamplex
Conv (onv(kereel-3333, stride-2)
Spik-Driven CNN Blockx1
SepConv (kernee77k stride-1)
x
x 2C
128
stride=1)
DownSamplext
Conv Cormmeeel333322 stride-2)
SepConv (kermeee=1k))))1] str stride=1)
T
4C
128
256
Spik-Driven CNN Blockx2
ChannelConv (kernee=333) stride=1)
DownSamplex1
Conv( stride-2)
3
Spike-Driven Spatiotemporal
SDHA (channeel
T x
H
8C
256
512
16
16
Transformerx6
Channe1MLP (ratio=4)
DownSamplex
Conv (kernel-3x3, stride-2)
Spike-Drived Driven Spatiotemporal TransformerX2
SDHA (channel=10C)
T
H
18
x
W
16
x 10C
320
640
ChannelML2 (ratio-4)
tificial Neural Networks (ANNs). Following the analysis in (Zhou et al., 2022; Yao et al., 2024a), the energy cost (MAC)
Table 8. Results on ImageNet-1K dataset. All the SNN-hased methods trained directly without distillation. "Spike", "Param", "Step", ,, and "Ace" denote Spike-Driven, the number of parameters, spiking time-sepp and accuracy. The best result among the SNN
of ANNS is measured in ciwpiill--andaaccmmuee operations. Given the number of floating-poin operations (FLOPs) in a layer, the energy cost for ANNs can be pressed as:
methods are in bold.
Param Power Step Accf (M)
Methods
Architecture
Spike
AANSs = FLOPs * EMAC-
(27)
(mJ)
T
(%)
RSB (Wightman al., 2021)
60
53.4
1
where EMAC is the energy consumed per MAC operation. In contrast, SNNs only involve additions and extractions, which are less energy-intensive. The energy cost for SNNs measured in accumulate (AC) operations, and computations involving non-spiking neurons in the preceding layer can be skipped. Thus, the energy cost for SNNs is given by:
ANN
VIT (Dosovitskiy al., 2021)
86
81.0
1
79.7
ANN2SNN
VGG-16 (Hu et al., 2023)
138.4
44.9
7
73.0
SEW-Res (Fang al., 2021)
60.2
12.9
4
69.2
CNN-Based
MA-SNN (Yao et al., 2023)
x
78.4
7.3
4
76.3
SNN
MS-Res-SNN (Hu al., 2024)
SpikFormer
x
29.7
11.6
4
73.4
(Zhou et al., 2022)
66.3
21.5
4
74.8
EsNNs p + FLOPs EMAC:
(28)
15.1
4.0
1
71.5
SeteeSrikkFonmer (Yao et al., 2024a)
15.1
16.7
where P is the spiking rate. For reference, 32-bit floating-point operation in 45nm technology consumes EMAC = 4.6p./ for MAC and EAC = 0.9p// for AC op- erations (Horowitz, 2014).
Transformer -Based SNN
55.4
13.0
1
78.0
55.4
52.4
15.1
4.0
1
71.5
SpikeVidenFormer
15.1
16.8
4
73.6
(Ours)
55.4
13.1
1
78.5
55.4
52.6
4
79.9
F. ImageNet Pre-training
Implementation details. ImageNet-1K 1K (Deng et al., 2009) is widely used benchmark for vision tasks, containing 1.3M training and 50K validation images across 1K classes. Since this a single-image task without temporal modeling, the primary objective is pre-train the backbone model for the downstream video-based vision tasks. To maintain scalability and generalization with previous work, Meta- SpikeFormer, we adopt the same model architecture settings, utilizing channel sizes C' of 32 and 64, which result in 15.IM and 55.4M parameters, respectively. The input image size 224 x 224. The model is first trained with a time-step for 200 epochs on NVIDIA A6000 GPUs, followed
by fine-tuning with a time-step T 4 for an additional 20 epochs. All other training configurations remain consistent with those in (Yao et al., 2024a).
Experiment Results. In Tab. we compare Spike Former with previous methods in terms of parameter count, power consumption, and accuracy. Our method achieves 73.6% and 79.9% accuracy for the small (15.IM) and large (55.4) 4M) model sizes, respectively, surpassing the state-of- the-art Meta- Sptta-SpikeFrmer by 0.4% and 0.2%. Notably, the major distinction between these two methods lies in the
14

---

SpikeVideoFormer: An Efficient Spike-Driven Video Transformer
Table 9. Results on ImageNet-C dataset.
Methods
Noise Gaussian Shot Impulse Blur Defocus Glass Motion Zoom Weather Snow Frost Fog Bright Digital Contrast Elastic Pixelate JPEG
ResNet50
29.6
27.9
24.5
38.8
26.1
37.9
34.5
30.1
36.7 43.6
66.8
38.8
44.8
47.0
55.1
VeneRe
34,6
33.4
31.9
37.8
35.7
37.4
34.0
25.2
36.8
30.1
62.4
28.5
48.7
63.3
61.0
Ours
33.1
34.8
38.0
39.1
38,9
38.1
35.9
28.3
37.0
42.1
66.2
34.5
49.3
61.4
61.8
use of spike-driven Hamming attention in our design ver- sus dot- dot-product attention, underscoring the effectiveness of
Input Event Stream
Inpul Video Clip
FF W
our proposed approach. Additionally, our directly trained SNN model outperforms the ANN-based VIT (Dosovitskiy et al., 2021), achieving 79.9% accuracy compared to 79.7%, marking the first instance of an SNN model surpassing an Transformer-based ANN model in this benchmark.
Spike Vidkeeiddooooee oomer
Regression Head
We also evaluate our approach on the ImageNet-C dataset to assess object recognition performance in noisy en- vironments. We compare method with VOneRes- Net50 (Dapello et al., 2020) and ResNet50 (He et al., 2016) in Tab. 9.
Avetogg Fealin
BER
d
G. Video Classification
Figure 6. Workflow human pose tracking using our proposed SpikeVideoFormer The regression head consists of three parallel linear layers to regress the SMPL shape parameters.
For the Kinetics-400 dataset, we train our model for 30 epochs using the AdamW optimizer with a batch size of 32 on 8 NVIDIA A6000 GPUs. The initial learning rates are set to 6e-5 for the pre-trained backhone and 6e-4 for the randomly initialized calssification head. The learning rate is adjusted by a AnnaainAneel schedule with a maximum of 35 epochs. For all model variants, we sample 32 frames per video clip using a temporal frame stride of 2, and the spatial resolution to 224 x 224. During inference, adopt the 3 view strategy from (Liu et al., 2022) where each video is uniformly sampled into four temporal clips. For each clip, the shorter spatial side is resized to 224 pixels, and we then extract three 224 x 224 crops along the longer spatial axis. The final score is computed as the average across all views.
rameters, global translations, and 3D and 2D joint positions to train our model. The total loss function is defined as:
L = =Apuneff + Ashhape Lshape + Aurams Ltrams
(29)
+ >aDCaD + A2DC2D.
where Apose Ashape Atrans ATD, and AND are the respective loss weights. For the pose loss, we use the 6D representa- tion of rotations. To measure the difference between the predicted and target poses, compute the geodesic dis- tance in SO(3) as follows:
T 21
Tr (R (0) (8))
Cpose ΣV
arccos²
H. Human Pose Tracking
2
The workflow is illustrated in Fig. 6. The output spike features from Spike VideoFormer are passed through the regression head to predict parametric human poses over time, represented by the SMPL body model (Loper et al., 2015). In the regression head, we apply 2D average pooling, followed by three parallel linear layers regress the SMPL shape parameters a T SMPL pose parameters 0 E RTx72 x72 and global translations à E T respectively. Additionally, using predefined camera parameters, we can also compute the 3D and 2D joint positions.
where R(-) transforms the 6D rotational representation to the 3 3 rotation matrix, and is the joint index. For the other losses, we compute the Fuclidean distances between the predicted and target values:
T
Lahape
|1 (1)
11n11
Lians
||d م
t=1
LN
Σ
3 11?
t=1
Training loss. Following previous work (Zou et al., 2021), we use a combination of losses for the pose and shape pa-
T
24
LaD =
Σ
B 11?
t=1
15

---

SpikeVideoFormer: An Efficient Spike-Driver Video Transformer
Images (Iaput)
SpikFormer (Zhou et al., 2022)
Meta-SpiieFormer (Yao al., 2024a)
VideeeeeeeFrrrr (Ours)
Events (Input)
SpikFormer (Zhou al., 2022)
Meta-SpiikeFormer (Yao et al., 2024a)
SpikeVideoFormer (Ours)
Figure 7. Qualitative Results of Our Spike VideoFormer on Human Pose Tracking.
Additional Implementation Details. The input dimensions for both modalities are T x 256 256 x 3. The event stream is evenly split into 37 segments over time, with each segment transformed into an event frame. In these frames, pixel value of 0 indicates no events occurred at that position during the time period, while a pixel value of signifies that events occurred at that position. These event frames then resized match the input shape required by the pre-trained Spike VodeoFormer.
types of augmentation: (1) randomly selecting video clips event streams of varying durations (1, 2, 3, 4) seconds for T = 8 and (4, 8, 16, 32) seconds for T = 32 with temporal frame stride of (1. 2. 3, 4) correspondingly: (2) applying a random spatial rotation to the input clip, ranging from -20 to 20 degrees. During testing, the durations are and 16 seconds for T 8 and T 32 models, respectively.
I. Video Semantic Segmentation
For training, the T = 8 and T = 32 models are trained for 30 and 20 epochs, respectively, using batch size of 64 and 8 across 8 NVIDIA A6000 GPUs. The learning rate is initially set to 6e-4 for the head and 6e-5 for the pre-trained backbone. The learning rate is adjusted by a Cosine ingLR schedule with maximum of 35 and 25 epochs for the T 8 and T 32 models, respectively. To ensure robustness against both fast and slow motions, we apply two
The workflow is illustrated in Fig. 9. Our Spike FideoFormer encoder extracts four scales of spike features, each with the following dimensions: T x x
W. H W. x T H x 4 x 2C, 2
2
T x H x W are then separately forwarded into the Memory Read and Fuse modules. In this module, the feature from the last time step is used as the query, while the previous T time steps
& x 4C', and 7' x T6
H
X T6
W
x 2C. These features
16

---

SpikeVideoFormer: An Efficient Spike-Driven Video Transformer
space-time context
A6000 GPUs. The initial learning rate set to le-3 and is adjusted using
query context
schedule, with maxi- mum of 400 and 80 epochs for the two datasets, respectively, Finally, the Memory Read and Fusion module is trained for 100 epochs on CityScapes and 20 epochs on VSPW, with a learning rate of le-4.
right arm/elhow occluded
11:19
0.01
0.28
0.14
0.133
0.13
0.13
8:12
Spike Video oomer [Encoddr)
0.19
0.1
0.30
811
SpikeFPN (Deooder)
Quories (T T
Mensry Roid and
1001
3.00
0.19
0.19
0.1.
0.15
0.13
8.11
6 m
6- 042
11:12
0.71/
0:14
h:1
0.93
0.09
0.15
0.12
0.10
0.15
0.18
8.16
K
Keys (7 =
Ruchage Speco
space-time context
Bx
query context
Blanning Allentien
12
right arm/elbow occluded
Figure 9. Pipeline video semantic segmentation using SpikeV. ideoFormer as encoder, spike-driven Hamming attention as Mem- ory Read and Fusion module, and SpikeFPNN decoder.
2-
101
6.04
0099
1:
0..44
0/22
1:
0.15
Queries (T -
101
6.14
0.07
0:15
0.24
6.2G
0:13
0.15
8.03
0.06
0:15
0.22
0.36
6-
1m
8.03
0.07
8:15
0.25
0.22
13
6.1-
6.64
0:20
0.36
8-
6.03
0.06
0.3.4
0.24
0.22
0.333
0.28
Keys (T =
8)
Figure 8. Visualization of Attention Maps in Human Pose Tracking
of features serve as the key and value for cross-attention via SDHA. The fused features of the last step across all four scales are subsequently used as pyramid features in the decoder SpikeFPN, which predicts pixel-level semantic categories.
Figure I0. Qualitative Results of Our SpikeVideoFormer on VSS.
For training, we use crop sizes of 480 480 for the VSPW dataset and 768 768 for CityScapes. During testing, crop sizes of 480 x 863 and 512 x 1024 are applied to these datasets, respectively. Data augmentation during training includes random resizing, flipping, cropping, and photo- metric distortion. The encoder and decoder are trained for 300 epochs on the CityScapes dataset and 60 epochs on the VSPW dataset, with a batch size of 16 across NVIDIA
17


Reviews:
[{"review_id": "5cyBjz4MB6", "reviewer": "Reviewer_oaZD", "summary": "The paper introduces SpikeVideoFormer, an efficient spike-driven video Transformer that leverages normalized Hamming similarity and joint space-time attention to achieve linear temporal complexity. It outperforms existing SNN-based models in video classification, human pose tracking, and video semantic segmentation while matching ANN-based methods in accuracy with significant efficiency gains.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer oaZD, \n\nWe appreciate your time and effort in reviewing our paper. Below, we provide a point-by-point response to your questions.\n\n---\n\n**Claims And Evidence:**\n\n- **We report the latency in the table below**, based on tests conducted using the same hardware setup—a single A6000 GPU and an AMD EPYC 7543 CPU. The task is human pose tracking, comparing the performance of GLoT (ANNs) and our SpikeVideoFormer (SNNs). The input is an RGB video clip of shape $T\\times 256\\times 256\\times 3$, with a batch size of 1. We conducted tests using 1,000 samples and averaged the time consumption as the latency. As the temporal length $T$ increases from 8 to 32 (4x), GLoT (quadratic attention) experiences a 9.8x latency increase, whereas SpikeVideoFormer (linear attention) shows only a 4.6x increase. GLoT’s increase is lower than the expected ~16× due to its use of a ResNet followed by a Transformer, where ResNet has linear complexity concerning temporal length. \n\n|Method|Timestep|Power (mJ)|Latency (ms)|PA-MPJPE (mm)|\n|:-|:-|:-:|:-:|:-:|\n|GLoT (ANNs)| $T=8$ |487.5|**303**|39.9|\n||$T =32$| 4046.1|**2972 (9.8x)**|46.5|\n|SpikeVideoFormer (SNNs)|$T=8$|96.0|**235**|39.8|\n||$T =32$|391.2|**1087 (4.6x)**|47.5|\n|\n\n---\n\n**Methods And Evaluation Criteria:**\n\n- **Q1**: **Yes, our proposed SDHA can also benefit spike-driven transformers in the image domain.** As demonstrated in the table below (also in Appendix F, Table 8), applying SDHA leads to an accuracy improvement of 0.4% (15.1M model) and 0.2% (55.4M model) on ImageNet. In this experiment, our method follows the same model architecture as Meta-SpikeFormer, except for the inclusion of SDHA.\n\n|Method|Attention|Param|Power(mJ)|Top-1 Accuracy|\n|:-|:-:|:-:|:-:|:-:|\n|Meta-SpikeFormer|SDSA|15.1|16.7|73.2|\n|Ours|**SDHA**|15.1|16.8|73.6 (**+0.4**)|\n|Meta-SpikeFormer|SDSA|55.4|52.4|79.7|\n|Ours|**SDHA**|55.4|52.6|79.9 (**+0.2**)|\n|\n\n- **Q2**: We respectfully believe that our proposed method offers valuable contributions and introduces new insights compared to previous ANN-based works, specifically: \n  - Space-time joint attention in ANNs suffers from quadratic computational complexity, and most related methods propose different space-time attention designs to reduce this complexity (as shown in Table 1). However, to our knowledge, **we are the first attempt to show that space-time spiking attention designs share the same linear complexity.** The experiments demonstrate that our spike-driven solution achieves performance comparable to recent ANN-based methods while offering significant efficiency gains, with improvements of x16, x10, and x5 on three video-based tasks.\n  - **Our primary contribution lies in the design of the first Spiking Video Transformer** with effective spike-driven attention design (Proposition 3.1). This approach achieves SOTA performance compared to existing SNN approaches, with over 15% improvement on human pose tracking and video semantic segmentation.\n\nThanks for the helpful questions. We will clarify the uniqueness and novelty in the final version.\n\n---\n\n**Experimental Designs Or Analyses:**\n\n- **Q1**: We are sorry that the statement of our method is unclear than intended. **To our knowledge, we are the first to explore spiking video Transformer.** Thanks for the constructive comment. We will clarify the novelty in the final version. \n\n- **Q2**: **Other Transformer-based SNNs are image-based approaches** that only use spatial attention. For a fair comparison on video-based tasks, we adapted these approaches to incorporate space-time joint attention. This setting is applied consistently across all three tasks in our work. We will clarify this point in the final version to avoid confusion.\n\n---\n\n**Supplementary Material:**\n\n- Thanks for pointing out this issue. We will include the supplementary materials as an appendix at the end of the main manuscript in the revised submission.\n\n---\n\n**Other Comments Or Suggestions:**\n\n- We will carefully address the formatting errors in the table captions, correct the notations in the equations, and fix any typos in the final version.\n\n---\n\nWe value your feedback and will ensure that the above results and discussions are thoroughly addressed in the final revision.", "claims_and_evidence": "The authors emphasize achieving linear temporal complexity with the proposed SpikeVideoFormer, but there does not appear to be a comparison in terms of latency to support this claim.", "methods_and_evaluation_criteria": "This work primarily proposes SDHA and space-time joint attention to enhance the performance of a spike-driven video transformer.\n- Among them, SDHA appears to be a general method that is not limited to video transformers but can contribute to improving the performance of general SNN transformer architectures. In that case, could transformers for the image domain also benefit from SDHA? This is something I am curious about.\n- On the other hand, space-time joint attention has already been used in existing ANNs, as mentioned in Section 3.4. Aside from replacing attention with SDHA, the novelty seems insufficient for it to be considered a main contribution. Could the authors further clarify the distinguishing aspects?", "theoretical_claims": "I have checked the correctness of the proofs for the theoretical claims presented in the paper.", "experimental_designs_or_analyses": "- The choice of SNN baselines, which are spiking transformers used for image processing, may raise questions from readers. If the authors propose a video-specific model, shouldn't they compare it with other spiking video transformers? If none exist, they should clearly state that theirs is the first.\n- Among the three video-based vision tasks, space-time joint attention was applied to other Transformer-based SNNs only for human pose tracking (Table 2). Why is this considered a fair comparison? This approach does not seem consistent across other tasks.", "relation_to_broader_scientific_literature": "This work contributes to the broader scientific literature by expanding the role of spiking neural networks in the video domain.", "essential_references_not_discussed": "The necessary references are sufficiently cited in the paper.", "other_strengths_and_weaknesses": "Strengths\n- The writing is clear and easy to understand.\n- The use of Hamming similarity for attention scoring is interesting.\n- The effectiveness of the method is demonstrated through various video-based vision tasks.\n\nWeaknesses\n- Please refer to the points mentioned in other sections.", "other_comments_or_suggestions": "There are some formatting errors and typos in this paper.\n- All table captions should be placed above the tables.\n- Some notations for the equations in Section 3 appear to be missing.\n- The figure on page 4 needs a caption.\n- There are misplaced periods before and after a reference (p.2, line 99).", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g0Wyuovqtt", "reviewer": "Reviewer_9Qge", "summary": "The authors propose SpikeVideoFormer, an efficient spike-based Transformer to process videos with linear temporal complexity. Technically, a spike-based Hamming attention mechanism is proposed from a theoretical perspective. Then, the authors further analyze several spike-based attention modules for video processing tasks. Finally, three typical video tasks (i.e., classification, human pose tracking, and semantic segmentation) are conducted to evaluate the effectiveness of the proposed method. The results show that the proposed SpikeVideoFormer outperforms SOTA methods in accuracy and energy efficiency.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 9Qge,\n\nWe are grateful for your insightful feedback. Below, we provide a detailed response to each of your points.\n\n---\n\n**Other Strengths And Weaknesses:**\n\n---\n\n**W1 Inference Time (per video clip $T\\times 256\\times 256\\times 3$ as input)**\n\n- **We report the inference time in the table below**, tested on an A6000 GPU and AMD EPYC 7543 CPU for human pose tracking, averaged over 1,000 video clips with a batch size of 1. As $T$ increases from 8 to 32 (4x), GLoT (quadratic ANN attention) achieves the best performance but experiences a 9.8× increase in inference time, while VIBE (linear ANN-GRU) shows a 5.1x increase but performs the worst. Both SNN methods exhibit only 4.3× and 4.6× increases thanks to our proposed spiking space-time linear attention. \n\n|Method|Timestep|Power (mJ)|Inference Time (ms)|PA-MPJPE (mm)|\n|:-|:-|:-:|:-:|:-:|\n|VIBE (ANNs)| $T=8$ |392.1|**264**|46.3|\n||$T =32$| 1511.2|**1335**|53.6|\n|GLoT (ANNs)| $T=8$ |487.5|**303**|39.9|\n||$T =32$| 4046.1|**2972**|46.5|\n|Meta-SpikeFormer (SNNs)|$T=8$|95.7|**230**|45.7|\n||$T =32$|387.2|**1001**|54.5|\n|SpikeVideoFormer (SNNs)|$T=8$|96.0|**235**|39.8|\n||$T =32$|391.2|**1087**|47.5|\n|\n\n- **For edge device deployment**, SNNs, relying only on additions, significantly reduce power consumption and latency on neuron-computing devices [A, B]. ANNs, requiring floating-point multiplications, achieve high parallel acceleration on GPUs but at a high energy cost. On the AMD Xilinx ZCU104 [C], ANNs process at 691.2 GFLOPs/s, while SNNs reach 5529.6 GFLOPs/s—an 8x speedup. In 45nm technology [D], ANN multiplication consumes 4.6pJ, whereas SNN addition uses only 0.9pJ, a 5.1x energy reduction.\n\n[A] Neuromorphic computing at scale. Nature 2025.\n\n[B] Spike-based dynamic computing with asynchronous sensing-computing neuromorphic chip. Nature Communications 2024.\n\n[C] Firefly: A high-throughput hardware accelerator for spiking neural networks with efficient dsp and memory optimization. IEEE VLSI 2023.\n\n[D] 1.1 computing’s energy problem (and what we can do about it). IEEE ISSCC 2014.\n\n---\n\n**W2 Event**\n\n- We have explored long-term event-based human pose tracking in Table 2. \n- We present results on event-based action recognition (HAR-DVS [E]) and event-based semantic segmentation for scene understanding (DDD17 [G]) in the table below. \n\n|Method|Param|Acc|\n|:-|:-:|:-:|\n|ACTION-Net [F] (ANNs)|27.9M|46.9|\n|Meta-SpikeFormer (SNNs)|15.0M|47.5|\n|SpikeVideoFormer (Ours)|15.0M|47.9|\n|\n\n|Method|Param|mIoU|\n|:-|:-:|:-:|\n|EV-SegNet [G] (ANNs)|13.7M|54.8|\n|SpikeVideoFormer (Ours)|17.8M|55.5|\n|\n\n[E] Hardvs: Revisiting human activity recognition with dynamic vision sensors. AAAI 2022.\n\n[F] Action-net: Multipath excitation for action recognition. CVPR 2021.\n\n[G] EV-SegNet: Semantic segmentation for event-based cameras. CVPR 2019.\n\n---\n\n**W3 Energy**\n\n- Please refer to our response to W1 Inference Time. We have included latency (inference time) to evaluate efficiency in long-term tasks using our linear complexity method. Additionally, we have discussed the potential of deploying our SNN on specialized neurocomputing devices to further enhance its efficiency.\n\n---\n\n**W4 Ablation**\n\n- We have conducted additional analysis of time steps and hyper-parameters on Human Pose Tracking (evaluated on PA-MPJPE). The results are shown in the tables below.\n\n|Timestep $T$|4|8|16|24|32|\n|:-|:-:|:-:|:-:|:-:|:-:|\n|PA-MPJPE|39.8|39.8|42.7|45.6|47.5|\n|\n\n|Channel size $C$|32|48|64|\n|:-|:-:|:-:|:-:|\n|Param|15.1M|31.3M|55.4M|\n|PA-MPJPE|44.4|41.7|39.8|\n|\n\n|Blocks|4-Transformer|1-CNN+3-Transformer|2-CNN+2-Transformer|3-CNN+1-Transformer|4-CNN|\n|:-|:-:|:-:|:-:|:-:|:-:|\n|Param|12.4M|13.8M|15.1M|16.5M|18.0M|\n|PA-MPJPE|40.7|40.1|39.8|45.6|54.9|\n|\n\n---\n\n**W5 Clarity**\n\n- Our SpikeVideoFormer differs from existing Video Transformers in two aspects:\n  - Unlike **ANN-based Video Transformers**, which focus on **reducing quadratic complexity in space-time attention**, we highlight that spike-driven attention inherently achieves linear complexity (Table 1), making our approach more efficient and scalable.\n  - Unlike **existing SNN-based Transformers**, which focus on **single-image tasks only**, we introduce the first Spiking Video Transformer with an effective spike-driven attention mechanism (Proposition 3.1).\n\n---\n\n**Questions For Authors:**\n\n- **Q1:** A video comprises a sequence of RGB images with 3×8 bits/pixel. In contrast, an event stream can be represented as a sequence of event frames with 1 bit/pixel (24x more sparse), indicating event occurrences within the time interval.\n\n- **Q2:** We emphasize videos due to their practicality in real-world applications, highlighting the need for efficient and effective processing methods. Following prior works, we also incorporate event-based tasks in our experiments to further demonstrate the applicability of SNNs.\n---\n\nWe are grateful for your suggestions and will ensure that the above results and discussions are reflected in the final revision.", "claims_and_evidence": "Yes, the claims are very clear.", "methods_and_evaluation_criteria": "Yes, the authors select three typical video tasks to verify the effectiveness of SpikeVideoFormer.", "theoretical_claims": "Yes，I have checked the correctness of all proofs.", "experimental_designs_or_analyses": "The effectiveness of the proposed method has been extensively validated through numerous experiments; however, certain ablation studies warrant further exploration to provide deeper insights.", "relation_to_broader_scientific_literature": "Indeed, Transformers have become the primary architecture in deep learning. Investigating spike-based Transformers is particularly valuable from a power consumption standpoint. The authors introduce a new attention mechanism for video processing, advancing the field beyond prior approaches.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n\n1. Investigating low-power versions of Transformer architectures is a highly meaningful research direction.\n\n2. The authors propose a spike-based Hamming attention mechanism and provide extensive theoretical proofs to support it.\n\n3. The authors validate the effectiveness of SpikeVideoFormer on three downstream tasks and supplement the paper with detailed appendices for further clarification.\n\n\nWeaknesses:\n\n1. Inference Time Analysis. Although the authors provide a complexity analysis, the inference time of SpikeVideoFormer is not reported, including the inference times of several open-source baseline methods. It would be valuable to compare the inference times and discuss the feasibility of deploying SpikeVideoFormer-like architectures on edge devices. This analysis would significantly enhance the practical relevance of the work.\n\n2. Real-World Event-based Vision Tasks. The selection of three typical video tasks is commendable. However, the authors do not explore real-world event camera tasks, such as long-term event stream processing, action recognition in real-world videos, or scene understanding. Including such tasks would further demonstrate the versatility and applicability of the proposed method in practical scenarios.\n\n3. Energy Efficiency Analysis. While the authors reference previous methods to analyze the energy efficiency of SNN algorithms, the use of AC or MAC operations for power consumption calculations may not be entirely convincing for SNNs. Given the critical importance of this metric, the rationale behind this approach should be thoroughly justified. The authors are encouraged to provide their insights or at least discuss this limitation in detail, as it significantly impacts the credibility of the energy efficiency claims.\n\n4. Ablation Studies and Parameter Analysis. The ablation experiments in the paper are relatively limited. Although the authors conduct three types of experiments, which may constrain the available space, it is essential to discuss the contributions of key parameters and modules. For instance, a more granular analysis of simulation time steps, hyperparameters would provide deeper insights into the proposed method's design and performance.\n\n5. Some Clarity. The authors should clarify how SpikeVideoFormer differs from existing Video Transformers. Beyond replacing activation functions with binary spikes, what are the unique design elements of this novel Spike Transformer architecture? A detailed discussion on these aspects would help readers better understand the innovation and contributions of the proposed method.", "other_comments_or_suggestions": "No", "questions_for_authors": "Please see the weaknesses and response each comment. Besides, two questions are listed below:\n\n1. In Table 2, what is the difference between video and event stream as inputs for SpikeVideoFormer?\n\n2. Why does the Spike-based Transformer structure emphasize video? Can it not process event streams? Doesn’t the title \"SpikeVideoFormer\" make the application seem too narrow?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rZ4Ks5JuWy", "reviewer": "Reviewer_4RFZ", "summary": "The authors present a novel model called the SpikeVideoFormer – a transformer network based on Spiking Neural Networks (SNN). They use Spike-Driven Hamming Attention (SDHA) instead of the usual dot product based self-attention. They claim their network to have a linear temporal complexity compared to the other model architectures that they explored. They show results on 3 tasks – video classification, video semantic segmentation and human pose tracking, achieving better performance compared to their ANN counterparts.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 4RFZ,\n\nWe sincerely appreciate your time and effort in reviewing our paper. Please find our point-by-point responses to your comments below.\n\n---\n\n**Claims And Evidence:**\n\n- Thanks for the value suggestion. Normally, Power = Watts * Time. According to [B, C], when comparing ANNs and SNNs, we typically assume the hardware operates for the same duration. **Therefore, computational efficiency measured in power is equivalent to that measured in watts.** As suggested by other reviewers, we have also reported latency (inference time) as an additional evaluation metric for computational efficiency in our response to Reviewer 9Qge (W1 Inference Time).\n- ANNs rely on **floating-point multiplications**, where GPUs can accelerate in parallel but with high energy costs. In contrast, SNNs use **binary spikes to propagate, requiring only additions**—a special case (0/1 binary matrix) that GPUs can speed up as well. However, power consumption and processing time can be further reduced on specialized neuron-computing devices [A, B], where multiplication operator is replaced with much faster addition operators.\n\n[A] Neuromorphic computing at scale. Nature 2025.\n\n[B] Spike-based dynamic computing with asynchronous sensing-computing neuromorphic chip. Nature Communications 2024.\n\n[C] Firefly: A high-throughput hardware accelerator for spiking neural networks with efficient dsp and memory optimization. IEEE VLSI 2023.\n\n---\n\n**Methods And Evaluation Criteria:**\n\n- **We evaluate our approach on the ImageNet-C dataset** to assess object recognition performance in noisy environments. Notably, Extreme Image Transforms [B] lacks ImageNet-C results, so we compare our method with VOneResNet50 [A] and ResNet50 [C].\n\n||Noise|Gaussian|Shot|Impulse|Blur|Defocus|Glass|Motion|Zoom|\n|:-|:-:|:-:|:-:|:-:|:-|:-:|:-:|:-:|:-:|\n|ResNet50 (25.6M) [F]||29.6|27.9|24.5||38.8|26.1|37.9|34.5|\n|VOneResNet50 [D]||34.6|33.4|31.9||37.8|35.7|37.4|34.0|\n|Ours (15M)||33.1|34.8|38.0| |39.1|38.9|38.1|35.9|\n|\n\n||Weather|Snow|Frost|Fog|Bright|Digital|Contrast|Elastic|Pixelate|JPEG|\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|ResNet50 (25.6M) [F]||30.1|36.7|43.6|66.8||38.8|44.8|47.0|55.1|\n|VOneResNet50 [D]||25.2|36.8|30.1|62.4||28.5|48.7|63.3|61.0|\n|Ours (15M)||28.3|37.0|42.1|66.2||34.5|49.3|61.4|61.8|\n|\n\n[D] Simulating a primary visual cortex at the front of CNNs improves robustness to image perturbations. NeurIPS 2020.\n\n[E] Extreme Image Transformations Facilitate Robust Latent Object Representations. arXiv 2023.\n\n[F] Deep residual learning for image recognition. CVPR 2016.\n\n---\n\n**Supplementary Material** & **Other Strengths And Weaknesses:**\n\n- We have included ground truths for human pose tracking and the spike attention maps on the [anonymous github page](https://anonymous.4open.science/w/AnonymousSpikeVideoFormer-5E5A/).\n\n---\n\n**Essential References Not Discussed:**\n\n- Visual cognitive neuroscience studies how factors like attention, motivation, emotion, and expectation shape visual perception and cognition [1]. While the first three enhance relevant stimuli processing, expectation suppresses predictable inputs [2-3]. Predictive processing suggests perception is inference-driven, refining sensory input through internal models shaped by context and experience [4]. Beyond vision, the visual cortex processes object names and activates in blind individuals, indicating broader cognitive roles in memory, imagery, and language [5-7]. Though its full scope remains debated, these insights inspire the brain-inspired spiking neural network (SNN) approach for complex visual tasks, enabling high-speed, low-energy neural processing.\n\n[1] The role of context in object recognition. Trends in Cognitive Sciences 2007.\n\n[2] How brains beware: neural mechanisms of emotional attention. Trends in Cognitive Sciences 2005.\n\n[3] Expectation (and attention) in visual cognition. Trends in Cognitive Sciences 2009.\n\n[4] An integrative, multiscale view on neural theories of consciousness. Neuron 2024.\n\n[5] Object domain and modality in the ventral visual pathway. Trends in Cognitive Sciences 2016.\n\n[6] The human imagination: the cognitive neuroscience of visual mental imagery. Nature Reviews Neuroscience 2019.\n\n[7] Reevaluating the sensory account of visual working memory storage. Trends in Cognitive Sciences 2017.\n\n---\n\n**Questions For Authors:**\n\n- The performance gain stems from space-time joint attention. As shown in our ablation study (Table 6), using spatial-only attention results in pose tracking error (PA-MPJPE) increase of more than 36%. Moreover, GLoT is limited by the quadratic complexity of ANN-based attention, requiring image features to be represented as a high-level single vector. In contrast, our SNN's linear complexity allows for more detailed spatial and temporal feature fusion through spike-driven attention.\n---\n\nWe are grateful for your thoughtful comments and will carefully integrate all results and discussions into the final version.", "claims_and_evidence": "The authors made claims about the computational efficiency of their model with power measured in milli Joules instead of Watts, which is what would be expected when using large power systems like GPUs. They should do a comparison in terms of Watts, especially when comparing the ANNs. \nAdditionally, ANNs benefit from the matrix multiplications being sped up by GPUs, does the same hold true for SNNs? Authors should give insights/comparison along those lines as well.  \nThe authors should give more specific insights into their training procedures w.r.t. hardware, compute costs, batch sizes,  etc.", "methods_and_evaluation_criteria": "I like the approach authors took in evaluating SpikeVideoFormer on a wide variety of tasks – segmentation, video understanding, human pose tracking. Given the SNNs ability to mimic the neuronal firing activity in the brain, they could also present a few tasks from cognitive psychology. For the primate visual system’s ability in tracking long range motion, they could evaluate their model on tasks like PathTracker for video or PathFinder for images. \nAn easier evaluation might be object recognition in noisy environments like Imagenet-C, and compare their methods against models of visual system like VOneNet or Extreme Image Transforms.", "theoretical_claims": "Proposition 3.1 is the theoretical aspect in the paper with its proof in the Appendix. On a broader look, it seems well written.", "experimental_designs_or_analyses": "The three tasks described in the paper for evaluating are set up well, besides my other comments related to reporting the metrics like watts and evaluating on tasks from cognitive psychology.", "relation_to_broader_scientific_literature": "The authors present an important and novel architecture useful for a lot of downstream video related tasks. This work is relevant to the broader community of neuro-inspired computing with potential applications in long term tracking.", "essential_references_not_discussed": "Given the authors talk about the affinity of their network to the human brain, they should include relevant references on the cognitive science literature for fundamental tasks like visual object recognition, tracking, etc.", "other_strengths_and_weaknesses": "Authors show impressive results in terms of their performance compared ot the other state-of-the-art literature for ANNs and SNNs. One of the ways to strengthen the paper for the reader would be to show the internal workings of the network through some kind of spike-maps (akin to saliency maps) so that the reader can get insights into where the network focuses. This would give insights into some of the explainability metrics of the model.", "other_comments_or_suggestions": "Some simplification of the writing to make the ideas more cogent would always help the reader.", "questions_for_authors": "For long-term tracking in the human pose tracking, authors mention their results being close to GLoT in ANNs. Did the authors try to improve the results with something like a memory component or gating to improve the long-term tracking abilities of the model?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rw6ZjweqaP", "reviewer": "Reviewer_ZrRF", "summary": "This manuscript introduces a video-based transformer model that implements spiking neural networks (SNNs) and Convolutional Neural Networks (CNN). The work highlights the efficiency of the proposed model in video-related tasks, particularly focusing on computational (parameters) and power efficiency.\nA key contribution is Hamming Attention, a mechanism that solves the dot-product problem in the SNN within the attenrion mechanism, while maintaining linear computational scaling with respect to tokens (spatial/temporal dimensions) (O(TND²)).\n\n\nThe model is validated across three tasks:\n\n-Human Pose Tracking\n\n-Video Classification\n\n-Video Semantic Segmentation\n\nafter rebuttal, I'll keep my recommendation", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ZrRF,\n\nWe greatly appreciate your time and effort in reviewing our work. Below are our point-by-point responses to your comments.\n\n---\n\n**Experimental Designs Or Analyses:**\n\n- Thanks for the constructive comment. For a qualitative comparison of video semantic segmentation, please refer to _Supp Figure 3_ on the [anonymous GitHub page](https://anonymous.4open.science/w/AnonymousSpikeVideoFormer-5E5A/). Additionally, video results are provided in _Supp Figures 5 and 6_. **The source code, results, and project website will be publicly released.**\n\n---\n\n**Relation To Broader Scientific Literature** & **Essential References Not Discussed:**\n\n- **We have cited this paper [A] (Line 477) in our work** and also applied in the experiment of Video Semantic Segmentation (Line 386). Integer-valued spike representation proposed in the suggested paper represents spikes as a set of integers, e.g. {0,1,2,3} (Integer-LIF=4), rather than {0,1} only. During inference the integer spike can be separated as a sum of {0,1} spikes, e.g., 3= 1+1+1, 2=1+1+0. This separation maintains spike-driven efficiency of SNN methods, while improving the model's representation ability.\n- **We further compare our method on the object detection task** against this work and Meta-SpikeFormer, as shown in the table below. Following prior work, we use spiking-based YOLO as the detection head and evaluate on the COCO 2017 dataset. Our method surpasses Meta-SpikeFormer by 0.6 mAP but lags behind SpikeYOLO by 0.2 mAP. However, unlike SpikeYOLO, which is specifically designed for object detection, our method is more general and applicable to a wide range of downstream tasks.\n\n|Method|Param|Timestep|Integer-LIF|mAP@50|\n|:-|:-:|:-:|:-:|:-:|\n|SpikeYOLO [A]|13.2M|1|4|59.2|\n|Meta-SpikeFormer|16.8M|1|4|58.4| \n|SpikeVideoFormer (ours)|16.9M|1|4|59.0|\n|\n\n[A] Integer-valued Training and Spike-Driven Inference Spiking Neural Network for High-Performance and Energy-Efficient Object Detection. ECCV 2024.\n\n---\n\n**Other Strengths And Weaknesses:**\n\n- Thanks for the valuable suggestion. Please refer to the response above under **Relation to Broader Scientific Literature** for the results related to the object detection task.\n- We present recent advancements in ANN-based Transformers for semantic segmentation as follows.\n\n  - Recent advancements in ANN-based Transformers have greatly enhanced semantic segmentation. SETR [1] pioneers a sequence-to-sequence approach, employing a pure Transformer encoder without convolutional layers. Segmenter [2] builds on a ViT backbone pre-trained on ImageNet and incorporates a mask transformer decoder to capture global context. SegFormer [3] further optimizes Transformer-based segmentation with a hierarchical encoder and a lightweight MLP decoder while eliminating positional encoding. Mask2Former [4] refines segmentation by restricting cross-attention to foreground regions using a masked attention operator. Recent CFFM [5] introduces coarse-to-fine feature assembling and cross-frame feature mining to capture both local and global temporal contexts. In contrast, we explore spiking video transformers to develop a faster, more energy-efficient approach for this task.\n\n[1] Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. CVPR 2021.\n\n[2] Segmenter: Transformer for semantic segmentation. ICCV 2021.\n\n[3] Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. ICCV 2021.\n\n[4] Masked-attention mask transformer for universal image segmentation. CVPR 2022.\n\n[5] Learning local and global temporal contexts for video semantic segmentation. IEEE TPAMI 2024.\n\n---\n\nWe sincerely appreciate your feedback and will ensure that all results and discussions are thoroughly reflected in the final version.", "claims_and_evidence": "-Solving dot-product's problem in SNN, replacing this with the Hamming Attention\n\n-Linear complexity with respect to tokens (spatial/temporal dimensions) (O(TND²))\n\n-Validation of the model in three task, surpassing the State-Of-The-Art in the SNN category.", "methods_and_evaluation_criteria": "-Dot product:\n\nBecause SNNs are sparse and do not have signals/elements at certain points (When Spike Query Contains No Elements), the dot product produces erratic Attension Maps.\n\n-Lineal complexity:\n\nIt is demonstrated that the model scale linearly when longer sequences are processed\n\n-The model is validated across three tasks:\n\nHuman Pose Tracking, Video Classification, Video Semantic Segmentation", "theoretical_claims": "-The derivative of Normalized Hamming Similarity is proved\n\n-The proposed Hamming-based attention for SNN is proved\n\n-The linear complexity with respect to the length of the tokens is experimentally proved", "experimental_designs_or_analyses": "No code or video is provided to veryffi the autentisity of the results\n\nIn attachments we can see some qualitative results comparing the task of human posse tracking.\n\nThe results of other models in the Video Semantic Segmentation task are not shown.", "relation_to_broader_scientific_literature": "While I am not deeply related, an improvement related to meta-spikeformer is made\n\nLuo, X., Yao, M., Chou, Y., Xu, b., Andli, g. Integervalued Training and Spike-Driven Inference Spiking Neural Network for High-Performance and Energy-Efficien to Bject Detection.eccv, 2024.\n\nHowever, Object Detection's task is omitted", "essential_references_not_discussed": "Object Detection's task is omitted", "other_strengths_and_weaknesses": "-Strengths\n\nThe manuscript is well-written and includes a wide variety of experiments supported by mathematical proofs.\n\n\n-Weaknesses\n\nWhile the manuscript focuses on developing spiking neural networks (SNNs), it omits addressing the task of Object Detection.\nAlso, recent advancements in transformers (ANN-based) for Semantic Segmentation are not adequately covered in the related work.", "other_comments_or_suggestions": "No comments", "questions_for_authors": "No questions", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received consistently positive reviews. The authors did an excellent job addressing the initial concerns raised by the reviewers through a clear and thorough rebuttal. The AC concurs with all reviewers and appreciate the thoughtful effort put into the response.

In addition, the AC believes this paper makes a solid contribution, presenting a novel method supported by impressive results. Therefore, the paper is recommended for acceptance. Please ensure that the clarifications provided in the rebuttal are incorporated into the camera-ready version to further strengthen the final submission.
Decision: Accept (poster)