Fully Spiking Neural Networks for Unified Frame-Event Object Tracking
Authors: Jingjun Yang
Liangwei Fan
Jinpu Zhang
Xiangkai Lian
Hui Shen
Dewen Hu
Venue: NeurIPS 2025 poster ()
Keywords: Object tracking
Spiking neural networks
Frame-Event fusion
Primary area: applications
TL;DR: a fully Spiking Frame-Event Tracking framework

Abstract: The integration of image and event streams offers a promising approach for achieving robust visual object tracking in complex environments. However, current fusion methods achieve high performance at the cost of significant computational overhead and struggle to efficiently extract the sparse, asynchronous information from event streams, failing to leverage the energy-efficient advantages of event-driven spiking paradigms. To address this challenge, we propose the first fully Spiking Frame-Event Tracking framework called SpikeFET. This network achieves synergistic integration of convolutional local feature extraction and Transformer-based global modeling within the spiking paradigm, effectively fusing frame and event data. To overcome the degradation of translation invariance caused by convolutional padding, we introduce a Random Patchwork Module (RPM) that eliminates positional bias through randomized spatial reorganization and learnable type encoding while preserving residual structures. Furthermore, we propose a  Spatial-Temporal Regularization (STR)  strategy that overcomes similarity metric degradation from asymmetric features by enforcing spatio-temporal consistency among temporal template features in latent space. Extensive experiments across multiple benchmarks demonstrate that the proposed framework achieves superior tracking accuracy over existing methods while significantly reducing power consumption, attaining an optimal balance between performance and efficiency.

Fully Spiking Neural Networks for Unified Frame-Event Object Tracking
Jingjun Yang"
Liangwei Fan"
Jinpu Zhang' Xiangkai Lian College of Intelligence Science and Technology
Hui Shen' Dewen Hu
National University of Defense Technology
(yjingjun, fanl Sanlinngee, zhangjinpu, lianxiangkai, shenhui, dwud) edu.. cn
Codes and models: https://githui: htspa///oocooocttott/////sseer
Abstract
The integration image and event streams offers a promising approach for achiev- ing robust visual object tracking in complex enviromments. However, current fusion methods achieve high performance the cost of significant computational overhead and struggle to efficiently extract the sparse, asynchronous information from event streams, failing to leverage the energy-efficient advantages of event-driven spiking paradigms. To address this challenge, Event Tracking framework called SpikeFET. This network achieves synergistic integration of convolutional local feature extraction and Transformer-based global modeling within the spiking paradigm, effectively fusing frame and event data. To overcome the degradation of translation invariance caused by convolutional padding, we introduce Random Patchwork Module (RPM) that eliminates posi- tional bias through randomized spatial reorganization and learnable type encoding while preserving residual structures. Furthermore, we propose a Regularization (STR) strategy that overcomes similarity metric degradation from asymmetric features by enforcing spatio-temporal consistency among temporal tem- plate features latent space. Extensive experiments across multiple benchmarks demonstrate that the proposed framework achieves superior tracking accuracy over existing methods while significantly reducing power consumption, attaining an optimal balance between performance and efficiency.
propose the first fully Spiking Frame-
Introduction
Visual target tracking has important application value in areas such as automatic driving and intelligent surveillance. Although traditional frame-based tracking methods [ 3 demonstrate satisfactory performance in conventional scenarios, robust tracking under complex environments involving low- light conditions, over-exposure, and high-speed motion remains a challenge. Event cameras offer new possibilities for tracking tasks due to their high dynamic range, microsecond response, and low power consumption [5), but their sparse and asynchronous characteristics makes it difficult to capture target appearance features. Some researchers [6, the performance and robustness of tracking by integrating motion information from frames with appearance information from events. However, most Artificial Neural Networks (ANNs)-based methods rely dense texture information for tracking and struggle to effectively model sparse event data. In contrast, Spiking Neural Networks (SNNs) demonstrate exceptional compatibility and ultra-low energy consumption advantages event stream processing through their bio-inspired spatiotemporal dynamics and event-driven sparse computational paradigm 9 [0]
7 g] have attempted to improve
Recently, an
'Theee authors contributed equally this work.
Correspondence
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

AUC Power Model
65
AUC (15)
SpikuFET-Trny
SOSTRNG
GEUTRIA
0
100
200
400
800
Theoretical Power (mJ)
GT
MonTnkakk
SDSTrack
VIPT
Ours
Figure 1: SpikeFET versus other tracking methods on COESOT.
Figure 2: Visualization results of SpikeFET-Base in low light and overexposure scenarios.
F-SpikeFormer 11 based on Metaformer 12 demonstrated that SNNs can achieve comparable performance to classical ANN-based vision transformer models in classification tasks. However, the application of SNNs in object tracking remains underdeveloped, facing critical bottlenecks in architecture-modality co-optimization. Current research predominantly focuses on ANN-SNN hybrid
architectures for unimodal tracking 13 14] While [15] extends this framework to multimodal
tracking, it suffers from compromised energy efficiency. Notably, SDTrack [16] implements fully spike-driven network but remains constrained to unimodal event-stream inputs, with interpolated noise in feature alignment causing error accumulation and performance degradation.
To address these challenges, we develop novel fully Spiking Frame- Event Tracking framework, named SpikeFET. We develop hierarchical tracking architecture with dual-branch feature extraction, single-branch fusion and bual-h prediction modules. This design effectively integrates local feature representation of convolutional networks with global modeling capabilities of Transformers, enhancing cross-modal fusion performance. In order to address the adverse impact of positional bias introduced by padding 11 18 in the convolution module on tracking accuracy, we propose
simple yet effective Randomized Patchwork Module (RPM). This module innovatively reorganizes target spatial distributions by randomly composing initial templates, online updated templates, and search frames into fused rectangular patches, while introducing learnable type encoding to explicitly identify the logical positions of each image patch. Without removing padding in residual units, RPM effectively mitigates padding-inducee degradation of translation invariance, significantly enhancing model robustness against target positional variations. In addition, further address the asymmetric boundary features of adjacent temporal template frames caused by RPM. we develop a Spatial- Temporal Regularization (STR) strategy that enhances both the temporal consistency between dual template target features in the latent space and the spatial consistency of boundary features, thereby improving the robustness of similarity measurement. In particular, SpikeFETT-iny achieves 2.7% higher AUC score on the FE108 8 dataset compared to the state-of-the-ae SDSTrack [19], while demonstrating 39x lower power consumption.
In summary, the main contributions of this paper can be summarized as follows:
We propose unified frame-event spiking tracker termed SpikeFET. To the best of our knowledge, it is the first work that employs a fully spiking neural network for unified frame-event object tracking.
We propose simple yet effective RPM that mitigates positional bias introduced by convolu- tional padding through randomized spatial reorganization, thereby significantly enhancing the model's robustness to target position variations.
We develop STR strategy that overcomes similarity degradation from asymmetric features by enforcing spatio-temporal consistency among temporal template features in latent space, enhancing the tracking accuracy and stability of the model.
Extensive experimental validation on multiple frame-event tracking benchmarks fully vali- dates the effectiveness our proposed SpikeFFET, and achieves an optimal balance between computational power consumption and performance and parameters, shown in Fig. !

---

2 Related Work
Visual Object Tracking. In recent years, deep learning has driven remarkable advancements in visual target tracking. Current mainstream approaches fall into three categories: First, CNN-based trackers like [7] utilize convolutional backbones for feature extraction and employ correlation mechanisms for target localization. However, their limited receptive fields restrict long-range dependency modeling, leading to reduced robustness in scenarios with rapid motion and occlusions. Second, hybrid Crn-Transf frameworks exemplified by TransT [2] and TMT [20] address this limitation by incorporating self-attention mechanisms into convolutional feature encoding, significantly improving spatio-temporal context modeling through complementary architecture design. Third, pure Transformer-bant based trackers S 21 demonstrate superior global relationship capture capabilities by unifying feature extraction and interaction within an antention-driven framework, showing promising potential for comprehensive context understanding.
Frame-Event based Object Tracking With the notable strengths of event data in low-light, fast- motion, and other complex scenes, frame-event fusion methods for single-object tracking have been gradually gaining attention in recent years. Some approaches 66 7 13
22 23 are based on
cross-modal fusion frameworks, like CEUTrrak [7]. simplify traditional two-branch architectures with a unified Transformer for synchronized multimodal feature extraction. TENet [23] improves event feature representation with lightweight multi-scale pooling and cross-modal mutual guidance for enhanced complementarity. Meanwhile, other methods (24 25 introduce prompt learning strategies, such as ViPT (25], which refines event features using pre-trained image models and attention mechanisms. In this paper, we propose dual- seaal--nnll-ddul framework for frame-event fusion, featuring independent feature extraction branches for each modality in the early stage and decoupled tracking heads after fusion to achieve efficient inter-modal complementarity and precise localization.
Spiking Neural Networks for Object Tracking SNNs show superior performance and efficiency in 27 28 through event-driven, low-power operation, excelling in time-sensitive domains like dynamic object tracking with bio-inspired asynchronous computation. Early frame- based SNN trackers [2] 31 integrated Siamese networks with temporal encoding using multi-step methods, and then compressed SNNs into single step models using knowledge distillation. For event- driven tracking B
vision tasks [T 26
32 33 STNet [L). and SNNTrack [14] enhanced spatiotemporal feature extraction but faced challenges in scenarios with insufficient texture information. SDTrack [16) is the first transformer-baat fusion models MMHT (15) utilized ANN-SNN hybrid architectures for spatiotemporal integration but did not fully exploit the energy-efficient advantages of SNNs. To overcome these limitations, we present SpikeFET,, the first fully spiking frame-event tracker. By adopting fully SNN-based multimodal fusion framework, SpikeFET fully unleashes the spatiotemporal modeling capabilities of spiking networks while preserving their energy-efficins computational properties.
spike-drive driven tracker, but also lacks robustness to spatial offsets. Frame-event
3 Methodology
As shown in Fig. 3 the overall framework of SpikeFET adopts a dual-single-duss structure based on SNN transformer. First, the image frames and event temporal frames are randomly combined using the RPM to mitigate the padding effects during feature extraction by convolutional layers. Subsequently. the combined multi-modal images are fed into feature extraction network, where Former Spike Blocks are employed to comprehensively extract features. Positional encoding, modality encoding, and type encoding are then incorporated into the feature embeddings, followed by dtrection-consister concatenation of the two modal images. The TransFormer Spike Block jointly processes and correlates these embeddings. Meanwhile, STR is introduced for dual template frames to maintain spatiotemporal consistency. Finally, the resulting feature embeddings are delivered to an SNN-based center-tracking head for prediction, with constraint to ensure consistency between the response maps generated by both modalities.
3.1 Spiking Frame-Event Tracking Network
In order to enhance the cross-modal feature fusion capability, we design a framework that integrates a dual branch feature extraction network and a dual modal response graph fusion mechanism. This framework consists of three core modules: dual branch modal specific feature extraction network,
3

---

Input Processing
Feature Extraction
Feature Fusion
Tracking
Featurm
Saytang good
Contoomme
Roegenoe
Irpot
Conrat +
Acki
CarsFurree
Block
Spiieo Ulock
Block
Moduliyy Encodings:
Liead
Fastus Map
Eaczalagg
Spike Block
IransP Block
Sepurvibe Spli: Comcoraton
Sepemblle Splie Commitation
Cuary
DN
Lawur
Liwar
Linsr
EN
Crmss Staped
Chers
Classdd Convoiatoon
Emns-Shaped Wirdew
Chasse MID
Spiliag Naaron
Figure 3: The overview of SpikeFET. Our SpikeFET consists of Input Processing, Feature Extraction, Feature Fusion and Tracking. Where the Event Stream simply transformed into event temporal frames using 334. Feature Extraction consists of DownSampling and ConvFormer Spike Block cascades. Feature Fusion consists of TransFormer Spike Block cascades. Tracking uses SNN Tracking Head, and Spiking Neuron use the Spike Firing Approximation (SFA)
cross-modal fusion network, and dual head decoupling tracking prediction with similarity fusion, as detailed Fig.
Modality Sppecit Feature Extraction Network The proposed dual-branch modality-specifee feature extraction network consists of a image branch and an event branch operating in parallel, where each branch comprises multiple stacked ConvFormer Spike blocks. Specifically, the ConvFormer Spike Block can be formulated as:
U = U + SepSpikeConv (U)
U" = U' ChannelConv(U')
(2)
SepSpikeConv (S (SN(C) (SN(U)))))
(3)
ChannelConv(U") Conv(SN(Con<<SV(D"))
where SepSpikeConv() denotes the token mixer, ChannelConv()) represents the channel mixer, Con- vpw1(.) and Convpw2(() are pointwise convolutions, Convdw(-) refers to depthwise convolution [35]
and Conv(.) indicates standard convolution. SN(.) stands for the spiking neuron layer. Note that BN layers are omitted here for notation simplicity.
Cross-Modal Fusion Network Before feeding the dual-modality feature maps into the fusion network, we introduce three standard learnable encodings: positional encoding Ep. modality encoding Em, and type encoding (see See 3.2). The dual-modality feature maps then summed with their corresponding three encodings:
(5) The image feature map U, and the event feature map U, are then concatenated as U [U,; U.]. The generated feature map U serves as the input to the fusion network for cross-mode modsl interactive learning.
U' U+ Em E

---

The cross cross-modal modal fusion network consists of Transformer Spike Blocks, which can be formulated as:
U' U SepSpikeConv(U)
(6)
U" U' CSWin-SSA(U')
(7)
UT U" + ChamnelMLP(U")
(8)
where CSWin-SSA(+) denotes the Cross- Windows [36 Spiking Self-Attention module. Specifically, the CSWin-SSA module can be expressed as:
Qs SN(Linear(U)), Ks = SN(Linear(U)), Vs - SN(Linear, (U))
(9)
U' = Linear 7 (CSWinSSA(Qs: L Ks. Vs))
(10)
SSA(Qs. Ks, Vs) = SN Vs * scale)
(11)
where Linear(-) denotes linear transformation, 7 is the channel expansion factor k77
CSWinSSA(+)
represents the CSWinSSA operator, and SSA(-) is the spiking self-attention operator within the CSWinSSA operator. To address the challenge of large values generated by matrix multiplication, scaling factor (scale) is introduced to regulate the magnitude of the results. Further details of the WSninSSA operator are provided in the Appendix
Decoupled Tracking Prediction With Similarity Fusion In the design of the tracking head, we maintain structural continuity with the dual-modal feature extractor. Specifically, we construct the Decoupled Spike Tracking Heads similar to SDTrack [16], utilizing parallel dual branches to process heterogeneous modal features.
In object tracking tasks, the response maps output by the network play a critical role in target localization. To achieve sufficient fusion of image and event modalities, propose enhancing feature alignment by constraining the similarity between the response maps of the two modalities, implemented via weighted focal loss function LewF (38) (see the Appendix [.I more details). The final loss function is expressed as: Les = LGWF (Rf/T, Rg/T), where R, and Rg are the response maps of two modes, 7 is the temperature coefficient, and the empirical setting is
3.2 Randomized Patchwork Module
The widely used padding in residual networks [17 [8] significantly degrades single-object tracking performance by breaking translation invariance, while traditional methods of removing padding 17] [ 18] would disrupt the network structure. To address this critical issue, while preserving the advantages of the original network core architecture,
have innovatively proposed a simple yet effective Randomized Patchwork Module (RPM). Specifically, prior to modality-speciff feature extraction, Two adjacent temporal template images Z1, Zo T 7'xCH, W. and the search image E X2''CxH,,XXX xW, are fed into the RPM. As shown in Fig. 4 RPM randomly employs either horizon- tal concatenation U₁ E R Txxx H, (W, +W.) or vertical concatenation U, E Tx (H,+H, +H,)) to generate rectangular fused inputs. This design randomly distributes the search image across four orientations (top, bottom, left, right) of the concatenated region, creating diversified input patterns. Meanwhile, enforce consistent concatenation direction (horizontal vertical) between image frames and event streams to ensure spatial consistency across modalities.
Additionally, as shown in Fig. 3 the learnable type encoding E, explicitly identifies the logical positions of image patches through additive integration with visual features, eliminating spatial ambiguities caused by random patch permutation while suppressing noise. Through visual analysis of the heatmap distribution in output images (Fig. 6), the effectiveness of the module is clearly demonstrated.
3.3
Regularization
Although RPM dynamically reorganizes the spatial distribution of input data mitigate target localization bias, the boundary regions of adjacent temporal template frames (e.g., Z, and Z₂) still manifest asymmetric feature sources, such as distinct features from other images or padding regions induced by convolutional networks. As illustrated in Fig. 5 the left boundary of Z, corresponds to padding regions, while the left boundary of 72 coincides with the right region of 7f while their bottom boundaries respectively align with different regions of the search frame X. This spatial distribution
5

---

Randomized Patchwork Module (RPM)
Spatial- Tempial Regularization (STR)
Similarity Less
3-236-
2 1ill
(T))
3+ 128-28122
2
U
=Padding
Irputs
bypes Outputs
Figure 4: Detailed design of proposed RPM. Randomly combine template frames Z and Z2 with search frame X.
Figure 5: Illusion of the proposed STR. Applying similarity loss regularization to two temporally ad- jacent template frames.
inconsistency inherently disrupts the strict translation invariance of convolutional networks, leading to feature representation deviations identical spatial positions across template frames Z₁ and Ze and consequently causing significant degradation similarity measurement. Notably, we also found that constraining the similarity of latent feature space between adjacent template frames in the temporal domain can establish a more robust similarity metric space.
Building upon the aforementioned insights, we propose the Temporal Regularization (STR) strategy to constrain the similarity measurement dual-template frame features. The core idea behind this strategy is to leverage two temporally adjacent template frame features with spatial-temporal consistency for relation modeling with the search frame features, thereby enhancing model robustness. Specifically, after the cross- moosl fusion network completes multi-modal information integration, we construct a similarity constraint on the fused features output by temporally adjacent template frames. By applying mean squared error (MSE) regularize the feature discrepancies between dual templates, we establish a similarity supervision signal with apatial-tempone consistent characteristics. This constraint be formulated as the following loss function:
h
sim
(12)
hww
where F, Rhxw and F2 Rbxw denote the output feature maps of temporally adjacent template frames after passing through the feature fusion network, where h and w represent the spatial dimensions of the feature maps. The loss function guides the network extract more generalizable feature representations from spatiotemporally continuous video data by minimizing the discrepancies between spatially corresponding points in the dual-template features.
3.4 Training and Inference
For training, we adopt the same training procedure as OSTrack B], employing weighted focal
loss (38] for classification and combination of L1 loss and generalized IoU loss [39] for regression. For cross- mrdal fusion and the STR strategy, we employ the similarity based loss functions LRes and Lsim previously described. Finally, the overall loss function is formulated as:
C = C.. + Ajoo C + Ac1 + L + \iouCh + ACI
(13)
Lunck L aLCss +f.aim
(14)
where Aiou = and ÀLI = are regularization parameters as 44. a 1 and 8 0.5 are hyperaarameters used to balance the contributions of different loss functions. Ablation studies hyperparameters are shown in Appendix F
For inference, we perform weighted summation and fusion of the image frame and event temporal frame response maps output by the dual-head decoupled tracking head to produce robust target localization results. This is expressed as:
R XRr + (1 XyRF
(15)
where 0.5 is a hyperparameter to balance the responses of different modalities. should be noted that we do not employ dynamic template update strategy. Instead, we consistently replicated

---

the first frame to use two template frames (both derived from the first frame) and one search frame for inference. At the same time, we follow common practice by applying a Hanning window penalty to leverage positional priors in tracking 2 3
Experiments
4.1
Implementation details
Our proposed SpikeFET was implemented with PyTTrrch 1.12 in Python 3.8 and trained on two NVIDIA RTX 4090 GPUs. The network input consists of triplet image group comprising one search image and two distinct template images. When constructing training samples from randomly sampled video sequences, we expanded the search regions and template regions to 4x and 2x their original bounding box sizes, respectively, followed by resizing them to 256x256 and 128x 128. Common data augmentation techniques such as horizontal flipping and brightness jittering were applied to the image sets. The model was trained using the AdamW optimizer. The optimizer has a cosine annealing scheduling over 50 epochs, where each epoch contained 60,000 image triplets. Please refer the Appendix for more details.
Similar to current mainstream multi-modal object tracking frameworks (such as VIPT 25 SD- STrack [19], TENet (23)), which commonly adopt OSTrack 3 frame-datasets pre-trained models for parameter fine-ture sonettuning or knowledge distillation, we also follow their standard configuration process. Specifically, we adapt our SpikeFET into single-modal SpikeET model, pre-trained frame- datasets such as COCO [4). LaSOT 41). TrackingNet [42]. and GOT-1OK 4 to fine-tune our SpikeFET network. For the single-moda SpikeET model, remove feature extraction branch and the spiking tracking head branch from SpikeFET, along with eliminating modality encoding. To balance speed and accuracy,
propose three variants with distinct architectures and parameters: SpikeFET--ass (Fig. 3) with larger parameter capacity, and SpikeFET-Ting and SpikeET optimized for computational efriciencc. Among these, only SpikeFET-Base undergoes frame-based dataset pre-training, while SpikeFET-Tiny and SpikeET adopt ImageNet-11 K [44) to pre-train the backbone, consistent with OSTrack (3. To quantitatively evaluate the performance of tracking systems, employ three core metries: Area Under the Curve (AUC), Precision Rate (PR), and theoretical power consumption. For detailed description of the theoretical power consumption caleulation method, please refer to Appendix
4.2 Datasets
We evaluated proposed SpikeFET using three large-scale frame-event single-object tracking datasets: FE108 (8), VisEvent [6], and COESOT [7]. All three datasets were captured using the DAVIS346 camera, which features spatial resolution of 346 x 260, dynamic range of 120 dB, and a minimum latency of 20 us.
Specifically, The FE108 dataset 8 contains 108 synchronized frame-event sequences captured in indoor environments, totaling approximately 1.5 hours of data. covers 21 distinct target objects and is divided into 76 training sequences and 32 testing sequences. The target bounding box annotations were generated using Vicon motion capture system. The VisEvent (6] dataset originally collected 820 frame-event pairs, divided into 500 training sequences and 320 testing sequences. Following [33 we removed sequences with missing event data or misaligned timestamps, resulting in refined dataset of 205 training sequences and 172 testing sequences. Multimodal networks and unimodal networks use 320 test sequences and 172 test sequences, respectively. COESOT 7 as a larger-scale benchmark, contains 578K frame-event pairs divided into 827 training sequences and 527 testing sequences. Collected across diverse indoor and outdoor scenarios, it covers 90 target categories annotated with 17 challenging visual attributes. All target bounding boxes were manually annotated.
For fair comparison, we trained SpikeFET using the full training set of VisEvent and evaluated on the test set containing 320 sequences, while SpikeET was trained on pruned version of the VisEvent training set and tested on a subset of 172 sequences. Note that on SpikeFFET, we only use raw event frames from the dataset on VisEvent, not event temporal frames.
7

---

4.3 Main Results
We evaluated the performance of our proposed SpikeFET-Tiny and SpikeFET-Base on several benchmarks, including FE108 8 VisEvent [6], and COESOT [7]. Meanwhile, for comparison with event-based trackers, we constructed a baseline model named SpikeET that using only event streams as input.
Table I:
Comparison with state-of-the-ar trackers FE108 8 VisEvent [6], and COESOT 77. We train and measure energy consumption on the VisEvent dataset. The best three results are shown in red, blue and green fonts. indicates that Trame-based trackers extended frame-event based trackers through early fusion. f indicates models pre-trained on image-frame tracking datasets.
FE108
VisEvent 6
COESOT 77]
Methods
Architecture
(M)
Power (mJ)
AUC(%)
PR(3)
AUC(%)
PR(S)
AUC(%)
PR(%)
DiMP50=
45
$7.1
85.1
47.8
67.0
58.9
67.1
PrDiMP50+
59.0
87.7
47.6
57.9
69.6
SiamRCNN
47
49.9
65.9
60.9
71.0
T:DiMP50+
60.3
91.2
60.1
72.2
T5ansTS50
60.5
ToMP101-
61.8
91.1
59.9
67.2
ANN
FENet
262.2
63.1
91.8
OSTrack 3
92.52
262.29
53.4
69.5
$9.0
70.7
CEUTrack 7
97.82
265.60
55.6
84,5
53.1
69.1
62.7
76.0
HRCHUUTrack
22
97.82
239.79
63.2
71.9
HRMonTruck
100.20
68.5
96.2
ViPT
25
93.36
134,93
65.2
92.1
59.2
75.8
65.7
73.9
SDSTrack1 19
107.80
719.10
65.8
92.6
59.7
76.7
63.7
71.7
ANN-SNN
SpikeFET-Tiny SpikeFET-Baset
29.33
18.36
68.5
96.5
$6.8
73.5
64.0
77.9
SNN
105.48
102.61
68.7
97.0
59.0
75.3
68.5
81.7
Frame-Event Tracking As shown in Tab. compared to several state-of-the-art trackers, our SpikeFET demonstrates significant advantages across most datasets. On the FE108 (8] dataset, TonnFFET-Tiny achieves an AUC score of 68.5%, matching the previous state-of-the-an HRMon- Track while surpassing it by 0.3% in PR score and outperforming all other trackers by substantial margin. Compared to the Tiny variant, the Base model further improves performance metrics, but there a tendency towards overfitting. Notably, although the model size the base model is comparable to the previous Base models, its theoretical energy consumption is significantly lower than equivalently scaled models. Additionally Compared with the latest SDSTrack (19) on ANN, our Tiny has improved its AUC score by 2.7% the FE108 g dataset, with parameters only one-third of SDSTrack [19], while reducing power consumption by 39 times.
On the more complex COESOT 7 dataset, SpikeFET- Base model achieves 2.7% and 7.7% improvements AUC and PR scores respectively compared to the state-of-the-ar MMHT (15), while the Tiny model also surpass the latest SDSTrack [19] by 0.3% in AUC. This demonstrates that our method exhibits stronger capabilities in handling large-scale complex scenarios.
On the VisEvent O dataset, our Tiny model outperforms the majority of trackers. However, for the Base model, our SpikeFET performs comparatively worse than ViPT 25 and SDSTrack [9). We attribute this primarily to the fact that SpikelETT is particularly well-suited for handling datasets with challenging scenarios and large-scale datasets, such as FE108 [ and COESOT 7 In such scenarios, the inherent advantages of event cameras become evident, allowing SpikeFETT to demonstrate exceptional performance. In contrast, ViPT (25) and SDSTrack [19] excel in less challenging
scenarios, such as VisEvent [6].
At the same time, shown in Tab. can be observed that SpikeFET-Base achieved increases of 0.2%, 2.2%, and 4.5% in AUC metrics compared to SpikeFET-Ting on the FE108 (8) VisEvent [6] and COESOT (7) datasets (arranged in ascending order data volume), with the performance gap progressively widening. Tab. rreeells that on FE108 88 a dataset featuring numerous challenging performance fully leveraged: its AUC is only 4.1% lower than SpikeFET-Ting while outperforming all event- tracking algorithms. In contrast, it underperforms by 17.4% on VisEvent [6]. This further illustrates the superiority of SpikeFET in challenging scenarios and complex datasets.
scenarios (e.g., overexposure, low Vllmination)-SSpikel

---

Table 2: Comparison with state-of-the-ard trackers on two event-based tracking benchmarks. The works in the table directly adopt the results of the SDTrack [6
report.
FE108
8
VisEvent
G
Methods
Architecture
Param (M)
Power (mJ)
AUC(%)
PR(%)
AUC(%)
PR( (%)
DiMP50 45
256.37
31.5
44.2
PrDiMP50 46
258.37
32.2
46.9
ATOM 49
30.199
28.6
47.4
SiamRPN 550
203.88
24.7
38.4
STARK 4
28.23
58.88
57.4
89.2
34.1
46.8
SimTrack 51
88.64
93.84
56.7
88.3
34.6
47.6
ANN
OSTrack 3
92.52
98.90
32.7
46.4
ARTrack 52
202.56
174.80
56.6
88.5
33.0
43.8
SeyTrack
133
90.60
302.68
53.5
85.5
28.6
43.3
HiT [54
42.22
19.78
55.9
88.5
34.6
47.6
GRM [5
99.83
142.14
56.8
89.3
33.4
47.7
HIPTrack 56
120.41
307.74
50.8
81.0
32.1
45.2
STNel
13
20.55
58.5
89.6
35.0
50.3
ANN-SNN
SNNTrack
31.40
8.25
35,4
50.4
SDTrack
107.26
30.52
59,9
91.5
37.4
51.5
SNN
SpikeET
22.36
8.80
64.4
94.7
39.4
54.0
These results confirm the energy efficiency characteristics SNNs and their potential for edge devices. Furthermore, due the inherent compatibility between SNNs and event data, combined with the high-efficiency design of our SpikeFETT network, SNNs demonstrate promising potential to surpass ANNs in frame-event tracking tasks.
Event-based Tracking As shown in Tab. without using fusion networks, our SpikeET with RPM and STR strategy outperforms all event-based trackers on both datasets, achieving state-of-the--e performance. Specifically, the FE108 ( dataset, SpikeET with 22.36M parameters and ultra- low energy consumption improves AUC by 4.5% and PR score by 3.2% compared to the previous best tracker SDTrack [16), even surpassing most trackers listed in Tab. T On the VisEvent 6 dataset, SpikeFT achieves 2.0% higher AUC and 2.5% better PR than SDTrack [6]] Notably, while consuming only 0.55ml improvement. These results effectively demonstrate the advantages our proposed RPM and STR modules, while further validating the energy efficiency SNNs and their potential for edge device deployment.
energy than SNNTrack [14] method delivers 4.0% AUC
4.4 Ablation Study
Effectiveness of proposed components We analyze the effects of the proposed RPM, E, and STR. Our baseline model, similar to CEU- Track (7]. processes six input images from both frame and event modalities with separate fea- ture extraction. These features concatenated before entering the fusion network, then sepa- rated prior to the prediction head and processed through decoupled tracking heads. As shown in Tab. 3 the baseline equipped with RPM (#3) achieves significant improvements. On FE108 [8] with numerous extreme scenarios, RPM substantially increases AUC by 34.7% and by 49.9% over baseline, while on VisEy-
With Without RPM Figure 6: Visualization of position bias learnt in the model w/ and w/o RPM.
Input
ent (6. it improves AUC by 10.4% and PR by 11.9%. More specifically, illustrated in Fig. without RPM method, the peak response in the output image fails to precisely indicate the target position, exhibiting significant deviation. In contrast, when RPM is employed, the target area is accurately

---

highlighted with high response activation. Quantitative analysis confirms that our model achieves enhanced tracking accuracy and robustness through RPM.
Furthermore, we conducted further experiments using one template frame and one search frame per modality, totaling four inputs. Without RPM, the six-input configuration (#baseline) yields only marginally higher performance than the four-input setup (#) 1) on the FE108 [8] dataset. Similarly,
when RPM is employed, the six-input configuration (#3) also shows only slight performance advantage over the four-input configuration (#2) on the FE108 [ dataset. This conclusively demon- strates that the primary innovation driving baseline improvement RPM's effective mitigation of padding-induced induced degradation in translation invariance. In contrast, utilizing additional data provides minimal performance gains relative RPM's s contribution. Additionally, the six six-input setting allows for the further incorporation of STR.
padding
Further additions of E, (#4) and STR (#5) #3 demonstrate effective enhancements, particularly with STR improving FE108 8 AUC by 0.6%. This proves that while retaining the advantage of RPM in dynamically reorganizing spatial distributions, the STR strategy effectively suppresses matching deviations caused by feature spatial misalignment, significantly improving model performance. Finally, integrating both E, and STR with RPM (#6) yields greater gains: 1.7% AUC and 1.5% PR improvements on FE108 [8) along with 1.3% AUC and 1.1 % PR improvements on VisEvent [6]
Table 3: Ablation studies on RPM, E, and STR Table 4: Ablation studies on the effect of the fusion (Random Patchwork Module, Type Encodings, methods. Cemporl Regularization).
Type
Mathed
ALCUS)
PETOS
PREWT
AUCRS)
Vislbvent
PREETT
Buseline
Inputs
RPM E,
STR
AUC(S))
32.1
FETOS
PR[S5]
AUL((S)
15.1
Vidlven
PR(3T
60.3
Baseline
Modal
SpikeET Uname)
SpikeET (Event
SpikeFETTTTnn
64.4
51.8
94.7
76.2
41.0
51.6
71.1
57.1
30.8
Pic-4
67.3
125
55.6
72.1
93.9
54.5
Backboue
Concat Add
71.9
x
66.8
98.9
35.5
72.4
Wie modality Encodings
96.1
55.8
72.6
66.8
95.0
55.7
72.3
Wio Puna-Haad
67.0
54.5
55.6
72.
67.4
95.8
55.7
72.5
Tracking
Wio Response Loss
67.1
95.5
56.1
72.9
68.5
96.5
56.8
73.5
W/o Inference Fusion
72.8
Effectiveness of the fusion methods We evaluate the effectiveness of our fusion approach. As shown in entries #2 and #3 in Tab. [ our frame-event fusion framework outperforms using either modality alone, demonstrating that our framework can effectively leverage the strengths of both modalities. The results from entry #4 reveal that concatenating the two modal images before feature extraction leads to performance degradation. The underlying reason may be that using independent parameters helps the model distinguish data types, enabling better learning of modality-speciil features. In entry #5, we investigate an alternative approach of combining image features with event features via addition. This results in lower performance compared our default method. Entry #6 demonstrates that sppecfi encoding more effectively differentiates modal features. Tracking experiments (entries #7, #8, and #9) confirm the necessity of our Decoupled Tracking Prediction With Similarity Fusion. Without this design, performance degrades to varying degrees. For more experiments, please refer to Appendix. F
We compare the tracking results of SpikeFET-Bass with the state-of-the-art tracker shown in Fig. 2 Despite facing challenges such as exposure and low light conditions, our tracker achieves robustness and excellent tracking accuracy. Additional visualization results provided in the Appendix. A
Visualization results
5 Conclusion
In this article, we propose full spiking neural network SpikeFET for unified frame-event object tracking. This network effectively integrates frame and event data in the spiking paradigm and achieves collaborative integration of convolutional local feature extraction and Transformer: based global modeling. Moreover, we design the RPM to overcome the degradation of translation invariance caused by convolution filling through random spatial reorganization, while preserving the residual structure. We also propose the STR strategy that overcomes the degradation of similarity metrics caused by asymmetric features by forcing spatiotemporal consistency between temporal template features in the latent space. The effectiveness and power consumption advantages of our SpikeFET have been demonstrated through extensive experiments multiple benchmarks.
10

---

References
[1] Luca Bertinetto, Jack Valmadre, Joao F Henriques. Andrea Vedaldi, and Philip HS Torr. Fully-convolttioaal siamese networks for object tracking. Computer vision-ECCV 2016 workshops: Amsterdam, the Netherlands, October 8-10 and 15-16, 2016, proceedings, part HI 14, pages 850-865. Springer, 2016.
[2] Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang, and Huchuan Lu. Transformer tracking. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 8126-8125 2021.
[3] Botao Ye, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Joint feature learning and relation modeling for tracking: A onc-stream framework. In European conference computer vision, pages 341-357. Springer, 2022.
[4] Bin Yan, Houwen Peng, Jianlong Fu, Dong Wang, and Huchuan La. Learning spatio-temporal transformer visual tracking. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10448-1045,, 2021.
[5] Jiqing Zhang, Kai Zhao, Bo Dong. Yingkai Fu, Yuxin Wang, Xin Yang. and Baocai Yin. collaborative feature representation for robust visual object tracking. The Visual Computer, 37(9):2671 2683, 2021.
[6] Xiao Wang. Jianing Li, Lin Zhu, Zhipeng Zhang. Zhe Chen, Xin Li, Yaowei Wang, Yonghong Tian, and Feng Wu. Visevent: Reliable object tracking via collaboration of frame and event flows. IEEE Transactions on Cybernetics, 54(3):1997-200 2023.
[7] Chuanming Tang, Xiao Wang, Ju Huang, Bo Jiang, Lin Zhu, Jianlin Zhang, Yaowei Wang, and Yonghong Tian. Revisiting color-event based tracking: A unified network, dataset, and metric. arXiv preprint 1rXiv:2211. 1010, 2022.
[8] Jiqing Zhang, Xin Yang. Yingkai Fu, Xiaopeng Wei, Baocai Yin, and Bo Dong. Object tracking by jointly exploiting frame and event domain. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2021.
[9] Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike-based machine intelligence with neuromorphic computing. Nature, 575 2019.
[10] Catherine D Schuman, Shruti R Kulkarni, Maryam Parsa, J Parker Mitchell, Prasanna Date, and Bill Kay. Opportunities for neuromorphic computing algorithms and applications. Nature Computational Science, 2(1):10 19, 2022.
[11] Man Yao, Xuerui Qiu, Tianxiang Hu, Jiakui Hu, Yuhong Chou, Keyu Tian, Jianxing Liao, Luziwei Leng, Bo Xu, and Guoqi Li. Scaling spike-driven transformer with efficient spike firing approximation training. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2025.
[12] Weihao Yu, Chenyang Si, Pan Zhou, Mi Luo, Yichen Zhou, Jiashi Feng, Shuicheng Yan, and Xinchao Wang. Metaformer baselines for vision. IEEE Transactions Pattern Analysis and Machine Intelligence, 46(2):8969912, 2023.
[13] Jiqing Zhang, Bo Dong, Haiwei Zhang, Jianchuan Ding, Felix Heide, Baocai Yin, and Xin Yang. Spiking transformers for event-based single object tracking. In Proceedings of the IEEE/CVF conference on Computer Vision and Puttern Recognition, pages 8801-8810, 2022.
[14] Jiqing Zhang, Malu Zhang, Yuanchen Wang, Qianhui Liu, Baocai Yin, Haizhou Li, and Xin Yang. Spiking neural networks with adaptive membrane time constant for event-hased tracking. IEFE Transactions on Image Processing, 2025.
[15] Hongze Sun, Rui Liu, Wuque Cai, Jun Wang, Yue Wang, Huajin Tang, Yan Cui, Dezhong Yao, and Daqing Guo. Reliable object tracking by multimodal hybrid feature extraction and transformer-based fusion. Neural Networks, 178: 106493,, 2024.
[16] Yimeng Shan, Zhenbang Ren, Haodi Wu, Wenjic Wei, Rui-Jie Zhu, Shuai Wang, Dehao Zhang, Yichen Xiao, Jicyuan Zhang, Kexin Shi, et al. Sdtrack: A bascline for event-based tracking via spiking neural networks. arXiv preprini arXiv:2503.0877
2025.
[17] Bo Li, Wei Wu, Qiang Wang. Fangyi Zhang. Junliang Xing, and Junjie Yan. Siamrpn++: Evolution of siamese visual tracking with very deep networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4282 4291, 2019.

---

[18] Zhipeng Zhang and Houwen Peng. Deeper and wider siamese networks for real-time visual tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4591-4600, 2019.
[19] Xiaojun Hou, Jiazheng Xing, Yijic Qian, Yaowei Guo, Shuo Xin, Junhao Chen, Kai Tang, Mengmeng Wang. Zhengkai Jiang. Liang Liu, al. Sdstrack: Self-distillation symmetric adapter learning for multi- modal visual object tracking. In Proceedings of the IEEF//VF Conference on Computer Vision and Pattern Recognition, pages 66551-26561. 2024.
[20] Ning Wang, Wengang Zhou, Jie Wang, and Houqiang Li. Transformer meets tracker: Exploiting temporal context for robust visual tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1571-1580) 2021.
[21] Chenxu Luo, Xiaodong Yang, and Alan Yuille. Exploring simple 3d multi-object tracking for autonomous driving. In Proceedings of the IEEF/CVF international conference on computer vision, pages 10488-10497, 2021.
[22] Zhiyu Zhu, Junhui Hou, and Dapeng Oliver Wu. Cross-modal orthogonal high-rank augmentation for rgb-event transformerrtrackerss. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22045 22055,22255222 2023.
[23] Pengeheng Shao, Tianyang Xu, Zhangyong Tang, Linze Li, Xiao-Jun Wu, and Josef Kittler. Tenet: targetness entanglement incorporating with multi-scale pooling and mutually-guided fusion for rgb-c object tracking. Neural Networks, 183: 106940 2025.
[24] Zongwei Wu, Jilai Zheng, Xiangxuan Ren, Florin- Alexandru Vasluianu, Chao Ma, Danda Pani Paudel, Luc Van Gool, and Radu Timofte. Single-model and any modality for video object tracking. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 191669 2024.
[25] Jiawen Zhu, Simiao Lai, Xin Chen, Dong Wang, and Huchuan Lu. Visual prompt multi modal tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9516-9526, 2023.
[26] Lin Zhu, Xiao Wang, Yi Chang, Jianing Li, Tiejun Huang, and Yonghong Tian. Event-based video reconstruction via polential-assistd spiking neural network. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1994-6604, 2022.
[27] Jesse Hagenaars, Federico Paredes- Paredes-Vallés, and Guido De Croon. Self-supervised learning of event- based optical flow with spiking neural networks, Advances in Neural Information Processing Systems, 34:7167-7179, 2021.
[28] Jinye Qu, Zeyu Gao, Tielin Zhang, Yanfeng Lu, Huajin Tang, and Hong Qiao. Spiking neural network for Altralow-latene, and high- high-accurate object detection. IEEE Transactions on Neural Networks and Learning Systems, 2024.
[29] Wenzhuo Liu, Shuiying Xiang, Tao Zhang. Yanan Han, Yahui Zhang, Xingxing Guo, Licun Yu, and Yue Hao. S4-kd: A single step spiking siamfc++ for object tracking with knowledge distillation. Neural Networks, 188: 107478, 2025.
[30] Yihao Luo, Min Xu, Caihong Yuan, Xiang Cao, Liangqi Zhang, Yan Xu, Tianjiang Wang, and Qi Feng. Siamsan: Siamese spiking neural networks for energy-efficiere object tracking, In International conference artificial neural networks, pages 182 194. Springer, 2021.
[31] Shuiying Xiang, Tao Zhang, Shuqing Jiang, Yanan Han, Yahui Zhang, Xingxing Guo, Licun Yu, Yuechun Shi, and Yue Hao. Spiking siamfe++: Deep spiking neural network for object tracking. Nonlinear Dynamics, 114(10)8417-8422 2024.
[32] Jiahang Cao. Mingyuan Sun. Ziqing Wang. Hao Cheng, Qiang Zhang, Shibo Zhou, and Renjing Xu. Spiking neural network as adaptive event stream slicer. arXiv preprint arXiv:2410.02249, 2024.
[33] Mingcheng Ji, Ziling Wang, Rui Yan, Qingjic Liu, Shu Xu, and Huajin Tang. Setn: Event-based object tracking with energy- energy-efficient deep convolutional spiking neural networks. Frontiers in Neuroscience, 17:1123698, 2023.
[34] Liangwei Fan, Yulin Li. Hui Shen, Jian Li, and Dewen Hu. From dense to sparse: low-latency and speed-robust weent-base object detection. IEEE Transactions on Intelligent Vehiclex, 2024.
[35] François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1251-1 2017.
12

---

[36] Xiaoyi Dong, Jianmin Bao, Dongdongg Chen, Weiming Zhang, Nenghai Yu. Lu Yuan. Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In Proceedings of the IEEL/CVF conference on computer vision and puttern recognition, pages 12124-12134, 2022.
[37] Jiangning Zhang, Xiangtai Li, Jian Li, Liang Liu, Zhucun Xue, Boshen Zhang, Zhengkai Jiang. Tianxin Huang. Yabiao Wang, and Chengjie Wang. Rethinking mobile block for efficient altention-base based models. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1389-1400. IEEE Computer Society, 2023.
[38] Hei Law and Jia Deng. Corneret: Detecting objects as paired keypoints. In Proceedings of the European conference on computer vision (ECCV), pages 734-750, 2018.
[39] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and a loss for bounding box regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 658 06886666226666 2019.
[40] Tsung-Ys Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13, pages 740 755. Springer, 2014.
[41] Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hcxin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling, Lasot: A high-quality benchmark for large-scale single object tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5374-5383 2019.
[42] Matthias Muller, Adel Bibi, Silvio Giancola. Salman Alsubaihi. and Bernard Ghanem. Trackingnet:
A large-scale dataset and benchmark for object tracking in the wild. In Proceedings of the European conference on computer vision (ECCV), pages 300-317, 2018
[43] Lianghua Huang, Xin Zhao, and Kaiqi Huang. Got-10k: A large nigh-diversity benchmark for generic object tracking in the wild. IEEE transactions pattern analysis and machine intelligence, 4365:11522- 1577. 2019.
[44] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Iece, 2009.
[45] Goutam Bhat, Martin Danelljan, Lue Van Gool, and Radu Timofte. Learning discriminative model prediction for tracking. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6182 6191, 2019.
[46] Martin Danelljan, Luc Van Gool, and Radu Timofte. Probabilistic regression for visual tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7183-7192. 2020.
[47] Paul Voigtlaender, Jonathon Luiten, Philip HS Torr, and Bastian Leibe. Siam r-cnn: Visual tracking by re-detection. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 6578-6588, 2020.
[48] Christoph Mayer. Martin Danelljan, Goutam Bhat, Matthieu Paul, Danda Pani Paudel. Fisher Yu, and Luc Van Gool. Transforming model prediction for tracking. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 8731-8740, 2022.
[49] Martin Danelljan, Goutam Bhat, Fahad Shahbaz Khan, and Michael Felsberg. Atom: Accurate tracking by overlap maximization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4660-46600 2019.
[50] Li, Junjie Yan, Wei Wu, Zheng Zhu, and Xiaolin Hu. High performance visual tracking with siamese re- gion proposal network. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 8971-8980, 2018
[51] Boyu Chen, Peixia I.i, Lei Bai, Lei Qiao, Qiuhong Shen, Bo Li, Weihao Gan, Wei Wu, and Wanli Ouyang. Backhone is all your need: A simplified architecture for visual object tracking. In European conference on computer vision, pages 375-392. Springer, 2022.
[52] Xing Wei, Yifan Bai, Yongchao Zheng. Dahu Shi, and Yihong Gong. Autoregrossive visual tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2023.
13

---

[53] Xin Chen, Houwen Peng, Dong Wang, Huchuan Lu, and Han Hu. Seqtrack: Sequence to sequence learning for visual object tracking. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 14572-14581, 2023.
[54] Ben Kang, Xin Chen, Dong Wang, Houwen Peng. and Huchuan Lu. Exploring lightweight hierarchical vision transformers for efficient visual tracking. Proceedings of the IEEE/CVF international conference on computer vision, pages 9612-9621, 2023.
[55] Shenyuan Gao, Chunluan Zhou, and Zhang. Generalized relation modeling for transformer tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18686-18692) 2023.
[56] Wenrui Cai, Qingjie Liu, and Yunhong Wang. Hiptrack: Visual tracking with historical prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19258- 19267, 2024.
[57] Wachirawit Ponghiran and Kaushik Roy. Spiking neural networks with improved inherent recurrence dynamics for sequential learning. In Proceedings of the AAAI conference artificial intelligence, volume 36, pages 8001-8008, 2022.
[58] Guobin Shen, Dongcheng Zhao, Tenglong Li, Jindong Li, and Yi Zeng. Are conventional snns really efficient? a perspective from network quantization. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 27538-27547, 2024.
[59] Zhenzhi Wu, Hchui Zhang, Yihan Lin, Guoqi Li, Meng Wang. and Ye Tang. Liaf-net: Leaky integrate and analog fire network for lightweight and efficient spatiotemporal information processing. IEEE Transactions on Neural Networks and Learning Systems, 33(1 1):22996222, 2021.
[60] Nicholas Chen. Pseudo-lahels for supervised learning on dynamic vision sensor data, applied to object detection under ego-motion. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 644-653, 2018.
[61] Ana I Maqueda, Antonio Loquercio, Guillermo Gallego, Narciso Gareía, and Davide Scaramuzza, Event- based vision meets deep learning steering prediction for self-driving cars. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 5419-5427, 2018
[62] Xiao Wang, Shiao Wang, Chuanming Tang. Lin Zhu, Bo Jiang, Yonghong Tian, and Jin Tang. Event stream- based visual object tracking: A high-resolution benchmark dataset and novel baseline. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 19248-19257, 2024.
[63] Mark Horowitz. 1.1 computing's energy problem (and what do about it). In 2014 IREE international solid-state circuits conference digest of technical papers (ISSCC), pages 10-14. IEEE, 2014.
[64] Man Yao, Jiakui Hu, Tianxiang Hu, Yifan Xu, Zhaokun Zhou, Yonghong Tian, Bo Xu, and Guoqi Li. Spike- driven transformer v2: Meta spiking neural network architecture inspiring the design of next-generation neuromorphic chips. arXiv preprint 2344:2440,,,663,, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We claim our contributions in the abstract (Lines 5-19) and introduction (Lines
60-73).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of proposed work in Appendix 1
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Our proposed method aims to design an effective structure Sec. to improve the tracking performance. And we verify the effectiveness of our method through experiments. (Sec. + Appendix F)
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our contribution is specific model. We describe the proposed methods in Sec. 3 and provide the implementation details in Sec. 4.1 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We publicly available data and provide the exact commands and environ- ment required for running to reproduce the results Appendix [ As a baseline, we will open source the code and provide it in supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.com cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
y /nips. /twpp.cc////p/ http:ppwwwigggieeeeeeeeeeaa
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We provide the training and test details in Sec. 3.4 Sec. 4.1 and Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Although our RPM introduces randomness, the results are stable in several random seed experiments. Therefore, we used a fixed random seed for the experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provided the type of compute workers in Sec. 4.1 and the required theoretical power consumption in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research conducted in the paper conforms with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential positive societal impacts of the work on Appendix H
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite the original paper that produced the code package or dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Our core method development does not involve LLMs as any important, original, or non-standard components.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
21

---

Appendix
A
More Visualization Results
We present more results of SpikeFET and other methods on the COESOT 7 dataset here,
GT
SDSTrack
ViPT
SpikeFET
Figure 7: Partial visualization results on the COESOT dataset.
22

---

GT
SDSTrack
ViPT
SpikeFET
Figure 8: Partial visualization results on the COESOT
7 dataset.
23

---

B Preliminary
B.1
Spiking Neuron
SNNs derive biological plausibility from their event-driven operation. Conventional training faces dilemma: enforcing strict 0, spikes induces gradient quantization errors, while mitigating errors via multi-bit spikes (57 [58] or continuous approximations [59] erodes event-driven efficiency. We introduce the Spike Firing Approximation (SFA) [Л, framework combining integer training with spike-driven inference, to optimize neuronal firing patterns.
We postulate that information transmission between spiking neurons is governed by spike firing rates.
The firing rate of spiking neuron is operationally defined as:
1 T
a'
s
(16)
where a' denotes the spike firing rate, and S ['tt) represents the spike train sequence at layer I over discrete time steps.
During training, we implement integer activation training by approximating a' with integer values.
This is achieved by replacing the temporal summation Σ 1=1 St [t] with tigle-timestep integer-valued activations:
syr |clip 0, 07]
(17)
where S/y denotes the integer activation within [0,T], with T representing the maximum activation threshold. And clip(x', 0, (tnexxx',,,T)oosiiiii consttrains x' to the interval [O,T], followed by rounding the nearest integer:
In inference, we perform spike activation inference:
Σ M
(18)
where $'11 denotes the spike train sequence SFA (Spike Firing Approximation) inference. The spatial input spiking neurons in layer (I+1) computed as:
X = -ii+ d'= wl+1 S/ = ( Wi+1
T Σ S'
(19)
The spike sequence ('(( consists solely of Os and Is, enabling all MAC (Multily-Accumulate) operations be converted into sparse AC (Accumulation Operations), thereby ensuring spike-driven computation during inference.
B.2 Input Representation
In this work. the input consists of two complementary modalities: frame--aae-fixee rate-fieee--e image frames and a continuous event stream E(x, p), where (x. y) denotes pixel coordinates, I represents timestamps, and p indicates polarity. To adapt deep learning architectures, event streams are typically converted into lmage-like representations [34 60 61]. Compared with complex event representation methods in single-object tracking (such event voxels [62] and GTP (6)), we employ simple Time- Integrated Image of Events (34 to encode the event stream within an accumulated temporal window t. This approach maps events into a 3D tensor while preserving temporal information and polarity, ensuring compatibility with deep learning methods and facilitating fusion with existing image frames modalities. Specifically, the conversion process first partitions the event stream along the temporal axis into set of B bins. The polarity values of events within each bin accumulated and normalized

---

to the [0, 255] range using an activation function. The transformation is formulated as follows:
4 = B 1 - 11
(20)
INo 11
S 11, =a
Σ 8(σ xi)ō(y - v:)0(t (5)
(21)
255
(aa) 1 1
(22)
Where o(-) and 6()) represent the activation function and Dirac delta function, respectively. IN. denotes the total number of events. The generated event time image S E IR Bxxxxx and the corresponding image frames processed using standard methods to extract template patches and search patches from both modalities. These patches are then jointly fed into the SpikeFET model.
C
More Implementation Details
C.1 Response Loss
We adopt the weighted focal loss 38 for by constraining the similarity. Specifically, we denote the ground truth target center and the corresponding low-resolution equivalent and p [ya,Pyy respectively. The Gaussian kernel is then applied to generate the groundtruth heatmap: =
(x-p,) p.) ((v--pp)) where 6 represents the object size-adaptive standard deviation Weighted Focal (GWF) loss function formulated as:
exp
38
20
Thus, the Gaughtan-Weighted
Lowr = -E ay
(1 P my a log Pay) (1 Pay) (P Pay)" a log(1 Pzy),
if ay =1
(23)
otherwise
where a and y are hyperparameters and are set 2 and 4, respectively, as suggested in 62 The response maps of both modalities are normalized by dividing them by a temperature coefficient (empirically set 2). The final loss function is expressed as: LRes = Lawr (R TRp/T, R,//T)
C.2 Metries
When evaluating algorithms in the field of Spiking Neural Networks (SNNs), researchers often adopt theoretical power consumption estimation methods simplify the analysis of hardware implementation details. Specifically, the energy cost of Artificial Neural Networks (ANNs) is calculated as FLOPs multiplied by EMAC while the energy cost of SNNs determined by FLOPs multiplied by Exc and the network spike firing rate. In 45 nm technology. the energy consumption for MAC Multiply-ACeumullat and AC (ACcumulate) operations EMAC 4.6p. and Exc 0.9p.J, respectively [63] For spiking-hasse baaed convolutional or multilayer perceptron (MLP) layers, only the additional time step and the spike firing rate per layer need to considered. this paper, we can calculate the peak emissivity of each layer, so the energy consumption of each layer is FLOPs multiplied by EAC multiplied by the peak emissivity of each layer. The subtle difference is that the network structure will affect the number of additions triggered by a single peak. For example, when using different convolution kernel sizes matrix multiplication, the energy consumption of the same spiky tensor is different. caleulate the theoretical power consumption using the following formula [64]
EANN EMAAC (FL FLMEP)
(24)
M
N
ESNN = T x EAC X
(m)
(m² Conv
R
x MLP
+T x EMAC X FL (25)
n=1
where denotes the time step length; (mm) and (n)) represent the spike firing rates of the m-th convolutional layer and the n-th fully connected (MLP) layer, respectively, defined the ratio of
non-zero elements the spike tensor. FI-Comy and Fl FL correspond to the FLOPs of their respective
(n)
layers, and FLCow represents the FLOPs of the first and last convolutional layer in the tracking head.
25

---

More specifically, the FLOPs of the m-th Conv layer in ANNs are:
FLConv = (km)² ftm
Wm Cm-1 Cm
(26)
Where km the kernel size, (b Wm) the output feature map size, Cm-1 and Cy are the input and output channel numbers, respectively. The FLOPs of the n-th MLP layer in artificial neural networks is:
FLMLP in On
(27)
where in and On are the input and output dimensions of the MLP layer, respectively. In order to provide readers with a clear understanding of spiking firing rate, we have provided detailed spiking firing rates of a SpikeFET model in Tab.
D Detailed configuration and hyperparameters of SpikeFET models and training
On the Frame-Event tracking benchmark, used two scales of SpikeFET in Tab. and trained the model our paper using hyperparameters Tab. 6
Table 5: Configuration of backbone models for different SpikeFET (considering only single branch)
stage
Tokens
Layer Specification
Tiny
Base
Downsampling
Dim
32
64
H
W
SepSpikeConv
MLP ratio
1
ConvFormer
2
Spike Block
Channel Conv
Conv ratio
# Blocks
Downsampling
Dim
64
128
H
SepSpikeConv
ML. ratio
2
ConvFormer
4
Spike Block
Channel Conv
Conv ratio
4
# Blocks
Downsampling
Dim
128
256
H
SepSpikeConv
MLP ratio
3
ConvFormer
8
8
Spike Block
Channel Conv
Conv ratio
4
# Blocks
2
Downsampling
Dim
256
512
SepSpikeConv
ML. ratio
2
H
W
X
TransFormer Spike Block
CSWin-SSS
Gamma ratio
16
16
Channel MLP
MLP ratio
# Blocks
Downsampling
Dim
320
SepSpikeConv
MLP ratio
H
W
5
TransFormer Spike Block
CSWin-SSA
Gamma ratio
4
16
16
Channel MLP
MLP ratio
# Blocks
2
Table Hyper- for training on SpikeFET
Hyper-parameter
Finetune
Directly Training
Traning
Dual-mode
Single-modal
Single-modal
Model size
Tiny
Base
Tiny
Base
Timestemp
4
8
4
8
Epochs
50
50
300
Batch size
32
16
80
40
Optimizer
ADAMW
Learning rate
6e-4
7.5e-5
6e-4
7.5e-4
Learning rate decay
Cosine
Warmup eopchs
5
5
20
Weight decay
0.05
26

---

E CSWin-SSA
CSWin Transformer 36 is an efficient and effective Transformer based general visual task backbone, which we have extended to pulse based Cross-Shaped Windows Spiking Self-Attention (CSWin-SSA). The CSWin-SSA first generates key (Ks). query (Qs). and value (Vs) vectors by linearly transforming the imput U. Then, it performs y-fold channel expansion on Vs to enhance representation. Next, the CSWinSSA operator applies cross-modal self-attention to establish dynamic associations between template and search region features, enabling adaptive extraction of target features. CSWinSSA is achieved by performing self-attention within the horizontal and vertical parallel stripes that form a cross-shaped window. According to the multihead self-attention mechanism, the input feature U E RT*HXXXC will be first linearly projected to K heads, and then each head will perform local self-attention within either the horizontal or vertical stripes. Specifically, the CSWin-SSA module can be formulated as:
Qs SN(Linear (U)). Ks -SN(Linear(U)) Vs = SN(nine (U))
(28)
Qs QQs. Q Q Ks -KK'. Kä.KK..,KKK KS Vs - [V'. VS. S..... V
(29)
Yi SSA(QS ik KS V
(30)
H-SSA* [Y Y. k...., Y
(31)
SSA(Qs, Ks, Vs) = SN Q5 Vs * scale)
(32)
where Q R(suxW)xC and M H/sw, i = ... The vertical stripes self-attention can be similarly derived, and its output for 1tth head is denoted as V-SSAx K, are the same. To address the challenge of large values generated by matrix multiplication, scaling factor (scale) is introduced to regulate the magnitude of the results.
Assuming that the natural image has directional deviation, divide the K heads into two parallel groups on average (each group has K/2 heads, K usually an even number). The first group of heads performs horizontal stripe self attention, while the second group of heads performs vertical stripe self attention. Finally, the outputs of these two parallel groups will be reconnected together.
U' Linear (Concat(head,,.... hc)d
(33)
(H-SSA((X) k == 1..... K/2 (X)SA&(X), K/2 + 1...., K
head
(34)
F Experiments of the hyper-parameters
In the proposed SpikeFET framework, the hyper-parameters involved are Lies weight a and Lsim weight 8, for which we only present ablation studies around the optimal parameters.
The ablation study of a We introduced the ablation study of a in Tab. 7 When a is greater than 1, the fusion response map of the two modalities overly depends the worst modality, leading to decrease in metrics, whereas when a is less than 1, the response maps of image frames and event frames cannot be adequately aligned. Therefore, select = 1.
Table 7: The ablation study of c..
0.5
a 1
a = 2
AUC(%)
56.3
56,8
55.8
The ablation study of 3 The ablation study of [ is shown in Tab. g When 3 = 0.5, we achieved the best performance. When 3 is too large, excessive emphasis on consistency between two template frames can lead to different features tending towards consistency, resulting in errors. When /3 is too small, it cannot fully make the two template frames consistent in time and space, and the effect not significant enough.
27

---

Table 8: The ablation study of B3.
B = 0.1
3 0.5
AUC(%)
56.4
56.8
56.2
G
Runtime and FLOPs
Here present the specific values of MAC (Multppy-ACccmmulae and AC (ACcumulate), along with runtime comparison between SpikeFET/SpikeET and other methods on an RTX 4090 GPU for reference. Results for SpikeFET and SpikeET are shown in Tab. and Tab. TO respectively.
Table 9: Comparison of MAC, AC, and runtime between SpikeFET and other methods.
Methods
Architecture
Speed (FPS)
MACIAC
FE108
VisEvent
COESOT
AUC(%)
PR(%)
AUC(%)
PR(%)
AUC(%)
PR(%)
DiMP50+
57.1
85.1
47.8
67.0
58.9
67.1
PrDiMP50
56
59.0
87.7
47.6
65.3
57.9
69.6
SiamRCNN
49.9
60.9
71.0
TrDiMP50
20
60.3
91.2
60.1
72.2
TransT300
63.9
93.0
47.4
65.0
60.5
72.4
59.9
ANN
FENet S
92.45
57
63.1
91.8
OSTrack
3
113.29
$7.02
53.4
69.5
59.0
70.7
CEUTrack
57.74
$5.6
84.5
53.1
62.7
76.0
HRCEUTrrakk
123.38
52.13
63.2
71.9
HiRMonTTrakk
68.5
96.2
ViPTi
25
135
29.33
65.7
SDDTrack
19
42.03
156.33
65.8
92.6
59.7
76.7
63.7
71.7
ANN-SNN SNN MMHT
[5
85.48
34,98
63.0
93.6
55.1
73.3
65.8
74.0
SpikeFET-Ting
106
30.34
68.5
96.5
56.8
73.5
64.0
77.9
SNN
SpikeRET-BBe
121.24
68.5
Table 10: Comparison of MAC, AC, and runtime between SpikeET and other methods.
FE108
VisEvent
Methods
Architecture
Speed (FPS)
MACIAC
AUC(%)
PR(%)
AUC(%)
PR(%)
DiMP50 45
71.93
55.73
31.5
44.2
PrDiMP50
46
71.82
55.73
32.2
46.9
ATOM 49
121.11
44.321
28.6
47.4
SiamRPN [50]
518.55
6.57
24.7
38.4
STARK
172.19
12.8
57.4
89.2
34.1
46.8
SimTrack [5
266.25
20.4
56.7
88.3
34.6
47.6
ANN
OSTrack
204.38
21.5
54.6
87.1
32.7
46.4
ARTrack 52
97.47
38
56.6
88.5
33.0
43.8
SeqTrack
533
113.09
65.8
53.5
85.5
28.6
43.3
HiT [54
437
4.3
55.9
88.5
34.6
47.6
GRM 555
153.18
30.9
56.8
89.3
33.4
47.7
HIPTTrack 56
110.23
66.9
50.8
81.0
32.1
45.2
STNet 13
163.40
58.5
89.6
35.0
50.3
ANN-SNN
SNNTrack 14
9.17
35.4
50.4
SDTrack [T6]
30.52
66.56
91.5
37.4
51.5
SNN
SpikeET
185.87
9.78
64.4
94.7
39.4
54.0
H
Impact Statement
This paper proposes fully spike-based frame-event tracking framework, and demonstrates the effectiveness and necessity of SNNs in target tracking through extensive experimental validation.
28

---

This work provides novel approach and establishes baseline for achieving low-pown wwwpowee, high- performance, and robust visual object tracking, aiming to inspire further research and development in energy-efficient and high-performance tracking systems. At the same time, it provides new paradigm for developing low-power edge vision computing, demonstrating potential applications in real-time perception fields such as autonomous driving, and promoting the engineering process of neural morphological computing technology. However, the misuse of Object tracking technology can have negative impact on personal privacy.
I Limitations
Although SpikeFET has achieved commendable accuracy and robustness in frame-event tracking through SNNs, its sparse characteristic results in insufficient pre-training tracking datasets, leading to suboptimal performance on VisEvent [6] compared to state-of-the-art trackers. Meanwhile, deploying the tracker edge computing platforms (e.g., brain-inspired chips neuromorphic chips) could further exploit SNNs' low-power characteristics. These challenges will constitute critical research directions for future exploration.
29

---

Table 11: Layer spiking firing rates of model SpikeFET-Tiny on VisEvent.
Image
Event
Downsampling
Cony
PWConv
0.3645 0.2723
SepSpikeConv
DWConv
0.3314
0.3330
Stage
ConvFormer Spike Block
PWConv2
0.3526 0.3703
Conv1
0.4293 0.3656
Channel Conv
Conv2
0.0740 0.0771
Downsampling
Conv
0.3203 0.2694
PWConyl
0.1791 0.1616
SepSpikeConv
DWConv
0.2381 0.2243
Stage 2
ConvFormer Spike Block
PWConv2
0.1712 0.1702
Conv1
0.2680 0.2442
Channel Conv
Conv2
0.0371 0.0302
Downsampling
Cony
0.2245 0.2031
PWConv1
0.1876
0.1719
SepSpikeConv
DWConv
0.2484 0.2379
ConvFormer Spike Block
PWConv2
0.1300 0.1248
Conv1
0.1818 0.1721
Channel Conv
Stage 3
Conv2
0.0197 0.0175
PWConvl
0.2642 0.2465
SepSpikeConv
DWConv
0.1486
0.1412
ConvFormer Spike Block
PWConv2
0.1160 0.1102
Conv1
0.1960 0.1874
Channel Conv
Conv2
0.0133 0.0116
Downsampling
Conv
0.2335 0,2050
SepSpikeConv Conv-1/2/3
0.1408
Qs
0,2372
Kgs
0.1077
CSWin-SSA
Vs
0,1851
TransFormer Spike Block1
Qs Qs(k)Vs) 0.5644
Linear 0.7385
Linear 0,2194
Channel MI.P
Linear
0.0156
SepSpikeConv Conv-1/2/3
0.1811
stage4
Qs
0.1901
Ks
0.0584
CSWin-SSA
Vss
0.0939
TransFormer Spike Block1
Qs (s(VVss)
0.1180
Linear
0.5973
Linear 0.2867
Channel MLP
Linear 0.0150
SepSpikeConv Conv-1/2/3
0.1782
Qs
0.1834
Ks
0.0460
CSWin-SSA
Vs 0.1147
TransFormer Spike Block2
Qs (k(KVVs)
0.1
Linear
0.5349
Linear Linear
0.2977
Channel MLP
0.0140
Continued on next page
30

---

Table 11 continued from previous page
Image
Event
SepSpikeCony Conv-1/2/3
0.1816
Qs
0.1633
Ks
0.0532
CSWin-SSA
Vs 0.0909
TransFormer Spike Block3
Qs(K3V 0.0962
Linear
0.4846
Linear 0.2922
Channel MLP
Linear
0.0103
SepSpikeConv Conv-1/2/3
0.1733
Qs
0.1925
Ks
0.0422
CSWin-SSA
Vs
0.0808
TransFormer Spike Block4
Qs (s3V)) 0.0676
Linear 0.3879
Linear 0.2574
Channel MLP
Linear
0.0116
SepSpikeConv Conv-1/2/3
0.1558
Qs
0.1886
K's
0.0532
CSWin-SSA
Vss
0.1221
TransFormer Spike Block5
Qs(K
0. 0.1294
Linear
0.5527
Linear
0.1854
Channel MLP
Linear
0.0146
Downsampling
Conv
0.1564
SepSpikeConv Conv-1/2/3
0. 001000
Qs
0.1951
Kss
0.0369
CSWin-SSA
Vs
0.0487
TransFormer Spike Blockl
Qs (s(VV)s)
0.0449
Linear
0.3142
Linear
0.2651
Channel MLP
stage5
Linear 2
0.0032
SepSpikeConv Conv-1/2/3
0.0900
Qs
0.1978
Ks
0.0165
CSWin-SSA
Vs
0.0073
TransFormer Spike Block2
Qs(K Vs)
0.0148
Linear
0.1635
Linear
0.1314
Channel MLP
Linear 2
0.0054
Conv 1
0.1519 0.1511
Conv2
0.0648 0.0665
Ctr
Conv3
0.0800 0.0752
Conv4
0.1429 0.1439
Conv5
Conv1
0.1519 0.1511
Conv2
0.0818 0.0865
Tracking
Offset
Conv3
0.0912 0.0844
Conv4
0.1273 0.1213
Continued next page
31

---

Table 11 continued from previous page
Image
Event
Conv5
Conv1 I
0.1511 0.151I
Conv2
0.0818 0.0865
Size
Conv3
0.1126 0.1112
Conv4
0.1526 0.1531
Conv5
32


Reviews:
[{"review_id": "9p7rH10EWU", "reviewer": "Reviewer_kyP6", "summary": "This paper proposes SpikeFET, a fully spiking neural network (SNN)-based framework for frame-event object tracking. By addressing the limitations of high computational overhead and inefficient sparse event modeling, the authors introduce a Random Patchwork Module (RPM) to mitigate translation invariance degradation and a Spatial-Temporal Regularization (STR) strategy to improve similarity metrics. Experimental results demonstrate superior performance on multiple benchmarks (FE108, VisEvent, COESOT) while achieving significant power reduction.", "questions": "1. ​​RPM Implementation Details​​: How is the concatenation direction (horizontal/vertical) determined for RPM? Is this choice data-dependent or fixed?\n​​2. Hyperparameter Sensitivity​​: Were α and β in the loss functions tuned on the validation set, or derived from prior work?", "limitations": "Overfitting to datasets with abundant event data may limit applicability to low-event scenarios.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths​​:\n1. RPM effectively reduces positional bias via randomized spatial reorganization and learnable type encoding.\n2. STR enforces temporal consistency between template features, improving similarity metrics.\n3. Strong performance gains on challenging datasets and comprehensive ablation studies.\n\n\n​​Weaknesses​​:\n\n1. The worse performance on VisEvent compared to ViPT and SDSTracker requires further analysis and raises concerns about dataset-specific optimizations.\n2. Complex loss function design, which requires hyperparameter analysis.", "was_revised": true, "final_justification": "Because of the rebuttal, the score changes to BA", "rebuttal": "We thank reviewer kyP6 for the constructive comments, We provide our feedbacks as follows.\n## W1: The worse performance on VisEvent compared to ViPT and SDSTracker requires further analysis and raises concerns about dataset-specific optimizations.\nThank you for the comment. In reality, Spiking Neural Networks (SNNs) have lagged behind Artificial Neural Networks (ANNs) on most benchmarks in recent years, mainly because of their inherently sparse computations. However, as SNNs continue to evolve, the performance gap is steadily narrowing. With our proposed design, SNNs have achieved comparable or even superior performance to ANNs on object tracking tasks.\n\nAdditionally, we posit that SpikeFET is particularly well-suited for processing large-scale datasets and data containing challenging scenarios. In these challenging conditions, the inherent advantages of event cameras become evident, allowing SpikeFET to demonstrate superior performance. In contrast, ViPT and SDSTrack excel in less demanding scenarios. As shown in Table 1, it can be observed that SpikeFET-Base achieved increases of 0.2%, 2.2%, and 4.5% in AUC metrics compared to SpikeFET-Tiny on the FE108 [8], VisEvent [6], and COESOT [7] datasets (arranged in ascending order of data volume), with the performance gap progressively widening. Table 2 reveals that on FE108 [8]—a dataset featuring numerous challenging scenarios (e.g., overexposure, low illumination)—SpikeET's performance is fully leveraged: its AUC is only 4.1% lower than SpikeFET-Tiny, while outperforming all event-based tracking algorithms. In contrast, it underperforms by 17.4% on VisEvent.\n\nBased on this, SpikeFET outperforms ViPT [25] and SDSTrack [19] on COESOT [7] and FE108[8], while slightly underperforming on VisEvent[6] with only a marginal performance drop. Additionally, it exhibits lower power consumption compared to ViPT [25] and SDSTrack [19], which stands as one of SpikeFET's key advantages. Corresponding analyses can be found in lines 251-256. Due to space constraints in the main text, we did not elaborate further, but we will add this in the revised version.\n## W2: Complex loss function design, which requires hyperparameter analysis.\nDue to the loss function paradigm for object tracking [3], we accordingly selected six loss functions from two modalities, as shown in Eq. (13), and retained the same hyperparameters. In addition, we introduced two newly proposed loss functions, $L_{res}$ and $L_{sim}$, resulting in a total of eight loss functions, as detailed in Eq. (14). The corresponding hyperparameter analysis can be found in Appendix E, where we present ablation studies only in the vicinity of the optimal parameters. The ablation experiments validate our hypotheses. This will be clarified in the final version.\n## Q1: RPM Implementation Details​​: How is the concatenation direction (horizontal/vertical) determined for RPM? Is this choice data-dependent or fixed?​​\n(1) The RPM we proposed determines the connectivity direction for each batch through random selection, where every batch may choose either horizontal or vertical orientation. We must align every sample within the same batch to the same direction so that they can be properly stacked into that batch. Such randomization can create greater positional variability.\n(2) The selection of connection directions is randomly chosen for any data—it is neither data-dependent nor predetermined.\n## Q2: Hyperparameter Sensitivity​​: Were α and β in the loss functions tuned on the validation set, or derived from prior work?\nThe $\\alpha$ and $\\beta$ hyperparameters in our loss function were both tuned on the validation set, primarily for adjusting model stability. To maintain consistency with the magnitude range of hyperparameters in object tracking, we performed tuning within the range of 0.1 to 5.\n\nWhen $\\alpha$ is excessively large, the fusion results become overly dependent on the poorer-performing modality. When $\\alpha$ is too small, the model cannot achieve sufficient fusion. When $\\beta$ is excessively large, it overly forces inconsistent features to align. When $\\beta$ is too low, it fails to fully achieve the STR effect. After tuning, we selected $\\alpha=1$ and $\\beta=0.5$ as the final parameters for the model.\n## Limitations: Overfitting to datasets with abundant event data may limit applicability to low-event scenarios.\nThank you very much for your question. We believe that in low-event scenarios, relying solely on the event modality would lead to poor tracking performance. Given this limitation, we introduced frames to compensate for the missing texture information in event data. Our proposed SpikeFET architecture effectively fuses image and event features within an SNN, fully leveraging the advantages of both modalities. As a result, SpikeFET achieves robust performance not only in low-event scenarios such as static or slow-moving scenes, but also in high-event scenarios like fast motion, thereby overcoming the performance degradation observed when using event data alone. We will add these visualization results in the revised version."}, {"review_id": "kJLcJQsoLW", "reviewer": "Reviewer_gksX", "summary": "This paper proposes SpikeFET, a fully spiking neural network (SNN) framework for unified frame-event object tracking. Built upon a dual-single-dual architecture, the framework effectively integrates convolutional and Transformer-based SNN modules for local and global modeling, while preserving the low-power advantages of SNNs. The introduced Randomized Patchwork Module (RPM) and Spatio-Temporal Regularization (STR) address key architectural limitations—namely translation invariance degradation due to convolutional padding and temporal misalignment between features. Experiments on three major benchmarks (FE108, VisEvent, COESOT) demonstrate that SpikeFET achieves state-of-the-art tracking performance with significantly lower power consumption.", "questions": "1. While the methodology is technically solid, the model relies heavily on pretraining with frame-based datasets (e.g., COCO, LaSOT), which raises concerns about its ability to generalize to event-only scenarios. If image frames are unavailable and only event data is used, can the model maintain its performance?  Can it generalize well? \n2. Also, would the pre-trained model show different effects when using both datasets?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The first fully SNN-based framework for frame-event fusion tracking, with novelty and relevance.\n2. The architecture is well-designed, combining convolutional and Transformer-style SNN components to exploit both spatial and temporal information.\n3. RPM and STR modules are effective in addressing structural weaknesses and enhancing feature consistency.\n4. The experimental section is thorough, with comparisons to strong baselines and detailed ablation studies.\n\nWeaknesses:\n\n1. While the methodology is technically solid, the model relies heavily on pretraining with frame-based datasets (e.g., COCO, LaSOT), which raises concerns about its ability to generalize to event-only scenarios. \n2. If image frames are unavailable and only event data is used, it is uncertain whether the model can maintain its performance, and it is also unclear whether it can generalize well. \n3. Additionally, it is not known whether the pre-trained model would show different effects when using both datasets.", "was_revised": true, "final_justification": "I appreciate the authors’ efforts in responding to the comments. I will maintain my first score.", "rebuttal": "We thank Reviewer gksX for the valuable comments.\n## W1 and Q1: While the methodology is technically solid, the model relies heavily on pretraining with frame-based datasets (e.g., COCO, LaSOT), which raises concerns about its ability to generalize to event-only scenarios.\nWe sincerely appreciate the reviewer’s insightful comments and apologize for any confusion caused by our previous unclear descriptions. We would like to clarify that our model does not rely solely on pre-training with frame-based datasets to achieve strong performance. In particular, for both SpikeFET-Tiny and SpikeET, we do not use frame-based pre-training, yet they outperform existing non-pre-training methods. For a fair comparison with approaches that do incorporate pre-training, such as ViPT [25] and SDSTrack [19], we pre-trained SpikeFET-Base using the same datasets—GOT-10k [43], TrackingNet [42], COCO [40], and LaSOT [41]. As a result, our model also achieved state-of-the-art performance on the COESOT [7] and FE108 [8] datasets.\n\nAdditionally, there is no need to concern yourself about our model's generalization capabilities in purely event-based scenarios. As shown in Table 2, SpikeET—the event-based variant of SpikeFET—demonstrates superior performance on event-exclusive datasets even without pre-training, achieving state-of-the-art (SOTA) results.\n## W2 and Q1: If image frames are unavailable and only event data is used, it is uncertain whether the model can maintain its performance, and it is also unclear whether it can generalize well.\nAs demonstrated in W1 and Table 2, our model achieves superior performance on event-only data while exhibiting enhanced generalization capability. The improved generalization ability of our model on event stream data stems from the inherent suitability of SNNs for processing sparse event stream data, which enables more effective learning from event-based representations.\n## W3 and Q2: Additionally, it is not known whether the pre-trained model would show different effects when using both datasets.\nThank you for the insightful comment. We believe that, in theory, pre-training with frame-event paired datasets should yield better results than current methods. However, the open-source community currently lacks a substantial volume of such paired datasets, preventing us from empirically validating this hypothesis. Nevertheless, in our subsequent work, we will verify this using a large-scale dataset we are currently collecting and will release it, filling this gap in the open-source community."}, {"review_id": "Ikom02XpxA", "reviewer": "Reviewer_dnLE", "summary": "The authors present SpikeFET, the first full SSN based object tracker using both events and frame data. The network adds a Random Patchwork Module and Spatial-Temporal Regularization to improve spatial-temporal invariance and consistency. Features are extracting independently for frames and events of image triplets. The features are then fused with a Spiking based transformer. By feeding two samples of the template into the tracking network, end performance can dramatically improve. SpikeFET is extensively tested across 2 benchmarks and compared with a diverse set of baselines. SpikeFET can improve tracking performance while minimizing power use of the system.", "questions": "1.\tWhat is the expected processing rate of SpikeFET? How big are the event bins? How much delay is in the SNN between input to output spike?\n2.\tWhat is the difference between table 1 and 2? The caption suggests table 2 is on event only input, but at least [44] and [45] are frame-based methods. The authors might consider merging the tables and adding a column indicating the method inputs to be clearer. \n3.\tBased on the ablation, RPM is the primary component. How much of the improvement is due to simply providing the network with another sample of the template (i.e. more data), than a fundamental insight into model architecture/training processes?\n4.\tCan the authors comment on parameter scaling SpikeFET? On FE108, it seems 3x the parameters had very little effect, while on COESOT it had a larger effect. \n5.\tSince frame-based cameras require more energy than event cameras, how does using a fused approach compare to an event only system in terms of overall energy used.\n\nIf the authors can clarify these points, I will raise my score.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n-\tSpikeFET is a novel method for object tracking with fused event + frame data.\n-\tSpikeFET has state of the art performance. The authors compare against a variety of other methods including frame only, event only, and frame+event methods by reporting results on 3 different datasets.\n-\tThe method diagrams aide understanding SpikeFET’s flow and how RPM and STR operate.\n-\tThe ablation study effectively compares the major steps in SpikeFET (RPM, Et, STR), and the choices in fusion method.\n\nWeakness\n\n-\tThe paper only visualizes 2 samples of tracking results, making it difficult to evaluate the method’s success.\n-\tInference mode is lacking detail. For instance, how are the 2 template images chosen?\n-\tEvaluation is theoretical/in simulation only and power usage is missing for some baseline methods.\n-\tMany important details are relegated to the appendix instead of the main paper (i.e. network input and datasets).\n-\tLots of baseline methods are shown, but a description of method differences between SpikeFET and them is lacking.\n-\tThe primary novelty that contributes to improvement over baselines seems to be in passing more data into the network, which is not surprising or significant.", "was_revised": true, "final_justification": "The authors resolved most of my concerns. Although the paper is not state of the art in all aspects, given its the first fully SNN method for this task, its an interesting direction for future research. Physical testing on neuromorphic chips would have made this a more solid accept.", "rebuttal": "We sincerely thank Reviewer dnLE for the constructive and valuable comments. The concerns are addressed as follows.\n## W1: The paper only visualizes 2 samples of tracking results, making it difficult to evaluate the method’s success.\nThank you for the reviewers' comments. We have conducted extensive visualizations, and SpikeFET consistently demonstrates strong performance. However, due to space limitations, we only included two figures in the current manuscript. We will incorporate more visualization results into the revised version.\n\n## W2: Inference mode is lacking detail. For instance, how are the 2 template images chosen?\nWe appreciate the reviewers for identifying this limitation, which was an oversight in our work. In our inference process, we did not employ a dynamic template update strategy. Instead, we consistently replicated the first frame to use two template frames (both derived from the first frame) and one search frame for inference. We will provide this clarification after lines 213–214 in the revised manuscript.\n## W3: Evaluation is theoretical/in simulation only and power usage is missing for some baseline methods.\nWe sincerely appreciate the reviewer for highlighting this critical limitation. Currently, most mainstream studies primarily compare theoretical power consumption, lacking practical power measurements. This gap exists because a model's actual power consumption is hardware-dependent. **To ensure fair comparisons across methods, we adopted a widely-used hardware-agnostic approximation method [62, 63] for estimating power consumption in both ANNs and SNNs**. We are now conducting tests on relevant chips and will deploy our SNNs algorithm onto neuromorphic chips for practical power evaluations in future work.\n\nWe acknowledge that for certain baselines of SpikeFET—such as DiMP50 [44], PrDiMP50 [45], SiamRCNN [46], TrDiMP50 [20], TransT50 [2], ToMP101 [47], and HRMontrack [22]—we referenced metrics reported in existing literature. This approach was necessary because these methods originally started as frame-based trackers and were extended to frame-event fusion trackers using early fusion approaches. Due to our limited access to implementation specifics, we were unable to reproduce their algorithms or precisely calculate theoretical power consumption. Consequently, we derived an approximated computational result based on our reproduction efforts, which will be supplemented in the revised manuscript. For FENet [8], we obtained a theoretical power consumption of 262.2 mJ, which also exceeds that of SpikeFET-Base.\n\nFor certain baselines of SpikeET, such as DiMP50 [44], PrDiMP50 [45], SiamRCNN [46], and ATOM [48], their theoretical power consumption values are detailed in the following table:\n\n|Method|DiMP50 [44]|PrDiMP50 [45]|SiamRCNN [46]|ATOM [48]|\n|-|-|-|-|-|\n|**Power (mJ)**|256.37|258.37|30.199|203.88|\n\nSpikeET **maintains the lowest power consumption**, and we will include this comparative analysis in the revised manuscript.\n## W4: Many important details are relegated to the appendix instead of the main paper (i.e. network input and datasets).\nWe apologize for any confusion caused by placing important details in the appendix due to space constraints; we will reformat and incorporate these details in the revised edition.\n## W5: Lots of baseline methods are shown, but a description of method differences between SpikeFET and them is lacking.\nWe thank the reviewer for this valuable comment. To the best of our knowledge, the proposed SpikeFET is the first fully-spiking frame–event fusion tracker, whereas all baselines—e.g., DiMP50 [44], OSTrack [3], CEUTrack [7], ViPT [25], and SDSTrack [19]—are built upon artificial neural networks (ANNs). MMHT [15], on the other hand, adopts an ANN-SNN hybrid architecture and is therefore not a fully-spiking tracker either. Furthermore, SpikeFET borrows the overall pipeline from CEUTrack but replaces the patch embedding with several convolutional layers, employs two spiking tracking heads, and converts every component into SNNs form. ViPT [25] is likewise built upon OSTrack [3] and treats the event modality as an auxiliary cue, whereas SDSTrack [19] extends OSTrack [3] with a self-distillation scheme. Earlier methods such as DiMP50 [44] and SiamRCNN [46] adopt Siamese architectures. We will add this clarification in the revised manuscript.\n## W6 and Q3: The primary novelty that contributes to improvement over baselines seems to be in passing more data into the network, which is not surprising or significant.\nWe clarify that the performance improvement is not achieved by adding more data. In addition, SNNs currently underperform ANNs in complex computer vision tasks such as detection and tracking. For the first time, we propose a fully-spiking fusion tracking network that achieves performance on par with ANNs — and even surpasses ANNs on some datasets — while consuming significantly less power than ANNs. The analysis is as follows:\n\nIn our ablation study (Table 3 #1), we used **six input images** (line 279). This configuration demonstrates that when using only **two template frames and one search frame** per modality without the RPM and adopting conventional input tokens, the model performance significantly deteriorates. Particularly on the FE108 dataset, this setup causes ~50% performance degradation compared to configuration #2.\n\nAdditionally, we conducted experiments without RPM using one template frame and one search frame per modality, resulting in four total inputs (two modalities × two inputs each). The results are presented in the table below:\n|Configuration|4-input|6-input|6-input with RPM|\n|-|-|-|-|\n|**AUC (%)**|30.8|32.1|66.8|\n|**PR (%)**|43.9|45.0|94.9|\n\nWe observe that without RPM, the six-input configuration yields only marginally higher performance than the four-input setup on FE108 [8]. This conclusively demonstrates that the primary innovation driving baseline improvement is RPM's effective mitigation of padding-induced degradation in translation invariance. In contrast, utilizing additional data provides minimal performance gains relative to RPM's contribution. We sincerely appreciate your emphasis on these important related works. We will incorporate the ablation results for the four-input configuration into the revised manuscript.\n## Q1: What is the expected processing rate of SpikeFET? How big are the event bins? How much delay is in the SNN between input to output spike?\n(1) We stack the event streams into event frames for training and inference based on the characteristics of each dataset. For COESOT [7], we use a 40ms time window; for FE108 [8] and VisEvent [6], we use the time interval between adjacent image timestamps. Thus, SpikeFET processes event stream data from each 40ms window or the image sampling interval, i.e., the processing rate is 25Hz or matches the image frame rate. In theory, a 10ms window or faster could be used, but to ensure a fair comparison with existing methods, we chose 40ms. \n(3) The end-to-end latency measurements for SpikeFET-Tiny, SpikeFET-Base, and SpikeET on an RTX 4090 GPU are 106 Hz, 49.26 Hz, and 185.87 Hz respectively.\n## Q2: What is the difference between table 1 and 2? The caption suggests table 2 is on event only input, but at least [44] and [45] are frame-based methods. The authors might consider merging the tables and adding a column indicating the method inputs to be clearer.\nWe thank the reviewer for pointing out the error in our paper. We incorrectly listed identical results for DiMP [44] on FE108 [8] in two different tables; this was an oversight on our part. Table 1 reports methods that are trained and evaluated on frame-event fused data, whereas Table 2 reports methods trained and evaluated solely on event data. In Table 2, [44] and [45] denote frame-based models that were trained and tested on event data. After the event-to-frame conversion described in Appendix A.2, the event stream is transformed into event frames, enabling frame-based models to process event data. We appreciate the suggestion and will merge the two tables in the revised version, adding a column to indicate the input modality.\n## Q4: Can the authors comment on parameter scaling SpikeFET? On FE108, it seems 3x the parameters had very little effect, while on COESOT it had a larger effect.\nAs discussed in lines 243–244 of the original manuscript, we believe that the FE108 [8] dataset is both too small and too simple, causing our SpikeFET-Base model to tend toward overfitting. By contrast, COESOT [7] is larger and more challenging—for example, the ratio of training sequences is 76 : 827 in their favor—so COESOT [7] exerted a greater influence. The near-identical performance of Tiny and Base on FE108 [8] is likewise observed in SDTrack [16], which improves AUC by only 0.9 % and PR by merely 0.2 % relative to other datasets, a marginal gain.\n## Q5: Since frame-based cameras require more energy than event cameras, how does using a fused approach compare to an event only system in terms of overall energy used.\nWe clarify that because event images are sparse while frame images are dense, the spike firing rate of event images is lower than that of frame images. This means that frame-based cameras require more energy. Furthermore, fusion methods require two separate branches to process both modalities, resulting in higher power consumption than processing either single modality alone. Therefore, the power consumption of a fusion network is ultimately higher than that of an event-only network in theory, but still lower compared to that of ANNs.\n## Q6: Figure 1 should make it clear the \"power\" is a theoretical value and not a real world measurement of the systems.\nThank you for pointing out this issue. We will provide an accurate definition of this term and make the necessary revisions in the revised manuscript."}, {"review_id": "tgAVPFFmFg", "reviewer": "Reviewer_VsCN", "summary": "This paper proposes a fully SNN-based framework for multimodal tracking with frame and event data. The architecture comprises a ConvFormer Spike Block for modality-specific feature extraction and a TransFormer Spike Block for feature fusion. The core contributions are: (1) the Randomized Patchwork Module (RPM), which mitigates convolution-induced positional bias via stochastic spatial reordering; and (2) a Spatial-Temporal Regularization (STR) strategy that enhances the consistency of dual-template features in latent space. Experiments on multiple datasets demonstrate the effectiveness of the proposed method.", "questions": "See above Weaknesses", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:** \n\n•  The exploration of a fully SNN-based framework for event-driven tracking is commendable, as most existing methods rely on hybrid ANN-SNN designs.\n\n•  The proposed method achieves performance on par with ANN-based models, highlighting the effectiveness of the fully SNN architecture.\n \n**Weaknesses:**\n \n•  The model is pretrained on large-scale RGB datasets such as COCO [40], LaSOT [41], TrackingNet [42], and GOT-10K. However, the fairness of comparison with other methods is unclear.\n\n•  The proposed RPM and SDTrack modules treat the search and template frames similarly. Moreover, RPM is designed to mitigate the degradation of translation invariance caused by padding, yet it is only applied in one direction. It remains unclear whether padding artifacts persist in the other directions.\n\n•  Besides parameter count, inference speed is also a critical metric for tracking tasks. Please provide a runtime comparison with other methods.\n\n•  The paper does not specify the timestamp configuration used in the experiments.\n\n•  As shown in Table 3, the baseline performs poorly while the addition of RPM leads to a large performance gain. Such a dramatic improvement warrants further explanation and justification.", "was_revised": true, "final_justification": "The authors’ responses have addressed most of my concerns, and I have accordingly increased my score. However, the reported runtime performance still requires more detailed information, as there appear to be discrepancies between the speeds presented in this work and those reported in the original papers.", "rebuttal": "We thank reviewer VsCN for the valuable and constructive comments. We address the concerns as follows.\n## W1: The model is pretrained on large-scale RGB datasets such as COCO [40], LaSOT [41], TrackingNet [42], and GOT-10K. However, the fairness of comparison with other methods is unclear.\nThanks for pointing out the misunderstanding regarding Table 1. We clarify that to ensure a fair comparison with pre-training methods such as ViPT [25] and SDSTrack [19], we followed their approach by pre-training SpikeFET-Base on COCO [40], LaSOT [41], TrackingNet [42], and GOT-10K [43]. The results demonstrate that SpikeFET-Base outperforms both methods on FE108 [8] and COESOT [7]. In contrast, to compare with most non-pre-trained baselines, SpikeFET-Tiny and SpikeET were not pre-trained on these datasets. Their results also show superior performance over the majority of non-pretrained methods, except for MMHT's [15] AUC performance on COESOT [7]. We will explicitly highlight this distinction in the revised table and corresponding textual description.\n## W2: The proposed RPM and SDTrack modules treat the search and template frames similarly. Moreover, RPM is designed to mitigate the degradation of translation invariance caused by padding, yet it is only applied in one direction. It remains unclear whether padding artifacts persist in the other directions.\nWe thank the reviewer for the comment. We clarify that our proposed RPM is fundamentally different from the Intrinsic Position Learning (IPL) introduced in SDTrack [19]. First, IPL is motivated by the idea that a joint positional embedding can enhance the cross-correlation performance; its goal is to effectively learn positional information. In contrast, RPM stems from the observation that the padding in convolutional layers introduces a systematic bias during tracking and degrades translation invariance; its purpose is to mitigate this degradation. Second, IPL adds redundant padding information to the input image, which hurts SDTrack's [19] performance, whereas RPM actively reduces such redundancy and further boosts accuracy. Finally, although at inference RPM randomly selects only one direction, during training it considers all four possible placements—top, bottom, left, and right—of the search region within the padded image, thereby eliminating the biasing effect of padding in the unused directions.\n## W3: Besides parameter count, inference speed is also a critical metric for tracking tasks. Please provide a runtime comparison with other methods.\nWe clarify that runtime is not a primary metric considered in this work, as **it can be reduced through hardware acceleration.** However, **based on suggestions, we have added a comparison of the runtime between SpikeFET/SpikeET and other methods on an RTX 4090 GPU.** For certain baselines of SpikeFET, such as DiMP50 [44], PrDiMP50 [45], SiamRCNN [46], TrDiMP50 [20], TransT50 [2], and ToMP101 [47], we have referenced the AUC metrics from other papers. This is because these methods, originally frame-based trackers, were extended to frame-event fusion trackers using early fusion approaches. Due to our lack of detailed understanding of their specific implementations, we were unable to replicate their algorithms and thus could not accurately calculate their inference times. The overall results are presented in Rebuttal Table A. Specifically: our SpikeFET-Base inference time outperforms those of SDSTrack [19] and HRMonTrack [22]. Beyond this, the inference time of SpikeFET-Tiny is comparable to other methods and falls within an acceptable range. Additionally, our SNN algorithm can be deployed on neuromorphic chips in the future to further reduce inference time.\n**Rebuttal Table A.** Comparison of inference time between SpikeFET and other baselines, with inference time tested on RTX 4090 using the same testing methodology as OSTrack [3].\n|Method|FENet [8]|OSTrack [3]|CEUTrack [7]|HRCEUTrack [22]|HRMonTrack [22]|ViPT [25]|SDSTrack [19]|MMHT [15]|SpikeFET-Tiny|SpikeFET-Base|\n|-|-|-|-|-|-|-|-|-|-|-|\n|**Inference Speed (FPS)**|92.45|127.8|116.53|123.38|39.7|135.43|42.03|85.48|106.23|49.26|\n\nFor SpikeET, we tested all methods except SNNTrack [14] and SDTrack [16] due to their non-public implementations. The overall results are presented in Rebuttal Table B. Specifically: our SpikeET inference time outperforms most networks and remains comparable to other methods within an acceptable range. Additionally, our SNN algorithm can be deployed on neuromorphic chips in the future to further reduce inference time. We appreciate your emphasis on these critical points. All testable methods will be included in the revised version for inference speed evaluation.\n**Rebuttal Table B.** Comparison of inference time between SpikeET and other baselines, with inference time tested on RTX 4090 using the same testing methodology as OSTrack [3].\n|Method|DiMP50 [44]|PrDiMP50 [45]|ATOM  [48]|SiamRPN [49]|STARK [4]|SimTrack [50]|OSTrack [3]|\n|-|-|-|-|-|-|-|-|\n|**Inference Speed (FPS)**|71.93|72.82|121.11|518.55|172.19|266.25|204.38|\n|**Method**|**ARTrack [51]**|**SeqTrack [52]**|**HiT [53]**|**GRM [54]**|**HIPTrack[55]**|**STNet [13]**|**SpikeET**|\n|**Inference Speed (FPS)**|97.4|113.09|437|153.18|110.23|163.40|185.87|\n## W4: The paper does not specify the timestamp configuration used in the experiments.\nAs detailed in Appendix A.2, we convert the event stream into an event frame by uniformly partitioning a temporal slice of the stream into B = 3 segments and mapping the events in each segment into one of the three RGB channels. On COESOT [7] we employ a 40 ms slice for each segment, while on FE108 [8] and VisEvent [6] we instead collect the events that fall between the timestamps of two consecutive image frames, split that interval into three equal sub-intervals, and map them to the RGB channels; the resulting event frames are used for training and inference.\n## W5: As shown in Table 3, the baseline performs poorly while the addition of RPM leads to a large performance gain. Such a dramatic improvement warrants further explanation and justification.\nWe thank the reviewer for this insightful observation; it is precisely the motivation behind our RPM module. We attribute the observed phenomenon to two key factors.\n\nFirst, padding severely undermines translation invariance and thereby degrades performance, a problem that has already been noted in works such as SiamRPN++ [17] and SiamDW [18], whose accuracies increased markedly once padding effects were mitigated. With sparse event data—where informative content is already limited—the detrimental impact of padding becomes even more pronounced. Consequently, our RPM module yields larger accuracy gains.\n\nSecond, FE108 [8] outperforms VisEvent [6] by a wider margin because FE108 [8] contains not only simple scenes (stationary camera, moving object in a fixed environment) but also challenging conditions such as high-exposure and low-light sequences. Our SNN-based SpikeFET excels at leveraging the sparse event information in these challenging settings, leading to larger improvements on FE108 [8] than on VisEvent [6]."}]

Author rebuttal:

Meta-review: The authors introduce a fully spiking tracking method based on frames and events, SpikeFET. They introduce two mechanisms, Randomized Patchwork Module (RPM) and Spatio-Temporal Regularization (STR), to improve translation invariance and better temporal feature alignment. The results are state of the art.

The reviewers raised several issues, from unclarity on the fairness of comparisons to whether and how pretraining is performed based on frames, the choosing of the template frames, and whether improvements come from more data only or the RPM. These issues were successfully addressed by the authors.

A discussion evolved on the measurement of the power savings by using an SNN for the frame-event fusion. The authors base themselves on well-established theoretical measures, counting MACs and ACs. A reviewer remarks that the numbers are based on old technology. Moreover, it has been remarked that an implementation on neuromorphic hardware and actual measurements would be better. This is true, but the procedure does align with common practice in the field.

Another issue is the explanation of the large difference between the performance with and without RPM, more than doubling the AUC and PR. The authors put this on the detrimental influence of the padding on translation invariance. Although this may be true, more in-depth analysis of this matter would be desirable. This could involve other data sets, or padding not with the searched for pattern but with other, unrelated data. 
 
All in all, the reviewers find most of their main concerns addressed.
Decision: Accept (poster)