PASS: Path-selective State Space Model for Event-based Recognition
Authors: Jiazhou Zhou
Kanghao Chen
Lei Zhang
Lin Wang
Venue: NeurIPS 2025 poster ()
Keywords: Event Camera
Object Recognition
Arbitrary Temporal Frequency
Primary area: applications

Abstract: Event cameras are bio-inspired sensors that capture intensity changes asynchronously with distinct advantages, such as high temporal resolution. Existing methods for event-based object/action recognition predominantly sample and convert event representation at every fixed temporal interval (or frequency). However, they are constrained to processing a limited number of event lengths and show poor frequency generalization, thus not fully leveraging the event's high temporal resolution. In this paper, we present our PASS framework, exhibiting superior capacity for spatiotemporal event modeling towards a larger number of event lengths and generalization across varying inference temporal frequencies. Our key insight is to learn adaptively encoded event features via the state space models (SSMs), whose linear complexity and generalization on input frequency make them ideal for processing high temporal resolution events.  Specifically, we propose a Path-selective Event Aggregation and Scan (PEAS) module to encode events into features with fixed dimensions by adaptively scanning and selecting aggregated event presentation. On top of it, we introduce a novel Multi-faceted Selection Guiding (MSG) loss to minimize the randomness and redundancy of the encoded features during the PEAS selection process. Our method outperforms prior methods on five public datasets and shows strong generalization across varying inference frequencies with less accuracy drop (ours -8.62% v.s. -20.69% for the baseline). Moreover, our model exhibits strong long spatiotemporal modeling for a broader distribution of event length (1-10^9), precise temporal perception, and effective generalization for real-world scenarios. Code and checkpoints will be released upon acceptance.

PASS: Path-selective State Space Model for Event-based Recognition
Jiazhou Zhou
Kanghao Chen
AI Thrust, HKUST(GZ)* International Digital Economy Academy jzhou297@connect. www.hhuutteee edu
AI Thrust, HKUST(GZ))
kchen879@connect. hkust-gz... edu.
Lei Zhang
Lin Wang!
International Digital Economy Academy 1e leizhang@idea edu.
School Electrical and Electronic Engineering,
Nanyang Technological University
1 linwang@ntu. edu.t. sg
Abstract
Event cameras are bio-inspired sensors that capture intensity changes asyn- chronously with distinct advantages, such as high temporal resolution. Existing methods for event-based object/action recognition predominantly sample and con- vert event representation every fixed temporal interval (or frequency). However, they are constrained to processing limited number of event lengths and show poor frequency generalization, thus not fully leveraging the event's high temporal resolu- tion. In this paper, present our PASS framework, exhibiting superior capacity for spatiotemporal event modeling towards a larger number of event lengths and gener- alization across varying inference temporal frequencies. Our key insight is to learn adaptively encoded event features via the state space models (SSMs), whose linear complexity and generalization on input frequency make them ideal for processing high temporal resolution events. Specifically, we propose Path-selective Event Aggregation and Scan (PEAS) module to encode events into features with fixed dimensions by adaptively scanning and selecting aggregated event presentations. On top of it. introduce novel Multi-faceted Selection Guiding (MSG) loss minimize the randomness and redundancy of the encoded features during the PEAS selection process. Our method outperforms prior methods five public datasets and shows strong generalization across varying inference frequencies with less accuracy drop (ours -8.62% v.s. -20.69% for the baseline). Overall, PASS exhibits strong long spatiotemporal modeling for a broader distribution of event length (1-10"), precise temporal perception, and generalization for real-world scenarios.
Introduction
Event cameras bio-inspired sensors that trigger signals when the relative intensity change exceeds threshold, adapting to scene brightness, motion, and texture. Compared with standard cameras, event cameras output asynchronous event streams, instead fixed frame rates. They offer distinct advantages, such as high dynamic range, microsecond temporal resolution, and low latency [20, 15, cameras have been applied to address various vision tasks, such as
73, 6]. Due to these merits, object/action recognition [9, 4, 31, 74, 76, 77, 54, 7, 17, 16, 35, 56, 55, 48, 1].
Project page: httpps//giithub.com http://gtthubb.cco//acooouooooe. aaiaaaaauuugaaand/AAaSSaS Hamepagee.. Corresponding Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Distribution of event length
Comparison
Train & Val sampling frequencies
[1,107]
Previous methods
F
Train 20 Hz Val 80Hz
Train 20 Hz
[1,10"]
Val 40Hz
PASS our methods
Train 20
[1,10']
Vol 80Hz
Figure I: Compared to previous event-based recognition methods limited to a narrow distribution of event length and poor temporal frequency generalization, our method, PASS, advances spatial- temporal event modeling across a broader distribution of event length ranging from to 10 and demonstrates superior temporal frequency generalization.
The spatiotemporal richness of events introduces complexities in data processing and necessitates models that can efficiently process and interpret them. To address this problem, existing methods predominantly sample and aggregate them at every fixed temporal interval, i.e., frequency. In this way, the raw stream can be converted into different event representations [75, 80, 2, 54, 66, 44, 32,5), 58]. In general, existing methods mainly follow two representative model structures: (a) step-by-step structure models [66, 69, 77, 75, 74, 30, 16] and (b) recurrent structure models [54, 79]. The former processes all time-step event frames in parallel, employing local-range and long-range temporal modeling sequentially, as shown in Fig. 2 (a). By contrast, the latter process event frames sequentially each time step. updating memory feature that affects the next input, as illustrated in Fig. (b).
However, current models face two pivotal challenges, as shown in Fig. 1. 1) Limited distribution of event length. Event cameras offer high temporal resolution, naturally generating dense event sequences. This necessitates models effectively processing events across broad distribution of event length, especially for high-speed scenarios or long-duration event streams [80]. However, current event-based recognition datasets [8, 21, 47] are restricted to a limited number of event lengths (106 1 (see appendix for existing dataset summary) and face computational bottlenecks for large event lengths due to quadratic attention complexity, thereby constraining exploration of spatiotemporal relationships across broad distribution of event lengths. 2) Limited inference frequency generalization. While event-based cameras offer high temporal resolution beneficial for recognizing objects and actions in high-speed, dynamic visual scenarios [80], current recognition models significantly degrade when inference frequencies differ from the training one, thereby limiting the full potential of these high-resolution event streams. For example, as shown Fig. 5 (b), the model trained at 60 Hz with existing event sampling strategies demonstrates poor generalization, with performance dropping up to 20.69% when evaluated at 20 Hz and 100 Hz sampling frequencies.
Recently, the selective state space model (SSM) has rivaled the previous backbone like vision transformer in performance while significantly reducing memory usage due to linear- scale complexity, showing robust generalization across 1D audio [24] and 2D image signals [42] when evaluated at varied frequencies. Given the inherent spatiotemporal richness due to events' high temporal resolution. natural motivation arises for harnessing the exceptional power of SSM for event spatiotemporal modeling. To this end, we propose PASS, novel framework for recognizing event streams capable of processing broad distribution of event length ranging from 10 to 10 and generalizing across varying inference frequencies, as depicted in Fig. 1. By harnessing the linear complexity and strong input frequency generalization of SSM. PASS delivers exceptional recognition performance and frequency generalization. It brings two key technical breakthroughs.
Firstly, since the large number of event length could cause difficulties for SSM in effectively learning the spatiotemporal properties from events, as SSM's hidden state updates rely heavily on the sequence length and feature order. To this end, we propose novel Path-selective Event Aggregation and Scan (PEAS) module to aggregate and convert events into sequence features with fixed dimensions Concretely, as shown in Fig. 3, a selection mask is first learned from the original event frame representation to facilitate the frame selection. Then, the bidirectional event scan is conducted on the selected perimeters to convert them into sequence features. This adaptive process ensures the

---

event scan path is end-to end-to-end learnable and responsive to every event input, thus enabling our PASS effectively process event streams across broad distribution of event length (Tab. 4).
Secondly, the varying sampling frequencies hinder the generalization of SSM during inference, as empirically verified in Tab. 5. This suggests that alterations in the input sequence length and order due to sampling frequency shifts greatly affect model performance. For this reason, we propose novel Multi-faceted Selection Guiding (MSG) loss. minimizes the randomness of the PEAS module event frame selection process caused by the random initialization of the selection mask's weight. Our MSG loss better facilitates alleviating the redundancy of the selected event frames, thus strengthening model generalization across varying inference frequencies (Tab. 5).
Extensive experiments across five public and three proposed datasets demonstrate PASS's 's superior performance. It outperforms previous methods by +3.45%, +0.38% +8.31 % +2.25% and +3.43% on the public PAF, SeAct, HARDVS, V-Caltech101. and N-Imagenet datasets, respectively. Additionally, PASS shows superior generalization across varying inference frequencies, with a maximum accuracy drop of -8.62% compared to -20.69% for previous methods. Given the absence of event-based recog- nition datasets with a large number of event length, we created two synthetic datasets and recorded one real-world dataset with around 10 event length: ArDVSI00 covers 100 action transitions with different meta-actions, GemArDVS100 100 features the same meta-actions yet in different combinations to evaluate the model's fine-grained temporal recognition ability, and Real-ArDVS10 dataset contains 10 recorded action transitions to assess the model's 's real-world generalization. Our PASS exhibits strong long spatiotemporal modeling across a broad distribution of event length (1-10%), precise temporal perception, and effective generalization for real-world scenarios, achieving 97.35%, 89.00% and 100% Top-1 accuracy on ArDVS100, TeMArDVS, and Real-DVS10 datasets. Our main contribution can be summarized as follows:
We propose PASS, novel framework for recognizing events across broad count distribu- tion (event length range from 10 to 10") and generalizing to various inference frequencies.
We introduce the PEAS module to convert asynchronous events into ordered sequence features, alongside MSG loss to promote effective event spatiotemporal modeling.
Extensive experiments prove PASS's superior performance and strong inference frequency generalization. The proposed ArDVS100, TemArDVS100 1000, and Real-ArDVS10 datasets prove the model's long spatiotemporal modeling, fine-graine fine-grained temporal perception, and real-world effectiveness, respectively.
2
Related Works
Event Input
Event Input
1...... ...
Logits
EN Logits
#g W
tiaN
(a) Step-by-step Structure Model
(b) Recurrent Structure Model
Long-range
Local-range
Local-range
Current
Latent
Memory
Encoder
Encoder
Feature
Vectors
Vectors
Update
Figure 2: Comparison of two model structures for previous event-based recognition methods.
Event-based Object/ Action Recognition. Existing event-based recognition works cover two main tasks: object recognition [75, 74, 15, 29, 73, 18, 25, 10, 35, 38] and action recognition [77, 66, 54, 65, 17, 48, 64, 39]. Specifically, object recognition captures stationary objects around 10 events, whereas action recognition records dynamic human actions with approximate e² events. These methods tackle high temporal resolution event spatiotemporal complexity via two key approaches, as shown in Fig. 2: 1) step-by-sten structure models and 2) recurrent structure models. Initially, the events are sampled into slices at fixed time intervals. The step-by-step structure models then use off- the-shelf backbones to extract local-range spatiotemporal features from event slices and then perform
3

---

Spatiotempore
Path-selective Event Aggnegation and Scan Module
Multi-faceted Selection Guiding Less
Modeling Module
Exent Stampi ing 6 Frome Ago Hlam
Sumbel Soffmar
Frow
(Ro.3)
Recogeil Prediction
Event Comora
Clssirrcaioon
Score Score Predictor
Wortar
Lapers
B B-Mamba Black
Soffmar Mlor
Intenframe ovent Matuol jufo wireron Lewn Cowerill
Event Counts Distribution
trtff!
Putch
buning Training
Event Histogram
axis Position Embedding
CLS Tokens
Buring Evaluation
Bidirectional Event Soon
Eirsum Matrix Production
Joint Event Count Histoyrom
Y-axis Pasition Embedcling
Spatial Embeddings
Volue Addition
Temporal Embeddings
Figure 3: Overview of our proposed PASS framework.
long-range temporal modeling using various methods, such as simple average operation [77, 75], proposed modules [66, 69], and loss guidance [74, 30J. Recurrent structure models [54, 79], on the other hand, process the event slices sequentially, updating their hidden state based on the input at each time step. Both structures ensure adaptability to varying event lengths. However, step-by-step structure models struggle with high computational complexity, especially for handling more events in high-speed and long-duration scenarios. Recurrent structure models tend to forget the initial information due to their simplistic recurrent design and require longer training time due to their inability to process data in parallel. Additionally, evidenced in Tab. 5, existing methods struggle to generalize across different inference frequencies, which is essential for applications high-speed, dynamic visual scenarios [80]. In this work, we aim to improve event basd- recognition across broad distribution of event length with improved generalization across varying inference frequencies.
State Space Model (SSM). It has recently demonstrated considerable effectiveness in capturing the dynamics and dependencies of long sequences. Unlike transformers [3, 40] with quadratic complexity, SSMs [23, 59, 60, 14] offer superior performance through linear complexity and show robust generalization across ID audio [24] and 2D image signals [42] when evaluated at varied frequencies. Mamba [22] distinguishes itself by introducing a data-dependent SSM layer, selection mechanism, and hardware-leve performance optimization. It motivates subsequent works in the vision [78, 67, 46), video [34, 45], and point cloud [72, 36] domains. Nikola et al. [80] first integrates SSMs with recurrent VIT framework for event-based object detection to enhance the adaptability for varying sampling frequencies by low-pass band-limiting loss. Subsequent research explored applying SSMs, particularly Mamba [22], to event-based tasks, including action recognition [52, 5], tracking [26, 52, 62], detection [68], Unlike prior work, our work seeks to recognize event streams of broader distribution of event length and generalize across varying inference frequencies.
3 Preliminaries
Event Stream. Event cameras capture object movement by recording the pixel-level log intensity changes, rather than capturing full-frame fixed intervals for conventional cameras. The asyn- chronous events, denoted as = {e,(x; yı. t,, i 1,2, 2, ..... N, reflects the brightness change e; for pixel the timestamp t, with coordinates (It, yı), and polarity ps E {1, -1} [15, 73]. Here, 1 and 1 represent the positive and negative brightness changes.
SSM for Vision. SSMs [23, 59, 14, 60] originate from the principles of continuous systems that map an input ID sequence æx1) E RL into the output sequence y()) E R& through an underlying hidden state ((t)) e N. Specifically, it formalized by dh(t)/dt Ah(t) + Bx(t) and y(t)

---

Event Scan Path
Event Scon Path
1/f
2/1
(P 1)/f P/f
Aggregated Event Frames
1/f
2/1
(P
1)/f PJf
Aggregated Event Fromes
(a) Fixed Time Window
(b) Fixed Event Count
Figure 4: Illustration of event frame aggregation.
Ch(tt Dr(t), where A E RNXN, B E RNx,, C R DE IR the state matrix, the input projection matrix, the output projection matrix, and the feed-forward matrix.
4 Proposed Method
Overview. The PASS framework, depicted in Fig. 3, processes events across a wide distribution of event length using our PEAS module and MSG loss, followed by the spatiotemporal modeling module for prediction. It comprises three components: (1) the PEAS module (Sec.4.1) for event sampling, event frame aggregation, and path-selective event selection. Then bidirectional event sean to encode events into sequence features with fixed dimensions. (2) On top of PEAS, the MSG loss LMSG (Sec.4.2) is proposed for minimizing the randomness and redundancy of encoded features; (3) the event spatiotemporal modeling module (Sec.4.3) to predict the final recognition results.
4.1 Path-selective Event Aggregation and Scan (PEAS) Module
We aim to recognize event streams across a wide distribution of event length. The events are first converted into event presentations, where we select the event frame presentation with fixed event length based on experiment results (see Sec. 5.3). The number of resulting aggregated event frames P can vary greatly due the high temporal resolution of events. This variability introduces complexity for spatiotemporal event modeling. Furthermore, due to SSM's recurrent nature, its hidden state update is greatly affected by the input sequence length and feature order, especially when modeling the long- rangrange temporal dependencies. To reduce this variability, we propose our PEAS module, which consists of the following four components to encode events across wider distribution of event length into sequence features with fixed dimensions in an end-to-end learning manner.
Event Sampling and Frame Aggregation. Unlike sequential language with compact semantics, events & [e:(2; Vi, P:)) E N i = 1,,2, N denotes the asynchronous intensity change at the pixel (I.,, yi) at time t, with polarity p. E (1, -1]. The complexity of spatiotemporal event data requires efficient processing of this high-dimensional data. Following previous methods [75, 80, 2, 54], we sample events with duration 7 every fixed temporal windows 1/f. where denotes the sampling frequency, e.g. 50 ms time windows 1/ʃ corresponding to sampling frequency f 20//z. We group number of events G each sampling time, as shown in Fig. 4 (b). This sampling method is more effective and robust than grouping events within fixed time windows as illustrated in Fig. 4 (a), as evidenced in the following Sec 5.3. Therefore, we obtain P = f event groups E' E PxGx4 Then, we utilize the event frame representation [75] to transform the event groups É' into a series of event frames F Px H xW x3, This transformation enables the use of traditional computer vision methods designed for frame-based data.
Path-selective Event Scan. With the aggregated event frame input F, we then conduct our path- selective event scan to reduce the variability of events. Concretely, as shown in Fig. 3, with the aggregated event frames F E Pxx x3 input, we utilize lightweight score predictor composed of two 3D convolutional layers, followed by an activation function to generate a selection mask M E Rx where K represents the number of selected frames and represents the number of original frames. The elements of M are either 0 or 1, with each marking the position of the selected event frame. Due to the con-differentiable nature of the max operation applied after the standard Softmax function to produce class probabilities, we employ the differentiable Gumbel Softmax [28] for backpropagation during training. The standard Softmax used for inference to facilitate the training process. Next, utilize the Einsum matrix-matie multiplication between the selection mask
5

---

M and the original event frames F to obtain the final selected event frames F' E <RX The above process ensures that F' can be derived from the original event frame input F in an end-to-end learning manner. Next, with the obtained selected event frames F' E I x3, we convert F' into a ID sequence using the bidirectional event scan, following the spatiotemporal scan proposed in [34]. As illustrated in Fig. 3, this scan elegantly follows the temporal and spatial order, sweeping from left to right and cascading from top to bottom. In this way, unlike scanning the original P event frames, our PEAS module can adaptively skip multiple event slices and encode the events across wide event distribution (10 to 109) into encoded features with fixed dimensions.
4.2 Multi-faceted Selection Guiding (MSG) Loss
While the proposed PEAS module is differentiable and capable of learning through back proo--ppppgtton, the basic multi-class cross-entropy loss, LCLS, is inadequate for effectively guiding model optimiza- tion. This is because the selection of event frames is stochastic at the onset of training due to the random weight initialization of the PEAS module. Consequently, during training, the model can only optimize performance based on the distribution of these randomly selected frames, rather than improving the PEAS module for adaptive selection of input events. To facilitate effective optimization, propose the MSG loss that addresses two crucial challenges: 1) reducing the randomness of the selection process to ensure the selected sequence features can encapsulate the entirety of the sequence; and 2) guaranteeing that each selected event feature stands out from the others, thus eliminating redundancy. To be specific, the MSG loss comprises two components, which will be detailed in the subsequent subsections.
Within-Frame Event Information Entropy (WEIE) Loss: We introduce within- event information entropy loss LWEI to reduce the randomness of frame selection, which arises from the random initialization of the PEAS module (Sec. 4.1). LWEIE EIE quantifies the image information entropy of each event frame. As shown in Fig. 3, the WEIE loss for the padding frame Fig. 3 (b) is zero. In contrast, the WEIE loss for the non-padding frame Fig. 3 (a) is greater than zero. Intuitively, higher WEIE loss indicates that the selected event frame contains more information and richer details. Thus, maximizing L EIE helps enhance model optimization to minimize randomness in the selection process. Specifically, we first calculate the frequency histogram pk hist(gray( (F()) for each selected event frame Fg. where K indicates the number of selected event frames, gray(.) converts RGB event frames to grayscale and hist(.) indicates histogram statistics frequency. Then the LWEIE is calculated as follows:
K N
LWEIE = >1
ph log Ph /K
where N is the number histogram bins; K indicates the number of selected event frames.
Inter-frame Event Mutual Information (IEMI) Loss: On top of the WEIE loss to quantify the information entropy for each event frame, we additionally propose the inter-frame event mutual information loss LIEMI to reduce the redundancy among selected event frames. LIEMI quantifies the mutual information [53] between every two consecutive event frames. As shown in Fig. 3, the Lw EIE for Fig. (c) and Fig. (d) are greater than the LWETE E for Fig. (d) and Fig. 3 (e). Intuitively, lower IEMI loss indicates greater differences between the frames. Thus, minimizing IEMI loss guides the model in maximizing the difference between selected event frames. Specifically, Lw EIE is composed of the joint event length histogram hist(.) between every two consecutive event frames F and F along with their spatial coordinates C₂ and Cy to indicates the position information. We compute LW EIE within every consecutive event frame F E K to lower computational cost. The IEMI loss LIEMI is formulated as follows:
joint hist(gray (F() + gray(FF C + +,)),
(2)
K-1
N
N
LIEMI
pk (oine(i,) log(P(()P())PP)))) (t.)))
(3)
K-I
k=1
where N indicates the number of histogram bins and K is the number of selected event frames.

---

Total Objective: Given the final prediction class y and the ground-trute class Y. the total objective composed by the MSG loss Lassa with three components and the commonly used multiclass cross-entropy loss LCuS:
Liotal = LIEMI - SWEIR++&ae +Cr.sw, Y).
LMSG
4.3 Event Spatiotemporal Modeling Module
After the PEAS module followed by the MSG loss, event inputs are transformed into the event frame sequence F' E R Given the inherently longer sequences because of the event stream's 's high temporal resolution, we leverage the SSM for event spatiotemporal modeling with linear complexity. As shown in Fig. 3, we first employ the 3D convolution with kernel size x 16 x 16 for patch embedding to transform the event frames into L non-overlapping spatiotemporal tokens Ie E RE where L = x H x W/16 x 16 and C' refer to feature dimensions. The SSM model, designed for sequential data, is sensitive token positions, making preserving spatiotemporal position information crucial. Thus, we concatenate a learnable classification token X,J, E BIXC at the start of the sequence and then add learnable spatial position embedding P. E [L++))xC and temporal embedding P. E Tx to obtain the final input sequence = [Kels. Tel + + Pt. Next, the input sequence x passes into L layers of stacked B-Mamba blocks. [22]. Note that the bidirectional event scan is actually conducted in the B-Mamba blocks for code implementation. Finally, the [CLS] token extracted from the final layer's output and forwarded to the classification head, which consists of the normalization layer and the linear classification layer for the final prediction y.
Experiments and Evaluation
5.1 Experiments settings
Public Datasets: Five publicly available event datasets are evaluated in this paper. including PAF [41]. SeAct [77], HARDVS [63], N-ImageNet [29] and N-Caltech101 [43].
Our ArDVS100, Real-ArDVS10 and TemArDVS100 Dataset. Existing datasets only provide events within a limited distribution of event length (10% for objects and 10" for actions). We introduce the ArD and TemArDVS across broad distribution of event length (10% to 10"), synthesized by concatenating event streams with varying meta actions, thus capturing action transitions over time. Specifically, ArDVS100 and TemArDVS datasets contain 100 action classes, with event durations of Is to 256s and 14s to 214s respectively. TemArDVS offers fine-grained temporal labels for more accurate action temporal recognition, distinguishing actions like 'sit down then get up' from "get up then sit down,' while the ArDVS100 dataset treats them as the same. We allocated 80% for training and 20% for testing (evaluating). Additionally, to assess the model's ss real-world applicability, we created real-world dataset, named Real-ArDVSig, comprising event-based actions lasting from 2s to 75s, encompassing 10 distinct classes selected from the ArDVS100 datasets. The train and validation (test) split ratio is 7:3.
Table 1: Model structure settings.
Model
Layer
Dim D
Param.
FLOPS(G)
Inference Time(ms)
FPS
Tiny (T)
24
192
7M
1.1
4.1
243.9
Small (S)
24
384
25M
4.3
15.7
63.7
Middle (M)
32
576
74M
12.7
40.4
24.7
Model Architecture & Experimental Settings: In alignment with VIT [13], we modify the depth and embedding dimensions to match models of comparable sizes, including Tiny (T), Small (S), and Middle (M). We adopt the pre- pre-trained VideoMamba [34] model checkpoints for initialization. All ablation studies, unless specifically stated, use the Tiny version on the PAF dataset at a sampling frequency of 0.8 Hz with 16 selected event frames. We reproduced [80] from their official GitHub repository and evaluated on our proposed and event-based recognition datasets for comparative analysis. The detailed model structure settings, parameter estimation, and computational complexity
7

---

Table 2: Comparison with previous methods for event-based object recognition.
Table 3: Comparison with previous methods for event-based action recognition.
Object Recognition (Around It events)
Action Recognition (Around events)
Model
Param.
CaliechT01
Top-1 Accuracy(S5
N. Imagenet
Model
Param.
PAP
Top (ccuracy(3))
SeAre
HARDVS
EST
48,93
EV-ACT [17]
92.60
EDGCN [9]
0.77M
83,50
EventTransAct [7]
57.81
Matrix- [4]
84.31
EvT [54]
0.4SM
61.30
E2VID [50]
10M
86.50
TIPIONT [ST]
92.70
D1S
86.81
Speck [21]
MEM [3]]
90.10
57.89
ESTP [63]
ASA [70]
51.22
47.10
S5-VI--B-KE [80]
17.5M
SS-VIE--KIO [80]
17.5M
92.93
58.21
74.85
S5. B K(2) [80]
17.5M
88.44
SS-VIT-B-K(1)) [80]
17.5M
92.
57.37
Even:Dance [74]
26M
ExACT
471M
94.33
66.07
90.10
PASST-K(U)
7M
88.29
48.74
PASS-I-K(8)
91.38
51.72
98.40
PASS-T-K(()
PASS-S-K(I)
(2()
25M
90.92
53.74
56.10
PASS-S S-K(
25M
96.55
93.33
62.07
60.34
98.41-8.31
201
PASS M. K()
PASS-M-K(2)
74M
94.60
94,20
61.32,3.45
61.12
PASS-M-X((6))
PASS-M-K(N)
98.28-1449
96.55
66.38
98.20
Table 4: Results of event-based action recognition with around 10th events).
Arhitrary-duratio Event Recognition (Around 10" events)
Top-I Accuracy(55)
Model
Param,
ArDVSTO0
Real-ArDVSIO
TemArDVS100
S5-VIT-B 33-VIT-B-K(16) [80]
91.58
90.00
60.26
17.5M
S5- 3S-VIT-B-KKEL ((32) [80]
93.39
93.33
79.62
90.20
80.00
59.20
7M
PASS-T-K(32)
93.85
93.33
89.00
PASS-S K(16)
94,90
90.00
62.90
25M
PASS-S-K(32)
96.00
100,00
73.41
PASS-M-AII
96.00
93.33
71.06
74M
PASS-M-A
97.35
100.00
82.50
are outlined in Tab. Note that model parameters are estimates, changing with category count and selected event frames K.
5.2
Experiments Results
5.2.1 Event-based Recognition Results
Results for recognizing event streams around 10th events. We evaluate PASS on N-Caltech101 and N-Imagenet. 'K' indicates the number of selected event frames. As shown in Tab. 2, our PASS-M- K(2) secures a notable advantage, outperforming EventDance [74] by +2.25% for the N-Caltech dataset and MEM [31] by +3.43% for the N-Imagenet dataset. It achieves superior accuracy (+1.28%) with 2.5x fewer parameters (7M vs 17.5M) than S5- ViT-B-KIE K(2) on the N-Caltech101 datasets, proving the superiority of PASS in effectively recognizing second-level event streams.
Results for recognizing event streams around 107 events. Tab. 3 presents recognition results on three datasets with 1s to 10s event streams. Our PASS-M-K(2) outperforms previous methods, exceed- ing ExAct [77] by +3.45% and +0.38% on the PAF and SeAct datasets, respectively. Additionally, the PASS-S-K(16 achieves remarkable 98.41% Top-1 accuracy HARDVS dataset, surpassing ExAct [77] by +8.31% while using 25M parameters with reduced computational demands.
Results for recognizing event streams around 10 events. In Tab. 4, we evaluate PASS our ArDVS100, TemArDVS100, and Real-ArDVSIO datasets. On the ArDVS100 dataset, our PASS- M-K(32) attains 97.35% accuracy, outperforming [80] by 3.96% and highlighting its potential arbitrarydduration event stream recognition. On the challenging TemArDVS100 dataset, our PASS-T-K(32) achieves 89.00% accuracy, surpassing [80] by 9.38% and demonstrating superior spatiotemporal action transition recognition. PASS-S-K(32) achieved 100% accuracy on the Real- ArDVS10 dataset, showcasing its effectiveness for real-world event-based action recognition.

---

100
100
100
Top-1 Accuracy (96
95
95
4.02%
Top-1 Accuracy (96)
95
90
80
90
85
-10.96%
Top-1 Accuracy
85
20.89%
85
80
80
80
-29.2%
75
75
75
70
Train Time Window
70
Time Window Event Counts
Train
65
65
85
Train
20Hz.
Olurs
Train 100Hz
60
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
Evaluation Frequencies, Hz
(b)
Evaluation Frequencies,
(c)
Evaluation Frequencies, Hz
Figure 5: Model generalization results across varying inference frequencies f training on PAF dataset with sampling frequencies at (a) 20Hz, (b) 60Hz, and (c) 100Hz.
Table 5: Ablation study PEAS module & LMsG-
Table 6: Ablation study on LMSG.
Settings
PAP (A()))
ArDVS100 (K(16)
LMSG
PAF(K(16))
Top](S))
Top](())
Leus
LIEMI
LWEIE
Topl(%)
No Sampling
92.90%
92.31%
92.98%
Random Sampling
92.98%
92.23%
X
93.75%
PEAS
93.33%
92.84%
PEAS LMSG
94.83% %
93.85%
/
/
94.83 9
5.2.2
Inference frequencies Generalization results.
Datasets & Experimental settings We trained PASS-S on the PAF dataset at 20 Hz, 60 Hz, and 100 Hz frequencies and evaluated across 20 Hz 100 Hz to evaluate its inference frequency generalization. Two frame aggregation methods were considered as the baseline, namely fixed 'Time Windows' and fixed 'Event count'. (Refer to Sec. 5.3 for more explanation and discussion.)
Results & Discussion As shown in Fig. 5, regardless of whether the model is trained at low, medium, high frequencies, our models demonstrate consistently strong performance across various inference frequencies, with maximum performance drop of only 8.62% when our PASS model trained at 60 Hz and evaluated at 100 Hz. This finding underscores their robustness and generalizability compared to the baseline methods ('Time Windows' and 'Event count'), ), which experience significant performance declines, such 18,96% -2066%, -29 23% for 'Time Windows' trained 20 Hz, 60 Hz, and 100 Hz and evaluated at 60 Hz, 100 Hz, and 20 Hz, respectively.
5.3 Ablation Study
We perform ablation studies on our PASS framework to evaluate the effectiveness of the PEAS module (Sec. 4.1), LMSG loss (Sec. 4.2).
Impact of PEAS module LMSG loss. As shown in Tab. 5, the baseline 'Random Sampling' randomly selects K event frames and achieves 92.98% and 92.23% accuracy on the PAF and ArDVS100 datasets, respectively. By introducing PEAS, we improve accuracy to 93.33% and 92.84%, representing a performance gain of +0.35% and +0.61% %, demonstrating its ability to preserve critical information. PEAS improves accuracy over baseline 'No Sampling" (+0.43% on PAF, +0.53% on ArDVS100), suggesting that the selected frames retain task-relevan information despite compression. When combining the PEAS module (Sec. 4.1) with LMSG loss, the full model reaches 94.83% 63% and 93.85% accuracy with performance increase of +1.85% and +1.62% PAF and ArDVS datasets, thus showing the effectiveness of LMSG loss to reduce the randomness and redundancy of the encoded features.
Effectiveness of Multi-faceted Selection Guiding Loss LMSG. As presented in Tab. 6, we ablate the three components of LMSG (Eq. 4). As the baseline, the LCLS stands for the standard cross- entropy loss, which achieves a Top-1 accuracy of 92.98%. By employing the LrEMI (Eq. 2), we attain 93.75% accuracy with 0.77% performance gain. The integration of LWEIF (Eq. 1) yields an additional 1.85% increase in accuracy. summary, all proposed components positively impact the final classification, thereby demonstrating their effectiveness.

---

Table 7: Ablation study on event representation.
N-Caltech101 (K(1)
PAF (K
Representation
Top](%)
Top5(%)
Top](%)
1op5(5%)
Frame(Grray) [75]
90.48%
97.53%
93.33%
100.00%
Frame(RGBB) [75]
90.94%
97.82%
94.83%
100.00%
Voxel [11]
90.19%
97.02%
92.47%
100.00%
TBR [27]
90.24%
97.13%
91.72%
100.00%
EST [19]
90.54%
97.66%
93.04%
100.00%
Original Event Frames
PEAS
Selected Event Fromes
Original Event Frames
FEAS
Selected Event Frames
soem
Epoch
Training...
Epoch 100
Figure 6: Visualization of PEAS module with MSG loss.
Event representation. Tab. 7 displays the impact of five commonly used event representations. The RGB frame [75] representation attains Top-1 accuracy rates of 90.94% on the N-Caltech101 dataset and 94.83% on the PAF dataset, surpassing the performance of the other three frame- baaed event representations, including gray frame [75], Voxel [1]] and TBR [27] and one classical learnable event representation EST [19), validating the use of the RGB frame representation in the SSM model, as its pre-training image data has a smaller distribution gap with the RGB event frames.
The visualization demonstration for the PEAS module. Fig. presents the original event frames and the K selected ones at the start and end of the training process. The black parts indicate the padded zero-value frames among a batch. To accommodate varying event lengths and maintain consistent input sizes for batch training, frame padding essential. In Fig. 6, the black parts represent the padded zero-valued frames within mini-batch At epoch O, the PEAS module randomly selects event frames, resulting in unnecessarily padded frames and redundant event frames with repetitive information. After 100 epochs, the eight chosen frames exclude redundant frames and non-informative padding, demonstrating the effectiveness of the PEAS module and the MSG loss.
Conclusion
In this paper, we present novel PASS framework for recognizing events. Extensive experiments prove that our PASS outperforms existing state-of-the-an approaches across five publicly available datasets. Our framework exhibits remarkable performance capabilities, successfully recognizing events across a wide event distribution (106 to 109) as validated through our custom-developed ArDVS100, Real-ArDVS10, and TemArDVS datasets. Moreover, PASS also shows strong general- ization across varying inference frequencies. We hope this method can pave the way for future model design for recognizing events for high-seed dynamic visual scenarios.
Acknowledgements
This work is supported by the Science Foundation of China (NSF) under Grant No. 62206069 (affiliated with Guangzhou HKUST Fok Ying Tung Research Institute) and the MOE AcRF Tier SSHR-TG Incubator Grant FY24 under Grant No. RSTG7/24.
10

---

References
[1] Amon Amir, Brian Taba, David Berg, Timothy Melano, Jeffrey McKinstry, Carmelo Di Nolfo, Tapan Nayak, Alexander Andreopoulos. Guillaume Garreau, Marcela Mendoza, et al. A low power, fully event- based gesture recognition system. In Proceedings of the IEEE conference on compuder vision and pattern recognition, pages 7243-7252, 2017. 1,26
[2] Yin Bi, Aaron Chadha, Alhabib Abbas, Eirina Bourtsoulatze, and Yiannis Andreopoulos. Graph-based spatio-temporal feature learning for neuromorphic vision sensing. IEEE Transactions on Image Processing, 29:9084 90999899-999,,,, 2020. 2, 26
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subhiah, Jared Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, ct al. Language models are fow-shot learners. Advancey in neural information processing systems, 33: 13:1877-1901, 2020. 4
[4] Marco Cannici, Marco Ciccone, Andrea Romanoni, and Mattco Matteucci. A differentiable recurrent surface for asynchronous best- data. Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX 16, pages 136-152. Springer, 2020. 1, 8
[5] Jiaqi Chen, Yan Yang. Shizhuo Deng. Da Teng, and Liyuan Pan. Spikmamba: When snn meets mamba in event-based human action recognition. In Proceedings of the 6th ACM International Conference on Multimedia in Asia, pages 1-8, 2024. 4
[6] Kanghao Chen, Hangyu Li, Jiazhou Zhou, Zeyu Wang, and Lin Wang. Lase-e2v: Towards language- guided semantic-aware event-to-video reconstruction. Advances in Neural Information Processing Systems, 37:70406- 70430, 2024,
[7] Tristan de Blegiers, Ishan Rajendrakumar Dave, Adeel Yousaf, and Mubarak Shah. Eventtransact: A video transformer-baat based framework for event-camera based action recognition. In 2023 IEEE/RSJ International Conference on Intelligent Roboty and Systems (IROS), pages 1-7. IEEE, 2023. 1,8
[8] Pierre De Touremiire, Davide Nitti, Etienne Perot, Davide Migliore, and Amos Sironi. A large scale event-based detection dataset for automotive. arXiv preprint arXiv:2001.08440 2020.
[9] Yongjian Deng, Hao Chen, and Youfu Li. A dynamic gen with rooseeepresnnnttionn distillation for event- based learning. In Proceedingy of the AAAI Conference on Artificial Intelligence. volume 38, pages 1492-1500, 2024. 8,8
[10] Yongjian Deng, Hao Chen, Hai Liu, and Youfu Li. A Voxel Graph CNN for Object Classification With Event Cameras. CVPR, 2022.
[11] Yongjian Deng, Hao Chen, Hai Liu, and Youfu Li. voxel graph cnn for object classification with event cameras. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 1172-1181, 2022. 10
[12] Yiting Dong, Yang Li, Dongeheng Zhao, Guobin Shen, and Yi Zeng. 100k large-scale neuromor- phic dataset towards fiivacy-preserving bullying recognition. Advances in Neural Information Processing Systems, 16:1992-1 2023.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner. Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition scale. arXiv preprint arXiv:2010. /9929, 2020. 7
[14] Daniel Y Fu, Tri Dao, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. arXiv preprint 22Xiv:2212.140522 2022. 23
[15] Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, Jörg Conradt, Kostas Danilidis, et al. Event-based vision: A survey, IEEE transactions on pattern analysis and machine intelligence, 44(1):154- 2020. 3.4.4
[16] Yue Gao, Jiaxuan Lu, Siqi Li, Yipeng Li, and Shaoyi Du. Hypergraph- based multi-view action recognition using event cameras. IEEE Transactions on Pattern Analysis Machine Intelligence, 2024, 1, 2
[17] Yue Gao, Jiaxuan Lu, Siqi Li, Nan Ma. Shaoyi Du, Yipeng Li, and Qionghai Dai. Action recognition and benchmark using event cameras, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 1, 8, 26

---

[18] Daniel Gehrig, Antonio Loquercio, Konstantinos G Derpanis, and Davide Scaramuzza. End-to-end learning of representations for asynchronous event-based data. In ICCV, 2019.
[19] Daniel Gehrig, Antonio Loquercio, Konstantinos G Derpanis, and Davide Scaramuzza. End-to-end learning of representations for asynchronous event-based data. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 5633-5643, 2019. 8, 10
[20] Daniel Gehrig and Davide Scaramuzza. Low-lateney automotive vision with event cameras. Nature, 629(8014): 10:4-00040000 2024.
[21] Mathias Gchrig, Willem Aarents, Daniel Gehrig, and Davide Scaramuzza. Dsec: sterco event camera dataset driving scenarios. IEEE Robotics and Automation Letters, 6(3):4947 4954, 2021.
[22] Albert and Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 2023. 4, 7, 23,
[23] Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. Advances in Neural Information Processing Systems, 15:35971-35998 2022. 4, 23
[24] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces, arXiv preprint arXiv:2111.00396, 2021. 2,
[25] Fuqiang Gu, Weicong Sng, Tasbolat Taunyazov, and Harold Soh. Tactilesgnet: A spiking graph neural network for event-based tactile object recognition. In IROS, 2020. 3
[26] Ju Huang. Shiao Wang, Shuai Wang. Zhe Wu, Xiao Wang, and Bo Jiang. Mamba-fetrack: Frame-event tracking via state space model. In Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pages 3-18. Springer, 2024.
[27] Simone Undri Innocenti, Federico Becattini, Federico Pernici, and Alberto Del Bimbo. Temporal binary representation for event-based action recognition. In 2020 25th International Conference on Pattrn Recognition (ICPR), pages 10426- 10432, IEEE, 2021. 10
[28] Eric Jang. Shixiang Gu and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144 2016. 5
[29] Junho Kim, Jachyeok Bae, Gangin Park, Dongsu Zhang, and Young Min Kim. N-imagenet: Towards robust, fine-grained object recognition with event cameras. Proceedings of the IEEE/CVF international conference on computer vision, pages 2146-2155 2021. 3, 7, 8, 26, 27
[30] Junho Kim, Inwoo Hwang, and Young Min Kim. Ev-tta: Test-time adaptation for event-based object recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17745-17754, 2022. 2.4
[31] Simon Klenk, David Bonello, Lukas Koestler, Nikita Araslanov. and Daniel Cremers. Masked event modeling: Self-supervised pretraining for event cameras. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2378-2388, 2024. 1,8
[32] Xavier Lagorce, Garrick Orchard, Francesco Galluppi, Bertram E Shi, and Ryad B Benosman. Hots: a hierarchy of event-based time-surfaces for pattern recognition. IEFE transactions pattern analysis and machine intelligence, 1346 1346-139,, 2016. 2
[33] Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. Cifar10-dvs: an event-stream dataset for object classification. Frontiers in neuroxcience, 11:309, 2017. 26
[34] Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang. Limin Wang, and Yu Qiao. Videomamba: State space model for efficient video understanding. In European Conference on Computer Vision, pages 237-255. Springer, 2025. 4, 6, 7,27
[35] Yijin Li, Han Zhou, Bangbang Yang, Ye Zhang, Zhaopeng Cui, Hujun Bao, and Guofeng Zhang, Graph- based asynchronous event processing for rapid object recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 934 943, 2021. 1,
[36] Dingkang Liang, Xin Zhou, Wei Xu, Xingkui Zhu, Zhikang Zou, Xiaoqing Ye, Xiao Tan, and Xiang Bai. Pointmamba: A simple state space model for point cloud analysis. arXiv preprint arXiv:2402.1077 10739, 2024.
12

---

[37] Yihan Lin, Wei Ding, Shaohua Qiang, Lei Deng, and Guoqi Li. Es-imagenet: A million event-stream classification dataset for spiking neural networks. Frontiers in neuroscience, 15:726582, 2021. 26
[38] Chang Liu, Xiaojuan Qi, Edmund Y Lam, and Ngai Wong. Fast classification and action recognition with event-hased imaging. IEEE Access, 10:55638-55649, 2022.
[39] Qianhui Liu, Dong Xing, Huajin Tang, De Ma, and Gang Pan. Event-based action recognition using motion information and spiking neural networks. In IJCAI, pages 1743-1749, 2021. 3, 26
[40] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lec. Vilbert: Pretraining task agnnosici visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems, 32, 2019. 4
[41] Shu Miao, Guang Chen, Xiangyu Ning, Yang 7i, Kejia Ren, Zhenshan Bing, and Alois Knoll. Neu- romorphic vision datasets for pedestrian detection, action recognition, and fall detection. Frontiers in neurorobotics, 13:38, 2019. 7, 26, 27
[42] Eric Nguyen, Karan Goel, Albert Gu, Gordon Downs, Preey Shah, Tri Dao, Stephen Baccus, and Christo- pher Ré. S4nd: Modeling images and videos as multidimensional signals with state spaces. Advances in neural information processing systems, 35:2846-2861, 2022. 2,
[43] Garrick Orchard, Ajinkya Jayawant, Gregory Cohen, and Nitish Thakor. Converting static image datascts to spiking neuromorphic datasets using saccades. Frontiers in neuroscience, 9:437, 2015. 26, 27
[44] Garrick Orchard, Cedric Meyer, Ralph Etienne-Cummings. Christoph Posch, Nitish Thakor, and Ryad Benosman. Hfirst: A temporal approach to object recognition. IEEE transactions on pattern analysis and machine intelligence, 37(10):2088-2040 2015. 2
[45] Jinyoung Park, Hee-Seon Kim, Kangwook Ko, Minbeom Kim, and Changick Kim. Videomamba: Spatio- temporal selective state space model. In European Conference on Computer Vision, pages 1-18. Springer, 2025.
[46] Badri N Patro and Vijay S Agneeswaran. Simba: Simplified mamba- based architecture for vision and multivariate time series. arXiv preprint arXiv:2403.1536, 2024. 4
[47] Etienne Perot, Pierre De Tournmiire Davide Nitti, Jonathan Masci, and Amos Sironi. Learning to detect objects with I megapixel event camera. Advances in Neural Information Processing Systems, 11166669666522 2020. 2
[48] Chiara Plizzari, Mirco Planamente, Gabriele Goletto, Mareo Cannici, Emanuele Gusso, Matteo Matteucci, and Barbara Caputo. E2 (go) motion: Motion augmented event stream for egocentric action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19935-15 -9947, 2022. 1,3
[49] Christoph Posch, Daniel Matolin, and Rainer Wohlgenannt. A quga 143 db dynamic range frame-free pwm image sensor with lossless pixel-level vidco compression and time-domain eds. IEEE Journal of Solid-State Circuits, 46 46(1):259 275, 2010. 27
[50] Henri Rebecq, René Ranftl, Vladlen Koltun, and Davide Scaramuzza. Events-to- video: Bringing modern computer vision to event cameras. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3857-3866, 2019. 8
[51] Hongwei Ren, Yue Zhou, Haotian Fu, Yulong Huang, Renjing Xu, and Bojun Cheng. Tipoint: A tensorized point cloud network for lightweight action recognition with event cameras. Proceedings of the 31st ACM International Conference on Multimedia, pages 8026 8034,, 2023. 8
[52] Hongwei Ren, Yue Zhou, Jiadong Zhu, Haotian Fu, Yulong Huang, Xiaopeng Lin, Yuetong Fang, Fei Ma, Hao Yu, and Bojun Cheng. Rethinking efficient and effective point based networks for event camera classification and regression: Fventmamba. arXiv preprint 2405 16, 2024. 4
[53] Daniel B Russakoff, Carlo Tomasi, Torsten Rohlfing, and Calvin R Maurer. Image similarity using mutual information of regions. In Computer Vision- ECCV 2004: 8th European Conference on Computer Vision, Prague, Czech Republic, May 11-14, 2004. Proceedings, Part III 8, pages 596 607. Springer, 2004. 6
[54] Alberto Sabater. Luis Montesano, and Ana C Murillo. Event transformer. a sparse-aware solution for efficient event data processing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2677-2686, 2022. 1, 2,
13

---

[55] Alberto Sabater. Luis Montesano, and Ana C Murillo. Event transformer+. multi pulti-purpose solution for efficient event data processing. IEEE Transactions on Puttern Analyxis and Machine Intelligence, 2023. 1
[56] Simon Schaefer, Daniel Gehrig, and Davide Scaramuzza. Aegnn: Asynchronous event- brent-based graph neural networks. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 12371-12381, 2022.
[57] Teresa Berrano-Gotarreooig and Bernabé Linares- Barares-Barranco. Poker-dvs and mnist-dvs, their history, how they were made, and other details. Frontiers in neuroscience, 9:481, 2015. 26
[58] Amos Sironi, Manuele Brambilla, Nicolas Bourdis, Xavier Lagorce, and Ryad Benosman. Hats: Histograms of averaged time surfaces for robust event-based object classification. In Proceedings of the IEEF conference computer vision and pattern recognition, pages 1731-1740, 2018. 2, 26
[59] Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. Simplified state space layers for sequence modeling, arXiv preprint arXiv:2208.04993 04933, 2022. 4, 23
[60] Jue Wang, Wentao Zhu, Pichao Wang, Xiang Yu, Linda Liu, Mohamed Omar, and Raffay Hamid. Selective structured state-spaces for long-form video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vixion and Puttern Recognition. pages 6387-6397, 2023. 4, 23
[61] Qi Wang, Zhou Xu, Yuming Lin, Jingtao Ye, Hongsheng Li, Guangming Zhu, Syed Afaq Ali Shah, Mohammed Bennamoun, and Liang Zhang. Dailydvs-200 A comprehensive benchmark dataset for event-hased action recognition. arXiv preprint arXiv:2407.05
2024. 26, 27, 29
[62] Xiao Wang, Shiao Wang, Xixi Wang, Zhicheng Zhao, Lin Zhu, Bo Jiang, et al. Mambaevt: Event stream based visual object tracking using state space model. arXiv preprint arXiv:2408. 10487, 2024. 4
[63] Xiao Wang, Zongzhen Wu, Bo Jiang. Zhimin Bao, Lin Zhu, Guoqi Li, Yaowei Wang, and Yonghong Tian. Hardvs: Revisiting human activity recognition with dynamic vision sensors. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 5615-5623, 2024. 8, 26, 27, 28, 29
[64] Bochen Xie, Yongjian Deng, Zhanpeng Shao, Hai Liu, and Youfu Li. Vmv-gen: Volumetric multi-view based graph can for event stream classification. IEEE Robotics and Automation Letters, 1()):1976-1983, 2022. 3
[65] Bochen Xie, Yongjian Deng, Zhanpeng Shao, Hai Liu, Qingsong Xu, and Youfu Li. Event voxel set transformer for spatiotemporal representation learning on event streams, arXiv preprint 2033:22333. 2023. 3
[66] Bochen Xie, Yongjian Deng, Zhanpeng Shao, Qingsong Xu, and Youfu Li. Event voxel set transformer for spatiotemporal representation learning on event streams. IEEE Transactions on Circuits and Systems for Video Technology, 2024. 2, 3,
[67] Rui Xu, Shu Yang, Yihui Wang, Bo Du and Hao Chen. A survey vision mamba: Models, applications and challenges. arXiv preprint arXiv:2404. 18861, 2024. 4
[68] Nan Yang, Yang Wang. Zhanwen Liu, Meng Li, Yisheng An, and Xiangmo Zhao. Smamba: Sparse mamba for event-based object detection. arXiv preprint arXiv:2501. /1911, 2025. 4
[69] Man Yao, Huanhuan Gao, Guangshe Zhao, Dingheng Wang, Yihan Lin, Zhaoxu Yang, and Guogi Li. Temporal-wise attention spiking neural networks for event streams classification. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 10221-10230, 2021. 2,4
[70] Man Yao, Jiakui Hu, Guangshe Zhao, Yaoyuan Wang, Ziyang Zhang, Bo Xu, and Guoqi Li. Inherent redundancy in spiking neural networks. In Proceedings of the IEFF/CVF international conference on computer vision, pages 16924-16934, 2023. 8
[71] Man Yao, Ole Richter, Guangshe Zhao, Ning Qiao, Yannan Xing, Dingheng Wang, Tianxiang Hu, Wei Fang, Tugba Demirei, Michele De Marchi, et al. Spike-based dynamic computing with asynchronous sensing computing neuromorphic chip. Nature Communications, 15(1):4464, 2024. 8
[72] Tao Zhang, Xiangtai Li, Haobo Yuan, Shunping Ji, and Shuicheng Yan. Point could mamba: Point cloud learning via state space model. arXiv preprint arXiv:2403.0076. 2024. 4
[73] Xu Zheng, Yexin Liu, Yunfan Lu, Tongyan Hua, Tianbo Pan, Weiming Zhang, Dacheng Tao, and Lin Wang. Deep learning for event-based vision: A comprehensive survey and benchmarks. arXiv preprint arXiv:2302. 08890, 2023, 3,
14

---

[74] Xu Zheng and Lin Wang. Eventdance: Unsupervised source-free cross-modal adaptation for event- based object recognition. Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17448-17458, 2024. 8,4,8
[75] Jiazhou Zhou, Xu Zheng, Yuanhuiyi Lyu, and Lin Wang. E-clip: Towards label-efficient event-based open open-workd world understanding by clip. arXiv preprint arXiv:2308.03135, 2023. 5,
[76] Jiazhou Zhou, Xu Zheng, Yuanhuiyi Lyu, and Lin Wang. Eventbind: Learning unified representation bind them all for event-based open-worke world understanding. In European Conference on Computer Vision, pages 4944.. Springer, 2024.
[77] Jiazhou Zhou, Xu Zheng, Yuanhwiyi Lyu, and Lin Wang. Exact: Language-guided conceptual reasoning and uncertainty estimation for event-based action recognition and more. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18633-18643, 2024. 2, 3, 4, 7, 8, 26, 27
[78] Lianghui Zhu, Bencheng Liao, Qian Zhang. Xinlong Wang. Weayu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint MoXiv:2401.0944 2024, 4, 25
[79] Nikola Zubić, Daniel Gehrig, Mathias Gehrig, and Davide Scaramuzza. From chaos comes order: Ordering event representations for object recognition and detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12846 12856, 2023. 2,
[80] Nikola Zubic, Mathias Gchrig, and Davide Scaramuzza. State space models for event cameras. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5819-5828, 2024. 4, 5, 7, 8, 23, 25
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. The claims are supported by our experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper examines the limitations of the authors' research in sec. 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
16

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes].
Justification: The paper provides complete details to replicate its key experimental findings in sec. 5.1.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [No]
Justification:
We don't provide open access to data and code in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nipp cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Codeeuomiisseeeeeeee
/nips. wwtpccccpppcccaeeccccccccccccclt / ssssssooiccyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
The paper comprehensively details training and testing parameters, including data splits, hyperparameters, optimizer selection in sec. 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No].
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes].
Justification:
The paper details the computational resources the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https / //turreuripp..e/ httppppllli.eittcaieeeeeeeeeeeee
Answer:[Yes]
Justification:
The research fully complies with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]..
Justification: The paper critically examines the potential positive and negative societal implications of the research.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification: Our research poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We explicitly mention and respect the creators and original owners.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification:
The paper introduces no new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not research human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our paper does not research human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
21

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [No]
Justification: The LLM serves solely writing, editing, formatting tool without affecting the core methodooogy.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https http//rrps..ccooofeeeeeeeeeee wct.///ooooeeecee220555555555550 for what should or should be described.
22

---

Appendix
In this appendix, we provide more details about model implementation, experimental settings, and datasets to complement the main paper. Additional analysis and discussions also incorporated. Below is the table of contents:
Model A
Technical Details of SSMs A.1
PyTorch-style Pseudocode for PEAS Module A.2
Experiments
Experiment Settings B.1
Event Frame Sampling Settings B.2
Reproduction Setrings for [80] B.3
More Experiment Results B.4
Dataset Details
ArDVS100 Dataset C.I
Real-ArDVS10 Dataset C.2
TemArDVS100 Dataset C.3
Dataset Comparision C.4
Publicly Available Dataset C.5
Discussion
A Model
A.1
Technical Details of SSMs
State Space Models (SSMs) [23, 59, 14, 60] originate from the principles of continuous systems that map input ID sequence ætt) E R into the output sequence y(t) E R' through an underlying hidden state htt)= E R. Specifically, it is formalized by dh(t)/d dt Ah(t) Bx(t) and y()
C'h(t) + (t( where A E R.NN B E Rx1,, C E RN D Nx1 are the state matrix, the input projection matrix, the output projection matrix, and the feed-forwart matrix.
dh(t)/dt = Ah(t) + Bx((),
(5)
v(t) Ch(t) + Dx(().
where E NNN, B E R.RI C.RIIICEE CE CERI, D IR NxI are the state (or system) matrix, the input projection matrix, the output projection matrix and the ferw-forward matrix.
The discretization process of SSMs is essential for integrating continuous-time models into deep- learning algorithms. [60]. We adopt Mamba [22] strategy, treating D as fixed network parameters while introducing timescale parameter to transform the continuous parameters A, B into their discrete counterparts Á, ß, formulated as follows:
Å -= eTp((AA))
(7)
B = (AA) (erp(AA) - -) AB
(8)
ht Ål--1 + Bai,
(9)
21 = CCb.
(10)
Compared to previous linear time-invariant SSMs, Mamba proposed selective sean mechanism that directly derived the parameters B, C, and A from the input during the training process, thus enabling better contextual sensitivity and adaptive weight modulation.
23

---

Algorithm PyTorch-style Pseudocode for the Proposed PEAS Module
B, C, H. W: Batch size, Channel, Width, Height
P. K: Amount of input and output event frames
Input event frames with shape (B, P, C, H. W) yy: Output selected frames with shape (B, K, W)
ScorePredictor(z) # Two-layer CNN network
Predict scores for each event frame (B, K, P)
self.training Differentiable selection during training
selection mask F.gumbelssottmaxe softmax(predd_eee score, dim=2, hard=True)
else: Hard selection during evaluation
idx_att idx_argmax s. mdmmm2, keepdim=True)[I]
selection Dalection_mask>oo torch. zeros sicttes)ssaateerr(ii (dim-e2 index=idx_argman argmax, value=1.0)
B, K. P selection brtpecttionmmssshhhppp
indices torch. wheresselection mask.eq(1))
Sort from largest to smallest corresponding to the time sequence
indices cooossssseedd_eee = wevehargsootindiies] reesaiiees/ reshape K), dim=l) Rearrange mask based on temporal sequence
For in range(B):
selection_ mask[i,
selection mask[[i
:]
Perform frame selection using the mask
= torch.einsumf 'kp. bethw" + "bepkhw', selection_mann x))
Sum over time dimension
= y.sum(dim=3) # (B.CKK.H.W)
A.2 PyTorch-style Pseudocode for PEAS Module
In Algorithm 1, we present the PyTorch-style pseudocode of the proposed PEAS module facilitate
readers' understanding.
Mask Selection (MS) Loss: Due to the arbitrary length of event streams with different numbers of
input event frames, frame padding is necessary to maintain consistent input sizes to ensure
Ori
Pad
Corresponding Position
M
0
1
0
0
1
0
0
1
(ms(Eqq44) 4)
Figure Illustration of components for the proposed MS loss.
training among mini-batch. Therefore, we propose an MS loss LMS filter the padded frames during selection. Specifically, as shown in Fig. 7, given the original event frame input F e x and the selection mask M K the LMs loss sums the mask value M, = Ori 1..., Ori Pad at the corresponding position of the padding frame in Fj.j = Ori+1 Ori+ Pad,
which is formulated as follows:
K
Pad
LMS
Σ
Σ
Mr,/(K x Pad),
(11)
Ori = P. and Pad indicate the number of selected event frames, original event frames, and padding frames, respectively.

---

B Experiments
B.1 Experiment Settings
We utilize the default hyperparameters for the B-Mamba layer [78], setting the state dimension to 16 and the expansion ratio to 2. Additionally, we adjust the stochastic depth ratio to O, 0.15, and 0.5 for the Tiny, Small, and Middle versions, respectively. We utilize the AdamW optimizer with cosine learning rate schedule with the initial 5 epochs for linear warm-up. Unless a special statement is made, the default settings for the learning rate and weight decay are 1e-3 and 0.05, respectively. The model is trained with 100 epochs for PAF, SeAct, and N N-Caltech101 datasets and 50 epochs for HARDVS, N-Imagenet, ArDVS100, TemArDVS100, and Real-ArDVS10 datasets. We employ BFloat 66 precision during training to improve stability. For data augmentation, we implement random scaling, random cropping, random flipping, and data mixup of the event frames during the training phase.
B.2 Event Frame Sampling Settings
The additional experiment settings of sampling frequency and aggregated event count per frame for different datasets are presented in Tab. 8.
Table 8: The sampling frequency and aggregated event count per frame for different datasets
Dataset
Sampling Frequency
Aggregated Event Count / Frame
N-Caltech101
200 Hz
50,000
N-Imagenet
50 Hz
2,000,000
PAF
80 Hz
100,000
SeAct
80 Hz
80,000
HARDVS
100 Hz
80,000
ArDVS100
50 Hz
80,000
Real-ArDVS10
50 Hz
80,000
TemArDVS100
50 Hz
80,000
B.3 Reproduction Settings for [80]
We reproduced [80] from their official GitHub repository and evaluated it on our proposed and event-based recognition datasets for comparative analysis. The direct comparison is not feasible due to fundamental differences: 1) task (object detection vs. object recognition). 2) network structure (detection vs. classification head), 3) SSM backbone (S4D, S5 vs. Mamba), 4) evaluation datasets (detection vs. recognition), and 5) evaluation metrics (mAP vs. accuracy).
For the above reasons, to ensure fair comparison, we replaced our Mamba backbone with [80]'s S5- S5-ViT-B model and substituted its YOLOX detection head with our classification head based on its GitHub repository PyTorch implementation. We did not the S4D backbone due to its lower performance compared to the S5 model, as reported by [80]. We adopted [80]'s event voxel representation, creating event voxels based on 50 ms time windows corresponding 20 Hz sampling frequency, divided into T= 10 discrete bins. Following [80]. we applied data augmentation techniques such random horizontal flips and zooming. The training was performed on the PAF and SeAct datasets for 100 epochs and on HARDVS ArDVS100, TemArDVS100, and Real-ArDVS datasets for 50 epochs. We also integrated the PEAS module, with K indicating the number of selected event frames.
B.4 More Experiment Results
Frame Aggregation Method: Time Windows vs. Event count. We illustrate two event aggregation methods in Fig. 4, where 'Event count' aggregation leads to varying aggregation temporal ranges and 'Time Windows' keeps them consistent. As shown in Fig. 5 (b), 'Event count' performs better compared to "Time Windows'. For example, 'Event count' achieves 96.55 % accuracy compared to 94.83% % for "Time Windows' when both trained and evaluated at 60 Hz. However, 'Event count' and 'Time Windows' experience -1.66 % and -18997% performance drops respectively, when evaluating
25

---

Table 9: Comparison of existing datasets with our ArDVS100 dataset.
Temporal Fine- line-grained
Dataset
Year
Sensors
Object
Scale
Class
Real
Duration(s)
Labels
MNISTDVS [43]
2013
DAV1S128
Image
30,000
10
X
N-Caltech101 [43]
2015
ATIS
Image
8,709
101
X
X
0.3s
N-MNIST [57]
2015
ATIS
Image
70,000
X
X
0.3s
CIFAR IO-VS [33]
2017
DAVIS
Image
10,000
10
X
X
1.2s
N-ImageNet [29]
2021
Samsung-
Image
1,781,167
1,000
X
X
0.1s
ES-InageNet [37]
2021
Image
1,306,916
1,000
X
DvsGesture [1]
2017
DAVIS128
Action
1,342
11
X
6s
N-CARS [58]
2018
ATIS
Car
24,029
2
X
0.1s
ASLAN-DVS [2]
2019
DAVIS240
Hand
100,800
24
X
X
0.1s
PAF [F]4
2019
DAVIS346
Action
450
10
5y
HMDB-DVS [2]
2019
DAVIS240c
Action
6,766
51
19s
UCF-DVS [2]
2019
DAVIS240c
Action
13.320
101
25s
DailyAction [39]
2021
DAVIS346
Action
1,440
12
58
HARDVS [63]
2022
DAVIS346
Action
107,646
300
58
THUEACTSO [17]
2023
CeleX-V
Action
10,500
50
2s-58
THUEACSOCHL [17)
2023
DAVIS346
Action
2,330
50
25-65
Bullying10K [12]
2023
DAVIS346
Action
10,000
1s-20s
SeAct [77]
2024
DAVIS346
Action
580
58
2x-
DailyDVS-200 [61]
2024
DVXplorer Lite
Action
22,046
2s-20s
ArDVS100
2024
DAVIS346
Action
8,000
100
1s-263s
Real-ArDVS10
2024
DAVIS346
Action
100
2s-75x
TemArDVS100
2024
DAVIS346
Action
8,000
100
14y-214s
Table 10: Model generalization results across different inference frequencies f on PAF dataset.
Top-I Accuracy & Perlormance Drup
Train f
Settings
Val f
20 Hz
40 Hz
60 Hz
80 Hz
100 Hz
Time Windows
93.10
89.65.34 3.4
74. 14.1
72.41 2049
68.97.3413
20
Event Counts
94.83
87.93
75. 15.56
75.86
70.69.3u14
Event Counts PAST-SSM-S
93.10
89.65.3.s
89.65.34
86.214x9
84.48
Time Windows
19.3
87.93
94.83
89.65
75.86
60 Hz
Event Counts
81.03asst
89.65
96.55
87.93
79.89.uu
Rvent Counts PAST-SSM-S
89.66.as
93.1 Laas
96.55
91.38-<17
87.93
Time Windows
87.93.34
91.37.
91.37a
91.37
100 Hz
Event Counts
67.24-27.
86.2 1.az
89.65.x1
98.1-am
94.83
Event Counts PAST-SSM-S
89.66.x1
94,83
13.1
93.1-1m
94.83
at 100 Hz. This result leads us to propose the PEAS module to improve model generalization across inference frequencies.
The statisties result for model generalization across varying inference frequencies. In Tab. 10, we present the specific statistics result for Fig.7 in the main paper for future comparison.
C Dataset Details
C.1 ArDVS100 Dataset
ArDVS100 contains 100 different action series with varying durations synthesized by concatenating the randomly selected event streams from the HARDVS [63] dataset. The ArDVS100 contains 8000 event stream, which durations range from 1.46x to 263. 36,.66s, with mean of 45.62s. To maintain brevity, Tab. 11 details only the selected 10 action series. We can observe that different classes feature distinct action series with varying meta-action counts, thus resulting in differing durations.
26

---

C.2 Real-ArDVS10 Dataset
The Real-ArDVS10 dataset was recorded using the DVS346 event camera, which has a resolution of 346 x 240 pixels. The Aral-A dataset includes ten action series randomly selected from the ArDVS100 dataset. During recording, participants stood before an event camera and performed meta-actions sequentially as instructed. Ten individuals (8 male, 2 female) contributed to the dataset, with detailed meta-action descriptions provided in Tab. 12.
C.3 TemArDVS100 Dataset
ArDVS100 contains 100 different action series with varying durations synthesized by concatenating the randomly selected event streams from both HARDVS [63] and DailyDVS-200 [61] datasets. The ArDVS 100 is made up of 8000 event streams, whose durations range from 14.53s to 213.54s, with a mean of 93.87s. For presentation simplicity, we just illustrate the selected 8 action series with detailed action descriptions in Tab. 13. Classes 1 to 4 and 97 to 100 share the same four meta-actions but form distinct action series through varying meta-action combinations, allowing the TemArDVS 100 dataset to provide fine-grained temporal labels for more precise action recognition.
C.4 Dataset Comparision
We compare our proposed ArD, Real-ArDVSIO, and TemArDVS100 datasets with existing event-based based recognition datasets. As shown in Tab. 9, previous datasets contain second- level event streams lasting from 0.1s 20s, while our proposed Real-ArDVS10 and TemArDVS 100 datasets provide minute-level duration event streams lasting from Is to 265s, 2s to 75s and 14s to 215s, respectively. We believe these proposed benchmarks will provide enhanced evaluation platforms for recognizing event streams of arbitrary durations and inspire further research in this field.
C.5 Publicly Available Dataset
Five publicly available event-based datasets are evaluated in this paper follows: 1) PAF [41], also known as DVS Action, is an indoor dataset featuring 450 recordings across ten action categories lasting around 5s. 2) SeAct [77]
newly released dataset for event-based action recognition, covering 58 actions within four themes lasting around 2s-10s. This work uses only class-level labels despite available caption- level labels. 3) HARDVS [63] is currently the largest dataset for best- action recognition, comprising (07,646 recordings of 300 action categories. It also has an average duration of 5s and a resolution of 346 260. 4) N-ImageNet [29] is derived from the ImageNet- dataset, where the RGB images are displayed on monitor and captured by a moving event camera. includes 1, 1,781, 1,67, event streams with 480 x 640 resolution across 1,000 unique object classes. 5) N-Caltech101 [43] contains event streams captured by event camera in front of mobile 180 x 240 ATIS system [49] with the LCD monitor presenting the original RGB images in Caltech101. There are 8,246 samples comprising 300 ms in length, covering 101 different types of items.
D Discussion
Model Limitation. We observe that larger VideoMamba tends to overfit during our experiments, resulting in suboptimal performance. This issue is not limited our models but is also observed in VMamba [22] and VideoMamba [34]. Future research could explore training strategies such as self-distillation and advanced data augmentation to mitigate this overfitting.
Broader Impact. Recognition is critical vision task with widespread applications like robot naviga- tion. Traditional
methods can degrade due motion blur and lighting variations. Event cameras exclusively capture moving objects, providing resilience to rapid motion and illumination changes while consuming minimal power. This method may be useful for high-level recognition tasks. As data-driven approach, the method's performance is sensitive to data biases. Careful attention to the data collection process is essential to ensure reliable and accurate results.
27

---

back Table 11: Meta-action descriptions for 10 selected action series classes in the ArDVS dataset. ArDVS100 includes 100 action series of varying durations, created by randomly concatenating event streams from HARDVS [63] to capture temporal action transitions.
Clsss Indox
Descriptin
Class Index
Descriptio
Class
Acliont50 Place
Class
Actionl:88 Bow Strighh Ann Rowing
Action
Upper
Swing Amns
Astion273.
Wrap
Action014- Alternate Front Kick
Action215. Skew Head Biye
Action 185- clench list and running
Ailion11-- Lefl illopsoss muscle stretching
Action195- Touch the back the head
Action185 clench
fist sturt running
buckward sliding steps
Astion(113 Single Leg Jump
Action200 Standing Long lump
Astion 19- Touch the neck and the heal
2c Hip Up Kick Jump
Action199. Tanch the forehand
Action 199. Touch the forehend
Action 20. Bore Hand Hard Pull Boar Action
Action Chest
Actionf226 Bow Side Flat
Action01 Body
rotation
Action032- Eaveshhopping action
Class 50
Action024- Side Lift Swivel Arm
gyonssties kicking exercises
Class
Action186- Rubbing Hands for Heating
AstionlA6. stapler
Action097- Left Back Stretch
Action237- Hard Pull Swing
Aclion218- Standing Twist
Action213- Standing Right Rear Lift
Right
Action12 Wandering
Action274- Chest
Astion236- Fye Exercise
Action266- Anow Squat Knee Lift
Action1933 Leg hending side sitting
Action257. Simplifted Thi Chi Tower Knee Depcession Step
Action1223 Side Lift
Aclion208- Knock Calculator
Actionl82 Shout
Action2i44
Action
Step
Action183. Bining Lips tue
Action1 Bare Hand Squat
Action022 Side Lunge Squat true
Action069- Right Lunge Twist Stretch
Actiont61 Right swag Itue
Aclion024- Side Lift Swivel Arm
Action214- Stanling Left Ley true
Actionl15- Opening closing steps
Active052. Cross waist punch
Astion|A6. stapler mice
Aclion122: Body Rolaton Moveneet
Action169- Wave true
A3tion133- Stup Fingers
Activat99- Left Front Thigh Stroich true
Actioe050- Step Place
ActionD18. Alteriating
Strike
Class 100
Action119. Bore Hand Squa
Action24-- Standing Jump Transformaiinn time
Holding
Action237-
Swing
Class
Action211 Standing Lefi Leg Lift
Action207: Salute
Action 11- Amm Prese Down true
Actionll1: Left iliopsoas musele stretching true
Action218- Kneck Calculator
Action253- Simplineed Tai Chi Twin Peaks Through Ears troe
Action199. Touch waist
Action147- Scratch
and cheeks
Action
Aclion067- Right thigh front streich
Action289. Cover the Eyes and the Legs true
Action008- 9th set of broudcast gymmesiics side movements
Action 106- Left hand circle true
Table 12: Meta-action descriptions for action series classes in the Real-ArDVS10 dataset. This dataset, captured by camera, features real human action transitions for randomly selected action series from the ArDVS100 dataset classes.
Description
Description
Class 4
Actionl18-- Front and rear foot pads
Actioa049- Jumping Rope in Place
Class
Action250- Standing Touch Toe
Class
Action 149. Surrender
Action127. Touch Shoulder
Action143 [43. Twist waist
Action086- hiss action
Wipe Nock
jumping jack
Action292. Air Kiss
Action171 171. Cover Tars
Class 72
Action881 Shoulder
Action277-
Class 30
Action273. Shoolder Wrap
Action023. Side Lift
Action215- Skew Head Biye
Actioa168- Block the Sun
Action250= Standing Touch Ioe
Actioa211 Mummy Jump
Class 36
Action0666 Right single swing arm
Action296- Applause
step forward
Action 85 clench your list and start sumning
Action (0S Leli hand circle
Action195-
of head
Action174- Beating
Action039 Forward and backward sliding steps
Action131. Tie Hair
Class 40
Action240- Standing Long Jump
Actionl822 Shout
Action206- Hip Up Kick Jump
Action240- Standing Long Jump
Action199. Touch the lorehead
Action273. Shoulder Wrap
Lell
Left iliopsoas muscle stretching
Action Touch the forchead
Action2 22- Comb Hair
Action013 1-99h of brondenst gymnastics kicking exercises
Class 90
Action067. Right thigh front strerch
Action250- Standing Touch Toc
Action255. Simplified Chi as sealed
Action (05. Left hand raised
Action047. In Place Wide Distance Run
16 jumping jack
jumping jack
Action244- Standing Lett Leg Lift
Action031. Make Face
Actioal08- Lelt Oblique Pull Down Half Squat
Actica016. 6- Alternate Froot Kick Jump
Action078. Right iliopsons musele stretch
28

---

Table 13: Meta-action descriptions for selected action series in the TemArDVS100 dataset. TemArDVS100 includes 100 action series of varying durations created by randomly combining event streams from HARDVS [63] and DailyDVS-200 [61]. TemArDVS100 features action series with identical meta-action but different combinations, enabling fine-grained temporal labeling of action transitions.
Closs Indes
Desciipiin
Clas Iodes
Descripaon
Tinow the ball in lund imo baskzl
Walt Rimeent with your Elic ou and oycs lonking sssigh shooe
Thrn off the top the wazer dispesser or sisk
your
cyex lioking striiht shandl
Clao
Turn oer top of the wate: dispenser sink.
Actiontiil
Cumpasiso
Companionn inee
deperser
Class
Walk forwoed with cheat
Throw
linking ensight shouel
Class
Tien ofl
Actostio Head
water digeraee sink
Actiontio Head to Head Compurison true
Walk fewwadd with cheit and eyes looking stright thead
Actimi33 35- Plasing TG Chi inse
Pui the that is tand or in the ubll.
Actionr2. Daw Cirile E.how thee Action242. Stanling
Action111- Mamme tump mus Action355 Playine lase
Clsa
Munmy
Actiou063..
Action272
Rakes both lands, make
esierd oetword from inside.
Tear
Pat on hat thet is hand on table
Action242 Standing Right Lift trme
Athinn 33- Playtng Tia Chi ins:
Tear giece of pepcr.
Tor piece of poper:
Activn063 Right humeral tnpe exzenionn te
Astion242.
Clsa
Class
Playing
Actin221 21 Mummy Ina:
Put
land oable.
Actiooni33 Rightiument siple estensioa tiee Aclioe242 Stoding Right Log Lift osc.
Action222 Dipo Clicle at Elbo tue Raise one both fand, make fol, and caioal unlward from the insik:
29


Reviews:
[{"review_id": "Vz3csvB1hn", "reviewer": "Reviewer_oNQK", "summary": "This work tackles two existing problems in event-based action/object recognition: restricted distribution of event sequence lengths and generalization across varying frequencies, by creating a new kind of event information representation method. It proposes the Path-selective Event Aggregation and Scan (PEAS) module to encode event streams of varying length into fixed length representations, and presented Multi-faceted Selection Guiding (MSG) Loss, which contains 2 kinds of information loss, to optimize the event representation generated by PEAS module. The method is evaluated on a series of datasets, and demonstrates superior performance against the existing baselines. Ablation studies shows the effectiveness of PEAS module and the newly created loss functions.", "questions": "Questions*\n(1) For the fixed length event frames generated by PEAS, when using SSMs as the backbone for object/action recognition, does the SSM actually utilize the event frames, or is its utilization of the initial information more limited than the later information?\n(2) The limited-length event streams generated by PEAS inevitably lead to information loss. How should this information loss be evaluated, and in which application scenarios is such loss acceptable, and in which cases might it have a significant impact?\n(3) What’s the authors’ thoughts on improving the spatiotemporal event modeling module?", "limitations": "Limitations are presented in the above Weaknesses section.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n(1) This work proposes a kind of representation learning method, which is capable of effectively encoding frame streams of varying event lengths and frequencies into fixed length representations, and achieves superior experimental results.\n(2) Through reduction of event length, it partially avoids the problem of SSMs easily forgetting initial information when processing long event sequences, and provides inspiration for effectively shortening the sequence length that SSMs need to handle.\nWeaknesses:\n(1) The information loss caused by PEAS module has not been quantitatively studied.\n(2) The proposed method in this paper, as a lossy representation approach, performs effectively on relatively simple tasks such as action/object recognition, but it is unclear whether it would be effective on more complex tasks.", "was_revised": true, "final_justification": "The author clearly addressed my questions in weaknesses and questions sections.", "rebuttal": "# Rebuttal to Reviewer oNQK\nWe sincerely appreciate the reviewer's recognition of our work's contributions, particularly regarding:\n- Novel Method: Our proposed PEAS effectively encode variable-length event streams into fixed representations, achieving superior superior performance against the existing baselines.\n- SSM Optimization: Addresses SSMs' forgetting problem by shortening sequences while preserving critical information.\n- Innovative Loss: Introduces MSG Loss to optimize frame selection, validated through ablations.\n- Strong Experiments: Demonstrates superior performance across multiple datasets with thorough analysis.\n- Generalization across Inference Frequencies: Demonstrates PASS's superior performance in handling varying inference frequencies.\n\nWe also sincerely appreciate the reviewer’s insightful feedback and constructive critiques. Below, we address each concern in detail, with additional clarifications and experimental support.\n## Question 1: SSM Utilization of PEAS-Generated Frames.\n*Concern:*\n\nThe reviewer asks whether the SSM backbone effectively utilizes the fixed-length event frames or disproportionately favors later information.\n\n*Response:*\n\nWe acknowledge the reviewer’s concern about potential biases in SSM’s utilization of event frames. Due to SSM’s recurrent nature, its hidden state updates can be influenced by input sequence length and feature order, particularly for long-range dependencies. To mitigate this, our PEAS module encodes events into fixed-length representations, ensuring balanced utilization of temporal information.\n\nKey evidence supporting this:\n\n- *Ablation Study (Table 4 main paper):* The additional baseline 'No Sampling' (using all event frames) achieves lower accuracy (92.90% on PAF, 92.31% on ArDVS100) compared to PEAS (93.33%, 92.84%), indicating that raw long sequences harm performance due to SSM’s limitations.\n\n- *Comparison to Random Sampling:* 'PEAS' outperforms 'Random Sampling' (+0.35% on PAF, +0.61% on ArDVS100), demonstrating its ability to preserve critical information.\n\n\n| Method | PAF | ArDVS100 |\n|--------|-------|---------|\n| No Sampling | 92.90% | 92.31% |\n| Random Sampling | 92.98% | 92.23% |\n| PEAS | 93.33% | 92.84% |\n| PEAS + L_MSG| 94.83% | 93.85% |\n\nTable 4: Ablation study for the PEAS module with additional 'No Sampling' experiment result. We will update the Table 4 (main paper) in the final version.\n\n\n## Question 2: Evaluating Information Loss in PEAS.\n*Concern:*\n\nThe reviewer raises concerns about quantifying information loss and its acceptability in different scenarios.\n\n*Response:*\n\nWe admit the PEAS module can introduce some randomness for the frame selection at the onset of training due to the random weight initialization of the PEAS module. Consequently, during training, the model can only optimize performance based on these randomly selected frames, rather than improving the PEAS module for adaptive selection of input events. Howerver, evaluating information loss is challenging since there is no universally accepted metric to define and measure it. In this paper, we addressed it through indirect and qualitative analyses.\n\n- *Quantitative Evaluation (Table 4 in main paper):*\n    - 'PEAS' improves accuracy over 'No Sampling' (+0.43% on PAF, +0.53% on ArDVS100), suggesting that the selected frames retain task-relevant information despite compression.\n    - 'PEAS' boosts accuracy compared with 'Random Sampling' (+0.35% on PAF, +0.61% on ArDVS100), demonstrating its ability to preserve critical information.\n    - With MSG Loss ('PEAS + $L_{MSG}$'), performance further improves to 94.83% (PAF) and 93.85% (ArDVS100), validating that $L_{MSG}$ mitigates randomness in frame selection. The MSG loss consists of two parts: (1) WEIE Loss is designed to reduce the randomness of the selection process to ensure the selected sequence features can encapsulate the entirety of the sequence, and (2) IEMI Loss is designed to guarantee that each selected event feature stands out from the others. They are motivated by the information entropy and mutual information from the information theory so as to guide the PEAS module to select the most representative event frames without introducing information loss.\n    \n- *Qualitative Analysis (Figure 6 in supplementary):* It demonstrates that the PEAS module can select the most representative event frames that are more informative and discriminative for the event recognition task after 100 training epochs. Currently, this part is presented in the supplementary material due to page limitations. Considering that this part is very important and has been mentioned by multiple reviewers, we will move this part to the main text in the final version.\n\n- *Task Suitability:*\n    - We can adjust the hyperparameter K, which is the selected number of event frames, to adapt to the different applications. \n    - Fast-moving scenarios: Larger K may be needed to capture rapid changes as the event camera triggers more events and the resulting event frames differ a lot.\n    - Slow-moving scenarios: Smaller K may be enough to capture the main information of the event stream as the resulting event stream is dense.\n\n\n## Question 3: Discussion on Improving the Spatiotemporal Event Modeling.\n*Concern:*\n\nThe reviewer asks for thoughts on enhancing the event modeling module.\n\n*Response:*\n\nThe improvement of the spatiotemporal modeling of event data can be discussed from two aspects: event representation and network structure.\n- *Event Representation (Sec. 3.1):*\n    - Currently, there is no commonly used event representation for event data. Event representation varies from task to task and adapts to different network architecture. \n    - In this paper, we use the event frame as the event representation. Ablations in Table 6 (main paper) show that the event frames are optimal for the spatiotemporal modeling of event data using SSM.\n- *Network Structure (Sec. 3.2):*\n    - Different network architectures are tailored to specific event representations for spatiotemporal modeling. For instance, event point clouds and graph representations are compatible only with specialized networks, such as point transformers or graph neural networks.\n    - In this work, we address the challenge of processing long event streams, where the number of aggregated event frames varies significantly due to the high temporal resolution of events. This variability complicates spatiotemporal modeling. \n    - Therefore, we propose the PEAS module to select the most representative event frames. followed by SSM to model the spatiotemporal information based on the selected event frames. The PEAS can be viewed as a dynamic path selection module, which can adapt to the different event stream lengths. As shown in Table 3 (main paper), our approach achieves robust performance across a wide event distribution (10^6 to 10^9 events), attaining 97.35% on ArDVS100. Moreover, it demonstrates fine-grained temporal recognition, scoring 89.00% on the challenging TemArDVS100 dataset.\n\n\n## Question 4: Applicability to Other Tasks.\n\n*Concern:*\n\nThe reviewer questions whether PASS generalizes to complex tasks.\n\n*Response:*\n\nWe appreciate the suggestion to extend our method to other tasks. However, due to time constraints, we are unable to conduct comprehensive experiments to evaluate its performance on additional tasks at this stage. We plan to extend our method to other tasks like event-based object detection and event-based action recognition in future work.\n\nWe would also like to clarify that event recognition remains a challenging task, as evidenced by current benchmark results. For instance:\n\n- On the N-Imagenet object recognition task, state-of-the-art methods achieve only 40-60% accuracy (our method reaches 61.32%, as shown in Table 1 in the main paper), significantly lower than the ~90% accuracy typical for standard ImageNet classification.\n\n- On the SeAct action recognition dataset, existing methods attain 50-60% accuracy (our method achieves 66.38%, Table 2 in the main paper).\n\n- For long-length event streams (~10^9 events), baseline methods achieve 79.62% accuracy on the TemArDVS100 dataset with fine temporal labels, while our method reaches 89.00% (Table 3 in the main paper).\n\nThese results demonstrate that while event recognition performance has saturated on small-scale datasets, substantial improvements are still needed for large-scale or temporally complex scenarios."}, {"review_id": "8npDnFo8CN", "reviewer": "Reviewer_ot6Q", "summary": "This manuscript introduces PASS, a Path-Selective State Space Model framework designed for event-based object and action recognition. \n\nThe primary objective is to address limitations in existing methods that inadequately leverage the high temporal resolution of event data and suffer from poor generalization across varying event lengths and sampling frequencies. \n\nPASS utilizes a Path-Selective Event Aggregation and Scan (PEAS) module for adaptively aggregating and encoding event frames. It further employs a Multi-Faceted Selection Guiding (MSG) loss to reduce randomness and redundancy among the selected features. \n\nExtensive evaluations across five public datasets and three newly introduced datasets (ArDVS100, TemArDVS100, Real-ArDVS10) demonstrate that PASS outperforms state-of-the-art methods, exhibiting robust generalization across diverse event lengths and inference frequencies.", "questions": "1. Based on Weakness #1: Can you clarify which component—PEAS module or MSG loss—plays a more significant role in improving generalization across varying inference frequencies? Can you add discussions on the setting formulation with existing works [R1] and [R2]? Meanwhile, please discuss on the differences and similarities with [R4].\n\n---\n2. Based on Weakness #2: Can you provide further interpretability analyses or visualizations that elucidate how the PEAS module selects frames, and how MSG loss reduces redundancy?\n\n---\n3. Based on Weakness #3: Have you performed detailed computational complexity or runtime analyses under realistic deployment scenarios? How might these findings affect practical applications?\n\n---\nAdditional questions (minor):\n\n4. Clearly define acronyms (PEAS, MSG, SSM, etc.) upon first usage for better readability.\n\n---\n5. Provide detailed computational resource requirements (e.g., GPU type, inference time) explicitly.\n\n---\n6. Do your proposed datasets sufficiently cover diverse real-world scenarios? How might the model perform under drastically different event scenarios not explicitly tested in your experiments?\n\n---\n7. Consider additional qualitative visualizations of intermediate model outputs to improve interoperability.\n\n---\n8. Discuss potential scalability issues and future steps to address them (e.g., model compression techniques).\n\n---\n9. Please add more discussions on tasks beyond recognition, such as event-based object detection [R3, R4] and semantic segmentation [R5, R6].\n\n---\nReferences:\n- [R1] Daniel Gehrig and Davide Scaramuzza. Low-latency automotive vision with event cameras. Nature, 629(8014):1034–1040, 2024.\n\n- [R2] Dongyue Lu, Lingdong Kong, Gim Hee Lee, Camille Simon Chane, and Wei Tsang Ooi. FlexEvent: Towards flexible event-frame object detection at varying operational frequencies. arXiv preprint arXiv:2412.06708, 2024.\n\n- [R3] Mathias Gehrig and Davide Scaramuzza. Recurrent vision transformers for object detection with event cameras. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13884–13893, 2023.\n\n- [R4] Nikola Zubíc, Mathias Gehrig, and Davide Scaramuzza. State space models for event cameras. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5819–5828, 2024.\n\n- [R5] Zhaoning Sun, Nico Messikommer, Daniel Gehrig, and Davide Scaramuzza. ESS: Learning event-based semantic segmentation from still images. In European Conference on Computer Vision, pages 341–357, 2022.\n\n- [R6] Lingdong Kong, Youquan Liu, Lai Xing Ng, Benoit R. Cottereau, and Wei Tsang Ooi. OpenESS: Event-based semantic scene understanding with open vocabularies. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15686–15698, 2024.", "limitations": "- The performance of the proposed framework might heavily rely on the tuning of hyperparameters such as the number of selected frames (K) or aggregation frequency, potentially requiring considerable tuning effort for new datasets.\n\n- Larger variants of the model reportedly show signs of overfitting. While acknowledged, explicit strategies for mitigating this, beyond general suggestions, are not provided in sufficient detail.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n(+) The manuscript aims to address known challenges in existing event-based recognition methods, particularly the limitations in handling diverse event lengths and sampling frequencies. The proposed PEAS module aggregates events with adaptive selection and scanning, improving representation efficiency. The MSG loss tries to address redundancy and randomness, enhancing model generalization.\n\n(+) Comprehensive experiments conducted on both existing benchmarks and newly introduced datasets convincingly demonstrate the superior performance and generalization capabilities of the PASS framework.\n\n(+) The manuscript is overall clear, organized, and effectively communicates the methodology, findings, and observations.\n\n---\n### Weaknesses\n(-) While extensive experiments are presented, detailed ablations specifically dissecting the relative contributions of the PEAS module versus MSG loss across diverse datasets would strengthen the manuscript's insights. Also, the idea of conducting frequency-adaptive event-based recognition has been studied in existing literature, while there is no discussions on the relevant works [R1] and [R2].\n\n(-) The introduction of multiple new components (PEAS and MSG loss) makes the model complex. Additional analysis regarding interpretability or insights into the internal decision-making process would help clarify their roles.\n\n(-) The complexity of the PEAS module, coupled with SSM layers, may pose computational challenges. Explicit analyses or discussions regarding computational overhead, runtime, and memory consumption would be beneficial.\n\n---\nReferences:\n- [R1] Daniel Gehrig and Davide Scaramuzza. Low-latency automotive vision with event cameras. Nature, 629(8014):1034–1040, 2024.\n\n- [R2] Dongyue Lu, Lingdong Kong, Gim Hee Lee, Camille Simon Chane, and Wei Tsang Ooi. FlexEvent: Towards flexible event-frame object detection at varying operational frequencies. arXiv preprint arXiv:2412.06708, 2024.", "was_revised": true, "final_justification": "I have read the response, as well as the responses for review comments from other reviewers.\n\nBased on what I learn from the review comments and responses, I believe part of the concerns have been addressed. Therefore, I am leaning towards maintaining the 4 - borderline accept rating.\n\nI suggest that the authors incorporate all clarifications, modifications, and new experiments into the revised manuscript, to ensure that the quality meets what NeurIPS always looks for.", "rebuttal": "# Rebuttal to Reviewer ot6Q\n\nWe sincerely value the reviewer's recognition of our work's contributions, particularly regarding:\n- Novel Technical Contribution:\n    - Introduces PASS framework with two key innovations:\n        - PEAS module: Adaptive event aggregation with path selection \n        - MSG loss: Reduces feature redundancy through end-to-end optimization\n- Comprehensive Empirical Validation:\n    - Outperforms SOTA across 8 datasets\n    - Demonstrates superior generalization:\n        - Across event lengths (1-10^9 events)\n        - Under varying inference sampling frequencies\n    - Introduces valuable new benchmarks:\n        - ArDVS100/TemArDVS100 for long-sequence analysis\n        - Real-ArDVS10 for real-world applicability\n- Presentation Quality\n    - Clear and well-organized manuscript\n    - Thorough ablation studies and analysis\n\nWe sincerely appreciate the reviewer’s thoughtful critique, which has helped us improve the manuscript. Below, we address each weakness and question with concrete revisions:\n\n## Q1. Component Ablation & Prior Work Discussion\n\n*Concern:*\n\n- Needs clearer ablation of PEAS vs. MSG contributions\n- Additional discussion of frequency-adaptive prior work ([R1], [R2], [R4])\n\n*Response:*\n\n| Method | PAF | ArDVS100 |\n|--------|-------|---------|\n| Random Sampling | 92.98% | 92.23% |\n| PEAS | 93.33% | 92.84% |\n| Random Sampling + L_MSG | 92.64% | 91.89% |\n| PEAS + $L_{MSG}$ | 94.83% | 93.85% |\n\nTable 1: Ablation study for the PEAS module with additional 'Random Sampling + $L_{MSG}$ ' experiment result. We will update the corresponding table 4 of the main paper in the final version.\n\n- Added new ablation studies in Table 1, showing:\n    - MSG Loss Requires Learnable Selection:\n        - Random Sampling + $L_{MSG}$ performs worse (92.64%/91.89%) than baseline random sampling (92.98%/92.23%)\n        - This confirms MSG loss cannot meaningfully guide completely random selection\n    - PEAS Enables Effective Optimization:\n        - PEAS alone outperforms random sampling (93.33%/92.84% vs 92.98%/92.23%)\n        - PEAS + MSG achieves best results (94.83%/93.85%), demonstrating the importance of learnable frame selection and also proves MSG loss's ability to optimize PEAS's selection process\n    - Conclusion: The MSG loss provides meaningful guidance only when combined with a learnable selection mechanism (PEAS), as it requires gradient pathways for optimization.\n\n\n| Method | Training Frequency (Hz) | Inference Frequency (Hz) |\n|--------|-------|---------|\n| R1| 20 | 20 36 90 180 |\n| R2| 20 | 20 36 90 180 |\n| R4| 20 | 20 40 80 100 200|\n| Ours| 20 60 100 | 20 40 60 80 100 |\n\nTable 2: The differences and similarities for evaluation settings of the inference frequencies generalization among the existing works [R1], [R2], and [R4]. We will add this table in the supplementary in the final version.\n\n- Added discussion in Table 2, showing:\n    - The existing works [R1], [R2], and [R4] only evaluate the inference frequencies generalization on the 20 Hz training frequency with inference frequencies ranging from 20 Hz to 200 Hz with different intervals.\n    - Our PASS model evaluates the inference frequencies generalization on the 20 Hz, 60 Hz, and 100 Hz training frequencies and the inference on the 40 Hz, 60 Hz, 80 Hz, 100 Hz, and 200 Hz inference frequencies.\n    - Our evaluation settings are more comprehensive; they not only investigate the inference frequency generalization on a model trained with 20 Hz and applied to 40 to 200 Hz, but also investigate models trained with 60 Hz and 100 Hz and applied to 20 to 200 Hz and 20 Hz to 80 Hz, respectively.\n\n## Q2. Interpretability analyses for the PEAS module\n*Concern:*\n\n- Needs clearer visualization of PEAS’s frame selection\n\n*Response:*\n\n- Added in revision: Currently, the qualitative analysis of the PEAS module was presented in the supplementary Figure 6 due to page limitations. Considering that this part is very important for your valuable suggestion, we will move this part to the main text in the final version.\n- Explanation for PEAS frame selection: At epoch 0, the PEAS module randomly selects event frames, resulting in unnecessarily padded frames and redundant event frames with repetitive information. After 100 epochs, the eight chosen frames exclude redundant frames and non-informative padding, resulting in selected frames that are semantically meaningful. \n- Key findings: This visualization result demonstrates the effectiveness of the PEAS module and the MSG loss. We can conclude that the MSG loss can effectively guide the PEAS module to select the most representative event frames, thus minimizing the randomness and redundancy of the encoded features during the PEAS selection process.\n\n## Q3. Computational complexity analyses\nReviewer Concern:\n\n- Needs runtime/memory analysis\n\nResponse:\n\n- Added detailed benchmarks (Table 3):\n    - As suggested, we provide additional detailed computational resource requirements on one A100 GPU with 80GB memory.\n    - As shown in Table 3, when our PASS model is applied in the real-world scenario, we can select different model versions based on the application requirements. For example:\n        - If the application requires high accuracy, we can select the Middle model with 74M parameters and 12.7G FLOPS and 24.7 FPS. \n        - If the application requires low latency, we can select the Tiny model with 7M parameters, 1.1G FLOPS, and 243.9 FPS.\n\n| Model | Layer | Dim D | Param. | FLOPS(G) | Inference Time(ms) | FPS |\n|--------|-------|---------|-------|-------|-------|-------|\n| Tiny (T) | 24 | 192 | 7M | 1.1 | 4.1 | 243.9 |\n| Small (S) | 24 | 384 | 25M | 4.3 | 15.7 | 63.7 |\n| Middle (M) | 32 | 576 | 74M | 12.7 | 40.4 | 24.7 |\n\nTable 3: The computational complexity and memory consumption of the PASS model. We will update the corresponding table 7 in the supplementary in the final version.\n\n\n## Additional questions (Minor)\n\n### Q4. Datasets coverage\n- We have tried our best to evaluate our PASS model across a wide range of event lengths and event frequencies, covering five publicly available event datasets: PAF, SeAct, HARDVS, N-ImageNet, and N-Caltech101. \n- Given the existing datasets only provide events within a limited distribution of event length (10^6 for objects and 10^7 for actions). We introduce the ArDVS100 and TemArDVS across a broad distribution of event length (10^6 to 10^9), synthesized by concatenating event streams with varying meta actions, thus capturing action transitions over time. \n- Additionally, to assess the model’s real-world applicability, we created a real-world dataset, named Real-ArDVS10, comprising event-based actions lasting from 2s to 75s, encompassing 10 distinct classes selected from the ArDVS100 datasets.\n\n### Q5. Discussion on potential scalability issues\n\nPASS demonstrates strong scalability through PEAS adaptively encoding (handling 10^9 events) and MSG loss to guide the PEAS module to select the most representative event frames. While larger models (74M params) show diminishing returns, we propose three potential improvements:\n\n- Model compression via quantization like INT8, INT4, etc.\n- Dynamic K-selection adapting to event density.\n- Hardware optimizations like CUDA kernel fusion.\n\n### Q6. Modification for readability\n\nWe appreciate this helpful suggestion for improving readability. In the revised manuscript, we will clearly define all acronyms upon first use:\n- PEAS: Path-selective Event Aggregation and Scan\n- MSG: Multi-faceted Selection Guiding loss\n- SSM: State Space Model \n- WEIE: Within-Frame Event Information Entropy \n- IEMI: Inter-Frame Event Mutual Information\n\n### Q7. Discussions on tasks beyond recognition\n\nWe appreciate this valuable suggestion. In future work, we will expand discussions on broader applications:\n- Object Detection: PASS's adaptive frame selection (PEAS) could enhance [R3]'s event transformers by improving long-term temporal modeling and complement [R4]'s SSM detection framework through more efficient event representation.\n- Semantic Segmentation: The MSG loss's redundancy reduction may benefit [R5]'s feature learning, while PEAS could optimize [R6]'s open-vocabulary pipeline by filtering noisy events.\n\n### Q8. Hyperparameter K tuning\n\nWe appreciate this valuable suggestion. In future work, we will expand discussions on dynamic hyperparameter K tuning adapting to:\n- event length\n- event density\n\n### Q9. Strategies for mitigating the overfitting issue of large variants of SMM model\nWe propose three potential strategies to address overfitting in our large variants of the SMM model:\n- Stochastic Path Dropout: Randomly masking 20% of PEAS paths during training to reduces overfitting\n- Temporal Augmentation: Event frame shuffling improves generalization\n- Regularized MSG Loss: Added L2 penalty on frame selection scores (λ=0.1)"}, {"review_id": "3frDCPNDOM", "reviewer": "Reviewer_Tnzz", "summary": "This paper addresses event-based recognition in long event streams. Leveraging efficient state space models, this work introduces a Path-selective Event Aggregation and Scan (PEAS) module for adaptive event encoding and a novel Multi-faceted Selection Guiding (MSG) loss to minimize feature redundancy. The approach outperforms prior methods on multiple datasets and establishes new benchmarks for long-duration event recognition.", "questions": "1) Question about fair baseline scaling: Does PASS outperform SSM baselines when model size (params) and frame count (K) are equalized?\n\nSuggestion:\n- Benchmark against S5-ViT-M-K(16) (matching PASS-M-K(16)'s 74M params and K=16).\n- Compare with EventDance+SSM (EventDance upgraded with S5-ViT-M).\n\n2) Please clarify: Does PASS maintains an accuracy advantage over *equally scaled* SSM baselines on N-Caltech101/PAF (object/action recognition)? If  SSM baselines are not equally scaled, the performance gains could be attributable to the larger ViT or SSM.", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "While prior work typically uses fixed temporal intervals for event representation, this work leverages state-space models (SSMs) to enable an adaptive sampling frequency of long video streams and thus handle ultra-long sequences. It achieves SOTA performance while efficiently processing event streams of length up to 10^9, setting new performance benchmarks. \n\nWeaknesses:\n- Clarity can be significantly improved, especially the presentation of experimental results. It is not clear \n\n- Evaluation is insufficient. The  ablation studies fail to quantify contributions of PEAS/MSG versus SSM-ViT backbones. Comparison against scaled baselines (e.g., S5-ViT-M-K(16)) is insufficient, as the baselines seem to use different scaling parameters than this approach. This needs to be clarified to support the claims of superior performance (please see the details in Questions).", "was_revised": true, "final_justification": "I have read the authors’ responses to both my review and those of the other reviewers. While the rebuttal provides helpful clarifications, the results do not convincingly demonstrate that the proposed PEAS and MSG modules yield meaningful performance gains beyond what is achieved by simply adopting a stronger, larger SSM backbone. The use of a lossy fixed-$k$-frame representation raises concerns about the method’s applicability to tasks requiring fine-grained temporal or structural reasoning, as also pointed out by other reviewers.", "rebuttal": "# Rebuttal to Reviewer Tnzz\nWe sincerely value the reviewer's recognition of our work's contributions, particularly regarding:\n\n- Innovative Method: Introduces the PEAS module for adaptive event encoding, and the novel MSG loss minimizes feature redundancy.\n- Technical Impact: First SSM adaptation for ultra-long event streams (up to 10⁹ events).\n- SOTA Performance: Outperforms prior methods on multiple datasets.\n- Dataset Contribution: Establishes new benchmarks for long-duration event recognition\n\nAdditionally, we greatly appreciate the reviewer's thoughtful assessment and valuable suggestions for improving our work. Below we address each concern with concrete revisions:\n\n## Q1. Baseline Scaling & Fair Comparison (Primary Concern)\n\nWe sincerely appreciate the reviewer's thoughtful feedback regarding baseline comparisons. Below, we provide a detailed justification for our scaling methodology and experimental design:\n\n### 1. Scaling Methodology of PASS\nOur PASS framework builds upon the VideoMamba SSM backbone [1], with the PEAS module (a lightweight two-layer 3D convolutional scorer) serving as the primary parameterized component. Since PEAS’s non-parametric operations (e.g., frame selection) introduce negligible parameters, the total model size is dominated by the SSM backbone. For fair comparisons, we adopt the standard ViT scaling protocol, producing Tiny (7M), Small (25M), and Middle (74M) variants by adjusting depth and embedding dimensions proportionally.\n\n### 2.  Challenges with Suggested Baseline Modifications\n- *Challenges with Suggested S5-ViT-M-K(16):* The S5-ViT baseline [2] combines an S5 [3] SSM with a ViT for event-based object detection. While we faithfully reproduced their official implementation (detailed in supplementary line 655 - line 670), substituting their Base ViT with a Middle variant *S5-ViT-M-K(16)* would introduce an unfair comparison for two reasons:\n    - Architectural Mismatch: S5-ViT is a hybrid SSM+ViT model, while PASS is purely SSM-based. Scaling the ViT component independently disrupts parameter parity (e.g., S5-ViT-M’s parameters diverge from PASS-M’s 74M).\n    - No corresponding ViT-M model: There is no corresponding ViT-M model publicly available since the ViT paper only provides the Small (22M), Base (86M), Large (307M), and Huge (632M) versions; none of them match our PASS-M (17.5M) parameter count.\n\n- *Challenges with Suggested EventDance+SSM*: EventDance [4] utilizes the convolution network (ResNet-18, 34, 50) combined with an EvFlowNet [5] as the flow estimation model and E2VID [6] as the reconstruction model. The whole architecture is designed for cross-modal adaptation from image to event modalities. Appending an SSM to this already complex pipeline would:\n    - Architectural Conflict: EventDance uses ResNet+EvFlowNet [5]+E2VID [6] for cross-modal adaptation. SSM insertion would require complete pipeline redesign.\n    - Skew Parameter Comparisons: ResNet-50 alone introduces ~25M parameters. The combined model size would dwarf our SSM-centric approach\n\n### 3. Additional Comparative Analysis\nDue to the above reasons, we tried our best to make a fair comparison with S5-ViT by adding the additional evaluation results for S5-ViT-B-K(2) on the N-Caltech101 datasets and S5-ViT-B-K(2) on the PAF, SeAct, and HARDVS datasets. The results are shown in the following table 1 and table 2:\n\n| Method | Param. | N-Caltech101 | N-Imagenet |\n|--------|-------|---------|-------|\n| S5-ViT-B-K(1) | 17.5M | 88.32% | - |\n| S5-ViT-M-K(2) | 17.5M | 88.44% | - |\n| EventDance | 26M | 92.35% | - |\n| PASS-T-K(1) | 7M | 88.29% | 48.74% |\n| PASS-T-K(2) | 7M | 89.72% | 48.60% |\n| PASS-S-K(1) | 25M | 90.92% | 53.74% |\n| PASS-S-K(2) | 25M | 91.96% | 56.10% |\n| PASS-M-K(1) | 74M | 94.20% | 61.12% |\n| PASS-M-K(2) | 74M | 94.60% | 61.32% |\n\nTable 1: Comparison with previous methods for event-based object recognition with additional evaluation results for S5-ViT-B-K(2) on the N-Caltech101 datasets. We will update Table 1 (main paper) in the final version.\n\n*Key Insight:*\n\nPASS-T-K(2) achieves superior accuracy (+1.28%) with 2.5× fewer parameters (7M vs 17.5M) than S5-ViT-B-K(2) on the N-Caltech101 datasets.\n\n\n| Method | Param. | PAF | SeAct | HARDVS |\n|--------|-------|---------|-------|-------|\n| S5-ViT-B-K(8) | 17.5M | 92.93% | 58.21% | 74.85% |\n| S5-ViT-B-K(16) | 17.5M | 93.12% | 57.37% | 95.98% |\n| PASS-T-K(8) | 7M | 91.38% | 51.72% | 98.40% |\n| PASS-T-K(16) | 7M | 94.83% | 49.14% | 98.37% |\n| PASS-S-K(8) | 25M | 93.33% | 60.34% | 98.20% |\n| PASS-S-K(16) | 25M | 96.55% | 62.07% | 98.41% |\n| PASS-M-K(8) | 74M | 98.28% | 65.52% | 98.05% |\n| PASS-M-K(16) | 74M | 96.55% | 66.38% | 98.20% |\n\nTable 2: Comparison with previous methods for event-based action recognition with additional evaluation results for S5-ViT-B-K(16) on the PAF, SeAct, and HARDVS datasets. We will update Table 2 (main paper) in the final version.\n\n*Key Insight:*\n\nPASS-T-K(16) achieves 94.83% performance on the PAF datasets, which is better than the S5-ViT-B-K(16) performance (93.12%) with fewer parameters (7M vs. 17.5M). Besides, our PASS-M-K(16) achieves 98.37% performance on the HARDVS datasets, which is better than the S5-ViT-B-K(16) performance (93.12%) with fewer parameters (74M vs 17.5M). The above results show PASS's consistent performance improvement and better parameter efficiency when we enlarge the selected event frame number.\n\n### 4.  Broader Advantages of PASS\nBeyond parameter efficiency, our method demonstrates:\n- Fine-grained Temporal Recognition: 89.00% on the TemArDVS100 dataset (vs. 79.62% for S5-ViT).\n- Real-World Robustness: 100% accuracy on the Real-DVS10 dataset.\n- Inference Frequency Generalization: Max accuracy drop of -8.62% (vs. -20.69% baseline).\n- Strong Long Spatiotemporal Modeling: Consistent performance across a broad distribution of event length (1-10^9).\n\nThese advantages stem from our novel PEAS module and MSG Loss, which enable effective spatiotemporal event modeling (see ablation studies in table 4 of the main paper). We will incorporate these additional results in the final version and welcome further discussion on evaluation protocols.\n\n---\n[1] Li, K., Li, X., Wang, Y., He, Y., Wang, Y., Wang, L., & Qiao, Y. (2024, September). Videomamba: State space model for efficient video understanding. In European conference on computer vision (pp. 237-255). Cham: Springer Nature Switzerland.\n\n[2] Nikola Zubíc, Mathias Gehrig, and Davide Scaramuzza. State space models for event cameras. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5819–5828, 2024.\n\n[3] Smith, J. T., Warrington, A., & Linderman, S. W. (2022). Simplified state space layers for sequence modeling. arXiv preprint arXiv:2208.04933.\n\n[4] Zheng, X., & Wang, L. (2024). Eventdance: Unsupervised source-free cross-modal adaptation for event-based object recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 17448-17458).\n\n[5] Zhu, A. Z., Yuan, L., Chaney, K., & Daniilidis, K. (2018). EV-FlowNet: Self-supervised optical flow estimation for event-based cameras. arXiv preprint arXiv:1802.06898.\n\n[6] Rebecq, H., Ranftl, R., Koltun, V., & Scaramuzza, D. (2019). Events-to-video: Bringing modern computer vision to event cameras. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3857-3866).\n\n\n## Q2. Experiment Result Reresentation\nWe sincerely appreciate the reviewer’s valuable feedback regarding the presentation of our experimental results. We acknowledge that the organization and clarity of our results could be improved, and we propose the following revisions for the final version:\n\n### 1. Restructured Results Section:\n    - We will reorganize Section 5.2 into clear subsections:\n        - 5.2.1: Event-based recognition results\n        - 5.2.2: Inference frequencies generalization results\n        - 5.2.3: Event frame selection visualization results\n\n### 2. Improved Table Presentations:\n    - All tables will be reformatted to include:\n        - Clearer headers with unified terminology\n        - Better grouping of comparable methods\n        - Consistent precision in reported metrics (all accuracies to 2 decimal places)\n\n### 3. Supplemental Analysis:\n    - We will include in the supplement:\n        - Complete per-dataset breakdowns\n        - Training curves\n        - Additional ablation studies\n\nWe believe these changes will significantly improve the clarity and impact of our results presentation. We sincerely thank the reviewer for this constructive suggestion, which will undoubtedly strengthen our paper’s quality."}, {"review_id": "uD4FESqJ1b", "reviewer": "Reviewer_7nZC", "summary": "This paper proposes PASS, a framework for event-based recognition that handles varying event lengths and sampling frequencies. The authors introduce a PEAS module for adaptive event frame selection and an MSG loss to guide this process. By utilizing state space models for efficient spatiotemporal modeling, PASS achieves strong performance and generalization across multiple benchmarks.", "questions": "See weakness.", "limitations": "The authors mentioned the limitations in the conclusion.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n1. The paper is overall well-written and easy-to-read.\n2. The paper is well-motivated and technically sound, addressing key limitations in event-based recognition, such as handling highly variable event lengths and poor generalization to different sampling frequencies. The proposed PASS framework effectively addresses these issues by adaptively selecting informative event frames.\n3. The method outperforms prior work by significant margins (e.g., +8.31% on HARDVS) across object and action recognition tasks.\n4. The authors provide three new datasets (ArDVS100, TemArDVS100, Real-ArDVS10), supporting broader evaluation of long and realistic event sequences.\n\n### Weaknesses\n1. While the PEAS module is designed to select informative frames via MSG loss, the interpretability of the selected paths remains unclear. The paper would benefit from qualitative analyses or visualizations showing which frames are selected and why, in order to justify that the selections align with semantically meaningful moments in the event stream.\n2. The PEAS module selects a fixed number of frames from the event stream, but the temporal coherence of the selection is not guaranteed. In particular, selected frames can be concentrated in the early part of the event sequence or capture redundant moments, potentially missing critical parts or temporal dependencies. \n3. Table 4 effectively evaluates the components of the proposed PASS model. To fully justify the individual contribution of the two components, it would be helpful to include an experiment where frames are selected via random sampling instead of using the PEAS module, and only the MSG loss is applied.", "was_revised": true, "final_justification": "My concerns have been addressed. I keep my rating at \"Borderline accept\"", "rebuttal": "# Rebuttal to Reviewer 7nZC\n\nWe sincerely value the reviewer's recognition of our work's contributions, particularly regarding:\n- Innovative Method: Introduces a PEAS module for adaptive event frame selection and an MSG loss to guide this process.\n- Technical Impact: Outperforms SOTA by +8.31% (HARDVS) and handles highly variable event lengths with generalization to different sampling frequencies.\n- New Benchmarks: Introduces 3 datasets for broader evaluation of long and realistic event sequences.\n- Clear Presentation: well-written, easy-to-read, well-motivated, and technically sound.\n\nAdditionally, we greatly appreciate the reviewer's thoughtful assessment and valuable suggestions for improving our work. Below we address each concern with concrete revisions:\n\n## Q1. PEAS Module Interpretability\n- Added in revision: Currently, the qualitative analysis of the PEAS module was presented in the supplementary Figure 6 due to page limitations. Considering that this part is very important for your valuable suggestion, we will move this part to the main text in the final version.\n- Explanation for PEAS frame selection: At epoch 0, the PEAS module randomly selects event frames, resulting in unnecessarily padded frames and redundant event frames with repetitive information. After 100 epochs, the eight chosen frames exclude redundant frames and non-informative padding, resulting in selected frames that are semantically meaningful. \n- Key findings: This visualization result demonstrates the effectiveness of the PEAS module and the MSG loss. We can conclude that the MSG loss can effectively guide the PEAS module to select the most representative event frames, thus minimizing the randomness and redundancy of the encoded features during the PEAS selection process.\n\n## Q2. Temporal coherence of the PEAS selection\n- Selection randomness: We admit the PEAS module can introduce some randomness for the frame selection at the onset of training due to the random weight initialization of the PEAS module. Consequently, during the start of training, the model can only optimize performance based on the distribution of these randomly selected frames, rather than improving the PEAS module for adaptive selection of input events. \n- Solution: MSG loss to facilitate effective optimization.\n    - WEIE Loss: To reduce the randomness of the selection process to ensure the selected sequence features can encapsulate the entirety of the sequence.\n    - IEMI Loss: To guarantee that each selected event feature stands out from the others.\n    - Both of the two losses are motivated by the information entropy and mutual information from the information theory so as to guide the PEAS module to select the most representative event frames without the loss of temporal coherence. \n- Evidence:\n    - The ablation study in Table 4 (main paper), 'PEAS + $L_{MSG}$,' proves the effectiveness of the MSG loss, which achieves 94.83% and 93.85% accuracy gain on the PAF and ArDVS100 datasets, respectively. \n    - The qualitative analysis of the PEAS module in Figure 6 (Supplementary) also proves the temporal coherence of the selection event frames.\n\n## Q3. Ablation Study with Random Sampling + MSG Loss\n\nWe thank the reviewer for this valuable suggestion. We have conducted the additional ablation study comparing random sampling with learned selection (PEAS), both with and without MSG loss. The updated results in the following Table 4 (main paper) reveal several key insights:\n\n| Method | PAF | ArDVS100 |\n|--------|-------|---------|\n| Random Sampling | 92.98% | 92.23% |\n| PEAS | 93.33% | 92.84% |\n| Random Sampling + L_MSG | 92.64% | 91.89% |\n| PEAS + $L_{MSG}$ | 94.83% | 93.85% |\n\nTable 4: Ablation study for the PEAS module with additional 'Random Sampling + L_MSG' experiment result. We will update Table 4 (main paper) in the final version.\n\n*Key Findings:*\n- 1. MSG Loss Requires Learnable Selection:\n    - Random Sampling + MSG performs worse (92.64%/91.89%) than baseline random sampling (92.98%/92.23%)\n    - This confirms MSG loss cannot meaningfully guide completely random selection\n- 2. PEAS Enables Effective Optimization:\n    - PEAS alone outperforms random sampling (93.33%/92.84% vs 92.98%/92.23%)\n    - PEAS + MSG achieves best results (94.83%/93.85%), demonstrating the importance of learnable frame selection and also proves MSG loss's ability to optimize PEAS's selection process\n\n*Conclusion:* The MSG loss provides meaningful guidance only when combined with a learnable selection mechanism (PEAS), as it requires gradient pathways for optimization.\n\nThis analysis has been added to Table 4 in the main paper and will be discussed in Section 4.3 of the revised manuscript."}]

Author rebuttal:

Meta-review: This paper introduces an interesting extension of SSM models for longer-range event recognition through frame selection. The idea is novel and has impact potential, and the reviewers generally had a positive inclination toward the work. The authors have provided evidence addressing most of the main concerns raised during review.

One important issue raised was whether the additional model complexity is necessary, or if scaling a standard SSM architecture would achieve similar performance. While the authors present results suggesting their design is beneficial, this comparison remains a central baseline and could be emphasized more strongly. Incorporating clearer evidence and discussion on this point in the final version would help strengthen the paper.

Overall, the contributions are promising, and the reviewers appreciated both the direction and the thorough author responses. Addressing the above points in revision will further clarify the value of the approach.
Decision: Accept (poster)