PLEIADES: Building Temporal Kernels with Orthogonal Polynomials
Authors: Yan Ru Pei
Olivier JMD Coenen
Venue: NeurIPS 2025 poster ()
Keywords: orthogonal polynomials
spatiotemporal
event-based
convolutional neural networks
object detection
Primary area: deep_learning
TL;DR: using polynomials to build spatio-temporal networks

Abstract: We introduce a class of neural networks named PLEIADES (PoLynomial Expansion In Adaptive Distributed Event-based Systems), which contains temporal convolution kernels generated from orthogonal polynomial basis functions. We focus on interfacing these networks with event-based data to perform online spatiotemporal classification and detection with low latency. By virtue of using structured temporal kernels and event-based data, we have the freedom to vary the sample rate of the data along with the discretization step-size of the network without additional finetuning. We experimented with three event-based benchmarks and obtained state-of-the-art results on all three by large margins with significantly smaller memory and compute costs. We achieved: 1) 99.59% accuracy with 192K parameters on the DVS128 hand gesture recognition dataset and 100\% with a small additional output filter; 2) 99.58% test accuracy with 277K parameters on the AIS 2024 eye tracking challenge; and 3) 0.556 mAP with 576k parameters on the PROPHESEE 1 Megapixel Automotive Detection Dataset.

PLEIADES: Building Temporal Kernels with Orthogonal Polynomials
Yan Ru Pei* NVIDIA Corporation Santa Clara, CA 95051 yanrpei yanrpei@gmail com
Olivier Coenen" Independent Researcher Encinitas, CA 92024 ol olivierr@llvveercceeee.ccmm
Abstract
We introduce a class of neural networks named PLEIADES (Polyynomial Ex- pansion Adaptive Distributed Event-based Systems), which contains temporal convolution kernels generated from orthogonal polynomial basis functions. We focus on interfacing these networks with event-based data to perform online spa- tiotemporal classification and detection with low latency. By virtue of using structured temporal kernels and event-basee data, we have the freedom to vary the sample rate of the data along with the discretization step-size of the network without additional finetuning. We experimented with three event-based benchmarks and obtained statc-of-the of-tth--tttttttte results all three by large margins with significantly smaller memory and compute costs. We achieved: 1) 99.59% accuracy with 192K parameters the DVS128 hand gesture recognition dataset and 100% with small additional output filter; 2) 99.58% test accuracy with 277K parameters on the AIS 2024 eye tracking challenge; and 3) 0.556 mAP with 576k parameters on the PROPHESEE Megapixel Automotive Detection Dataset.
Introduction
Temporal convolutional networks (TCNs) Lea al.) [016] have been staple for processing time series data from speech enhancement (Pandey and Wang 2019) to action segmentation [Lea et al. 2017]. However, in most cases, the temporal kernel is very short (usually size of 3), making it difficult for the network to capture long-range temporal correlations. The temporal kernels are intentionally kept short, because keeping long temporal kernel with large number of trainable kernel values usually leads to unstable training. In addition, they require a large amount of memory to store the weights during inference. One popular solution for this has been parameterize the temporal kernel function with a simple multilayer perceptron (MLP), which promotes stability [Romero et al. 2021 and more compressed parameters, but often significantly increases computational load.
Here, we introduce a method of parameterization of temporal kernels, named PLEIADES (PoLyno- mial Expansion In Adaptive Distributed Event-based Systems), that in many memory and computational costs compared to explicit convolutions. The design fairly modular, and be used drop-in replacement for any DD-like convolutional layers, allowing them to perform long temporal convolutions effectively. In fact, we augment previously proposed (1+2)D causal spatiotemporal network (Pei al. 2024] by replacing temporal kernels with this new polynomial parameterization. This new network architecture serves as the backbone for a wide range of online spatiotemporal tasks ranging from action recognition object detection.
reduce the
Even though our network can be used for any spatiotemporal data (e.g. videos captured with conventional cameras), we investigate in this work the performance of our network event-based based
Work done while Brainchip Inc.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

data (e.g. data captured by an event camera). Event cameras are sensors that generate outputs events [-1,+1} responding to optical changes in the scene's luminance [Gallego al. 2020], and can generate sparse data on an incredibly short time scale, usually at 1 microsecond. Event cameras can produce very rich temporal features, capturing subtle motion patterns, and allow for flexible adjustment of the effective frames per second (FPS) in our network by varying the timebin size. This makes it suitable to test networks with long temporal kernels sampled at different step sizes. For readers interested in the performance conventional camera data, we report preliminary results on the KITTI 2DOD task in Appendix [3.3]
The code for building the structured temporal kernels, along with pre-trained PLEIADES network for evaluation on the DVS128 dataset is available here: https://tthbu.ccm//eaaabee/ee
2 Related Work
2.1 Long Temporal Convolutions and Parameterization of Kernels
When training a neural network containing convolutions with long (temporal) kernels, it is usually not desirable to explicitly parameterize the kemeel values for each time step. First of all, the input data may not be uniformly sampled, meaning that the kernels need to be continuous in nature, making explicit parameterization impossible In this case, the kernel is treated as a mapping from an event timestamp to a kernel value, where its mapping is usually achieved via a simple MIP [Qi et al. 2017] Romero et al. 2021 Poli et al.] 2023). In cases where the input features are uniformly sampled, explicit parameterization of the values for each time step becomes possible in theory. However, certain regularization procedures need to be applied (Fu et al. 2023 in practice, otherwise the training may become unstable due to the large number of trainable weights. Storing all these weights may environments (for edge or mobile devices).
also be unfavorable in memory-constrainee
The seminal work proposing a memory encoding using orthogonal Legendre polynomials in a recurrent state-space model is the Legendre Memory Unit (L.MU) (Voelker al., 2019), where Legendre polynomials (a special case of Jacobi polynomials) are used. The HiPPO formalism (Gu et al. 2020) then generalized this to other orthogonal functions including Chebyshev polynomials, Laguerre polynomials, and Fourier modes. Later, this sparked a cornucopia of works interfacing with deep state-space models including S4 [Gu et al., 202 H3 [Fu al., 2022]. and Mamba [Gu and Dao 2023), achieving impressive results on a wide range of tasks from audio generation to language modeling. There are several common themes among these networks that PLEIADES differs from. First, these models typically only interface with 1D temporal data, and usually try to flatten high dimensional data into ID data before processing [Gu et al. [021a] Zhu et al. 2024]. with some exceptions [Nguyen et al., 2022]. Second, instead of explicitly performing Limte-winddw temporal convolutions, a running approximation of the effects of such convolutions is performed, essentially yielding a system with infinite impulse responses where the effective polynomial structures are distorted [Stöckel. [02T] Gu et al.) 2020). And in the more recent works, the polynomial structures are tenuously used only for initialization. but then made fully trainable. Finally, these networks mostly use an underlying depthwise structure [Howard et al. 2017) for long convolutions, which may limit the network capacity, albeit reducing the compute requirement of the network.
2.2 Spatiotemporal Networks
There are several classes of neural networks that can process spatiotemporal data (i.e. videos and event frames). For example, a class of networks combines components from spatial convolutional networks and recurrent networks, with the most prominent network being ConvLSTM (Shi et al. 2013 These types of models interface well with streaming spatiotemporal data, but are oftentimes difficult to train (as with recurrent networks in general). On the other hand, we have a class of easily trainable networks that perform (separable) spatiotemporal convolutions such as the R(2+1)D and P3D networks (Tran et al.) 20188, QQu et al., 2017]. but they were originally difficult to configure for online inference they do not assume causality. However, it is easy configure the temporal convolutional layers as causal during training, such that the network can perform efficient online inference with streaming data via the use of circular buffering (Pei et al. 2024) incorporating spike-based (Shrestha and Orchard 2018 or event-based [vvaov et al. 2022 processing.
It requires uncountable-infinn number of "weights" to explicitly parameterize continuous function.

---

2.3 Event-based Data and Networks
An event produced by an event camera can be succinctly represented as a tuple E = (p. z, y, t),, where p denotes the polarity, I and y are the horizontal and vertical pixel coordinates, and is the timestamp of the event. collection of events can then be expressed as {E,, E..... J. To feed event-based data into conventional neural networks, events are often discretized into uniform timebins yielding tensors generally shaped (2, H. W, T), where H & W are the height width of the sensor/layer and T is time period. Different discretizations have been explored in the past. The simplest approach is to count the number of events in each timebin [Maqueda et al. 2018]. Others include replacing each event with fixed or trainable kernel [Zhu et al., [2019] Gehrig al. 2019] before evaluating the contribution of that kernel to given bin. Here, we use direct binning and event-volume binning methods PPei et al., 2024]. yielding the 4d tensor (2, H, W) to our network, noting that we retain the polarity unlike previous works (Zhu et al. 2019].
The most popular class of event-based networks is spiking neural networks, which generate spikes {0, +1} with continuous timestamps at each neuron, usually described with predetermined fixed internal dynamics [Gerstner and Kistler 2002 Gerstner et al., 20144. These networks can be efficient during inference, as typically they only need to propagate 1-bit signals, but they are also difficult to train without specialized techniques to efficiently simulate the neural dynamics and ameliorate the spiking behaviors. The SLAYER model [Shrestha and Orchard 2018 computes the neuron response using a kernel from the spike response model (SRM), taking only limited forms such as decaying exponent & alpha function. The kernel is used to temporally convolve the input spikes. PLEIADES generalizes the impulse-response kernel of SLAYER by making the kernel fully trainable convolution kemnl,, thus fully adapted to input and network structure.
A recent line of work (in parallel to ours) is to interface event-based processing with deep state- space modeling [Schöne et al. 2024 Soydan et al., 2024), while we here still retain the orthogonal polynomial structures (see Section 2.1). Other works have proposed training using differentiable functions, surrogate gradients, to bridge the discontinuous gap of spikes to use backpropagation [Neftci al. 2019], so that spiking networks can be trained like standard neural networks.
3 Temporal Convolutions with Polynomials
In this section, we discuss: 1) how the temporal kernels are generated from weighted sums of polynomial basis functions; 2) how the temporal kernels are discretized in timebins; 3) how the convolution with the input feature tensor can be optimized by changing the order of operations. In the following, we index the input channel with c, the output channel with d, the polynomial degree or basis with n, the spatial dimensions with æ and y, the input timestamp with I, the output timestamp with f', and the temporal kernel timestamp with T.
3.1 Building temporal kernels from orthogonal polynomials
Jacobi polynomials p(a.9) function:
(T) are class of polynomials that are orthogonal relative to a weighting
h 3) [" pla..) (T) m (T) - 71 7) dr 5pm=
where 8pm is the Kronecker delta, which is 1, or O, if = m, or n + m, respectively, hence establishing the orthogonality condition. A (a..)) is a normalization constant that we ignore. A continuous function can be approximated by taking the weighted sum of these polynomials up to a given degree N, where the weighting coefficients (10, 7I.... } are the trainable network parameters.
When this parameterization is used in ID convolutional layer typically involving multiple input and output channels, then naturally we require a set of coefficients for each pairing of input and output channels. More formally, if we index the input channels with and the output channels with d, then the continuous kernel connecting c d can be expressed as
N
ked (T)(T)) = Σ
Yod p(a..) n=0
(I)
3

---

Training with such structured temporal kernlls restricts expressivity compared to temporal kernels parametrized with weight parameter each timebin by not permitting discrete jumps across bins. However, there are several key advantages in using structured continuous kernels that largely over- compensate for the reduced expressivity. First, this implicit parameterization allows for resampling of the kernels during discretization,, meaning that the network can interface with data sampled at different rates without additional fine-tuning (see Section 3.2). Second, having a functional basis will allow an intermediate subspace to store feature projection, which can sometimes improve memory and computational efficiency (see Section 3.3). Finally, since Jacobi polynomial basis associated with underlying StuvllLouville differential equation, this injects physical inductive biases into our network, making the training more stable and guiding it to a better optimum (see Section 5.1 for an empirical proof).
3.2 Discretization of the convolution kernels
In the current implementation of our network, which interfaces with time binned inputs, a discretiza- tion of the temporal kernels needed. One method is integrating the temporal kernels over the time bins of interest.
We start by defining the antiderivative of the temporal kernels
Kor(T)= = /-] k'd(T') dr' =
N Vod (aamPlaan) (7') dr'
n=0
(2)
N
Vednn pla.9) n+1
(T) const,
1=0
where the constant term does not depend on and can be ignored. To evaluate the integral of kea in the time bin (To, + Ar], that denote the discrete kea[To]. we take the difference
Rod[ro] Kea(To + AT) - Kea(To)
N
pla.t) (To + Ar) pla.0) (TO)
%od
(n+11
(3)
n=0
Σ
Tod
(0
[TO].
n=0
where P is the appropriately defined discrete polynomials, to obtain the discrete form of Eq. Eq. be considered a generalized matrix multiplication where the dimension n (the polynomial basis dimension) is contracted, see Section 3.3 Fig. provides a schematic representation of the operations to generate discretized temporal kernels for multiple channels.
Under this discretization scheme, it is very easy to resample the continuous temporal kernels (either downsampling or upsampling), interface with data sampled at arbitrary rates, i.e. arbitrary bin sizes for event-based data. The network can be trained at a given step size Ar, but adapted to perform inference different rate (either faster or slower), without any additional tuning. The discretized polynomial basis can be regenerated using the equations above with new Ar, and everything else in the network can remain unchanged B
3.3 Optimal order of operations
Given discretized kernels, the notation is simplified by employing the Einstein notation einsum, where repeating indices are summed over by convention. The equation above is rewritten as
Tratt
Yedn nr
"This true if the scale of input data invariant under resampling. For event- event-based data accumulated into bins. the contribution, that is, the integral over time of an event must remain independent of the discretization; the bin values have to be rescaled by a factor reciprocal to the new bin size relative to the original one. For example, if the bin size is doubled, then the bin values need to be appropriately halved.

---

discretized values
discretized values
C
coefficients
0.46
0.20
channels
-0.69
-0.69
-0.58
0.73
X
channels
0.20
0.42
-0.96 0.94
Figure 1: Generating discrete temporal kernels for multiple channels, based trainable coefficients and fixed basis orthogonal polynomials. Here, temporal kernels one per channel, is generated from basis polynomials discretized over timebins. The shaded areas represent the discretized polynomial values. The kernel coefficients may be organized as 3 x matrix, and the discretized basis polynomials may be organized of coefficients) then yields the final discretized kernels for the channels discretized over 5 timebins 3 matrix.
x 5 matrix. The matrix multiplication of the two (contraction
where the repeating index n is summed over (or contracted) on the right-hand side, corresponding to summing over the polynomial basis. See Appendix [.. for a detailed description of the contraction rules. The temporal convolution of a kernel kijr with a spatiotemporal input feature tensor u. to obtain the output y discrete time t', position (3r, y) and output channel dl is written
Vdzyy! Vodn Mate Veryts
where (P) is the convolution operator matrix, sparse Toeplitz matrix generated from P (see Aopendix/AA3) If depthwise convolution is performed [Howard et al. 2017]. the equation simplifies to Veryt' Yen Mate (P) Vesyt- Here, only one channel index e needed, as the connections are parallel thus the input and output channels do not mix. The kernels are assumed to be separable into spatial and temporal convolutional kernels; thus, the temporal indices do not interact with the spatial indices and y. Each temporal kernel is applied separately to each spatial bin.
All einsum operations are associative and commutative so we have full freedom over the order of contractions. For example, we can first generate the temporal kernels from the orthogonal polynomials, then perform the convolutions with the input features (the typical order of operations. from left to right). However, equally valid, we can also first convolve the basis polynomials with the input features separately, then weigh and accumulate these results using the polynomial coefficients. This can be written as Yedn (Mate Meayt) = "eadn Terput' Odayy' in einsum form, where represents the intermediate projections. Note that this freedom of ordering of contractions is not possible for unstructured temporal kernels since there is no intermediate basis on which to project anything.
In practice, we select the contraction path optimize memory computational usage [Gray and Kourtis 2021]. depending on the training hardware and cost considerations. Memory and computational costs can be calculated for any contraction path, given the dimensions of the contraction indices (tensor shapes) (see Appendix [A. for cost calculations). By leveraging the opt_eine einssm library, with tailored adjustment of its cost-estimation rules (see Appendix A.3). can automatically determine an optimal contraction path. Further refinements and applications are discussed in the Pei al., 2025).
related literature (Gray and Kourtis 202T Peil 2025
The coefficients are the trainable parameters, and form a compressed representation of the binned temporal kernel Kedr when the polynomial expansion is less than the number bins n & т. End- to-end networks with temporal layers convolving with PLEIADES kernels can leverage standard optimization methods such as backpropagatior
'Einstein summation (einsum) restores permutation invariance to otherwise non-commutative matrix multi- plications tensor contractions by explicitly labeling contraction indices. An optimal forward contraction path automatically implies an optimal backward path, so standard autodif- ferentiation suffices.
5

---

4 Network Architecture
Tempord block
Spatial block
oulpul
Ul conv
DWS
lul
DW5
event pakarily
spatio-temporaal block
coTV
corvv
conv
8
16
32
48
64
80
96
112
128
256
236
heght
I0.1111
(1.3.3
110.111
[1.3.3] 3)
0.1.11
[1.3.3]
10.111
11.3.3]
0.1.1
11.3.3)
10.11
(1..3) [1111]
widlt
Figure 2: A representative network used for eye tracking. The backbone consists of 5 spatiotemporal blocks, Full convolutions are denoted by darker blue blocks (full conv), depthwise-separabbl convolu- tion by lighter blocks (DWS conv). The detection head is inspired by CenterNet, with the modification that the 3 x 3 convolution is made depthwise-separabe and a temporal layer is prepended to it.
The main network block is spatiotemporal convolution block, factorized as a (1+2)D convolution. A ID temporal convolution performed on each spatial bin (pixel) followed by 2D spatial convolution on each temporal frame; this is similar to R(2+1)D convolution block [Tran et al. 2018), and
the (1+2)D network for online eye tracking [Pei et al. 2024). Each of the temporal and spatial convolutional layers may be factorized as a depthwise separable (DWS) layer (Howard et al. 2017 to further reduce computational costs. For every temporal kernel (every channel pairing, every layer, and every network variant),
use -0225 and 3 = -.25 for the Jacobi polynomial basis functions, with degrees up to 4, which were determined empirically to be suitable for all of our experiments See Appendix 1.1.2 for details.
As additional notes, the choice of (a, 5) does not affect the expressivity of the temporal kernels, and all choices of (a. yield polynomial bases spanning the same space. What it really influences here is the training dynamies, and full mathematical justification of why a / 0.25 yields superior training dynamics is beyond the scope of this work D Nevertheless, there is justification at an informal level. The choice of = 3 = 0.25 is an "intermediate" hetween Legendre polynomials (a =a= 3 0) and Chebyshev polynomials (a B -055), which balances the benefits of both. For instance, Legendre polynomials are more balanced, but slower to represent sharp boundary transitions; Chebyshev polynomials are better for representing edge-localized patterns, but more numerically unstable.
Key ancillary design decisions (beyond polynomial kernels) are studied more thoroughly in our previous work Pei et al. 2024
Strict causality: All iperations-iicludiinn temporal convolutions are causal, enabling low-latency online inference.
Hybrid normalization: After every temporal convolution, we perform a causal Group Normalization with groups 4. And after every spatial convolution, we perform a Batch Normalization. This hybrid strategy is inspired by Gordon et al. [2019].
Lightweight activations: ReLU after every convolution layer and inside each DWS layer keeps implementation simple for edge devices and encourages activation sparsity.
For tasks requiring object tracking or detection (see Sections 5.2 and 5.3) 1, we attach a temporally smoothed CenterNet detection head to the backbone (see Fig. 2 consisting of DWS temporal layer, 3 DWS spatial layer, and a final pointwise layer [Zhou al. [011]]. with ReLU activations in between. Since our backbone is already spatiotemporal in nature and capable of capturing long- range temporal correlations, we do not use any additional recurrent heads (e.g. ConvLSTMs) or temporal- bansd loss functions [Perot et al. 2020
'Other choices of and 3 were also performant, but slightly worse than this particular choice. However, the degree choice is relatively important. Note that degree 10 would make kernel essentially "free". which would actually harm performance, especially in the low-latency regime (see Section 5.1 5.1 and Appendix (....)... This requires extensions theories of neural tangent kernel and dynamic mean field theory etc.

---

5 Experiments
We conduct experiments standard computer vision tasks with event-based datasets. For all bascline experiments, we preprocess the event data into 4d tensors of shape (2, H, T"), with the 2 polarity channels retained. General details of data and training pipelines are given in Appendix [ With the exception of the Prophesee GEN4 experiments (Section we run 25 trials for each experiment and report the mean and standard error (which assumes normal distribution of noise).
5.1 DVS128 Hand Gesture Recognition
The DVS128 dataset (CC BY 4.0) contains recordings of 10 hand gesture classes performed by different subjects [Amir et al. 2017]. recorded with a 128 x 128 dynamic vision sensor (DVS) camera. We use simple backbone consisting of spatiotemporal blocks. The network architecture is almost the same as that shown in Fig. 2 with the exception that the detection head is replaced by spatial global average pooling layer followed by simple 2-layer MLP to produce classification logits (technically a pointwise Conv ID layer during training). The output produces raw predictions at 10 ms intervals, which already by themselves are surprisingly high-quality. With additional output filtering on the network predictions, the test accuracy can be pushed to 100% (see Table [ In addition, we compare the PLEIADES network with counterpart that uses unstructured temporal kernels, or simply a Conv(1+2)D network (Pei et al., 2024). and find that PLEIADES performs better with smaller number of parameters (due to polynomial compression).
Table I: The raw 10-class test accuracy of several networks on the DVS128 dataset. Output filtering is performed only the networks indicated by an asterisk. PLEIADES is evaluated (mean & standard error) only after all temporal layers have processed non-zero, valid frames, resulting in an inherent warm-up lateney of 0.44 seconds (see Se.t 0.15 second majority filter on raw PLEIADES outputs attains 100% accuracy with minimal compute overhead, at the cost of added latency.
Model
Accuracy
Parameters
MACs / see
Conv(1+2)D
99.17
196 K
0,429 B
ANN-Rollouts Kugele et al. 2020
97.16
500 K
10.4 B
TrueNorth CNN [Amir et al. 2017
96.59
18 M
SLAYER [Shrestha and Orchard
2018
93.64
PLEIADES PLEIADES filtering*
99.59 (0.02)
192 K
0.499 B
100.00 (0.00)
192 K
0.499 B
Prior work lacks a standardized evaluation protocol: some models operate online, whereas others process entire clips before predicting. We therefore report uccuracy-latency trade-off curves for each PLEIADES variant, yielding multiple Pareto frontiers. Latency defined as the number of event frames observed since the sequence starts, multiplied by the bin size. Enforcing non-zero inputs at every temporal layer imposes an intrinsic delay of latency L(k 1) Ar 450 ms in our baseline Zero Sero-padding unseen frames removes this warm-up, letting the model predict after only a few bins at slight cost in early-time accuracy. If latency is secondary, we can instead post-filter the outputs- e.g., with causal majority or exponential filter [Amir et al.] [017] to gain a few accuracy points, at the expense of extra delay equal to the filter length. Appendix B. details these latency-accuracy trade-offs.
We test two mechanisms to get the network respond faster (Fig. 3 left). (i) Smaller bins: keeping the temporal kernel length fixed at k 10 timebins, we reduce the bin size AT from 10 ms (baseline) to 5 ms, halving the effective temporal window. (ii) Causal prefix masking: during training we randomly drop an initial block of forcingg the model to depend on more recent inputs and shortening its functional response window 9 Accuracy- latency traces for both strategies are plotted in the left panel; implementation details of the masking augmentation appear in Appendix B.I
Next, we compare the previous results with keeping the kernel window constant 100 ms and changing the bin sizes. Starting from the network trained at AT = 10 ms. we resample the temporal
'Here, we used layers (L), kernel size (k) of 10, and time step (AT) of 10 ms. "This does not alter theoretical latency but improves accuracy when fewer frames are available; it can, however, slightly hurt accuracy when the full sequence is provided.
7

---

Accuracy (%)
80-
Accuracy (%)
80
50
10mc CHP-ERN
50
20ms stepsine (nesempled)
10ms slapsize
Sen stoodize (recomplcd)
karmels
10
20
30
50
70
100
200
300
500 700
1000
100
200
300
400
500
600 700
1000
Latency (ms)
Latency (ms)
Figure 3: (Left) Accuracy vs. latency curves for different PLEIADES variants with changing temporal window determined as a kernel size of 10 timebins but with different bin sizes on the DVS128 dataset. A masking augmentation is optionally used randomly mask out the starting frames of dataset segments during training in order to stimulate faster responses in the network. (Right) Accuracy vs. latency curves for different PLEIADES variants with fixed temporal window of 100 ms for each temporal layer, but having different bin sizes. The benchmark network trained with kernel size of 10 timebins and 10 step size, and the other variants are resampled without additional fine-tuning, network variant trained without PLEIADES structured temporal kernel is also displayed baseline reference (free kernels).
kernels without weight tuning by re-discretizing the polynomial bases (Sec. 3.2) and re-binning the
events ÅT E {5, 20} ms, upsampling and downsampling, respectively (Fig right). The number of timesbins, k, is chosen so that k Ar 100 ms for k 20 and k: timebins, respectively, preserving the size of the time window. The resulting eccuracy-latency curves remain largely unchanged. For reference, the same plot includes Conv(1+2)D baseline with unconstrained (free) non-PLEIADES kernels.
5.2 AIS2024 3ET+ Event-based Eye Tracking
Table 2: The 10-pixel, 5-pixel, and 3-pixel tolerances for the CVPR 2024 AIS eye tracking challenge. The performances of other models are extracted from [Wang et al. 2024].
Model
p10
p5
p3
Parameters
MambaPupil
99.42
97.05
90.73
CETM
99.26
96.31
83.83
7.1M
(I2)(
99.00
97.97
94.58
1.1M
ERVT
98,21
94.94
87.26
150K
PEPNet
97.95
80.67
49.08
640K
PLEIADES + CenterNet 99.58 (0.03)
97.95 (0.03)
94.94 (0.04)
277K
The performance of our network on the 3ET+ dataset, introduced in the CVPR AIS 2024 eye-tracking challenge [Wang et al. 2024 investigated [0] We adopt the DVS 128 hand-gesture backbone, adjusting the timebin to 5 ms. The 2-layer hhad is replaced by CenterNet-ste detector and loss, following (Pei et al. 2024), but we predict only pupil centre points, omitting bounding-box size. See Fig. 2 for the architecture.

---

Table 3: Performance of PLEIADES with CenterNet detection head versus the baselines in Peroo
al. 2020) and more recent models. Frame rate (FPS) is derived from the bin size 50 ms for the baselines and 10 ms for ours.
Model
mAP
Parameters
MACs sec
FPS
RED [Perot et al. 2020)
0.43
24.1 M
20
Gray-Retina Perot et al. 2020
0.43
32.8 M
2060 B
20
S5-ViT-B [Zubic et al. 2024
0.478
18.2 M
20
GET-T
[Png et al. 2023
0.484
21.9 M
PLEIADES CenterNet
0.556
0.576 M
122.5 B
100
5.3 Prophesee GEN4 Roadscene Object Detection
The Prophesee GEN4 Dataset road-scene object detection dataset collected with megapixel event camera [Perot et al. 2020 I1 The dataset spans around 14 hours of rural/urban driving under day and night conditions. While seven classes are annotated, we follow the original protocol [Perot et al.) 2020] and report mAP only for cars and pedestrians. Our detector uses spatiotemporal hourglass backbone with 10 ms timebin (Ar 10 ms) and a CenterNet detection head (Sec. 4): no non-max max suppression (NMS) is applied, as CenterNet's 's design and the model's 's temporal coherence already suppress spurious boxes. Architectural and training specifics appear in Appendix [B.2 [.2] Remarkably,
our model trains and infers at 100 Hz (10 ms bins) with minimal memory and compute overhead, avoiding the steep accuracy drop typically reported for tighffreuency inference [Zubic et al., (202)..
6 Limitations
key limitation of our design is the memory overhead incurred by the finite-window temporal filters: each layer must buffer the most recent & feature maps, so the cost grows linearly with the kernel length k and multiplicatively with spatial resolution. On high-resolution streams or longer kernels this cache can dominate the footprint, hindering deployment edge devices.
Fortunately, the polynomial structure our temporal kernels offers a principled workaround. Instead of retaining all & frames, can maintain a compact set of online basis projections - inner products between the incoming features and fixed polynomial basis functions - updated recursively each time step [Stöckel 1 Gu al. 2020). These running coefficients play the role of hidden states in recurrent neural networks: to obtain the layer output we need only point-wise multiplication between the coefficients and the learned kernel weights, followed by the usual spatial convolution. Conceptually, this is the forward direction of the convolution thcorem - replacing convolution with product in an appropriate transform domain.
This reformulation collapses the memory requirement from O(HHW) to O(nHW), where n < the number of basis functions, and aligns our model with recent deep state-space architectures Guu jet al. 2021bla). Integrating such recurrent updates into future variants could substantially reduce memory while retaining the long temporal context that finite-window convolutions provide.
7 Conclusion
We presented PLEIADES, fully-causal spatiotemporal architecture whose temporal filters expressed as linear combinations of orthogonal polynomials. This structural choice gives the model analytically controllable receptive windows and yields state-of-the-art performance across all event- based vision benchmarks considered, while remaining markedly robust to time-step resampling-nu additional fine-tuning was required when the bin size was halved or doubled.
Publicyy available on Kaggle; in the absence a specific license, Kaggle's standard terms apply. The challenge organizers have confirmed that our use is permitted. !l Prophesee has allowed using the data in academic context given proper citation, which have provided.

---

Even in its present vanilla CNN instantiation, PLEIADES is exceptionally lean: it stores only the polynomial coefficients and small circular buffers, resulting in a memory and FLOP footprint well below contemporary counterparts. Nevertheless, two avenues could unlock further gains:
-ppasati regularization: Event streams are naturally sparse; injecting interme- diate loss terms that penalize dense activations could cut energy consumption and latency without sacrificing accuracy.
Spiking conversion:
The orthogonal-polynoiin filters implicitly realize high-order linear dynamics. Re-interpreting these filters as dynamical synapses would let us swap conventional ReLUs for analytically tractable spiking units whose responses exceed the expressiveness of standard leaky integrate-and-fir neurons. Such a reformulation would align PLEIADES with neuromorphic hardware, marrying convolutionall-eetwer training pipelines with the event-driven efficiency of spiking execution.
Taken together, these properties position PLEIADES as versatile foundation for future low-latency, low-power event- based perception systems, spanning both conventional GPUs and emerging neuro- morphic accelerators.
Acknowledgments and Disclosure of Funding
We would like to acknowledge Nolan Ardolino, Kristofor Carlson, and Anup Varase (listed in alphabetical order) for discussing ideas and offering insights for this project. would also like to thank Daniel Endraws for performing quantization studies on the PLEIADES network, and Sasskia Brüers for help with producing the figures.
References
Amon Amir, Brian Taba, David Berg. Timothy Melano, Jeffrey McKinstry, Carmelo Di Nolfo, Tapan Nayak, Alexander Andreopoulos, Guillaume Garreau, Marcela Mendoza, et al. A low power, fully event-based gesture recognition system. In Proceedings of the IEFE conference computer vision and patter recognition, pages 7243-7252, 2017.
Daniel Y Fu, Tri Dao, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré, Hungry Hungry Hippos: Towards language modeling with state space models. arXiv preprint arXiv:2212.14052. 2022.
Daniel Y Fu, Elliot L Epstein, Eric Nguyen, Armin W Thomas, Michael Zhang, Tri Dao, Atri Rudra, and Christopher Ré. Simple hardware-efficient long convolutions for sequence modeling. In International Conference on Machine Learning, pages 10373-10391, PMLR, 2023.
Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taha, Andrea Censi, Stefan Leutenegger, Andrew J Davison, Jörg Conradt, Kostas Danilidis, et al. Event-based vision: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(1):154-11 2020.
Daniel Gehrig, Antonio Loquercio, Konstantinos G Derpanis, and Davide Scaramuzza. End-to-end learning of representations for asynchronous event-based data. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5633-5643, 2019.
Wulfram Gerstner and Werner M. Kistler. Spiking Neuron Models: Single Neurons, Populations, Plasticity. Cambridge University Press, 2002.
Wulfram Gerstner, Wemer M Kistler, Richard Naud, and Liam Paninski. Neuronal dynamics: From single neurons to networks and models of cognition. Cambridge University Press, 2014.
Ariel Gordon, Hanhan Li, Rico Jonschkowski, and Anelia Angelova. Depth from videos in the wild: Unsu- pervised monocular depth learning from unknown cameras. In Proceedings of the IEEF/CV International Conference on Computer Vision, pages 8977-8986, 2019.
Johnnie Gray and Stefanos Kourtis. Hyper-optimized tensor network contraction. Quantum, 5:410, 2021.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.0000 0000522....... 2023.
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33:1474- 1487, 2020.
10

---

Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.003% 2021a.
Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao. Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in neural information processing systems, 34:572-585, 2021b.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.0486/ 2017.
Dmitry Ivanov, Aleksandr Chezhegov, and Denis Larionov. Neuromorphic artificial intelligencce systems. Frontiers in Neuroscience, 16:959626, 2022.
Alexander Kugele, Thomas Pfeil. Michael Pfeiffer, and Elisabetta Chicca. Efficient processing of spatio-temporal data streams with spiking neural networks. Frontiers in neuroxcience, 14:512192, 2020.
Colin Lea, Rene Vidal, Austin Reiter, and Gregory D Hager. Temporal convolutional networks: A unified approach to action segmentation. Computer Vision ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15- 16, 2016, Proceedings, Part III 14, pages 47 47-54 54. Springer, 2016.
Colin Lea, Michael D Flynn, Rene Vidal, Austin Reiter, and Gregory D Hager. Temporal convolutional networks for action segmentation and detection. In proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 156-165, 2017.
Ana Maqueda, Antonio Loquercio, Guillermo Gallego, Narciso Gareía, and Davide Scaramuzza. Event-based vision meets deep learning on steering prediction for self-driving cars. In Proceedings of the IEEE conference on computer vision and pultern recognition, pages 5419-5427, 2018.
Emre O Neftei, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient based optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51-63. 2019.
Eric Nguyen, Karan Goel, Albert Gu, Gordon W Downs, Precy Shah, Tri Dao, Stephen A Baccus, and Christopher Ré. S4ND: Modeling images and videos multidimensional signals using state spaces. arXiv preprint arXiv:2210.06588. 2022.
Ashutosh Pandey and DeLiang Wang. Tenn: Temporal convolutional neural network for real-time speech enhancement the time domain. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6875-6879. IEEE, 2019.
Yan Ru Pei. Let SSMs be ConvNets: State-space modeling with optimal tensor contractions. arXiv preprint axXiv:2501.13256 2025.
Yan Ru Pei. Sasskia Brüers, Sébastien Crouzet, Douglas McLelland, and Olivier Coenen. A Lightweight Spatiotemporal Network Online Eye Tracking with Event Camera. Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition Workshops, 2024.
Yan Ru Pei, Ritik Shrivastava, and FNU Sidharth. Optimized real-time speech enhancement with deep ssms on raw audio. In Proc. Interspeech 2025, pages 51-55, 2025.
Yansong Peng. Yueyi Zhang, Zhiwei Xiong, Xiaoyan Sun, and Feng Wu. Get: Group event transformer for event-based vision. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6038-6048, 2023.
Etienne Perot, Pierre Tournemire, Davide Nitti, Jonathan Masci, and Amos Sironi, Learning to detect objects with a megapixel event camera. Advances in Neural Information Processing Systems, 33:16639-1665 16652, 2020.
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. In International Conference on Machine Learning, pages 28043-28078. PMLR, 2023.
Charles R Qi. Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660. 2017.
Zhaofan Qiu, Ting Yao, and Tao Mei. Learning temporll representation with pseudo-3d residual networks. In proceedings of the IEEE International Conference on Computer Vision, pages 5533-554) 2017.
11

---

David W Romero. Anna Kuzina, Erik J Bekkers, Jakub M Tomezak, and Mark Hoogendoorn. CKConv: Continuous kemel convolution for sequential data. arXiv preprini arXiv:2102.02611, 2021.
Mark Schöne, Yash Bhisikar, Karan Bania, Khaleelulla Khan Nazeer, Christian Mayr, Anand Subramoney. and David Kappel. Stream: A universal state-space model for sparse geometric data. arXiv preprint arXiv:2411.1263, 2024.
Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung. Wai-Kin Wong, and Wang-chun Woo. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. Advances neural information processing systems, 28, 2015.
Sumit B Shrestha Garrick Orchard. SLAYER: Spike layer error reassignment time, Advances in neural information processing systems, 31, 2018.
Taylan Soydan, Nikola Zubić, Nico Messikommer, Siddhartha Mishra, and Davide Scaramuzza, S7: Selective and simplified state space layers for sequence modeling. arXiv preprint arXiv:2410.03466 2024.
Andreas Stöckel. Discrete function bases convolutional neural networks. arXiv preprint arXiv:2103.05009 2021.
Du Tran, Heng Wang. Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference on Computer Vision and Puttern Recognition, pages 1450-6459, 2018.
Aaron Voelker, Ivana Kajié, and Chris Eliasmith. Legendre Memory Units: Continuous-ting representation in recurrent neural networks. Advances in neural information processing systems, 32, 2019.
Zuowen Wang, Chang Gao, Zongwei Wu, Marcos V. Conde, Radu Timofte, Shih-Chii Liu, Qinyu Chen, et al. Event-Based Eye Tracking. AIS 2024 Challenge Survey. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2024.
Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects as points. arXiv preprint 074700904.078550 2019.
Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Daniliidis. Unsupervised event-based learning of optical flow, depth, and egomotion. In Proceedings the Pattern Recognition, pages 989 997. 2019.
the IEEE/CVF Conference Computer Vision and
Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision Mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09411 2024.
Nikola Zubic, Mathias Gehrig, Davide Scaramuzza. State space models event cameras. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 5819-5828, 2024.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: As outlined in the abstract, we introduced network constructed with orthog- onal polynomials adapted for event-based data, formulated in Sections and 4 The empirical results given in the abstract are also presented in detail in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In Section we addressed limitation of the network, which is the memory cost of maintaining a circular buffer. A solution is proposed for future investigations, which is to integrate recent deep state-space modeling methods into our network.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
13

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: There are two main theoretical components to this work. First, we show the discretization of our temporal kernels in Eq. which allows for natural temporal resampling based on the input bin sizes. The derivation is based on the anti-derivative formulation and is mathematically rigorous. Second, we have the optimal order of contraction as discussed in Section 3.3 and given a detailed treatment in Appendix [A] The derivation based on the assumption that optimality is based on theoretical memory and computational costs, and does not account for HW -peccicciicc characters such as limiting assumption is acknowledged in the main text.
This potentially
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section and Appendix B provide full experimental details for reproducing our results. Our code will also be made available upon as supplementary material, and will be open-sourced upon potential acceptance of the paper.
and
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
14

---

(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All the datasets used in this study are open sourced. Section and Appendix B provide full experimental details for reproducing results. Our code will also be made available upon as supplementary material, and will be open-sourced upon potential acceptance of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (kttps://nips.com cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips /nipps.cc/pbllle.. httppi/c//gideeeeoeeeeiieeeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The full training and testing details are given in Appendix B The splits we use are also the standard splits for the corresponding datasets.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
15

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: With the exception of the Prophesee GEN4 experiment, we run 25 trials for each experiment, and report the mean and standard error. The Prophesee GEN4 experiment is computationally expensive,
only afford to one trial. But with training large models on large datasets, is generally expected that the experiment noise would be rather small.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots. The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The GPU Type, training batch sizes (and the corresponding VRAM), and the total training wall times are all listed in Appendix B
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https://neurips httpr//p....cbubbl//ccccccccceee
Answer: [Yes]
Justification: The authors have carefully reviewed every item in the Code of Ethics, and confirmed that work conforms to them in every aspect.
Guidelines:
16

---

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed. The model that will be released a small classification and detection model and is non-generative nature. is very unlikely that the model will be used for malicious purposes.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
17

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cite and mention the license of all the datasets used in the study. In addition, when reporting the results of existing models for comparison, we also properly cite the corresponding work. For unknown licenses, contacted the owners directly for their permission to use their assets in our work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com datasees has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will upload the model architecture and pre-trained weights as supple- mentary material. We also plan open-source them upon potential paper acceptance. Documentation is included via a README. file detailing setup and usage. In addition, custom license will be included in the open-source release.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
18

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We do not use LLMs at all the core method development in this research.
Guidelines:
The answer NA means that the method development in this research does not involve L.MMs as any important, original, or non-standard components.
Please refer to our LLM policy /attps://neur// cc/ /ct////ernncee/2225////////////e for what should or should not be described.
19

---

A Optimal Contraction Order Memory and Compute
A.1 The Rules of Einsum
The rules of contracting an einsum expression can be summarized as follows:
At every contraction step, any two operands can be contracted together, as the einsum operation is associative and commutative.
For any indices appearing in the two contracted operands but not the output and other
operands, these indices can be summed away for the intermediate contraction results.
Any ordering of contractions (or contraction path) following these rules is guaranteed to yield equivalent results.
A simple example is multiplying three matrices together, or Da = Bja In the first stage, we can first choose to contract Aij and Bjs. which would yield an intermediate result of Max where the index j is contracted away as it does not appear in the output DR. In the second stage, we then contract Ma and Cki to arrive at the output Da.
We can also choose to do the contractions in any other order, and the result will remain the same. As more extreme example, we can even perform outer product first Majki Aig Ckt, noting that cannot contract away the j and l: indices yet as they appear in Byk still. The contractions of j and then need to be left to the second stage contraction, Da Mitu Intuitively, we feel that this is a very suboptimal way of doing the multiplication of three matrices, and we can formalize why this is by looking at the memory and computational complexities of performing a contraction.
A.2 Memory and Compute Requirements of Contraction
If we assume that we are not performing any kernel fusion, and explicitly materializing and saving all intermediate tensors for backpropagation, then the extra memory and compute incurred by each contraction step is as follows:
The memory needed to store the intermediate result simply the size of the tensor, or equivalently the product of the sizes of its indices.
The compute needed to evaluate the intermediate result is the product of the sizes of all indices involved in the contraction (repeated indices are counted only once).
Again, we can use the Da = Aij where we assume that the index sizes to be i, j. k. and I. Doing the first stage contraction Md = A., By₁ will require ik units of extra memory and ijk units of compute, and doing the second stage contraction Da = M1 Cи will require no extra memory besides that for storing the output and ikl units of compute. This gives us total extra memory requirement of ik units and total compute requirement of ijk: + ikl units.
On the other hand, if we perform the outer product Mijki = A:y first, this will require ijkl units of extra memory and ijkl units of compute. The second stage contraction Da = Mista will require ijkl units of compute. Therefore, the total memory requirement of this contraction path is djkl units and the total compute requirement is 2ijkl units, both being significantly worse than the first contraction path, regardless of the sizes of the tensors.
A more subtle example (the only remaining contraction path) is contracting the operands from back to front, which we verify requires total memory of jl units and a total compute of jkl izl units. So this is only more memory optimal than the first contraction path if ji < ik, and more compute optimal if jkl + isl < ijk ikl, which may not be immediately obvious from inspection as the optimality now depends on the sizes of the tensors.
Note that it is assumed that every tensor involved in the einsum expression requires gradient from backpropagation, in the context of neural network training. This is why we identify the size of each intermediate result as "additional memory". as they need to be stored as tensors used for gradient computation. addition, it is not difficult to see that for einsum operations, the computational costs
From here on, we will consistently use this abuse of notation where the same letter will be used to denote both the index and the corresponding dimensional size of that index.
20

---

required for backpropagation are exactly double that of the forward computation. Therefore, we only need consider the forward pass of the einsum expression, which is what we have been doing.
Importantly, note that this argument for memory and computational costs of gradient computations is only true under the assumption of reverse-mode automatic differentiation (backpropagation), which is what is used in almost all modern machine learning frameworks. In other words, we do not consider more general forms of automatic differentiation such as the forward-mode variant. Another important note is that in practice if the operations are memory-bound, then the computational cost estimates may not be useful for training time estimation.
A.3 Convolution with Parameterized Temporal Kernel
Recall in the main text that the equation for performing a full convolution with a polynomially parameterized temporal kernel is
ydayt Mexxyt Ydne-Mnert
(5)
where the convolution operator tensor (P) based on the discretized polynomial basis functions P given by the following Toeplitz matrix for each degree basis (assuming that kernel size is with the discretized timestamps being {To, T2, 73, ):
M(P)
P
0
Pln
P 70
0
0
P(
P Pm
P[To]
0
(6)
P(ra]
P [(r2]
P(m]
P mr
0
Pla
P(ral
P(
[m
P(rol
P[Ta]
P[ra]
P[ra]
P(ri]
P(ro]
where for valid-type convolution omit the first four of the matrix.
Note that we need to make two modifications the memory and compute calculation rules in Section to adapt for the sparse and Toeplitz structure of the convolution matrix M. First is that the memory required for storing any tensor containing both t' is guaranteed to be some form of convolution kernel, so it should only contribute memory factor of N, (the kernel size) instead of Ne, Second is that any contraction two tensors with one containing | and the other containing t, t' is guaranteed to be temporal convolution, should similarly contribute compute factor of Ne for valid-type convolutions and N, for same-type convolutions. For our implementation, monkey patch these modifications into the opt_einsul package used to provide memory and FLOP estimations of einsum expressions.
Table 4: The memory and compute requirements for each possible contraction path, where we are using a slight abuse of notation by allowing the index to represent the dimensional size of that index the "extra memory" and "total compute" columns. The initial equation cxyt, dnc, ntt 't't t is always assumed. We assume here that N, = N₂ for simplicity (equivalent to performing same-type convolutions).
Contraction Path
Extra Memory
Total Compute
dnxyt,nt tt -> dxyt? dnæyt
nzyt + dr) nayt(et CT + de) dner deaylT
ncxyt' dnc -> dxyt' neayt
>> cxyt,dct't -> dxyt? cryl
Following the prescription given above for calculating the memory and computational requirements for performing contractions, we summarize the requirements of each contraction path for the temporal convolution in Table We only consider the case for full convolutions, but the case for depthwise convolutions is analogous.
The first contraction path first contracts the input with the polynomial coefficients, then convolves the intermediate result with the basis functions. The second contraction path first convolves input with the basis functions, then contracts the intermediate result with the polynomial coefficients.
21

---

The third contraction path first generates the temporal kernels from the polynomial coefficients and basis functions, then convolves them with the imput features. In most cases, we see that the last contraction path is most memory efficient in typical cases, or when < dn. However, the optimality for computational efficiency more subtle and requires comparison of dn(c + T), ne(T + d), and der.
B Details of Experiments
To convert events into frames, we choose the binning window to be 10 ms, unless otherwise specified. This time step kept fixed throughout our network, as we do not perform any temporal resampling through the network. For the DVS128 and AIS2024 eye-tracking experiments, we perform simple direct binning along with random affine augmentations (with rotation angles up to 10 degrees, translation factors up to 0. and spatial scaling factors up to 1.1). For the Prophesee roadscene detection, we perform event-volume binning (analogous to bilinear interpolation), with augmentations consisting of horizontal flips at 0.5 probability and random scaling with factors from 0.9 to 1.1.
Recall that our network performs valid-type causal temporal convolutions which reduces the number of frames by (kernel size- 1) per temporal convolution. To avoid introducing any implicit temporal paddings to our network, we prepend extra frames (relative to the labels) to the beginning of the input segment. The total number of extra frames is then (number of temporal layers) x (kernel size - 1).
For all training runs, we use the Adam optimizer with learning rate of 0.001 and weight decay of 0.001 (with PyTorch default keywords), along with the cosine decay learning rate scheduler (updated every step) with warmup period of around 0.01 of the total training steps. The runs are performed with automatic mixed precision (float 16) with the model torch. compile le 'd. All training jobs are done on single NVIDIA A30 GPU.
For the total training epochs and walltimes (on a single A30 GPU):
DVS128: 100 epochs and around 32 minutes, using a batch size of 64,
3ET+: 100 epochs and around 9 minutes, using batch size of 64.
Prophesee GEN4: 25 epochs and around 8 hours, using a batch size of
The batch sizes are set such that nearly saturates the available VRAM on the NVIDIA A30 GPU, or around 24 GB. With the exception of the Prophesee object detection experiments (Appendix [.2). we ran 25 trials for each experiment to report the mean and standard error estimators of the metrics.
B.1 DVS Hand Gesture Recognition
Following the standard benchmarking procedure on this dataset, we only train and evaluate on the first 1.5 seconds of each trial, and filter the "other" class where the subject performs random gestures not falling into the 10 predefined classes.
As mentioned, the network requires least (number of temporal layers) (kernel size 1) + 1 frames of inputs to guarantee that every temporal layer is processing "valid" nonzero input features. To generate output predictions with less than this number of frames, can prepend zeros to layer inputs where needed to match the kernel size. This simulates the behavior of initializing the buffers of the temporal layers with zeros during online inference.
If the number of input frames is greater than (number of temporal layers) x (kernel size - 1) + 1. then the network will produce more than one output prediction. If the latency budget allows, we can apply a majority filter to the classification predictions of the network, such that there is more confidence in the predictions.
To foree the network to respond faster, we apply custom random temporal masking augmentation sample-wise with 1/2 probability. The random masking operation works by selecting a frame uniformly random from the first frame to the middle frame of the segment, then the selected frame and every frame preceding it is completely set to zero. This means that the network will be artificially biased to respond to more recent input features during inference, thereby effectively decreasing its response latency.
22

---

B.1.1 Input Sparsity
We perform 4-bit quantization (with quantization aware training) on the gesture recognition network, and find that the network can achieve very high sparsity even without applying any regularization loss, given that interfaces with event-based data and uses ReL.U activations (which is sparsity promoting).
Table 5: Input sparsity for each layer of the gesture recognition network backbone under 4-bit quantization.
Layer
Input Sparsity
Conv(I+2)D
0,99
Cony(I+2)D
0.94
(1+2(
0.94
Conv(1+2)D
0.79
Conv(I+2)D
0.68
B.1.2 Effect of Polynomial Degree and Jacobi Parameters
We tested the effect of polynomial degree the performance of the network at the low-latency regime of 200 ms, where the network sees less than half the amount data needed to fill its temporal buffers. Perhaps counterintuitively, having too high of polynomial degree actually degraded performance (and having too low of a degree also underperforms). This highlights the importance of selecting an appropriate degree: that is expressive enough learn meaningful patterns from limited data, yet not so flexible that it overfits to temporal noise and fails to generalize when the full temporal context unavailable.
Table The effect of polynomial degree on the performance, for the DVS128 dataset latency of 200 ms, with Jacobi parameters fixed at (a, 3) (-0.25, -0225)
Degree
Accuracy
55.4 (0.03)
4
73,2 (0.03)
5
67.6 (0.04)
60.2 (0.02)
52.5 (0.02)
free
52.3 (0.02)
As shown in Table 7 the effect of varying the Jacobi parameters (a., is relatively minor compared to the impact of polynomial degree. While (-0.25, -0.25) yields the highest accuracy, the differences across other parameter settings are within roughly two standard deviations, suggesting that the influence these parameters may only be borderline significant.
Table 7: The effect of Jacobi parameters the performance, for the DVS128 dataset latency of 200 ms, with degree fixed at 4.
Jacobi Parameters (a, 8)
Accuracy
-0.25, -0.25)
73.2 (0.03)
-0.25, 0)
72.9 (0.02)
( --0255, 0.25)
72.8 (0.03)
(0, -0.25)
72.8 (0.04)
(0, 0)
72.7 (0.03)
(0, 0.25)
72.6 (0.03)
(0.25, 00.25)
72.9 (0.02)
(0.25, 0)
72.8 (0.04)
(0.25, 0.25)
72.6 (0.03)
23

---

B.2 Prophesee GEN4 Roadscene Object Detection
Following recipe similar to the original paper, we remove bounding boxes that are less than 60 pixels in the diagonal. In addition, we perform event-volume binning which simultaneously performs spatial resizing from (720, 1280) (160, 320) and temporal binning of 10 ms. For data augmentations, we perform horizontal flips at 0.5 probability and random scaling with factors from 0.9 to 1.1.
The CenterNet detection head produces feature frames where each frame is spatially shaped (40, 80). Each pixel contains 7 + 2 + outputs containing 7 class logits (center point heatmaps). the bounding box height and width scales, and the bounding box center point de and ye offsets. We perform evalua- tions directly on these raw predictions, without any output filtering (c.g. no non-max suppression). The network is trained on the full 7 road-scene classes of the dataset, and the mAP is evaluated the cars and pedestrians classes, at confidence thresholds from 0.05 to 0.95 in steps of 0.05 and averaged using trapezoid integration.
See Table. [ for details on the model architecture, which resembles an hourglass structure. Unless otherwise indicated, the temporal kernel size is assumed be 10, causal and valid-type. The spatial kernel size is assumed to be 3 x 3, where the spatial stride can be inferred from the output shape of the layer. DWS denotes both the temporal and spatial layers in the Conv (onv(1+2)) ++) block depthwise- separable. The BottleNeck block is similar (but not identical) to the IRB block in MobileNetV2; it residual block with the residual path containing three Conv2D layers with ReLU activations in between: a depthwise 3 x 3 Conv2D followed by a pointwise Conv2D quadrupling the channels followed by pointwise Conv2D quartering the channels.
Before each decoder layer, the input feature first upsampled spatially by factor x 2. is then summed with intermediate output feature from encoder layer that has the same spatial shape. To match the temporal shapes, the beginning frames are truncated if necessary. The remaining frames projected with pointwise convolutional layer (a long-range skip connection).
Table 8: The PLEIADES CenterNet architecture used for the Prophesee dataset.
Layer
Output Shape
Channels
Input
(2, 7; 160, 320)
Encoder
Conv(I+2)D
(32, 7 9, 80, 160)
16 +
BottleNeck 2D
(32, T 9, 160)
32 +232 + 128 + 32
DWS Conv(I+2)D
(64, T' 18, 40, 80)
-2-45 - 64
BottleNeck 2D
(64. T 18,40, 80)
64- 64 256 64
DWS Conv(1+2)D
(96. T 27, 20)
64-80 80 96
DWS Conv2D
(128, 27, 10,20)
96
DWS Conv2D
(256, 27, 1,)
128 + 256
Decoder
Upsample
(256, T 27, 10,20)
DWS Conv2D
(256, T 27, 10, 20)
256 > 256
Upsample
(256, -'- 27, 20,40)
DWS Conv2D
(256, T' -27, 27, 20,20,40
256 + 256
Upsample
(256, 27, 40)
DWS Conv2D
(256, T 27, 40,
256 256
CenterNet Head
DWS C+))(
(128,T T 27, 40,
256
256 128
pointwise Conv
[11, 27, 40, 80)
128 11
24

---

B.3 Results on KITTI 2DOD task
See Table 9 for results of PLEIADES on the KITTI 2D object detection task compared other standard networks optimized for conventional camera data. Our network architecture is the same used for the Prophesee GEN4 road-scene object detection task as given in T
Model
mAP
Parameters
MACs sec
RGBD Fusion (YOLOv2)
0,482
349 B
SimCLR (ResNet50)
0.570
26 M
82 B
PLEIADES + CenterNet 0.576
0.57 M
18 B
Table 9: Comparison of models in terms of mAP, number of parameters, and MACS per second. Note that the MACs/sec measure assumes an FPS of 10, which is used in the KITTI recordings.
25


Reviews:
[{"review_id": "gj1iNH6e7L", "reviewer": "Reviewer_VVJk", "summary": "This paper introduces PLEIADES, a family of fully‐causal spatiotemporal convolutional networks whose temporal kernels are parameterized as linear combinations of low‐order Jacobi polynomials. By structuring the temporal filters in this way, the authors achieve substantial parameter savings and enable a single model to operate at multiple temporal resolutions without retraining. Integrated into a depthwise‐separable (1+2)D backbone and paired with a CenterNet‐style detection head.I appreciate the paper’s focus on parameter efficiency and its use of orthogonal polynomials to enforce causality and multiresolution flexibility.", "questions": "Why choose degree-4 Jacobi polynomials with α=β=–0.25 specifically? Provide either a theoretical argument or an empirical comparison against other orthogonal bases (e.g., Chebyshev, Legendre) and different degree/α/β settings. Showing a small grid‐search or error-bound analysis would clarify that this choice is principled rather than arbitrary. Broaden Evaluation Beyond Event Cameras. Does the polynomial-parameterized kernel extend effectively to other modalities (e.g., standard RGB video, audio, or sensor time series)?", "limitations": "Please see above", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The paper provides an interesting read. Using orthogonal polynomials to constrain temporal filters is a clever way to add additional inductive bias. Additionally, the same trained network can nicely handle input streams sampled at different rates without fine-tuning.\n\nHowever, the paper would benefit from deeper theoretical and empirical grounding:\n\n- Lack of principled basis selection: The choice of degree‐4 Jacobi polynomials with α=β=–0.25 appears ad hoc. There is no derivation or discussion of why these specific polynomial parameters are optimal, nor comparison to other orthogonal bases.\n\n- Sparse ablation studies: Key hyperparameters—polynomial degree, α/β settings, temporal kernel length, and buffer size—are insufficiently explored. A more thorough sensitivity analysis would help practitioners understand trade‐offs between expressivity, latency, and memory.\n\n- Narrow experimental scope: All evaluations focus on event‐camera data. Without tests on conventional video, audio, or other time‐series domains, it’s unclear whether PLEIADES generalizes beyond this niche.\n\n- Missing baselines: The paper does not compare against other structured temporal modules (e.g., S4, Fourier‐parameterized filters) or compact recurrent units, leaving open whether the polynomial approach truly outperforms these alternatives.\n\nOverall, while PLEIADES presents an intriguing architectural twist, its conceptual novelty and practical applicability would be strengthened by more principled justification, broader evaluations, and clearer empirical analyses.", "was_revised": true, "final_justification": "The authors actively answer my questions, I am increasing my score to weak accept.", "rebuttal": "Dear Reviewer VVJk, we thank you for the careful evaluation of the paper. We would like to address some of the points you raised.\n\nThe choice of the Jacobi parameters α = β = –0.25 is indeed based on empirical selection. In Table 7 of Appendix B.1.2, we **compared this choice against other sets** of (α, β) values, and determined this to be relatively performant. While the theoretical justification was not fully detailed in the paper, we take this opportunity to clarify the rationale behind our choice. We first note that the choice of  (α, β) does not affect the expressivity of the temporal kernels, and all choices of (α, β) yield polynomial bases spanning the same space. What it really influences here is the **training dynamics**, and a full mathematical justification of why α = β = –0.25 yields superior training dynamics is beyond the scope of this work, requiring extensions to theories of neural tangent kernel and dynamic mean field theory etc. Nevertheless, there is a justification at an informal level. The choice of α = β = –0.25 is an \"intermediate\" between Legendre polynomials (α = β = –0) and Chebyshev polynomials (α = β = –0.5), which balances the benefits of both. For instance, Legendre polynomials are more balanced, but slower to represent sharp boundary transitions; Chebyshev polynomials are better for representing edge-localized patterns, but more numerically unstable.\n\nThe **accuracy–latency tradeoff** is a central focus of our study and is highlighted in Figure 3 of the main text. In this experiment, we systematically increase the network’s effective buffer size—allowing for greater latency—to observe corresponding gains in accuracy. This reveals several key Pareto frontiers for our model. Another central aspect of our work is **the effect of temporal kernel length**. A distinctive feature of our approach is the ability to freely adjust or “resample” the granularity of the kernel. This flexibility is discussed theoretically in Section 3.2 and empirically illustrated in the multiple subplots of Figure 3, each corresponding to a different temporal resolution. In addition, our comparisons with other state-of-the-art models across three tasks include key metrics such as **parameter count and computational cost**. Lastly, as noted above, Appendix B.1.2 analyzes the impact of polynomial degree and the choice of (α,β). Taken together, we believe these empirical studies provide a comprehensive assessment of the relevant tradeoffs.\n\nThe point about extending our work to RGB / conventional cameras is an important one, and we thank the reviewer for pointing this out. Our network can indeed be easily adapted for conventional cameras, as mentioned in the last paragraph of the Introduction, and we are currently performing empirical studies along this direction (e.g. object detection, segmentation, planning with RGB cameras). However, we choose to focus specifically on event cameras in our current work, as it is a relatively under-explored area, with huge potential for efficient deployment on the edge (mobile devices). Our choice of a relatively simple and light-weight network also fits this theme. The extension to audio and time series data is an interesting one. In theory, we can remove the spatial convolutional layers, keeping the temporal ones for time series processing, so it would be an interesting direction for future study (with comparisons to S4, Mamba, DeltaNet, etc).\n\nWe would like to clarify that we do compare against **structured state-space models** or SSMs. For example, in Table 2, we compare against **MambaPupil** for event-based eye tracking. And in Table 3, we compare against **S5-ViT-B** on the Prophesee object detection dataset. In both cases, we outperformed them in both performance and efficiency. For comparison against SSMs architecture-wise, we provided a comparison in Section 2.1, and highlighted several differences: 1) our network is spatio-temporal, 2) we preserve the polynomial basis, 3) we have a more flexible connectivity structure. Finally, in Section 6, we discuss how SSMs can be incorporated into our network.\n\nWe sincerely appreciate your time and thoughtful feedback. We hope our responses have clarified the concerns you raised, and we kindly ask that you consider revisiting your evaluation in light of our rebuttal. If any aspects remain unclear or raise further questions, we would be happy to provide additional clarification."}, {"review_id": "jAymd6V68K", "reviewer": "Reviewer_cGrC", "summary": "The paper presents PLEIADES, a causal spatio-temporal network whose temporal filters are written as short Jacobi-polynomial expansions. This replaces a long 1-D convolution with just a few learned coefficients, giving wide temporal context while keeping the parameter count modest. Because the filters are continuous in time, they can be re-sampled, so one trained model can run at several frame rates without extra tuning. Inserted into a straightforward (1 + 2)D backbone, the approach scores 99.6 % on DVS128 gestures, 99.6 % on the AIS eye-tracking task, and 0.556 mAP on the Prophesee automotive data. The design is aimed at real-time, resource-constrained event-vision systems.", "questions": "Q1. Empirical Validation of Physical Inductive Bias\nThe manuscript asserts that Jacobi polynomials inject a “physical inductive bias” via their Sturm–Liouville connection, leading to “more stable training” and “better optima,” yet provides no training‐dynamics data in Section 5.1.\no\tRequest: Please include training‐loss curves, gradient‐norm trajectories, and convergence‐rate comparisons between PLEIADES and a free‐kernel (Conv(1+2)D) baseline.\nQ2. Jacobi Parameter and Degree Sensitivity\nThe choice α=β=−0.25 and polynomial degree N=4 appears empirical.\no\tRequest: Provide a sensitivity analysis over α,β∈{−0.5,0,0.5} and degrees N∈{2,4,6,8}, with accuracy/stability results, and give guidelines for choosing these hyperparameters in new domains.\nQ3. Generalization Beyond Event Vision\nAll experiments are on event‐camera benchmarks, leaving open whether PLEIADES applies to other temporal tasks.\no\tRequest: Test PLEIADES on at least one non‐event task (e.g. video‐frame classification or audio‐sequence modeling) or on longer time windows, and report performance to assess broader applicability.\nQ4. Ablation of Core Components\nIt is unclear which design elements drive the reported gains.\no\tRequest: Provide ablation studies isolating (a) polynomial‐kernel parameterization, (b) (1+2)D factorization, (c) hybrid normalization, and (d) strict causality, showing each component’s effect on accuracy and latency.\nEvaluation Criteria:\nMeeting Q1 and Q2 with clear empirical data would significantly boost my confidence. Demonstrating cross‐domain performance (Q3) would broaden the paper’s impact. Comprehensive ablations (Q4) would clarify the key innovations and could raise clarity/originality scores.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\nQuality: \nThe paper demonstrates good technical quality with rigorous mathematical foundations and comprehensive experimental validation. The use of Jacobi polynomials for temporal kernel parameterization is mathematically sound, leveraging well-established orthogonality properties and connections to Sturm-Liouville differential equations. The experimental methodology is thorough, with 25 trials per experiment, proper statistical reporting with standard errors, and testing across diverse event-based benchmarks. The resampling experiments provide particularly compelling evidence of the method's theoretical advantages, showing stable performance across different temporal discretizations without retraining.\nClarity:\nThe paper is generally well-written with mathematical exposition. Figure 1 effectively illustrates the core concept of generating discrete temporal kernels from polynomial coefficients. The progression from theoretical foundations (Section 3) to practical implementation (Section 4) and experimental validation (Section 5) follows a logical structure. The Einstein notation is used appropriately to simplify complex tensor operations, and the contraction optimization discussion provides valuable implementation insights.\nSignificance:\nThe work addresses an important problem in event-based computing where traditional approaches struggle with temporal modeling efficiency. The achieved performance improvements are substantial - PLEIADES reduces parameters by 62-96% compared to competing methods while maintaining or exceeding accuracy. The 100 FPS inference capability on complex detection tasks represents a significant practical advancement. The resampling capability without retraining has broad implications for deploying models across different sensor configurations and hardware platforms.\nOriginality:\nThe application of orthogonal polynomials to neural network temporal kernels is novel and theoretically motivated. While orthogonal polynomials have been used in state-space models (LMU, HiPPO, S4), their application to explicit finite-window temporal convolutions with resampling capabilities is original. The connection between polynomial structure and computational optimization through flexible contraction ordering is a creative contribution that bridges mathematical theory with practical implementation concerns.\nWeaknesses\nQuality:\nThe parameter selection for Jacobi polynomials (α=β=−0.25, degree up to 4) appears purely empirical without theoretical justification or sensitivity analysis. The paper lacks ablation studies examining the contribution of different architectural components, making it difficult to assess which design choices are critical. The comparison baselines, while reasonable, may not represent the absolute state-of-the-art in all cases, and some methods (like TrueNorth CNN from 2017) are becoming dated.\nClarity:\nThe paper suffers from several presentation issues that impede understanding, particularly for readers without deep expertise in numerical analysis.\n\tConcept-to-notation gap: The important orthogonality definition for Jacobi polynomials (lines 117-118) lacks equation numbering. The paper transitions too abruptly from the intuitive concept of \"polynomial kernels\" to dense Einstein-summation formulas without providing concrete examples. For instance, after introducing Jacobi polynomials in Section 3.1, the paper immediately jumps to the general form in equation (1)  without showing what a simple degree-2 kernel would look like in explicit vector form. This leaves readers struggling to map abstract symbols to concrete operations.\n\tPhysical inductive bias inadequately supported: Section 3.1 claims that Jacobi polynomials provide \"physical inductive biases\" due to their connection with Sturm-Liouville differential equations, making training \"more stable\" and guiding networks to \"better optima.\" However, this crucial claim is merely asserted without empirical evidence. The paper lacks training dynamics comparisons, gradient norm trajectories, or convergence analysis that would substantiate these theoretical advantages over free kernels.\n\tNotation inconsistencies: The manuscript inconsistently alternates between continuous kernels and discrete implementations without clearly establishing the discretization relationship. Additionally, Figure 1 labels coefficients as γ_n  while Section 4 references different variables, creating unnecessary confusion. \n\tEinstein notation overuse: While Einstein summation is powerful, its repeated use for both channel mixing and polynomial basis summation obscures the actual data flow and operation order. The same operations presented once in plain matrix notation or pseudocode would significantly improve readability, especially for equation (4) where multiple indices interact simultaneously.\n\tIncomplete mathematical exposition: Critical implementation details like numerical integration of orthogonal polynomials, gradient computation through the polynomial parameterization, and numerical stability considerations are either missing or relegated to appendices. This makes it difficult for readers to understand or reproduce the method.\nThese clarity issues collectively create barriers to understanding and adoption, particularly problematic given the paper's goal of providing a practical alternative to existing temporal modeling approaches.\nSignificance: \nThe evaluation is limited to event-based vision tasks, raising questions about generalizability to other temporal modeling domains. The memory limitation discussion reveals a fundamental scalability issue (linear growth with kernel length and multiplicative growth with spatial resolution) that could limit practical deployment. While a theoretical solution is proposed using recursive updates, this remains unimplemented and unvalidated. The method's advantages may diminish for very long temporal sequences where the polynomial approximation becomes less effective.\nOriginality:\nWhile the specific application is novel, the work builds heavily on existing foundations (orthogonal polynomials, (1+2)D convolutions, CenterNet heads). The connection to state-space models is acknowledged but not deeply explored,  the paper could better position itself relative to this rich body of work. The claimed novelty of \"structured temporal kernels\" may be overstated given prior work on parameterized temporal filters in various forms.", "was_revised": true, "final_justification": "Based on the additional information provided by the reviewers, my concerns have been eased and I have raised my score.", "rebuttal": "We thank the reviewer for their comprehensive and thoughtful review. We aim to address the comments and questions as thoroughly as possible within the scope of the rebuttal.\n\n- The choice of α = β = –0.25 is an \"intermediate\" between Legendre polynomials (α = β = –0) and Chebyshev polynomials (α = β = –0.5), which **balances the benefits of both**. For instance, Legendre polynomials are more balanced, but slower to represent sharp boundary transitions; Chebyshev polynomials are better for representing edge-localized patterns, but more numerically unstable. A full formal proof of why this choice is optimal is currently beyond the scope of this work, and would involve complex learning dynamics theory.\n\n- In Appendix B.1.2 we performed sensitivity analyses of the effects of polynomial degree, alpha, and beta. The main conclusion was that the choice of a degree being 4 was an important one, whereas the choice of (α, β) appeared less significant. Therefore, we believe degree 4 polynomials to be at the sweet spot being having expressivity while preventing overfitting (bias-variance tradeoff)\n\n- Besides TruthNorth, all the other networks are fairly modern. For instance, all the networks for eye-tracking were from the 2024 CVPR AIS challenge. And S5-ViT-B is a modern network from 2024 combining state-space modeling and transformers. We decided to include TruthNorth because it is a seminal network for event-based vision.\n\n- Due to space constraints, we focused on the aspects of Jacobi polynomials most relevant to our method, rather than providing a full theoretical background. As the reviewer noted, our approach integrates concepts from functional analysis, numerical approximation, and tensor operations. We aimed to present these concisely while maintaining coherence, with Figure 1 serving as a visual summary of the core ideas.\n\n- For comparison against the performance of free kernels, we showed in the right subplot of Fig. 3 that **having polynomial temporal kernels was better at almost all latency settings**. We acknowledge that we did not dig too deeply at the level of training dynamics, and thank the reviewer for pointing this out as a gap.\n\n- In Section 3.2, we clearly detailed the discretization of the continuous temporal kernels, and from there on, the kernels are referenced in their discretized form, denoted by the overline above k.\n\n- α, β, and γ actually refer to different parameters. (α, β) are parameters defining the Jacobi polynomial basis (via the weight function).  γ refers to the polynomial coefficients. This is detailed in Sections 3.1 and 3.2\n\n- We believe Einstein summation notation here actually makes the channel connectivity more apparent, and allows for cleaner notations. It is also crucial to the discussion of optimal contraction patterns as detailed in Appendix A. Additionally, it is also more directly translatable to np.einsum and torch.einsum, and has become increasingly prevalent (e.g. Mamba2). That being said, the introduction of the notation in Section 3.3 may have been too abrupt, and the paper would benefit from a longer exposition of this notation.\n\n- The beauty of PLEIADES is that we **do not require manual derivation of the backward pass** (gradient computation), say, compared to the SLAYER model [Shrestha 2018]. Everything is differentiable, so autograd (tf, torch, jax) would be able to handle everything for us, as evident in the attached source code. This is stated at the end of Section 3 as \"PLEIADES kernels can leverage standard optimization methods such as backpropagation\". In addition, the optimal contraction order in the forward pass also implies the optimality of the backward pass, as mentioned in footnote 4 and Appendix A. \n\n- While a formal analysis of gradient stability would be valuable, we note that standard practices—such as normalization layers and adaptive optimizers like AdamW—can obfuscate the implicit effects of our polynomial parameterization on training dynamics. Nonetheless, we agree this is an interesting direction for deeper theoretical investigation.\n\n- As the reviewer noted, memory cost scales linearly with temporal buffer size and is further multiplied by spatial dimensions—an inherent challenge for (1+2)D ConvNets. However, our network’s causal structure limits this cost to the buffer (not the full sequence), and the resampling feature allows for shorter temporal kernels via larger step sizes. While we acknowledge this bottleneck as a consideration for edge deployment, PLEIADES remains a strong candidate given its parameter and compute efficiency—often **1–2 orders of magnitude lower** than comparable models.\n\n- Even though PLEIADES builds on top of previous works, we do not think it is incremental. The usage of trainable polynomials in an (1+2)D network setting, to our knowledge, is completely novel and offers major performance and efficiency boosts over prior arts.\n\nTo address the questions directly:\n\n- Q1 While we are unable to provide a comprehensive set of training dynamics analyses within the short rebuttal window, we kindly ask the reviewer to consider whether Figure 3 provides sufficient evidence that polynomial parameterization plays a meaningful role for PLEIADES.\n- Q2 We hope that Appendix B.1.2 addresses this concern. Based on our experiments across three diverse tasks, degree 4 consistently provided strong performance, suggesting it is a robust choice in practice. While we acknowledge that a more exhaustive sweep could be informative, our findings indicate that the exact choice of (α,β) has relatively limited impact compared to the degree. We believe these observations offer a reasonable justification for our selected configuration.\n- Q3 We appreciate the reviewer’s suggestion to explore conventional camera tasks. While we agree that extending PLEIADES to RGB-based domains such as object detection and segmentation is a promising direction—and one we are actively pursuing—we have chosen to focus this work on event-based tasks. This domain remains relatively under-explored and presents unique challenges and opportunities for efficient temporal modeling, which we believe deserve dedicated attention. We hope the reviewer understands our decision to keep the scope focused in this initial study.\n- Q4 Our primary focus in this work is on the effectiveness of the polynomially parameterized temporal kernels, which we have comprehensively analyzed through ablation studies. The other components—such as (1+2)D factorization, hybrid normalization, and strict causality—were adopted based on prior art and serve a supporting role rather than being central to our contribution. We believe a full exploration of these components, while valuable, is best addressed in future work focused on architectural variations.\n\nWe sincerely appreciate your thoughtful feedback and the opportunity to clarify our contributions. We hope our responses have been helpful in addressing your concerns, and we would be grateful if you would consider them in any potential revision of your evaluation. Please feel free to let us know if you have any further questions or comments—we’d be happy to clarify."}, {"review_id": "e2S3mVtPlL", "reviewer": "Reviewer_iy2F", "summary": "This paper introduces PLEIADES, a neural network architecture for online spatiotemporal processing, particularly with event-based data, by constructing temporal convolution kernels. Instead of using explicitly parameterized, discrete kernels, like temporal convolutions, PLEIADES generates them from a linear combination of orthogonal polynomial basis functions, i.e., Jacobi polynomials. Benefiting from this design, it is efficient, flexible, and high-performing.", "questions": "1. What is the key difference between the proposed method and previous work like Mamba, and what is the reason for the performance improvement over them?\n2. How many trials do you conduct on each benchmark, for your method and other methods?\n3. Could you provide experimental results on general object tracking benchmarks, such as LaSOT?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The motivation of the paper is reasonable, and the proposed method is promising for its ability to tackle different sampling rates without fine-tuning.\n2. The paper is clearly written and easy to follow.\n3. The paper provides a powerful new tool and backbone for the community by setting a new state of the art on three major and diverse tasks.\n\nWeaknesses:\n1. The proposed method stems from established and well-known concepts (orthogonal polynomials, (1+2)D convolutions, event-based processing), and a short discussion could be included to help distinguish the novelty of this work.\n2. The number of trials should be given for each experiment.\n3. Apart from using temporal kernels, other methods, such as Mamba, also present superior performance for sequential modeling. Discussion and experiment should be incorporated for comparison.", "was_revised": true, "final_justification": "I appreciate the response from the authors. Since my questions are addressed, I hold the original score.", "rebuttal": "We thank reviewer iy2F for the positive feedback. We'd like to take the chance here to address the comments / questions.\n\nThe reviewer is indeed correct that the network is based on the (1+2)D conv pattern. In the introduction, we mentioned that \"we augment a previously proposed (1+2)D causal ... network by replacing its temporal kernels with this new polynomial parameterization\", to highlight the key novelty here being our **polynomial parameterization of the temporal kernels**. However, we do realize this can be moved earlier in the introduction, so the novelty is immediately clear.\n\nAs noted in Section 5 and in item 7 of the reproducibility checklist, we conducted **25 trials** for each experiment and reported the mean and standard error. In the result tables, the standard deviation is shown in brackets for our PLEIADES network. For the other methods, we report results as published in their respective papers, as some of these models are not open-source and were therefore not available for re-evaluation.\n\nFor **comparison against SSMs**, we compared against MambaPupil for eye-tracking and S5-ViT-B for object detection, in addition to theoretical discussions in Section 2.1 and 6. \n\nIn Section 2.1, we highlighted several key architectural differences of PLEIADES over existing SSMs: 1) our network is spatio-temporal, 2) we preserve the polynomial basis, 3) we have a more flexible connectivity structure. And in Section 6, we discuss how SSM designs can be incorporated into PLEIADES.\n\nThe focus of our work was on event-based tasks, so we did not include conventional camera datasets in the paper. That being said, we are currently actively exploring the application of PLEIADES to RGB camera tasks such as object detection, segmentation, and planning. Initial results suggest a 5x reduction of parameters while achieving similar results compared to the smaller models in the YOLO series when benchmarked on the KITTI 2DMOT. And LaSOT is certainly a dataset that we could benchmark on in the future (along with other tracking benchmarks such as VOT, TrackingNet, GOT-10k)"}]

Author rebuttal:

Meta-review: PLEIADES presents a robust, fully‑causal spatiotemporal convolutional backbone that parameterizes temporal kernels with short Jacobi‑polynomial expansions. By reducing the number of learned coefficients, the authors achieve substantial parameter savings while preserving or surpassing performance on three diverse event‑based benchmarks. 
The authors have effectively addressed the reviewers' concerns, leading all reviewers to agree on a "borderline accept" rating for the paper. ACs concur with this positive assessment and recommend acceptance of the paper.
Decision: Accept (poster)