Bridging the latency gap with a continuous stream evaluation framework in event-driven perception

Domain: biology
Published: 2026-03-16
Editorial outcome: major_revision

Abstract
Neuromorphic vision systems process continuous event streams and offer transformative potential for real-time applications. However, their evaluation remains tethered to methodologies from RGB imaging. These approaches convert asynchronous event streams into synchronized frames and ignore perception latency, creating a critical gap between benchmarks and real-world performance. To address this, we introduce the STream-based lAtency-awaRe Evaluation (STARE) framework. STARE integrates two core components: Continuous Sampling, maximizing model throughput to reduce the impact of latency, and Latency-Aware Evaluation, quantifying latency-induced online accuracy. To rigorously validate STARE, we developed ESOT500, a high-dynamic object tracking dataset with 500 Hz annotations. Experiments reveal that latency severely degrades online accuracy by over 50%. We further introduce two model enhancement strategies: Asynchronous Tracking, a fast-slow architecture that boosts model throughput, and Context-Aware Sampling, which dynamically adapts input to handle low event density cases. Overall, our work bridges the latency gap between models’ theoretical potential and real-world deployment.

Introduction
Biological vision systems excel at perceiving dynamic environments through continuous, adaptive sensing1–3. In contrast, artificial vision systems, even those inspired by neural mechanisms, typically rely on discrete and frame-based processing inherited from conventional RGB cameras4–8, as shown in Fig. 1a–b. This mismatch introduces a critical limitation: the temporal discontinuity of static frames inherently induces perceptual delay between sensory inputs9–13. Such delay accumulates in real-world applications, degrading performance in tasks requiring rapid reactions, such as autonomous navigation or human-robot interaction9,14–17. Event cameras, which are neuromorphic sensors that asynchronously encode pixel-level intensity changes at microsecond resolution, promise to bridge this gap by capturing continuous event streams3,18–21. However, the traditional perception framework, which forces continuous event stream into fixed-rate event frames3,4,21–24, as shown in Fig. 1b, reintroduces perceptual delays and discards the sensor’s inherent capacity for real-time, event-driven computation. Moreover, such framework typically evaluates model performance by comparing each model output against the ground truth of the corresponding input frame. This evaluation paradigm assumes instantaneous computation, failing to account for the influence of perception latency. We define perception latency as the total time elapsed from the moment an event is triggered to the moment a downstream application receives the perception model’s corresponding output, which is a critical factor in real-world deployment. In such scenarios, downstream applications such as robotic control require continuous access to the most recent outputs of perception models, and even minor latency can lead to outdated predictions and compounding errors. This mismatch between evaluation and real-time application is especially pronounced in neuromorphic vision, where event cameras generate data at over 1 MHz, yet existing frameworks process these streams as low-frequency frames (typically ~25 Hz), discarding temporal precision and penalizing lightweight, high-speed algorithms by confining them to suboptimal input frequencies.

To unlock the potential of event cameras in real-time applications, recent advances in perception frameworks have attempted to reduce latency by either optimizing frame-based processing pipelines9,25–28 or redesigning the perception framework to directly process the online streaming data29–32. Despite these improvements, evaluation methodologies remain predominantly based on fixed frame-rate sequential data. To address this issue, we introduce the STream-based lAtency-awaRe Evaluation (STARE) framework, which is designed to transcend traditional limitations and realistically assess event-driven models through two core components, as shown in Fig. 1c. First, Continuous Sampling schedules the model to immediately process the latest events right after the previous cycle, maximizing model throughput for real-time use. Second, Latency-Aware Evaluation integrates latency into accuracy metrics: it aligns each timestamped ground truth with the latest available prediction, directly quantifying accuracy degradation caused by outdated model outputs. An example of the perception result from STARE is shown in Fig. 1d.

Given that existing event datasets are constrained by low frame-rate annotations and designed for the traditional latency-ignored framework, we developed a new event dataset with high-frequency annotations: ESOT50033, a 500 Hz object tracking dataset in which each annotation simulates a real-world downstream query (Fig. 2). This annotation density ensures resistance to temporal aliasing34 and supports rigorous Latency-Aware Evaluation (Fig. 3). Leveraging STARE with ESOT500, we demonstrate that perception latency reduces accuracy by over 50% (Fig. 4a–b). Robotic experiments further validate this finding: a 55% increase in perception latency leading to complete task failure (Fig. 5).

To address the accuracy degradation stemming from latency, we propose two model enhancement strategies that underscore the need for advanced architectural designs and operational paradigms for event-driven perception systems (Fig. 6). First, Asynchronous Tracking, which draws inspiration from harnessing the continuity of event streams to improve model throughput and thus fulfills the dense query requirements from downstream applications9,25, adopts a dual-component architecture: a heavyweight base model generates high-accuracy yet low-frequency predictions, while a lightweight residual model is integrated to rapidly refine the most recent output of the base model. Second, Context-Aware Sampling, which originates from our key observation that model performance correlates with the event density surrounding the target object. This method dynamically adjusts the model’s activation state based on this contextual event density. Experimental results demonstrate the efficacy of these strategies: Asynchronous Tracking improves latency-aware accuracy by up to 60% (from 31.83 to 51.06 AUC) via a 78% enhancement in model throughput (from 118 Hz to 210 Hz), whereas Context-Aware Sampling further enhances robustness, yielding over 51% performance improvement (from 18.73 to 28.29 AUC) in challenging scenarios (Fig. 7).

Our approach breaks free from the static, latency-ignored constraints of traditional vision research by centering on a critical, underaddressed principle: temporal congruence between event cameras’ continuous sensing, model computation, and real-world task demands. Unlike prior work that forces event streams into frame-based pipelines or treats latency as an isolated metric, we present a cohesive solution that integrates STARE (a stream-based latency-aware evaluation framework that mirrors real deployment), ESOT500 (a dense 500 Hz dataset for validating the framework), and our asynchronous/context-aware strategies, thereby narrowing the divide between the theoretical speed of event cameras and their practical utility in real-time scenarios. This represents the main contribution of our work.

Results
The stream-based latency-aware evaluation framework
We developed the STream-based lAtency-awaRe Evaluation (STARE) framework: a realistic, rigorous benchmarking tool explicitly tailored to event-driven vision systems, designed to transcend the limitations of conventional latency-ignored paradigms. STARE’s design is anchored in two core, mutually reinforcing principles that govern both the operation of models in online streaming scenarios and the quantification of their performance under real-time constraints, namely Continuous Sampling and Latency-Aware Evaluation.

Continuous sampling in STARE. The traditional framework operates offline by converting a continuous event stream into a sequence of fixed-rate frames and processing them sequentially, as shown in Fig. 1b. However, adapting this fixed-rate method for online operation would disrupt the event stream’s continuity, lowering model throughput and creating inefficiency. The core issue is the inherent misalignment between a model’s variable inference latency and the fixed frame interval. This misalignment inevitably causes idle time for both faster and slower models, as they must wait for the next frame to arrive, thereby artificially limiting maximum throughput10–13. More details can be found in Section “Formal Definition”.

To address this, STARE employs the Continuous Sampling strategy during the model’s perception stage. As shown in Fig. 1c, this approach leverages the continuity of the event stream by scheduling the model to sample the most recent events for processing immediately upon completing the prior processing cycle. As illustrated by the comparison between Fig. 1b, d, Continuous Sampling eliminates idle time and enables the model to operate at maximum throughput, which is critical for real-time applications such as robotic control or high-speed interaction.

Latency-aware evaluation in STARE. The second core principle of STARE lies in its evaluation protocol, which explicitly integrates perception latency into the final accuracy metrics. As shown in Fig. 1b, the traditional framework evaluates a model’s prediction against the ground truth at the input’s timestamp, implicitly assuming that predictions are available instantaneously6,7,35–39. Such an assumption neglects the inevitable delay between sensory input and computational output, thereby overlooking the critical influence of perception latency on real-time performance9–13.

To address this, as shown in Fig. 1c, STARE employs a Latency-Aware Evaluation protocol, which simulates the interaction between the perception model and the downstream application. The core of this protocol involves comparing sparse model outputs against a dense sequence of high-frequency ground truth annotations, with each annotation representing a real-time query. For any query at time t query, the most recent perception result prior to that time (i.e., t output ≤ t query) is retrieved to calculate the performance metric (e.g., AUC, IoU).

This mechanism inherently highlights the impact of latency. When a model exhibits high latency, its sparse outputs result in a single prediction being repeatedly utilized as the best available estimate for a sequence of queries. For example, as shown in Fig. 1c, the perception result generated at time t i+2 is reused to evaluate queries after t i+2, such as t j where t i+2 < t j < t i+3, thereby accumulating error over time. This approach provides a direct quantification of the performance degradation attributable to perception latency, yielding a faithful assessment of a model’s suitability for real-time deployment.

The stream-based latency-aware evaluation dataset
To enable rigorous validation of the STARE framework and address the limitations of existing event datasets, which lack dense temporal annotations for Latency-Aware Evaluation, we introduce the ESOT500 dataset. ESOT500 comprises two subsets with distinct sensor resolutions: ESOT500-L (346×260) and ESOT500-H (1280×720), covering diverse indoor/outdoor scenes, object classes, and challenging environmental conditions (Fig. 2a–b). A comparative analysis with related event datasets is presented in Fig. 2c, while detailed protocols for data capture, annotation, training/test splits, and implementation are provided in Supplementary Note 3. Latency-Aware Evaluation (a core goal of STARE) requires two critical dataset attributes, as identified in prior analysis: (1) dense ground truth to simulate the high-frequency queries of real-world downstream applications (e.g., robotic control)40–44, and (2) high temporal resolution to capture high-dynamic object motion without the temporal aliasing inherent in low-frequency annotations34,45. ESOT500 is purpose-built to satisfy both requirements via its 500 Hz annotation rate, as elaborated below.

Simulating dense downstream queries with 500 Hz annotations. The 500 Hz annotation frequency, which is ESOT500’s defining feature, generates a dense ground truth sequence that mimics the continuous, high-rate query demands of downstream applications40–44. This enables STARE to directly assess latency-induced accuracy: by comparing a model’s sparse, timestamped outputs with ESOT500’s dense query stream, the framework captures how latency causes predictions to become outdated relative to real-time task demands, which would be impractical with low-frequency datasets.

High-fidelity capture of high-dynamic object motion. The 500 Hz annotation rate enables high-fidelity capture of high-dynamic object motion, addressing a critical shortcoming of low-frequency datasets, which struggle to resolve rapid, transient motion transitions. From an information-theoretic perspective, sampling continuous high-dynamic motion at discrete, low frequencies acts as a lossy channel45: the original fast-changing trajectory serves as the input signal, while sparse annotations produce degraded outputs that omit key motion details. As the speed or variability of object motion increases, this degradation becomes more severe (Fig. 3a)34, leading to substantial discrepancies between the true high-dynamic motion and trajectories interpolated from low-frequency sparse annotations (Fig. 3e).

To quantify this distortion, we introduce the Reconstruction Error (RE) metric, which measures motion information loss at low sampling frequencies by: (1) downsampling ESOT500’s 500 Hz ground truth to a target frequency; (2) reconstructing the trajectory via linear interpolation; and (3) computing the error relative to the original 500 Hz data. As shown in Fig. 3b–d, RE is pronounced at conventional low frequencies (e.g., 25 Hz) but gradually declines as the sampling rate approaches 500 Hz. Fig. 3e presents several visual examples. These results confirm that high temporal resolution is critical to preserve the integrity of high-dynamic motion. The RE curve is expected to be even steeper in more complex tasks (e.g., multi-object detection in dynamic scenes6, 6D pose estimation for high-speed targets25, visual odometry in rapid locomotion7), as higher degrees of freedom introduce greater uncertainty. These results provide quantitative evidence that ESOT500’s high-frequency annotations are crucial for validating event-driven models in high-dynamic scenarios. A formal definition of RE is provided in Section “Reconstruction Error” (Methods).

Experimental results and analysis
Event-driven perception evaluation with STARE. To quantify the impact of perception latency on event-driven perception, we leverage the STARE framework to evaluate state-of-the-art models. This evaluation centers on the ESOT500 dataset, tailored for high-dynamic, low-latency assessment, and extends to external benchmarks to verify generalizability.

Experimental setting. We selected representative trackers spanning diverse model families, offline accuracy, and inference speeds (Fig. 4a–b): Siamese-based (PrDiMP1846), Transformer-based (MixFormer47), GNN-based (KeepTrack48), RNN-based (KYS49) and Segmentation-Centric (RTS50). We also incorporated models specifically designed for event streams, such as those operating directly on raw events (EGT29), grid-like representations21 (HDETrack5), and a recent cross-modal tracker using the Mamba architecture (Mamba-FETrackV251). Trackers were assessed under two paradigms: (a) Traditional framework: Event streams were preprocessed into 20 Hz fixed-rate frames (with varying event durations per frame)22. (b) STARE framework: Trackers used Continuous Sampling, processing the most recent events within different sampling window sizes. More details about the experimental settings can be found in Supplementary Note 1.

Evaluation on ESOT500. The results, summarized in Fig. 4a–b, reveal two key findings. First, most models experienced a substantial accuracy drop by up to 50% when moving from the traditional framework to STARE. This demonstrates that accuracy degradation caused by perception latency has been largely overlooked by the traditional evaluation framework. While the accuracy drop was generally observed, the results under STARE highlighted the superiority of some lightweight models over heavy ones. As shown in Fig. 4f–h, some models that achieved higher offline accuracy in the traditional framework were outperformed by more lightweight models under STARE. For example, MixFormer47 outperformed KeepTrack48 under STARE, whereas KeepTrack performed better in the traditional framework. This result is consistent with our real-world robotic ping-pong experiment in Fig. 5c. Notably, such performance ranking reversals are observed even on our RTX 3090 GPU with high computational power. It is reasonable to infer that the impact of latency is much more pronounced on resource-constrained devices, which are common in robotics and autonomous systems52,53. These findings provide direct experimental evidence that the traditional framework can yield misleading conclusions about a model’s practical capability, underscoring the necessity of a latency-aware evaluation paradigm such as STARE.

Second, the performance of most trackers under STARE followed a unimodal trend with respect to the sampling window size. The average peak performance appeared around a 20 ms window, reflecting a trade-off: longer windows gather more information but risk redundancy, while shorter windows avoid redundancy but miss critical data. Drawing on this finding, we noticed the effectiveness of Continuous Sampling was influenced by sampling window size. Hence, we made a deeper observation on Continuous Sampling by comparing the model performance under STARE and a frame-based latency-aware evaluation. Under the frame-based latency-aware evaluation, the event stream was converted into event frames in advance, and the model was constrained to sample events only at the discrete timestamps of these frames. More details about the frame-based latency-aware evaluation can be found in Section “Formal Definition” (Method). In contrast, under STARE with Continuous Sampling, the model sampled and processed the most recent events immediately after the previous processing was complete, not requiring any fixed-rate preprocessing. We evaluated model under these two settings with varying sampling window size. As shown in Fig. 4e, Continuous Sampling improved the model’s online accuracy by 51–129% with various sampling windows, demonstrating that Continuous Sampling is able to improve the model throughput thereby enhances accuracy by leveraging the continuity of event data. Furthermore, Continuous Sampling worked the most effectively with a sampling window of 50 ms, which best balanced window size and information richness on our dataset. It revealed that the window size in Continuous Sampling should be carefully considered to unlock a model’s full real-time potential.

The above experiments highlighted the impact of latency on model performance. To comprehensively evaluate a model under STARE in different latency conditions, we conducted experiments using a latency simulator, different physical GPU & CPU configurations, and resource contention from a parallel task. The results are shown in Fig. 4i–l, respectively. In all cases, we found that the accuracy dropped consistently under STARE as latency increased, demonstrating the consistent negative impact of perception latency on model performance.

Evaluation on other datasets. Apart from the proposed ESOT500 dataset, we also applied STARE to two other commonly-used event-based tracking datasets: FE10821 and VisEvent24. As shown in Fig. 4c–d, the performance of models on these datasets also followed the unimodal distribution with respect to the sampling window size, similarly to our findings on ESOT500. It is worth noting that the optimal sampling window for Continuous Sampling on FE108 (e.g., 20 ms for KYS49) is much shorter than that on VisEvent (e.g., 50 ms for KYS). This is likely because the objects in FE108 are hung on a stick and swung, generating high-density events. As a result, a short sampling window in this case would lead to more proper information aggregation.

Evaluation on ping-pong robot player with STARE. To examine the real-world impact of perception latency, we developed an event-based ping-pong robot player. This platform enabled us to test the immediate impact of latency on applications that demand rapid responses. The system consisted of a robotic arm holding a bat, an event camera, a ball launcher, an event-driven tracker, and a robot control policy. These components formed a tightly coupled perception-action loop. The event camera recorded the ball’s trajectory at 1 MHz. The tracker, operating at tens of Hz, requested the latest event data and provided estimated 3D positions of the ball. The robot control policy queried the tracker at 2 kHz, computing hitting trajectories, and sent control commands to the arm at 200 Hz, guiding it to strike the ball. The complete setup is illustrated in Fig. 5a. An example of a successful robot hitting back a ping-pong ball is shown in Fig. 5b. More details are provided in Supplementary Note 2.

To quantify the impact of perception latency on the ping-pong robot player, we conducted experiments under five conditions with results summarized in Fig. 5c. In the first three settings, we applied STARE to the MixFormer47 model and simulated different levels of perception latency by varying the model processing speed. Increasing the processing speed from 23.0 Hz to 55.3 Hz improved the success rate from 0 to 7 out of 20 trials, demonstrating that reduced latency contributed to real-world task success. The fourth setting served as a critical ablation study to STARE by removing Continuous Sampling operation. When the event stream was converted into 40 Hz frames, the model success rate dropped to 2 out of 20 trials, lower than the second setting (5 out of 20) with a close processing speed but with Continuous Sampling operation. This demonstrated that the traditional frame-based online setting failed to unlock the model’s real-time potential. In the final setting, we selected KeepTrack48 that has a higher traditional offline accuracy than MixFormer (62.87 AUC vs 61.56 AUC) but has a lower processing speed (33.2 Hz). KeepTrack achieved a lower task success rate (1 out of 20 trials), highlighting the negative impact of high processing latency even though the model had a higher offline accuracy.

Overall, these findings emphasize two essential requirements for evaluating event-driven systems. First, perception latency should be explicitly considered in evaluation to ensure that performance metrics reflect realistic deployment conditions rather than idealized assumptions. Second, to fully realize the capabilities of event-driven models, the evaluation methodology should preserve the continuous nature of event streams instead of constraining them into fixed-rate frames, which can suppress real-time potential even when the nominal throughput appears high.

Strategies for continuous event-driven perception
To mitigate latency-induced performance loss while aligning with the intrinsic continuity of event streams, we propose two targeted model enhancement strategies: Asynchronous Tracking (boosts throughput via dual-model collaboration) and Context-Aware Sampling (optimizes input efficiency via dynamic event selection).

Asynchronous tracking. Increasing model throughput is a direct way to produce more timely predictions, which is critical for satisfying the high-frequency query demands from downstream applications (e.g., robotic control). Event streams inherently provide continuous visual information at near-arbitrary timestamps, theoretically supporting ultra-high throughput. But this potential is not fully utilized by single-model architectures, which are limited by the computational latency of feature extraction. To leverage this continuity, we develop Asynchronous Tracking: a dual-model framework that uses a lightweight residual model to recursively update predictions from a heavyweight base model, thereby boosting throughput without sacrificing accuracy.

Figure 6a details the architecture: the heavyweight base model extracts high-quality features from the event stream to estimate target positions, but results in significant computational latency. While the base model runs inference, the lightweight residual model processes the latest incoming events to refine the base model’s features and predictions in real time. This design has two key advantages: it avoids redundant, computationally expensive feature extraction (by reusing the base model’s shared features) and integrates the most recent event information (preserving temporal continuity).

Experimental validation (Fig. 7a) confirms the strategy’s efficacy: Asynchronous Tracking increases model throughput by 78% (from 118 to 210 Hz) and lifts latency-aware accuracy by 60% (from 31.83 to 51.06 AUC). For comparison, we also explored a single-model alternative, the Predictive Motion Extrapolation, which extrapolates future target positions via velocity vectors output by the base model (details in Method Section “Predictive Motion Extrapolation”). Both strategies aim to boost throughput by updating base model predictions, but Asynchronous Tracking outperforms Predictive Motion Extrapolation across all settings (Fig. 7b). We attribute this performance gap to the limitations of motion extrapolation in high-dynamic scenarios, particularly critical for ESOT500’s 500 Hz annotated high-dynamic motion. To quantify this, we define the unpredictability score (negative accuracy gain of Predictive Motion Extrapolation relative to a non-predictive baseline), where higher scores indicate more unpredictable, namely high-dynamic motion. As shown in Fig. 7b, a model achieves sharply decreased accuracy while unpredictability scores increase, confirming why predictive motion extrapolation underperforms.

Context-aware sampling. Our earlier STARE evaluations (Section “Event-Driven Perception Evaluation with STARE”) revealed a key insight: model performance exhibits a unimodal distribution with respect to event sampling window size. Specifically, too small a window lacks sufficient motion information, while too large a window introduces redundant information. This finding motivated Context-Aware Sampling: a dynamic input refinement strategy that adapts the sampling window while activating/deactivating model inference based on event density surrounding the target’s last known location. The goal is to balance computational efficiency (avoiding unnecessary and inaccurate inference) and perception accuracy (preventing the target from slowly drifting).

Qualitative examples in Fig. 6b–c illustrate the mechanism: When event density is low (indicating slow or no motion, shown in Fig. 6b), inference is deactivated, and the last valid prediction is reused. This can reduce computational latency without accuracy loss. To avoid target drift from prolonged deactivation (Fig. 6c), the strategy enforces periodic reactivation to re-localize the target using newly accumulated events. The detailed formulation of the event density threshold and reactivation interval is provided in Section “Context-Aware Sampling” (Method). Experimental results (Fig. 7a, c) confirm its effectiveness: Context-Aware Sampling improves performance across most settings, and when combined with Asynchronous Tracking, achieves a 61% accuracy gain (from 31.83 to 51.12 AUC), as shown in Fig. 7a. Its effectiveness is most pronounced in sparse-event scenarios. Fig. 7c shows the strategy lifts accuracy by over 51% from an average of 18.73 to 28.29 AUC, demonstrating its robustness in challenging scenarios with sparse events.

Methods
In this section, we employ VOT as a concrete example to formally describe the STARE framework and compare it with latency-ignored and frame-based approaches.

Framework
Formal definition
Preliminary and Notation. Given an event stream

E = {e i } N i= e 0 , (1)

we regard the sampled event stream segment as

E l,r = {e i } r i=l , (2)

where e i = (xi, t i, p i), including event coordinate, timestamp and polarity; N e is the max event index and N e+1 is the total events number of an event stream; l, r are event indexes, corresponding to the left and right bounds of the sample event stream segment. In this work, the basic sampling strategy can be divided into two types, one fixes the time duration of sampled event stream, while another fixes the sample event number. Following that, the indexes l, r can be achieved by two indexing functions:

1) Given the window size of sampling time duration L, the time-fixed indexing function φ τ can be defined as

[l τ, r τ] = φ τ (E, t, L), (3)

where

r τ = max{i | (xi, t i, p i) ∈ E, t i ≤ t}, (4)

l τ = min{i | (xi, t i, p i) ∈ E, t i ≥ max(t − L, t 0)}: (5)

2) Given the window size of sampled events number N, the number-fixed indexing function φ μ can be defined as

[l μ, r μ] = φ μ (E, t, N), (6)

where

r μ = max{i | (xi, t i, p i) ∈ E, t i ≤ t}, (7)

where l μ = max(r μ − N, 0): (8)

Before feeding the sampled event stream E l,r into the perception model, we need to convert the event stream into a specific representation that the model can process. The general conversion process is denoted by the function χ

Eb l,r = χ(E l,r): (9)

As to the relevant event representations, we make a brief description in Supplementary Note 1.

As illustrated in Section “Results”, to meet the Latency-Aware Evaluation paradigm, each ground truth bounding box Bj within an event stream sequence is attached with a timestamp t j. Specifically, we assume that the time starts from zero, and the total time duration of an event stream is T. We mark the ground truth set as

Q = {(Bj, t j)} N j= gt 0 , (10)

where N gt is the max ground truth index and N gt+1 is the total number of ground truth and B 0 serves as the template bounding box for tracking. If the sampling frequency is H segments per second, we can calculate that

N gt = bT ∗ Hc − 1: (11)

Frame-based Latency-ignored Evaluation. As discussed in Section “Introduction” and illustrated in Fig. 8, frame-based latency-ignored evaluation involves processing preprocessed event frames one by one to calculate the offline accuracy. We mark the sequence of event frames as

Eb f = {Eb l i, r i} N i= f 0 , (12)

where N f is the max event frame index and N f+1 is the total number of event frames. With the introduced notations above, we can get

N f = N gt, (13)

and

[l i, r i] = φ τ (E, i/H + 1, L): (14)

Since each event frame in Eb f should be fed into the model, we can get the output bounding box set described as

Q offline = {B b k} N k f = 1, (15)

and the corresponding output-ground-truth pairs set described as

D offline = {(Bj, B b j) | Bj ∈ Q, B b j ∈ Q offline} N j= gt 1 : (16)

With D offline, we can calculate standard accuracy metrics, such as AUC and AP.

Frame-based Latency-Aware Evaluation. As illustrated in Fig. 9, the frame-based latency-aware evaluation framework runs models on fixed-rate frames, with event sampling restricted to periodic, fixed timestamps (e.g., t k, t k+1). This design introduces a misalignment between the model’s processing latency and the frame interval, leading to unavoidable idle time periods where the model is forced to wait for the next frame before processing new events.

The outputs contain not only the bounding boxes but also corresponding output timestamps for latency-aware evaluation. The output set can be marked as

Qb streaming = {(B b k, bt k)} N k s = streaming 1, (17)

where N streaming is the max output index and the total number of timestamped outputs. To implement the latency-aware evaluation, we still need to match each item in Q with an item in Qb streaming. As shown in Fig. 9, the matching principle and the matched output-ground-truth pair set D streaming can be described as

D streaming = {(Bj, B b k j) | (Bj, t j) ∈ Q, (B b k j, bt k j) ∈ Qb streaming} N j= gt 1 , (18)

where each item in D streaming satisfies

k j = max{k | bt k ≤ t j, 8(B b k, bt k) ∈ Qb streaming}: (19)

With D streaming, we can calculate metrics in the same way as frame-based latency-ignore evaluation, however, to show the latency-aware results.

Stream-based Latency-Aware Evaluation. As shown in Fig. 1c, in STARE, perception models get the input from continuous event streams instead of discrete frames. Algo. 1 of Supplementary Note 5 illustrates the stream-based tracking process implemented in our work.

In general, we start by setting the first ground truth bounding box B 0’s timestamp as the current world time t curr. The stream-based tracking process then begins, wherein each cycle, we sample the event stream using the indexing function φ based on t curr and the sampling window ω. The event stream E l,r is then converted into Eb l,r with the specific event representation χ. We then feed Eb l,r into the perception model to predict a new bounding box and return the computational latency Δt. Then, we update t curr, store the new timestamped output in Qb STARE, and repeat the process. Notably, the perception model with B 0 is initialized.

According to the tracking, we can get the timestamped bounding boxes, described as

Qb STARE = {(B b k, bt k)} N k S = T 1 ARE, (20)

where the N STARE is the max output index and also the total number of outputs. The method of aligning Q and Qb STARE is consistent with the frame-based latency-aware evaluation. The matched output-ground-truth pair set D STARE can be described as

D STARE = {(Bj, B b k j) | (Bj, t j) ∈ Q, (B b k j, bt k j) ∈ Qb STARE} N j= gt 1 , (21)

where for each item in D STARE satisfies

k j = max{k | bt k ≤ t j, 8(B b k, bt k) ∈ Qb STARE}: (22)

We can use the same way to calculate accuracy metrics as the frame-based latency-free evaluation with D STARE, to get the perception model’s performance on stream-based tracking.

Analysis. From the above description, we can see that for a given event stream E and sampling setting (fps and window size), both frame-based latency-ignore evaluation and frame-based latency-aware evaluations share the same event frame set Eb f as input. But the frame-based latency-aware evaluation will skip some coming frames. Therefore, we can know that

N streaming ≤ N gt − 1, (23)

with equality if and only if

Δt ≤ 1/H, (24)

where Δt is the model’s processing latency and 1/H is the frame interval. In the frame-based latency-aware evaluation, if the model is currently processing the input, the coming frames will be skipped. When the model finishes the inference, it briefly idles while waiting for the next frame. Consequently, (19) can be simplified as

k j = j − 1: (25)

On the contrary, the STARE framework schedules the perception model to operate on the event stream continuously, called Continuous Sampling, avoiding idle time caused by waiting for subsequent frames. The perception model can instantly receive input right after generating the current bounding box, which leads to a higher frequency of outputs and maximally exploits the real-time capability of the perception system.

With the same algorithm for tracking and evaluation as in frame-based latency-aware evaluation, we can get

N streaming ≤ N STARE, (26)

with equality if and only if

Δt = λ ∗ 1/H, λ ∈ Z+: (27)

Moreover, in the STARE framework, it follows that

N gt − 1 < N STARE, if Δt < 1/H: (28)

In the real-world online scenarios, both H and N gt can be considered infinite, while Δt cannot be equal to zero. However, in computer-simulated evaluations, they are all finite. Thus, a higher value of H and N gt aids in a more precise evaluation of a perception model’s real-time performance, to some extent motivating the creation of our ESOT500 dataset.

Reconstruction error
A significant secondary benefit of ESOT500’s 500 Hz annotation rate is its ability to faithfully capture high-dynamic object motion without the temporal aliasing34 inherent in lower-frequency datasets. Furthermore, the process of recording a continuous real-world motion trajectory at a discrete frequency can be viewed as a lossy information channel45. The original, continuous trajectory serves as the input signal, while the set of low-frequency annotations is the channel’s output. The fundamental question is: how much information about the original signal is preserved after passing through this channel?

To quantify this, we define the Reconstruction Error (RE), which serves as a practical measure of the channel’s fidelity. The RE evaluates our ability to reconstruct the original high-frequency trajectory B high, given only a downsampled set of low-frequency annotations B low. Specifically, for a given low frequency f low, we generate B low by subsampling B high. We then use a linear interpolator L to reconstruct the trajectory between every two consecutive low-frequency samples Bt0 i and Bt0 i+1. The RE at frequency f low is then the accumulated loss between the reconstructed trajectory B recon and the original high-frequency ground truth:

RE(f low) = E t∈[t0 i, t0 i+1] [Loss(Bt, L((Bt0 i, Bt0 i+1), t))], (29)

where Loss(⋅, ⋅) is a metric such as 1 − IoU. Here we adopt L2 distance.

A high reconstruction error signifies that the sampling frequency is too low to capture the signal’s complexity, leading to severe information loss, which is also known as temporal aliasing34. This error is thus a direct, quantitative measure of aliasing, reflecting the gap between the information content of the original signal.

Details and insights of model enhancement
Predictive motion extrapolation. We define the function κ to represent the linear compensation function based on model-predicted velocity, which can be formally described as

κ(B b, vb, Δt) = B b + vb ∗ Δt, (30)

where the latency Δt is a scalar, and B b and vb are both four-dimensional vectors, including the information of coordinates of the top-left corner and the size of the bounding box.

The predictive head is trained by adding the loss of predictive inference result B pred, which is obtained by Eq. (30), in addition to the loss of inference result of current input, B curr. The improved loss formula can be expressed as

Loss(B curr, B b curr) + Loss(B pred, B b pred), (31)

where

B b pred = κ(B b curr, vb curr, latency): (32)

The loss of B b pred and B b curr follows the same calculation formula defined by the specific perception model. Furthermore, considering the tracked object does not always move in a straight line, the parameter latency is set to a small value (e.g., 8 ms) to facilitate model convergence. In fact, the predictive module is more effective at predicting the short future, and when the required compensation time is too long, it can lead to incorrect predictions.

An intriguing finding during debugging was that simply equipping the model with predictive capability, even when its velocity prediction is not used for the final output, already boosts STARE performance. If we further use κ to adjust B b curr according to vb curr and Δt, the performance can be improved further.

Asynchronous tracking. The sequential perception pipeline described in Algo. 1 of Supplementary Note 5 inherently suffers from processing latency. During the inference time Δt of the perception model, any newly arriving events are queued and not immediately processed. This creates a temporal gap between the latest available sensory information and the model output, thus hindering true low-latency perception.

To mitigate this, we introduce an asynchronous tracking paradigm inspired by recent advancements in event-based vision9. As shown in Fig. 6a, this approach decouples the main tracking process into a heavyweight base model and a lightweight residual model. The base model is responsible for comprehensive feature extraction and state estimation, while the residual model provides rapid, incremental updates during the base model’s inference period.

Let the base model G base be composed of a feature extracting backbone G b and a prediction head G h. In a standard cycle, the backbone first extracts the feature map h curr from the input event segment Eb curr:

h curr = G b(Eb curr, B b last): (33)

Then, the prediction head G h computes the bounding box B b base from this feature map:

B b base = G h(h curr): (34)

While the base model is computing h curr and B b base, a new stream of events arrives. We term the latest events in this new stream, within the current sampling window, as the residual event segment E res. The residual model G res is a lightweight network designed to process these new events quickly. It takes the latest backbone feature h curr from the base model and the new residual event frame Eb res = χ(E res) as input to generate an updated feature map h updated:

h updated = G res(h curr, Eb res): (35)

This updated feature, which now incorporates the most recent motion information, is immediately fed into the base model’s prediction head G h to yield a refined, lower-latency bounding box B b async:

B b async = G h(h updated): (36)

Notably, we schedule the residual model based on the speed ratio between the base model and the residual model running in parallel. For example, we might execute the residual model three times within the processing interval of a single base model run to ensure the speed stability of the parallel running. By generating B b async during the inference period, this asynchronous approach effectively utilizes the computational downtime of the sequential perception loop, leading to a more timely and accurate perception of the object state.

Context-aware sampling
Event density calculation. Following the given notations, we give a formal description of the function ϕ to calculate the event density ρ with respect to the bounding box area B− of the last active output, which is defined as

ρ = ϕ(B−, E l,r) = X x∈B− ψ(x, E l,r), (37)

and

ψ(x, E l,r) = 1, if ∃(xi, t i, p i) ∈ E l,r, s.t. x = xi 0, otherwise (38)

The calculated density represents the number of event-triggered pixels within B−, instead of the full event number within B−. This is because, for perception models, whether there is an event triggered at a certain pixel position is far more critical than the total number of events triggered at that position.

Furthermore, for Eq. (37), it should be noted that since the reference event density is constantly updated and the size of B− has significant uncertainty, it is advisable not to divide the area of B− when calculating ρ.

Activated sampling judgement. With given notations, we give here a formal description of how we implement the active sampling judgement function J, as

J(ρ, ρ−, d) = true, if ρ > α ∗ ρ− true, if ρ > β ∗ ρ− and d > γ false, otherwise (39)

where ρ and ρ− represent the event density of current sampling and last active sampling; d represent the lasting time of being inactive; α and β represent the high and low threshold ratios; γ represent the duration required to trigger the low threshold β. In our experiments, α and β are typically set to 0.5 and 0.05 respectively, and γ is set to 50 ms.

As detailed in Section “Strategies for Continuous Event-driven Perception”, every inactive sampling gives additional event data to the next cycle of tracking, enhancing its accuracy. However, the state of being inactive should not last for too long period, unless the object is really motionless, so we set an upper bound γ.

Triggering of the strategy. Given that some perception models rely on previous outputs for prediction assistance (e.g., tracking algorithms require the last bounding box to define the search area), Context-Aware Sampling can enhance prediction result even if it is not frequently triggered in a sequence. We present the trigger ratio (or sparsity) and the corresponding STARE performance improvement for a subset of ESOT500 sequences with the setting of 2 ms sampling window in Fig. 7c. More details are available in Supplementary Note 8.

Modified STARE pipeline. The Algo. 1 of Supplementary Note 5 illustrates the STARE pipeline. The modified pipeline, extended to Predictive Motion Extrapolation (blue) and Context-Aware Sampling (orange), can be seen in Algo. 2 of Supplementary Note 5. The modified pipeline for Asynchronous Tracking can be seen in Algo. 3 of Supplementary Note 5.

In the provided algorithm, it is evident that the Context-Aware Sampling serves as an enhancement that is independent of specific tracking algorithms. This strategy is designed to mitigate issues originating from the input data or from suboptimal parameter configurations. Conversely, the Predictive Motion Extrapolation represents a form of enhancement that inherently depends on the neural network structure of the perception model, and its implementation approach and associated challenges vary across different kinds of perception models.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "The paper introduces a lot of terms that are undefined, which makes is *extremely* difficult to understand what the authors\nare trying to convey. Here is a partial list from reading just Section 1:\n* line 24: latency-free\n* line 25: pre-slice\n* line 27-28: temporal alignment between inputs and outputs\n* line 29: perceptual stage\n* line 31: outdated prediction (what is a prediction and how can it be outdated?)\n* line 64: hardware-induced temporal distortions (what are temporal distortions and how does hardware induce them?)\n* line 69: real-time constraints (what are they?)\n* line 71: computational readiness\n* line 74: perceptual query\n* line 78: output density, temporal misalignment\n* line 89: this temporal super-resolution (where is super-resolution so far?)\n* line 96-97: temporal coherence\n* line 97: physically grounded supervision signals (what are the signals? why are they supervised? why are they physically\ngrounded?)\n* line 109: retinotopic adaptability (retinotopic usually refers to the mapping between information on the retina and the visual\ncortex, but I don't think you meant that here; retinotopic adaptability is perhaps just a new term that you invented; you\nprobably meant to just say visual adaptation)\nFigure 1a is never properly explains. It would have nicely warmed up a reviewer who is not immediately familiar with your\nundefined terms with a proper explanation. Fundamentally, the writing is very implicit about the exact application scenario\nthe authors care about: can you describe, as an example, an end-to-end application that your proposed framework\nhelps/applies to? Can you also give an example of what you call \"latency-induced performance degradation\"? Otherwise,\nit's unclear what is the real motivation of the work.\nHow does the STARE framework handle situations where computational delays vary dynamically due to resource\ncontention (e.g., on shared hardware)?\nI know VOT is the focus of the paper, but can you discuss how STARE ca be adapted to other perception tasks beyond\nvisual object tracking, such as semantic segmentation or SLAM, and if so, what modifications would be necessary?\nTo what extent could Predictive Motion Extrapolation be improved using higher-order dynamics (e.g., acceleration-based\nmodels) instead of first-order velocity estimation?\nHow sensitive is the performance of Context-Aware Sampling to the chosen event density threshold and its decay strategy\nfor reactivation?\nAnother minor point about writing: almost every single paragraph is tens of lines long, which is makes it *extremely* difficult\nto get an overall structure of a section/subsection.", "reviewer_2": "I co-reviewed this manuscript with one of the reviewers who provided the listed reports. This is part of the Nature\nCommunications initiative to facilitate training in peer review and to provide appropriate recognition for Early Career\nResearchers who co-review manuscripts.", "reviewer_3": "The authors propose a well-motivated approach to evaluate neural network architectures working on DVS event streams.\nRightfully noting that current ML algorithms do not leverage the very high temporal resolution of these sensors and instead\nfall back to frame-based representations, their method takes inference time (latency) into account to provide a more practical\nevaluation of performance. The latter is enabled by the conception of an event-based dataset for visual object tracking with\nhigher temporal resolution (500 Hz) for the groundtruth than prior in the literature.\nThis dataset serves for an extensive benchmark of a wide range of algorithms, and it turns out that the performance hierarchy\nbetween state-of-the-art models is altered when evaluated in their framework. Finally, the authors propose two architectural\ninnovations adapted for their framework, that further boost the accuracy of their models.\nIt goes without saying that the number of contributions (annotated dataset, evaluation metrics, benchmark, etc.) is large and\nreflects a great amount of work. The authors did a good job at explaining hyperparameters and their code repository is clean\nand ready for reproducibility.\nI would like to first give some comments on the form:\n- in general, references should be more properly formatted, not only with the authors title and year, but also the doi, journal\netc. In particular, refs 11-13, 30, 28, 29, 36, 37 should be completed.\n- on this note, ref. 38 appears twice (same as ref. 22)\n- the reference of each dataset should be included in Fig.2a\n- still in Fig.2a, the authors must provide a more exhaustive comparison of the different datasets, by including key features\nthat should enlighten future researchers to choose one dataset over another. I am thinking about recording device, spatial\nresolution and total dataset length (e.g., in hours). This should be done even if it does not necessarily help\nadvertising the dataset. Authors used the DAVIS346 sensor, while more modern sensors have come out in the recent years.\n- Concerning the dataset, Fig.2b shows poor variety and RGB frames are dark, which is a heavy downside for any\nmultimodal approach in future research. Either the dataset has more environmental variety and other examples should\nreplace the current ones, or there should be a explanation in the caption or main text.\n- Still on Fig.2b, why are we only seeing one bounding box per frame ? There should be one for each object appearing in the\nscene. Are they present in the annotations ? Is it an aesthetic choice for the figure ?\n- I also have some problems with the writing style and paper structure overall. While the authors did a great job at introducing\nthe need for their approach, I do not see why very relevant other works are only found in supplementary materials.\n- please introduce the different models in section 2A \"evaluation metrics and STARE\"\n- line 237, the reference to Fig.1d is irrelevant.\n- please add a reference for the DAVIS346 neuromorphic sensor\n- Similarly, I don't think their experiments on FE108 should necessarily be in supplement, as they can also provide some\nvaluable insights.\n- missing reference for OSTrack in Table 1\n- In general, I think that the type of plots presenting experimental results (Fig.3 and 4b-e) could be better chosen. For\nexample, in Fig.3, the reader is presented with a variety of models each having different throughput (frequency). The latter\nbeing in the legend makes it hard to read and it is a pity that this feature is not used as an dimension in a plot (in an\naxis, as the marker diameter...)\n- Fig.3 has several panels d.\n- While the number of models and conditions tested is impressive, Table 1 is hard to read and should be more thoroughly\ndescribed in the results section. If a model is included in the benchmark, it must be motivated and described in the main text.\n- I would have appreciated a figure on both computational latency compensation and context aware sampling, since these\nare presented as major contributions.\n- the authors motivated both of these innovations well, but did not explain whether similar approaches have already been\nproposed in the neuromorphic sensing and computing litterature\n- unfinished sentenvce at line 570.\n- this is more subjective but I feel that the quality of most figures could be improved (I'm thinking notably about Fig.1c-d and\nFig.2c-d)\n- Fig.4e is not described in the text.\n- line 570, unfinished sentence.\nNow, as I understand it, the main contribution that enables all others is the dataset with its tight annotation. Since the\nannotations are first at low frequency and then interpolated at high frequency, I wonder how complex it is to enhance the\nannotation of any given dataset. I understand that it still requires manual curation, but if the bulk of the upgrade is done\nautomatically, wouldn't it be already possible to apply this to other, already existing datasets ?\nSecond, since authors argue that high-frequency movements are generally not captured in prior datasets, I deem it is\nimportant to establish more quantitatively their presence in the proposed dataset. Typically, I don't know whether sport\nequipments, bikes and bottles (?) really need 500 Hz annotation frequency. Conversely, the \"Bird\" class with rapid wing\nflapping presented as a motivation is not present in the categories of ESOT500. Additionally, if such rapid movements do\nindeed exist here, how do models perform specifically on these particular movements ? On slower ones ?\nI would also appreciate more details on the data acquisition, especially the camera movements and the types of scenes.\nAdditionally, how was the splitting done between train/test ? Based on which heuristics ?\nThird, do authors believe that their proposed computational latency compensation and context aware sampling could still be\nemployed in the traditional framework and improve performances (even if maybe not optimal) ? Could context aware\nsampling be useful for datasets with lower annotation frequency, but slow movements ?\nFurthermore, it is nice to see that context aware sampling manages to induce improvements in throughput, but what is the\nachieved sparsity ?\nWhat is the hardware platform ? What makes it a \"platform\" rather than a standard computer equipped with some GPUs ?\nWere there any specific innovation on the software or hardware sides to make it a good testbench for STARE.\nI am slightly surprised this approach is completely new, since it attempts to adress important concerns well known by\nresearchers in the field. This paper looks solid in terms of contribution and methodology, and its message is clear. However,\nit is currently in a quite unpolished state and an important number of modifications remain to be done for now before any\ndecision relative to publication or rejection can be made.", "reviewer_4": "The authors present a timely and much-needed approach for fair benchmarking in event-based vision, demonstrating that\naccuracy improves when event-frame slicing is aligned with motion in dynamic scenes. To address the limitations of fixed-\nrate event-frame slicing, they introduce STARE, a stream-based, latency-aware evaluation framework. This contribution may\npave the way for future methods aimed at more effectively evaluating event-driven data. The authors validate their approach\non object tracking datasets, reporting AUC performance for both STARE and traditional time-window slicing, with a\nperformance peak observed at a 20 ms sampling window.\nThe manuscript tackles a highly relevant challenge in the event-based vision community and is generally well written, with a\ncoherent and engaging narrative. I greatly appreciated reading it, as it addresses a critical issue in our field and convincingly\ndemonstrates the advantages of event-driven (ED) cameras. The work underscores the importance of fair comparisons that\nreflect the superior temporal resolution of ED systems. I consider this contribution both innovative and necessary for the\nadvancement of our domain.\nThat said, the Introduction covers a broad topic but would benefit from refinement and greater focus. Expanding it with more\nspecific context would help to better emphasise the significance of the research question. In particular, the distinctions\nbetween frame-based and stream-based approaches are not clearly articulated by page three, which may leave readers\nunfamiliar with the field struggling to follow the argument. In comparison, the remaining sections are more clearly written. I\nrecommend revising the Introduction to ensure the foundational concepts are well explained before delving into technical\ndetail.\nI would also highlight the following sentence as particularly effective: \"Neuromorphic perception must prioritize temporal\ncongruence over static-frame accuracy.\" This statement succinctly captures the manuscript’s central thesis and should be\nmore prominently featured earlier in the text, possibly guiding the reader through the rest of the paper.\nOne surprising omission is a thorough engagement with the neuromorphic literature on bioinspired temporal processing.\nGiven that the authors rely on such concepts to support their claims, the discussion of biological plausibility appears\nunderdeveloped. For instance, the statement that \"the bio-inspired strategies introduced in this work could inspire further\ninnovations in neuromorphic computing by emulating the predictive and adaptive mechanisms of biological systems\" lacks\nadequate support from relevant literature on biologically plausible models. As it stands, the connection feels overstated. To\nstrengthen this aspect, I recommend integrating a more substantive review of the neuromorphic work on visual attention and\nmotion estimation. For reference, I have included below a sample list of relevant studies that may serve as a starting point.\nThis list is not exhaustive, and additional relevant works can certainly be found:\nD’Angelo, G., Perrett, A., Iacono, M., Furber, S., & Bartolozzi, C. (2022). Event driven bio-inspired attentive system for the\niCub humanoid robot on SpiNNaker. Neuromorphic Computing and Engineering, 2(2), 024008.\nMilde, M. B., Bertrand, O. J., Ramachandran, H., Egelhaaf, M., & Chicca, E. (2018). Spiking elementary motion detector in\nneuromorphic systems. Neural computation, 30(9), 2384-2417.\nClerico, V., Snyder, S., Lohia, A., Kaiser, M. A. A., Schwartz, G., Jaiswal, A., & Parsa, M. (2024). Retina-inspired Object\nMotion Segmentation. arXiv preprint arXiv:2408.09454.\nD'Angelo, G. V., Clerico, V., Bartolozzi, C., Hoffmann, M., Furlong, P. M., & Hadjiivanov, A. (2025). Wandering around: A\nbioinspired approach to visual attention through object motion sensitivity. Neuromorphic Computing and Engineering.\nFor the reasons outlined above, a clear rationale for the selection of the Predictive Module and Context-Aware Sampling\nmodels should be added and briefly explained. This would clarify the level of bioinspiration involved and justify the\nnecessity of these mechanisms as the most suitable choices for this manuscript, as this is currently not evident.\nIn contrast, the section on the Neuromorphic Vision Dataset ESOT500 is, in my view, the most well-written and compelling\npart of the manuscript. For instance, the discussion of the 'upper limit on the duration of the inactive state' demonstrates a\ncommendable attention to detail, which strengthens the methodological rigour of the work.\nFinally, the placement of the figures is problematic. They appear to be positioned far from the corresponding text, making it\ndifficult to determine whether they are introduced in the correct order or context. I would also suggest improving the captions,\nas most details are not described.\nThe following detailed comments are offered with the aim of strengthening the manuscript:\nMinors:\n“Experiments demonstrate that conventional metrics overestimate … of latency-aware assessment.” This sentence is\nunclear. In what sense do they overestimate real-time performances?\nI believe the abstract is somewhat inconclusive. After reading it, I cannot fully grasp what you are specifically proposing, and\nsome of the sentences are unclear, “of event …latency-induced errors by 59%”. You talk about motion and context-aware\nsampling, and the last sentence is overly general. I’d consider rewriting the abstract.\n“Biological vision.. Fig. 1b.” There is a bit of a jump here, in the Figure, you do not clearly represent either biological vision or\nadaptive sensing.\n“ sensor’s inherent capacity.” typo, too much space between “sensor’” and “s”.\n“latency-free” feels odd; they do have latency, and it’s higher, which brings us to the advantage of using ED cameras. Also,\nthey could work online, but the problem would remain the latency. I’d rephrase this sentence as it seems confusing.\n“while ignoring the temporal alignment between inputs and outputs” what do you mean here?\n“event frames with latency compensation techniques.” Reference here please, also I’d briefly explain these techniques.\n“that mirrors biological vision’ s real-time constraints.” Same typo of big space after the apostrophe.\n“frame-based latency-free methods,” I am really not convinced by the term “latency-free”\n“This design enables microsecond level analysis of latency-induced performance degradation.” This sentence is confusing.\nWhat do you mean by performance degradation? I would be clearer on the specific performance you’re referring to→,\nlatency? accuracy?\n“super-resolution”, I’d say, high temporal resolution→ same here “, grounded supervision signals”\n“To accommodate frame-based …representations (2, 18).” The references here are not sufficient. I’d suggest briefly\nexplaining the content so that the reader doesn’t need to look up and read the articles just to understand your point.\nJust a small advice, I’d add the term STARE also into Figure 1b\n“Traditional approaches …fixed-rate frames (Fig. 1d)” English, I’d rephrase.\n“dichotomy that misrepresents real-world performance (27–31).” How? Please explain every affirmation.\nTable 1 is far away from where it is introduced in the text, the reader loses the flow.\n“underperform …query demands (Table 1).” Table 1 is gigantic, I’d briefly explain the important points and results from the\nTable to be discussed later, so that the reader can immediately spot the values and understand the merit of your work. I can\nsee you do it later, but going back and forth to read the names and look for those names in the Table made me lose the\npoint. I’d highlight those results better in the table (coloured box?). Also, please explain why it is impressive or significant\nthat “the low-FPS ToMP101 (32) can outperform the high-FPS ATOM (33), while the low-AUC ToMP50 (32) can surpass the\nhigh-AUC ToMP101”\n“(performed by 5 expert annotators)” The word “expert” feels odd here.\n“follow the [t, x, y,w, h] format with occlusion” It’s intuitive, but please explain the variables.\n“Predictive Motion Extrapolation which leverages motion-sensitive (24) event streams” ref should go after “streams”\n“Predictive Motion … critical visual inputs (25).” Again, there is no explanation about the reason behind these specific\nmethod choices.\n“efference copies” Please explain the term, not everyone is familiar with it (especially researchers not familiar with\nneuroscience and bioinspiration). Also, I’d add a reference for it, it is indeed important and aligns well with the temporal\ndomain of ed cameras.\n“500Hz and mirror the millisecond…biological motor control,” unless you add a specific reference, this might feel too vague.\nIs it always the case? (might not be true for specific cases)\n“Figs. 4b, c & e reveal … of accuracy-speed trade-offs.” This point is important and should somehow come out more clearly\nat the end of the Introduction.\n“Furthermore, several challenging …, including Table. 1, Figs.” The word biological plausibility feels a bit too much here, in\nwhat way you have presented any biological, real neuroscientific data, and how did you compare to those? I’d rather not use\nit here, and I’d try to be more specific on the task at hand. There are different levels of abstraction in bioinspired models, and\nbiological plausibility would need more justification and description; I’d just avoid it.\nPlease define AUC: Area Under the Curve at the start.\n“to optimize model inputs.” …. while computing the performance…\n“ effectiveness of our .. compensation strategies.” It is not clear here what compensation is.\n“As mentioned above, …perception models in an offline latency-free manner.” This is repeated over and over throughout the\nmanuscript, it’s redundant. (Again, latency-free it’s an odd terminology to describe fixed frame-rate)\n“Moreover, the phenomena …..2 to 3 percentage points.” I’d improve English here.\n“drops of up to 60” no period here.\nIn Eq. 1, what is Ne? E. 12 same problem, I think I got it, but please clarify any variable. All the equations need a detailed\nexplanation of all the variables.\n“Algo. illustrates the stream-based tracking process implemented in our work.” Something is wrong here; please check it.\n“to Predictive Module (blue) … Algo. 6. In the provided algorithm” I’d add the same colour-coded explanation in the caption\nof the Algo.\nMajors:\n“faster algorithms may achieve… temporal misalignment.” So this is indeed an important point to discuss, and I don’t think\nit’s properly addressed here. If this is your research question and what you prove with your experiments, I would expand this\npoint as it would arguably give merit to your work and would definitely benefit the entire event-driven community.\nAgain, here, “Our experiments …. fragility in event-based perception.”, the reader does not remember exactly all the details,\nand it is not clearly understood what exactly you mean by “systemic fragility in event-based perception” and what the drops\nrefer to. It would be important to restructure the introduction and specify in detail the story that you want to convey with this\nwork. As a result, “In order to address this fragility, we derive two … and Context-Aware Sampling.” The reader does not\nunderstand the need for these two biological mechanisms to address a fragility that is not properly explained before.\n“Context-Aware Sampling…adaptability of biological vision”, again, here you are dropping a term that would need to be\nexplained to be understood. What kind of retinotopic adaptability? There is an entire field on biologically plausible motion\ndetection and visual attention with neuromorphic sensing, AND also computing that is completely missed to be mentioned\nhere. Why did you choose 24? Why that one specifically? Also, there are plenty of event-based and spiking neural networks\non visual attention mechanisms. It would be important to understand the reasons behind your choices.\n“maximizes output density to match consecutive queries.” How do you do that?\nI’m on the third page, and it is still not clear what the authors are representing in Figure 1 a) and b). Why should 1 b be\ncontinuous when I can clearly see that the time windows to create the event frames are not consecutive? I actually would\nassume that is true for Figure 1a). Also, why in Figure 1a) is there no representation of the event frames for the time windows\nafter the first one (they are green squares)?\n1c) if you call it latency-free and streaming perception, why do I not find streaming perception in Figure 1c? What is F?\n(frame)? Please explain every little detail of your Figures. There is no explanation for the Idle.\nFigure 1: Again, the term latency-free is odd. Anyway, (Figure 1a)) if it is in fixed rate, why is there that big portion of time in\nthe middle (I assume that is time, please put the units everywhere)? Figure 1b), again, here given the perspective is not clear\nwhether the time windows are the same and spaced in time equally, also, why is it not continuous? Somehow, in Figure 1b\nseems that the past frame gives the latency-aware perception for the current output time. Is this true?\nFigure 2: No explanation of the colour lines, green and blue dashed in 2d. What is on the y-axis? What is the unit of the x-\naxis? How should I read this figure?\nAn important consideration when using event-based cameras is power consumption. While this may be beyond the scope of\nthe current work, it would still be worth mentioning. Would anything change if STARE were used instead? → You later talk\nabout it, which makes my comment even more important: “To explore a wider range .. of experiments in Fig. 3d:”\nGiven the clear categorisation in Figure 2d, it would be valuable to compare the accuracy across these categories, with and\nwithout using STARE. This could provide insight into which subjects or objects are most affected by the loss in accuracy.\n“The 59% latency-aware accuracy… through event density modulation” I do not find the 59% accuracy the authors are\nreferring to; I don’t understand if I am missing something here. In what way are you validating the foveal-peripheral\nadaptation? There is no spatio-temporal distribution of log-polar receptive fields, not even an approximation. Why do you say\nthat?", "reviewer_4_code": "I didn’t have time to thoroughly test the code, but the repository appears well organised. One concern is that it states\ncompatibility only with Linux, which could be a limitation for users on other operating systems. It would be helpful to clarify\nwhether the code can be run on other platforms as well."}, "author_response": "Point by point response on NCOMMS-25-16641A\nTheauthorswouldliketothanktheeditorandreviewersfortheirconstructivecommentsandsug-\ngestions that have helped improve the quality of this manuscript. The manuscript has undergone\na thorough revision according to the editor and reviewers’ comments. Please see below our re-\nsponses. For the reviewers’ convenience"}]

Decision letter