In-sensor analog optoelectronic processing of concurrent event and memory signals for dynamic vision sensing

Domain: chemistry
Published: 2025-12-26
Editorial outcome: major_revision

Abstract
Efficient dynamic vision requires capturing instantaneous changes and temporal context, yet existing image and event sensors rely on power-hungry digital processing. Here, we introduce an in-sensor dual-response architecture that concurrently generates analog event spikes and persistent memory tails. A prototype sensor integrates phosphor pairs with silicon photodiodes and transimpedance amplifiers to achieve microsecond- and millisecond-scale dual kinetics. Measurements during light-emitting diode replay reconstruct event frames that match software frame differences, while the slow channel behaves as a linear reservoir of motion history. A single memory frame fed to a convolutional neural network enables accurate classification of human actions (93.1%) and vehicle trajectories (98.0%), as well as speed estimation with errors of 2.15 km/h. Integration with a compressive optical neural network frontend mapping 4900 inputs to 16 per frame yields 93.3% action classification accuracy. By eliminating analog-to-digital conversion and digital accumulation, this approach enables ultralow-latency, ultralow-power neuromorphic vision.

Introduction
Efficiently perceiving dynamic scenes in real time is critical for applications such as robotics, autonomous driving, and intelligent surveillance1–6. Traditional frame-based vision sensors capture full images at fixed intervals (Fig. 1a, top), providing rich spatial context but suffering from high data redundancy: static backgrounds are repeatedly processed, elevating bandwidth requirements, power consumption, and processing latency5,7–9.

Event-based vision sensors—also known as neuromorphic or dynamic vision sensors (DVSs)—have emerged as compelling alternatives. Unlike conventional imagers, DVSs asynchronously output pixel-level brightness changes, creating sparse event streams characterized by microsecond latency, high temporal resolution, and high dynamic range (Fig. 1a, middle)10–13. By filtering out redundant static information, DVSs drastically reduce data throughput and energy consumption while minimizing motion blur during rapid movements12,14. These advantages have positioned event-based sensors as key enabling technologies for energy-constrained, real-time tasks such as robotic navigation and automotive safety.

However, DVS outputs are inherently stateless, limiting their ability to capture persistent context or motion trajectories15,16. In conventional pipelines (Fig. 1a, b), motion persistence is reconstructed by (i) computing pixel-wise differences between raw frames (RFs) to produce event frames (EFs)—or directly capturing EFs with a DVS—(ii) converting those event signals via analog-to-digital conversion (ADC), and (iii) digitally accumulating them into memory frames (MFs). Such multi-step digital processing introduces additional latency, power consumption, and hardware complexity12,15,17–19. Hybrid/event-intensity sensors such as Dynamic and Active-pixel Vision Sensor (DAVIS) and Asynchronous Time-based Image Sensor (ATIS) provide both asynchronous events and absolute intensity12,20–22. Recovering longer-term temporal context, however, typically requires off-sensor buffering or digital accumulation, which increases system-level latency and complexity.

Recent advances have shown that embedding short-term memory functionality directly within the sensor can greatly enhance dynamic perception9,10,23–25. Retinomorphic photomemristor arrays10 and 2-dimensional (2D) heterostructure-based sensors26–28 inherently encode temporal motion history, enabling “reservoir-in-sensor” processing that eliminates redundant data transfer and provides instantaneous access to accumulated motion context. Yet most existing implementations rely on luminescent or photoconductive materials (e.g., long-persistent phosphors or cumulative photoconductors) whose memory responses are primarily triggered by externally encoded event spike-like pulses, reflecting per-pixel intensity changes29–34. This limitation prevents individual pixels from capturing both instantaneous events and persistent motion context in a single stage. Extending in-sensor temporal processing beyond conventional DVS pipelines, a retinomorphic photodiode consolidates event sensing, band-pass dynamics, and light adaptation within a single diode and reports a dynamic range exceeding 200 dB35, demonstrating retina-like temporal behavior under extreme lighting. Event-driven sensors with in-sensor computation generate programmable, amplitude-encoded spikes using dual-polarity photodiodes for in-sensor spiking-neural-network operations7. These advances validate the promise of in-sensor temporal processing, yet their external interfaces remain predominantly spike-centric, with longer-term temporal context typically reconstructed off-sensor via event accumulation or windowing.

Here, we introduce an in-sensor dual-response pixel architecture that executes instantaneous event detection and temporal integration directly in the analog domain at the pixel level (Fig. 1c). Each pixel comprises two parallel photosensor channels that differ in their fast-response kinetics and in the amplitude of their persistent luminescence tails (ms scale). A real-time analog differential measurement of their outputs yields (i) event spikes whenever illumination changes, arising from mismatches in fast response kinetics, and (ii) memory tails that persist after each spike, arising from differences in slow-rise and slow-decay dynamics. This approach dramatically reduces system latency, power consumption, and complexity.

We validate our architecture on the Weizmann Human Action dataset, achieving a structural similarity index (SSIM) of 0.94 for reconstructed EFs and good agreement with a linear reservoir model36,37 for memory signals. When fed into lightweight convolutional neural networks (CNNs), our MF input outperforms event- or frame-only baselines in action recognition. We further demonstrate that single MF inputs enable vehicle trajectory classification with 98% accuracy and speed estimation with a mean absolute error (MAE) of 2.15 km/h. Finally, an optical neural network (ONN)-based compressive encoder38–42 (4900→16) achieves 93.3% accuracy, enabling efficient analog feature encoding. This architecture eliminates the need for ADC and digital accumulation, reduces end-to-end latency by more than an order of magnitude, and projects per-channel power below 1 mW—marking a significant step toward ultralow-latency, ultralow-power neuromorphic vision.

Results
Event and memory signal detection by response speed mismatch

Figure 2a illustrates the pixel-level sensor architecture, hereafter referred to as an analog event-memory sensor (AEMS). Sensors A and B incorporate silicate (Sr2SiO4:Eu2+) and garnet (Lu3Al5O12:Ce3+) phosphors, respectively, each coupled to a Si photodiode (PD) and transimpedance amplifier (TIA) (see Methods and Supplementary Note 1). Time-resolved photoluminescence (TRPL) measurements showed that the intrinsic photoluminescence lifetimes of the silicate and garnet are approximately 1.5 μs and 200 ns, respectively (Fig. 2b). In addition, the silicate exhibits a pronounced persistent luminescence (after-glow) tail, whereas garnet’s persistent luminescence is negligible.

In our standard configuration (Fig. 2c), Sensor A’s TIA (feedback resistance, Rf = 1.5 MΩ) exhibits a 3dB bandwidth (f3dB) of 9 kHz (τ = 1/2πf3dB = ~18 μs), whereas Sensor B’s TIA (Rf = 4.75 MΩ) has a f3dB of 3 kHz (τ = ~53 μs). In this case, the effective electrical response is dominated by the TIAs’ low-pass characteristics, rather than the much shorter intrinsic PL lifetimes of the phosphors. Under a 100-ms LED on/off square-wave, Sensors A and B show 10–90% rise/fall times tr of approximately 50 μs and 120 μs, respectively, closely matching the estimate tr ≈ 0.35/f3dB (39 μs and 117 μs).

The differential output Vout(t) = VA(t) − VB(t) exhibits an immediate positive spike at LED turn-on owing to Sensor A’s faster kinetics and a negative spike at turn-off (Fig. 2c, red). In this configuration, spikes span approximately 100 μs, but adjusting the TIAs’ feedback resistances can reduce them to as short as 2.4 μs (Supplementary Figs. 3; 2d). Raising the cutoff frequencies above the phosphors’ decay rates could narrow spikes further, approaching the intrinsic PL timescales of the materials.

Following each spike, the silicate’s persistent luminescence, which is nearly absent (less than 5% amplitude) in the garnet channel, produces a slowly decaying tail in Vout (Fig. 2c, blue). Details of the long-tail characterization—combining TRPL for the early (μs) regime and broadband PD-TIA measurements for the long-time (ms) regime—are provided in Supplementary Note 1.3. The convolution of the two TIAs’ time constants with the silicate after-glow kinetics yields analog traces encoding both instantaneous events and temporal memory (dominant decay time τ ~ 45 ms). Stepped-intensity experiments confirm that each brightness transition elicits repetitive spikes, each of which causes an immediate increase or decrease in the tails that gradually decay over time (Fig. 2e).

Under our configuration, EF spikes and MF tails have opposite polarity, enabling real-time separation (see Supplementary Note 2 for the EF/MF recovery method). The opposite case‒when the silicate channel is read with a TIA whose bandwidth is lower than that of the garnet channel‒yields spikes and tails with the same polarity (Supplementary Fig. 5). We also observed memory-fading behavior, in which the degree of tail signal accumulation and decay varied systematically with the temporal profile of input brightness steps, closely matching residual-state dynamics in reservoir computing. Fitting these dynamics with a linear reservoir (leaky-integrator) model36,37

Ti[n] = α · Ti[n−1] + β · Si[n−1]

where Ti[n] is the internal memory tail state of channel i at frame n, and Si[n−1] is the external event spike input. The per-frame retention (decay) factor α is estimated from the after-glow fits (α = ~0.895 for τ = ~45 ms with Δt = 5 ms). We set β = 1 − α to enforce unity steady-state gain. Using α ∈ {0.85, 0.90, 0.95} in Fig. 2f brackets the fitted range and faithfully reproduces the observed analog-memory tails. The τ is tunable at the materials stage43 and, in principle, can be reconfigured in operando44, with details provided in Supplementary Note 1.5.

Real video tests with analog event–memory sensors

To validate our analog event + memory sensing approach, we replayed pixel-wise brightness time-traces from dynamic video clips onto AEMSs using controlled LED stimuli. We selected 93 clips from the Weizmann Human Action Dataset45 (each comprising 21 RFs at 70 × 70 pixels across ten action classes; representative frames are shown in Fig. 3a and Supplementary Fig. 11) and converted each pixel’s 8-bit intensity into a corresponding LED intensity sequence (see Supplementary Note 2).

Figure 3b presents example LED intensity waveforms for selected pixels alongside the corresponding AEMS voltage responses. Each luminance transition produces a rapid differential spike followed by a slower decaying tail. Figure 3c shows a 2D spatiotemporal map of the AEMS voltage outputs over time for the 70 pixels in the 43rd column of each frame from the dataset in Fig. 3a. By aggregating spike occurrences across all pixels at each timestep, we reconstructed EFs (Fig. 3d, top panels), which closely match digitally computed frame differences (see Supplementary Fig. 13 and Supplementary Video 1 for comparisons between AEMS-reconstructed and digitally computed EFs). This process was carried out across the entire dataset, and the system operated stably over the long acquisitions without observable drift in either spike or memory responses (see also Supplementary Fig. 9 for accelerated-cycling data). Consequently, the reconstructed EFs achieve an average SSIM ~ 0.94 and MAE ~ 0.02 (Supplementary Fig. 14).

In parallel, we integrated the tail amplitudes across pixels to form MF images (Fig. 3d, bottom), capturing motion persistence in a manner analogous to frame accumulation. These analog memory images align qualitatively with the outputs of a linear reservoir model applied to the EF inputs (Supplementary Fig. 13), supporting their functional fidelity. Because they reflect gradual decay after spike accumulation, the MFs effectively preserve motion traces over time. Notably, unlike the case where a linear reservoir model is applied directly to RF inputs34, our MFs are generated from EF-based dynamics and therefore preserve spike polarity (±) and suppress redundant background, yielding richer temporal history with reduced data redundancy (see Supplementary Note 3).

We further evaluated our method on the UCSD Pedestrian Dataset46, which features real-world urban scenes with complex visual clutter, including multiple overlapping pedestrians and moving vehicles (Fig. 3e; Supplementary Fig. 15). Despite this complexity, our analog framework enabled robust background suppression and clear delineation of motion trajectories. In the reconstructed EFs, static background elements such as trees and roads were suppressed, while dynamic objects exhibited sharp intensity transitions (SSIM of 0.95 when compared to digitally computed frame differences), confirming spatial fidelity. The corresponding MFs revealed smooth decaying trails consistent with the underlying motion paths.

Learning and evaluation for dynamic image classification

To quantify the benefit of each information channel, we trained lightweight CNNs on three synthetic modalities derived from the Weizmann Human Action Dataset (93 clips, 21 frames each; Fig. 4a). Each clip was augmented twentyfold, and only the last 10 frames—where sufficient temporal context exists—were used to generate RFs, EFs, and MFs. This yielded 18,600 samples per modality, split 80/20 for training and validation. Training used hardware-calibrated EF/MF datasets that embed measured AEMS characteristics—noise statistics, short-/long-term drift, nonlinear responses captured by look-up table (LUT)-based calibration curves, and inter-frame spatiotemporal correlations—and testing used independent AEMS-measured RFs and EFs/MFs (Supplementary Note 2.3).

In single-modality experiments, RF-based models exhibited the poorest performance, with both training and validation accuracies remaining below 45% after 20 epochs (Fig. 4b; Supplementary Fig. 16a). EF benefited modestly from calibration—training/validation behavior improved—but test accuracy remained 69% (Supplementary Fig. 16b), consistent with EF’s frame-difference-like nature and ambiguity in single-frame cues. MF showed the largest gain with calibration, improving from pre-calibration accuracies of about 91%, 85%, and 82% (training/validation/test) to 97%, 96%, and 93%, with stable convergence by epoch 17 (Supplementary Fig. 16c). This reflects the calibrated inclusion of accumulated noise, drift, LUT nonlinearity, and inter-frame correlation absent in purely digital simulations.

For a minimal two-channel fusion, EF plus MF reached accuracies of about 96%, 96%, and 92% (training/validation/test)—an improvement of about 3.3 percentage points over the pre-calibration fusion—yet slightly below MF alone (Fig. 4b; Supplementary Fig. 16d). Confusion matrices (Fig. 4c) show that action pairs ambiguous under EF are effectively separated when MF is used, underscoring MF’s strength in encoding accumulated, continuous spatiotemporal patterns. Overall, progressing from RF to EF to MF under hardware-calibrated training narrows the train/validation/test gap and improves generalization, with MF delivering the highest test accuracy on measured data.

Evaluation of vehicle dynamics in intersection environments

For an object moving at speed v, the continuous-space memory trail along its trajectory, A(ξ), follows an exponential profile A(ξ) ∝ exp(−ξ/ℓ), where ξ denotes the arc length along motion, and ℓ = vτ is the characteristic trail length. This property implies that the AEMS-based memory filter (MF) provides a natural cue for trajectory and speed estimation (see Supplementary Note 4 for the quantitative EF and MF framework). To evaluate our in-sensor event-plus-memory (EF+MF) pipeline under deployment-relevant intersection scenarios, we generated a synthetic dataset of 510 short video clips depicting vehicles traversing a four-way intersection (Fig. 5a). Vehicles entered from the top, left, or right roads and exited via straight, left-turn, or right-turn trajectories (Fig. 5b shows 50 representative paths), with speeds varying from 30 to 60 km/h by adjusting per-frame displacements. From each clip, we extracted a 36 × 36 pixels patch covering the exit region (red box) and sampled 21 frames at 10 ms intervals (see Supplementary Note 5.1 for details).

Per-pixel intensity time series from each patch were replayed through an AEMS, yielding analog responses from which we reconstructed EFs and MFs. For both classification and regression, we split these EF and MF sequences into 90% training and 10% unseen test sets. Figure 5c shows final-frame EFs and MFs for three representative combinations of speeds and trajectory (see Supplementary Fig. 23 for the full 21-frame sequences). While EFs vary only subtly with speed and direction, MFs—by integrating the analog tails—clearly reveal motion trails encoding both trajectory and velocity.

Under a strict single-frame, equal-latency setting (decision at the end of the last frame), a lightweight CNN on the final frame shows that the EF-only model overfits (58.8% test accuracy), whereas the MF-only model achieves 98.0% (Fig. 5d; Supplementary Fig. 25). For speed estimation, a CNN regressor trained with Huber loss47,48 attains MAE (MAPE) of 4.77 km h⁻¹ (10.6%) with EF and 2.15 km h⁻¹ (4.9%) with MF (Fig. 5e; Supplementary Fig. 25c), corresponding to about ±5.3 km h⁻¹ at 95% confidence for a single MF frame. All speed-estimation experiments use intersection scenes with high-contrast crosswalk backgrounds, in which alternating bright/dark stripes overlap the MF tail. To contextualize background effects, Supplementary Fig. 24 profiles the MF tail with and without crosswalks—showing near-exponential decay without crosswalks and local envelope distortion with crosswalks—and demonstrates that background-aware training mitigates this bias. While multi-frame optical-flow and hybrid optical-flow-plus-event pipelines can achieve lower MAE when given longer temporal windows, they entail higher decision latency and memory; detailed accuracy–latency trade-offs and model/training specifics are provided in Supplementary Note 5.3.

Efficient motion analysis via ONN-AEMS hybrid pipeline

High-resolution dynamic image sequences impose a significant computational burden when processed digitally. To address this, we propose integrating an ONN with our AEMS-based analog processing pipeline, enabling most computationally intensive operations to be performed optically and in the analog domain (Fig. 6a). Each 70 × 70 pixels frame is compressed to a 16D feature vector via a single-pass matrix–vector multiplication (MVM). In our implementation, the RF is replicated into 16 parallel channels (optical “fan-out”), each modulated by a pre-trained 4900 × 16 weight mask encoded in greyscale. The modulated outputs of all 4900 pixels per channel are summed onto a photodetector (optical “fan-in”), completing a full 4900 × 16 MVM in one shot (see Supplementary Note 6.1).

Over 21 consecutive frames, the ONN produces a 16D output vector per frame (Fig. 6b). These are converted into a 16-channel LED drive, and the AEMS’s analog spike-and-tail responses are recorded (Fig. 6c). From these recordings, we extract: (i) compressed EFs (cEFs) — the spike amplitudes at each of the 20 inter-frame transitions — and (ii) compressed MFs (cMF) — the tail amplitudes immediately following those transitions. Both cEFs and cMFs are thus represented as 20-frames × 16-channel matrices.

From the AEMS outputs of 4650 augmented video sequences (90% training including 20% for validation, and 10% testing), we select 13 cEFs and 13 cMFs (26 frames total × 16 channels = 416D) per sequence (Fig. 6d). This corresponds to a 247-fold reduction from the original 102,900D input (21 frames × 70 × 70 pixels).

A lightweight classifier (64-unit ReLU dense layer followed by a 10-unit softmax) was trained to predict one of ten action classes. After training (Supplementary Fig. 28), we achieved ~93.3% classification accuracy on the test set (Fig. 6e). Most action classes are well separated, except for some confusion between “Two-hands wave” and “One-hand wave,” likely due to their similar spatiotemporal signatures in the compressed 16D feature space.

We next investigated the impact of frame count on classification accuracy. Starting from the full set of 20 cEFs and 20 cMFs, we progressively reduced the number of paired inputs from 40 to 2 (i.e., n cEF + n cMF, for n = 20 to 1). Figure 6f shows the resulting test accuracy as a function of compression ratio, which increases from 160-fold (n=20) to 3216-fold (n=1). While accuracy declines moderately beyond approximately one-thousand-fold compression, it remains above 90% even at three-hundred-fold (red dots). Compared to idealized simulations (blue dots), our measured accuracy is a few percentage points lower—likely due to optical misalignments, sensor noise, and the fact that only the classifier (not the ONN compressor) is retrained on real data. These simulations suggest that with improved alignment, hardware fidelity, and full end-to-end training, accuracy could exceed 96% even under extreme compression.

By offloading MVM to the ONN and extracting event and memory signals using AEMS, our hybrid pipeline dramatically reduces computational load and enables real-time, low-power action recognition at the sensor frontend. In our prototype, ONN outputs are still digitally stored and replayed via LEDs due to OLED and camera refresh limitations, which prevent fully real-time optical–analog processing. These constraints also dominate the end-to-end latency—OLED refresh (16.7 ms at 60 Hz) and camera integration (10–20 ms)—for a total of about 27–37 ms. In a production-ready system, the OLED fan-out would be replaced by a microlens array41,49 (or a diffractive optical element, DOE), and high-speed photodetector arrays41,50 would capture each channel directly, eliminating all digital storage and replay steps. The weight mask would be implemented as a fixed-pattern passive transmissive element (e.g., chromium-on-glass attenuation, etched-glass phase, or a metasurface), so no periodic refresh would be required during operation. The remaining task—amplifying, offsetting, and rescaling each of the 16 continuous-valued ONN outputs—would likewise be performed entirely in the analog domain via integrated banks of low-noise TIAs and simple sample-and-hold (S/H) or level-shifter circuits co-packaged with the pixel array. Under this architecture, the latency would be set by the PD-TIA bandwidth and S/H timing, enabling μs-scale operation. A detailed description of the deployable architecture and latency assumptions, along with process and packaging considerations for monolithic (or highly integrated) implementations, is provided in Supplementary Note 6.2 (Supplementary Table 3).

Methods
Fabrication of fluorescent PDMS films
To enable analog optical sensing, two types of phosphors—Sr2SiO4:Eu2+ (silicate) and Lu3Al5O12:Ce3+ (garnet)—were embedded in a PDMS matrix (Silgard™ 184, Dow Corning) to form flexible fluorescent films. The PDMS base and curing agent were mixed at a 10:1 weight ratio, and phosphor powders were added at a 1:8 phosphor-to-PDMS ratio and homogenized. The mixture was degassed under vacuum for 30 minutes, cast into a 60 mm petri dish, and pre-cured at 70 °C for 1 hour. TRPL decays were measured using a Horiba Delta Diode-375L pulsed laser diode (λ = 375 nm) and a time-correlated single-photon counting (TCSPC) module integrated into a Fluoromax-4 system.

AEMS assembly and photoelectrical response measurement
Each AEMS unit consisted of two parallel optical channels, each comprising a phosphor-PDMS film, a long-pass filter (>495 nm, Thorlabs), and a silicon photodiode with integrated TIA (PDA100A2, Thorlabs). To ensure identical responsivity to input light, the two sensor arms (silicate and garnet) were carefully aligned in both position and angle. Analog illumination was driven by a 255-nm UV LED modulated by a National Instruments USB-6423 DAQ, amplified using an HP 6827A amplifier. The differential voltage Vout = VA − VB was sampled at 40 kS/s using the same DAQ device.

EF/MF image construction and motion analysis
To evaluate dynamic visual sensing, video clips from the Weizmann Human Action dataset (21 frames, 70 × 70 pixels) and the Crossroad dataset (21 frames, 36 × 36 pixels) were replayed frame-by-frame into the AEMS via UV LEDs. Each 8-bit pixel intensity (0–255) was mapped to a corresponding LED brightness level. The AEMS output signals were processed to extract differential spike responses (event data) and decaying tail responses (memory data), which were used to construct EFs and MFs.

These EF/MF representations then served as training data for CNN-based models. For human action classification (Weizmann), we used hardware-calibrated, simulation-generated EF/MF datasets parameterized by measured device characteristics. For vehicle trajectory and speed prediction (Crossroad), we used EF and MF images reconstructed from AEMS measurements. The CNN architecture consisted of a single convolutional layer with 64 filters (3 × 3), ReLU activation, 2 × 2 max-pooling, and fully connected layers. The classifier output layer predicted either one of ten action classes (Weizmann) or three trajectory types (Intersection). Training used the Adam optimizer (learning rate = 0.001, batch size = 32), with early stopping based on validation accuracy.

ONN–AEMS hybrid pipeline implementation
To demonstrate optical–electronic hybrid computing, we constructed a pipeline integrating an ONN with the AEMS array. We first trained a simulation-based neural network with the same architecture as the ONN-AEMS pipeline—comprising a fully connected compressor (FCNN), temporal differencing, reservoir integration, and a dense classifier. The trained FCNN weights (4900 × 16) were then implemented optically. Each RF (70 × 70 pixels) was split into 16 replicated images and displayed on an OLED panel. These were optically projected onto 16 distinct greyscale weight masks encoded on an LCD panel, enabling one-shot 4900 × 16 optical MVM. The modulated light was focused onto a scientific camera (Thorlabs CC215MU), and each of the 16 projected images was integrated over its 4900 pixels to yield a 16-D compressed output vector.

This feature vector was replayed via LED into the AEMS array to extract dynamic temporal signals. Each sample consisted of 13 cEFs and 13 cMFs, flattened into a 416-D input vector (13 × 16 × 2). A lightweight classifier with one hidden layer (64 ReLU units) and a final softmax layer was trained to classify human actions from these temporally compressed analog signals.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "Fundamental approaches to motion target perception at the detector level primarily follow two paradigms: the differential approach, which analyzes target motion trends by outputting inter-frame information differences (exemplified by event cameras), and the integrative approach, which quantifies the relationship between signal intensity and time to memorize motion trajectories through quasi-volatile storage effects (e.g., persistent photoconductivity and its associated reservoir neural networks). This paper presents an innovative fusion of both differential and integrative methods to enhance motion target perception, demonstrating notable novelty. However, the work significantly lacks characterization and mechanistic analysis of the core device/materials principles. I therefore recommend acceptance contingent upon the authors thoroughly characterizing and elucidating the fundamental operational mechanisms of their proposed device.\n1. The slow, persistent decay of luminescence in the silicate is a critical origin for the formation of \"MF\". Consequently, understanding the mechanism underlying this persistent luminescence is paramount for \"MF\". In the PL testing presented in Figure 2, more complete decay data must be provided. This data should be fitted to determine the lifetimes at different stages of the decay process, clearly indicating the distinct mechanisms dominating each stage. A detailed explanation of the principle governing this persistent luminescence phenomenon should accompany the analysis.\n2. A key advantage of the \"MF\" mode is its ability to circumvent the need for frame-by-frame or inter-frame readout. By integrating signal over multiple frames, it significantly reduces the total data output, thereby lowering demands on bandwidth and computational resources. However, this advantage is difficult to achieve under the differential paradigm. Merging multiple frames in a differential approach would inherently obscure the fine details of abrupt changes in moving targets – precisely the information it aims to capture. Please elaborate on how your solution addresses this critical challenge of preserving transient dynamic details while achieving data reduction.\n3. The detectable range of motion velocities in this work is fundamentally constrained by the lifetime of the persistent luminescence. When targets move significantly slower or faster than this range, or when high-speed and low-speed targets appear simultaneously, the \"MF\" process is highly susceptible to interference. For instance, in the application scenario depicted in Figure 5, can the system still reliably detect and distinguish both a high-speed vehicle and a low-speed pedestrian when they are present concurrently? Furthermore, in this same mixed-traffic environment, would the \"EF+MF\" process be adversely affected by noise if the background illumination is sufficiently high? Please provide a concrete quantitative framework describing the dynamic interplay between the system frame rate, the decay characteristics of the persistent luminescence, and the velocities of moving targets.‘\n4. Again, regarding the persistent luminescence material property: how is the lateral diffusion of the persistent luminescence signal prevented after localized irradiation? Ensuring that this phenomenon remains confined to the irradiated pixel and does not bleed into neighboring pixels is foundational to the successful execution of the \"EF+MF\" concept. Given its critical role in the core functionality of the proposed device, I strongly recommend that the authors perform and present detailed characterization specifically addressing this spatial confinement property, accompanied by a thorough mechanistic explanation.\n5. Given that the lifetime of the persistent luminescence critically determines the detectable range of moving target velocities, a fundamental question emerges: Is this lifetime a tunable or reconfigurable parameter in the proposed material/system?", "reviewer_2": "In this manuscript (ID: NCOMMS-25-53691-T) entitled “In-sensor Analog Optoelectronic Processing of Concurrent Event and Memory Signals for Dynamic Vision Sensing”, Kim et al. propose a prototype sensor that integrates silicate and garnet phosphors with Si photodiodes and transimpedance amplifiers to achieve fast and slow kinetics. The authors demonstrate vehicle trajectory classification and speed estimation based on in-sensor dual-response sensing architecture that concurrently generates analog event spikes and persistent memory tails. By eliminating analog-to-digital conversion and digital accumulation, the proposed approach enables ultralow latency and ultralow power consumption neuromorphic vision. The study presents an interesting dynamic vision sensor incorporating event spikes and persistent memory directly in the analog domain at the pixel level. However, there are some obvious flaws in this work, and the article needs to be thoroughly revised before it can be published in the prestigious journal Nature Communications. Please be advised with the following comments.\n(1) The event-driven vision sensor based on two photodiodes with different response kinetics has been reported in the previous work (Nature Electronics 6, 870-878 (2023)). The authors should comprehensively discuss the advancements and advantages regarding both functionality and performance compared with previous event-based vision sensors including DAVIS, ATIS and recent published works (e.g. Nature Nanotechnology (2025). https://doi.org/10.1038/s41565-025-01973-6).\n(2) The study uses silicate/garnet phosphors paired with TIAs of different bandwidths to concurrently generate event spikes and memory tails. Why were silicate (long-persistent) and garnet (short-persistent) specifically chosen? What is the uniqueness of these materials? Is there a quantitative model (e.g. carrier dynamics) demonstrating that this combination maximizes the response speed disparity?\n(3) The authors claim “ultralow-latency processing” of the ONN-AEMS hybrid pipeline, but the current workflow in experiments fails to achieve fully end-to-end optical-analog processing. The latency bottlenecks induced by the OLED refresh rate and camera integration time remain unquantified. The feasibility of monolithic integration within single chip should also be further discussed.\n(4) The array-level scalability and applicability of memory frames need to be assessed. The decay times (> 1 ms) may cause inter-frame crosstalk in high-speed scenarios (e.g. 1000 fps). The fidelity under shortened pulse intervals needs to be tested. The uniformity of phosphor-film would affect pixel-to-pixel response consistency, and the multi-pixel statistics are necessary.\n(5) What is the area overhead of the dual-channel pixels versus conventional single-channel DVS pixels? The authors are suggested to discuss the CMOS process compatibility including thermal budget for phosphor integration. Besides, the power claim of “<1 mW/channel” lacks detailed support. The power distribution of the TIA+PD and the conditions need to be specified.\n(6) The vehicle dynamic tests use a synthetic dataset rather than real-road videos. The real-world complexities like illumination changes, occlusions and multi-object interferences are underestimated. The speed estimation MAE (2.15 km/h and 4.77 km/h) lacks comparison to baselines (e.g. optical flow + event cameras).", "reviewer_3": "This manuscript reports an analog event-memory sensor (AEMS) architecture that concurrently generates event spikes and memory tails at the pixel level, validated using both human action recognition and vehicle dynamics tasks. The approach is further integrated with an optical neural network (ONN) front end for feature compression, demonstrating high classification accuracy with significant latency and power advantages over conventional digital pipelines. The concept of dual-response in-sensor computing is timely and relevant for ultralow-power dynamic vision systems.\nHowever, while the overall technical direction is promising, several aspects require further clarification and experimental validation to strengthen the manuscript’s system-level robustness and scalability. The following comments aim to provide constructive suggestions to improve the completeness and applicability of the work.\n1. EF/MF generation is demonstrated using a single-device AEMS setup with simulated video input. In a large-scale pixel array, practical issues such as simultaneous multi-point illumination, optical crosstalk between adjacent pixels, and electronic interference are likely to affect the clean separation of event and memory signals. The authors should discuss potential mitigation strategies and outline a hardware validation roadmap to ensure that the dual-response concept remains effective in full-array implementations.\n2. The method for determining the temporal filtering parameters (e.g., decay time constants) used to separate EF and MF signals is not fully described. Fixed parameters may lead to overlaps between events and memory tails when scene motion is rapid or when multiple objects move simultaneously. The authors should consider whether adaptive tuning—based on scene statistics or real-time feedback—could improve robustness, and if so, how it could be implemented in hardware.\n3. All experiments use UV LED projection under controlled conditions. In real-world applications, broadband or spectrally varying illumination and oblique incidence angles could alter phosphor emission kinetics and TIA response, potentially reducing EF/MF separation fidelity. It would be valuable to characterize the spectral and angular sensitivity of the system, and to assess performance under mixed lighting conditions to ensure operational stability outside of the laboratory.\n4. The analog memory tail relies on the persistent luminescence properties of the silicate phosphor, which may degrade under prolonged UV exposure or high-frequency operation. Such degradation could cause MF amplitude drift over time. The authors are encouraged to provide accelerated aging or cycling endurance data to quantify long-term stability and retention performance.\n5. The CNN classifiers are trained on simulated data and then evaluated on AEMS-measured outputs, without co-optimizing for real-world analog noise, drift, and nonlinearity. A closed-loop training approach—where the model is trained directly on hardware outputs—could improve classification robustness and inform optimal sensor parameter tuning for EF/MF separation.\n6. The frame rate directly impacts spike resolution and memory tail fidelity, especially in large-scale arrays. Higher frame rates may reduce integration time for memory signals, while lower frame rates may cause aliasing or temporal overlap. The authors should report how classification accuracy, latency, and EF/MF separation quality vary with frame rate. Additionally, system-level metrics such as throughput, per-pixel power consumption, and end-to-end latency should be provided to clarify the practical performance envelope in real-world operation."}, "author_response": "We sincerely thank the reviewer for providing insightful comments on our manuscript. We greatly appreciate the opportunity to address and clarify the issues raised in the reviewer's report, and we believe that the additional analyses and revisions have significantly strengthened our manuscript. Our responses to the points raised in the report and the corresponding changes made in our revised version (highlighted in red) are provided below.\n\n[... Author response text for Round 1 ...]"}, {"round": 2, "reviewer_comments": {"reviewer_1": "The authors have addressed my concerns satisfactorily and have effectively improved the manuscript. I recommend acceptance.", "reviewer_2": "The authors have fully addressed my previous comments, i am now happy to recommend this work for publication in Nature Communications.", "reviewer_3": "Thank you for your detailed and constructive responses to the previous review.\nThe authors clearly answered all the questions raised before. The explanations about the device operation, data handling, speed detection, and spatial control are easy to understand and supported with new figures and notes. The added tests and simple models help show how the system works in different situations. The revised paper is now clear, complete, and well organized.\nBased on the addressed concerns and the enhanced quality of the manuscript, I recommend the work for publication in Nature communications."}, "author_response": ""}]

Decision letter