Ultrafast visual perception beyond human capabilities enabled by motion analysis using synaptic transistors

Domain: physics
Published: 2026-02-10
Editorial outcome: major_revision

Abstract
Optical flow, inspired by biological visual systems, calculates spatial motion vectors aiming to enable robots to excel in dynamic environments. However, current algorithms, despite human-competitive task performance on benchmark datasets, suffer from significant time delays, limiting practical deployment. Here, we introduce a neuromorphic temporal-attention hardware that emulates the interaction between the retina and the lateral geniculate nucleus (LGN) to extract temporal motion cues directly in hardware. Using a two-dimensional synaptic transistor array, the system encodes brightness changes and accumulates them in analog, non-volatile states, generating compact regions of interest (ROIs). These ROIs then act as inputs to conventional downstream optical flow and vision algorithms, enabling ultrafast motion analysis. At the hardware level, the synaptic transistor offers high-frequency response (~100 μs), non-volatility (>10,000 s), and endurance (>8,000 cycles). Compared to state-of-the-art algorithms, our approach demonstrates a 400% speedup, surpassing human-level performance while maintaining or improving accuracy through temporal priors.

Introduction
Optical flow, originally introduced by James J. Gibson in the 1950s and inspired by biological visual perception, estimates motion vectors within a visual scene1–3. Over the decades of development, current state-of-the-art algorithms such as RAFT and GMFlow have demonstrated impressive performance on benchmark datasets. By leveraging two-dimensional gradient representations of pixel movement, optical flow offers clear, intermediate motion representations compared to end-to-end approaches, enabling it to excel in dynamic scene analysis at performance levels comparable to human capabilities4–9. However, translating these achievements to real-world applications is still challenging due to the high computational overhead required to process visual inputs in real time. For instance, while Tesla’s Autopilot employs occupancy networks to achieve latency as low as ~10 ms10,11, performing optical flow analysis and object segmentation on a 1920×1080 resolution image can require over 0.6 s on an Nvidia V100 GPU, 4X humans (Supplementary Notes 1, 2). Such delays are unacceptable for time-sensitive applications like autonomous driving, where a one-second delay at highway speed can reduce the safety margin by up to 27 m, significantly increasing safety risks. Thus, due to this obvious time delay, the field deployment of optical flow seems to be unrealistic12–15.

Optical flow is initially aimed to mimic the processing pipeline of biological visual systems, but cannot faithfully replicate their high processing efficiency in practice, which is because biological vision excels at processing large volumes of visual information efficiently through dynamically focusing on regions where motion occurs16,17. Specifically, biological vision, including the retina and lateral geniculate nucleus (LGN), emphasizes temporal change and dynamically routes information toward locations where motion occurs, acting as a temporal-attention function that detects moving regions before high-level complex processing18,19. This suggests a design principle for artificial vision: a fast, low-overhead function that dynamically detects motion regions, thereby accelerating downstream motion analysis by focusing computation where changes occur. Replicating this dynamic processing in artificial systems is challenging due to the intrinsic inflexibility of conventional CMOS-based technologies, which cannot readily adjust their processing functions in response to varying stimuli20,21. Fortunately, neuromorphic devices such as synaptic transistors and memristors offer synapse-like characteristics that can emulate the processing functions of biological synapses. Their intrinsic adaptability enables on-device adaptation and local feature extraction functions, which in turn lowers both energy consumption and processing latency. This approach has already been applied across multiple sensory modalities. In a typical implementation, a front-end sensor first transduces external stimuli into electrical signals (voltage or current), which are then transmitted to synaptic devices for processing. For tactile perception, the sensor is typically a piezoresistive or piezoelectric film that converts mechanical force into an electrical response. Based on this architecture, artificial tactile receptors are widely studied—including rapidly adapting mechanoreceptors and nociceptors—using neuromorphic hardware22–25. For example, Wang et al. proposed a memristor-based perceptual signal processing method, which can emulate multiple essential tactile receptors and sensory neurons with a single memristor22. For auditory processing, neuromorphic device arrays can capture and process auditory temporal patterns directly in hardware, enabling low-latency, low-power front ends for sound recognition and localization26–28. For visual processing, the high spatial redundancy of video frames aligns perfectly with the local, in-memory processing capability of synaptic devices22,29–31. For example, Baek et al. recently reported a neuromorphic neuron with dendritic morphology based on silicon-nanowire transistors that can perform nonlinear integration of excitatory/inhibitory synaptic inputs while considering spatial distribution dependency. Its built-in direction selectivity enables on-device motion detection, and the power consumption of processing event-based pulses from the 1000 pixels is about 0.1–2.0 mW32. More broadly, the on-device adaptation capability of synaptic devices makes human-like ultrafast visual processing possible22,33–35.

Here, inspired by the biological LGN system, we propose a neuromorphic hardware that directly detects motion changes using two-dimensional neuromorphic synaptic transistors. The array encodes brightness derivatives and accumulates these temporal changes in analog, non-volatile states, as shown in Fig. 1. The resulting array state generates temporal motion cues, i.e., compact maps of regions of interest (ROIs) that guide and accelerate downstream optical flow calculation. In our demonstration, conventional optical flow calculation methods (e.g., Farneback, GMFlow, and RAFT) are applied only inside these ROIs, not the entire image, ultimately accelerating various tasks such as motion prediction, segmentation, and tracking. In summary, the neuromorphic hardware rapidly extracts motion change information, which is then passed to conventional processors for downstream optical flow computation in our demonstration (Supplementary Movie 1). At the device level, leveraging the superior properties of two-dimensional materials, such as atomic thickness and enhanced electrostatic control, the developed floating gate synaptic transistors demonstrate high-frequency response (~100 μs), robust retention (>104 s), and exceptional endurance (>8000 cycles). In our experiments, we deployed our pipelines across various application scenarios—including vehicle operation, UAVs, robotic arms, and sports activities—to perform tasks like motion prediction, object segmentation, and object tracking. The results demonstrate that our method significantly accelerates processing times, achieving an average of ~400% speedup and surpassing human-level speeds (~150 ms) in most cases. Notably, by incorporating spatial-temporal consistency of motion information, our spatiotemporal approach maintains or improves accuracy, such as a 213.5% performance increase in the vehicle scenario. These advancements empower robots with ultrafast and accurate perceptual capabilities, enabling them to handle complex and dynamic tasks more efficiently than ever before.

Results
Floating gate synaptic transistor
In our neuromorphic motion extraction hardware, synapse arrays serve to embed temporal information from external visual scenes. To achieve precise encoding and long-term retention of this information, the synapse array must exhibit synapse-like characteristics—adjusting its state in response to external stimuli—and non-volatile properties to maintain stored data36. To further ensure high-frequency processing capabilities and long-term system stability, we have designed floating gate synaptic transistors based on a two-dimensional van der Waals heterostructure as neuromorphic devices that generate temporal motion cues directly in hardware. Based on the Fowler-Nordheim tunneling mechanism, the floating gate synaptic transistor precisely regulates the charge in the floating gate through the gate voltage, thus realizing the long-term stable storage of information and guaranteeing the continuity and reliability of produced motion cues in the time dimension. Fig. 2a demonstrates a schematic structure of this floating gate synaptic transistor. From bottom to top, the synaptic transistor includes a gold film (serving as the control gate), an aluminium oxide (Al2O3) blocking layer, a multilayer graphene (MLG) floating gate, a thin hexagonal boron nitride (h-BN) tunneling layer, and a molybdenum disulfide (MoS2) channel. In operation, gate-source voltage (Vgs) pulses are applied to the control gate (with the source grounded) to modulate the drain-source current (Ids). Comprehensive details on the fabrication process of the floating gate synaptic transistor and the Raman characterization of its heterostructure are provided in Supplementary Figs. 1 and 2. Under these conditions, the MoS2 channel’s output characteristic confirms a good ohmic contact with the Cr/Au interface (Supplementary Fig. 3). In terms of memory behavior, the transfer curve of this synaptic transistor at a fixed drain-source voltage (Vds = 1 V) depicts a clockwise memory window that reaches 11.2 V when the Vgs is swept from −10 V to +10 V (Fig. 2b). Furthermore, the memory window increases with the maximum applied Vgs, as presented in Supplementary Fig. 4. When applying Vgs pulses, this synaptic transistor displays obvious synapse-like characteristics. As shown in Fig. 2c, the change in conductance is positively related to the number of applied pulses. The calculation method for the pulse-number linearity is detailed in the Supplementary Fig. 22. The modulation mechanism is elucidated by the energy band diagram: negative Vgs pulses drive holes into the floating gate and elevate the device’s conductance, while positive Vgs pulses facilitate electrons tunneling into the floating gate and reduce its conductance. Additional details about the operating mechanism can be found in Supplementary Fig. 5. This modulation can be controlled by varying the amplitude and duration of applied voltage stimuli. As depicted in Fig. 2d, the increase in conductance correlates positively with both the amplitude and duration of the negative Vgs pulses. We successfully achieved weight regulation at a lower negative Vgs amplitude of 7 V by enhancing the gate coupling ratio37,38, as detailed in Supplementary Note 7. Supplementary Fig. 6 illustrates the variation of conductance with the amplitude of the positive Vgs pulses. With respect to response speed, this floating gate synaptic transistor demonstrates rapid operating speed, achieving a current switching of 60 μA (from a low- to high-conductance state) under −15 V Vgs pulse with a duration of 100 μs (Fig. 2e). This ~100 μs response time is suitable for high-frequency visual information processing. Moreover, the synaptic transistor exhibits repeatable programming characteristics and multiple analog states (Fig. 2f), enabling the precise encoding of external information as its state. In terms of endurance, up to 8000 programming/erasing cycles can be achieved under positive and negative Vgs pulses (±15 V, 1 ms), with the Ids at Vds = 1 V remaining at ~10−9 A and 10−5 A in the low- and high-conductance states, respectively (Fig. 2g). Furthermore, this synaptic transistor displays non-volatile behavior. Both the low- and high-conductance states are maintained for over 104 s (Fig. 2h), confirming their non-volatility in storing external stimuli data. The excellent retention characteristics are primarily attributed to the intrinsic physical properties of the material and the optimized device structure design, as detailed in the Supplementary Note 6. Compared with other reported devices (Fig. 2i), this synaptic transistor requires a low Vgs amplitude for weight modulation and presents excellent retention37,39–47. The excellent performance of the device, including fast response, long endurance, and stable retention, can be attributed to the selection of materials and thicknesses of each functional layer, the enhanced gate coupling ratio37, and the establishment of atomically sharp and flat interfaces39,48–50; a detailed comparison is provided in Supplementary Fig. 7 and Supplementary Table 1. When scaling a single synaptic transistor to a 4×4 array as shown in Fig. 2j, the developed fabrication process can be found in the Methods section and Supplementary Fig. 8. After encapsulation (Fig. 2k), the array can be interfaced with external circuits via pins or connectors, facilitating integration with other system components. Such scalability paves the way for the development of commercial chips. The variation among multiple devices, as exhibited in Fig. 2l, demonstrate the consistent synaptic modulation behavior.

Temporal motion cue generated by neuromorphic devices
To directly generate temporal motion cues at the hardware level, we propose the imaging architecture illustrated in Fig. 3a. In this design, a conventional imaging array could serve as the front-end sensor, converting external stimuli into an analog voltage signal. This signal is then processed along two parallel paths: one is digitized to form the conventional image representation, while the other converts the analog signal and modulates a synapse array to record temporal information. Specifically, the circuit needed for voltage conversion, shown in Fig. 3b, c consists of a differential processing part and an amplitude conversion part. The differential processing part extracts changes in light intensity, while the amplitude conversion part generates voltage pulses applied to the synaptic transistor array, reflecting the temporal information of the current visual scene. In the differential processing part, a high-pass filter first differentiates the light intensity, and an operational amplifier (op-amp) then amplifies these changes within a suitable operating range for subsequent processing (Fig. 3b and Eq. 1):

Vi,j(t) = a dI/dt = a [I(t) − I(t − Δt)] (1)

where V is the output voltage of the differential processing unit; Δt refers to the sampling interval; (i, j) represents the spatial coordinates of the temporal information, corresponding in size to the synapse array; I is the analog visual voltage transmitted from the imaging array (which may be resized to match the synapse array size); and a is a proportionality coefficient ensuring the voltage remains within a suitable operating range. In the following amplitude conversion part, an absolute circuit is constructed to extract the absolute voltage change, focusing on the magnitude of the light intensity change rather than its direction (Eq. 2):

V^i,j = |Vi,j| (2)

Then, a reconfigurable op-amp generates a corresponding modulation pulse based on the current amplitude of the absolute voltage V^i,j. The relationship follows Eq. 3:

Ve i,j = p1 * (V^i,j − b1), V^i,j > Vth
Ve i,j = p2 * V^i,j − b2, V^i,j ≤ Vth (3)

where Ve i,j is the modulation voltage Vgs applied to the synapse array, p1, p2 are different proportional coefficients, b1, b2 represent different modulation biases, and Vth is a preset threshold. Under the effect of modulation voltage Ve i,j, the synaptic transistor at position (i, j) of array is modulated to a resistance state related to the nature of the light intensity change. By comparing it with the threshold Vth, dramatic changes caused by potential moving objects and mild changes caused by background movement or noise are separated and translated into negative and positive Vgs pulses, respectively, resulting in different trends of device states switching. The pulse width and the maximum supported processing frequency are detailed in Supplementary Note 13. When analyzing this synapse array, the temporal motion information can be inferred from the distribution of Ids values. For example, devices with high Ids values (low-resistance under the modulation of negative Vgs pulses) that clusters spatially indicate regions containing moving objects. More details of this circuit and the manipulation of analog visual voltage can be found in Supplementary Fig. 10.

In our implementation, we employed a commercial camera to capture 800×800 images within a vehicle scenario and processed the visual input using the 4×4 synapse array (Fig. 3d). During processing, the visual stimuli captured by the commercial image sensor are resized by averaging the light intensity of a matrix of m×n pixels into a basic unit. Here, m and n are set to 200. In this configuration, external visual stimuli are translated into modulation voltages for the synapse array according to the voltage conversion circuit. Thus, the temporal information of the current visual scene is mapped onto this transistor array. For instance, when a pedestrian suddenly runs in front of the car (Fig. 3d, e), a noticeable change in the analog visual voltage occurs at position (2, 4), leading to a negative Vgs pulse. As a result, the synaptic transistor at (2, 4) switches to a high-conductance state, causing its Ids value to increase significantly under a fixed Vds. As shown in Fig. 3f, the distribution of measured Ids across the 4×4 synaptic transistor array clearly reflects the temporal dynamics of the current visual scene, highlighting the presence of a moving object on the right side. To further facilitate integration with conventional visual processing methods, these temporal data are transformed into values ranging from 0 to 255, as shown in Fig. 3g using a logarithmic mapping (see “Methods”), making them compatible with common image processing libraries such as Python’s OpenCV (cv2) package. Through this pipeline, the temporal information encoded by the neuromorphic devices can be seamlessly combined with the original image (Fig. 3h).

Accelerated movement velocity calculation
After extracting temporal information of the current visual scene from the synapse array states, this data can be transformed into temporal cues that accelerate velocity estimation, ultimately producing the optical flow assisted by temporal cues (Fig. 4a). Specifically, the conversion into temporal cues involves two main steps. First, there is a binarization process based on a predefined threshold. Second, a connectivity analysis that includes defining connectivity, labeling connected components, and expanding these regions is performed (Fig. 4b). The resulting list of connected regions serves as the temporal cues, as shown in Fig. 4c. In addition to the vehicle operation scenario, the UAV operation scenario with a resolution of 160×160 is also processed using the same 4×4 synaptic transistor array. By comparing the original image and the generated temporal cues, it is observed that the constructed temporal cues are as effectively highlight potential moving objects. During the subsequent velocity inference, the temporal cues serve as ROIs that help automatically filter the areas where movement velocities need to be calculated (Fig. 4d). This filtering process speeds up velocity calculations compared to processing the entire image. Additionally, the neuromorphic pipeline seamlessly integrates with current velocity inference approaches, whether they are based on traditional computer vision techniques or neural networks.

In our implementation, we demonstrate the adaptability of our method using three representative algorithms for movement velocity calculation: the traditional Farneback algorithm and the neural network-based GMFlow and RAFT algorithms, as shown in Fig. 4e. Integration with other algorithms, such as FlowFormer, is demonstrated in Supplementary Figs. 11 and 13. These algorithms vary in their operational characteristics and should be selected based on the practical working environment. For instance, Farneback is suitable for less demanding scenarios, while GMFlow and RAFT are more appropriate for situations requiring higher accuracy and adaptability, albeit at increased computational and technical costs. Nevertheless, the capability of integrating various movement velocity calculation methods ensures that the neuromorphic pipeline is applicable to unstructured environments, which can vary significantly. Taking the vehicle operation scenario as an example, the temporal cues filter the visual input so that only highlighted regions are forwarded to the subsequent velocity inference stage. In practical implementations, to improve robustness and resolution, the selected region is slightly padded to include peripheral information about the moving area. As shown in Fig. 4f, the detected motion region is slightly padded, and the movement information can be calculated using various velocity inference algorithms. Details on the strategy of padding and handling multiple moving objects can be found in Supplementary Note 5.

In the vehicle scenario, processing examples are shown in Fig. 4g. Even during periods of high motion—when the running pedestrian occupies a large portion of the scene—our method remains faster (Fig. 4i). Compared to conventional optical flow methods, our spatiotemporal approach enables the detection of potential motion regions (1–2 ms) as shown in Supplementary Fig. 15, and the average total velocity inference times for Farneback, GMFlow, and RAFT are reduced to 13.0%, 37.2%, and 19.6%, respectively (Fig. 4j). For the UAV scenario analysed with Farneback as shown in Fig. 4h, the average inference time is reduced to 51.0% of the original duration (Supplementary Fig. 12).

Demonstration of fundamental tasks
Optical flow assisted by temporal cues integrates both temporal and spatial motion information (e.g., calculated spatial movement velocity), and it can support fundamental tasks that enable autonomous vehicles, UAVs, and robots to perceive, understand, and interact with their environments autonomously and intelligently (Fig. 5a). During task execution, these motion cues are first decomposed. Then, temporal motion cues selectively filter spatial cues to focus only on regions with potential motion information rather than processing the entire scene. This selective focus significantly accelerates subsequent task execution. The filtered spatial cues are then combined with visual input to execute task-specific algorithms. The implementation pipeline is detailed in Supplementary Note 4. Aligning with the vision of utilizing optical flow to help robotic systems efficiently perceive the dynamic world, similar to biological visual systems, the above pipeline for processing visual information in our method mirrors the human visual system. This includes a perception unit corresponding to the human eye, a synapse array that extracts temporal motion cues analogous to the LGN, and task-specific algorithms that perform high-level processing similar to the visual cortex. Additional descriptions of the human visual system can be found in Supplementary Note 1.

In our implementation, the visual inputs encompass various scenarios, including vehicle operation, UAV operation, sports activities (e.g., table tennis from the UCF dataset)51, and grasp operation (Supplementary Movie 2). The settings for visual processing are provided in the Methods section and in Supplementary Table 5. Utilizing optical flow assisted by temporal cues, essential tasks such as motion prediction, object segmentation, and object tracking are performed on these visual inputs. Detailed task-specific algorithms can be found in the Methods section and Supplementary Fig. 16. As shown in Fig. 5b, spatial motion cues are calculated using multiple velocity inference methods and employed to execute various tasks. Additional results are presented in Supplementary Figs. 17–21. Notably, the table-tennis sequences did not exhibit missed detections despite the small object size, owing to an appropriately chosen sensor-to-synapse pooling size, as detailed in Supplementary Notes 11–12. The overall accelerated processing, which includes both velocity inference and task execution in the neuromorphic pipeline, achieves processing times comparable to human perception—approximately 150 ms (Supplementary Note 1).

In addition to high processing efficiency, these tasks are evaluated using standard metrics, including the Structural Similarity Index Measure (SSIM), pixel accuracy (PA), and Intersection over Union (IoU), and neuromorphic pipeline achieves comparable performance to conventional pipeline (Fig. 6a). In certain scenarios, such as vehicle operation, UAV operation (small), and grasping, neuromorphic pipelines significantly outperforms conventional methods. On average, the accuracy improvements are 213.5%, 157.4% and 740.9%, respectively. The results of significance testing, summarized in Supplementary Table 11, statistically confirm these improvements and underscore the robustness and reproducibility of our approach. More detailed task performance data can be found in Supplementary Table 2. Although device metrics such as uniformity and linearity (see Supplementary Table 6) are within commonly reported ranges for synaptic transistors, the demonstrated analog weight updating under cumulative same-polarity voltage pulses enables reliable storage of temporal motion patterns sufficient for our tasks. Design optimizations and algorithmic corrections used in the experiments to achieve such high accuracy can be found in Supplementary Note 8. In terms of processing efficiency (Fig. 6b), the ability of our method accelerates full-spectrum visual processing, including both velocity inference and task-specific algorithms. Thus, a faster response can be observed compared to conventional optical flow methods. When using Farneback for velocity inference, the acceleration ratio—i.e., the ratio of the original whole processing time to the whole accelerated processing time—ranges from 12.5 to 58.0%, with an average of 27.5%. For GMFlow-based tasks, the acceleration ratio ranges from 4.7 to 36.7%, with an average of 20.6%. For RAFT-based tasks, the acceleration ratio ranges from 16.7 to 53.3%, with an average of 29.1% (Supplementary Table 3). After examining all 33 groups of tasks, the average acceleration ratio is 26.1%, which corresponds to an approximate 4X speedup. When fitting the acceleration ratio of velocity inference and the percentage of filtered regions based on temporal cues in the vehicle scenario, clear linear relationships are observed for both Farneback and RAFT methods, with all R² values exceeding 0.94 (Fig. 6c). However, velocity inference using GMFlow does not follow this trend due to its unique operational characteristics (Supplementary Fig. 14). Other velocity inference methods, such as FlowFormer, exhibit similar linear acceleration trends (Supplementary Fig. 13). In the acceleration of task execution, similar linear relationships can be observed using data points from Farneback-based vehicle scenario processing. This general acceleration, which correlates with the size of filtered visual input based on temporal cues, demonstrates the effectiveness of the proposed approach in enhancing both velocity inference and task execution performance. As a result, the neuromorphic motion extraction hardware pipeline enables real-time visual processing capabilities that are comparable to, or even exceed, human-level perception (Fig. 6d and Supplementary Table 4). A comparison of our method with other state-of-the-art neuromorphic visual approaches is provided in Supplementary Table 12.

Nevertheless, in challenging scenarios involving ego-motion or out-of-distribution motion, the performance of our method may degrade. To evaluate its robustness under such conditions, we conducted a comprehensive evaluation detailed in Supplementary Note 10 and Supplementary Movie 3. Specifically, we performed two sets of experiments: (i) controlled sequences captured using a hand-held phone, with concurrent IMU recordings of device motion, and (ii) real-world recordings from an in-car dashcam. In the first experiment, the observed speedup decreased to 170%, while in the second, the acceleration ratio was 74.8%, corresponding to a reduction in speed-up from 400 to 134% (1/74.8%). Regarding accuracy, as shown in Supplementary Table 8, performance degrades in complex scenes compared with sparse-motion scenarios. This is consistent with the limitations of conventional optical flow methods, which also perform poorly under such conditions52,53. The observed degradation arises primarily from the constraints of the deployed optical flow computation and downstream task algorithms. The core role of our neuromorphic hardware is to extract motion regions and generate ROIs, thereby accelerating downstream processing, making it powerful in scenes with sparse motion. Furthermore, we discuss potential motion compensation and fallback strategies that could be incorporated to further enhance system robustness (Supplementary Note 10).

Methods
Device fabrication
The bottom control gates in the floating gate synaptic transistors were prepared via electron beam lithography (EBL) on 285 nm SiO2/Si substrate, followed by thermal evaporation. Atomic layer deposition (ALD) technique was used to deposit the gate dielectric aluminum oxide (Al2O3). MoS2/h-BN/MLG heterostructure in a single device was prepared by mechanical exfoliation from bulk materials (Nanjing MKNANOTech. Co., Ltd., https://www.mukenano.com), and precisely positioned on the Al2O3 via the dry-transfer method. MoS2/h-BN/MLG structures in a floating gate synaptic transistor array were fabricated from chemical vapor deposited materials (Six Carbon Technology Shenzhen), and stacked via wet transfer, then patterned through reaction ion etching according to the structural design. Furthermore, 5/50 nm Cr/Au drain-source electrodes on MoS2 channel were defined sequentially by EBL, thermal evaporation and lift-off.

Device characterization
Electrical measurements of the floating gate synaptic transistor were conducted by a semiconductor parameter analyzer (Bl500A, Keysight) under atmospheric conditions. The thickness of the two-dimensional material was measured by Bruker Mulyi-Mode 8 AFM. Raman characterization was tested using a confocal Raman spectrometer (WITec alpha300R) with a 532 nm laser as the excitation source.

Logarithmic mapping
To transform the drain-source current of the floating gate transistor to a range of 0–255, a logarithmic mapping is employed as shown in Eq. 4:

s = [log10(Ids) / log10(Imax)] * 255 (4)

where Ids represents the drain-source current, and s is the transformed current scaled within the range of 0 to 255.

Task algorithms
In the task of motion prediction, it involves predicting the position of a moving object. For the filtered visual input, by using reference frames and employing the Lanczos interpolation method, the moving object in the next moment can be inferred. The Lanczos interpolation formula is Eqs. 5 and 6:

g(x) = Σ g(k)Ln(x − k) (5)

Ln(x) = sin(πx)sin(πx/n) / (π^2 x^2 / n), x = 0
1, x = 0 (6)

where g(x) represents the interpolated value at position x, g(k) represents the pixel values at integer positions k in the current frame, Ln(x) is the Lanczos kernel function, and n is the size of the kernel.

For object segmentation, the optical flow assisted by temporal cues is first converted to polar coordinates. This transformation separates the original movement velocity information into direction (angle) and magnitude (distance) components, making it easier to analyze motion information. Notably, this step only manipulates the ROI that includes significant moving objects inferred from the motion pattern layer, thereby omitting regions with slight noise caused by environmental changes, such as slow variations in lighting. After the transformation, the image is converted from RGB to HSV colorspace, where the direction and magnitude of motion velocity layers are represented using the hue and value channels, respectively. This process is beneficial for subsequent processing because it allows more intuitive manipulation of color-based information: the hue channel encodes the direction of motion, while the value channel encodes the magnitude of motion. This separation simplifies the process of identifying and segmenting moving objects based on their motion characteristics. When the motion information of the ROI is represented in the HSV color space, thresholding operations along with erosion and dilation operations can be applied to create a binary mask that accurately segments the moving objects. Thresholding isolates the relevant motion information, while erosion and dilation help refine the segmentation by removing small noise and closing gaps in the detected objects, respectively. This process results in a clear and precise segmentation of moving objects, thus enabling subsequent tasks such as object tracking and interaction in dynamic environments.

In object tracking, coordinate conversion, which is similar to that used in object segmentation, is applied first. This conversion separates motion information into direction and magnitude components. Following this, morphological opening is performed to smooth boundaries and remove noise. Then, using the contour detection algorithm, multiple bounding boxes are detected. Next, non-maximum suppression is applied to eliminate redundant detections and retain the most significant objects. This step ensures that only the most prominent moving objects are tracked across frames, improving the accuracy of the tracking process. Unlike conventional optical flow, which can be disturbed by background movements leading to unnecessary tracking, our pipeline focuses solely on the ROI regions that include potential moving objects. This targeted approach enhances tracking precision and reduces computational overhead by ignoring irrelevant background motion.

When evaluating the performance of the above tasks, metrics including SSIM, PA, and IoU are calculated to quantify the quality of prediction, segmentation, and tracking (Eq. 7–10), respectively:

SSIM(x, y) = (2μxμy + C1)(2σxy + C2) / (μx^2 + μy^2 + C1)(σx^2 + σy^2 + C2)

where μx and μy are the average values of predict result x and ground truth y, σx^2 and σy^2 are the variances of x and y, σxy is the covariance of x and y, and C1 and C2 are constants to stabilize the division with weak denominator.

PA = Σn ii / Σt i

where n ii is the number of pixels correctly classified for class i, and t i is the total number of pixels in class i. Here, PA calculates the percentage of correct segmented pixels.

IoU = Σn i=1 IoU si / n

IoU si = |A∩B| / |A∪B|

where |A∩B| is the area of overlap between the tracking mask A and the ground truth mask B, and |A∪B| is the area of union between the tracking mask A and the ground truth mask B, IoU si represents the IoU between the i-th bounding box and the ground truth. All evaluation metrics (SSIM, PA, and IoU) are calculated over the entire image area for both conventional and our method to ensure fair comparison.

Running environment
Performance evaluations of the neuromorphic pipeline with the Farneback method for velocity inference are performed on the 12th Generation Intel® Core™ i9-12900H processor. In contrast, performance evaluations of the neuromorphic pipeline utilizing neural network–based velocity inference methods, including RAFT, GMFlow and FlowFormer, are conducted on a server outfitted with an NVIDIA V100 GPU and an Intel® Xeon® Platinum 8260 CPU operating at 2.40 GHz.

Visual processing
To demonstrate the scalability of our approach, visual input data—encompassing UAV (small) operations, table tennis, and grasping scenarios—are simulated using the synapse array based on our fabricated synaptic transistor (Supplementary Fig. 9).

Visualization of optical flow
Following the work by Baker et al., optical flow vectors are mapped to a color-coded image52. In this visualization approach, color hue represents the direction of motion, and color intensity/saturation correspond to the magnitude of motion.

Data availability
All data supporting this study and its findings are available within the article, its Supplementary Information and associated files. Source data have been deposited in Figshare under accession code https://doi.org/10.6084/m9.figshare.30977674.

Code availability
All the necessary codes used in the tactile experiments and visual experiments, and their descriptions are available in https://github.com/RTCartist/Neuromorphic-Spatiotemporal-Optical-Flow.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "This manuscript introduces a neuromorphic hardware-based approach to spatiotemporal optical flow by integrating temporal motion cues with spatial gradients through two-dimensional synaptic transistor arrays. The authors report substantial performance gains while achieving or surpassing human-level motion prediction, segmentation, and tracking speeds. While the novelty of the authors’ approach is impressive, several important points need to be addressed before the manuscript can be considered for publication.\n1. The paper introduces “spatiotemporal optical flow” as a novel concept and applies it within a neuromorphic framework, which is indeed innovative. However, the Introduction section lacks a broader survey of artificial synapse-based signal processing beyond visual stimuli. Since voltage signals can be derived from other input modalities, a contextual overview of related signal-processing trends is need not to mislead the readers.\n2. It is unclear whether the innovation lies in the specific design and construction of the synaptic transistor device or primarily in its application to optical flow. If the development of their 2D-based synaptic transistor itself is also the main focus, the underlying material science and device-level rationale are only briefly mentioned. A deeper explanation of why this device is particularly suitable (over other types of synaptic transistor) for spatiotemporal optical flow would strengthen the manuscript.\n3. Figure 2 implies that the pulse-number linearity and array-level uniformity are not particularly high, compared to recently-reported synaptic devices. The paper does not clarify whether the authors applied design optimizations, fabrication techniques, or algorithmic corrections to address these limitations. Commenting on whether improving these characteristics could further boost performance or how the possible error can be calibrated would be valuable.\n4. In Figure 2i, the authors emphasize long retention of their device, yet the mechanisms responsible for this superior retention are not deeply explored. A more detailed discussion on the material and structural factors driving this performance would enhance the scientific depth of this manuscript.\n5. Other works (e.g., Nano Energy 89, 106439 (2021); Nat. Nanotechnol. 17, 27 (2022)) have demonstrated photo-memory devices that directly convert light to stored electrical signals. It would strengthen the paper to explain the practical advantages of the authors’ split architecture (sequentially using an imaging array and synaptic-transistor temporal cues) over such designs, particularly regarding efficiency, energy use, or system complexity.\n6. There is a minor typo in Figure 3a: “voltgae” should be corrected to “voltage.”\n7. The demonstrations mainly cover conditions with static or slow-moving backgrounds and a single fast-moving object. It is unclear how the system performs when the background itself is moving rapidly (e.g., when mounted on a moving vehicle) or under out-of-distribution motion patterns like sudden directional changes or objects approaching/receding. Have fallback strategies been designed for such cases, or can the authors quantify how much the reported ~400% speedup would degrade under these conditions? Would motion compensation or background separation mechanisms be required?\n8. The paper should provide quantitative comparisons of how input resolution and synapse-array block size (e.g., 4×4) affect key metrics such as IoU, processing speed, ROI accuracy, and miss rate. Providing tables or graphs showing how much temporal-cue information is lost at different resolutions would clarify the method’s tradeoffs.\n9. The supplementary demo shows that the system struggles to predict and track small, fast-moving objects like a table tennis ball (~10 px, ~20 px/frame). Reporting the detection failure rates for such conditions and analyzing how they impact the overall pipeline performance would help delineate the method’s limitations.\n10. Although the paper mentions the synapse identification time (1–2 ms), it does not provide a detailed breakdown of the total pipeline latency. A stacked chart or detailed summary showing contributions from ADC conversion, data transfer, synapse-array readout, ROI mask generation, and GPU computation would improve transparency and help identify bottlenecks.\n11. The reported performance improvements (e.g., 213.5%, 157.4%, 740.9%) lack accompanying standard deviations or p-values, making it difficult to assess statistical significance or reproducibility. Providing detailed statistical metrics (mean ± standard deviation, n ≥ 3) and significance testing would enhance the credibility of the results.\n12. While the GitHub repository is referenced, the ROI preprocessing and optical-flow integration code is not available. Could the authors provide the full codebase to ensure reproducibility", "reviewer_2": "I co-reviewed this manuscript with one of the reviewers who provided the listed reports. This is part of the Nature Communications initiative to facilitate training in peer review and to provide appropriate recognition for Early Career Researchers who co-review manuscripts.\nWhile the GitHub repository is referenced, I was unable to locate the ROI‑preprocessing and optical‑flow integration code used in the manuscript. For full reproducibility, please provide these missing modules (or indicate where they will be released) along with any necessary instructions for running them.", "reviewer_3": "This study proposes a novel spatiotemporal optical flow method that uses the dynamic switching behaviour of neuromorphic devices to encode motion information directly into the device state, thereby achieving ultra-fast optical flow calculation. Compared with the current in-sensor computing scheme that mainly relies on the device response characteristics to encode visual information, the encoding process of this method is adjustable. In addition, this method has no special requirements for the front-end optical imaging technology, and can even be a commercial optical sensor device, which greatly enhances the practicality of actual deployment. This encoding strategy can be extended to various dynamic visual tasks and effectively overcome the delay bottleneck in visual processing, which is very helpful for a variety of vision-dependent intelligent robot scenarios. If the author can well address my following questions about the mechanistic and technical aspects of materials and engineering, I recommend accepting this manuscript.\nHere are my main concerns:\n- In the introduction, the authors emphasize that floating-gate transistors achieve high-frequency response, strong retention, and excellent durability due to the characteristics of two-dimensional materials. Please describe the impact of atomic thickness and enhanced electrostatic control on these performance indicators. Is there a specific device design or material selection that can further optimize these characteristics?\n- In device testing, the authors reported that the Vgs amplitude required for weight modulation is lower than that of other studies. What aspects of the transistor design enable this reduction?\n-In the section “Motion Cues Generated by Neuromorphic Devices”, the authors describe the process of converting visual stimuli into motion cues, but do not specify the voltage pulse width applied to the neuromorphic device. Can you explain the pulse width range used, any limitations (e.g., minimum or maximum duration), and the maximum processing frequency it supports?\n-In this method, when modulating the neuromorphic device for subsequent optical flow calculations and task execution, device differences may introduce errors. How does neuromorphic optical flow mitigate the impact of device-to-device differences while maintaining or even improving processing accuracy?\n-In the discussion, the authors claim that the encoding method can be integrated with object detection networks such as YOLO. To demonstrate its versatility, please provide experimental results to demonstrate such integration. What specific requirements does the encoding method impose on downstream algorithms (scope of application)?\nHere are some minor issues:\n-Add reference numbers for other papers shown in Figure 2i\n-The core code flow has been demonstrated in the supplementary material. Please upload the code to the GitHub repository to reproduce the results.", "reviewer_4": "The authors propose a neuromorphic optical flow approach that encodes temporal information in a synaptic transistor array. The so-obtained temporal information is subsequently used for downstream tasks, to e.g., segment moving objects. Down-selecting the region of interest in this manner leads to substantial speed-ups in processing.\nI have a few main concerns with the article:\n1. Optical flow.\nThe authors focus on “neuromorphic optical flow”. However, if I understand correctly, the authors actually do not determine optical flow with the synaptic array, but values encoding the change in brightness over time (Equations 1-3). This is quite confusing. Optical flow is the apparent motion of objects in the scene due to relative motion between the objects and the observer. This apparent motion, captured by a 2D image sensor, has two dimensions (in the x- and y-direction). This is not the output of the synaptic transistor array, though. Indeed, later optical flow methods are applied to the regions of interest selected with the transistor array.\nOn the one hand, this should be made clear much earlier in the article. On the other hand, this is a bit of an anti-climax, since brightness changes can also be determined really efficiently in software. This is then limited by the camera’s frame rate, but it would form a good comparison for the authors’ study to show the benefits of the custom transistor array. Hence, the authors should in my eyes compare their method with a software-based one that determines the value of Eq 3 in software on images and performs the same down-stream tasks.\n2. Task difficulty\nThe method is applied to different tasks. In a first case, 800 x 800 pixels are downscaled to 4 x4 regions to which the array is applied. Large moving objects can then be detected. It is not really clear how good this method works. A major concern for me is that substantial ego-motion in well-lit, textured scenes, would trigger lots of brightness changes, leading to the array lighting up everywhere. This is very common for ego-motion when it includes rotations. The pedestrian task shows dark images in which the car is driving very slowly. The UAV operation images show a drone against a uniform background (in the video also against the sky over a landscape). Real-world applicability seems far away given these tasks.\n3. Event-based cameras.\nDuring the reading of the article, I have been continuously wondering when event-based cameras (neuromorphic cameras) would be mentioned. These are similar in spirit, and many algorithms have been devised (both neural and non-neural) to extract optical flow from the events that come from such cameras. These cameras offer the opportunity to have low-latency, low-power optical flow determination.\nThe authors only mention event-based cameras in the “Future directions” part, mentioning that: “By incorporating event-based cameras as the front-end sensor, the synapse array in the neuromorphic optical flow can be modulated by event streams, potentially delivering similar acceleration benefits for event-based optical flow methods.” How would this work? Events are ON- or OFF events, and the processing of brightness over time happens in the hardware before. It is non-evident to me how the proposed transistor array would be combined with neuromorphic cameras. Instead, this approach could (or should) be compared with them.\nAlthough I am sympathetic to the idea of down-selecting regions of interest based on motion cues, the concerns above currently do not make me very enthusiastic about the article."}, "author_response": "We sincerely appreciate the valuable time and efforts the reviewers have spent reviewing our manuscript. The insightful comments and feedback are instrumental in improving the quality of our work. We believe we have addressed all of the reviewer’s comments, resulting in a manuscript that is more rigorous in content and clearer in presentation. Our point-by-point responses to the reviewers’ comments are as follows.\nReviewer #1\nThis manuscript introduces a neuromorphic hardware-based approach to spatiotemporal optical flow by integrating temporal motion cues with spatial gradients through two-dimensional synaptic transistor arrays. The authors report substantial performance gains while achieving or surpassing human-level motion prediction, segmentation, and tracking speeds. While the novelty of the authors’ approach is impressive, several important points need to be addressed before the manuscript can be considered for publication.\nResponse:\nThank you for recognizing the novelty of our work and the performance improvement brought by our method. Our detailed responses to your valuable comments are provided below.\nComment #1\n1. The paper introduces “spatiotemporal optical flow” as a novel concept and applies it within a neuromorphic framework, which is indeed innovative. However, the Introduction section lacks a broader survey of artificial synapse-based signal processing beyond visual stimuli. Since voltage signals can be derived from other input modalities, a contextual overview of related signal-processing trends is need not to mislead the readers.\nResponse:\nThank you for your kind suggestion on providing a detailed overview of current signal-processing trends, the original manuscript has been revised as follows (Red represents modified or newly added text):\n(Article) On Page 3: Fortunately, neuromorphic devices such as synaptic transistors and memristors offer synapse-like characteristics that can emulate the processing functions of biological synapses. Their intrinsic adaptability enables on-device adaptation and local feature extraction functions, which in turn lowers both energy consumption and processing latency. This approach has already been applied across multiple sensory modalities. In a typical implementation, a front-end sensor first transduces external stimuli into electrical signals (voltage or current), which are then transmitted to synaptic devices for processing. For tactile perception, the sensor is typically a piezoresistive or piezoelectric film that converts mechanical force into an electrical response. Based on this architecture, artificial tactile receptors are widely studied—including rapidly adapting mechanoreceptors and nociceptors—using neuromorphic hardware1–4. For example, Wang et al. proposed a memristor-based perceptual signal processing method, which can emulate multiple essential tactile receptors and sensory neurons with a single memristor1. For auditory processing, neuromorphic device arrays can capture and process auditory temporal patterns directly in hardware, enabling low-latency, low-power front ends for sound recognition and localization5–7. For visual processing, the high spatial redundancy of video frames aligns perfectly with the local, in-memory processing capability of synaptic devices1,8–10. For example, Baek et al. recently reported a neuromorphic neuron with dendritic morphology based on silicon-nanowire transistors that can perform nonlinear integration of excitatory/inhibitory synaptic inputs while considering spatial distribution dependency. Its built-in direction selectivity enables on-device motion detection, and the power consumption of processing event-based pulses from the 1,000 pixels is about 0.1–2.0 mW11. More broadly, the on-device adaptation capability of synaptic devices makes human-like ultrafast visual processing possible 1,12–14.\nComment #2\n2. It is unclear whether the innovation lies in the specific design and construction of the synaptic transistor device or primarily in its application to optical flow. If the development of their 2D-based synaptic transistor itself is also the main focus, the underlying material science and device-level rationale are only briefly mentioned. A deeper explanation of why this device is particularly suitable (over other types of synaptic transistor) for spatiotemporal optical flow would strengthen the manuscript.\nResponse:\nThank you for offering us an opportunity to clarify. The primary novelty of our work is the proposed spatiotemporal optical flow framework, which couples temporal information generated in situ by the neuromorphic synapse array with spatial gradients extracted from image frames, thereby accelerating the entire pipeline—from velocity inference to downstream high-level processing tasks. This framework is general and can be integrated with other neuromorphic synaptic devices, including ferroelectric memristors, phase change memory, etc. Through the non-volatile characteristics of synaptic devices, our method enables efficient temporal cue generation directly in hardware, significantly reducing external latency compared to purely software-based approaches.\nTo better illustrate that, the following content has been added to the discussion part:\n(Article) On Page 10:\nContribution of neuromorphic optical flow methods\nThe primary novelty of our work is the proposed spatiotemporal optical flow framework, which couples temporal information generated in situ by the neuromorphic synapse array with spatial gradients extracted from image frames, thereby accelerating the entire pipeline—from velocity inference to downstream high-level processing tasks. This framework is general and can be integrated with other neuromorphic synaptic devices, including ferroelectric memristors, phase change memory, etc. Through the non-volatile characteristics of synaptic devices, our method enables efficient temporal cue generation directly in hardware, significantly reducing external latency compared to purely software-based approaches.\nComment #3\n3. Figure 2 implies that the pulse-number linearity and array-level uniformity are not particularly high, compared to recently-reported synaptic devices. The paper does not clarify whether the authors applied design optimizations, fabrication techniques, or algorithmic corrections to address these limitations. Commenting on whether improving these characteristics could further boost performance or how the possible error can be calibrated would be valuable.\nResponse:\nWe agree that pulse–number linearity and array-level uniformity are important for reliable operation. Compared to recently reported synaptic devices as shown in Table R1, our device metrics fall within the ranges commonly observed for synaptic transistors. The demonstrated analog weight updating under cumulative voltage pulses of the same polarity allows for the storage of temporal motion patterns, which is sufficient for our task. To further optimize the experiment results, the following design optimizations and algorithmic corrections have been utilized:\n(Article) On Page 9: Although device metrics such as uniformity and linearity (see STable 6) are within commonly reported ranges for synaptic transistors, the demonstrated analog weight updating under cumulative same-polarity voltage pulses enables reliable storage of temporal motion patterns sufficient for our tasks. Design optimizations and algorithmic corrections used in the experiments to achieve such high accuracy can be found in SNote 8.\nDesign optimizations and algorithmic corrections utilized to ensure robust processing abilities\n(SNote 8) On Page 15: To address pulse-number nonlinearity, specific design optimizations have been implemented. Specifically, to ensure robust processing abilities, the voltage pulses used to modulate the synaptic array is scaled by the reconfigurable op-amp to reside within the regime in which the synaptic transistor remains highly linear. As illustrated in Fig. 2c and Fig. R1, the device exhibits high linearity in this voltage window. Based on this measurement, the coefficients in Eq. 3 (plus , plus , bia and bia ) are tuned to make the modulation voltage 1 2 1 2 coincide with this voltage window, as shown in Fig. 3d. Thus, when deploying spatiotemporal optical flow with synapse devices, pulse-sweep experiments are performed firstly to determine the voltage window, and then reconfigurable op-amp parameters are adjusted to match this range.\nFor the residual array-level non-uniformity, such variations can be considered as inherent ‘noise’, adding to the ideal uniform conductance map15–18. To mitigate this non-uniformity, we employ two strategies in series: (i) a reconfigurable thresholding scheme during binarization to maintain sensitivity despite variability, and (ii) a spatial-continuity merge that leverages the connectivity of object motion to produce robust ROIs.\n(i) Reconfigurable thresholding scheme during binarization: The binarization threshold s used to translate the analog state information s into temporal cues s’ (see Fig. 4c and Eq. th R1) can be adjusted to accommodate device variations. In practice, lowering s increases th sensitivity so that states with added variation still map to s’ = 255 and are treated as potential motion. Through this method, when using Farneback algorithm for motion prediction in grasping tasks, despite artificially introduced device variations of up to 5%, equivalent performance can be achieved by setting an appropriate binarization threshold (e.g., s = 250; th Fig. R3). In our experiments, thresholds shown in STable 5 are set based on measured device variations, ensuring reliable extraction of dynamic motion regions for subsequent velocity inference and downstream tasks.\n255,s≥s\ns'=  th (1)\n0,otherwise\n(ii) Spatial continuity inherent to object motion. The inherent spatial continuity of moving objects implies that the physical extent of a single object is connected and coherently arranged in space. Therefore, even if the synaptic array response exhibits non-uniformities, merging these responses to create a ROI region ensures that the resulting mask effectively captures the entire moving object, as shown in Fig. R4. By integrating reconfigurable thresholding and spatial continuity considerations, our approach achieves robust visual processing performance despite device non-idealities.\nThrough these two strategies, the region of moving objects can be accurately extracted and covered, maintaining or even improving processing accuracy through adding temporal prior knowledge.\n(STable 6) Table R1. Comparison with prior works on linearity and array-level uniformity. The calculation method can be found in Fig. R1 and Fig. R2.\nLinearity (potentiation/depression) Array-level uniformity References\n1.5/5.9 N/A 19\n1.9/0.5 N/A 20\n1.51/−0.38 N/A 20\n1.3/−0.3 N/A 21\nN/A <12% 22\n0.96/−0.11 N/A 23\n1.53/0.47 9.76% 24\n1.26/0.80 11.38% This work\n(Fig. S22) Fig. R1. Linearity calculation. To calculate the nonlinearity of the floating gate synaptic transistor, the following equation is used with the data extracted from Fig. 2c24:\n((Gm −Gm )×w+Gm )1/m,if m≠0\nmax min min\n\nG=  G (2)\nG ×( min )w,if m=0\n min G\n max\nwhere G and G are the maximum and minimum conductance during potentiation\nmax min\n(conductance increase) and depression (conductance decrease), respectively; m is the parameter reflecting nonlinearity, and w is an internal variable, 0 to 1. The ideal linearity corresponds to m = 1 (perfect linear conductance update). The calculated m with extracted data from Fig. 2c is 1.26 for potentiation and 0.80 for depression.\n(Fig. S23) Fig. R2. Array-level uniformity analysis. Using the conductance distribution in Fig. 2l, the array-level uniformity—reported as the coefficient of variation, σ/μ×100%, is 11.38% (evaluated at the 16th programming pulse). Here, μ is the arithmetic mean of the measured conductance values across the array, and σ is the corresponding sample standard deviation.\n(Fig. S24) Fig. R3. Performance evaluation under simulated array variability. In simulation experiments, the ideal synaptic transistor update at cell (i,j) is denoted as Δw ideal(t+Δt). Device variability is introduced by adding a Gaussian noise term proportional to ij the ideal update value:\nΔw var(t+Δt) = Δw ideal(t+Δt)+ σΔw (t)ξ ,\nij ij ij ij\nwhere ξ is standard normal random variable, σ is the variation factor, Δw ideal(t+Δt) is the ideal ij ij\nupdate value without variability, and Δw var(t+Δt) is the updated value accounting for variability.\nij\na, Velocity inference and task execution results at a 1% variation factor. b, Results at a 3% variation factor. c, Results at a 5% variation factor. d, Prediction accuracy comparison using SSIM across the above variation settings, demonstrating consistent performance despite increased variability.\n(Fig. S25) Fig. R4. Enhancing system robustness via spatial continuity merging. Using a grasping scenario as an example, spatial continuity effectively merges the detection regions, enabling easy capture of the entire moving object. Therefore, merging based on spatial continuity significantly enhances robustness against device variations or response inconsistencies.\nPotential improvements\n(SNote 8) On Page 16: Further improvements in the device metrics, including pulse-number linearity and array-level uniformity, can increase the precision of ROI coverage, more completely capturing moving objects while avoiding background noise. In practical terms, this reduces the burden of forming temporal cues; the re-tuning of the morphological merge step could be simplified or even omitted, because the uniform array state would already cleanly present contiguous motion regions. Finally, the resulting cleaner ROIs would reduce background coverage and, in turn, decrease the number of pixels passed to downstream algorithms, improving overall throughput and end-to-end latency.\nComment #4\n4. In Figure 2i, the authors emphasize long retention of their device, yet the mechanisms responsible for this superior retention are not deeply explored. A more detailed discussion on the material and structural factors driving this performance would enhance the scientific depth of this manuscript.\nResponse:\nWe are grateful for the reviewer’s valuable comments. The extended retention time of the floating gate synaptic transistor ensures the seamless continuity of temporal information during neuromorphic optical flow processing. To more clearly clarify the mechanism underlying the device's excellent retention characteristics, we add the following text to the main text and supplementary information.\n(Article) On Page 4:\nBased on the Fowler-Nordheim tunneling mechanism, floating gate synaptic transistor precisely regulates the charge in the floating gate through the gate voltage, thus realizing the long-term stable storage of information and guaranteeing the continuity and reliability of neuromorphic optical flow processing in the time dimension.\n(Article) On Page 5:\nThe excellent retention characteristics are primarily attributed to the intrinsic physical properties of the material and the optimized device structure design, as detailed in the SNote 6.\n(SNote 6) On page 12: The excellent retention characteristic primarily arises from the intrinsic physical properties of the materials and the design of the device architecture. Specifically, in material selection, multilayer graphene (MLG) was employed as the floating gate layer due to its exceptional charge storage capacity and high density of states, which significantly enhances charge capture efficiency and ensures more stable retention performance.\nAtomically flat hexagonal boron nitride (h-BN), with no dangling bonds, serves as an ideal uniform tunneling layer for efficient charge injection. Additionally, its inherently low defect density enables it to effectively suppress charge leakage pathways, thereby enhancing the overall performance of the device. In atomic thin-layer MoS channels, the quantum 2 confinement effect in the vertical dimension precisely restricts the charge tunneling pathways, significantly reducing lateral leakage currents in floating-gate devices and thereby markedly enhancing charge capture efficiency. Furthermore, due to the absence of dangling bonds on the surface of two-dimensional materials, the van der Waals heterostructure composed of two-dimensional materials with distinct functional layers exhibits flat atomic interfaces. This significantly mitigates interfacial charge scattering, thereby ensuring superior retention performance.\nAt the structural design level, the ability of floating-gate synaptic transistors to achieve long-term stable storage of information stems from two core factors: first, the Fowler-Nordheim (F-N) tunneling mechanism inherently enables precise regulation of the charge within the floating gate; second, the unique structure of the floating gate possesses the inherent capability to stably trap charges. Furthermore, precise control over the thickness of h-BN tunneling layer is a critical parameter in optimizing the structural design of synaptic devices and significantly influences their overall performance. Through careful selection of the appropriate h-BN thickness, we have achieved three key objectives: First, ensuring that the floating gate layer can effectively retain charge after the removal of the gate voltage pulse; Second, maintaining a sufficiently low tunneling barrier to enable efficient charge transport under low gate voltage excitation; Third, enhancing the long-term operational stability and reliability of the device. Our device employs a 13-nm-thick h-BN tunneling layer, which exhibits superior long-term retention characteristic, sustaining performance for up to 104 s with negligible degradation. As the thickness of hBN decreases, the device's storage performance deteriorates markedly35. This degradation is predominantly attributed to the reduced thickness of h-BN, which results in a lower tunnel barrier and allows the injected charge to escape from the floating gate into the channel. As a result, the compromised ability of the floating gate to retain charges gives rise to an increase in leakage current and elevated off-state conductivity, which ultimately adversely impacts the storage properties. It is worth noting that the utilization of excessively thick tunneling layers is not advisable, particularly given that this would result in a higher charge tunneling energy barrier. If the channel material is excessively thick, it leads to weak electrostatic control capability because the electric field generated by the gate cannot adequately penetrate the entire channel thickness to modulate carriers. Consequently, higher-amplitude or wider-pulse-width gate voltage pulses would be required36. Thus, the thickness of the tunneling layer should be carefully optimized to achieve an optimal trade-off between data retention and operational efficiency. In previous reports, the barrier height between the tunneling layer (h-BN) and the channel layer (MoS ) was approximately 2 eV29. By selecting 2 an appropriate h-BN thickness, we can achieve a suitable barrier height, thus enabling the device to exhibit superior retention performance.\nComment #5\n5. Other works (e.g., Nano Energy 89, 106439 (2021); Nat. Nanotechnol. 17, 27 (2022)) have demonstrated photo-memory devices that directly convert light to stored electrical signals. It would strengthen the paper to explain the practical advantages of the authors’ split architecture (sequentially using an imaging array and synaptic-transistor temporal cues) over such designs, particularly regarding efficiency, energy use, or system complexity.\nResponse:\nWe appreciate your feedback about discussing the advantages of our separate design with photo-memory devices that simultaneously perform sensing, memory, and processing. The following discussion has been added to the Supplementary Information.\n(Article) On Page 10:\nNeuromorphic optical flow can also be realized with photo-memory devices; a comparison between our separated design and integrated photo-memory implementations is provided in SNote 9.\n(SNote 9) On Page 17:\nComparison between the proposed separate design and the integrated photo-memory devices\nOur proposed framework employs a sequential architecture comprising imaging and synaptic transistor arrays, where visual information initially captured by the front-end imaging sensor is converted into voltage pulses through design interfacing circuits. These pulses modulate the synaptic transistor array, encoding temporal information in terms of array states. In comparison, photo-memory devices37,38 can directly integrate sensing, memory and processing in a single device based on their inherent photo-responsive properties. Based on these characteristics, these devices can, in principle, generate similar temporal cues for neuromorphic optical flow methods.\nThrough decoupling the sensing of raw visual signals from memory and processing, the primary advantage of our architecture is its enhanced flexibility. The interfacing circuit that links the image array to the synaptic array can be re-tuned on demand, enabling the system to adapt to changing environments. For instance, when detecting traffic lights at night, the preset threshold V can be raised due to high contrast between the light source and its dark th surroundings, whereas a lower V is appropriate under daytime illumination. Conversely, th integrated photo-memory devices inherently couple sensing and encoding. Therefore, the adaptability and reliability of such systems will be decreased under extreme or variable illumination conditions. In addition, our architecture allows straightforward integration with diverse imaging front ends, which can adapt to diverse working conditions and facilitate potential commercialization.\nConsidering efficiency and energy consumption, the additional conversion circuits introduce minimal incremental power usage—estimated at 10 mW in total. This includes about 25 µW per BPW34 photodiode under moderate illumination and roughly 9.6 mW for the operational amplifiers OPA2320 (op-amps). This incremental energy is relatively insignificant when compared to the substantial power demands of downstream processing units, such as an Intel i9-12900H processor (minimum assured power of 35 W). As for complexity, the interface consists primarily of filters and op-amps, components already common in imaging pipelines and easily scaled with standard analog-IC design practices.\nIn summary, while photo-memory devices could offer elegant all-in-one sensing and processing, our split architecture has higher flexibility and the ability to integrate with commercial hardware, making it the more deployable option for robotics and autonomous-vision systems.\nComment #6\n6. There is a minor typo in Figure 3a: “voltgae” should be corrected to “voltage.”\nResponse:\nThanks for your careful checking. Typo has been corrected.\nRevised Figure 3a\nFigure 3. a, Architecture of the imaging system integrating temporal information with the original image. b, Mathematical representation of the circuit used to extract temporal information. c, Circuit diagram. d, Visual input containing analog visual voltage changes and the corresponding V applied to the synaptic transistor at location (2, 4). e, Visual input for gs the subsequent frame following d. f, Temporal information derived from the I values of the ds synaptic transistor array states. g. Temporal information scaled to the range of 0 to 255. h. Examples of processed visual input in the vehicle scenario between frames 25 and 26, and between frames 38 and 39.\nComment #7\n7. The demonstrations mainly cover conditions with static or slow-moving backgrounds and a single fast-moving object. It is unclear how the system performs when the background itself is moving rapidly (e.g., when mounted on a moving vehicle) or under out-of-distribution motion patterns like sudden directional changes or objects approaching/receding. Have fallback strategies been designed for such cases, or can the authors quantify how much the reported ~400% speedup would degrade under these conditions? Would motion compensation or background separation mechanisms be required?\nResponse:\nWe thank the reviewer for highlighting the importance of evaluating our system under dynamic scenarios with rapidly moving backgrounds, sudden directional changes, and varying object trajectories such as approaching or receding movements. To address this concern, we have included a comprehensive evaluation of our system’s performance in these challenging conditions within the Supplementary Information. We conducted two sets of experiments: (i) controlled sequences captured with a hand-held phone while the IMU records its own motion, (ii) real-world recordings from an in-car dashcam. In the first set of experiments, the speedup degraded from ~400% to about 170% speedup, and in the second experiment, the speedup degraded from ~400% to about 134% speedup, equivalent to an acceleration ratio of 74.8%. In both cases, the neuromorphic method remains faster than the conventional method. Additionally, we discuss potential fallback strategies, including motion compensation and background separation mechanisms, that could be integrated to enhance robustness further.\n(Article) On Page 9:\nNevertheless, in challenging scenarios involving ego-motion or out-of-distribution motion, the performance of our method may degrade. To evaluate its robustness under such conditions, we conducted a comprehensive evaluation detailed in SNote 10. Specifically, we performed two sets of experiments: (i) controlled sequences captured using a hand-held phone, with concurrent IMU recordings of device motion, and (ii) real-world recordings from an in-car dashcam. In the first experiment, the observed speedup decreased to 170%, while in the second, it declined from around 400% to 134%, corresponding to an acceleration ratio of 74.8%. Despite these reductions, the neuromorphic approach consistently outperforms conventional methods. Furthermore, we discuss potential motion compensation and fallback strategies that could be incorporated to further enhance system robustness (SNote 10).\n(SNote 10) On Page 18:\nPerformance evaluation under dynamic scenarios\nIn dynamic scenes involving ego-motion or out-of-distribution motion, background regions also exhibit obvious motion. Here, ”background” denotes objects that are stationary or slow-moving in the world (lab) frame. Nevertheless, our method continues to identify regions of interest (ROIs) to achieve acceleration based on the relative velocity contrast between background and moving objects of interest. This follows directly from the moving pattern encoding principle of our method: the modulation of the synapse array depends on the absolute light intensity derivative |∂I/∂t|, which scales with relative moving speed (Fig. R5). Let v denote camera cam ego motion and v object motion; the relative speed v is\nobj rel\nv = v −v (3)\nrel obj cam\nAs shown in Fig. R5, the background satisfies v ≈ 0 and thus v = ||v ||, whereas cross-obj rel cam traffic or opposing motion typically yields v > ||v ||, producing larger |∂I/∂t| compared to rel cam background information. Therefore, setting an appropriate threshold voltage V utilized in Eq. th 3 can omit background by modulating those corresponding synaptic devices into high resistance, while moving objects of interest modulate the corresponding synapse array region toward low resistance and are selected as ROIs.\nCo-moving objects with small v are de-prioritized to preserve throughput, similar to how rel human drivers prioritize crossing pedestrians over similarly moving vehicles. When such targets are required (e.g., lead-vehicle tracking), V can be lowered to recover them.\nth\nTo validate robustness, we conducted two sets of experiments: (i) controlled sequences captured with a hand-held phone while the IMU measured v during which V is linearly cam th scaled with v (ii) real-world recordings from an in-car dashcam. Acceleration and accuracy cam are quantified using Farneback optical flow for velocity inference, which represents the baseline performance of our pipeline (Fig. 6c). In both cases, ROI quality is maintained, and end-to-end speed-ups are preserved.\nPerformance evaluation\nIn the handheld phone capturing experiments, to consider safety, one participant is asked to hold the camera while walking or running at predefined speed levels. A second participant simultaneously ran within the scene. For each camera speed, the trajectories of both the camera and the runner followed three distinct motion patterns (Fig. R6a), covering out‑of‑distribution cases such as abrupt direction changes and approaching or receding motion. During the experiments, the phone’s on‑board IMU is continuously logged; its readings are integrated to obtain velocity, and the peak value is taken as the moving speed of the camera, as shown in Fig. R6b. Four levels are tested: baseline walking (~0.5 m/s) and three running speeds—low (~1.5 m/s), medium (~2.5 m/s), and high (~3.5 m/s). These visual data collected in the above speed levels with three moving patterns are simulated by a 20 × 13 synapse array, with V proportionally scaled according to the camera ego motion velocity v th cam at different running speeds:\nV =αv +β (4)\nth cam\nwith α and β tuned to 2 and -2 and held fixed across all runs.\nDetailed results can be found in Figs. R7–R18 and Tabel R2, demonstrating consistent improvements in processing efficiency and accuracy over conventional optical flow methods.\nSpecifically, acceleration ratios and accuracy improvements were as follows. As shown in Fig. R6c, in baseline speed, the average acceleration ratio is 55.9%, and the accuracy improvement ratio is 116.1%; in low-speed motion, the average acceleration ratio is 57.1%, and the accuracy improvement ratio is 113.2%; in medium-speed motion, the average acceleration ratio is 62.6%, and the accuracy improvement ratio is 109.4%; in high-speed motion, the average acceleration ratio is 62.1%, and the accuracy improvement ratio is 111.3%. Note that camera ego-motion introduces significant variations in background appearance. As neuromorphic optical flow selectively extracts critical motion features while filtering out irrelevant background fluctuations, a slight decrease in performance metrics is expected for the prediction task that requires comprehensive global image analysis. However, for object segmentation and tracking tasks, the selective extraction of key motion information substantially enhances overall performance. The marked improvement in processing efficiency and accuracy over conventional methods demonstrates that neuromorphic optical flow provides robust processing capabilities even under ego-motion conditions by adaptively tuning the threshold voltage V .\nth\nMoreover, in the multiple object detection scenario depicted in Fig. R19, the average acceleration ratio achieved is 29.1%, accompanied by an accuracy improvement ratio of 102.2%. Furthermore, as observed in representative key frames, our method accurately extracts velocity information for multiple objects simultaneously.\nIn challenging real-world scenarios captured by vehicle-mounted cameras, visual data from three distinct scenarios are simulated using a 20 × 13 synapse array with finely tuned threshold voltages. Similar results are observed in Fig. R20 to Fig. R22 and Table R3, demonstrating that the velocity information of key moving objects within these scenes can be accurately extracted, significantly accelerating processing while maintaining accuracy. Notably, the IoU metric in the object tracking task exhibited comparatively lower performance relative to other tasks. This limitation primarily stems from the downstream tracking algorithm, which, despite accurately extracting velocity information, occasionally identifies multiple objects that are not necessarily the primary objects of interest, thus decreasing overall tracking accuracy. On average, an acceleration ratio of 74.8% is achieved, alongside an accuracy improvement ratio of 101.7% (Fig. R23). These results collectively underscore the robustness and efficacy of neuromorphic optical flow, demonstrating performance improvements in processing speed without affecting accuracy even under more demanding operational conditions.\nPotential motion compensation and fallback strategies\nAs shown in the hand-held smartphone experiments, adapting the threshold voltage V based th on camera ego motion velocity ensures that key temporal moving patterns are mapped into the synaptic array state, thereby facilitating downstream inference. Specifically, V is linearly th scaled with camera ego motion. In subsequent deployments, ego-motion from an IMU (or an equivalent sensor) can be used to dynamically compensate global motion and maintain robust ROI selection. Furthermore, fallback strategies can also be implemented to improve system robustness. For instance, upon detecting rapid, scene-wide intensity fluctuations, the system could temporarily revert to computing optical flow for the entire frame until signal stability is restored. As demonstrated in Fig. 6 and Fig. S15, this fallback adds only ~1–2 ms for ROI identification—negligible relative to the end-to-end latency—while preserving accuracy across diverse, challenging scenarios.\n(STable 7) Table R2. Detailed performance metrics in the handheld phone capturing experiments.\nScenario tpreo tpren tsego tsegn tobo tobn SSIMo SSIMn PAo PAn IoUo IoUn\nPattern 1 0.11 0.06 0.13 0.07 0.11 0.06 0.93 0.88 0.92 0.99 0.18 0.22\nBaseline\nPattern 2 0.13 0.05 0.14 0.05 0.12 0.04 0.93 0.87 0.89 0.98 0.12 0.20\nspeed\nPattern 3 0.12 0.09 0.13 0.10 0.11 0.09 0.90 0.87 0.61 0.64 0.08 0.13\nPattern 1 0.09 0.05 0.11 0.06 0.08 0.05 0.91 0.84 0.79 0.90 0.19 0.34\nLow-\nspeed Pattern 2 0.09 0.05 0.10 0.05 0.08 0.04 0.93 0.90 0.88 0.98 0.24 0.31\nmotion\nPattern 3 0.09 0.05 0.10 0.06 0.08 0.05 0.87 0.81 0.80 0.88 0.20 0.20\nPattern 1 0.09 0.06 0.10 0.07 0.08 0.06 0.83 0.79 0.81 0.91 0.27 0.37\nMedium-\nspeed Pattern 2 0.09 0.04 0.10 0.04 0.08 0.04 0.84 0.77 0.85 0.95 0.24 0.29\nmotion\nPattern 3 0.09 0.06 0.11 0.07 0.09 0.07 0.77 0.70 0.62 0.68 0.12 0.13\nPattern 1 0.09 0.06 0.10 0.07 0.09 0.06 0.79 0.68 0.69 0.78 0.13 0.17\nHigh-\nspeed Pattern 2 0.09 0.05 0.11 0.06 0.09 0.05 0.83 0.73 0.74 0.82 0.09 0.14\nmotion\nPattern 3 0.10 0.07 0.11 0.07 0.09 0.06 0.73 0.61 0.56 0.67 0.06 0.07\nMultiple objects 0.09 0.03 0.10 0.03 0.09 0.03 0.96 0.96 0.98 0.99 0.39 0.41\nIn Table R2, the subscript o denotes the conventional pipeline, and n denotes the neuromorphic (ours) pipeline. Accordingly, t /t are the per-frame times for the motion preo pren prediction task, t /t are the per-frame times for the object segmentation task, and t /t sego segn obo obn are the per-frame times for the object tracking task. SSIM /SSIM are the Structural Similarity o n Index Measure about prediction accuracy, PA /PA are pixel accuracy, and IoU /IoU are o n o n intersection-over-union. For all time metrics, lower values are better; for SSIM, PA, and IoU, higher values are better. All times are reported in seconds per frame, and the accuracy metrics are unitless in [0, 1].\n(STable 8) Table R3. Comprehensive performance metrics from the in-car dashcam experiments.\nScenario tpreo tpren tsego tsegn tobo tobn SSIMo SSIMn PAo PAn IoUo IoUn\nScenario 1 0.32 0.25 0.41 0.29 0.30 0.25 0.86 0.83 0.78 0.79 0.16 0.16\nScenario 2 0.35 0.25 0.42 0.29 0.34 0.27 0.88 0.85 0.80 0.81 0.16 0.17\nScenario 3 0.29 0.22 0.40 0.27 0.28 0.22 0.87 0.87 0.87 0.90 0.09 0.10\nTable R3 reports the same metrics as Table R2: subscripts o and n denote the conventional and neuromorphic pipelines, respectively. Lower is better for t , t , and t ; higher is better pre seg ob for SSIM, PA, and IoU. Times are in seconds per frame; accuracy metrics are unitless in [0, 1].\n(Fig. S26) Fig. R5. Model relating synapse array modulation to scene-relative speed. a, Basic setting. b, Relative speed determines the magnitude of dI/dt that drives synapse array modulation. Comparing this signal to the hardware threshold voltage V separates th background from moving objects of interest.\n(Fig. S27) Fig. R6. Setting and results evaluation in the handheld phone capturing experiments. a, Experimental setup with three motion patterns. b, Camera ego-motion speed estimated from the on-board IMU. c, Processing time and task performance across speed levels.\n(Fig. S28) Fig. R7. Performance analysis of motion pattern 1 under baseline velocity conditions. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S29) Fig. R8. Performance analysis of motion pattern 2 under baseline velocity conditions. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S30) Fig. R9. Performance analysis of motion pattern 3 under baseline velocity conditions. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S31) Fig. R10. Performance analysis of motion pattern 1 under low-speed conditions. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S32) Fig. R11. Performance analysis of motion pattern 2 under low-speed conditions. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S33) Fig. R12. Performance analysis of motion pattern 3 under low-speed conditions. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S34) Fig. R13. Performance analysis of motion pattern 1 under medium-speed conditions. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S35) Fig. R14. Performance analysis of motion pattern 2 under medium-speed conditions. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S36) Fig. R15. Performance analysis of motion pattern 3 under medium-speed conditions. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S37) Fig. R16. Performance analysis of motion pattern 1 under high-speed conditions. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S38) Fig. R17. Performance analysis of motion pattern 2 under high-speed conditions. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S39) Fig. R18. Performance analysis of motion pattern 3 under high-speed conditions. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S40) Fig. R19. Performance analysis when detecting multiple objects. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S41) Fig. R20. Performance analysis of challenging scenario 1. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S42) Fig. R21. Performance analysis of challenging scenario 2. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S43) Fig. R22. Performance analysis of challenging scenario 3. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S44) Fig. R23. Average performance analysis of challenging scenarios.\nComment #8\n8. The paper should provide quantitative comparisons of how input resolution and synapse-array block size (e.g., 4×4) affect key metrics such as IoU, processing speed, ROI accuracy, and miss rate. Providing tables or graphs showing how much temporal-cue information is lost at different resolutions would clarify the method’s tradeoffs.\nResponse:\nWe thank the reviewer for highlighting the need for quantitative comparisons of how input resolution and synapse array size influence key performance metrics. To address this comment, we have performed additional theoretical analysis and experimental verification.\nImportantly, the critical parameter affecting performance is the sensor-to-synapse pooling block size p×q (Fig. R24), which is jointly determined by the input resolution (m,n) and the array size M×N. This coupling yields p=m/M and q=n/N; hence, changing either resolution or array size changes the same underlying variable: the number of pixels pooled per synapse. Thus, in the following experiments, the number of pixels pooled per synapse is changed, and the corresponding results are evaluated.\n(SNote 11) On Page 21:\nEffects of sensor-to-synapse pooling size on neuromorphic optical flow performance\nWhen mapping scene visual information of resolution (m,n) to the synapse array of size M×N, each synaptic device receives the spatial average of a local group of neighboring pixels. This group is termed the sensor-to-synapse pooling block of size p×q pixels (Fig. R24). Under the uniform tiling, it can be calculated that p=m/M and q=n/N. Let the raw light intensity be I(u,v,t)∈Rm×n, and u and v represent location. Thus, the temporal derivative of visual voltage r I driving synapse (i,j) is:\ni,j\ndI 1 (i+1)p−1(j+1)q−1∂I (u,v,t)\ni,j =   r (5)\ndt pq ∂t\nu=ip v=jq\nThis spatial averaging suppresses the high-frequency noise of raw visual data. When p×q is small, motion detail is preserved, but noise increases. With appropriate adjustment of V , th background can still be separated from moving objects of interest. Conversely, if p×q becomes too large relative to the object scale, motions are averaged into the background, reducing ROI separability even after retuning V .\nth\nQuantitative evaluation\nTwo representative scenarios, vehicle and grasp, are simulated using the synapse array of different pooling sizes. In the following analysis, we set p = q and report results using a single pooling parameter p.\nIn vehicle operation, with input resolution held fixed, increasing the synapse array to reduce the pooling size from 200 to 100 and 50 pixels, the acceleration ratio and task accuracy remain essentially unchanged. This stability arises because of retuning the threshold voltage V . In th contrast, enlarging the pooling size to 400 (corresponding to a 2×2 synapse array), both acceleration and accuracy significantly degrade (Fig. R25b). This performance reduction primarily arises from the increased spatial smoothing inherent at larger pooling block sizes, causing critical motion information to blend into background signals and reducing the accuracy of region-of-interest (ROI) identification. Note that the ROI accuracy can be reflected in the Intersection-over-Union (IoU) metric in object tracking tasks. The reduction in ROI accuracy can lead to either excessive background inclusion or incomplete object coverage; thus, both task accuracy and processing acceleration will be degraded. Regarding miss rate, a noticeable increase is observed at an input resolution of 400, characterized by two consecutive frame omissions during motion (miss rate 0.0392), while performance at higher resolutions remained robust. Detailed results can be found in Table R4.\nSimilar results can be obtained for the grasping operation (Fig. R26). Here, performance metrics remained largely consistent when moderately increasing the pooling block size from the original 80 to 160 with input resolution fixed. However, when further increasing the pooling block size to 320, a substantial decline in these metrics is observed. It indicates that our neuromorphic optical flow approach exhibits considerable tolerance to pooling block size adjustments within a certain range. Nevertheless, when the pooling block size significantly exceeds the size of target objects, performance metrics sharply decrease. Notably, in the grasping scenario, the miss rate consistently remained zero, as the key object was continuously detected during velocity inference. However, incomplete object representation occasionally occurred at excessively low resolutions, as shown in Fig. R26a, impacting overall task performance metrics.\nIndicated by these results, the neuromorphic optical flow method demonstrates a reasonable tolerance to small pooling block size, but its performance could be degraded when the pooling block size is far larger than the moving details. In our experiments, robust performance is achieved by the selected resolution, which aligns closely with the size scale of key visual features. With advancements in synapse-array integration, higher-density arrays are expected to further enhance the capacity of neuromorphic optical flow methods, ensuring that the pooling size remains sufficiently small.\n(STable 9) Table R4. Quantitative evaluation of processing performance for varying sensor-to-synapse pooling block sizes (p×q).\nScenario Block size Gpre Gseg Gob Spre Sseg Sob Miss rate\n50 0.9980 1.0039 1.8972 0.0647 0.0625 0.0656 0\n100 0.9987 1.0051 1.9848 0.0787 0.0750 0.0792 0\nVehicle\n200 0.9992 1.0050 1.7963 0.0822 0.0789 0.0830 0\n400 1.0029 1.0053 0.6143 0.2779 0.2703 0.2910 0.0392\n40 0.9998 1.0033 28.7344 0.0624 0.0528 0.0568 0\n80 0.9998 1.0031 20.6503 0.1346 0.1252 0.1346 0\nGrasping\n160 0.9993 1.0034 25.5042 0.0932 0.0796 0.0899 0\n320 0.9987 1.0022 4.8949 0.2001 0.1820 0.2022 0\nIn Table R4, G denotes accuracy gain, defined as the ratio of the neuromorphic metric to the conventional baseline, where values > 1 indicate improved accuracy; S denotes the acceleration ratio, defined as the neuromorphic processing time relative to the conventional baseline, where values < 1 indicate the neuromorphic pipeline is faster. The subscripts pre, seg, and obj refer to the prediction, segmentation, and object tracking tasks, respectively. The miss rate is computed as the number of consecutively missed frames divided by the total number of processed frames.\n(Fig. S45) Fig. R24. Sensor-to-synapse pooling. An input frame of resolution (m,n) is mapped to a synapse array of size M×N by spatially averaging local p×q pixel neighborhoods per synapse.\n(Fig. S46) Fig. R25. Performance analysis when changing the input resolution of vehicle scenarios.\n(Fig. S47) Fig. R26. Performance analysis when changing the input resolution of grasp scenarios.\nComment #9\n9. The supplementary demo shows that the system struggles to predict and track small, fast-moving objects like a table tennis ball (~10 px, ~20 px/frame). Reporting the detection failure rates for such conditions and analyzing how they impact the overall pipeline performance would help delineate the method’s limitations.\nResponse:\nWe appreciate the reviewer’s suggestion to evaluate the system’s limitations regarding small, fast-moving objects, such as a table tennis ball (~10 px, ~20 px/frame). Upon further analysis, we confirm the consistent detection of the moving ball across all frames in our evaluated scenarios. Possible detection failure, impact assessments on overall pipeline performance, and limitations under varying motion conditions have now been comprehensively incorporated into the Supplementary Information to transparently delineate the boundaries and performance metrics of our method.\n(Article) On Page 8:\nNotably, the table-tennis sequences did not exhibit missed detections despite the small object size, owing to an appropriately chosen sensor-to-synapse pooling size, as detailed in SNotes 11–12.\n(SNote 12) On Page 23:\nLimitations in Tracking Small, Fast-Moving Objects\nWhen processing visual information, small and fast-moving objects present are more difficult to detect and may lead to occasional misses. However, our system demonstrates the capability to robustly detect the continuous motion dynamics of such small objects in the table tennis scenario, as shown in the following Fig. R27.\nThis performance is a direct result of the pooling analysis detailed in Response to Comment 8. By selecting an appropriate sensor-to-synapse pooling block size, the temporal moving signal of objects can be distinguished from background information and is not averaged out. In this experiment, a pooling block size of 10×10 pixels is suitable, as it aligns with the ball’s image footprint of approximately 8×8 pixels. After adjusting the threshold voltage (V ), the th intensity fluctuations caused by the ball remain above the detection criterion and can be reliably identified (miss rate 0%). Consequently, increasing the array size allows for a reduction in pooling block size, which improves sensitivity to small targets and enhances the detection capability of the neuromorphic optical flow.\nNevertheless, a detection miss can occur if the object footprint is substantially smaller than the pooling block size (Eq. R6), particularly when the sensor array size is fixed.\nA  A (6)\nobj pool\nwhere A and A are the areas (in pixels) of the object and the pooling block. In practice, obj pool we operationalize ”≪” as A /A ≤ 0.1, i.e., one order of magnitude smaller. Besides, an obj pool object moving too rapidly may exit the frame between captures, also leading to a miss. As a result, these objects will not be considered in the subsequent velocity inference and task execution. As a result, the local SSIM, PA, and IoU can drop to 0, while the global metrics for the entire image may still appear acceptable. Fallback full-frame processing can be activated to detect such movement. Specifically, the system temporarily computes optical flow for the full frame. As demonstrated in Fig. 6 and Fig. S15, this fallback introduces only ~1–2 ms of extra processing time for identifying potential moving regions. This delay is negligible compared to the overall pipeline latency, and it ensures 100% accuracy ratio in the same tasks as the original method since all regions are calculated.\n(Fig. S48) Fig. R27. Detection results of small, fast-moving objects (in table tennis scenarios). Object tracking results are illustrated in the following frames to demonstrate detection performance. The table-tennis ball was successfully detected across all frames. In Frame 5, the brief loss of tracking accuracy results primarily from limitations of the tracking algorithm itself; however, the velocity inference for this frame, derived from spatial cues provided by synapse array state information, correctly identifies the region of interest, underscoring the robustness of the underlying detection mechanism.\nComment #10\n10. Although the paper mentions the synapse identification time (1–2 ms), it does not provide a detailed breakdown of the total pipeline latency. A stacked chart or detailed summary showing contributions from ADC conversion, data transfer, synapse-array readout, ROI mask generation, and GPU computation would improve transparency and help identify bottlenecks.\nResponse:\nThank you for highlighting the need to provide a detailed breakdown of the total pipeline latency. We have added the following stacked analysis and bottleneck discussion to the Supplementary Information.\n(SI):\n(STable 10) Table R5. Total pipeline latency. Our pipeline comprises three stages: (1) Temporal cue collection via the synaptic array (covering ADC conversion, synapse-array readout, and data transfer), (2) Identification of potential motion regions (ROI mask generation), and (3) Accelerated velocity inference and task execution (GPU/CPU) as shown in Fig. R28.\nEstimated Share of\nStage Components\nlatency (ms) total\n1. High-pass filtering\n2. Op-amp tracking\n(1) Temporal cue collection via 3. Synaptic device\n1.2 2.9%\nsynaptic array programming\n4. ADC conversion/readout\n5. Data transfer\n(2) ROI mask generation OpenCV (Python) 1–2 4.9%\n(3) Accelerated velocity inference &\nGPU/CPU kernels 37.6 92.2%\ntask execution\nSpecifically, within stage 1, for high-pass filtering, the resistor-capacitor filter used here (R=1 kΩ, C=10 nF) contributes a settling time of approximately 46 µs, consistent with the time constant τ (4.6 τ rule for first-order systems). For signal amplification, the gain–bandwidth product of the op-amp, which tracks the filtered signal, does not significantly influence this settling time. The synaptic transistor programming pulse and device settling contribute approximately 1 ms. For the analog-to-digital conversion, using a 16-bit SAR ADC (ADS8168) throttled to 0.5 MSPS for extra settling head-room, one conversion, including sample-and-hold (2 µs), 24-clock serial read-out (1.5 µs), acquisition (2 µs) and a 10 % guard-band, takes ≈ 6 µs, so the full 16-channel scan for a 4×4 synapse array completes in ≈ 0.10 ms. Finally, USB-3 bulk data transfer to the host PC (32-byte payload at 400 MB/s) adds an insignificant latency of approximately 0.02 ms. The time consumption of this stage is estimated at 1.2 ms. At stage 2, currently performed using OpenCV in Python, it is measured as ~1–2 ms. Accelerated velocity inference and task execution (stage 3), executed on a CPU or GPU, takes an average of 37.6 ms, representing a substantial improvement compared to the original duration of 288.2 ms. This corresponds to an 86.96% reduction in latency, making it the dominant contributor to end-to-end processing efficiency improvement (see Fig. 6).\n(Fig. S49) Fig. R28. Time breakdown of each stage in the entire pipeline. Temporal cue collection via the synaptic array requires approximately 1.2 ms, potential motion region identification has an upper-bound estimate of 2 ms, and accelerated velocity inference and task execution take around 37.6 ms.\nComment #11\n11. The reported performance improvements (e.g., 213.5%, 157.4%, 740.9%) lack accompanying standard deviations or p-values, making it difficult to assess statistical significance or reproducibility. Providing detailed statistical metrics (mean ± standard deviation, n ≥ 3) and significance testing would enhance the credibility of the results.\nResponse:\nWe appreciate the reviewer’s suggestion to provide additional statistical context to support our reported improvements. The performance improvements initially presented reflect averaged outcomes from a single scenario across multiple tasks, each employing different velocity-inference methods. To strengthen statistical rigor and reproducibility, we have added detailed statistical analyses, including mean ± standard deviation (n ≥ 3) and significance testing.\n(Article) On Page 8:\nThe comprehensive statistical metrics can be found in Fig. R29, Fig. R30, and Fig. R31, clearly indicating that the main source of the observed performance enhancement is the improved accuracy in object tracking tasks. Specifically, these performance improvements are attributed to the additional environmental knowledge embedded in the temporal cue. For instance, in the RAFT-based object segmentation for grasping operations (Fig. 5b), RAFT cannot infer velocity accurately due to its limited generalization. However, the temporal cue provides a boundary constraint, enhancing segmentation accuracy. Similarly, in object tracking tasks, the temporal cue highlights the region containing the moving object while excluding irrelevant regions. As a result, the tracking accuracy is improved by reducing the impact of noise. The results of significance testing, summarized in Table R6, statistically confirm these improvements and underscore the robustness and reproducibility of our approach.\n(Fig. S50) Fig. R29. Statistical analysis (mean ± SD) of performance improvements achieved using different velocity inference algorithms in the vehicle operation scenario across all processed frames. Metrics shown include SSIM for prediction, PA (normalized by a factor of 100) for segmentation, and IoU for object tracking.\n(Fig. S51) Fig. R30. Statistical analysis (mean ± SD) of performance improvements achieved using different velocity inference algorithms in the UAV operation (small) scenario across all processed frames. Metrics shown include SSIM for prediction, PA (normalized by a factor of 100) for segmentation, and IoU for object tracking.\n(Fig. S52) Fig. R31. Statistical analysis (mean ± SD) of performance improvements achieved using different velocity inference algorithms in the grasping scenario across all processed frames. Metrics shown include SSIM for prediction, PA (normalized by a factor of 100) for segmentation, and IoU for object tracking.\n(STable 11) Table R6. Significance testing for performance improvements across different velocity-inference algorithms and operational scenarios. This table summarizes the statistical significance of performance improvements using three distinct metrics—SSIM (for prediction), PA (for segmentation), and IoU (for object tracking). Three statistical tests are performed:\n1. Shapiro p: p-value from the Shapiro-Wilk normality test, assessing whether the data distribution deviates significantly from normality. Lower values (typically <0.05) indicate significant deviation from a normal distribution.\n2. t p: p-value from the Student’s t-test, measuring the statistical significance of differences between two normally distributed groups. Values <0.05 indicate statistically significant differences.\n3. Wilcoxon p: p-value from the Wilcoxon signed-rank test, a non-parametric test used when data distributions significantly deviate from normality. Again, values <0.05 denote statistically significant differences.\nScenario-specific results are detailed, demonstrating the robustness and reproducibility of the observed improvements across various operational contexts and algorithms.\nInference\nScenario Metric Shapiro p t p Wilcoxon p\nalgorithm\nSSIM 1.81E-02 6.82E-01 6.60E-01\nFarneback PA 5.75E-06 2.31E-09 9.63E-10\nIoU 3.75E-01 4.16E-04 9.52E-04\nSSIM 3.22E-03 1.89E-02 4.12E-02\nAutodriving GMFlow PA 2.15E-09 7.40E-01 7.19E-02\nIoU 3.65E-01 2.68E-05 1.91E-05\nSSIM 6.29E-02 1.91E-01 3.11E-01\nRAFT PA 1.93E-05 6.26E-02 3.53E-02\nIoU 2.99E-01 3.91E-05 2.67E-05\nSSIM 5.73E-01 7.03E-01 5.78E-01\nFarneback PA 2.44E-11 9.34E-02 1.30E-01\nIoU 6.64E-03 2.31E-10 1.20E-08\nSSIM 2.90E-01 4.86E-21 2.47E-07\nUAV (small) GMFlow PA 5.21E-06 1.77E-15 5.82E-11\nIoU 4.43E-01 2.15E-13 5.82E-11\nSSIM 3.96E-01 3.69E-16 2.46E-07\nRAFT PA 1.93E-07 3.45E-02 2.63E-02\nIoU 5.16E-02 1.82E-02 5.03E-02\nSSIM 1.24E-02 1.41E-13 4.94E-11\nFarneback PA 1.01E-05 5.10E-33 8.33E-18\nIoU 1.33E-09 1.24E-35 8.33E-18\nSSIM 9.38E-11 3.23E-108 8.29E-18\nGrasping GMFlow PA 1.44E-08 2.14E-94 8.33E-18\nIoU 1.14E-08 1.14E-37 8.08E-18\nSSIM 6.88E-03 6.69E-85 8.16E-18\nRAFT PA 2.72E-07 1.14E-53 8.59E-18\nIoU 1.09E-08 1.16E-37 8.08E-18\nComment #12\n12. While the GitHub repository is referenced, the ROI preprocessing and optical-flow integration code is not available. Could the authors provide the full codebase to ensure reproducibility.\nResponse:\nWe appreciate the reviewer highlighting this point. To fully ensure reproducibility, we have now uploaded the complete codebase, including the ROI preprocessing and optical flow algorithms, to the GitHub repository https://github.com/RTCartist/Neuromorphic-Spatiotemporal-Optical-Flow.\nReviewer #2\nReviewer #2 (Remarks to the Author):\nI co-reviewed this manuscript with one of the reviewers who provided the listed reports.\nThis is part of the Nature Communications initiative to facilitate training in peer review\nand to provide appropriate recognition for Early Career Researchers who co-review\nmanuscripts.\nReviewer #2 (Remarks on code availability):\nWhile the GitHub repository is referenced, I was unable to locate the ROI-preprocessing\nand optical-flow integration code used in the manuscript. For full reproducibility, please\nprovide these missing modules (or indicate where they will be released) along with any\nnecessary instructions for running them.\nResponse:\nThank you for recognizing the contribution of our work and feedback that helps improve the quality of our work. Point-by-point responses have been provided and necessary codes to reproduce our results have been uploaded.\nReviewer #3\nReviewer #3 (Remarks to the Author):\nThis study proposes a novel spatiotemporal optical flow method that uses the dynamic switching behaviour of neuromorphic devices to encode motion information directly into the device state, thereby achieving ultra-fast optical flow calculation. Compared with the current in-sensor computing scheme that mainly relies on the device response characteristics to encode visual information, the encoding process of this method is adjustable. In addition, this method has no special requirements for the front-end optical imaging technology, and can even be a commercial optical sensor device, which greatly enhances the practicality of actual deployment. This encoding strategy can be extended to various dynamic visual tasks and effectively overcome the delay bottleneck in visual processing, which is very helpful for a variety of vision-dependent intelligent robot scenarios. If the author can well address my following questions about the mechanistic and technical aspects of materials and engineering, I recommend accepting this manuscript.\nResponse:\nThank you for recognition of our work. Our detailed responses to your valuable comments are provided below.\nComment #1\n- In the introduction, the authors emphasize that floating-gate transistors achieve high-frequency response, strong retention, and excellent durability due to the characteristics of two-dimensional materials. Please describe the impact of atomic thickness and enhanced electrostatic control on these performance indicators. Is there a specific device design or material selection that can further optimize these characteristics?\nResponse:\nThank you for your valuable comments. Leveraging the superior properties of two-dimensional materials, such as atomic thickness and enhanced electrostatic control, the developed floating gate synaptic transistors demonstrate high-frequency response, robust retention, and exceptional endurance.\nThe atomic thickness substantially shortens the carrier migration path, effectively mitigates scattering effects, improves carrier mobility, and consequently establishes a robust foundation for achieving high-frequency response. Enhanced electrostatic control enables more efficient modulation of the channel charge density via gate voltage, leading to improved response speed and a lower operating voltage. Furthermore, due to the quantum confinement effect of two-dimensional materials, the electric field demonstrates enhanced control over the vertical dimension of atomic-scale channel materials, leading to improved charge capture efficiency and comprehensive device performance.\nBased on the material selection and device architecture design, the fabricated floating-gate synaptic transistors exhibit a high-frequency response (~100 μs), robust retention (>104 s), and exceptional endurance (>8,000 cycles). We selected 6-nm-thick MoS as the channel 2 material, 13-nm-thick h-BN as the tunneling layer, and 8-nm-thick graphene as the floating gate layer to design and fabricate van der Waals heterojunctions. In terms of transistor design, we adopt extended floating gate, and utilize high-K dielectric Al O as the blocking layer to 2 3 enhance electrostatic control.\nAccordingly, the manuscript is revised as below:\n(Article) On Page 5:\nThe excellent performance of the device, including fast response, long endurance, and stable retention, can be attributed to the selection of materials and thicknesses of each functional layer, the enhanced gate coupling ratio29, and the establishment of atomically sharp and flat interfaces 25,39–41; a detailed comparison is provided in Fig. S7 and STable 1.\n(SNote 7) On Page 14:\nEnhanced electrostatic control enables more efficient modulation of the channel charge density via gate voltage, leading to a lower operating voltage. Zhou et al. demonstrated through theoretical calculations based on the two-triangle-barrier modified F-N tunnelling model that enhanced electrostatic control can significantly reduce operating voltage29.\n(SNote 6) On Page 13:\nTo further optimize the overall performance of the device, it can be achieved by replacing channel materials with higher mobility, optimizing the thickness of individual functional layers, shortening the channel width, adopting high-K gate dielectric layer (e.g., Al O /HfO ), and 2 3 2 conducting annealing treatment (H /Ar at 400 °C for 3 hours) to improve the quality of the van 2 der Waals interface.\nComment #2\nIn device testing, the authors reported that the Vgs amplitude required for weight modulation is lower than that of other studies. What aspects of the transistor design enable this reduction?\nResponse:\nThank you for your valuable comments. We successfully achieved weight regulation at a lower V amplitude of 7 V by enhancing the gate coupling ratio29,42. To provide a clearer explanation gs of the reason why a relatively small gate voltage amplitude is sufficient for weight modulation, we have included the following text in the main text and the Supplementary Information.\n(Article) On Page 5:\nWe successfully achieved weight regulation at a lower negative V amplitude of 7 V by gs enhancing the gate coupling ratio35,36, as detailed in SNote 7.\n(SNote 7) On Page 14:\nIn terms of transistor design, we select thin channel materials, optimize the tunnel layer thickness, adopt extended floating gate, and utilize high-K dielectric Al O as the blocking layer 2 3 to enhance the gate coupling ratio29,42. Due to the quantum confinement effect inherent in two-dimensional materials, the careful selection of thin channel materials can significantly enhance electrostatic control capabilities. The tunnel layer thickness was carefully optimized to ensure charge retention in the floating gate layer after the removal of the gate voltage pulse while maintaining a relatively low tunneling barrier (~2 eV), enabling efficient tunneling under smaller gate voltages. By connecting a large metal electrode to the float gate, the overlapping area between the float gate and the control gate is increased, thereby enhancing the capacitance of the blocking layer and subsequently improving the gate coupling ratio. The control gate capacitance is increased by using a high-K dielectric Al O to enhance the gate coupling ratio, 2 3 thereby significantly reducing the voltage required for floating gate charge injection.\nComment #3\nIn the section “Motion Cues Generated by Neuromorphic Devices”, the authors describe the process of converting visual stimuli into motion cues, but do not specify the voltage pulse width applied to the neuromorphic device. Can you explain the pulse width range used, any limitations (e.g., minimum or maximum duration), and the maximum processing frequency it supports?\nResponse:\nThank you for highlighting the need to clarify the experimental details regarding voltage pulse width and processing frequency. We have added a detailed discussion into our Methods section as follows:\n(Article) On Page 6:\nBy comparing it with the threshold V , dramatic changes caused by potential moving objects th and mild changes caused by background movement or noise are separated and translated into negative and positive V pulses, respectively, resulting in different trends of device state gs switching. The pulse width and the maximum supported processing frequency are detailed in SNote 13.\n(SNote 13) On Page 24:\nPulse width requirements and maximum processing frequency:\nIn our vehicle and UAV experiments, the voltage pulse width applied to the synaptic transistor array was set at 1 ms, determined by balancing real-time processing requirements and device response characteristics.\nFor real-time processing requirements, the pulse width should be shorter than the sampling interval of the front-end image array to avoid accumulated delay; otherwise, new frame data might arrive while the synaptic array is still processing the information of the previous frame. Given typical visual information frame rates between 25 Hz and 60 Hz (approximately 17–40 ms per frame), the selected pulse width of 1 ms comfortably meets the real-time processing constraint, being at least an order of magnitude shorter than the sampling interval. This acts as a practical upper bound for the pulse width.\nRegarding device limitations, our fabricated synaptic transistor exhibits rapid switching capabilities, achieving a response time of approximately 100 μs, as illustrated in Fig. 2e. This response time sets the theoretical lower bound for the pulse width required for obvious state switching. Considering this, the chosen 1 ms pulse exceeds the minimum device response duration, ensuring reliable and stable operation.\nGiven these constraints, the maximum theoretical processing frequency of our system can be inferred. Since the synaptic array operates in series with the front-end imaging array, the maximum processing frequency is primarily limited by the response speed of synaptic transistors. According to the minimum response time (~100 μs) as shown in Fig. 2, the synaptic array could theoretically support a maximum processing frequency of up to approximately 10 kHz (1/100 μs). In this setting, the synaptic array states are just fully updated before subsequent frames are captured by the imaging array. However, in practical considerations, factors such as data transfer latency and additional computational overhead necessitate a safety margin of approximately one orders of magnitude. Thus, conservatively estimating, the maximum processing frequency up to approximately 1 kHz. When further improving the response speed of the synapse device, the maximum processing frequency can be increased.\nComment #4\nIn this method, when modulating the neuromorphic device for subsequent optical flow calculations and task execution, device differences may introduce errors. How does neuromorphic optical flow mitigate the impact of device-to-device differences while maintaining or even improving processing accuracy?\nResponse:\nThank you for highlighting the potential error introduced by device variations, e.g., array-level non-uniformity. We have incorporated a detailed explanation into the revised manuscript, as follows:\n(SNote 8) On Page 15:\nFor the residual array-level non-uniformity, such variations can be considered as inherent ‘noise’, adding to the ideal uniform conductance map15–18. To mitigate this non-uniformity, we employ two strategies in series: (i) a reconfigurable thresholding scheme during binarization to maintain sensitivity despite variability, and (ii) a spatial-continuity merge that leverages the connectivity of object motion to produce robust ROIs.\n(i) Reconfigurable thresholding scheme during binarization: The binarization threshold s used to translate the analog state information s into temporal cues s’ (see Fig. 4c and Eq. th R7) can be adjusted to accommodate device variations. In practice, lowering s increases th sensitivity so that states with added variation still map to s’ = 255 and are treated as potential motion. Through this method, when using Farneback algorithm for motion prediction in grasping tasks, despite artificially introduced device variations of up to 5%, equivalent performance can be achieved by setting an appropriate binarization threshold (e.g., s = 250; th Fig. R32). In our experiments, thresholds shown in STable 5 are set based on measured device variations, ensuring reliable extraction of dynamic motion regions for subsequent velocity inference and downstream tasks.\n255,s≥s\ns'=  th (7)\n0,otherwise\n(ii) Spatial continuity inherent to object motion. The inherent spatial continuity of moving objects implies that the physical extent of a single object is connected and coherently arranged in space. Therefore, even if the synaptic array response exhibits non-uniformities, merging these responses to create a ROI region ensures that the resulting mask effectively captures the entire moving object, as shown in Fig. R33. By integrating reconfigurable thresholding and spatial continuity considerations, our approach achieves robust visual processing performance despite device non-idealities.\nThrough these two strategies, the region of moving objects can be accurately extracted and covered, maintaining or even improving processing accuracy through adding temporal prior knowledge.\n(Fig. S24) Fig. R32. Performance evaluation under simulated array variability. In simulation experiments, the ideal synaptic transistor update at cell (i,j) is denoted as Δw ideal(t+Δt). Device variability is introduced by adding a Gaussian noise term proportional to ij the ideal update value:\nΔw var(t+Δt) = Δw ideal(t+Δt)+ σΔw (t)ξ ,\nij ij ij ij\nwhere ξ is standard normal random variable, σ is the variation factor, Δw ideal(t+Δt) is the ideal ij ij\nupdate value without variability, and Δw var(t+Δt) is the updated value accounting for variability.\nij\na, Velocity inference and task execution results at a 1% variation factor. b, Results at a 3% variation factor. c, Results at a 5% variation factor. d, Prediction accuracy comparison using SSIM across the above variation settings, demonstrating consistent performance despite increased variability.\n(Fig. S25) Fig. R33. Enhancing system robustness via spatial continuity merging. Using a grasping scenario as an example, spatial continuity effectively merges the detection regions, enabling easy capture of the entire moving object. Therefore, merging based on spatial continuity significantly enhances robustness against device variations or response inconsistencies.\nComment #5\n-In the discussion, the authors claim that the encoding method can be integrated with object detection networks such as YOLO. To demonstrate its versatility, please provide experimental results to demonstrate such integration. What specific requirements does the encoding method impose on downstream algorithms (scope of application)?\nResponse:\nWe sincerely appreciate the opportunity to clarify the integration capability of our encoding approach with downstream algorithms, such as YOLO. The following discussion and supporting experimental results have been incorporated into Supplementary Information:\n(SI):\n(Fig. S53) Fig. R34. Integration with YOLO neural networks. a, Pipeline about utilizing temporal cues to accelerate downstream high-level processing tasks. In this pipeline, temporal cues generated by our method first highlight regions containing potential motion. Subsequently, only these dynamically filtered regions are passed to high-level processing functions, such as YOLO-based object detection. By selectively focusing computational resources on regions of interest (ROIs) rather than processing entire frames, significant computational savings and improved efficiency can be achieved. b, Integration with YOLO neural networks, verifying effective object detection within identified ROIs. Furthermore, our encoding method can be extended to a broader range of downstream algorithms beyond YOLO. The requirement for seamless integration is the flexibility of downstream algorithms to handle inputs of varying dimensions corresponding to different ROI sizes. If an algorithm strictly enforces fixed input dimensions, resizing the ROI can mitigate compatibility issues to some extent. However, this resizing may also affect detection accuracy, which should be carefully evaluated.\nComment #6\n-Add reference numbers for other papers shown in Figure 2i\n-The core code flow has been demonstrated in the supplementary material. Please upload the code to the GitHub repository to reproduce the results.\nResponse:\nWe thank the reviewer for this suggestion. References for the comparative studies presented in Figure 2i have now been added and clearly indicated in the revised manuscript. Additionally, the core implementation code, previously provided in the Supplementary Information, has now been uploaded to a publicly accessible GitHub repository https://github.com/RTCartist/Neuromorphic-Spatiotemporal-Optical-Flow.\nRevised Fig. 2i\nReviewer #4\nThe authors propose a neuromorphic optical flow approach that encodes temporal information in a synaptic transistor array. The so-obtained temporal information is subsequently used for downstream tasks, to e.g., segment moving objects. Down-selecting the region of interest in this manner leads to substantial speed-ups in processing.\nResponse:\nThank you for recognition of our work. Our detailed responses to your valuable comments are provided below.\nComment #1\nThe authors focus on “neuromorphic optical flow”. However, if I understand correctly, the authors actually do not determine optical flow with the synaptic array, but values encoding the change in brightness over time (Equations 1-3). This is quite confusing. Optical flow is the apparent motion of objects in the scene due to relative motion between the objects and the observer. This apparent motion, captured by a 2D image sensor, has two dimensions (in the x- and y-direction). This is not the output of the synaptic transistor array, though. Indeed, later optical flow methods are applied to the regions of interest selected with the transistor array.\nIn the one hand, this should be made clear much earlier in the article. On the other hand, this is a bit of an anti-climax, since brightness changes can also be determined really efficiently in software. This is then limited by the camera’s frame rate, but it would form a good comparison for the authors’ study to show the benefits of the custom transistor array. Hence, the authors should in my eyes compare their method with a software-based one that determines the value of Eq 3 in software on images and performs the same down-stream tasks.\nResponse:\nWe appreciate the reviewer’s careful reading and the opportunity to clarify the role of synaptic transistor array within our neuromorphic optical flow pipeline. This feedback has strengthened our work. Below we address each concern and describe the clarifications and new experiments that have been incorporated into the revised manuscript and Supplementary Information.\n1. Why we use the term “neuromorphic optical flow”\nReviewer’s concern\nThe synaptic array does not directly output the optical flow vector field, but rather a map of brightness change (Eqs. 1–3), so the label “optical flow” is confusing.\nResponse\nThe reviewer correctly notes that the synaptic transistor array itself outputs temporal motion cues derived from brightness changes, rather than optical flow vectors directly. Subsequently, these hardware-derived temporal cues guide downstream algorithms (such as Farneback, RAFT, GMFlow, or FlowFormer) to compute optical flow fields. We use the term “neuromorphic optical flow” to emphasize three core points:\n1. Biological analogy: Our approach closely mirrors the biological retina-LGN interaction in visual processing, where spatiotemporal integration begins at the hardware-level front-end (conventional imaging sensor combined with neuromorphic synaptic array). This aligns with the neuromorphic computing philosophy, which aims to replicate biological visual perception mechanisms for human-like, efficient, and robust visual processing.\n2. Contribution of neuromorphic synapse array: The synaptic transistor array naturally encodes temporal derivatives of brightness intensity due to its intrinsic non-volatile; compared to software methods, . This intrinsic capability of the synapse array enables efficient temporal cue generation directly in hardware, significantly reducing external latency compared to purely software-based approaches.\n3. Difference between our approach and conventional optical flow method: Compared to conventional optical flow methods, our approach involves additional temporal cues generated by neuromorphic hardware. This added temporal information in neuromorphic optical flow can significantly accelerate task-specific computations compared to conventional methods relying solely on spatial data.\nTo explicitly avoid confusion, we have added the following clarification to the manuscript introduction:\n(Article) On Page 4:\nIt should be noted that the synaptic transistor array encodes a derivative of brightness based on the non-volatile characteristics of neuromorphic synaptic transistors, and the optical flow field (map of velocity) is obtained by velocity inference methods applied inside the potential motion regions revealed by those temporal cues.\n2. Maximum processing frequency supported by neuromorphic synapse array\nGiven the fast response characteristics of synaptic transistors, our approach could theoretically support a maximum processing frequency of approximately 1 kHz, offering significantly enhanced temporal resolution and reduced motion blur, which is beneficial for applications such as high-speed robotics and autonomous driving. The following discussion has been added to Supplementary Information.\n(SNote 13) On Page 24:\nSince the synaptic array operates in series with the front-end imaging array, the maximum processing frequency is primarily limited by the response speed of synaptic transistors. According to the minimum response time (~100 μs) as shown in Fig. 2, the synaptic array could theoretically support a maximum processing frequency of up to approximately 10 kHz (1/100 μs). In this setting, the synaptic array states are just fully updated before subsequent frames are captured by the imaging array. However, in practical considerations, factors such as data transfer latency and additional computational overhead necessitate a safety margin of approximately one orders of magnitude. Thus, conservatively estimating, the maximum processing frequency up to approximately 1 kHz. When further improving the response speed of the synapse device, the maximum processing frequency can be increased.\nWe agree that a similar ROI map can be developed using software. The following simulation has been included in the Supplementary Information section:\n(SNote 14) On Page 25:\nTo evaluate the information processing, we simulated the process of translating light intensity changes into modulation voltage pulses, which are then used to update the synaptic transistor states necessary for optical flow calculation and task acceleration, as illustrated in Fig. R35.\nIn the vehicle scenario, software-based simulation achieved an average processing time per frame of 2.98 ms, with negligible impact on downstream task accuracy. Similarly, for the UAV scenario, the average simulation time per frame is 3.71 ms while maintaining the task accuracy. By contrast, the total time for neuromorphic hardware, the re-configurable operational amplifier (settling time ≈46 µs), synaptic transistor programming pulse, and device settling (total ~1 ms), significantly reduces latency relative to the purely software-based approach (33.6% and 27.0% acceleration ratio in the vehicle and UAV scenario, respectively). In addition, the pulse width can be shortened to further reduce the pipeline processing time using neuromorphic hardware, as discussed in the maximum processing frequency part (SNote 13). Therefore, the neuromorphic hardware implementation demonstrates higher efficiency, particularly suited to applications demanding high real-time responsiveness.\n(Fig. S54) Fig. R35. Comparative analysis between simulated software-based synaptic array processing and actual neuromorphic hardware processing. a, Simulation pipeline. b, Time consumption of simulation pipeline per frame for the vehicle scenario when achieving same downstream task accuracy compared to hardware experiments. c, Time consumption of simulation pipeline per frame for the UAV scenario when achieving same downstream task accuracy compared to hardware experiments.\nComment #2\nThe method is applied to different tasks. In a first case, 800 x 800 pixels are downscaled to 4 x4 regions to which the array is applied. Large moving objects can then be detected. It is not really clear how good this method works. A major concern for me is that substantial ego-motion in well-lit, textured scenes, would trigger lots of brightness changes, leading to the array lighting up everywhere. This is very common for ego-motion when it includes rotations. The pedestrian task shows dark images in which the car is driving very slowly. The UAV operation images show a drone against a uniform background (in the video also against the sky over a landscape). Real-world applicability seems far away given these tasks.\nResponse:\nThanks for your valuable comments to demonstrate the applicability of our method. To address this concern, we have compared our method with state-of-the-art visual processing work in the neuromorphic computing field, discussed the effect of resolution, and added the visual processing experiments in dynamic scenarios with self-motion and challenging scenario. The following discussion and experiments have been added to our manuscript and supplementary information.\n(STable 12) On Page 99:\nComparison with other works\nCompared to other state-of-the-art approaches listed in Table R7, our experiments utilize data sources that more closely resemble real-world visual information, covering diverse scenarios including vehicle driving, drone flight, robotic arm operation and table tennis. This enhanced applicability arises from the flexibility of our system architecture, which allows the encoding method of external visual data into the synapse array state information to be adjusted according to varying scene requirements.\n(STable 12) Table R7. Comparison with other works.\nJayachandran Huang et Baek et\nThis work Zhang et al. 38 Zhou et al.43\net al.44 al.45 al.11\nFloating Two- Two-\nMonolayer\nNeuromorphic device synaptic retinomorphic Programmable dimensional Multibranch\ndevice transistor photodiode\nMoS2\noptical dendristor\nphotodetector\narray sensor\nField-\nLaboratory Laboratory Laboratory\nData sources based Simulation Simulation\nbased data based data based data\ndata\nVehicle\ndriving RC car\nMoving\napproaching\npedestrian\nDrone another car\nflight\nButterfly\nMoving light\nScenarios Waving hands motion\nRobotic spots\ndetection\narm\nMoving RC car\noperation\nremote control approaching\n(RC) car wall\nTable\ntennis\nTo further demonstrate the applicability of our method in dynamic real-world scenarios, the following theoretical analysis and experimental evaluation have been added:\n(SNote 10) On Page 18:\nPerformance evaluation under dynamic scenarios\nIn dynamic scenes involving ego-motion or out-of-distribution motion, background regions also exhibit obvious motion. Here, ”background” denotes objects that are stationary or slow-moving in the world (lab) frame. Nevertheless, our method continues to identify regions of interest (ROIs) to achieve acceleration based on the relative velocity contrast between background and moving objects of interest. This follows directly from the moving pattern encoding principle of our method: the modulation of the synapse array depends on the absolute light intensity derivative |∂I/∂t|, which scales with relative moving speed (Fig. S36). Let v denote camera cam ego motion and v object motion; the relative speed v is\nobj rel\nv = v −v (8)\nrel obj cam\nAs shown in Fig. S36, the background satisfies v ≈ 0 and thus v = ||v ||, whereas cross-obj rel cam traffic or opposing motion typically yields v > ||v ||, producing larger |∂I/∂t| compared to rel cam background information. Therefore, setting an appropriate threshold voltage V utilized in Eq. th 3 can omit background by modulating those corresponding synaptic devices into high resistance, while moving objects of interest modulate the corresponding synapse array region toward low resistance and are selected as ROIs.\nCo-moving objects with small v are de-prioritized to preserve throughput, similar to how rel human drivers prioritize crossing pedestrians over similarly moving vehicles. When such targets are required (e.g., lead-vehicle tracking), V can be lowered to recover them.\nth\nTo validate robustness, we conducted two sets of experiments: (i) controlled sequences captured with a hand-held phone while the IMU measured v during which V is linearly cam th scaled with v (ii) real-world recordings from an in-car dashcam. Acceleration and accuracy cam are quantified using Farneback optical flow for velocity inference, which represents the baseline performance of our pipeline (Fig. 6c). In both cases, ROI quality is maintained, and end-to-end speed-ups are preserved.\nPerformance evaluation\nIn the handheld phone capturing experiments, to consider safety, one participant is asked to hold the camera while walking or running at predefined speed levels. A second participant simultaneously ran within the scene. For each camera speed, the trajectories of both the camera and the runner followed three distinct motion patterns (Fig. R37a), covering out‑of‑distribution cases such as abrupt direction changes and approaching or receding motion. During the experiments, the phone’s on‑board IMU is continuously logged; its readings are integrated to obtain velocity, and the peak value is taken as the moving speed of the camera, as shown in Fig. R37b. Four levels are tested: baseline walking (~0.5 m/s) and three running speeds—low (~1.5 m/s), medium (~2.5 m/s), and high (~3.5 m/s). These visual data collected in the above speed levels with three moving patterns are simulated by a 20 × 13 synapse array, with V proportionally scaled according to the camera ego motion velocity v th cam at different running speeds:\nV =αv +β (9)\nth cam\nwith α and β tuned to 2 and -2 and held fixed across all runs.\nDetailed results can be found in Figs. R38–R49 and Tabel R8, demonstrating consistent improvements in processing efficiency and accuracy over conventional optical flow methods.\nSpecifically, acceleration ratios and accuracy improvements were as follows. As shown in Fig. R37c, in baseline speed, the average acceleration ratio is 55.9%, and the accuracy improvement ratio is 116.1%; in low-speed motion, the average acceleration ratio is 57.1%, and the accuracy improvement ratio is 113.2%; in medium-speed motion, the average acceleration ratio is 62.6%, and the accuracy improvement ratio is 109.4%; in high-speed motion, the average acceleration ratio is 62.1%, and the accuracy improvement ratio is 111.3%. Note that camera ego-motion introduces significant variations in background appearance. As neuromorphic optical flow selectively extracts critical motion features while filtering out irrelevant background fluctuations, a slight decrease in performance metrics is expected for the prediction task that requires comprehensive global image analysis. However, for object segmentation and tracking tasks, the selective extraction of key motion information substantially enhances overall performance. The marked improvement in processing efficiency and accuracy over conventional methods demonstrates that neuromorphic optical flow provides robust processing capabilities even under ego-motion conditions by adaptively tuning the threshold voltage V .\nth\nMoreover, in the multiple object detection scenario depicted in Fig. R50, the average acceleration ratio achieved is 29.1%, accompanied by an accuracy improvement ratio of 102.2%. Furthermore, as observed in representative key frames, our method accurately extracts velocity information for multiple objects simultaneously.\nIn challenging real-world scenarios captured by vehicle-mounted cameras, visual data from three distinct scenarios are simulated using a 20 × 13 synapse array with finely tuned threshold voltages. Similar results are observed in Fig. R51 to Fig. R53 and Table R9, demonstrating that the velocity information of key moving objects within these scenes can be accurately extracted, significantly accelerating processing while maintaining accuracy. Notably, the IoU metric in the object tracking task exhibited comparatively lower performance relative to other tasks. This limitation primarily stems from the downstream tracking algorithm, which, despite accurately extracting velocity information, occasionally identifies multiple objects that are not necessarily the primary objects of interest, thus decreasing overall tracking accuracy. On average, an acceleration ratio of 74.8% is achieved, alongside an accuracy improvement ratio of 101.7% (Fig. R54). These results collectively underscore the robustness and efficacy of neuromorphic optical flow, demonstrating performance improvements in processing speed without affecting accuracy even under more demanding operational conditions.\nPotential motion compensation and fallback strategies\nAs shown in the hand-held smartphone experiments, adapting the threshold voltage V based th on camera ego motion velocity ensures that key temporal moving patterns are mapped into the synaptic array state, thereby facilitating downstream inference. Specifically, V is linearly th scaled with camera ego motion. In subsequent deployments, ego-motion from an IMU (or an equivalent sensor) can be used to dynamically compensate global motion and maintain robust ROI selection. Furthermore, fallback strategies can also be implemented to improve system robustness. For instance, upon detecting rapid, scene-wide intensity fluctuations, the system could temporarily revert to computing optical flow for the entire frame until signal stability is restored. As demonstrated in Fig. 6 and Fig. S15, this fallback adds only ~1–2 ms for ROI identification—negligible relative to the end-to-end latency—while preserving accuracy across diverse, challenging scenarios.\n(STable 7) Table R8. Detailed performance metrics in the handheld phone capturing experiments.\nScenario tpreo tpren tsego tsegn tobo tobn SSIMo SSIMn PAo PAn IoUo IoUn\nPattern 1 0.11 0.06 0.13 0.07 0.11 0.06 0.93 0.88 0.92 0.99 0.18 0.22\nBaseline\nPattern 2 0.13 0.05 0.14 0.05 0.12 0.04 0.93 0.87 0.89 0.98 0.12 0.20\nspeed\nPattern 3 0.12 0.09 0.13 0.10 0.11 0.09 0.90 0.87 0.61 0.64 0.08 0.13\nPattern 1 0.09 0.05 0.11 0.06 0.08 0.05 0.91 0.84 0.79 0.90 0.19 0.34\nLow-\nspeed Pattern 2 0.09 0.05 0.10 0.05 0.08 0.04 0.93 0.90 0.88 0.98 0.24 0.31\nmotion\nPattern 3 0.09 0.05 0.10 0.06 0.08 0.05 0.87 0.81 0.80 0.88 0.20 0.20\nPattern 1 0.09 0.06 0.10 0.07 0.08 0.06 0.83 0.79 0.81 0.91 0.27 0.37\nMedium-\nspeed Pattern 2 0.09 0.04 0.10 0.04 0.08 0.04 0.84 0.77 0.85 0.95 0.24 0.29\nmotion\nPattern 3 0.09 0.06 0.11 0.07 0.09 0.07 0.77 0.70 0.62 0.68 0.12 0.13\nPattern 1 0.09 0.06 0.10 0.07 0.09 0.06 0.79 0.68 0.69 0.78 0.13 0.17\nHigh-\nspeed Pattern 2 0.09 0.05 0.11 0.06 0.09 0.05 0.83 0.73 0.74 0.82 0.09 0.14\nmotion\nPattern 3 0.10 0.07 0.11 0.07 0.09 0.06 0.73 0.61 0.56 0.67 0.06 0.07\nMultiple objects 0.09 0.03 0.10 0.03 0.09 0.03 0.96 0.96 0.98 0.99 0.39 0.41\nIn Table R8, the subscript o denotes the conventional pipeline, and n denotes the neuromorphic (ours) pipeline. Accordingly, t /t are the per-frame times for the motion preo pren prediction task, t /t are the per-frame times for the object segmentation task, and t /t sego segn obo obn are the per-frame times for the object tracking task. SSIM /SSIM are the Structural Similarity o n Index Measure about prediction accuracy, PA /PA are pixel accuracy, and IoU /IoU are o n o n intersection-over-union. For all time metrics, lower values are better; for SSIM, PA, and IoU, higher values are better. All times are reported in seconds per frame, and the accuracy metrics are unitless in [0, 1].\n(STable 8) Table R9. Comprehensive performance metrics from the in-car dashcam experiments.\nScenario tpreo tpren tsego tsegn tobo tobn SSIMo SSIMn PAo PAn IoUo IoUn\nScenario 1 0.32 0.25 0.41 0.29 0.30 0.25 0.86 0.83 0.78 0.79 0.16 0.16\nScenario 2 0.35 0.25 0.42 0.29 0.34 0.27 0.88 0.85 0.80 0.81 0.16 0.17\nScenario 3 0.29 0.22 0.40 0.27 0.28 0.22 0.87 0.87 0.87 0.90 0.09 0.10\nTable R9 reports the same metrics as Table R2: subscripts o and n denote the conventional and neuromorphic pipelines, respectively. Lower is better for t , t , and t ; higher is better pre seg ob for SSIM, PA, and IoU. Times are in seconds per frame; accuracy metrics are unitless in [0, 1].\n(Fig. S26) Fig. R36. Model relating synapse array modulation to scene-relative speed. a, Basic setting. b, Relative speed determines the magnitude of dI/dt that drives synapse array modulation. Comparing this signal to the hardware threshold voltage V separates th background from moving objects of interest.\n(Fig. S27) Fig. R37. Setting and results evaluation in the handheld phone capturing experiments. a, Experimental setup with three motion patterns. b, Camera ego-motion speed estimated from the on-board IMU. c, Processing time and task performance across speed levels.\n(Fig. S28) Fig. R38. Performance analysis of motion pattern 1 under baseline velocity conditions. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S29) Fig. R39. Performance analysis of motion pattern 2 under baseline velocity conditions. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S30) Fig. R40. Performance analysis of motion pattern 3 under baseline velocity conditions. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S31) Fig. R41. Performance analysis of motion pattern 1 under low-speed conditions. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S32) Fig. R42. Performance analysis of motion pattern 2 under low-speed conditions. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S33) Fig. R43. Performance analysis of motion pattern 3 under low-speed conditions. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S34) Fig. R44. Performance analysis of motion pattern 1 under medium-speed conditions. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S35) Fig. R45. Performance analysis of motion pattern 2 under medium-speed conditions. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S36) Fig. R46. Performance analysis of motion pattern 3 under medium-speed conditions. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S37) Fig. R47. Performance analysis of motion pattern 1 under high-speed conditions. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S38) Fig. R48. Performance analysis of motion pattern 2 under high-speed conditions. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S39) Fig. R49. Performance analysis of motion pattern 3 under high-speed conditions. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S40) Fig. R50. Performance analysis when detecting multiple objects. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S41) Fig. R51. Performance analysis of challenging scenario 1. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S42) Fig. R52. Performance analysis of challenging scenario 2. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S43) Fig. R53. Performance analysis of challenging scenario 3. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S44) Fig. R54. Average performance analysis of challenging scenarios.\nFor the concern about detecting small objects, the mapping process from sensor and synapse array is analyzed, and quantitative evaluation are performed in different synapse array size as follows:\n(SNote 11) On Page 21:\nEffects of sensor-to-synapse pooling size on neuromorphic optical flow performance\nWhen mapping scene visual information of resolution (m,n) to the synapse array of size M×N, each synaptic device receives the spatial average of a local group of neighboring pixels. This group is termed the sensor-to-synapse pooling block of size p×q pixels (Fig. R55). Under the uniform tiling, it can be calculated that p=m/M and q=n/N. Let the raw light intensity be I(u,v,t)∈Rm×n, and u and v represent location. Thus, the temporal derivative of visual voltage r I driving synapse (i,j) is:\ni,j\ndI 1 (i+1)p−1(j+1)q−1∂I (u,v,t)\ni,j =   r (10)\ndt pq ∂t\nu=ip v=jq\nThis spatial averaging suppresses the high-frequency noise of raw visual data. When p×q is small, motion detail is preserved, but noise increases. With appropriate adjustment of V , th background can still be separated from moving objects of interest. Conversely, if p×q becomes too large relative to the object scale, motions are averaged into the background, reducing ROI separability even after retuning V .\nth\nQuantitative evaluation\nTwo representative scenarios, vehicle and grasp, are simulated using the synapse array of different pooling sizes. In the following analysis, we set p = q and report results using a single pooling parameter p.\nIn vehicle operation, with input resolution held fixed, increasing the synapse array to reduce the pooling size from 200 to 100 and 50 pixels, the acceleration ratio and task accuracy remain essentially unchanged. This stability arises because of retuning the threshold voltage V . In th contrast, enlarging the pooling size to 400 (corresponding to a 2×2 synapse array), both acceleration and accuracy significantly degrade (Fig. R56). This performance reduction primarily arises from the increased spatial smoothing inherent at larger pooling block sizes, causing critical motion information to blend into background signals and reducing the accuracy of region-of-interest (ROI) identification. Note that the ROI accuracy can be reflected in the Intersection-over-Union (IoU) metric in object tracking tasks. The reduction in ROI accuracy can lead to either excessive background inclusion or incomplete object coverage; thus, both task accuracy and processing acceleration will be degraded. Regarding miss rate, a noticeable increase is observed at an input resolution of 400, characterized by two consecutive frame omissions during motion (miss rate 0.0392), while performance at higher resolutions remained robust. Detailed results can be found in Table R10.\nSimilar results can be obtained for the grasping operation (Fig. R57). Here, performance metrics remained largely consistent when moderately increasing the pooling block size from the original 80 to 160 with input resolution fixed. However, when further increasing the pooling block size to 320, a substantial decline in these metrics is observed. It indicates that our neuromorphic optical flow approach exhibits considerable tolerance to pooling block size adjustments within a certain range. Nevertheless, when the pooling block size significantly exceeds the size of target objects, performance metrics sharply decrease. Notably, in the grasping scenario, the miss rate consistently remained zero, as the key object was continuously detected during velocity inference. However, incomplete object representation occasionally occurred at excessively low resolutions, as shown in Fig. R57a, impacting overall task performance metrics.\nIndicated by these results, the neuromorphic optical flow method demonstrates a reasonable tolerance to small pooling block size, but its performance could be degraded when the pooling block size is far larger than the moving details. In our experiments, robust performance is achieved by the selected resolution, which aligns closely with the size scale of key visual features. With advancements in synapse-array integration, higher-density arrays are expected to further enhance the capacity of neuromorphic optical flow methods, ensuring that the pooling size remains sufficiently small.\n(STable 9) Table R10. Quantitative evaluation of processing performance for varying sensor-to-synapse pooling block sizes (p×q).\nScenario Block size Gpre Gseg Gob Spre Sseg Sob Miss rate\n50 0.9980 1.0039 1.8972 0.0647 0.0625 0.0656 0\n100 0.9987 1.0051 1.9848 0.0787 0.0750 0.0792 0\nVehicle\n200 0.9992 1.0050 1.7963 0.0822 0.0789 0.0830 0\n400 1.0029 1.0053 0.6143 0.2779 0.2703 0.2910 0.0392\n40 0.9998 1.0033 28.7344 0.0624 0.0528 0.0568 0\n80 0.9998 1.0031 20.6503 0.1346 0.1252 0.1346 0\nGrasping\n160 0.9993 1.0034 25.5042 0.0932 0.0796 0.0899 0\n320 0.9987 1.0022 4.8949 0.2001 0.1820 0.2022 0\nIn Table R10, G denotes accuracy gain, defined as the ratio of the neuromorphic metric to the conventional baseline, where values > 1 indicate improved accuracy; S denotes the acceleration ratio, defined as the neuromorphic processing time relative to the conventional baseline, where values < 1 indicate the neuromorphic pipeline is faster. The subscripts pre, seg, and obj refer to the prediction, segmentation, and object tracking tasks, respectively. The miss rate is computed as the number of consecutively missed frames divided by the total number of processed frames.\n(Fig. S45) Fig. R55. Sensor-to-synapse pooling. An input frame of resolution (m,n) is mapped to a synapse array of size M×N by spatially averaging local p×q pixel neighborhoods per synapse.\n(Fig. S46) Fig. R56. Performance analysis when changing the input resolution of vehicle scenarios.\n(Fig. S47) Fig. R57. Performance analysis when changing the input resolution of grasp scenarios.\nComment #3\nDuring the reading of the article, I have been continuously wondering when event-based cameras (neuromorphic cameras) would be mentioned. These are similar in spirit, and many algorithms have been devised (both neural and non-neural) to extract optical flow from the events that come from such cameras. These cameras offer the opportunity to have low-latency, low-power optical flow determination.\nThe authors only mention event-based cameras in the “Future directions” part, mentioning that: “By incorporating event-based cameras as the front-end sensor, the synapse array in the neuromorphic optical flow can be modulated by event streams, potentially delivering similar acceleration benefits for event-based optical flow methods.” How would this work? Events are ON- or OFF events, and the processing of brightness over time happens in the hardware before. It is non-evident to me how the proposed transistor array would be combined with neuromorphic cameras. Instead, this approach could (or should) be compared with them.\nAlthough I am sympathetic to the idea of down-selecting regions of interest based on motion cues, the concerns above currently do not make me very enthusiastic about the article.\nResponse:\nWe appreciate the reviewer’s careful reading and the opportunity to compare our method with event-based cameras (neuromorphic cameras) and clarify how our synaptic-transistor array can cooperate with event-based cameras.\n(SNote 15) On Page 26:\nComparison between our methods and neuromorphic cameras based optical flow computation\nIn our proposed method, the synaptic array encodes external visual information into its array states, reflecting potential regions of motion. Conceptually, this approach shares similarities with neuromorphic cameras (event-based cameras), particularly in its focus on capturing rapid changes in light intensity. However, there are key differences in the underlying principles and subsequent optical flow computation. For the working principle of event-based cameras, a differential processing is designed to extract changes in light intensity, denoted as ΔL. When ΔL exceeds the predefined threshold, ON- or OFF events are triggered asynchronously, signaling significant visual changes as shown in Eq. R11 and Eq. R12:\nΔLp(x,y,t)= Lp(x,y,t)−Lp(x,y,tp) (11)\nk\n\n\nΔLp(x,y,t)≥θ\n+\n→emit(x,y,t,+1)\n(12)\n\n\nΔLp(x,y,t)≤−θ\n−\n→emit(x,y,t,−1)\nwhere Lp(x,y,t) is the log-intensity measured at pixel (x,y) at time t, tp (x,y) is the time stamp of k the last event of polarity p ∈{+1,−1} (ON=+1, OFF=−1) at that pixel, and θ and θ are + − predefined thresholds determining the event-triggering criterion.\nIn our proposed framework, light intensity changes between frames are translated into voltage pulses flexibly, and array states are modulated by these pulses. This process can be expressed as:\nΔI(x,y,t)=I(x,y,t)−I(x,y,t−Δt) (13)\n\nV (t)=E(ΔI(x,y,t)) (14)\nx,y\n\nM (t+Δt)≈M (t)+U(V (t)) (15)\nx,y x,y x,y\nwhere I(x,y,t) is the light intensity recorded at pixel (x,y) at time t, Δt is the sampling interval, E denotes the voltage-conversion function defined by Eqs. 2 and 3, U denotes the approximate state-update rule of the synaptic device, and M (t) is the state information of synaptic array x,y at time t, which can be expressed in terms of resistance or conductance and experimentally accessed through the drain–source current I . Owing to device non-idealities, the relationship ds is written with “≈” rather than “=”. Compared to event streams, the synapse array state relies not only on modulation voltage induced by changes in light intensity but also on its current state. This means that synapse array states highlight rapid changes while also preserving historical change information, as illustrated in Fig. R58. This characteristic can provide cleaner and more actionable data for further high-level processing beyond mere optical flow computation. For example, the integration with the YOLO algorithm is shown in Fig. R59.\nThe above working principle also affects the needed operations for subsequent optical flow computation. During velocity inference, a large fraction of event-based optical flow methods first accumulate events in short windows into tensors, e.g., time surfaces (as in EV-FlowNet) or voxel grids with temporal bins (as in E-RAFT), before feature extraction and correlation46,47. In our case, each synaptic device performs this temporal accumulation in hardware, producing a non-volatile analogue time surface that updates via modulation pulses. Therefore, this hardware-level accumulation cuts both energy and processing time. Regarding region-of-interest (ROI) selection, we acknowledge the similarity: event streams already imply ROIs. Nevertheless, an explicit ROI mask derived from synaptic states can be applied within event pipelines, e.g., to crop or mask spatially dense convolutions/correlation to reduce computation. Thus, the synapse array is not a replacement for event sensors but a temporal front-end that accumulates temporal information in analogue and generates ROIs early in the pipeline that can be flexibly combined with downstream computation.\n(Fig. S55) Figure R58. Comparison of frame-based imaging, event-based processing, and synapse array state. a, Frame-based camera image. b, Processing pipeline in an event-based camera. In event-based cameras, the extracted light intensity change is visualized as D. c, Our proposed system architecture encodes external visual information into the modulation voltage applied to the synapse array. The resulting synapse array state (resistance map) integrates temporal history, providing richer dynamics than a voltage-only heat map V.\n(Fig. S53) Fig. R59. Integration with YOLO neural networks. a, Pipeline about utilizing temporal cues to accelerate downstream high-level processing tasks. In this pipeline, temporal cues generated by our method first highlight regions containing potential motion. Subsequently, only these dynamically filtered regions are passed to high-level processing functions, such as YOLO-based object detection. By selectively focusing computational resources on regions of interest (ROIs) rather than processing entire frames, significant computational savings and improved efficiency can be achieved. b, Integration with YOLO neural networks, verifying effective object detection within identified ROIs. Furthermore, our encoding method can be extended to a broader range of downstream algorithms beyond YOLO. The requirement for seamless integration is the flexibility of downstream algorithms to handle inputs of varying dimensions corresponding to different ROI sizes. If an algorithm strictly enforces fixed input dimensions, resizing the ROI can mitigate compatibility issues to some extent. However, this resizing may also affect detection accuracy, which should be carefully evaluated.\n(SNote 16) On Page 28:\nCombination with event-based cameras\nAfter replacing the frame imager with a standard Dynamic vision sensor(DVS)-style front end, our method can perform the mapping of temporal moving information directly on hardware. Starting from the hardware perspective of an event-based camera, which includes photoreceptors, differentiators, and ON/OFF comparators, there are two potential operating schemes that transform the event stream into programming voltages for the synaptic array.\n1. Clocked boxcar integration\nEvents within a short and programmable window T are integrated per pixel address, and a w corresponding ‘active’ or ‘silent’ voltage will be applied to the synapse array at the same location, as the following Eq. R16:\nC [k]=\n(k+1)T\nw e (t) dt (16)\nxy xy\nkT\nw\nwhere C [k] denotes the event count at pixel (x,y) during window k, and e (t) represents the xy xy events (ON corresponds to +1 while OFF corresponds to −1). A comparator realizes the pulse generation:\nV [k]=\n\n\nV\nact\n,C\nxy\n[k]≥θ\n(17)\nxy  V sil ,C xy [k]<θ\nwhere V [k] is a single pulse of width τ at the time boundary, θ is the threshold deciding which xy w kind of voltage pulse is applied. Here, V is the amplitude for negative ‘active’ V pulses that act gs reduce the resistance of our fabricated synaptic device, whereas V is the amplitude for sil positive ‘silent’ V pulses that push the synaptic device into a high-resistance state. Typical gs T values are sub-millisecond to a few milliseconds, selected to match the expected motion w temporal scale and allowable latency. In hardware implementation, the pixel front-end could be unmodified compared to conventional DVS, only periphery circuit is added: a minimal analog accumulator based on capacitor and a comparator can implement the boxcar (e.g., integrating events and comparing the count with a threshold); a small timing generator (clock or asynchronous tick) defines T ; a pulse generator deliver the chosen V or V to the w act sil address synaptic device location.\n2. DC bias with event overlays\nIn this scheme, the array is biased at a low DC level, and each incoming event adds a brief overlay pulse at its address. Compared to the previous scheme, this realizes synchronous, sub-window latency updates that are natural for event streams. With a rectangular template to constrain the pulse width and a per-pixel refractory gate to avoid write storms, the programming V voltage at each addressed synapse (x,y) is modelled as:\ngs\nV (t)=V +  Am (t )s(t−t ) (18)\nxy DC xy n n\nn:x =x,y =y\nn n\n1,t −tlast ≥τ\nm (t )=  n xy ref (19)\nxy n\n0,otherwise\n1,0≤t<τ\ns(t)=  p (20)\n0,otherwise\nwhere V is a low bias applied continuously to the synaptic device array, A is the overlay DC amplitude added to the bias for a short duration τ when an event occurs at (x,y), m (t ) is the p xy n per-pixel refractory gate that controls the programming pulses from overwriting at the time duration τ after the last event t last, and s(t) indicates the pulse width of the overlay pulse is ref xy τ . In this scheme, events with both polarities contribute identically to the synaptic array. The p refractory gate enforces a minimum inter-overlay interval τ to prevent excessive writes under ref bursty input. In hardware implementations, digital circuits or analog capacitors and comparators can be utilized to implement this refractory gate, while a pulse generator superposes an amplitude-limited overlap A on the DC bias for τ . Both these two schemes p only consider the absolute intensity change of temporal information and omit the light intensity polarity; if needed, polarity can be handled by (i) two synaptic arrays/banks per pixel (ON writes one bank, OFF the other).\nWe implemented both schemes by applying a simulated synapse array to real event streams recorded with a Prophesee event-based camera. In each scheme, the synapse array maps motion patterns into non-volatile device states, creating similar patterns as frame-based image sensors for subsequent optical flow calculations, as illustrated in Fig. S60. Compared to raw event streams, the synapse array not only emphasizes change information but also retains historical data. Moreover, it can be read once per window to create an explicit ROI mask for downstream processing. The time-surface-like characteristics and the involved ROI indicate the potential to connect the architecture of our design with event-based cameras, thereby enhancing their performance. The simulation code is available in the accompanying GitHub repository https://github.com/RTCartist/Neuromorphic-Spatiotemporal-Optical-Flow.\n(Fig. S56) Fig. R60. a, Raw event streams from Prophesee event-based vision systems. b, Synapse array state when processing event streams using the clocked boxcar integration scheme. c, Synapse array state when processing event streams using the DC bias with event overlays scheme.\nReferences:\n1. Wang, S. et al. Memristor-based adaptive neuromorphic perception in unstructured environments. Nat Commun 15, 4671 (2024).\n2. Yoon, J. H. et al. An artificial nociceptor based on a diffusive memristor. Nat Commun 9, 417 (2018).\n3. Hong, S. J. et al. Bio-inspired artificial mechanoreceptors with built-in synaptic functions for intelligent tactile skin. Nat. Mater. 1–9 (2025) doi:10.1038/s41563-025-02204-y.\n4. Donati, E. & Valle, G. Neuromorphic hardware for somatosensory neuroprostheses. Nat Commun 15, 556 (2024).\n5. Truong, S. N., Ham, S.-J. & Min, K.-S. Neuromorphic crossbar circuit with nanoscale filamentary-switching binary memristors for speech recognition. Nanoscale Res Lett 9, 629 (2014).\n6. Seo, S. et al. Artificial van der Waals hybrid synapse and its application to acoustic pattern recognition. Nat Commun 11, 3936 (2020).\n7. Gao, S. et al. Programmable Linear RAM: A New Flash Memory-based Memristor for Artificial Synapses and Its Application to Speech Recognition System. in 2019 IEEE International Electron Devices Meeting (IEDM) 14.1.1-14.1.4 (2019). doi:10.1109/IEDM19573.2019.8993598.\n8. Lee, J. et al. Light-enhanced molecular polarity enabling multispectral color-cognitive memristor for neuromorphic visual system. Nat Commun 14, 5775 (2023).\n9. Choi, C. et al. Curved neuromorphic image sensor array using a MoS2-organic heterostructure inspired by the human visual recognition system. Nat Commun 11, 5934 (2020).\n10. Huang, H. et al. Fully integrated multi-mode optoelectronic memristor array for diversified in-sensor computing. Nat. Nanotechnol. 20, 93–103 (2025).\n11. Baek, E. et al. Neuromorphic dendritic network computation with silent synapses for visual motion perception. Nat Electron 7, 454–465 (2024).\n12. Chen, W. et al. Essential Characteristics of Memristors for Neuromorphic Computing. Advanced Electronic Materials 9, 2200833 (2023).\n13. Wang, S. et al. Memristor-Based Intelligent Human-Like Neural Computing. Advanced Electronic Materials 9, 2200877 (2023).\n14. Liu, F. et al. Printed synaptic transistor–based electronic skin for robots to feel and learn. Science Robotics 7, eabl7286 (2022).\n15. Fu, J., Liao, Z., Liu, J., Smith, S. C. & Wang, J. Memristor-Based Variation-Enabled Differentially Private Learning Systems for Edge Computing in IoT. IEEE Internet of Things Journal 8, 9672–9682 (2021).\n16. Fu, J., Liao, Z. & Wang, J. Level Scaling and Pulse Regulating to Mitigate the Impact of the Cycle-to-Cycle Variation in Memristor-Based Edge AI System. IEEE Transactions on Electron Devices 69, 1752–1762 (2022).\n17. Jiang, H. et al. A novel true random number generator based on a stochastic diffusive memristor. Nat Commun 8, 882 (2017).\n18. Tuma, T., Pantazi, A., Le Gallo, M., Sebastian, A. & Eleftheriou, E. Stochastic phase-change neurons. Nature Nanotech 11, 693–699 (2016).\n19. Li, X. et al. Multi-terminal ionic-gated low-power silicon nanowire synaptic transistors with dendritic functions for neuromorphic systems. Nanoscale 12, 16348–16358 (2020).\n20. Zhu, J. et al. Ion Gated Synaptic Transistors Based on 2D van der Waals Crystals with Tunable Diffusive Dynamics. Advanced Materials 30, 1800195 (2018).\n21. Go, J. et al. W/WO3−x based three-terminal synapse device with linear conductance change and high on/off ratio for neuromorphic application. Appl. Phys. Express 12, 026503 (2019).\n22. Yang, C.-S. et al. All‑Solid‑State Synaptic Transistor with Ultralow Conductance for Neuromorphic Computing. doi:10.1002/adfm.201804170.\n23. Nikam, R. D., Kwak, M., Lee, J., Rajput, K. G. & Hwang, H. Controlled Ionic Tunneling in Lithium Nanoionic Synaptic Transistor through Atomically Thin Graphene Layer for Neuromorphic Computing. doi:10.1002/aelm.201901100.\n24. Seo, S. et al. The gate injection-based field-effect synapse transistor with linear conductance update for online training. Nat Commun 13, 6431 (2022).\n25. Jiang, Y. et al. A scalable integration process for ultrafast two-dimensional flash memory. Nat Electron 7, 868–875 (2024).\n26. Kang, J.-H. et al. Monolithic 3D integration of 2D materials-based electronics towards ultimate edge computing solutions. Nat. Mater. 22, 1470–1477 (2023).\n27. Lai, H. et al. Photoinduced Multi‑Bit Nonvolatile Memory Based on a van der Waals Heterostructure with a 2D‑Perovskite Floating Gate. Advanced Materials 34, 2110278 (2022).\n28. Li, G. et al. Photo-induced non-volatile VO2 phase transition for neuromorphic ultraviolet sensors. Nat Commun 13, 1729 (2022).\n29. Liu, L. et al. Ultrafast non-volatile flash memory based on van der Waals heterostructures. Nat. Nanotechnol. 16, 874–881 (2021).\n30. Lu, H., Wang, Y., Han, X. & Liu, J. An Ultrafast Multibit Memory Based on the ReS /h- 2 BN/Graphene Heterostructure. ACS Nano 18, 23403–23411 (2024).\n31. Migliato Marega, G. et al. A large-scale integrated vector–matrix multiplication processor based on monolayer molybdenum disulfide memories. Nat Electron 6, 991–998 (2023).\n32. Yang, Q. et al. Controlled Optoelectronic Response in van der Waals Heterostructures for In‑Sensor Computing. Adv Funct Materials 32, 202207290 (2022).\n33. Zha, J. et al. A 2D Heterostructure‑Based Multifunctional Floating Gate Memory Device for Multimodal Reservoir Computing. Advanced Materials 36, 2308502 (2024).\n34. Zhu, X., Li, D., Liang, X. & Lu, W. D. Ionic modulation and ionic coupling effects in MoS2 devices for neuromorphic computing. Nature Mater 18, 141–148 (2019).\n35. Tang, J. et al. A Reliable All-2D Materials Artificial Synapse for High Energy-Efficient Neuromorphic Computing. Advanced Functional Materials 31, 2011083 (2021).\n36. Yi, S.-G. et al. Artificial Synaptic Emulators Based on MoS2 Flash Memory Devices with Double Floating Gates. ACS Applied Materials & Interfaces (2018) doi:10.1021/acsami.8b10203.\n37. Du, J. et al. A robust neuromorphic vision sensor with optical control of ferroelectric switching. Nano Energy 89, 106439 (2021).\n38. Zhang, Z. et al. All-in-one two-dimensional retinomorphic hardware device for motion detection and recognition. Nat. Nanotechnol. 17, 27–32 (2022).\n39. Wu, L. et al. Atomically sharp interface enabled ultrahigh-speed non-volatile memory devices. Nat. Nanotechnol. 16, 882–887 (2021).\n40. Huang, X. et al. An ultrafast bipolar flash memory for self-activated in-memory computing. Nat. Nanotechnol. 18, 486–492 (2023).\n41. Wang, H. et al. Ultrafast Non‑Volatile Floating‑Gate Memory Based on All‑2D Materials. Advanced Materials 36, 2311652 (2024).\n42. Yu, J. et al. Simultaneously ultrafast and robust two-dimensional flash memory devices based on phase-engineered edge contacts. Nat Commun 14, 5662 (2023).\n43. Zhou, Y. et al. Computational event-driven vision sensors for in-sensor spiking neural networks. Nat Electron 6, 870–878 (2023).\n44. Jayachandran, D. et al. A low-power biomimetic collision detector based on an in-memory molybdenum disulfide photodetector. Nat Electron 3, 646–655 (2020).\n45. Huang, P.-Y. et al. Neuro-inspired optical sensor array for high-accuracy static image recognition and dynamic trace extraction. Nat Commun 14, 6736 (2023).\n46. Zhu, A. Z., Yuan, L., Chaney, K. & Daniilidis, K. EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras. in Robotics: Science and Systems XIV (2018). doi:10.15607/RSS.2018.XIV.062.\n47. Gehrig, M., Millhäusler, M., Gehrig, D. & Scaramuzza, D. E-RAFT: Dense Optical Flow from Event Cameras. in 2021 International Conference on 3D Vision (3DV) 197–206 (2021). doi:10.1109/3DV53792.2021.00030.\nResponse Letter to Reviewers’ Comments\nWe sincerely appreciate the valuable time and efforts the reviewers have spent reviewing our manuscript. The insightful comments and feedback are instrumental in improving the quality of our work. We believe we have addressed all of the reviewer’s comments, resulting in a manuscript that is more rigorous in content and clearer in presentation. Our point-by-point responses to the reviewers’ comments are as follows.\nReviewer #1\nThe authors have carried out the revision diligently and have addressed each detailed comment thoroughly, with substantial improvements and clarifications. The Introduction section and the novelty of the work are now more clearly highlighted, and the release of the full code via GitHub further enhances transparency and reproducibility. I appreciate the authors’ efforts in strengthening the manuscript, and I find it suitable for acceptance.\nResponse:\nWe sincerely thank the reviewer for their positive and encouraging comments. We truly appreciate the recognition of our efforts to improve the manuscript’s clarity, highlight its novelty, and enhance transparency and reproducibility. Your supportive feedback is greatly appreciated.\nReviewer #2 (Remarks to the Author):\nI co-reviewed this manuscript with one of the reviewers who provided the listed reports.\nThis is part of the Nature Communications initiative to facilitate training in peer review\nand to provide appropriate recognition for Early Career Researchers who co-review\nmanuscripts.\nResponse:\nWe are grateful to this reviewer for the time and effort dedicated in reviewing our manuscript. The insightful comments provided in earlier revisions have been instrumental to enhance the quality of our work.\nReviewer #3 (Remarks to the Author):\nI appreciate the detailed responses and revision the authors made. The authors have\nnow fully addressed my comments. I recommend publication of the work in Nature\nCommunications.\nReviewer #3 (Remarks on code availability):\nThe code works fine.\nResponse:\nWe sincerely thank the reviewer for the time and effort devoted to evaluating our manuscript. We are delighted that our revisions have fully addressed the reviewer’s concerns and appreciate the positive recommendation for publication.\nReviewer #4\nI first want to acknowledge the substantial effort by the authors to address my and the\nother reviewers’ comments. The authors’ response addresses some of the raised\nissues. However, I still have the following concerns and questions.\nResponse:\nWe sincerely thank you for your recognition of our efforts and for the detailed comments that helped improve our work. Our detailed responses to your valuable feedback are provided below.\nComment #1\nThe authors acknowledge that their neuromorphic device does not produce optical flow,\nbut identifies regions of motion for downstream processing.\nHowever, they continue to use the term for the method that they propose. I maintain\nthat this is misleading. Readers will easily miss or misread the added sentence in the\nintroduction.\nThe title still is “Neuromorphic spatiotemporal optical flow: Enabling ultrafast visual\nperception beyond human capabilities”. Since there are works in the literature that do\nactually determine optical flow in a neuromorphic fashion, this is actually already\nmisleading, especially when the authors later state in the introduction: “Here, we\npropose a neuromorphic optical flow method that leverages two-dimensional\nneuromorphic synaptic transistors to form temporal motion cues directly in\nhardware, …” Moreover, in Figure 1, the “movement velocity calculation” block lies in\nthe “neuromorphic” part of the diagram. However, traditional algorithms like Farnebäck\nare used for this. Hence, for correctness, this block should be part of the traditional\nalgorithm side of the diagram.\nInstead, the authors actually propose neuromorphic hardware for change detection. In\nline 71-72, the authors capture this very nicely: “because biological vision excels at\nprocessing large volumes of visual information efficiently through dynamically\nfocusing on regions where motion occurs16,17.”\nIn their response to my previous review, the authors state that they base themselves\non the “biological retina-LGN interaction”. This sounds like a much better justification\nto me. Although I am no expert on this area, reading a bit about the LGN reveals that\nthey perform the function of a (spatio-)temporal attention mechanism, with the\nemphasis on temporal.\nHence, I propose that the authors focus more on this aspect, reflecting the change in\nthe title and introduction. It should be clear that optical flow is a downstream\nprocessing task that will be applied only to regions with motion.\nResponse:\nWe fully agree with the reviewer’s observation that our approach is more accurately described as mimicking the biological retina–LGN interaction rather than computing optical flow directly in hardware. To avoid any potential misunderstanding, we have revised the title, abstract, introduction, relevant figures and relevant texts to better reflect the nature and scope of our work. Throughout the manuscript, we have removed the term neuromorphic optical flow and consistently referred to our system as a motion extraction hardware (or upstream motion extraction method), which helps in downstream optical flow and motion analysis tasks.\nThe revised sections are summarized as follows (revisions and new additions are shown in red):\n1. Title\nAs suggested by both the reviewer and the editor, we have revised the title to more accurately represent the core contribution of this work: Ultrafast visual perception beyond human capabilities enabled by motion analysis using synaptic transistors.\n2. Abstract\nOptical flow, inspired by biological visual systems, calculates spatial motion vectors aiming to enable robotics to excel in dynamic environments. However, current algorithms, despite human-competitive task performance on benchmark datasets, suffer from significant time delays, limiting practical deployment. Here, we introduce a neuromorphic temporal-attention hardware that emulates the retina-LGN interaction to extract temporal motion cues directly in hardware. Using a two-dimensional synaptic transistor array, the system encodes brightness changes and accumulates them in analog, non-volatile states, generating compact regions of interest (ROIs). These ROIs then act as inputs to conventional downstream optical flow and vision algorithms, enabling ultrafast motion analysis. At the hardware level, the synaptic transistor offers high-frequency response (~100 μs), non-volatility (>10,000 s), and endurance (>8,000 cycles). Compared to state-of-the-art algorithms, our approach demonstrates a 400% speedup, surpassing human-level performance while maintaining or improving accuracy through temporal priors.\n3. Introduction\nTo explicitly prevent confusion between our neuromorphic motion-detection front end and optical flow computation, the Introduction has been thoroughly revised as follows:\nArticle, Page 3, Lines 67-111:\nOptical flow is initially aimed to mimic the processing pipeline of biological visual systems but cannot faithfully replicate their high processing efficiency in practice, which is because biological vision excels at processing large volumes of visual information efficiently through dynamically focusing on regions where motion occurs1,2. Specifically, biological vision, including the retina and lateral geniculate nucleus (LGN), emphasizes temporal change and dynamically routes information toward locations where motion occurs, acting as a temporal-attention function that detects moving regions before high-level complex processing3,4. This suggests a design principle for artificial vision: a fast, low-overhead function that dynamically detects motion regions, thereby accelerating downstream motion analysis by focusing computation where changes occur. Replicating this dynamic processing in artificial systems is challenging due to the intrinsic inflexibility of conventional CMOS-based technologies, which cannot readily adjust their processing functions in response to varying stimuli5,6. Fortunately, neuromorphic devices such as synaptic transistors and memristors offer synapse-like characteristics that can emulate the processing functions of biological synapses. Their intrinsic adaptability enables on-device adaptation and local feature extraction functions, which in turn lowers both energy consumption and processing latency. This approach has already been applied across multiple sensory modalities. In a typical implementation, a front-end sensor first transduces external stimuli into electrical signals (voltage or current), which are then transmitted to synaptic devices for processing. For tactile perception, the sensor is typically a piezoresistive or piezoelectric film that converts mechanical force into an electrical response. Based on this architecture, artificial tactile receptors are widely studied—including rapidly adapting mechanoreceptors and nociceptors—using neuromorphic hardware7–10. For example, Wang et al. proposed a memristor-based perceptual signal processing method, which can emulate multiple essential tactile receptors and sensory neurons with a single memristor7. For auditory processing, neuromorphic device arrays can capture and process auditory temporal patterns directly in hardware, enabling low-latency, low-power front ends for sound recognition and localization11–13. For visual processing, the high spatial redundancy of video frames aligns perfectly with the local, in-memory processing capability of synaptic devices7,14–16. For example, Baek et al. recently reported a neuromorphic neuron with dendritic morphology based on silicon-nanowire transistors that can perform nonlinear integration of excitatory/inhibitory synaptic inputs while considering spatial distribution dependency. Its built-in direction selectivity enables on-device motion detection, and the power consumption of processing event-based pulses from the 1,000 pixels is about 0.1–2.0 mW17. More broadly, the on-device adaptation capability of synaptic devices makes human-like ultrafast visual processing possible7,18–20.\nHere, inspired by the biological LGN system, we propose a neuromorphic hardware that directly detects motion changes using two-dimensional neuromorphic synaptic transistors. The array encodes brightness derivatives and accumulates these temporal changes in analog, non-volatile states, as shown in Fig. 1. The resulting array state generates temporal motion cues, i.e., compact maps of regions of interest (ROIs) that guide and accelerate downstream optical flow calculation. In our demonstration, conventional optical flow calculation methods (e.g., Farneback, GMFlow, and RAFT) are applied only inside these ROIs, not the entire image, ultimately accelerating various tasks such as motion prediction, segmentation, and tracking.\nIn summary, the neuromorphic hardware rapidly extracts motion change information, which is then passed to conventional processors for downstream optical flow computation in our demonstration.\n4. Relevant texts and figures\nWe have revised Figure 1 to replace the “movement velocity calculation” block with “optical flow calculation”, clearly indicating that these steps are performed using traditional algorithms. We also highlight the neuromorphic hardware and update the pipeline name to emphasize that our hardware is solely responsible for extracting motion changes. Corresponding figure captions and textual descriptions have been updated for consistency and clarity.\nFurthermore, throughout the manuscript, we have removed the term “neuromorphic optical flow” and consistently refer to our system as motion extraction hardware (or the upstream motion extraction method), which helps in downstream optical flow and motion analysis tasks.\nRevised Figure 1. Neuromorphic motion extraction hardware and its application. a, Example application scenarios. b, Comparison between the proposed neuromorphic motion extraction hardware pipeline and the conventional optical flow pipeline. c, Performance comparison of processing time and accuracy between the two approaches.\nComment #2\n2. Background motion\nI appreciate that the authors have performed and included many additional experiments.\nThe authors compare their method with a “conventional” pipeline. However, there are\ntwo changes introduced by the authors that should be clearly pieced apart in the results\n(as a form of ablation study):\na. Optical flow is now only applied to motion areas.\nb. This motion is calculated with neuromorphic hardware.\nIn the new results, the computation time of the proposed method (“Neuromorphic”) is\ncompared with “Conventional”. It is important to know which part of the speed-up is\ndue to contribution a and which due to contribution b. The authors did already measure\nthis (and mentioned it in their reply), but it should be clearly visible in these result\ngraphs.\nMoreover, in terms of accuracy of the methods, it should be really clear how this is\ndetermined. Is the conventional dense flow evaluated in the entire image? And is the\nneuromorphic method only evaluated in the motion regions? If so, this would not be a\nfair comparison, as optical flow will be easier to determine in areas in which motion is\nevident and texture is present. I would love to hear the answers to this myself, but this\nshould also be really clear in the manuscript.\nFinally, I don’t see videos of the added experiments, which make it a bit hard to estimate\nhow much background motion there actually is in the new sequences. However, in the\nfigures in the supplementary, it is clear that when there is more motion, segmentation\nshows quite some false positives, with objects on the car hood or on road markings.\nThis does not seem like a very good performance to be honest. The results on the\nchallenging scenario (R53) seem downright unusable…\nResponse:\nWe sincerely thank the reviewer for these constructive comments, which have helped us clarify the origin of the speed-up, ensure fairness in the accuracy comparison, and strengthen the presentation of results under challenging conditions. We have made several additions and clarifications to the revised manuscript as detailed below.\n1. Clarifying the Source of Speed-up\nWe agree with the reviewer that the observed performance improvement stems from two distinct factors:\n(a) restricting optical-flow computation to motion regions only, and\n(b) employing neuromorphic hardware to generate these motion regions.\nIn our manuscript, we have separated the contributions from factors (a) and (b) in the performance comparison; in other words, the speed-up results are influenced solely by factor (a) or solely by factor (b). To ensure clarity and avoid misunderstanding, all related figures now include annotations specifying which factor contributes to the observed speed-up.\nRevised Figure 4. Annotations added to panels i and j to indicate the source of speed-up.\nRevised Figure 6. Annotations added to panel b to indicate the source of speed-up.\nRevised Figure S27. Annotations added to panel c to indicate the source of speed-up.\nThe caption has also been updated to include the note: “The acceleration of the sub-experiments also stems from applying optical flow within motion regions.”\nRevised Figure S44. Annotations added to indicate the source of speed-up.\nThe caption has also been updated to include the note: “The acceleration of the sub-experiments also stems from applying optical flow within motion regions.”\nRevised Figure S54. Annotations added to panel b to indicate the source of speed-up.\n2. Clarifying the Accuracy Evaluation\nTo ensure fairness in the comparison, all accuracy metrics, including SSIM, PA, and IoU, are computed over the entire image for both the conventional and our proposed methods in our work. For instance, in the visual prediction task, the predicted next-frame image is compared against the full-scale ground-truth frame to obtain these metrics. The following sentence has been added to the Methods section:\nArticle, Page 14, Lines 526-528:\nAll evaluation metrics (SSIM, PA, and IoU) are calculated over the entire image area for both conventional and our methods to ensure fair comparison.\n3. Performance in Challenging Scenarios\nWe acknowledge that our approach is less effective in highly complex scenarios, as clearly reflected in STable 8. Such degradation is intrinsic to optical flow methods, as reported in previous studies, where performance decline under complex motion remains a challenge21,22. This can be observed in the additional SI figures provided in our previous response (Figs. S41–S43, panel b accuracy comparison). The primary function of our neuromorphic hardware is to extract motion regions and generate regions of interest (ROIs), effectively accelerating downstream processing, especially in scenes with sparse motion. However, as we have clarified, it does not enhance accuracy in complex scenes where conventional optical flow methods already struggle.\nTo make this limitation explicit, the following content has been added:\nArticle, Page 9, Lines 344-358:\nNevertheless, in challenging scenarios involving ego-motion or out-of-distribution motion, the performance of our method may degrade. To evaluate its robustness under such conditions, we conducted a comprehensive evaluation detailed in SNote 10. Specifically, we performed two sets of experiments: (i) controlled sequences captured using a hand-held phone, with concurrent IMU recordings of device motion, and (ii) real-world recordings from an in-car dashcam. In the first experiment, the observed speedup decreased to 170%, while in the second, the acceleration ratio was 74.8%, corresponding to a reduction in speed-up from 400% to 134% (1/74.8%). Regarding accuracy, as shown in STable 8, performance degrades in complex scenes compared with sparse-motion scenarios. This is consistent with the limitations of conventional optical flow methods, which also perform poorly under such conditions21,22. The observed degradation arises primarily from the constraints of the deployed optical flow computation and downstream task algorithms. The core role of our neuromorphic hardware is to extract motion regions and generate regions of interest, thereby accelerating downstream processing, making it powerful in scenes with sparse motion. Furthermore, we discuss potential motion compensation and fallback strategies that could be incorporated to further enhance system robustness (SNote 10).\nTo further support this clarification, we have added supplementary videos demonstrating these challenging sequences, including cases with dynamic backgrounds and self-motion, to provide a clear visual understanding of the performance. More details can be found in the attached Supplementary Video 3.\n(Fig. S41). Performance analysis of challenging scenario 1. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S42). Performance analysis of challenging scenario 2. a, Velocity inferences and task results of key frames. b, Performance comparison.\n(Fig. S43). Performance analysis of challenging scenario 3. a, Velocity inferences and task results of key frames. b, Performance comparison.\nComment #3\nThe authors describe both the working of an event-based camera and their own method.\nThey then say: “This characteristic can provide cleaner and more actionable data for\nfurther high-level processing beyond mere optical flow computation.” I agree with this\nstatement. The processing of the proposed array is different from the event-based\ncamera, retaining different information. Instead of just a binary ON / OFF output, it\noutputs the change in light intensity. It is surprising however, to then read: “Thus, the\nsynapse array is not a replacement for event sensors but a temporal front-end that\naccumulates temporal information in analogue and generates ROIs early in the pipeline\nthat can be flexibly combined with downstream computation” Since the information\npassed on by the array is inherently different, it actually is a replacement / alternative /\ncomplement to event-based cameras.\nWhat follows in the rebuttal is a bit strange, as the authors propose to combine the\napproaches by basically counting events, independently of the polarity. This represents\nan *alternative* to their main proposed method for detecting motion regions, but not a\nway in which their proposed method can be applied to event cameras…\nWhat I would propose is that the authors would describe their proposed method as an\nalternative to event-based vision for detecting regions of motion. Ideally, a comparison\ncould be made in which regions of motion are identified as the authors propose based\non event counts, and effects are studied on downstream tasks. However, I feel that it is\nnot the amount of work that is the problem, but the clarity on what the contribution is\nand how this is supported by experimental evidence.\nSince the main contribution is to first identify regions of motion and then apply\ndownstream algorithms for, e.g., optical flow, the authors should discuss their\ncontribution with respect to previous similar works:\nSagar, J., & Visser, A. (2014, August). Obstacle avoidance by combining background\nsubtraction, optical flow and proximity estimation. In IMAV 2014: International Micro Air\nVehicle Conference and Competition 2014, Delft, The Netherlands, August 12-15, 2014.\nDelft University of Technology.\nhttps://www.imavs.org/papers/2014/paper40.pdf\nDenman, S., Fookes, C., & Sridharan, S. (2009, December). Improved simultaneous\ncomputation of motion detection and optical flow for object tracking. In 2009 Digital\nImage Computing: Techniques and Applications (pp. 175-182). IEEE.\nFrom the second article’s abstract: “The proposed algorithm uses the motion\nsegmentation results to inform the optical flow calculations and ensure that optical flow\nis only calculated in regions of motion, and improve the performance of the optical\nflow around the edge of moving objects.”\nResponse:\nWe thank the reviewer for this thoughtful and constructive clarification, and for agreeing our view that “this characteristic can provide cleaner and more actionable data for further high-level processing beyond mere optical flow computation.” We fully agree that our synapse-array architecture should be presented as an alternative neuromorphic approach to event-based vision for detecting motion regions. Regarding the experiment that counts events irrespective of polarity, this was included only to demonstrate that our neuromorphic hardware can also accumulate event-based input streams. We have clarified this point in the revision to avoid any confusion. The manuscript has been revised accordingly.\nArticle, Page 11, Lines 421-426:\nCompared with event-based cameras, which also extract motion changes rapidly, the proposed synapse array performs analog accumulation of light-intensity changes, producing a continuous-valued representation that reflects short-term motion history. Thus, the synapse array should be viewed as an alternative approach to event-based vision for detecting regions of motion. Its output, determined by the continuous conductance states of the neuromorphic hardware, differs fundamentally from the binary ON/OFF events produced by DVS sensors, providing cleaner and more actionable data for high-level processing beyond mere optical flow computation. A more detailed comparison is provided in SNote 15, and we further demonstrate that our neuromorphic hardware can accumulate event-based input streams (see SNote 16).\nAdditionally, we have revised or removed the previously confusing discussions in SNotes 15 and 16 to improve clarity as shown in the following content. The complete updated versions are provided in the revised Supplementary Information.\nSNote 15:\nThe above working principle also affects the needed operations for subsequent optical flow computation. During velocity inference, a large fraction of event-based optical flow methods first accumulate events in short windows into tensors, e.g., time surfaces (as in EV-FlowNet) or voxel grids with temporal bins (as in E-RAFT), before feature extraction and correlation25,26. In our case, each synaptic device performs this temporal accumulation in hardware, producing a non-volatile analogue time surface that updates via modulation pulses. Therefore, this hardware-level accumulation cuts both energy and processing time. Regarding region-of-interest (ROI) selection, we acknowledge the similarity: event streams already imply ROIs. Nevertheless, an explicit ROI mask derived from synaptic states can be applied within event pipelines, e.g., to crop or mask spatially dense convolutions/correlation to reduce computation. Thus, the synapse array is not a replacement for event sensors but a temporal front-end that accumulates temporal information in analogue and generates ROIs early in the pipeline that can be flexibly combined with downstream computation.\nSNote 16:\nTo demonstrate the versatility of our architecture, we further tested how the synaptic array could process event-based input streams. Starting from the hardware perspective of an event-based camera, which includes photoreceptors, differentiators, and ON/OFF comparators, there are two potential operating schemes that transform the event stream into programming voltages for the synaptic array.\nTo better position our contribution, we have added the following discussion:\nArticle, Page 10, Lines 375-389:\nComparison with ROI-first approaches in prior work\nThis ROI-first strategy has precedents in software and sensor work that restrict optical-flow computation to likely moving regions. For example, Sagar et al. detect foreground regions and combine foreground-focused processing with template/scale cues for monocular MAV obstacle avoidance; their pipeline demonstrates that processing only foreground regions can substantially reduce computation and improve obstacle confidence in constrained flight settings23. Denman et al. propose a combined optical-flow algorithm that uses foreground masks to limit flow computations and to improve flow near object boundaries24. Our work differs from these prior software pipelines in one fundamental respect: the temporal cues are generated directly in analog hardware by synaptic transistors with non-volatile, high-frequency response. This hardware generation obviates repeated frame-to-frame accumulation in software, enabling rapid ROI formation and thus ultrafast visual perception. In short, while Sagar & Visser and Denman et al. demonstrate the value of restricting flow to motion regions, our synaptic array provides a hardware temporal attention mechanism that produces those regions faster and with a different information representation.\nReferences:\n1. Adelson, E. H. & Bergen, J. R. Spatiotemporal energy models for the perception of motion. J. Opt. Soc. Am. A, JOSAA 2, 284–299 (1985).\n2. Borst, A. & Helmstaedter, M. Common circuit design in fly and mammalian motion vision. Nat Neurosci 18, 1067–1076 (2015).\n3. O’Connor, D. H., Fukui, M. M., Pinsk, M. A. & Kastner, S. Attention modulates responses in the human lateral geniculate nucleus. Nat Neurosci 5, 1203–1209 (2002).\n4. McAlonan, K., Cavanaugh, J. & Wurtz, R. H. Guarding the gateway to cortex with attention in visual thalamus. Nature 456, 391–394 (2008).\n5. Clifford, C. W. G. & Ibbotson, M. R. Fundamental mechanisms of visual motion detection: models, cells and functions. Progress in Neurobiology 68, 409–437 (2002).\n6. Zidan, M. A., Strachan, J. P. & Lu, W. D. The future of electronics based on memristive systems. Nat Electron 1, 22–29 (2018).\n7. Wang, S. et al. Memristor-based adaptive neuromorphic perception in unstructured environments. Nat Commun 15, 4671 (2024).\n8. Yoon, J. H. et al. An artificial nociceptor based on a diffusive memristor. Nat Commun 9, 417 (2018).\n9. Hong, S. J. et al. Bio-inspired artificial mechanoreceptors with built-in synaptic functions for intelligent tactile skin. Nat. Mater. 1–9 (2025) doi:10.1038/s41563-025-02204-y.\n10. Donati, E. & Valle, G. Neuromorphic hardware for somatosensory neuroprostheses. Nat Commun 15, 556 (2024).\n11. Truong, S. N., Ham, S.-J. & Min, K.-S. Neuromorphic crossbar circuit with nanoscale filamentary-switching binary memristors for speech recognition. Nanoscale Res Lett 9, 629 (2014).\n12. Seo, S. et al. Artificial van der Waals hybrid synapse and its application to acoustic pattern recognition. Nat Commun 11, 3936 (2020).\n13. Gao, S. et al. Programmable Linear RAM: A New Flash Memory-based Memristor for Artificial Synapses and Its Application to Speech Recognition System. in 2019 IEEE International Electron Devices Meeting (IEDM) 14.1.1-14.1.4 (2019). doi:10.1109/IEDM19573.2019.8993598.\n14. Lee, J. et al. Light-enhanced molecular polarity enabling multispectral color-cognitive memristor for neuromorphic visual system. Nat Commun 14, 5775 (2023).\n15. Choi, C. et al. Curved neuromorphic image sensor array using a MoS2-organic heterostructure inspired by the human visual recognition system. Nat Commun 11, 5934 (2020).\n16. Huang, H. et al. Fully integrated multi-mode optoelectronic memristor array for diversified in-sensor computing. Nat. Nanotechnol. 20, 93–103 (2025).\n17. Baek, E. et al. Neuromorphic dendritic network computation with silent synapses for visual motion perception. Nat Electron 7, 454–465 (2024).\n18. Chen, W. et al. Essential Characteristics of Memristors for Neuromorphic Computing. Advanced Electronic Materials 9, 2200833 (2023).\n19. Wang, S. et al. Memristor-Based Intelligent Human-Like Neural Computing. Advanced Electronic Materials 9, 2200877 (2023).\n20. Liu, F. et al. Printed synaptic transistor–based electronic skin for robots to feel and learn. Science Robotics 7, eabl7286 (2022).\n21. Baker, S. et al. A Database and Evaluation Methodology for Optical Flow. Int J Comput Vis 92, 1–31 (2011).\n22. Wang, Y. et al. Occlusion Aware Unsupervised Learning of Optical Flow. in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition 4884–4893 (IEEE, Salt Lake City, UT, 2018). doi:10.1109/CVPR.2018.00513.\n23. Sagar, J. & Visser, A. Obstacle avoidance by combining background subtraction, optical flow and proximity estimation. Proc. Int. Micro Air Vehicle Conf. and Competition (IMAV 2014), Delft, The Netherlands, 12–15 Aug 2014. Delft University of Technology (2014).\n24. Denman, S., Fookes, C. & Sridharan, S. Improved Simultaneous Computation of Motion Detection and Optical Flow for Object Tracking. in 2009 Digital Image Computing: Techniques and Applications 175–182 (2009). doi:10.1109/DICTA.2009.35.\nReviewer #4 (Remarks to the Author): I thank the authors for their considerable effort in replying to my previous remarks, and the modifications to the article - I think it now much better captures the scientific contribution of the authors. I just have one very minor remark left: please note that in the figures conventional is sometimes red and sometimes blue, while neuromorphic is also sometimes blue and sometimes red. It would be easiest for the reader if neuromorphic is always blue and conventional always red - giving neuromorphic a nicer color ; )\nResponse: We thank the reviewer for their positive assessment and for catching this inconsistency. We agree that a unified color scheme improves readability. We have revised all relevant figures in the main text to strictly follow this standard:\nNeuromorphic method: Always represented in Blue.\nConventional method: Always represented in Red."}, {"round": 2, "reviewer_comments": {"reviewer_4": "I thank the authors for their considerable effort in replying to my previous remarks, and the modifications to the article - I think it now much better captures the scientific contribution of the authors.\nI just have one very minor remark left: please note that in the figures conventional is sometimes red and sometimes blue, while neuromorphic is also sometimes blue and sometimes red. It would be easiest for the reader if neuromorphic is always blue and conventional always red - giving neuromorphic a nicer color ; )\n(Remarks on code availability)\n-"}, "author_response": "Response: We thank the reviewer for their positive assessment and for catching this inconsistency. We agree that a unified color scheme improves readability. We have revised all relevant figures in the main text to strictly follow this standard:\nNeuromorphic method: Always represented in Blue.\nConventional method: Always represented in Red."}]

Decision letter