NEOSTI - a neuromorphic electronic-opto spatial-temporal hybrid image sensor

Domain: physics
Published: 2026-03-26
Editorial outcome: major_revision

Abstract
Image sensors in machine vision systems face significant challenges related to energy efficiency and processing capability when storing, transferring, and processing massive amounts of data. In humans, over 80% of brain-processed information is obtained through the eyes, which are capable of detecting and synchronously processing information with extremely low overall power consumption. Inspired by the biomimetics, we propose a Neuromorphic Electronic-Opto Spatial Temporal Imager (NEOSTI), one of the smallest electronic-opto fully integrated, eye-sized vision systems enabling acquisition and operation in typical indoor/outdoor non-coherent environments, under both natural and artificial lighting conditions without any extra requirement of the light source. NEOSTI combines processing-pre-sensor in optical domain, processing-in-sensor with nonlinear acquisition capability while optical to electronic converting, and processing-near-sensor in electronic domain, enabling parallel data computing capabilities while sensing. NEOSTI also integrates a low complexity Binary Neural Network to process image semantic information. It attains competitive performance in several visual processing tasks.

Introduction
The human visual system is highly efficient in collecting and processing of the visible information. As a bionic device, the image sensor establishes a bridge between the real world in optical domain and artificial intelligence in electronic domain, but current image sensor technology falls short in meeting the requirement of real-time, low-power processing capability at the edge, which is critical for embodied intelligence such as intelligent robots, fully autonomous driving, and drones that require effective interaction with dynamic environments. Different processing technologies in either electronic domain or optical domain have been proposed in literature to break the bottleneck, but there is always a trade-off between processing performance and power/volume consumption.

Traditional artificial vision system relies on image sensors coupled with cloud processing or Artificial Intelligence (AI) processors, featuring higher image resolution, higher frame rate, and higher accuracy across various image processing tasks than human visual system, but paying for a several orders higher power consumption than the human visual system. In addition, the sub second wireless communication latency [1] between the sensor end and the computational cloud cannot meet the milliseconds to sub-milliseconds response time requirement risen from the embodied intelligence scenario.

In order to improve efficiency of the traditional artificial vision system, Processing-in-Sensor (PIS) [2–9] and Processing-Near-Sensor (PNS) [10–12] technologies have been introduced into the raw intensity acquisition process. PIS/PNS technology mitigates the delay and power consumption associated with redundant data transmission workload by integrating the sensing and computing components, enhances data processing efficiency as well as reduces the volume of data transmitted to the host through parallel processing at column or pixel levels. This approach aligns more closely with the operational principles of the visual systems observed in humans and various creatures. Nevertheless, the interdependence of the sensing and computing ends, influenced by the manufacturing process, implies that the performance of one end can impact the other, and the computing capabilities achievable on the sensing end are inherently limited.

In addition to PIS/PNS in electronic domain, optical domain calculation also provides a promising solution to increase computing speed without greatly increase the power consumption, as well as to overcome the bandwidth bottlenecks of PIS/PNS. In particular, the optical solution enables compression in the spatial domain of the image, resulting in a reduction of the amount of data to be processed on the sensor side. The development of the neural networks using optical methods, commonly referred to as optical neural networks (ONNs), has gained considerable traction as a prominent research area. Different ONNs have been proposed in literature [13–15], realizing parallel information processing by integrating optical architecture based on free space, aiming to accelerate deep learning computation. However, ONNs faces constraints related to power consumption, data rate, and on-chip processing capabilities. Additionally, many ONNs have restriction on light source, such as specific coherent light source, which cannot work in natural light condition.

In this article, we propose an all-in-one neuromorphic electronic-opto spatial-temporal image sensor (NEOSTI) design, which integrates various technologies to enhance data processing efficiency as well as to reduce sensor volume size. The NEOSTI integrated processing-pre-sensor(PPS) in optical domain, PIS with nonlinear acquisition capability and parallel data computing capabilities while sensing, and PNS with an on-chip binary neural network (BNN) for image semantic information processing. The NEOSTI attains competitive performance in five static and dynamic visual processing tasks. The experimental results reveal an accuracy of 96.4% in MNIST (10-category), 94.9% in UnityEyes (4-category), 91.7% in QuickDraw (10-category), 91.3% in Fashion MNIST (10-category), and 90.6% in Weizmann (video, 10-category), which demonstrate significant advancements in accuracy across multiple categories. These findings demonstrate a compact and efficient vision system with potential applications across a range of scenarios. Table 1 summarizes and compares the NEOSTI with representative frontier work in the fields discussed above. The NEOSTI is one of the smallest electronic-opto fully integrated vision systems that can operate in typical indoor and outdoor environments, under both natural and artificial lighting conditions, without any extra requirement of the light source, such as laser or coherent light source.

Results
Architecture of NEOSTI

The human visual system (Fig.1a) enables efficient information compression while acquisition. There are over 130 million photoreceptor cells located in retina converting illumination intensity into neural spike signal. However, since a significant amount of pre-processing occurs within the retina itself [27], only about 1.2 million fibers (axons) are required in the optic nerve, resulting in an equivalent compression ratio of over 100. In addition, the Lateral Geniculate Nucleus (LGN) in the visual pathway plays a role in attenuating temporal correlations within the image, which further reduces the total amount of neural signals to be processed in the visual cortex. Secondly, the retina consumes only about 3 mW for both intensity acquisition and fundamental feature extraction [28], realizing a 23.08 pW/pixel energy efficiency. While state-of-the-art consumer image sensors only achieve an efficiency around 30 nW/pixel [29] for only intensity acquisition. In addition, the human visual system enables an intra-scene dynamic range (DR) of over 120 dB [28] and an inter-scene DR over 180 dB. While consumer image sensors typically achieve a DR of only 60-70 dB, the most advanced of which reaches around 140 dB [30].

Inspired by biomimetics, NEOSTI (Fig.1b) incorporates similar functionalities, including computations throughout the entire process, closely mimicking the capabilities of biological systems. The workflow of NEOSTI (Fig.1c) starts from a sliding convolution in the optical domain. A mask is deployed in front of the photodetector array in the electrical domain, filtering the input images and extracting the spatial features, which effectively reduces the amount of data received on the sensor plane. Subsequently, visual information is further processed using nonlinear optoelectronic conversion together with a calculation-supported readout link of the pixel circuit. At last, a low-complexity on-chip BNN is implemented as the visual cortex, providing high-level information processing capabilities, and finally obtaining the end-to-end classification result.

NEOSTI can be equivalent to an algorithm model (Fig.1d), which combines the optical domain and the electrical domain to construct a 3-layer sliding convolutional neural network. Initially, the 3-layer neural network model is trained to obtain the parameters for each part of the network. Subsequently, the parameters of the first layer’s 5 × 5 convolution kernel are extracted, binarized and solidified into physical patterns. The remaining of the network is reprogramed into a reconfigurable chip. The convolution mask and chip are co-packaged forming a lensless vision chip, which is capable of being fabricated using a mature process, making it well-suited for large-scale production.

Pre-sensor Processing in Optical Domain

The human eye (Fig.2a) perceives a world full of vibrant colors, while each animal has its distinct method of processing visual information. For instance, the compound eyes (Fig.2b) of flies are highly specialized, capable of capturing images at remarkable speeds. Their visual nervous system is adapted to rapidly process moving objects, enabling them to detect and react to their environment with high efficiency. Pre-processing occurs directly within the fly’s eyes for promptly responding to movement in their surroundings. Similarly, in frogs, specialized neurons in the retina are responsible for detecting edges or sudden motions within a scene. These neurons aid frogs in their survival strategies for hunting prey or evading predators. By filtering and integrating varying levels of light intensity and boundary information, a sophisticated feature detection is performed to ensure rapid and precise visual processing.

NEOSTI (Fig.2c) serves as the ”eyes” for machines, such as robots, facilitating intelligent processing across a variety of scenarios. It combines the capability of human eyes for the acquisition of the imaging of a scene, and compound eyes for feature extraction. The incident scene passes through four convolution kernels on the mask (Fig.2d), resulting in four feature maps on the sensor. The sliding convolution calculation is completed through the projection relationship of the geometric optics. Each pixel value on the feature map is obtained in parallel with near-zero latency. The schematic diagram of the equivalent optical calculation of the mask can be represented in a 5 × 5 binary matrix (Fig.2e). Assume that the input light source consists of a series of point light sources, by adjusting the distance from the mask to the imaging plane, each pixel on the imaging plane can receive the convolutional result between the point light source within a specific region and the convolution kernel parameters on the mask. As a result, the sensor plane receives four parallel convolution feature maps (Fig.2f). The feature size of the mask is limited by both diffraction and optical blur. (Fig.2g). Leveraging the propagation characteristics of geometric optics, this process can be effectively represented as a window sliding convolution, which is performed in parallel. The PPS operates at the position of the lens or glasses within the human visual system, augmenting the system’s capabilities and enabling direct processing of optical domain data, which is an attribute absence in the human visual system. By processing information in the optical domain, data arriving at the PIS module can be compressed with near-zero latency and zero power consumption, thereby reducing power consumption in both photoelectric conversion and data processing stages.

Neuromorphic Nonlinear Photoelectric Response

The retina, situated at the back of the eye, is a layered structure responsible for capturing light and initiating the process of vision (Fig.3a). The cones and rods in the lowest layer serve as photoreceptors, generating bio-signals associated with light intensity. The signal is then conveyed and transmitted through cells in the middle region of the retina. Ultimately, the generated nerve impulses are traveling along the optic nerve tract. The impulses are subsequently relayed to the visual cortex in the brain.

In a Complementary Metal Oxide Semiconductor (CMOS) image sensor, a reverse-biased photodiode structure is employed as the photoreceptors for photoelectric conversion (Fig.3d). The photodiode generates a photocurrent that is positively correlated with light intensity. A pre-reset capacitor is discharged by the photocurrent, causing the voltage VPD to decrease at a rate contingent upon the light intensity, realizing a conversion from optical intensity to electrical signals. The signals are subsequently amplified and read out by the associated circuit. Typically, a voltage-readout pixel structure is utilized. It is also known as the accumulation-mode pixel structure (Fig.3b), in which the voltage drop, ∆VPD, over a certain exposure time is measured and read out via an analog-to-digital convertor (ADC) to obtain the corresponding digital code value (Fig.3c), resulting in a linear relationship for the light-to-voltage conversion (gray line in Fig.3i).

Conversely, the human visual system employs a nonlinear relationship between the light intensity and perception is employed due to the nonlinearity of photoreceptor response (Fig.3j). This nonlinear characteristic effectively expands the visual capacity to a wider range of light levels, particularly in extremely bright or dim environments.

Inspired by the human vision system, NEOSTI utilizes a pulse width modulated (PWM) based 3-layered readout pixel structure [31–33] (Fig.3e), realizing a retina-mimicking nonlinear photoresponse and pulse-based representation of light intensity. The fundamental principle of the pulse modulation method starts with setting a voltage threshold at the configuration stage. A pulse flip was triggered when the voltage on the photodiode reaches the preset threshold VINV during the exposure process (Fig.3f). This approach enables a nonlinear light-to-time conversion without the requirement of ADC, and allows operation at lower voltages. The pulse output format also facilitates low-power computation design at the circuit level processing, which will be discussed in the following section. To further align the nonlinear response of PWM pixel with the human eye, a dynamic scanning bias voltage, VG, is applied at point G (Fig.3g), making the threshold of the flipping voltage VINV to gradually increase over time. Compared to a fixed flipping voltage threshold, dynamic VINV stretches the response curve in low-light intensity ranges, extending the dynamic range of pixel under dark light condition (Fig.3h-i). Experimental results indicate that the PWM pixels of NEOSTI exhibit a nonlinear photoresponse function similar to that of visual cells to encode light intensity information (Fig.3k). Moreover, the simulated responses demonstrate that by adjusting the bias voltage VG, the pixel has the potential to mimic the adaptive function of the human eye, shifting the curve as a whole to accommodate varying lighting conditions (Fig.3l).

In-Sensor and Near-Sensor Processing

Behind the photosensitive area, an array of processing elements (PEs) is employed to receive the pulse signal, enabling parallel visual information in-sensor processing. The processing element is simultaneously connected to two adjacent PWM pixels (Fig.4a), performing direct summation or subtraction calculation on the two pulse signals from two neighboring pixels based on programmable parameters during the exposure process. The working times of the two counters within the PE are non-overlapping (Fig.4b), which reduces the counters’ working duration by 33% compared to conventional readout methods, lowering circuit dynamic power consumption. The summation calculation is performed using a left-shift module for efficient ”×2” operation, and a digital adder to sum the digital codes, while the subtraction calculation relies solely on counter DIFF, allowing counter SUM to be disabled to achieve greater performance advantages. The overall in-sensor processing forms part of a 2 × 2 binary convolution with a stride of 1, which effectively removes spatial-temporal redundancy and extract fundamental features from the input data.

Additionally, a BNN is implemented on-chip, providing advanced near-sensor information processing capabilities, and enhancing the ability to generalize across various tasks. Due to the concurrent exposure of each row of pixels and the simultaneous generation of intra-row calculation results by parallel PEs, the implementation of a pipeline architecture becomes imperative to efficiently execute convolution operations. A 4-stage pipeline dedicated to a convolution block (Fig.4c) is utilized in NEOSTI. The block encompasses a 2 × 2 convolution layer, a ReLU nonlinear activation function, and a 2 × 2 maximum pooling layer. This pipelined architecture is compactly parallelized, fully utilizing the row-by-row exposure time of the pixels to complete the pipeline operations without any additional time overhead, maximizing the utilization of on-chip hardware resources with high computational efficiency. There are in total two layers of convolution and pooling operations performed through the in-sensor and the near-sensor processing, yielding the fully-processed feature maps. By leveraging on-chip storage capabilities, the difference between frames can be directly calculated, which is a commonly-used operation for inter-frame processing. Combined with subsquent pipelined BNN structure, NEOSTI is endowed with enhanced ability for processing spatial-temporal hybrid information.

Evaluation of NEOSTI system

The performance of NEOSTI has been validated across multiple tasks, including MNIST[34], Fashion MNIST[35], Quick Draw[36] and UnityEyes[37], with an accuracy of 96.4%, 91.3%, 91.7%, and 94.9%, respectively (Fig.5a). In experiments, the opticalmaskkeptthesameconvolutionpattern,whiletheparametersofelectricallayerswere reconfigured for different tasks. The experimental results not only show the recognition performance of NEOSTI on various tasks, but also demonstrate that the programmableelectricalconvolutionlayersprovideNEOSTIwithgoodgeneralization ability between different tasks.

In order to further verify the respective functions of each part of NEOSTI, differentsetups,includingexperimentsusingonlyPPSoronlyPIS,wereverifiedseparately (Fig.5b). The NEOSTI shows an average accuracy improvement of 11.725% and 14.075%comparedwithonlyPISoronlyPPS.ComparedwithLeNet-5,asmallneural network in the digital domain, NEOSTI shows similar performance. In addition, we also verified the noise robustness of NEOSTNI (Fig.5c) using the peak signal-to-noise ratio (PSNR) calculated as
PSNR=10·log(MAXI)2/MSE=20·log(MAXI/√MSE) (1)
where MAXI represents the maximum pixel value, and MSE represents the noise level. When the light intensity becomes lower, the PSNR deteriorates. Experimental results show that when the PSNR decreases, NEOSTI does not significantly deteriorate in the accuracy of different characters. This feature is advantageous due to the fact that the optical domain processing enables the filtration of a portion of the noise. Furthermore, the neural network’s robust anti-interference capability enables better performance, particularly in low-light photo scenarios.

In addition to static image classification task, NEOSTI was also tested on the video analysis task, which requires not only the capability of spatial information acquistion, but also the capability of temporal information analysis between frames. NEOSTI is evaluated on an open dataset, Weizmann [38]. The Weizmann dataset (Fig.5d) includes multiple video sequences of ten different actions, including run, jump, bend, etc. In each video sequence, only one person performs the action. NEOSTI can convert sequence information into several feature maps (Fig.5g), with an accuracy of 90.6% (Fig.5e). The BNN-based feature extraction model serves as a general, real-time, and efficient data compressor that effectively reduces the input data volume by 81.3%, significantly reducing the computational and storage overhead of feature sequences while preserving intra-frame spatial features. In different-setup experiments, the only PPS setup shows a 58% accuracy on the same dataset (Fig.5f). This is mainly due to the fact that the processing of the optical domain is spatially without storage. The programmable on-chip PIS and PNS circuits can make up for this. Similarly to the test on static image classification task, when noise increases, NEOSTI exhibits good robustness (Fig.5h). The spatial domain filtering of the PPS module still plays a key role in processing noise.

Methods
Design of optical domain convolution mask

Anopticalmaskwasappliedinfrontofthepixelarrayinsteadofalens(Supplementary Fig. 1a) to realize the calculation of one-layer convolution.
As a result, there are four 5 × 5 binary convolution kernel patterns representing the pre-trained convolution kernel parameters (0 or 1) of the first layer of the BNN (Supplementary Fig. 1b). The feature size of the kernel pattern, ∆, was set to 40 µm (Supplementary Fig. 1c) to satisfy ∆≥ (2.44λd)/K to avoid the diffraction effect withalightsourcewavelength,λ,of405nm.Thepitchsize(d)wascalculatedtobe 1.28 mm (Supplementary Fig. 1a) to achieve a coverage of one-quarter sensor array (about 1.28 mm × 1.28 mm) by each kernel pattern, as well as to generate 4 non-overlap feature maps for the following layer. According to the theory of geometrical optics, the sensor plane captures the light signal S0,S1,...,Sm−1 as an outcome of sliding multiplication and accumulation between the light source, L0,L1,...,Ln−1, and the mask. This is a parallel computing process with no power consumption and almost zero latency.

The aperture ratio of the optical mask is approximately 50%. Additionally, the mask template is fabricated using a process where a coating is applied to a glass substrate. Due to losses caused by refraction and other factors in the glass substrate, when the pixel is set to be transparent, the transmission rate is approximately 90%. Therefore, the overall transmission rate is 45%. The light intensity attenuation caused by modulation, refraction, and other factors is approximately 3.47 dB.

Retina-inspired photoreceptors and pulse modulation

ThePWMpixelstructuregeneratesdifferentwidthspulsesignalstorepresentdifferent light intensities. This architectural configuration offers two benefits: 1) its dynamic rangeremainsunrestrictedbypowersupplyvoltage(VDD)constraints,facilitatingits operationevenatexceptionallylowerpowersupplylevers,thusrenderingitamenable forenergy-efficientcomputationaltasks;2)theresultantoutputsignalisdirectlydigital signal, obviating the conventional necessity for an ADC, and seamlessly integrating into computational processes. In this work, all the transistors are operated in the sub-threshold region under a VDD of 0.35 V. Its working process is realized by a current comparator composed of two transistors, MP and MN (Supplementary Fig. 2a). In the exposure phase, VPD is discharged by the bias current source, resulting in an exponential increase of the drain current of MP. Simultaneously, the decrease of VPD leads to a reduction in the drain current of MN. IP and IN are denoted as:
IP=IP0×exp((VDD-VG-VTH,P)/(npVT)) (3)
IN=IN0×exp((VPD-VTH,N)/(nnVT)) (4)
With exposure, IP gradually becomes larger than IN, causing the voltage at node VOUT is pulled up. At this moment, IP is approximately equal to IN, (3) and (4) yield:
VPD,INV=nn/(np+nn)*(VDD-VG,INV-VTH,P)+nn/(np+nn)*VT*ln(IP0/IN0)+VTH,N (5)
ThepulsewidthTINV,correspondingtothetimeelapsedfromthestartofexposure to the moment when the voltage VPD is flipped, is proportional to the incident light intensity (Supplementary Fig. 2b), which can be calculated as:
TINV=CPD*(VPD,RST-VPD,INV)/(Iph-IP0*exp((VDD-VG,INV-VTH,P)/(npVT)))
For dynamic scanning bias voltage applied in our work, VG is represented by TINV as:
VG,INV=VG0-k·TINV (7)
where k is the slope of the scanning bias voltage change with respect to time (Fig.3.g).

To balance the trade-off between power consumption and noise performance in different pixel working mode, two distinct reset strategies are implemented in the acquisition mode and the calculation mode, respectively. This is achieved by adding a cascode transistor MR to the typical PWM pixel structure (as shown in Supplementary Fig.2a), which controls the on/off state of the current path during the reset phase.

In the acquisition mode, the low-noise reset approach is applied to eliminate the threshold voltage mismatch noise, which is described as follows. The switch SW2 is closed during the reset phase, connecting the drain and gate of the NMOS transistor MN. When the charge is fully reset on the PD, there will be no current passing through this branch. Then the transistor MR is turned on, establishing a current path VDD-MP-MR-MN-VSS, which ensures the initial IP = IN. The reset voltage of the photodiode can be derived from (3) and (4) as:
VPD,RST=nn/(np+nn)*(VDD-VG,RST-VTH,P)+nn/(np+nn)*VT*ln(IP0/IN0)+VTH,N (8)
At the time TINV, The difference of the two voltages VPD,RST and VPD,INV is:
∆VPD=VPD,RST-VPD,INV=nn/(np+nn)*(VG,INV-VG,RST) (9)
The∆VPD issolelydependentonthedisparitiesbetweennp,nn andVG,effectively mitigatingvariationsinducedbythefabricationprocessandsuppressingcolumnFPN.

In the calculation mode, the transistor MR is turned off to interrupt the flow of DC current from MP to ground through MN to achieve a low-power reset, which reduces the average current in the reset phase to 0.17% of that in the on state, leading to a 31% reduction in the average power consumption per row time. Although the threshold mismatch noise is not eliminated, experiments conducted in the calculation mode show that a 1% pixel value deviation leads to only about a 0.2% decrease in classification accuracy. In addition, the PE eliminates most of the errors when calculating sum/differential of the adjacent pixels. As a result, the noise caused by non-ideal factors has little impact in the calculation mode.

The pipeline structure of neural network

Duetotheconcurrentexposureofeachrowofpixelsandthesimultaneousgeneration of intra-row calculation results by 64 PEs, the implementation of a pipeline architecture becomes imperative to efficiently execute convolution operations. Supplementary Fig.4a illustrates the design and workflow of a 4-stage pipeline dedicated to a convolution block. This block encompasses a 2 × 2 convolution layer, a ReLU function, and a 2 × 2 maximum pooling layer. The hardware structure of the convolution block is illustrated in Supplementary Fig.4b, which is implemented directly after the PEs of each column. The specific process of the 4-stage pipeline is described as follows:
Cycle 1: 64 PEs compute and generate the 2 × 1 outputs, PEOUT0(m) and PEOUT1(m), for the mth row of pixels, where, RES0 and RES1 are the convolution results of the two adjacent pixel values with the first and second rows of the convolution kernel K = [k0,k1;k2,k3], respectively.
For the nth column of pixels, RES0 and RES1 are calculated as
RES0(m,n)=k0·IN(n)+k1·IN(n+1)
RES1(m,n)=k2·IN(n)+k3·IN(n+1)
The output result of PEOUT0(m) will be cached to Cycle 2.
Cycle2:ThecachedresultPEOUT0(m)issummedwithPEOUT1(m+1)togenerate the 2 × 2 convolution results for the mth and (m+1)th row. The convolution results then pass through the ReLU activation function module, whose hardware structure is a 2-to-1 multiplexer controlled by the sign bit of the input (Supplementary Fig.4c) : if the sign bit is 0 (indicating a positive value), the original data is passed through; if the sign bit is 1 (indicating a negative value), a value of zero is output. The outputs of convolution and ReLU activation(denoted as CM(m)) are stored to Cycle 3.
CM(m)=ReLU(PEOUT0(m)+PEOUT1(m+1)) (12)
Cycle 3: The 2 × 1 maximum pooling,which is implemented in hardware using a comparator and a 2-to-1 multiplexer (Supplementary Fig.4d), is performed within the row direction. And the output result MP(m) is cached to Cycle 4.
MP(m,n/2)=Max(CM(m,n)+CM(m,n+1)) (13)
Cycle 4: Another 2 × 1 maximum pooling module is applied to the previously cached result MP(m) and the newly generated result MP(m+1) from the last stage, producing the final 2×2 maximum pooling result.
OUT(m/2,n/2)=Max(MP(m,n/2),MP(m+1,n/2)) (14)
With the pipelined calculating hardware structure, the processing results of one standard BNN layer can be completed synchronously with the pixel exposure, generating the feature map in a row-by-row scheme as well.
During the processing, the quantization bit-width varies with each operation. The specific bit-widths following each operation are listed in Supplementary Fig.8. The raw image is quantized as 10-bit unsigned integers. After processing through the two-layer electrical network, the resulting feature map is quantized as 15-bit signed integers.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "In “NEOSTI: A Neuromorphic Electronic-Opto Spatial-Temporal Hybrid Image Sensor”, Tianyi Liu and co-authors present a\nstudy on an all-in-one, fully integrated vision system that combines optical pre-processing, in-sensor and near-sensor\nprocessing, and on-chip binary neural network (BNN) computation. The system is capable of operating under both natural\nand artificial lighting conditions without the need for coherent light sources. This paper is innovative and is of interest to\npotential readers working in the field of machine vision and edge computing. There are some comments on this work listed\nas follows:\nQ1: The optical mask in front of the chip systems is not new. It is better for the author to discuss the feasibility of\nimplementing reconfigurable or programmable optical masks, or alternative approaches to enhance flexibility for different\ntasks or datasets.\nQ2: The manuscript introduces the use of pulse width modulated (PWM) to encode light intensity, which is then processed\nby on-chip units for arithmetic operations and subsequent processing by a binary neural network (BNN) in a pipelined\nfashion without the requirement of ADC. The computing process is all realized in digital domain. While this is an interesting\napproach, since current in-memory computing strategies usually conduct the matrix multiplication operation in analog\ndomain by converting the signal into an output voltage intensity, which is applicable to ANN with multi-valued weight\nparameters. The authors could explained more about whether the benefits of pulse width coding are unique to BNNs or\napplicable to a broader class of models.\nQ3: The system relies on a combination of in-sensor processing (PIS) and near-sensor binary neural network (PNS) to\nhandle dynamic scenes. The speed advantage of BNNs in conducting dynamic tasks is not sufficiently discussed or\ndemonstrated. A more detailed analysis of these points would significantly strengthen the paper and improve its clarity, such\nas the advantages of NEOSTI compares to established dynamic processing approaches such as in-sensor or in-memory\nrecurrent Neural Networks (RNNs)1 or dynamic vision sensors (DVS)2. Also, is the time resolution of this NEOSTI will be\nlimited by the row-by-row exposure mode?.\nQ3: While the manuscript describes the overall system architecture and the role of the on-chip BNN, it lacks a detailed\naccount of the BNN hardware implementation. For completeness and to facilitate reproducibility, I recommend that the\nauthors provide more information regarding the hardware structure of the BNN module.\nI recommend that the manuscript be considered for publication after the authors have adequately addressed the above\ncomments through further discussion and revisions.\n1. Chen, J. et al. Optoelectronic graded neurons for bioinspired in-sensor motion perception. Nature Nanotechnology 18,\n882-888 (2023).\n2. Zhou, Y. et al. Computational event-driven vision sensors for in-sensor spiking neural networks. Nature Electronics 6,\n870-878 (2023).", "reviewer_2": "The authors propose a new Neuromorphic Electronic–Opto Spatial–Temporal Imager to enhance data processing efficiency\nand reduce the overall sensor volume.\nOverall, I find that more information about the image sensor array performance and features is needed. Below, I provide\nsome comments to guide the authors in improving the article’s content:\n1. Include a table of specific image sensor features.\nThe following key parameters appear to be missing and should be provided:\n• Fill factor\n• Sensor dynamic range (the authors claim both indoor and outdoor operation)\n• Fixed-pattern noise\n• Pixel power consumption\n• Spatial contrast sensitivity (when computed from processed pixel outputs)\n• Sensor latency versus illumination\n• Any other relevant specifications\n2. Clarifications on pixel and image sensor operation/implementation:\n• What is the range of selectable integration times for system operation?\n• At the pixel level, what is the purpose of the cascode transistor, and what is its bias value?\n• Discuss how dark current limits the sensor’s dynamic range.\n• The design requires a time-to-digital converter (TDC) to assign intensity levels based on pixel time response. How many\nbits are used to implement the TDC? What is the range of pixel response times it can handle? How does TDC resolution\naffect pixel dynamic range?\n• Provide the system clock frequency.\n• One of the claimed advantages of the proposed architecture is low power consumption. Please elaborate on the trade-off\nbetween power consumption and frame rate.", "reviewer_3": "The approach presented in this paper is certainly interesting from a scientific perspective. Splitting operations and layers of a\nCNN-type network for image data processing (image recognition/classification tasks) between different physical elements\nlike static optical masks, focal-plane/pixel-level analog signal processing and binary neural network layers operated in the\ndigital domain, is an interesting and novel concept worth to be published. The presentation of innovation is sufficiently well\ndone, the idea, concept and methodology are well described and illustrated. Evaluation results are presented. However,\nseveral points need to be addressed, in particular the discussion section is very weak and misses out on several key\nquestions.\nIn my view, at least these points below need to be addressed before possible publication of this material:\n(1) Some claims are bold or imprecise:\n\"the smallest all-in-one eye size fully integrated vision system\", The presented vision system, with chip dimensions of 5mm x\n5mm is certainly not \"the smallest\" ever proposed. Just as one of many examples, Synsens's Speck vision System-on-Chip\ncombines a 128x128 (event-based) pixel array with a 9-layer spiking CNN processor on 30mm2 silicon.\n(https://arxiv.org/abs/2304.06793)\n\"Near-human performance in five static and dynamic visual processing tasks\" — \"near human\" is a very bold statement,\nparticularly given the toy-problem like nature of the experiments/data sets used for validation of the NEOSTI approach, see\nbelow (4).\n\"current image sensor technology falls short in meeting the requirement of embodied intelligence\", a broad statement without\nany clear meaning. Later, \"embodied intelligence\" is vaguely defined as possibly involving e.g. \"intelligent robots, fully\nautonomous driving, drones...\"\n\"NEOSTI utilizes a pulse width modulated (PWM) based non-linear readout pixel structure, realizing a retina-like 3-layered\nprocessing structure to generate pulse signals analogous to neural spikes\" — NEOSTI is far from being \"retina-like\" or from\ngenerating anything resembling \"neural spikes\". This is per se not a problem but does not need to be claimed this way.\n(2) More issues not related to technical content that need attention:\nSeveral language-related flaws: E.g. \"NEOSTI is ... system that can operable ...\", \"lenless\" — proof reading required\nFactually imprecise statements: \"consumer image sensor can only achieve a DR around 140dB\" — no, consumer imagers\ntypically reach 60-70dB, only specialized automotive multi-exposure image sensors go up to DR levels around/beyond\n140dB\"\nCitations and benchmarking: NEOSTI claims to be \"neuromorphic\" but does not compare itself to any other neuromorphic\nvision solutions (e.g. event sensor based) that can solve similar problems; e.g. Synsense's Speck, combining event pixel\narray and SNN processor on a single chip (https://arxiv.org/abs/2304.06793) nor with SoA industry Vision System-on-chip\nsolutions like Sony's IMX500, targeting similar use cases/applications (https://www.aitrios.sony-semicon.com/edge-ai-\ndevices/imx500).\nPulse modulation (including PWM) is a well known approach in vision sensing — prior work should be cited (e.g.\nhttps://ieeexplore.ieee.org/document/5701724).\n(3) Missing discussion on Static mask limitations:\nLooking at the early feature extraction capabilities of NEOSTI, the static optical mask functions as the first convolutional\nlayer of a 3-layer convolutional neural network. The features it extracts are likely some fundamentally low-level spatial\nfeatures, determined by the patterns etched onto the mask itself. The mask performs a 5x5 sliding convolution on the\nincoming light. I assume that this process is analogous to the initial feature extraction step in a typical CNN, which often\nidentifies basic elements like edges, corners, and textures. In the present case, the weights, implemented as the patterns on\nthe mask, are binary (transparent or opaque) and are derived from the pre-trained weights of a Binary Neural Network\n(BNN). This means the mask is optimized for the types of features relevant to its original training data. For a different use\ncase requiring the detection of fundamentally different low-level features, a new mask would need to be fabricated. So, while\nthis optical pre-processing is highly efficient in terms of power and speed, a static, fabricated mask does present limitations\nregarding flexibility and the range of addressable use cases.\nIn my view, a deeper discussion on this subject is missing and should include:\nLack of Adaptability: The mask itself cannot adapt to changing conditions or tasks at run-time. The system's adaptability\nrelies on the programmable on-chip electronic circuits (PIS and PNS) that process the feature maps created by the static\nmask.\nPotential for Sub-Optimal Performance on Unseen Tasks: The fixed front-end filtering might be sub-optimal for tasks that\nwere not considered during the initial training and mask design. The success of the system on a new task depends on\nwhether the features extracted by the fixed mask are still relevant and sufficiently informative for the subsequent electronic\nlayers to work with.\nThe paper suggests some kind of modular approach where optical components could be interchanged via a standardized\ninterface, allowing for task versatility. This implies that for different applications, one could swap out the mask for another\none with more suitable pre-trained patterns. How could such an approach be implemented in a real-world / commercial\nscenario? I believe some in-depth discussion on these points would significantly increase the quality and impact of the\npaper.\n(4) Insufficient discussion on Adaptability and Scalability:\nIn my understanding, the presented implementation as described seems to have a limited real-world applicability. The\nconcept (in the form of a fabricated prototype) is validated only on \"toy problems\", using very low-resolution input data with\nonly simple high-contrast grey scale or BW features. (MNIST, UnityEyes, QuickDraw, Fashion MNIST, Weizmann), and the\nNEOSTI approach, in its current form, would face considerable challenges when applied to more complex, higher-resolution,\nand full grayscale natural scenes. The limitations I think is the binary optical mask.\nHigher-Resolution Features: Scaling to a higher resolution would present a physical challenge due to light diffraction. The\nminimum feature size that can be created on the optical mask is governed by the wavelength of light and the distance\nbetween the mask and the sensor. Currently, the feature size is set to 40µm. To increase the image resolution, the pixels on\nthe sensor would need to be made smaller and more numerous. Consequently, the features on the mask would also need to\nshrink to match this new, finer pixel grid. However, as the feature size shrinks, the diffraction of light becomes a more\nsignificant issue, causing blurring that would defeat the purpose of the high-resolution sensor, i.e. the laws of physics\nimpose a practical floor on how small the mask's features can be, thereby limiting the maximum achievable resolution before\ndiffraction renders the optical computation useless. Please comment.\nGrayscale Information: The system's handling of grayscale (and color) is limited by two different components: the optical\nmask and the sensor design, i.e. what the sensor can detect and what the mask can compute. The sensor's output is\nquantized to 10-bit unsigned integers, allowing it to represent a range of intensity levels, not just black and white. In my\nunderstanding, the significant bottleneck is that the optical mask is binary. Therefore, the convolution performed in the\noptical domain is not a true multiplication-and-accumulation of numerical values. The system cannot perform optical\nconvolutions with grayscale kernels (e.g., a Sobel filter) and effectively binarizes the optical information before it reaches the\nsensor. To handle complex grayscale data optically, the mask would need to be replaced by a more complex component\n(kind of spatial light modulator), which could apply non-binary (grayscale) convolutional kernels. However, this would add\nsignificant complexity, and cost, defeating the primary advantage of the NEOSTI approach.\nPower Consumption: It is not fully clear to me what is included in the power consumption numbers of Table 1 and how\npower consumption scales with e.g. increasing resolution. Some discussion on this point would be helpful as power\nconsumption seems to be a major KPI of this approach.\nThe paper does hint at future possibilities for overcoming these limitations, suggesting that scalability could be improved by:\nIntegrating lenses coated with specific patterns instead of a simple mask.\nExpanding the number of optical layers by incorporating multiple, varied optical elements.\nLeveraging advancements in material science, such as quantum dots, to create all-optical nonlinear responses, which could\nallow for more complex computations beyond simple binary filtering.\nIn my view, a more in-depth discussion on how those extensions could address above limitations would improve the paper\nsignificantly."}, "author_response": "Responses to the reviewer's comments on the manuscript\nentitled \"NEOSTI - A Neuromorphic Electronic-Opto\nSpatial-Temporal Hybrid Image Sensor\", by T. Liu, Z.\nHuang, X. Wang, et. al\nWe would like to thank the reviewers for the constructive comments and feedback,\nwhich have helped us to substantially improve our manuscript. We have made\nconsiderable changes to the paper taking into consideration the reviewer’s suggestions.\nThe changes introduced are summarized below. We addressed all the reviewer’s\ncomments point by point after the following summary:\n1. We adjusted the structure of the original manuscript into two parts: a revised\nmain manuscript and a Supplementary information file, with careful formatting\nadjustments and revisions made to ensure full compliance with the Nature\nCommunications submission requirements.\n2. We added more detailed analysis into the Discussion and Method section of the\nmanuscript and the Supplementary information according to reviewers’\ncomments to enhance the clarity and completeness of the work.\n3. We added two additional benchmarking works on neuromorphic and industry\non-chip vision systems in the revised manuscript.\n4. We cited several new references to improve the completeness of the manuscript.\n5. We revised several statements and polished the language thoroughly to improve\nthe quality of the work.\nThe changes in the revised manuscript are highlighted in red color.\n1.1 Reviewer 1 states:\nThe optical mask in front of the chip systems is not new. It is better for the author to\ndiscuss the feasibility of implementing reconfigurable or programmable optical masks,\nor alternative approaches to enhance flexibility for different tasks or datasets.\nOur response:\nWe sincerely appreciate the reviewer’s suggestion.\nIn this experiment, a passive static mask was chosen as the optical encoding device,\nwhere the amplitude mask plate controls the transmittance of each unit via\nphotolithography coating technology. The PNS/PIS systems feature programmable\nunits for parameter adjustment, with the NEOSTI being partially reconfigurable for\ndifferent tasks or datasets.\nWhile some approaches use active programmable devices, such as spatial light\nmodulators (SLMs) or other liquid crystal devices, to modulate the phase or amplitude\nof incident light, others, like Zhou et al.1 have proposed reconfigurable diffractive\nprocessing units that can handle large-scale neural networks, reaching up to the million-\nneuron level. These systems incorporate a digital micromirror device (DMD), an SLM,\nand a complementary metal-oxide-semiconductor (CMOS) sensor. Similarly, Huo et\nal.2 developed a programmable optical neural network employing two layers of liquid\ncrystal modulators, featuring sparsely connected neurons. However, the use of active\nmodulators introduces additional power consumption, typically on the order of several\nwatts. Meanwhile, the input frame rate of the SLM (for example, Holoeye LC2012) is\nlimited to 60 Hz, which constrains the overall processing speed of the system and makes\nit unsuitable for high-speed scenarios. Hu et al.3 proposed the FourierCam architecture,\ndesigned to capture the pixel-wise temporal spectrum of dynamic scenes through\noptical coding using a digital micromirror device operating at 2 kHz. Although this\napproach enables high-speed modulation, the inclusion of high-speed components such\nas DMDs results in a bulky system that requires precise optical alignment, thereby\nhindering its deployment on mobile platforms.\nIn contrast, fixed and passive masks offer the advantage of performing computation\nwith zero power consumption. In this work, the optical mask modulates the amplitude\nof incident light. Li et al.4 further proposed a wavelength-multiplexed diffractive\noptical network composed of a series of pretrained metasurfaces. In this design, input\ncomputing with a reconfigurable diffractive processing unit. Nat. Photon. 15, 367–373 (2021).\nneuron array and functional learning. Nat. Commun. 14, 2535 (2023)\nPhotonics Research, 9(5): 701-713 (2021).\ndiffractive optical network. Advanced Photonics, 5(1): 016003-016003 (2023).\n2\ndata are encoded into distinct wavelengths, and different wavelengths passing through\nthe broadband diffractive optical network yield distinct modulation and computation\nresults. Through this mechanism, a large set of linear transformations can be executed\nindependently within the same diffractive network at different illumination\nwavelengths, enabling high-throughput intelligent machine-vision systems and\nhyperspectral optical processors. Although the metasurfaces in such systems are non-\nreconfigurable or non-programmable, different tasks can be assigned to distinct\nwavelength channels, each corresponding to a specific predesigned computational\nfunction. This approach effectively expands the modulation dimensionality within a\nfixed structure, allowing multiple tasks to be integrated into a single passive platform.\nTherefore, even fixed and passive optical masks possess the potential to support multi-\ntask processing through appropriate wavelength-domain encoding and design.\n[editorial note: panel a and panel c are redacted as they contain third-party material]\nFig. 1 Schematic diagram of an optical computing system based on different modulation devices.\n(a) Reconfigurable diffractive processing units with an SLM1. (b) Programmable optical neural\nnetwork with two liquid crystal modulators2. (c) FourierCam with a DMD3.\nMeanwhile, to address the trade-off between reconfigurability and power consumption,\nthe introduction of novel materials is necessary. Transparent non-emissive display\ndevices, such as electrochromic (EC) display5, maintain optical contrast and readability\nwhile offering advantages such as wide viewing angles, flexibility, and low power\nconsumption. The development of new high-performance electrochromic materials\nholds potential as the core modulation unit, with the capability to overcome the power\nlimitations inherent in traditional technologies. These materials can be fabricated into\nhigh-precision reconfigurable optical masks through an in-situ lithography and pixel-\ndefining layer integration process, reducing driving energy consumption by more than\nthree orders of magnitude compared to conventional liquid crystal devices. Moreover,\nit provides high modulation precision for multi-bits and distinct modulation effects for\ndifferent wavelengths, potentially expanding the modulation dimensions.\nWe have added the discussion in Discussion Section of the revised manuscript.\nRevised text:\nDiscussion (lines 334-350)\nusing the pixel confinement effect. Device. 1,6 (2023).\nthe PNS/PIS modules contain tunable parameters. Consequently, NEOSTI exhibits\npartial reconfigurability and demonstrates a certain degree of generalization across\ndifferent tasks. To achieve optical reconfigurability, employing spatial light modulators\n(SLMs) or similar modulation devices inevitably introduces power consumption and\nsystem complexity, resulting in a trade-off between efficiency and performance. With\nthe development of novel materials, such as electrochromic (EC) materials5 used in\ntransparent and non-emissive display devices, which offers a promising pathway. The\nadvancement of high-performance electrochromic materials holds great potential for\nserving as core optical modulation units, capable of overcoming the inherent power\nconstraints of conventional modulation technologies. Through in-situ lithography\ncombined with pixel-defining layer integration, these materials can be fabricated into\nhigh-precision, reconfigurable optical masks. Such an approach reduces driving energy\nconsumption by more than three orders of magnitude compared to traditional liquid\ncrystal devices. Moreover, it enables high-precision, multi-bit optical modulation and\ndistinct wavelength-dependent modulation effects, thereby expanding the\ndimensionality of the optical modulation dimension.\n1.2 Reviewer 1 states:\nThe manuscript introduces the use of pulse width modulated (PWM) to encode light\nintensity, which is then processed by on-chip units for arithmetic operations and\nsubsequent processing by a binary neural network (BNN) in a pipelined fashion without\nthe requirement of ADC. The computing process is all realized in digital domain. While\nthis is an interesting approach, since current in-memory computing strategies usually\nconduct the matrix multiplication operation in analog domain by converting the signal\ninto an output voltage intensity, which is applicable to ANN with multi-valued weight\nparameters. The authors could explained more about whether the benefits of pulse width\ncoding are unique to BNNs or applicable to a broader class of models.\nOur response:\nWe sincerely appreciate the reviewer’s insightful comment.\nAnalog-domain computation methods typically rely on charge accumulation in\ncapacitors. Although such approaches allow for multi-bit weight configuration, the\ncalculation results are inevitably affected by analog noise, device mismatch, and\ncapacitor nonlinearity in the computing circuits, limiting the accuracy and reliability.\nAnd the large capacitor arrays for analog-domain processing poses a conflict with the\narea and power constraints of edge devices. In contrast, digital-domain computation is\ninherently more robust, as it eliminates the influence of cumulative analog errors and\nthus improve computational accuracy. Moreover, digital-domain computation allows\nfor more flexible architectural design and can readilytake advantage of the performance,\nefficiency, and scalability gains brought by process technology improvements.\nIn our proposed PE, pulses generated by adjacent PWM pixels can be processed through\n4\nsimple logic operations to control the readout counters, directly yielding results adapted\nto BNN convolution, which reduces the power consumption of both pixel readout and\ncomputation circuits. The PE design in this work is optimized specifically for BNN\nmodels. Nevertheless, the concept of performing computation during the PWM pixel\nreadout process holds potential for further exploration and development, and it may be\nextended to support a broader range of models and applications, such as edge detection\nand image filtering, which similarly rely on spatially local pixel processing.\nWe have added the content to the Discussion Section of the revised manuscript.\nRevised text:\nDiscussion (lines 305-317)\nNEOSTI applies the Pulse Width Modulation pixel structure, which enables direct\nconversion of photoelectric signals into pulse-width representations, allowing the signal\nto be processed directly in the digital domain. Compared with analog-domain\ncomputation based on capacitor-array integration, digital-domain computation is\ninherently more robust, as it eliminates the influence of cumulative analog errors caused\nby noise, device mismatch, and capacitor nonlinearity, ensuring the accuracy and\nreliability of the processing results. Moreover, digital-domain computation allows for\nmore flexible architectural design. In NEOSTI, the Processing Element is elegantly\ndesigned to couple the PWM readout process with the BNN computing framework,\nachieving efficient in-pixel computation. The concept of integrating on-chip\ncomputation during the PWM pixel readout process holds great potential for further\nexploration and development to support a broader range of models and applications,\nsuch as edge detection and image filtering, which similarly rely on spatially local pixel\nprocessing.\n1.3 Reviewer 1 states:\nThe system relies on a combination of in-sensor processing (PIS) and near-sensor\nbinary neural network (PNS) to handle dynamic scenes. The speed advantage of BNNs\nin conducting dynamic tasks is not sufficiently discussed or demonstrated. A more\ndetailed analysis of these points would significantly strengthen the paper and improve\nits clarity, such as the advantages of NEOSTI compares to established dynamic\nprocessing approaches such as in-sensor or in-memory recurrent Neural Networks\n(RNNs) or dynamic vision sensors (DVS). Also, is the time resolution of this NEOSTI\nwill be limited by the row-by-row exposure mode?\nOur response:\nWe sincerely appreciate the reviewer’s insightful comment.\nThe BNN model integrated in NEOSTI provides general, real-time, and efficient feature\nextraction and compression capabilities, directly extracting compressed temporal\nfeature map sequences from input frames. The input image from the sensor has a\n5\nresolution of 64×64 with 10-bit precision in calculation mode, resulting in a data size\nof 40,960 bits, while the output of on-chip BNN consists of 8 groups of 8×8 feature\nmaps quantized to 15 bits, with a total size of only 7,680 bits. As a result, BNN-based\nfeature extraction effectively compresses 81.3% of the input data scale, significantly\nreducing the computational and storage overhead of frame feature sequences while\npreserving intra-frame spatial features, which achieves a balance between spatial and\ntemporal feature processing abilities. In contrast, RNNs are good at sequence\nprocessing but not well-suited for handling two-dimensional image data which has\nstrong local spatial correlations; DVS performs well in capturing rapidly changing or\nmoving scenes, but the event-stream output loses the global spatial features of each\nframe. Both of them focus much on temporal feature analysis, lacking sufficient\ncapability to capture spatial features within frames. We have added the discussion of\ndynamic processing in the revised manuscript.\nThe temporal resolution of NEOSTI is not constrained by the row-by-row exposure\nmode, since the subsequent PIS and PNS convolution operations also require a row-by-\nrow sliding kernel for calculation. The global exposure mode would not improve\nprocessing speed; instead, it would increase the overhead of readout and storage. In\nNEOSTI, the exposure and subsequent BNN calculations are executed in a low-\noverhead pipeline structure, ensuring that the acquired image can be processed row-by-\nrow in real time.\nRevised text:\nResults (lines 270-273)\nThe BNN-based feature extraction model serves as a general, real-time, and efficient\ndata compressor that effectively reduces the input data volume by 81.3%, significantly\nreducing the computational and storage overhead of feature sequences while preserving\nintra-frame spatial features.\nDiscussion (lines 290-294)\nCompared with other neuromorphic designs such as dynamic vision sensors (DVS),\nwhich perform well in capturing rapidly changing or moving scenes but lack sufficient\ncapability to handle static scenes, NEOSTI provides more general feature extraction\nand compression capabilities, making it suitable for a wider range of scenarios\nencompassing both dynamic and static scenes.\n1.4 Reviewer 1 states:\nWhile the manuscript describes the overall system architecture and the role of the on-\nchip BNN, it lacks a detailed account of the BNN hardware implementation. For\ncompleteness and to facilitate reproducibility, I recommend that the authors provide\nmore information regarding the hardware structure of the BNN module.\nOur response:\nThe on-chip BNN hardware architecture is adapted to the row-by-row exposure scheme,\nwith a dedicated calculation module deployed after the readout circuit of each column\nin the pixel array. The hardware structure of a basic convolution block is illustrated in\nthe figure below, in which the circuit design corresponds exactly to the four-stage\npipeline.\nFig. 2 (Supplementary Fig. 4b) The hardware structure of BNN convolution calculation circuits.\nDuring the first clock cycle after the pixel is exposed, the PE array processes the data. In the\nsecond clock cycle, the delayed data is used to compute the convolution feature map. The\nnonlinear function employs the ReLU function to minimize hardware overhead. In the subsequent\ntwo cycles, pairwise maximum operations within the row and maximum operations between rows are\nexecuted to complete a 2x2 maximum pooling.\nCycle 1: 64 PEs compute and generate the 2×1 convolution outputs, PEOUT0(m) and\nPEOUT 1(m), for the mth row of pixels, where\nRES0(𝑚,0) RES1(𝑚,0)\nRES0(𝑚,1) RES1(𝑚,1)\n⎛ ⎞ ⎛ ⎞\nPEOUT0(𝑚) = ⋮ , PEOUT1(𝑚) = ⋮\n⎜ ⎟ ⎜ ⎟\nRES0(𝑚,62) RES1(𝑚,62)\nRES0 and RES1 are th⎝e RcoEnSv0o(l𝑚ut,i6on3 )r⎠esults of the two adjacen⎝tR pEixSe1l( v𝑚a,lu6e3s) ⎠with the\nfirst and second rows of the convolution kernel , respectively. For the nth\n𝑘0 𝑘1\ncolumn of PE, RES0 and RES1 are calculated as\nK = [ ]\n𝑘2 𝑘3\nRES0(𝑚,𝑛) = 𝑘0 ⋅𝐼𝑁(𝑛)+𝑘1 ⋅𝐼𝑁(𝑛+1)\nThe output result of P\nR\nE\nE\nO\nS\nU\n1(\nT\n𝑚\n0(\n,\nm\n𝑛)\n) w\n=\nil\n𝑘\nl 2b\n⋅\ne\n𝐼\nc\n𝑁\nac\n(\nh\n𝑛\ne\n)\nd\n+\nto\n𝑘\nC3y\n⋅\nc\n𝐼\nl\n𝑁\ne\n(\n2\n𝑛\n.\n+1)\nCycle 2: The cached result PEOUT0(m) is summed with PEOUT1(m + 1) to generate\nthe 2×2 convolution results for the mth and (m+1)th row. The convolution results then\n7\npass through the ReLU activation function module, whose hardware structure is a 2-to-\n1 multiplexer controlled by the sign bit of the input (as shown in Fig.3): if the sign bit\nis 0 (indicating a positive value), the original data is passed through; if the sign bit\nis 1 (indicating a negative value), a value of zero is output. The outputs of convolution and\nReLU activation (denoted as CM(m)) are stored to Cycle 3.\nCM(𝑚) = ReLU(𝑃𝐸𝑂𝑈𝑇0(𝑚)+𝑃𝐸𝑂𝑈𝑇1(𝑚+1))\nFig. 3 (Supplementary Fig. 4c)Hardware implementation of ReLU operation. It is implemented as\na 2-to-1 multiplexer.\nCycle 3: The 2×1 maximum pooling, which is implemented in hardware using a\ncomparator and a 2-to-1 multiplexer (as shown in Fig. 4), is performed within the row\ndirection. And the output result MP(m) is cached to Cycle 4.\n𝑛\nMP(𝑚, ) = Max(CM(𝑚,𝑛),CM(𝑚,𝑛+1))\nimplemented as a comparator and a 2-to-1 multiplexer.\nCycle 4: Another 2×1 maximum pooling module is applied to the previously cached\nresult and the newly generated result from the last stage,\nproducing the final 2×2 maximum pooling result.\nMP(𝑚) MP(𝑚+1)\n𝑚 𝑛 𝑛 𝑛\nOUT( , ) = Max(MP(𝑚, ),MP(𝑚+1, ))\n2 2 2 2\nWith the pipelined calculating hardware structure, the processing results of one standard\nBNN layer can be completed synchronously with the pixel exposure, generating the\nfeature map in a row-by-row scheme as well.\nMethods Section of the revised manuscript.\nRevised text:\nMethods (lines 559-598)\nThe hardware structure of the convolution block is illustrated in Supplementary Fig.4b,\nwhich is implemented directly after the PEs of each column. The specific process of the\n4-stage pipeline is described as follows:\nCycle 1: 64 PEs compute and generate the 2×1 convolution outputs, PEOUT0(m) and\nPEOUT 1(m), for the mth row of pixels, where,\nRES0(𝑚,0) RES1(𝑚,0)\nRES0(𝑚,1) RES1(𝑚,1)\n⎛ ⎞ ⎛ ⎞\nPEOUT0(𝑚) = ⋮ , PEOUT1(𝑚) = ⋮\n⎜ ⎟ ⎜ ⎟\nRES0(𝑚,62) RES1(𝑚,62)\nRES0 and RES1 are th⎝e RcoEnSv0o(l𝑚ut,i6on3 )r⎠esults of the two adjacen⎝tR pEixSe1l( v𝑚a,lu6e3s) ⎠with the\nfirst and second rows of the convolution kernel , respectively. For the nth\n𝑘0 𝑘1\ncolumn of PE, RES0 and RES1 are calculated as\nK = [ ]\n𝑘2 𝑘3\nRES0(𝑚,𝑛) = 𝑘0 ⋅𝐼𝑁(𝑛)+𝑘1 ⋅𝐼𝑁(𝑛+1)\nThe output result of P\nR\nE\nE\nO\nS\nU\n1(\nT\n𝑚\n0(\n,\nm\n𝑛)\n) w\n=\nil\n𝑘\nl 2b\n⋅\ne\n𝐼\nc\n𝑁\nac\n(\nh\n𝑛\ne\n)\nd\n+\nto\n𝑘\nC3y\n⋅\nc\n𝐼\nl\n𝑁\ne\n(\n2\n𝑛\n.\n+1)\nCycle 2: The cached result PEOUT0(m) is summed with PEOUT1(m+1) to generate\nthe 2×2 convolution results for the mth and (m+1)th row. The convolution results then\npass through the ReLU activation function module, whose hardware structure is a 2-to-\n1 multiplexer controlled by the sign bit of the input (Supplementary Fig. 4c): if the sign\nbit is 0 (indicating a positive value), the original data is passed through; if the sign\nbit is 1 (indicating a negative value), a value of zero is output. The outputs of convolution and\nReLU activation(denoted as CM(m)) are stored to Cycle 3.\nCycle 3: The 2×1 maximum pooling, which is implemented in hardware using a\nCM(𝑚) = ReLU(𝑃𝐸𝑂𝑈𝑇0(𝑚)+𝑃𝐸𝑂𝑈𝑇1(𝑚+1))\ncomparator and a 2-to-1 multiplexer (Supplementary Fig.4d), is performed within the\nrow direction. And the output result MP(m) is cached to Cycle 4.\n𝑛\nMP(𝑚, ) = Max(CM(𝑚,𝑛),CM(𝑚,𝑛+1))\nCycle 4: Another 2×1 maximum pooling module is applied to the previously cached\nresult and the newly generated result from the last stage,\nproducing the final 2×2 maximum pooling result.\nMP(𝑚) MP(𝑚+1)\n9\n𝑚 𝑛 𝑛 𝑛\nOUT( , ) = Max(MP(𝑚, ),MP(𝑚+1, ))\n2 2 2 2\nWith the pipelined calculating hardware structure, the processing results of one standard\nBNN layer can be completed synchronously with the pixel exposure, generating the\nfeature map in a row-by-row scheme as well.\nSupplementary Fig.4b\nSupplementary Fig. 4b The hardware structure of BNN convolution calculation circuits. During\nthe first clock cycle after the pixel is exposed, the PE array processes the data. In the second clock\ncycle, the delayed data is used to compute the convolution feature map. The nonlinear function\nemploys the ReLU function to minimize hardware overhead. In the subsequent two cycles,\npairwise maximum operations within the row and maximum operations between rows are\nexecuted to complete a 2x2 maximum pooling.\n2.1 Reviewer 2 states:\nInclude a table of specific image sensor features.\nThe following key parameters appear to be missing and should be provided:\n• Fill factor\n• Sensor dynamic range (the authors claim both indoor and outdoor operation)\n• Fixed-pattern noise\n• Pixel power consumption\n• Spatial contrast sensitivity (when computed from processed pixel outputs)\n• Sensor latency versus illumination\n• Any other relevant specifications\nOur response:\nWe sincerely appreciate the reviewer’s insightful comment.\nThe specific features of our image sensor are summarized in the table below, which\nhave been added to Supplementary information as Supplementary Table 1. The spatial\ncontrast sensitivity and sensor latency versus illumination features are not applicable to\nour system, and detailed explanations are provided below.\nArea 25mm2\n20μm×20μm (Acquisition mode)\nPixel size\n40μm×40μm (Calculation mode)\n128×128 (Acquisition mode)\nArray size\n64×64 (Calculation mode)\nSystem clock frequency 1MHz\nFill factor 74%\nDynamic range 53.98dB (200~100,000 lux)\nFixed-pattern noise 0.78%\nPixel power consumption 96.37pW/pixel @ 1fps\nTable 1 (Supplementary Table 1) Summary table of chip specifications.\nThe following is an explanation of some image sensor features:\n Fill factor: The total area of a pixel is 400 μm2, and the area of the photodiode\nregion is 295.96 μm2. Fill factor = 295.96/400 = 74%.\n Dynamic range: The light-sensing range of NEOSTI’s pixel is measured to be\n200-100,000 lux, corresponding to a dynamic range of 20log(100000/200) =\n53.98dB. The photoresponse curve is shown in Fig.5.\npulse width modulated pixel structure. The relationship between photocurrent (light intensity) and\npulse width is nonlinear.\n Pixel power consumption: The pixel power consumption can be estimated from\nthe system power consumption and the size of the pixel array. At a frame rate of 1\nfps, the sensing power consumption of a 128×128 array is 1.579µW. Pixel power\nconsumption = 1.579µW/(128×128) = 96.37 pW/pixel.\n Spatial contrast sensitivity：NEOSTI employs a lensless image sensor design, so\nspatial contrast sensitivity is not applicable as in traditional optical systems.\n Sensor latency versus illumination：The latency of frame-based CMOS sensor\nis determined by exposure and readout circuits, and does not depend on\nillumination as event-based sensor or single-photon detector. In NEOSTI design,\nthe latency is directly equal to the frame exposure time due to the pipeline design.\nRevised text:\nMethods (lines 664-665)\nThe specific features of the image sensor is summarized in Supplementary Table 1.\nSupplementary Table 1\nSupplementary Table 1 Summary table of chip specifications.\n12\n2.2 Reviewer 2 states:\nClarifications on pixel and image sensor operation/implementation:\n(1) What is the range of selectable integration times for system operation?\nOur response:\nWe sincerely appreciate the reviewer’s insightful comments.\nIn our PWM-based pixel architecture, the selectable integration time is achieved\nthrough the configurable exposure delay time (the red part in Fig.6). The minimum\nintegration time is one row time (as shown in Fig.6a), where the exposure delay time is\nequal to 0 and the integration time corresponds to the counter working time. The\nmaximum integration time is equal to one frame time (as shown in Fig.6b), where the\nexposure delay time is set close to the frame time (approximately 1s for 1 fps). The\ncounter working time (the yellow part in Fig. 6) corresponds to the frequency and bit-\nwidth of readout counter. The minimum is determined by one clock period of the\ncounter (1μs), and the maximum is determined by the full counter range (1023μs for\nthe 10-bit counter). Since the pulse width of PWM pixel output reflects the light\nintensity, the actual integrate time is dynamically related to the illumination.\nWe have added the clarification of selectable integration time to Supplementary Note\n1.\nFig. 6 (Supplementary Fig.3) The timing diagrams of exposure control signals for different\nexposure times. a. Exposure control signals for one-row exposure time. b. Exposure control\n13\nsignals for one-frame exposure time.\nRevised text:\nSupplementary Note 1\nThe integration time (counter working time, the yellow part in Supplementary Fig.3) of\nthe counter is determined by both the clock period and the full-scale range. The\nminimum integration time is one counting cycle (1 μs), while the maximum is the full\ncounter range (1023μs for the 10-bit counter). Since the pulse width of PWM pixel\noutput reflects the light intensity, the actual integrate time is dynamically related to the\nillumination.\n2.3 Reviewer 2 states:\n(2) At the pixel level, what is the purpose of the cascode transistor, and what is its bias\nvalue?\nOur response:\nWe sincerely appreciate the reviewer’s insightful comments.\nThe cascode transistor M controls the on/off state of this current path during the reset\nR\nphase, which supports different requirements in dual-mode workflow: 1) eliminate\nnoise in acquisition mode, and 2) reduce power consumption in calculation mode.\nFig. 7 (Supplementary Fig.2a) The specific circuit diagram of each pixel circuit. The pixel circuit\nincludes the corresponding photodiode, current comparator, and row-shared bias circuit. Each 2x2\npixel unit is cascaded to form a super pixel in the calculation mode.\nIn acquisition mode, the transistor M is turned on during the reset phase to ensure that\nR\nthe reset PD voltage is essentially consistent with the voltage expression during\nexposure. The expressions of the reset PD voltage and the PD voltage when reversing\nare:\n𝑛𝑁 𝐼𝑃0\n𝑉𝑃𝐷,𝑅𝑆𝑇 = (𝑉𝐷𝐷 −𝑉𝐺,𝑅𝑆𝑇 −𝑉𝑡ℎ,𝑃)+𝑛𝑛𝑉𝑇ln ( )+𝑉𝑇𝐻,𝑁\n𝑛𝑃 𝐼𝑁0\n14\n𝑛𝑁 𝐼𝑃0\n𝑉𝑃𝐷,𝐼𝑁𝑉 = (𝑉𝐷𝐷 −𝑉𝐺,𝐼𝑁𝑉 −𝑉𝑡ℎ,𝑃)+𝑛𝑛𝑉𝑇ln ( )+𝑉𝑇𝐻,𝑁\nTherefore, all terms re 𝑛 l𝑃ated to the threshold voltage in 𝐼𝑁 0 can be eliminated as\nshown in Equation below, which effectively mitigating variations induced by the\nΔVPD\nfabrication process and suppressing column FPN.\n𝑛𝑛\nΔ𝑉𝑃𝐷 = 𝑉𝑃𝐷,𝑅𝑆𝑇 −𝑉𝑃𝐷,𝐼𝑁𝑉 = (𝑉𝐺,𝐼𝑁𝑉 −𝑉𝐺,𝑅𝑆𝑇)\nIn calculation mode, the transistor M\nR\nis turned 𝑛o𝑝ff to interrupt the flow of DC current\nfrom M to ground through M to achieve a low-power reset, which reduces the average\nP N\ncurrent in the reset phase to 0.17% of that in the on state, leading to a 31% reduction in\nthe average power consumption per row time. Although the threshold mismatch noise\nis not eliminated due to the reset PD voltage becomes VDD, experiments conducted in\nthe calculation mode show that a 1% pixel value deviation leads to only about a 0.2%\ndecrease in classification accuracy, demonstrating that the low-power reset has little\nimpact on processing results.\nTransistor M functions as a switch, so its bias value is digital, either VDD or 0.\nR\nWe have revised the Method Section of the manuscript to highlight the role of transistor\nM in controlling the reset mode.\nR\nRevised text:\nMethod (line 490-492)\nThis is achieved by adding a cascode transistor M to the typical PWM pixel structure\nR\n(as shown in Supplementary Fig.2a), which controls the on/off state of the current path\nduring the reset phase.\nMethod (line 509-511)\nIn the calculation mode, the transistor M is turned off to interrupt the flow of DC\nR\ncurrent from M to ground through M to achieve a low-power reset, which reduces the\nP N\naverage current in the reset phase to 0.17% of that in the on state, leading to a 31%\nreduction in the average power consumption per row time.\n2.4 Reviewer 2 states:\n(3) Discuss how dark current limits the sensor’s dynamic range.\nOur response:\nWe sincerely appreciate the reviewer’s insightful comments.\nIn PWM pixels, the integration of photocurrent on the capacitor determines the timing\nof the voltage crossing threshold, which is then converted into a pulse width signal.\nHowever, in the absence of illumination, the presence of dark current still introduces a\n15\nleakage path, causing charge to accumulate slowly over time. As a result, even under\ndark conditions, the pixel voltage will eventually reach the threshold and trigger a false\ninversion in the PWM output, which is dominated by the dark current instead of\nphotocurrent. This phenomenon limits the effective dynamic range on the low-\nillumination side. In our implementation, when the light intensity is 200 lux or lower,\nthe effect of dark current leads to a fixed digital output code of 8, indicating the lower\nbound of the dynamic range.\nWe have added a discussion on how dark current limits the sensor’s dynamic range to\nthe Supplementary Note 3.\nRevised text:\nSupplementary Note 3\nThe dynamic range of NEOSTI's PWM pixel is determined by the dark current I and\ndark\nthe resolution of the counters in PE module.\nIn PWM pixels, the integration of photocurrent on the capacitor determines the timing\nof the voltage crossing threshold, which is then converted into a pulse width signal.\nHowever, in the absence of illumination, the presence of dark current I still\ndark\nintroduces a leakage path, causing charge to accumulate slowly over time. As a result,\neven under dark conditions, the pixel voltage will eventually reach the threshold and\ntrigger a false inversion in the PWM output, which is dominated by the dark current\ninstead of photocurrent. This phenomenon limits the effective dynamic range on the\nlow-illumination side. In our implementation, when the light intensity is 200 lux or\nlower, the effect of dark current leads to a fixed digital output code of 8, indicating the\nlower bound of the dynamic range.\n2.5 Reviewer 2 states:\n(4) The design requires a time-to-digital converter (TDC) to assign intensity levels\nbased on pixel time response. How many bits are used to implement the TDC? What is\nthe range of pixel response times it can handle? How does TDC resolution affect pixel\ndynamic range?\nOur response:\nWe sincerely appreciate the reviewer’s insightful comments.\nThe quantization of counters in PE (serve as a TDC) is 10 bits. And the quantization\nbit-width of each stage in the BNN calculation pipeline are described in Method Section\n(lines 598-601) and in Supplementary Fig.8.\nthe PIS and PNS modules. During the processing, the quantization bit-width varies with each\noperation.\nThe pixel response time that can be handled depends on the available frame rate and\nsystem clock frequency. The maximum value is 7.8ms, corresponding to the maximum\nresponse time for row readout at 1fps; and the minimum value is 1 μs, corresponding\nto one clock period of the counter.\nIn PWM pixels, light intensity is encoded into pulse width and digitized by the counter\n(TDC). At high illumination, the pulse width becomes very short. Once the width is\nbelow one period of the counter, even higher light intensity levels are quantized to the\nsame digital code. This saturation effect limits the dynamic range on the high-\nillumination side. In our implementation, the upper bound of the dynamic range is\n100000 lux.\nWe have added a discussion on how TDC resolution affect pixel dynamic range sub-\nsection to the Supplementary Note 3.\nRevised text:\nSupplementary Note 3\nIn PWM pixels, light intensity is encoded into pulse width and digitized by the counter\n(serve as a TDC). At high illumination, the pulse width becomes very short. Once the\nwidth is below one period of the counter, even higher light intensity levels are quantized\nto the same digital code. This saturation effect limits the dynamic range on the high-\nillumination side. In our implementation, the upper bound of the dynamic range is\n100000 lux.\n2.6 Reviewer 2 states:\n(5) Provide the system clock frequency.\nOur response:\nWe sincerely appreciate the reviewer’s comments.\nThe frequency of system clock is 1MHz. We have included this information in the\n17\nnewly added Supplementary Table 1.\nRevised text:\nSupplementary Table 1.\nSupplementary Table 1 Summary table of chip specifications.\n2.7 Reviewer 2 states:\n(6) One of the claimed advantages of the proposed architecture is low power\nconsumption. Please elaborate on the trade-off between power consumption and frame\nrate.\nOur response:\nWe sincerely appreciate the reviewer’s insightful comments.\nThe power consumption of image sensors increases with higher frame rates.\nTheoretically, the power consumption of the pixel array and the digital circuits increases\napproximately linearly with the frame rate, while the bias circuit power remains almost\nunaffected. At low frame rates, as the data processing rate decreases, the digital\nprocessing power overhead can be further reduced by lowering the digital supply\nvoltage. NEOSTI supports a configurable frame rate range of 1-32 fps, in order to adapt\nto different application requirements for power consumption and frame rate. The\nmeasured power consumption results at 1 fps (low-power) and 32 fps (high-speed)\noperating modes are summarized in Table 2.\nLow-power (1 fps) High-speed (32 fps)\nAcquisition 1.579 μW 25.7 μW\nCalculation 2.351 μW 121.74 μW\nTable 2 (Supplementary Table 2) System power consumption at different work modes.\nWe have added the discussion on the impact of frame rate on power consumption in the\nDiscussion Section, and provided the power consumption measurement table in the\nSupplementary information.\nRevised text:\nAs for the impact of temporal resolution, the power consumption of the pixel array and\nthe digital circuits increases approximately linearly with the frame rate, while the bias\ncircuit power remains almost unaffected. At low frame rates, as the data processing rate\ndecreases, the processing power overhead can be further reduced by lowering the digital\nsupply voltage. NEOSTI supports a configurable frame rate range of 1-32 fps, in order\nto adapt to different application requirements for power consumption and frame rate.\nSupplementary Table 2\nSupplementary Table 2 System power consumption at different work modes.\n3.1 Reviewer 3 states:\nSome claims are bold or imprecise:\n(1) \"the smallest all-in-one eye size fully integrated vision system\", The presented\nvision system, with chip dimensions of 5mm x 5mm is certainly not \"the smallest\"\never proposed. Just as one of many examples, Synsens's Speck vision System-on-Chip\ncombines a 128x128 (event-based) pixel array with a 9-layer spiking CNN processor\non 30mm2 silicon. (https://arxiv.org/abs/2304.06793)\nOur response:\nWe sincerely appreciate the reviewer’s insightful comments.\nHere, we intended to emphasize that NEOSTI represents a minimally sized integrated\noptoelectronic hybrid sensing and processing system, rather than merely a single-\ndomain processing chip. Actually, the silicon area of our NEOSTI (25mm2) is smaller\nthan Synsens's Speck chip (30mm2). However, we fully agree that using an absolute\nqualifier could be inappropriate and potentially misleading, and have revised the\nstatement to the clearer and more accurate form: “… one of the smallest electronic-opto\nfully integrated, eye-sized vision systems”.\nRevised text:\nAbstract (lines 21-22)\nInspired by the biomimetics, we propose a Neuromorphic Electronic-Opto Spatial\nTemporal Imager (NEOSTI), one of the smallest electronic-opto fully integrated, eye-\nsized vision systems enabling acquisition and operation in typical indoor/outdoor non-\ncoherent environments, …\nIntroduction (lines 90-91)\nThe NEOSTI is one of the smallest electronic-opto fully integrated vision systems that\ncan operate in typical indoor and outdoor environments, …\n3.2 Reviewer 3 states:\n(2) \"Near-human performance in five static and dynamic visual processing tasks\" —\n\"near human\" is a very bold statement, particularly given the toy-problem like nature\nof the experiments/data sets used for validation of the NEOSTI approach, see below (4).\nOur response:\nWe sincerely appreciate the reviewer’s insightful comments.\nWe agree that the “near-human” description is too strong just based on the validation\nresults of 5 datasets. Our intention was to highlight that the proposed system, designed\n20\nin a manner analogous to biological visual processing, achieves competitive\nperformance and demonstrates strong generalization capability across multiple static\nand dynamic visual processing tasks. To avoid overstatement, we have revised the\ndescription in the manuscript to a more accurate and cautious form: “competitive\nperformance”. We believe this revision better reflects the scope and contribution of our\nwork without overstating its significance.\nRevised text:\nAbstract (lines 30-31)\nIt attains competitive near-human performance in several visual processing tasks.\nIntroduction (lines 82-83)\nThe NEOSTI attains competitive performance in five static and dynamic visual\nprocessing tasks.\nDiscussion (lines 286-287)\nNEOSTI demonstrates competitive performance across five static and dynamic visual\nprocessing tasks.\n3.3 Reviewer 3 states:\n(3) \"current image sensor technology falls short in meeting the requirement of\nembodied intelligence\", a broad statement without any clear meaning. Later, \"embodied\nintelligence\" is vaguely defined as possibly involving e.g. \"intelligent robots, fully\nautonomous driving, drones...\"\nOur response:\nWe sincerely appreciate the reviewer’s insightful comments.\nWe agree that our original statement about “current image sensor technology falling\nshort in meeting the requirement of embodied intelligence” was overly broad and\nlacked clarity. Our intention was to point out the limitations of existing image sensors\nin supporting real-time, low-power edge processing that are crucial for embodied\nintelligence applications. By “embodied intelligence,” we specifically refer to scenarios\nsuch as intelligent robots, autonomous driving, and drones, where sensing and\nprocessing must be tightly integrated to interact effectively with dynamic environments.\nWe have revised the manuscript to provide a clearer definition and more precise\ndescription of this concept.\nRevised text:\nIntroduction (lines 36-39)\n…, but current image sensor technology falls short in meeting the requirement of real-\ntime, low-power processing capability at the edge, which is critical for embodied\n21\nintelligence such as intelligent robots, autonomous driving, and drones that require\neffective interaction with dynamic environments.\n3.4 Reviewer 3 states:\n(4) \"NEOSTI utilizes a pulse width modulated (PWM) based non-linear readout pixel\nstructure, realizing a retina-like 3-layered processing structure to generate pulse signals\nanalogous to neural spikes\" — NEOSTI is far from being \"retina-like\" or from\ngenerating anything resembling \"neural spikes\". This is per se not a problem but does\nnot need to be claimed this way.\nOur response:\nWe sincerely appreciate the reviewer’s insightful comment.\nWe agree that NEOSTI’s design does not reproduce the exact neural structure and\nmechanism of the retina. Rather than biological equivalence, our intention is to describe\nthe functional analogy reflected in two aspects: (i) the non-linear photoresponse,\nand (ii) time-domain pulse representation of light intensity.\ni. The PWM pixel in NEOSTI exhibits a nonlinear photoresponse curve (as shown in\nFig.9) similar to the sigmoidal photoresponse curve observed in retinal\nphotoreceptors (as shown in Fig.10).\n In the mid-range of light intensity, the pixel response is approximately linear\nwith respect to the logarithm of light intensity, consistent with the logarithmic\nencoding of the retinal photoreceptors’ response to light.\n At high or low illumination levels, the pixel response gradually saturates,\ncorresponding to the decline of light sensitivity under strong or weak light\nconditions observed in biological vision systems.\nFig. 9 (Fig. 3k of the manuscript) Experimental results of the photoresponse curve of NEOSTI’s\npulse width modulated pixel structure. The relationship between photocurrent (light intensity) and\npulse width is non-linear.\n[editorial note: figure redacted due to containing third-party material]\nFig. 10 (Fig. 3j of the manuscript) Photoresponse curves of cones and rods and adaptability under\n22\ndifferent lighting conditions. (Adapted from Valeton & van Norren, 1983; and Boynton &\nWhitten, 1970)\nii. The PWM pixel in NEOSTI transfers the intensity information to time domain,\nwith a single-bit pulse signal output whose width encodes the light intensity. The\ninformation is represented in temporal domain, sharing a conceptual similarity with\nneural spike coding where the information is carried by temporal patterns rather\nthan amplitude variations.\nFurthermore, the PWM pixel and subsequent readout circuit (as shown in Fig. 11)\nare co-designed to fully exploit this temporal pulse encoding, integrating sensing\nand computation within the same hardware implementation, achieving an efficient\npre-processing capability analogous to the way biological visual system performs\nearly-stage information compression and feature extraction directly at the sensing\nlevel.\nFig. 11 (Fig. 4a of the manuscript) Architecture of the in-sensor processing element (PE). The PE\ncircuit mainly includes two counters and corresponding control logic, parameter registers, and\ncalculation logic, which can complete the readout of raw data and column-parallel calculation.\nWe have revised the manuscript to reduce statements of excessive analogy. We believe\nthe modification clarifies that our work is biomimetic in function but not biological in\nmechanism.\nRevised text:\nResults (lines 190-192)\nNEOSTI utilizes a pulse width modulated (PWM) based 3-layered readout pixel\nstructure, realizing a retina-mimicking non-linear photoresponse and pulse-based\nrepresentation of light intensity.\n3.5 Reviewer 3 states:\nMore issues not related to technical content that need attention:\n(1) Several language-related flaws: E.g. \"NEOSTI is ... system that can operable ...\",\n\"lenless\" — proof reading required\nWe sincerely appreciate the reviewer’s insightful comment.\nWe acknowledge the language-related flaws in the original manuscript and have\ncarefully proofread the paper to correct such grammatical and spelling errors. We\nbelieve the revised version has improved readability and linguistic accuracy.\nRevised text:\n1. Introduction (line 91)\nThe NEOSTI is one of the smallest electronic-opto fully integrated vision systems\nthat can operable operatein typical indoor and outdoor environments, …\n2. Results (line 128)\nThe convolution mask and chip are co-packaged forming a lenless lensless vision\nchip, …\n3. Introduction (line 34)\nAs a bionic device, the image sensor established establishes a bridge between the\nreal world in optical domain and artificial intelligence in electronic domain,\n3.6 Reviewer 3 states:\n(2) Factually imprecise statements: \"consumer image sensor can only achieve a DR\naround 140dB\" — no, consumer imagers typically reach 60-70dB, only specialized\nautomotive multi-exposure image sensors go up to DR levels around/beyond 140dB\"\nOur response:\nWe sincerely appreciate the reviewer’s insightful comment.\nWe acknowledge and appreciate the reviewer for pointing out the imprecise in our\ndescription of the dynamic range (DR) of consumer image sensors. The typical dynamic\nrange of CMOS image sensors applied in smart phones is around 60-70dB, while high\nDR image sensors require specialized techniques. For example, the OmniVision\nOX03J10 image sensor6 achieves up to 140 dB dynamic range by combining dual\nconversion gain pixels with multi-exposure HDR capture, which is designed for\nautomotive imaging applications.\nWe fully accept your suggestion and have revised the statement in a more\ncomprehensive and accurate form.\nRevised text:\nResults (lines 108-110)\nadvanced of which reaches around to 140 dB.\n3.7 Reviewer 3 states:\n(3) Citations and benchmarking: NEOSTI claims to be \"neuromorphic\" but does not\ncompare itself to any other neuromorphic vision solutions (e.g. event sensor based) that\ncan solve similar problems; e.g. Synsense's Speck, combining event pixel array and\nSNN processor on a single chip (https://arxiv.org/abs/2304.06793) nor with SoA\nindustry Vision System-on-chip solutions like Sony's IMX500, targeting similar use\ncases/applications (https://www.aitrios.sony-semicon.com/edge-ai-devices/imx500).\nOur response:\nWe thank the reviewer for the suggestion to include additional citations and\nbenchmarking. We have added the relevant specifications of SynSense's Speck event\ncamera and Sony's IMX500 (as shown in Table 3).\nOptical Sensing Calculation\nComputing Total Power Natural\nPath Energy Energy Equivalent\nReference Speed Consumption Light\nLength Efficiency Efficiency Algorithm\n(MOPs/s) (W) Input\n(mm) (pJ/pixel/fps) (TOPS/W)\n9-layer\nSynSense\n4-50 N/A N/A N/A 420-15,000 u spiking Yes\nSpeck\nCNN\nSony 1382 4.97 (peak) configurable\n4-50 25.12 178.8 m Yes\nIMX500 (peak) (8-32b) CNNs\nTable 3 (Part of Table 1 in the revised manuscript) Main experimental performance of NEOSTI\ncompared with state-of-the-art works.\nRegarding the concept of “neuromorphic”, our design emphasizes the biologically-\ninspired opto-electronic hybrid neural network processing capability integrated in\nNEOSTI together with the non-linear photoelectric response, which differs from the\nevent-based cameras (such as DVS) that focuses on sensing luminance changes and\ngenerating neural-like event spikes. The two approaches represent different\ninterpretations and implementation paths of the neuromorphic concept, where event-\ndriven designs centers on neural spike emulation, and NEOSTI emphasizes the co-\nintegration of opto-electronic hybrid sensing and processing. Event-based cameras\nperforms well at capturing fast motion and dynamic changes, but the event-stream\noutput loses the global spatial features of each frame, lacking sufficient capability to\nhandle static scenes. In contrast, our design offers more general processing capability\nunder a wider range of application scenarios.\nWe have added the discussion of the term “neuromorphic” to Supplementary Note 5,\nand added the comparison between NEOSTI and event-based sensors to the Discussion\nRevised text:\nTable 1\nTable 1 Main experimental performance of NEOSTI compared with state-of-the-art works. …\nSupplementary Note 5\nThe term \"neuromorphic\" has been used in a broad range of contexts, often referring to\nhardware systems that emulate neural computation or biological processing principles.\nIn NEOSTI design, we embody the concept of neuromorphic through the biologically-\ninspired, fully-integrated opto-electronic hybrid neural network processing capability\ntogether with the non-linear photoelectric response. While the event-driven designs,\nsuch as DVS, focus on sensing luminance changes and generating neural-like event\nspikes. These two approaches represent different interpretations and implementation\npaths of the neuromorphic concept, where event-driven designs centers on neural spike\nemulation, and NEOSTI emphasizes the co-integration of opto-electronic hybrid\nsensing and processing.\nDiscussion (lines 290-294)\nCompared with other neuromorphic designs such as dynamic vision sensors (DVS),\nwhich perform well in capturing rapidly changing or moving scenes but lack sufficient\ncapability to handle static scenes, NEOSTI provides more general feature extraction\nand compression capabilities, making it suitable for a wider range of scenarios\nencompassing both dynamic and static scenes.\n26\n3.8 Reviewer 3 states:\n(4) Pulse modulation (including PWM) is a well known approach in vision sensing —\nprior work should be cited (e.g. https://ieeexplore.ieee.org/document/5701724).\nOur response:\nThank you for your insightful comment.\nAs the reviewer pointed out, the PWM pixel structure is widely used in digital image\nsensor design. We appreciate and accept the reviewer’s comment, and have added this\nwork as Ref.[33] in the revised manuscript. The following two references regarding\nPWM pixels have also been added in the revision as Ref.[31"}]

Decision letter