Patent Description:
Passive vision in low-light environment is required for many military applications, such as reconnaissance or hidden maneuvers. Conventional sensors (e.g., complementary metal oxide semiconductor (CMOS) image sensors) used in vision devices lose light information when operating in low-light conditions when intensity of the light is below a sensitivity threshold or above a saturation threshold of the sensor. Increasing sensitivity of the conventional sensor does not provide an efficient solution, as this can lead to more frequent saturation of the sensor when the light is not sufficiently low, causing a loss of light information. In addition, high sensitivity sensors are fragile and expensive and can output noisy signals.

In comparison to conventional sensors, event-based (neuromorphic vision (NMV)) sensors accumulate light energy. The accumulated light energy is integrated until the integration reaches some fixed threshold δ. Upon reaching the threshold, a fixed binary signal is output (also referred to as the NMV sensor firing)' through a transmission line, and the integration process is reset. In a low-light setting, such sensors can have several advantages over conventional sensors.

Unlike conventional sensors, NMV sensors do not loose light information due to saturation. In other words, their dynamical range is significantly higher. NMV sensors are less affected by measurement noise in low-light settings. Additionally, binary signals output by NMV sensors are robust to transmission noise and require less energy and simpler hardware than non-binary signals.

Further advantages arise when polarization filters are used with NMV sensors for enhancing image reconstruction. Such polarization filters reduce light intensity. In a low-light environment, conventional sensors are more affected by this reduction of light intensity than NMV sensors.

However, image reconstruction from NMV sensor output remains challenged. To date, NMV systems recover an image for a particular pixel using only sensor information nearest to the location of the pixel and at the point in time that that pixel sensed an event. This leads to lesser quality of reconstructed images compared to images reconstructed from output of conventional sensors.

The reason for this reduction of image quality when reconstructed from NMV sensor output is that potential for information from NMV sensors is yet untapped. In particular, image and video signals produced using NMV sensors potentially have untapped hidden sparsity in spatial and temporal domains. Since video has significantly higher sparsity than images, the untapped hidden sparsity in spatial and temporal domains for video is greater than for images.

While conventional methods and systems have generally been considered satisfactory for their intended purpose, there is still a need in the art for vision systems to operate in low-light environments while overcoming the limitations of systems that use conventional sensors and methods and improving exploitation of available light information and yet untapped hidden sparsity in signals. <NPL>, discloses a spike camera with a video capture scheme and methods of decoding a spike stream for texture reconstruction. <NPL>, discloses real time compressive sensing video reconstruction in hardware. <NPL>, discloses a spike-to-image neural network that reconstructs a dynamic scene form a continuous spike stream.

The purpose and advantages of the below described illustrated embodiments will be set forth in and apparent from the description that follows. Additional advantages of the illustrated embodiments will be realized and attained by the devices, systems and methods particularly pointed out in the written description and claims hereof, as well as from the appended drawings. To achieve these and other advantages and in accordance with the purpose of the illustrated embodiments, in one aspect, there is provided a method of imaging as claimed in claim <NUM>.

In another aspect of the disclosure, an NMV system as claimed in claim <NUM> is provided.

In a further aspect, a compressive sensing and reconstruction (CSR) engine as claimed in claim <NUM> is provided for reconstructing images and/or video from output of an array of NMV sensors passively sensing light in a low-light environment.

Reference will now be made to the drawings wherein like reference numerals identify similar structural features or aspects of the subject disclosure. For purposes of explanation and illustration, and not limitation, a block diagram of an exemplary embodiment of a neuromorphic vision (NMV) system <NUM> in a low-lighting environment in accordance with the disclosure is shown in <FIG> and is designated generally by reference character <NUM>. Methods associated with vision and of NMV system <NUM> and processing for applying compressive sensing to output of NMV system <NUM> in accordance with the disclosure, or aspects thereof, are provided in <FIG>, as will be described. The systems and methods described herein can be used to provide improved vision and imaging system, such as for military applications (e.g., reconnaissance or hidden maneuvers), navigation (autonomous or vision assistance ), e.g., for driving, piloting, space travel, unmanned aerial vehicles, above or underwater marine navigation), personal or commercial security systems, wildlife observation systems, etc..

NMV system <NUM> is configured to sense light by an array of NMV sensors in a low-light environment, accumulate the sensed light, integrate the accumulated light for each of the NMV sensors, output event signals per NMV sensor and reset the corresponding NMV each time the integration has exceeded a threshold value. The event signals are provided for further processing in order to be reconstructed into an image.

NMV system <NUM> includes an NMV array <NUM> and NMV processor <NUM>. NMV array <NUM> includes a plurality of sensors <NUM>. The number of NMV sensors <NUM> included in NMV array <NUM> is not limited by the example shown in <FIG> and can be any number selected for the application used. NMV processor <NUM> stores information about each NMV sensor <NUM> in association with a sensor identification (ID). The information stored for each NMV sensor <NUM> can include, for example, location of the NMV sensor <NUM> within NMV array <NUM>, spectral information about NMV sensor <NUM>, polarization filter information about NMV sensor <NUM>, orientation of NMV sensor <NUM>, and any other information particular to NMV sensor <NUM> that can be used for performing compressive sensing and/or image reconstruction.

As explained in WIPO patent publication No. <CIT>, image data may be output from a sensor array into a digital retina that converts that image data into "spikes" using various image processing and data processing techniques. The digital retina includes digital circuitry that generates spike data indicative of a spike in association with a particular photoreceptor within the sensor array whenever the intensity value measured by that photo receptor exceeds a threshold. The digital retina can be implemented using various solid-state technology including, for example, complementary metal-oxide- semiconductor (CMOS) implemented technology, which can include, for example, one or more field programmable, gate arrays (FPGAs), graphics processing units (GPUs), or functionally or structurally similar devices integrated circuits and associated software and/or firmware provided in, for example, application specific integrated circuits (ASICs).

NMV array <NUM> can be exposed to light reflected from a scene <NUM> in a low-light environment and respond by outputting event signals when triggered to do so. NMV processor <NUM> receives and processes event signals output by individual NMV sensors <NUM> each time that NMV sensor <NUM> is triggered to output an event signal. The event signal is a binary signal and includes identification (ID) information that identifies the NMV sensor that output the event signal. NMV processor <NUM> timestamps receipt of the event signal. NMV processor <NUM> stores spatiotemporal spike patterns (SSPs) <NUM>, updating the SSPs <NUM> over time and outputting the SSPs <NUM> for processing in order that an image can be reconstructed from the SSPs. SSPs <NUM> can be output at regular intervals, upon request, or in response to satisfaction of a condition.

NMV sensors <NUM> sense and accumulate light. Even under low-light conditions, some light presents; even this small amount of light can be sensed and accumulated. Each NMV sensor <NUM> integrates the sensed light and stores an integrated charge until the integrated charge exceeds a predetermined threshold value. When the threshold value is exceeded, NMV sensor <NUM> outputs a binary signal to NMV processor <NUM> and is reset. The NMV sensor <NUM> can reset itself or can trigger an external device to reset the NMV sensor <NUM>. Once reset, the NMV sensor <NUM> stores zero integrated charge and starts accumulating sensed light anew.

NMV sensors differ from conventional sensors in that unlike conventional sensors, NMV sensors do not merely compute a linear function of image brightness at a location of each sensing element, but processing uses a nonlinear mapping function, local spatiotemporal filtering, and adaptation.

In low light conditions, unlike conventional sensors, NMV sensors <NUM> have the ability to capture all available light information. There is no loss of light information, such as due to lack of sensitivity or saturation, as occurs in conventional sensors. NMV sensors <NUM> are sturdier and less expensive than high sensitivity conventional sensors, which tend to be expensive and fragile.

Further advantages over conventional sensors include a reduced quantity of signal transmission. Only event signals that accumulated enough light are sent from NMV sensors <NUM>. Those NMV sensors <NUM> that did not yet accumulate enough light do not output event signals. Therefore, the amount of event signals is reduced compared to signals sent at regular intervals by conventional sensors, since all conventional sensors in an array transmit a signal at each interval, even when nothing is sensed. This difference is much more noticeable in low-light conditions when firing of NMV sensors is sparse and transmission of the binary output is inexpensive.

Another advantage relative to conventional sensors is that noise included in event signals is averaged during integration and thus is reduced. In addition, the binary signal used by the event signal is robust to transmission noise and requires less energy and simpler hardware than non-binary signals.

In normal light conditions, with regards to post-processing, compression of light information output by NMV sensor is lossy and non-optimal compared to raw data output using conventional compression methods such as JPEG. However, under low-light conditions, NMV sensors capture more light information than conventional sensors due to saturation in the latter. In addition, under low-light conditions enhanced compression (e.g., MPEG compression), traditionally used for images or videos, leads to expensive transmission while output of NMV sensors is sparse and its transition is inexpensive.

However, when compressive sensing techniques are available for application to raw event signals, there is no longer a need to for the raw event signals to compression methods, such as JPEG and MPEG, in order to be transmitted and stored. The ability to transmit and store raw event signals annuls the need for crosstalk between NMV sensors <NUM> or supervision of such crosstalk by a processing unit, which occurs in conventional compression methods, such as JPEG and MPEG.

With reference to <FIG>, a compressive sensing reconstruction (CSR) engine <NUM> receives the spatiotemporal spike patterns <NUM> and processes them with compressive sensing which is used for image reconstruction for output of a reconstructed image, such as example reconstructed image <NUM>. Compressive sensing is currently used for recovering image, video, or other numerical fields from data using underlying sparsity of the unknown numerical field.

Based on existing technology, compressive sensing cannot be applied to raw event signals from NMV sensors. While compressive sensing is currently available for use with video and images, it assumes a matrix transforming light intensities into measurements is fixed a priori and that measurements are linear functions of light intensities. These assumptions imply a different structure of the problem to be solved by compressive sensing than is present in an NMV system. To illustrate, compressive sensing techniques currently available are unable to recovery images and/or video from data presented in the form of rare spikes, such as the output from NMV sensors <NUM>.

Compressive sensing fusing CSR engine <NUM> includes a preprocessor <NUM> that prepares raw event signals output by NMV sensors <NUM> for compressive sensing and image and/or video reconstruction. The raw data includes time stamps of spikes output from NMV sensors <NUM> and identification of the corresponding NMV sensor <NUM> (which is used to retrieve location of the NMV sensor <NUM> in the NMV array <NUM>) that output the spike. Preprocessor <NUM> constructs a matrix of linear equations and solves the constraint optimization problem using the raw data. The matrix of the linear equation system depends on sensed data, namely timestamps at which identified NMV sensors <NUM> fired, which results in a structure of the CS problem that is different from existing art.

Preprocessor <NUM> computes matrices A and F for linear system AFa = δ, the linear system being integral to the optimization problem solved by CSR engine <NUM>. Matrices A, F are constructed using data (time stamps of spikes and corresponding NMV sensor identification) from NMV sensors <NUM> and sensor locations. Preprocessor <NUM> correlates the number of spikes for each NMV sensor <NUM> (which is data-driven and is coordinated with construction of matrix A). Preprocessor <NUM> further uses a spatial sparsifying basis, such as Fourier and or Wavelet (the basis used is predetermined and does not change).

CSR engine <NUM> solves compressive sensing by minimizing non-zero components of a sparse vector under data-driven linear constraints and reconstructs an image(s) and/or video.

NMV processor <NUM>, CSR engine <NUM>, and/or preprocessor <NUM> can be integrated in a single device or share one or more hardware or software components. Additionally, NMV processor <NUM>, CSR engine <NUM>, and/or preprocessor <NUM> can be implemented as physical or virtual devices. Whether implemented as physical or virtual device(s), each of NMV processor <NUM>, CSR engine <NUM>, and/or preprocessor <NUM> uses a local or remote hardware processing device that executes software instructions, which enables performance of the disclosed functions.

Communication between NMV processor <NUM>, CSR engine <NUM>, and/or preprocessor <NUM> can include wired or wireless communication, which can include communication via a network, such as an intranet, a local area network (LAN), and/or a wide area network (WAN).

With reference to <FIG>, application of CS to the raw event data is effective regardless of the configuration of array <NUM>. NMV sensors <NUM> can be positioned in a regular grid configuration, in a completely random configuration, or something between these two configurations. Individual NMV sensors <NUM> can be configured to have a different respective spectral ranges, use polarization filters of different respective types (wherein each type of polarization filter filters light at a different angle), and/or have different respective orientations for different angles of observation of a scene. Distribution of NMV sensors <NUM> having different spectral ranges, polarization, and/or orientation can be in accordance with a pattern, groupings, random, or some combination of thereof.

In comparison with conventional sensors, although polarization sensing can be desirable because it can produce significant advantages for image reconstruction, polarization filters are not suitable for low light environment due to light reduction caused by polarization filters. However, polarization filters do not have a negative effect when used with NMV sensors <NUM>, since the light threshold in NMV can be adjusted to compensate for loss of light in polarization filter, and the advantages of polarization filtering can still be had.

Whereas in conventional sensors that provide multispectral vision, when an individual rectangular sensor array is used for each spectral range, there is either an increase in the number of sensors used or there is a reduction in resolution. This is because conventional sensors operate based on local information in which the conventional sensors provide light information at a specified location of the conventional sensor. There is a blind spot at any location that does not have a sensor.

However, when using compressive sensing to process raw event signals output by NMV sensors <NUM>, there is no need in NMV array <NUM> for NMV <NUM> sensors to be grouped in rectangle according to spectral range. This is because compressive sensing operates with image/video as a whole and recovers it as a solution of an optimization problem. Coordinates of NMV sensors <NUM> are only used when constructing the linear equation system. When a sensor is removed, a row in the linear equation system is removed. Similarly, if an NMV sensor <NUM> is added or moved, coefficients of corresponding linear equations are adjusted. A missing NMV sensor <NUM> may have some effect on the solution, but does not lead to a blind spot. This freedom of distribution allows for multispectral vision without an increase in the number of sensors used or a reduction in resolution.

Whereas health monitoring of individual conventional sensors uses extra hardware, this affects the cost, weight, size, energy consumption and overall complexity of the vision system. However, when using compressive sensing to process raw event signals output by NMV sensors, monitoring of NMV array <NUM> can be performed by emulating sensor responses from recovered video signal, comparing actual NMV sensor responses with the emulated sensor responses, and determining whether any difference there between indicates dysfunctionality of an NMS sensor <NUM>.

More particularly, when using compressive sensing, any NMV sensor <NUM> can be removed from consideration without physically removing the NMV sensor <NUM>. Predictions can be made of output for that NMV sensor <NUM> and the prediction can be compared with actual output. Furthermore, accuracy of the prediction can be estimated. If a difference between actual and predicted output exceeds prediction accuracy, and especially if it does so consistently, then it is likely that something is wrong with the NMV sensor <NUM>. In addition, outlier data is readily distinguishable upon solving the optimization problems, wherein outlier data is output from NMV sensors <NUM> that are suspected to be defective. In this way, the NMV sensors <NUM> can be monitored independent of physical presence of the NMV sensors.

When an NMS sensor <NUM> is deemed dysfunctional, its output can be excluded from future processing for image reconstruction. No physical changes need to be made to the NMV sensor array <NUM>. With returned reference to <FIG>, the mathematical solution applied by preprocessor <NUM> to NMV output from NMV sensors <NUM> is described in greater detail. NMV sensors <NUM> of NMV array <NUM> are modelled as a set of n individual event-based sensors distributed in 2D domain Ω. It is assumed that the process is observed on a time interval [<NUM>, T]. A goal is to reconstruct a light intensity field L(x, y, t) dynamically at location x, y at time t over the whole domain Ω × [<NUM>, T].

No limitations are made on sensor placement, meaning within 2D domain Ω, NMV sensors <NUM> can be laid in a regular grid, or by completely random distribution or something between these two extremities. Coordinates of an ith sensor of NMV sensors <NUM> are denoted by (xi,yi). Integration of a data signal for each sensor i at moment t is represented using Equation (<NUM>): <MAT> Let ti,<NUM>,. , ti,n denote times when neuromorphic encoder for sensor i "fires" an event signal <NUM>,. Correspondingly, Ni is a total number of signals received from ith sensor on time interval [<NUM>, T]. A definition of neuromorphic encoding is represented by Equation (<NUM>): <MAT>.

By definition of Equation (<NUM>), the difference in the left-hand term is a light signal accumulated between two consecutive firings of an NMV sensor i at the moments ti,j and ti,j+<NUM>. This difference must be equal to δ in order for there to be a firing at moment ti,j+<NUM>.

Equation (<NUM>) can be equivalently rewritten in a form that describes a linear equation system, as represented by Equation (<NUM>): <MAT> where q = col(q<NUM>(t<NUM>,<NUM>),. , q<NUM>(t<NUM>,N<NUM>), q<NUM>(t<NUM>,<NUM>),. , q<NUM>(t<NUM>,N<NUM>),. , qn(tn,<NUM>),. , qn(tn,Nn)), <MAT>, A = diag(AN<NUM>,. , ANn), δ = the threshold for integration <MAT> <MAT>.

Equation (<NUM>) is a unity of equations (<NUM>) written for all sensors and all firing times and organized in matrix form.

The problem is formulated as reconstruction of the light field Q(x,y,t) under the conditions represented by Equation (<NUM>).

Reconstruction of a light intensity field from neuromorphically encoded signals is now discussed, in particular a reconstruction method for a field Q(x,y,t) of light intensities. Light field Q(x,y,t) is image/video to be restored. Light field Q(x,y,t) is static for restoration of an image and dynamic for restoration of video over a time interval (as opposed to a momentous scene). For each reconstruction process, which can be performed at discrete time intervals for reconstruction of video, light field Q(x,y,t) is fixed, either as a fixed image or as a video over a fixed time interval. Within reconstruction, light field Q(x,y,t) does not change. When a scene captured by NMV array <NUM> is a dynamic scene, reconstruction is repeated successively and processed in real time, also referred to as online, the reconstruction instances recovering video over respective, successive time intervals.

Scene reconstruction can be expressed as a problem of recovering a signal from sparse measurement. Compressive sensing methods are described in <NPL>). A main assumption behind this approach is sparsity of the signal in some appropriate basis. Practice shows that this assumption holds for video signal expansions in Fourier and Wavelet bases.

Following compressing sensing methodology as described by <NPL>), a basis <MAT> is used that allows sparse representation of Q, as represented by Equation (<NUM>), which is a conceptual step as follows: <MAT> where vector a = col(a<NUM>,. , aM) is a sparse vector, meaning a vector with relatively few non-zero components. Equation (<NUM>) expresses that image/video Q(x,y,t) can be presented as a linear combination of relatively few functions from the chosen basis {Bi}, such as Fourier basis or Wavelet basis, or any basis that allows representation of image/video in the form of Equation (<NUM>) with vector a being a sparse vector. Bi(x,y,t) are elements of the chosen basis, e.g., Fourier functions for Fourier basis, Wavelets for Wavelet basis, etc., without limitation to a specific basis. Fourier basis is selected, for example, as the spatial sparsifying basis. The Fourier basis has some advantages. First, Fourier basis is successfully used for sparsification in many image-related and/or physics-based practical applications (e.g., JPEG coding). Second, sparse Fourier transformation can be done more efficiently than a transformation associated with other standard sparsifying bases, as described by <NPL>).

Combining Equations (<NUM>) and (<NUM>) (also referred to as combining the event signals output by the NMV sensors, wherein the combination is formatted into a format that is compatible for an algorithm used to perform compressive sensing) the reconstruction problem can be formulated as represented by Problem (<NUM>): <MAT> where ∥ a ∥<NUM> is L<NUM> norm of vector a (wherein L<NUM> is the number of non-zero components in the vector a), <MAT> <MAT>.

With notations as introduced, Equation (<NUM>) can be written in the form q = Fa. Substituting q = Fa in Equation (<NUM>) provides: AFa = δ e, which is the constraint provided in Problem (<NUM>). Problem (<NUM>) thus captures all information obtained from the NMV sensors <NUM>, including times of spikes and the locations of the NMV sensors, formatted and expressed in a language that is compatible with compressive sensing. Thus in Problem (<NUM>) the number of non-zero terms is minimized in sparse representation Equation (<NUM>) of the image/video under the constraint that summarizes the measurements.

It is noted that sensing matrix AF is not set a priori, but rather is a data-dependent matrix that is dynamic over time, depending on measured signals. The measured signals are points in time when spikes are issued by identified NMV sensors. Solving the mathematical problem represented by Problem (<NUM>), image Q is reconstructed.

Methods for solving the optimization problem represented by Problem (<NUM>) are now described. Problem (<NUM>) is relaxed to a convex though non-differentiable problem. Several examples of such problems are represented in Problems (<NUM>) - (<NUM>): <MAT> <MAT> <MAT> where λ, ε are positive coefficients fixed a priori, ∥ a ∥* is sparsity-inducing norm of vector a, e.g., L<NUM> norm, ∥ ∥# is any norm, most commonly L<NUM> norm.

Correctness of relaxations of the problem represented by Problem (<NUM>) to one of the forms represented by Problem (<NUM>)-(<NUM>) depends on properties of the matrix (AF). Both theory and practice favor that matrix (AF) be created by sensors distributed randomly and independently over a domain, as described by <NPL>.

Any of Problems (<NUM>)-(<NUM>) can be solved by highly efficient methods, as described by <NPL>). In the case when high convergence speed rather than high accuracy of the solution is needed, Split-Bregman method is often used, as described by <NPL>).

Accordingly, measurements from an array of NMV sensors when processed using Equations (<NUM>)-(<NUM>) is converted into a form that is compatible with compressive sensing and optimization equations represented by Problems (<NUM>)-(<NUM>). The ability to apply compressive sensing to the output of NMV sensors is critical to image reconstruction from NMV sensor output. In this way, images and video can be recovered using a passive vision system that senses passively while operating in a low-light environment.

Once the image is reconstructed, online health monitoring of the sensors may be performed by identification and isolation of NMV sensors <NUM> which output signals that correspond to outliers in objective functions represented by Problems (<NUM>)-(<NUM>).

The method can be combined with a method described in <CIT> and <CIT>. This combination would provide a computationally efficient method for multi-spectral and/or multi-polarization imaging using a same number of NMV sensors as would be used for single-spectral imaging and/or non-polarized imaging using an array of conventional sensors.

<FIG> shows an exemplary and non-limiting flowchart <NUM> illustrating a method for applying compressive sensing to raw event data from NMV sensors, such as NMV sensors of NMV system <NUM> shown in <FIG> and <FIG>,and reconstructing an image and/or video in accordance with certain illustrated embodiments. The event data includes (spectro)temporal spike patterns that imply spatial information based on identification of NMV sensors that output corresponding event data. The event data can include spectral data or can be monochromal. The method can be performed by CSR engine <NUM> shown in <FIG>. Before turning to description of <FIG>, it is noted that the flowchart in <FIG> shows an example in which operational steps are carried out in a particular order, as indicated by the lines connecting the blocks, but the various steps shown in this diagram can be performed in a different order, in parallel, or in a different combination or sub-combination. It should be appreciated that in some embodiments some of the steps described below may be combined into a single step. In some embodiments, one or more additional steps may be included. In some embodiments, one or more of the steps can be omitted.

At block <NUM>, temporal integration output, including time stamps t<NUM>,. ,tNn, is received from NMV sensors of an NMV sensor array, such as NMV sensors <NUM> and NMV sensor array <NUM> shown in <FIG>. The timestamps are used for construction of the matrix F and matrix A used in Problem (<NUM>). In one or more embodiments, the temporal integration output can be (spectro)temporal, as the integration output can be integrated over spectrum. Furthermore, the integration output can be dynamic over time and can be received and processed in real time.

At block <NUM>, temporal integration output, such as temporal spike patterns shown in <FIG> and <FIG>, is transformed into a system of linear equations. This process of block <NUM> is described above with respect to Equations (<NUM>-<NUM>). The transformation into a system of linear equations can be dynamic over time and can be performed in real time.

The process of block <NUM> is described above with respect to conceptual Equation (<NUM>), such as by construction of matrix F as applied in Problem (<NUM>). The sparse representation of light field Q(x,y,t) implied by the matrix F can be dynamic over time and can be received and processed in real time.

At block <NUM>, reconstruction of an image is performed using sparse representation. The reconstruction can be dynamic over time and can be processed in real time. At block <NUM>, the reconstructed image is output, such as for display by a display device. Output of the reconstructed image can be dynamic over time and can be output in real time such that the output is video. At block <NUM>, optimization of the NMV system parameters, such as adjustments to firing thresholds, can be performed, with an objective of enhancement of image quality or adaptation of the NMV system to changing light conditions or the needs of a particular application (such as navigation). The optimization can be performed in real time. At block <NUM>, health monitoring of the NMV array can be performed. The health monitoring can be performed in real time. Blocks <NUM> and <NUM> are shown in dotted lines as an indication that they are optional and/or can be performed in parallel with one or another or blocks <NUM>-<NUM>.

Aspects of the present disclosure are described above with reference to block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. Features of the methods described include operations, such as equations, transformations, conversions, etc., that can be performed using software, hardware, and/or firmware. Regarding software implementations, it will be understood that individual blocks of the block diagram illustrations and combinations of blocks in the block diagram illustrations, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the block diagram block or blocks.

It is to be appreciated the embodiments of the disclosure include software algorithms, programs, or code that can reside on a computer useable medium having control logic for enabling execution on a machine having a computer processor. The machine typically includes memory storage configured to provide output from execution of the computer algorithm or program.

As used herein, the term "software" is meant to be synonymous with any code or program that can be in a processor of a host computer, regardless of whether the implementation is in hardware, firmware or as a software computer product available on a disc, a memory storage device, or for download from a remote machine. The embodiments described herein include such software to implement the logic, equations, relationships and algorithms described above. One skilled in the art will appreciate further features and advantages of the illustrated embodiments based on the above-described embodiments. Accordingly, the illustrated embodiments are not to be limited by what has been particularly shown and described, except as indicated by the appended claims.

With reference to <FIG>, a block diagram of an example computing system <NUM> is shown, which provides an example for configuration of NMV processor <NUM>, CSR engine <NUM>, and/or preprocessor <NUM>, any of which can optionally be embedded in a Device A1. Additionally, all or portions of NMV processor <NUM>, CSR engine <NUM>, and/or preprocessor <NUM> could be configured as software, and computing system <NUM> could represent such portions. Computing system <NUM> is only one example of a suitable system and is not intended to suggest any limitation as to the scope of use or functionality of embodiments of the disclosure described herein. Computing system <NUM> can be implemented using hardware, software, and/or firmware. Regardless, computing system <NUM> is capable of being implemented and/or performing functionality as set forth in the disclosure.

Computing system <NUM> is shown in the form of a general-purpose computing device. Computing system <NUM> includes a CPU/Processor <NUM>, storage <NUM>, an input/output (I/O) interface (I/F) <NUM> that can communicate with an internal component, such as optionally a user interface <NUM> and optionally an external component <NUM>.

The CPU/Processor <NUM> can include, for example, a Programmable Logic Device (PLD), a microprocessor, a DSP, a microcontroller, an FPGA, an ASIC, and/or other discrete or integrated logic circuitry having similar processing capabilities. The CPU/Processor <NUM> and the storage <NUM> can be included in components provided in an FPGA, ASIC, microcontroller, or microprocessor, for example. Storage <NUM> can include, for example, volatile and nonvolatile memory for storing data temporarily or long term, and for storing programmable instructions executable by the CPU/Processor <NUM>. Storage <NUM> can be a removable (e.g., portable) memory for storage of program instructions. I/O I/F <NUM> can include an interface and/or conductors to couple to the one or more internal components <NUM> and/or external components <NUM>.

Computer system <NUM> is only one example of a suitable system and is not intended to suggest any limitation as to the scope of use or functionality of embodiments of the disclosure described herein. Regardless, computer system <NUM> is capable of being implemented and/or performing any of the functionality set forth hereinabove.

These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flow diagram and/or block diagram block or blocks.

The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational operations to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the block diagram block or blocks.

Computer system <NUM> may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, and so on that perform particular tasks or implement particular abstract data types.

It is intended that the disclosure not be limited to the particular embodiment(s) disclosed, but that the disclosure will include all embodiments falling within the scope of the appended claims.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

It must be noted that as used herein and in the appended claims, the singular forms "a", "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a stimulus" includes a plurality of such stimuli and reference to "the signal" includes reference to one or more signals and equivalents thereof known to those skilled in the art, and so forth.

Claim 1:
A method of imaging comprising:
passively accumulating sensed light by an array (<NUM>) of neuromorphic vision, NMV, sensors (<NUM>) in a low-light environment;
integrating the accumulated light by each of the NMV sensors (<NUM>);
outputting a time-stamped event signal per sensor of the array of NMV sensors (<NUM>) upon an intensity value of the integrated accumulated light exceeding a threshold value;
resetting each NMV sensor (<NUM>) after outputting an event signal for new integration of sensed light;
combining the event signals, wherein combining the event signals includes converting the event signals output by the NMV sensors (<NUM>) into a linear equation system having its matrices constructed using the event signals and identification of corresponding NMV sensors (<NUM>) that output the respective event signals and locations of the NMV sensors (<NUM>) of the array (<NUM>); and
reconstructing an image and/or video based on the combined event signals using compressive sensing.