Patent ID: 12198465

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of the present disclosure will be described in detail on the basis of the accompanying drawings. Furthermore, in the following embodiments, the same reference numeral will be given to the same or equivalent portion or element, and redundant description thereof will be omitted.

A typical event based sensor (EBS) employs a so-called event-driven type driving method in which the existence or nonexistence of address event ignition is detected for every unit pixel, and a pixel signal and ignition time information are read out from a unit pixel in which the address event ignition is detected.

Furthermore, the unit pixel in this description represents a minimum unit of a pixel or unit pixel including one photoelectric conversion element (also referred to as “light-receiving element”) and can correspond to each dot in image data that is read out from an image sensor as an example. In addition, the address event represents an event that occurs for every address that is allocable to each of a plurality of the unit pixels which are arranged in a two-dimensional lattice shape. An event detection sensor responds to a change in intensity without being confined to the boundary of the integration time within frames of a traditional image sensor. Intensity change is correlated with a change in photocurrent, and if this change exceeds a constant threshold value it could be detected as an event.

FIG.1is a block diagram illustrating a schematic configuration example of an imaging device according to at least some embodiments of the present disclosure. As illustrated inFIG.1, for example, an imaging device100includes an imaging lens110, a solid-state imaging device or image sensor200, a recording unit120, a communication interface124, and a processor system or control system130. The various components of the imaging device100may be interconnected to one another by a communications bus128or signal lines. As examples, the imaging device100can be provided as or as part of a camera that is mounted in an industrial robot, an in-vehicle camera, or as part of or in connection with other devices or instruments.

The imaging lens110can include an optical system that collects light from within a field of view114. The collected or incident light is directed (e.g. condensed) onto a light-receiving surface of the image sensor200. In particular, the imaging lens110can collect light from within a selected area of a scene by directing the field of view114to encompass that portion of the scene. The light-receiving surface is a surface of a substrate on which photoelectric conversion elements of pixels310included in the image sensor200are arranged. The image sensor200photoelectrically converts the incident light to generate image data. As discussed herein, the image sensor200can include different sets of photoelectric conversion elements disposed on the same or different substrates. Moreover, the image sensor200can include photoelectric conversion elements that perform single or multiple functions. These functions can include event detection and imaging functions. In addition, the image sensor200can execute predetermined signal processing such as noise removal and white balance adjustment with respect to the generated image data. A result obtained by the signal processing and a detection signal indicating the existence or nonexistence of an address event ignition and ignition time information can be output by the image sensor200to the processor system130. A method of generating the detection signal indicating the existence or nonexistence of the address event ignition will be described later.

The recording system120is, for example, constituted by a flash memory, a dynamic random access memory (DRAM), a static random access memory (SRAM), or the like, and records data provided from the image sensor200.

The processor system130is, for example, constituted by a central processing unit (CPU) and the like. For example, the processor system130can include one or more general purpose processors, controllers, field programmable gate arrays (FPGAs), graphical processing units (GPUs), application specific integrated circuits (ASIC), or combinations thereof. Moreover, the processor system130can execute application programming or routines, stored as software or firmware in memory or data storage included in or interconnected to the processor system130to perform various functions and methods as described herein. For example, the processor system130can process data output from the image sensor200. For example, as described herein, the processor system130can process event detection signals output by the EBS sensor function or portion of the image sensor200and can control the imaging sensor function or portion of the solid-state imaging device, at least in part in response to the event detection signals. The processor system130can also control components of the imaging device100in addition to the image sensor200, such as the operation of the recording unit120, the communication interface124, focusing and shutter operations that might be supported by the imaging lens110, and the like. In accordance with further embodiments of the present disclosure, the processor system130can implement advanced processing capabilities, including but not limited to neural network and artificial intelligence capabilities and functions, as described herein.

Next, a configuration example of the image sensor200will be described in detail with reference to the accompanying drawings.

FIG.2is a view illustrating a lamination structure example of an image sensor200in accordance with at least some embodiments of the present disclosure. As illustrated inFIG.2, the image sensor200can have a structure in which a light-receiving chip201and a logic chip202are vertically laminated. A side of the light receiving chip201opposite the logic chip202is a light receiving surface204. In joining of the light-receiving chip201and the logic chip202, for example, so-called direct joining in which joining surfaces of the chips are planarized, and the chips are laminated with an inter-electron force can be used. However, there is no limitation thereto, and for example, so-called Cu—Cu joining in which copper (Cu) electrode pads formed on joining surfaces are bonded, bump joining, and the like can also be used.

In addition, the light-receiving chip201and the logic chip202are electrically connected to each other, for example, through a connection portion such as a through-silicon via (TSV) that penetrates through a semiconductor substrate. In the connection using the TSV, for example, a so-called twin TSV method in which two TSVs including a TSV that is formed in the light-receiving chip201and a TSV that is formed from the light-receiving chip201to the logic chip202are connected to each other on chip external surfaces, a so-called shared TSV method in which the light-receiving chip201and the logic chip202are connected with a TSV that penetrates through both the chips, and the like can be employed.

However, in the case of using the Cu—Cu joining or the bump joining in the joining of the light-receiving chip201and the logic chip202, both the light-receiving chip201and the logic chip202are electrically connected to each other through a Cu—Cu joint or a bump joint.

As can be appreciated by one of skill in the art after consideration of the present disclosure, an imaging device200implemented as connected light receiving201and logic202chips can include image sensor200components disposed as part of the light receiving chip201, with some or all of the processor system130components disposed as part of the logic chip202. Other components, such as the recording unit120and communication interface components can be distributed amongst one or both of the chips201and202. In accordance with still other embodiments, a data storage or other chip can be laminated and electrically connected to the light receiving201and logic202chips. Moreover, the light receiving chip can include multiple substrates joined to respective logic chips202or to a common logic chip202, for example where the image sensor200includes multiple sensor devices.

FIG.3is a block diagram illustrating a functional configuration example of the image sensor200according to at least some embodiments of the present disclosure. As illustrated inFIG.3, the image sensor200can include a drive circuit211, a signal processor212, an arbiter213, a column ADC220, and a pixel array300. Some or all of the components can be entirely or partially integrated into, or implemented by, the processor system130.

A plurality of unit cells or pixels310, also referred to herein simply as pixels310, are arranged in the pixel array300. Details of the unit pixels310will be described later. For example, each of the unit pixels310includes a photoelectric conversion element such as a photodiode, and a circuit that generates a pixel signal of a voltage value corresponding to the amount of charge generated in the photoelectric conversion element, hereinafter, referred to as a pixel circuit. Moreover, as discussed in greater detail elsewhere herein, the pixel circuit can include either or both of a first or imaging signal generation circuit and a second or address event detection readout circuit. Each photoelectric conversion element can be associated with a respective pixel circuit, or multiple photoelectric conversion elements can be associated with a common pixel circuit.

In this example, the plurality of unit pixels310are arranged in the pixel array300in a two-dimensional lattice shape. The plurality of unit pixels310may be grouped into a plurality of pixel blocks or groups, each including a predetermined number of unit pixels. Hereinafter, an assembly of unit pixels which are arranged in a horizontal direction is referred to as a “row,” and an assembly of unit pixels which are arranged in a direction orthogonal to the row is referred to as a “column.”

Each of the unit pixels310generates charges corresponding to an amount of light received at the respective photoelectric conversion element. In addition, at least some of the unit pixels310can be operated to detect the existence or nonexistence of address event ignition on the basis of whether or not a value of a current (hereinafter referred to as a photocurrent) produced by charges generated in the photoelectric conversion element or a variation amount thereof exceeds a predetermined threshold value. When the address event is ignited, a signal is output to the arbiter213.

The arbiter213arbitrates requests received from the unit pixels310performing the event detection function and transmits a predetermined response to the unit pixel310which issues the request on the basis of the arbitration result. The unit pixel310which receives the response supplies a detection signal indicating the existence or nonexistence of the address event ignition (hereinafter, simply referred to as “address event detection signal”) to the drive circuit211and the signal processor212.

The drive circuit211drives each of the unit pixels310and allows each of the unit pixels310to output a pixel signal to the column ADC220.

For every unit pixel310column, the column ADC220converts an analog pixel signal from the column into a digital signal. In addition, the column ADC220supplies a digital signal generated through the conversion to the signal processor212.

The signal processor212executes predetermined signal processing such as correlated double sampling (CDS) processing (noise removal) and white balance adjustment with respect to the digital signal transmitted from the column ADC220. In addition, the signal processor212supplies a signal processing result and an address event detection signal to the recording unit120through the signal line209.

The unit pixels310within the pixel array unit300may be disposed in pixel groups314. In the configuration illustrated inFIG.3, for example, the pixel array unit300is constituted by pixel groups314that include an assembly of unit pixels310that receive wavelength components necessary to reconstruct color information from a scene. For example, in the case of reconstructing a color on the basis of three primary colors of RGB, in the pixel array unit300, optical color filter materials can be deposited onto the pixels according to a predetermined color filter array to control light of desired wavelengths to reach the pixel surface. Specifically, a unit pixel310that receives light of a red (R) color, a unit pixel310that receives light of a green (G) color, and a unit pixel310that receives light of a blue (B) color are arranged in groups314aaccording to the predetermined color filter array.

Examples of the color filter array configurations include various arrays or pixel groups such as a Bayer array of 2×2 pixels, a color filter array of 3×3 pixels which is employed in an X-Trans (registered trademark) CMOS sensor (hereinafter, also referred to as “X-Trans (registered trademark) type array”), a Quad Bayer array of 4×4 pixels (also referred to as “Quadra array”), and a color filter of 4×4 pixels in which a white RGB color filter is combined to the Bayer array (hereinafter, also referred to as “white RGB array”). In addition, and as discussed in greater detail elsewhere herein, event detection pixels can be interspersed or included within the pixel array300. As also discussed in greater detail elsewhere herein, the event detection pixels may be provided as dedicated event detection pixels, which only perform an event detection function, or as combined event detection and image sensing pixels, which perform both event detection and image sensor functions.

FIG.4is a schematic view illustrating an array example of unit pixels310in the case of employing pixel groups314with an arrangement of unit pixels310and associated color filters in the color filter array configured to form a plurality of Bayer arrays310A. As illustrated inFIG.4, in the case of employing the Bayer array as the color filter array configuration, in the pixel array300, a basic pattern310A including a total of four unit pixels310of 2×2 pixels is repetitively arranged in a column direction and a row direction. For example, the basic pattern310A is constituted by a unit pixel310R including a color filter401of a red (R) color, a unit pixel310Gr including a color filter401of a green (Gr) color, a unit pixel310Gb including a color filter401of a green (Gb) color, and a unit pixel310B including a color filter401of a blue (B) color.

FIGS.5A-5Ddepict various configuration examples of an imaging device100, and in particular of arrangements of a solid-state imaging device or image sensor200pixels, in accordance with embodiments of the present disclosure. More particularly,FIG.5Adepicts an image sensor200having a first or EBS sensor530, which includes an array300of pixels310in the form of address event detection pixels503disposed on a first light receiving chip or substrate201a, and a second or imaging sensor540, which includes an array300of pixels310in the form of image sensing pixels502disposed on a second light receiving chip or substrate201b. As can be appreciated by one of skill in the art after consideration of the present disclosure, an imaging device100including separate EBS530and imaging540sensors can be configured with separate lens assemblies110that collect light from within the same or similar fields of view, or can be configured with a shared lens assembly110that directs light to the sensors530and540via a beam splitter. In accordance with embodiments of the present disclosure, the number of address event detection pixels503included in the EBS sensor530can be equal to the number of image sensing pixels502included in the imaging sensor540. Moreover, the area of each address event detection pixel503can be the same as the area of each image sensing pixel502. Alternatively, the EBS sensor530and the imaging sensor540can have different numbers of pixels310. For example, the image sensor200can include a EBS sensor530having a relatively low number of event detection pixels503, thereby providing a relatively low resolution, and an imaging sensor540having a relatively high number of image sensing pixels502, thereby providing a relatively high resolution. In accordance with at least some embodiments of the present disclosure, event detection and image sensing operations can be performed simultaneously.

FIG.5Bdepicts image sensor200with pixels310configured as combined or shared event detection and image sensing pixels501disposed on a single light receiving chip or substrate201. As can be appreciated by one of skill in the art after consideration of the present disclosure, the shared event detection and image sensing pixels501can be selectively operated in event detection or image sensing modes. Moreover, in accordance with at least some embodiments of the present disclosure, event detection and image sensing operations can be performed simultaneously with some pixels operating in event detection mode and some pixel operating in image sensing mode.

FIG.5Cdepicts image sensor200having an array of unit pixels310that includes a plurality of event detection pixels503and a plurality of image sensing pixels502formed on the same light receiving chip or substrate201. In the illustrate example, the majority of the unit pixels are in the form of image sensing pixels502, with a smaller number of event detection pixels503disposed amongst the image sensing pixels502. However, an image sensor200having both event detection503and image sensing502pixels disposed on the same light receiving chip or substrate201can include the same number of pixels502and503or can have more event detection pixels503than image sensing pixels502. In accordance with at least some embodiments of the present disclosure, event detection and image sensing operations can be performed simultaneously.

FIG.5Ddepicts an image sensor200having an array of unit pixels310that includes groups of shared event detection and image sensing pixels501, and groups of image sensing pixels502, formed on the same light receiving chip or substrate201. The individual groups can be configured as Bayer arrays that alternate between Bayer array groups of shared event detection and image sensing pixels501, and Bayer array groups of image sensing pixels502. Accordingly,FIG.5Dis an example of an image sensor200in which different shared event detection and image sensing pixels501can respond to light within different wavelength ranges. For example, the shared event detection and image sensing pixels501can be associated with color filters. Alternatively, the shared pixels501can all receive light within the same wavelength range. Although an equal number of groups containing equal numbers of respective pixels310are depicted in the figure, other configurations are possible. As can be appreciated by one of skill in the art after consideration of the present disclosure, the shared event detection and image sensing pixels501can be selectively operated in event detection or image sensing modes. Moreover, in accordance with at least some embodiments of the present disclosure, event detection and image sensing operations can be performed simultaneously.

FIG.5Edepicts an image sensor200having an array of unit pixels310that includes groups of shared event detection and image sensing pixels501, and groups of event detection pixels503, formed on the same light receiving chip or substrate201. The individual groups of shared event detection and image sensing pixels can be configured as Bayer arrays that alternate with groups of event detection pixels503. Although an equal number of groups containing equal numbers of respective pixels310are depicted in the figure, other configurations are possible. As can be appreciated by one of skill in the art after consideration of the present disclosure, the shared event detection and image sensing pixels501can be selectively operated in event detection or image sensing modes. Moreover, in accordance with at least some embodiments of the present disclosure, event detection and image sensing operations can be performed simultaneously.

FIG.5Fdepicts an image sensor200having an array of unit pixels310that includes groups of shared event detection and image sensing pixels501, groups of image sensing pixels502, and groups of event detection pixels503, all formed on the same light receiving chip or substrate201. Some or all of the individual groups of pixels can be configured as Bayer arrays. For instance, in at least one example configuration, groups of shared event detection and image sensing pixels501and groups of image sensing pixels can be configured as Bayer arrays, while each of the event detection pixels503can be configured to receive light from within the same wavelength range. For example, the shared event detection and image sensing pixels501and the image sensing pixels can be associated with color filters, and the event detection pixels503can be without color filters. Although an arrangement in which ½ of the pixels310are shared event detection and image sensing pixels501, ¼ of the pixels310are image sensing pixels502, and ¼ of the pixels310are event detection pixels503, other configurations are possible. As can be appreciated by one of skill in the art after consideration of the present disclosure, the shared event detection and image sensing pixels501can be selectively operated in event detection or image sensing modes. Moreover, in accordance with at least some embodiments of the present disclosure, event detection and image sensing operations can be performed simultaneously.

Next, a configuration example of a unit pixel310will be described.FIG.6Ais a circuit diagram illustrating a schematic configuration example of the unit pixel310according to at least some embodiments of the present disclosure, and in particular in accordance with embodiments that include pixels310configured as combined or shared event detection (EBS) and image sensor (IS) pixels501that perform both event detection and image sensor functions. As illustrated inFIG.6A, the unit pixel310includes, for example, a pixel imaging signal generation unit (or readout circuit)320, a light-receiving unit330, and an address event detection unit (or readout circuit)400. According to at least one example embodiment, the event detection readout circuit400can trigger operation of the image signal generation readout circuit320based on charge generated by a photoelectric conversion element (or photoelectric conversion region)333and based on operation of the logic circuit210. The logic circuit210inFIG.6Ais a logic circuit including, for example, the drive circuit211, the signal processor212, and the arbiter213inFIG.3. In accordance with at least some embodiments of the present disclosure, the logic circuit can be implemented in the processor system130. As described in greater detail elsewhere herein, the logic circuit210can make determinations as to whether to trigger operation of the image signal generation readout circuit320or the operation of image signal generation circuits320associated with other unit pixels310based on the output of the event detection readout circuit400or the output of other event detection readout circuits400.

For example, the light-receiving unit330includes a first or imaging transmission transistor or gate (first transistor)331, a second or address event detection transmission transistor or gate (second transistor)332, and a photoelectric conversion element333. A first transmission or control signal TG1 transmitted from the drive circuit211is selectively supplied to a gate of the first transmission transistor331of the light-receiving unit330, and a second transmission or control signal TG2 transmitted from the drive circuit211is selectively supplied to a gate of the second transmission transistor332. An output through the first transmission transistor331of the light-receiving unit330is connected to the pixel imaging signal generation unit320, and an output through the second transmission transistor332is connected to the address event detection unit400.

The pixel imaging signal generation unit320can include a reset transistor (third transistor)321, an amplification transistor (fourth transistor)322, a selection transistor (fifth transistor)323, and a floating diffusion layer (FD)324.

In accordance with at least some embodiments of the present disclosure, the first transmission transistor331and the second transmission transistor332of the light-receiving unit330are constituted, for example, by using an N-type metal-oxide-semiconductor (MOS) transistor (hereinafter, simply referred to as “NMOS transistor”). Similarly, the reset transistor321, the amplification transistor322, and the selection transistor323of the pixel imaging signal generation unit320are each constituted, for example, by using the NMOS transistor.

The address event detection unit400can include a current-voltage conversion unit410and a subtractor430. The address event detection unit400can further be provided with a buffer, a quantizer, and a transmission unit. Details of the address event detection unit400will be described in the following description in connection withFIG.7.

In the illustrated configuration, the photoelectric conversion element333of the light-receiving unit330photoelectrically converts incident light to generate a charge. The first transmission transistor331transmits a charge generated in the photoelectric conversion element333to the floating diffusion layer324of the image signal generation readout circuit320in accordance with the first control signal TG1. The second transmission transistor332supplies an electric signal (photocurrent) based on the charge generated in the photoelectric conversion element333to the address event detection unit400in accordance with the second control signal TG2.

When an instruction for image sensing is given by the processor system130, the drive circuit211in the logic circuit210outputs the control signal TG1 for setting the first transmission transistor331of the light-receiving unit330of selected unit pixels310in the pixel array300to an ON-state. With this arrangement, a photocurrent generated in the photoelectric conversion element333of the light-receiving unit330is supplied to the pixel imaging signal generation readout circuit320through the first transmission transistor331. More particularly, the floating diffusion layer324accumulates charges transmitted from the photoelectric conversion element333through the first transmission transistor331. The reset transistor321discharges (initializes) the charges accumulated in the floating diffusion layer324in accordance with a reset signal transmitted from the drive circuit211. The amplification transistor322allows a pixel signal of a voltage value corresponding to an amount of charge accumulated in the floating diffusion layer324to appear in a vertical signal line VSL. The selection transistor323switches a connection between the amplification transistor322and the vertical signal line VSL in accordance with a selection signal SEL transmitted from the drive circuit211. Furthermore, the analog pixel signal that appears in the vertical signal line VSL is read out by the column ADC220and is converted into a digital pixel signal.

When an instruction for address event detection initiation is given by the processor system130, the drive circuit211in the logic circuit210outputs the control signal for setting the second transmission transistor332of the light-receiving unit330in the pixel array unit300to an ON-state. With this arrangement, a photocurrent generated in the photoelectric conversion element333of the light-receiving unit330is supplied to the address event detection unit400of each unit pixel310through the second transmission transistor332.

When detecting address event ignition on the basis of the photocurrent from the light-receiving unit330, the address event detection unit400of each unit pixel310outputs a request to the arbiter213. With respect to this, the arbiter213arbitrates the request transmitted from each of the unit pixels310and transmits a predetermined response to the unit pixel310that issues the request on the basis of the arbitration result. The unit pixel310that receives the response supplies a detection signal indicating the existence or nonexistence of the address event ignition (hereinafter, referred to as “address event detection signal”) to the drive circuit211and the signal processor212in the logic circuit210.

The drive circuit211can also set the second transmission transistor332in the unit pixel310that is a supply source of the address event detection signal to an OFF-state. With this arrangement, a supply of the photocurrent from the light-receiving unit330to the address event detection unit400in the unit pixel310is stopped.

Next, the drive circuit211sets the first transmission transistor331in the light-receiving unit330of the unit pixel310to an ON-state by the transmission signal TG1. With this arrangement, a charge generated in the photoelectric conversion element333of the light-receiving unit330is transmitted to the floating diffusion layer324through the first transmission transistor331. In addition, a pixel signal of a voltage value corresponding to a charge amount of charges accumulated in the floating diffusion layer324appears in the vertical signal line VSL that is connected to the selection transistor323of the pixel imaging signal generation unit320.

As described above, in the image sensor200, a pixel signal SIG is output from the unit pixel310in which the address event ignition is detected to the column ADC220. In accordance with further embodiments of the present disclosure, a pixel signal is output from the unit pixels310within a group or sub array of unit pixels310associated with the address of the unit pixel310from which an address event detection signal has been provided.

Furthermore, for example, the light-receiving unit330, the pixel imaging signal generation unit320, and two log (LG) transistors (sixth and seventh transistors)411and414and two amplification transistors (eighth and ninth transistors)412and413in the current-voltage conversion unit410of the address event detection unit400are disposed, for example, in the light-receiving chip201illustrated inFIG.2, and other components can be disposed, for example, in the logic chip202that is joined to the light-receiving chip201through the Cu—Cu joining. Therefore, in the following description, in the unit pixel310, configurations which are disposed in the light-receiving chip201are referred to as “upper layer circuit”.

A configuration example of a group of unit pixels310configured as image sensing pixels502with a shared pixel imaging signal generation readout circuitry320in accordance with at least some embodiments of the present disclosure is depicted inFIG.6B. In this example, each photoelectric conversion element333is selectively connected to the floating diffusion324via a respective transfer gate331. In addition, the components of the pixel imaging signal readout circuit320are shared by the photoelectric conversion units333. In this example, four photoelectric conversion units333a-333d, and four corresponding transfer gates331a-331d, are shown. However, any number of photoelectric conversion units333and respective transfer gates331can be included in connection with a shared pixel imaging signal readout circuit320.

A configuration example of a unit pixel310configured as a single function address event detection pixel503and associated address event detection readout circuit400elements is depicted inFIG.6C. As shown, this example includes a single photoelectric conversion element333selectively connected by a transfer gate332to components of an address event detection readout circuit400. An event scan control block415controls operation of the address event detection readout circuit400.

FIG.7is a block diagram illustrating a schematic configuration example of the address event detection unit400according to at least some embodiments of the present disclosure. As illustrated inFIG.7, the address event detection unit400includes a current-voltage conversion unit410, a buffer420, a subtractor430, a quantizer440, and a transmission unit450. The current-voltage conversion unit410converts the photocurrent from the light-receiving unit330into a voltage signal and supplies the voltage signal generated through the conversion to the buffer420. The buffer420corrects the voltage signal transmitted from the current-voltage conversion unit410, and outputs a voltage signal after correction to the subtractor430. The subtractor430lowers a voltage level of the voltage signal transmitted from the buffer420in accordance with a row drive signal transmitted from the drive circuit211and, supplies the lowered voltage signal to the quantizer440. The quantizer440quantizes the voltage signal transmitted from the subtractor430into a digital signal, and outputs the digital signal generated through the quantization to the transmission unit450as a detection signal. The transmission unit450transmits the detection signal transmitted from the quantizer440to the signal processor212and the like. For example, when address event ignition is detected, the transmission unit450supplies a request for transmission of an address event detection signal from the transmission unit450to the drive circuit211and the signal processor212to the arbiter213. In addition, when receiving a response with respect to the request from the arbiter213, the transmission unit450supplies the detection signal to the drive circuit211and the signal processor212.

The current-voltage conversion unit410in the configuration illustrated inFIG.7can include the two LG transistors411and414, the two amplification transistors412and413, and a constant-current circuit415as illustrated inFIG.6A. For example, a source of the LG transistor411and a gate of the amplification transistor413are connected to a drain of the second transmission transistor332of the light-receiving unit330. In addition, for example, a drain of the LG transistor411is connected to a source of the LG transistor414and a gate of the amplification transistor412. For example, a drain of the LG transistor414is connected to a power supply terminal VDD. In addition, for example, a source of the amplification transistor413is grounded, and a drain thereof is connected to a gate of the LG transistor411and a source of the amplification transistor412. For example, a drain of the amplification transistor412is connected to a power supply terminal VDD through the constant-current circuit415. For example, the constant-current circuit415is constituted by a load MOS transistor such as a p-type MOS transistor. In this connection relationship, a loop-shaped source follower circuit is constructed. With this arrangement, a photocurrent from the light-receiving unit330is converted into a voltage signal in a logarithmic value corresponding to a charge amount thereof. Furthermore, the LG transistors411and414, and the amplification transistors412and413may be each constituted, for example, by an NMOS transistor.

FIG.8is a circuit diagram illustrating a schematic configuration example of the subtractor430and the quantizer440according to at least some embodiments of the present disclosure. As illustrated inFIG.8, the subtractor430includes capacitors431and433, an inverter432, and a switch434. In addition, the quantizer440includes a comparator441. One end of the capacitor431is connected to an output terminal of the buffer420, and the other end is connected to an input terminal of the inverter432. The capacitor433is connected to the inverter432in parallel. The switch434opens or closes a route connecting both ends of the capacitor433in accordance with a row drive signal. The inverter432inverts a voltage signal that is input through the capacitor431. The inverter432outputs an inverted signal to a non-inverting input terminal (+) of the comparator441. When the switch434is turned on, a voltage signal Vinit is input to a buffer420side of the capacitor431. In addition, the opposite side becomes a virtual ground terminal. A potential of the virtual ground terminal is set to zero for convenience. At this time, when a capacity of the capacitor431is set as C1, a potential Qinit that is accumulated in the capacitor431is expressed by the following Expression (1). On the other hand, both ends of the capacitor433are short-circuited, and thus an accumulated charge thereof becomes zero.
Qinit=C1×Vinit  (1)

Next, when considering a case where the switch434is turned off, and a voltage of the capacitor431on the buffer420side varies and reaches Vafter, a charge Qafter accumulated in the capacitor431is expressed by the following Expression (2).
Qafter=C1×Vafter  (2)

On the other hand, when an output voltage is set as Vout, a charge Q2 accumulated in the capacitor433is expressed by the following Expression (3).
Q2=−C2×Vout  (3)

At this time, a total charge amount of the capacitors431and433does not vary, and thus the following Expression (4) is established.
Qinit=Qafter+Q2  (4)

When Expression (1) to Expression (3) are substituted for Expression (4), the following Expression (5) is obtained.
Vout=−(C1/C2)×(Vafter−Vinit)  (5)

Expression (5) represents a subtraction operation of a voltage signal, and a gain of the subtraction result becomes C1/C2. Typically, it is desired to maximize (or alternatively, improve) the gain, and thus it is preferable to make a design so that C1 becomes large and C2 becomes small. On the other hand, when C2 is excessively small, kTC noise increases, and thus there is a concern that noise characteristics deteriorate. Accordingly, a reduction in the capacity of C2 is limited to a range capable of permitting noise. In addition, since the address event detection unit400including the subtractor430is mounted for every unit pixel310, a restriction on an area is present in capacities C1 and C2. Values of the capacities C1 and C2 are determined in consideration of the restriction.

The comparator441compares a voltage signal transmitted from the subtractor430and a predetermined threshold voltage Vth that is applied to an inverting input terminal (−). The comparator441outputs a signal indicating the comparison result to the transmission unit450as a detection signal. In addition, when a conversion gain by the current-voltage conversion unit410is set as CGlog, and a gain of the buffer420is set to “1”, a gain A of the entirety of the address event detection unit400is expressed by the following Expression (6).

[Math.1]A=C⁢Glog·C⁢1C⁢2⁢∑n=1Niphoto⁢_n(6)

In Expression (6), iphoto_n represents a photocurrent of an nth unit pixel310, and a unit thereof is, for example, an ampere (A). N represents the number of the unit pixels310in a pixel block and is “1” in this embodiment.

FIG.9is a block diagram illustrating a schematic configuration example of the column ADC according to at least some embodiments of the present disclosure. The column ADC220includes a plurality of ADCs230which are provided for every column of the unit pixels310. Each of the ADCs230converts an analog pixel signal that appears in the vertical signal line VSL into a digital signal. For example, the pixel signal is converted into a digital signal in which a bit length is greater than that of a detection signal. For example, when the detection signal is set to two bits, the pixel signal is converted into a digital signal of three or greater bits (16 bits and the like). The ADC230supplies a generated digital signal to the signal processor212.

Next, an operation of the image sensor200according to at least embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

First, an example of the operation of the image sensor200will be described by using a timing chart.FIG.10Ais a timing chart illustrating an example of the operation of the image sensor according to an embodiment of the present disclosure.

As illustrated inFIG.10A, at a timing TO, when an instruction for address event detection initiation is given by the processor system130, the drive circuit211raises the control signal TG2 applied to the gate of the second transmission transistor332of all of the light-receiving units330in the pixel array unit300to a high level. With this arrangement, the second transmission transistors332of all of the light-receiving units330enter an ON-state, and a photocurrent based on a charge generated in the photoelectric conversion element333of each of the light-receiving units330is supplied from each the light-receiving units330to each of a plurality of the address event detection units400.

In addition, in a period in which the control signal TG2 is at a high level, all of the transmission signals TG1 applied to the gate of the first transmission transistor331in each of the light-receiving units330are maintained at a low level. Accordingly, in this period, a plurality of the transmission transistors331in all of the light-receiving units330are in an OFF-state.

Next, a case where the address event detection unit400of an arbitrary unit pixel310configured to perform event detection detects address event ignition in a period in which the control signal TG2 is in a high level will be assumed. In this case, the address event detection unit400that detects the address event ignition transmits a request to the arbiter213. With respect to this, the arbiter213arbitrates the request, and returns a response for the request to the address event detection unit400that issues the request.

The address event detection unit400that receives the response raises a detection signal that is input to the drive circuit211and the signal processor212to a high level, for example, in a period of a timing T1 to a timing T2. Furthermore, in this description, it is assumed that the detection signal is a one-bit signal.

The drive circuit211to which a high-level detection signal is input from the address event detection unit400at the timing T1 lowers all control signals TG2 to a low level at a subsequent timing T2. With this arrangement, supply of a photocurrent from all of the light-receiving units330of the pixel array unit300to the address event detection unit400is stopped.

In accordance with embodiments of the present disclosure, where a determination by the processor system130is made that pixel imaging signal generation circuit320should be enabled, at the timing T2, the drive circuit211raises a selection signal SEL that is applied to a gate of the selection transistor323in the pixel imaging signal generation unit320of the unit pixel310in which the address event ignition is detected (hereinafter, referred to as “reading-out target unit pixel”) to a high level, and raises a reset signal RST that is applied to a gate of the reset transistor321of the same pixel imaging signal generation unit320to a high level for a constant pulse period, thereby discharging (initializing) charges accumulated in the floating diffusion layer324of the pixel imaging signal generation unit320. In this manner, a voltage, which appears in the vertical signal line VSL in a state in which the floating diffusion layer324is initialized, is read out by the ADC230connected to the vertical signal line VSL in the column ADC220as a reset-level pixel signal (hereinafter, simply referred to as “reset level”), and is converted into a digital signal.

Next, at a timing T3 after reading out the reset level, the drive circuit211applies a transmission signal TRG of a constant pulse period to the gate of the first transmission transistor331of the light-receiving unit330in the reading-out target unit pixel310. With this arrangement, a charge generated in the photoelectric conversion element333of the light-receiving unit330is transmitted to the floating diffusion layer324in the pixel imaging signal generation unit320, and a voltage corresponding to charges accumulated in the floating diffusion layer324appears in the vertical signal line VSL. In this manner, the voltage that appears in the vertical signal line VSL is read out by the ADC230connected to the vertical signal line VSL in the column ADC220as a signal-level pixel signal of the light-receiving unit330(hereinafter, simply referred to as “signal level”) and is converted into a digital value.

The signal processor212executes CDS processing in which a difference between the reset level and the signal level which are read out as described above is obtained as a net pixel signal corresponding to a light-reception amount of the photoelectric conversion element333.

Next, at a timing T4, the drive circuit211lowers the selection signal SEL that is applied to the gate of the selection transistor323in the pixel imaging signal generation readout circuit320of the reading-out target unit pixel310to a low level, and raises the control signal TG2 that is applied to the gate of the second transmission transistor332of all of the light-receiving units330in the pixel array unit300to a high level. With this arrangement, address event ignition detection in all of the light-receiving units330in the pixel array unit300is restarted.

FIG.10Bis a timing chart illustrating an example of an operation of an image sensor in accordance with other embodiments of the present disclosure. At a timing TO, when an instruction for address event detection initiation is given by the processor system130, the drive circuit211raises the control signal TG2 applied to the gate of the transmission transistor332associated with photoelectric conversion elements333of selectively activated address event detection units400. More particularly, some or all of the address event detection units400may be activated.

In addition, the transmission signal TG1 applied to the gates of the first transmission transistors331are maintained in a low level. Accordingly, the associated transmission transistors331are in an OFF-state.

In this example, an arbitrary address event detection unit400detects address event ignition at a time T1 during which the control signal TG2 is at a high level, and the associated transmission transistor332is in an ON-state. In response to the event trigger, image frame capture begins. The image frame capture can be a full frame image capture that involves all of the image sensing pixels502included in the pixel array300. Alternatively, an event detection by a particular event detection unit400can operate as a trigger for image capture of by a set of image sensing pixels502in a vicinity of the event detection unit400, or otherwise associated with the event detection unit400. Readout of signals obtained by the image sensing pixels can then be performed. Moreover, as discussed elsewhere herein, the processor system130can operate to control the frame rate of enabled image sensing pixels502or circuits320.

FIG.11illustrates aspects of the operation of an imaging device100in accordance with embodiments of the present disclosure according to at least some embodiments of the present disclosure. Initially, the imaging device100may be monitoring a scene (step1100) in a EBS mode. In at least some operating scenarios, monitoring a scene in EBS mode includes one or more pixels outputting EBS data to a processor in communication with the imaging device.

As the imaging device100monitors the scene, the EBS data output by the pixels may be analyzed by a processor (step1104). The processor may be configured to be capable of analyzing EBS data to detect changes in light intensity within the scene. As can be appreciated by one of skill in the art after consideration of the present disclosure, the shared event detection and image sensing501or address event detection503pixels can be operated such that events, in the form of changes in light intensity within the scene are detected. Moreover, in accordance with at least some embodiments of the present disclosure, the imaging device100may be operated to detect events continuously.

The detection operation can be performed by the drive circuit211, and/or through execution of application programming by the processor system130. As can be appreciated by one of skill in the art after consideration of the present disclosure, events are generally indicated by signals output from one or more event detection pixels501,503within the pixel array300.

In analyzing the EBS data, the processor may be capable of detecting a triggering event. A triggering event may be detected by the processor by identifying one or more of a plurality of possible patterns or otherwise event associated information in EBS data. For example, a triggering event may be detected by monitoring event density in EBS data and determining that an object exists. In some embodiments, EBS data may be used as an input to a neural network which may output a decision as to whether a triggering event has occurred. For example, a neural network may be trained to detect a face or a set of desired object categories in input EBS data or to otherwise detect a set of meaningful events in input EBS data.

If a triggering event is detected at step1108, the processor may generate a signal to switch the sensor into RGB mode in step1112. If a triggering event is not detected at step1108, the method may return to step1104in which EBS data is analyzed.

After a triggering event is detected at step1108, the RGB mode may be activated in step1112. In some embodiments, after the triggering event has been detected, a determination can be made relating to parameters that should be applied in collecting image data in the RGB mode. For example, the imaging system100can be operated to activate the entire frame to collect data in the RGB mode, or to activate a region of the frame to collect data in the RGB mode. In another example, the imaging system100can be operated to collect image data at a particular frame rate.

A determination can then be made as to whether to discontinue image sensing operations (step1116). In accordance with embodiments of the present disclosure, the acquisition of image information can continue for a predetermined period of time or until a predetermined number of frames of image data have been acquired. Accordingly, the acquisition of image information can be discontinued after an initial image or set of images has been acquired. In accordance with still other embodiments of the present disclosure, image information can continue to be acquired for as long as a detected object remains within the field of view114of the imaging system100. The acquisition of image information related to an object can be discontinued after the object is determined to have left the field of view of the imaging device100. As yet another alternative, the acquisition of image information related to an object can be continued until sufficient image information has been acquired to allow application programming executed by the processor system130of the imaging system104of an associated system, to perform object recognition and to determine that image acquisition operations associated with that object can be discontinued.

After a determination that image sensing operations can be discontinued, a determination can next be made as to whether operation of the image sensor system100should be discontinued (step1120). If operation is to continue, the process can involve switching from the RGB mode back to the EBS mode in step1124before returning to step1104. Otherwise, the operation can end at step1128.

FIGS.12A,12B, and12Care block diagrams illustrating a variety of systems for switching between EBS pixel signals and RGB pixel signals. As discussed above in relation toFIGS.5A-5D, various configurations of an imaging device100may be implemented in various embodiments. For example, as illustrated inFIG.5A, an image sensor200may have a first or EBS sensor530and a second or imaging sensor540. As illustrated inFIG.5B, an image sensor200may have pixels310configured as combined or shared event detection and image sensing pixels501which may be selectively operated in event detection or image sensing modes. As illustrated inFIG.5C, an image sensor200may have an array of unit pixels310including a plurality of event detection pixels503and a plurality of image sensing pixels502. No matter the type of image sensor200being used, the switching between event detection or EBS mode and the image sensing or RGB mode may be implemented with a switching system as illustrated inFIGS.12A,12B, and12C.

As can be appreciated inFIG.12A, EBS pixel data may be output by an EBS sensor1200and RGB pixel data may be output by an image sensor1204as described above in relation toFIGS.5A-5F. The EBS pixel data and RGB pixel data may be output simultaneously or separately depending on implementation. EBS pixel data may be input into an EBS event analysis system such as a processor or CPU1220in communication with the image sensor200as well as a computer system executing a neural network1212. In some embodiments, the CPU1220may be capable of executing the neural network itself and thus a separate neural network1212may not be necessary.

The neural network1212ofFIG.12Amay implement a convolutional neural network or some other type of analysis algorithm. The neural network1212may be capable of controlling a switch1208. In some embodiments, the switch1208may be controlled by the CPU1220. The switch1208may be, for example, a transistor. The switch1208may control the flow of data from the EBS pixels and the RGB pixels to an output circuit1216. In this way, the neural network1212may be capable of analyzing data from the EBS sensor1200and, based on analysis of the EBS pixel data, control whether EBS pixel data or RGB pixel data is output from the imaging device100. The neural network1212and/or CPU1220may be capable of controlling a frame rate or other data capture quality variable of the EBS sensor1200and/or RGB or image sensor1204via a feedback system1224. The entire frame of RGB data may be sent to the output circuit, or a region of the RGB frame may be sent to the output circuit. The frame rate or other data capture quality variable may be altered based on qualities of any detected object. For example, a faster object may warrant increasing a frame rate.

Switching logic may be used to switch a sensor from a EBS data mode to an RGB data mode and vice versa. In some embodiments, EBS data may be analyzed by a computer system capable of controlling a switch to switch the EBS/RGB switchable sensor between EBS and RGB mode. Analysis may be performed through a neural network or another method of data analysis. Depending on decision logic, an output circuit may output either EBS or RGB data from the sensor.

For example, a processor may be configured to process an output from a sensor operating in a EBS mode and/or a sensor operating in an RGB mode. The processor may be configured to output an event signal based on EBS data and/or output an image signal based on RGB data. The processor may further be configured to select between the EBS mode and RGB mode based on processing of EBS and/or RGB data.

As illustrated inFIGS.12B and12C, a single sensor with capabilities for both EBS data and image data may be used. For example, as illustrated inFIG.12B, EBS pixel data and RGB pixel data may be output by a sensor1228with both EBS pixels and RGB pixels as described above in relation toFIGS.5A-5F. The EBS pixel data and RGB pixel data may be output simultaneously or separately depending on implementation. EBS pixel data may be input into a EBS event analysis system such as a processor or CPU1220in communication with the image sensor200as well as a computer system executing a neural network1212. In some embodiments, the CPU1220may be capable of executing the neural network itself and thus a separate neural network1212may not be necessary.

The neural network1212ofFIG.12Bmay implement a convolutional neural network or some other type of analysis algorithm. The neural network1212may be capable of controlling a switch1208. In some embodiments, the switch1208may be controlled by the CPU1220. The switch1208may be, for example, a transistor. The switch1208may control the flow of data from the EBS pixels and the RGB pixels to an output circuit1216. In this way, the neural network1212may be capable of analyzing data from the sensor1228and, based on analysis of the EBS pixel data, control whether EBS pixel data or RGB pixel data is output from the imaging device100. The neural network1212and/or CPU1220may be capable of controlling a frame rate or other data capture quality variable of the sensor1200via a feedback system1224. The frame rate or other data capture quality variable may be altered based on qualities of any detected object. For example, a faster object may warrant increasing a frame rate.

In an alternative embodiments, as illustrated inFIG.12C, EBS pixel data and RGB pixel data may be output by a sensor1232with pixels capable of generating both EBS pixel data and RGB pixel data as described above in relation toFIGS.5A-5F. The EBS pixel data and RGB pixel data may be output simultaneously or separately depending on implementation. EBS pixel data may be input into a EBS event analysis system such as a processor or CPU1220in communication with the image sensor200as well as a computer system executing a neural network1212. In some embodiments, the CPU1220may be capable of executing the neural network itself and thus a separate neural network1212may not be necessary.

The neural network1212ofFIG.12Cmay implement a convolutional neural network or some other type of analysis algorithm. The neural network1212may be capable of controlling the sensor1232via the feedback system1224. In some embodiments, the feedback system1224may be controlled by the CPU1220. The feedback system1224may control the flow of data from the EBS pixels and the RGB pixels to an output circuit1216. In this way, the neural network1212may be capable of analyzing data from the sensor1228and, based on analysis of the EBS pixel data, control whether EBS pixel data or RGB pixel data is output from the imaging device100.

Whether an event warrants switching from EBS to RGB depends on the application. Many methods of switching which support a low power design may be used and certain embodiments may be as described herein.

For example, depending on application, one or more of the following methods may be used to determine when and whether to switch from EBS to RGB mode: a detection of a high EBS event density, detection of a low EBS event density, analysis of EBS data by a neural network, analysis of EBS data by a recurrent neural network, detection of EBS motion in a particular direction. It should be noted that such methods should not be considered as the only possible methods of determining when and whether to switch from EBS mode to RGB mode.

Data collected via the EBS mode may also be used to determine to send the entire frame of RGB data to the output or send a region of the RGB frame to the output. The EBS data may also be used to determine speed of an object and may be used to switch to a higher frame rate.

In one embodiment, a sensor may be switched from EBS mode to RGB mode when EBS event density exceeds a threshold amount in the entire scene or a predefined region of the scene. Such an embodiment may be useful for capturing motion. For example, a sensor set to switch from EBS mode to RGB mode based on EBS event density exceeding a threshold amount may be used to recognize a vehicle entering into a scene or to recognize a person entering a room, etc.

In some embodiments, the processor system130may be capable of using event detection data to determine a frame rate to apply to the RGB mode. The determined frame rate for the RGB mode can be based on the identity of the object as determined from the event detection data, the relative velocity of the object, or a degree of interest in an identified object. For example, a relatively high frame rate could be applied to an automobile, a moderate frame rate can be applied to a cyclist, and a relatively low frame rate can be applied to a pedestrian. A higher frame rate can be applied to an object moving at a faster apparent velocity than an object that is stationary or moving at a lower apparent velocity.

The various operations performed by the processing system130on the event detection data and/or the image data can include applying one or more neural networks to analyze the collected information.

Embodiments of the present disclosure can continue to operate event detection pixels502,503while image sensing pixels501,502are in operation. As noted elsewhere herein, event detection pixels502,503generally operate asynchronously. By continuing to operate the event detection pixels502,503, event detection functions can be performed continuously, without loss or diminution of temporal event detection performance of the imaging device100.

Accordingly, embodiments of the present disclosure provide imaging devices100with one or more pixel arrays300that are capable of performing both event detection and imaging operations. Moreover, the event detection pixels can be operated continuously, and the image sensing pixels can be operated selectively. Moreover, a frame rate applied for operation of the image sensing pixels can be selected based on characteristics of or an identification of the detected event or events. After a selected time period, after an event being imaged is no longer present, or after some other criterion has been met, operation of the image sensing pixels can be discontinued, while operation of the event detection pixels continues. Accordingly, continuous monitoring for events is provided in combination with selected imaging operations, thereby providing relevant image data while conserving power, data transmission, and data processing resources.

EBS sensors or sensors comprising EBS pixels may be capable of generating frames of data indicating changes in light intensity. For example, a positive change in light intensity may be reflected in a frame by a pixel of a value such as +1 or a particular color such as red. A negative change in light intensity may similarly be reflected in a frame by pixel of a particular value such as −1 or of another color such as blue. If a EBS pixel does not detect a change in light intensity, a zero value or a color such as white may be used.

In some embodiments, a EBS sensor or a sensor comprising EBS pixels may be capable of indicating an amount of change in light intensity. For example, a relatively high change in light intensity may be reflected by a pixel of a value of +1.00 while a relatively low, but positive, change in light intensity may be reflected by a value of +0.01 for example. The values +1.00 and +0.01 may be represented by an 8-bit digital value of 255 and 1, respectively. Similarly, a range of colors may be used to indicate amounts of change.

However, EBS cameras provide change information and time information only. For example, data from EBS sensors corresponding to an event for a pixel may correspond to three states: −1 indicates a negative change, +1 indicates a positive change, and 0 indicates no change. Information on the time of change may also be provided. EBS cameras alone do not provide color information or shades of gray. For this reason, EBS cameras are not general purpose cameras for capturing image or video information. The above references to EBS pixels being associated with pixels of colors should not be interpreted as EBS pixels being associated with colors from a scene but instead the use of colors only as a visualization of changes in light intensity.

When operating in EBS mode, a EBS/RGB switchable sensor may operate in a relatively lower power consumption state. When operating in RGB mode, the EBS/RGB switchable sensor may operate in a relatively higher power consumption state. For this reason, the EBS mode may be used for lower power and the RGB mode may be activated, or switched to, only as needed.

Switching logic may be used to switch a sensor from a EBS data mode to an RGB data mode and vice versa. In some embodiments, switching logic may be used to only switch the RGB data on and off. In some embodiments, EBS data may be analyzed by a computer system capable of controlling a switch to switch the EBS/RGB switchable sensor between EBS and RGB mode. Analysis may be performed through a neural network or another method of data analysis. Depending on decision logic, an output circuit may output either EBS or RGB data from the sensor.

For example, a processor may be configured to process an output from a sensor operating in a EBS mode and/or a sensor operating in an RGB mode. The processor may be configured to output an event signal based on EBS data and/or output an image signal based on RGB data. The processor may further be configured to select between the EBS mode and RGB mode based on processing of EBS and/or RGB data.

Whether an event warrants switching from EBS to RGB depends on the application. Many methods of switching which support a low power design may be used and certain embodiments may be as described herein.

In some embodiments, a switch between EBS and RGB mode may be triggered based on processing of EBS frames with a convolutional neural network (“CNN”). In such an embodiment, EBS frames may be fed to a classification CNN such as a LeNet, VGG16, ResNet, etc., to a detection CNN such as a RCNN, YOLO, SSD, etc., or other types of neural network. If a specific object, such as a person, a face or vehicle, is recognized or otherwise detected with a high probability, RGB mode may be triggered to capture a color image of the object for further analysis.

If the neural network decides the probability of a certain category of object, such as a person, a face or a car, exceeds a pre-defined threshold, the RGB mode may be triggered.

For example, one or more EBS frames may be used as an input to a CNN which may output a triggering decision. In some embodiments, a single EBS frame may be used as an input to generate a triggering decision. A single EBS frame may be a collection of EBS signals collected over a particular time frame such as 1 millisecond. In some embodiments, a number of EBS frames may be used as an input. For example, a series of EBS frames taken over a given time period, for example 1 second, may be used.

A CNN may comprise a number of layers and may be trained to detect one or more types of EBS-related events. For example, a CNN may comprise a number of convolutional layers (e.g., conv1, conv2, conv3, conv4, conv5, etc.) and one or more max pooling layers. A CNN may be trained through a process of inputting EBS frames showing known events. In some embodiments, a CNN may be trained to output a triggering decision in the event of detecting EBS data showing the occurrence of a particular event. A triggering decision may be as simple as a +1 for yes and a 0 for no. In some embodiments, a triggering decision may be more complex, for example, an identification of an event type for a detected event. For example, the CNN may detect an input with EBS data showing a high number of events which exceeds a pre-defined spatio-temporal density, or the CNN may detect an input with EBS data which is recognized by the CNN as being indicative of an existence of a particular object such as a person, a face or vehicle. The triggering decision may include information about the object as detected and/or recognized by the CNN.

A block diagram of certain embodiments of the present disclosure is illustrated inFIG.13. As can be appreciated, an image system capable of receiving EBS data1300and RGB data1304as inputs may be capable of generating an output. The EBS data1300may be output from a EBS sensor or other type of sensor capable of generating EBS data1300. The EBS data1300may be input into an output circuit1308capable of processing EBS data1300for further analysis by one or more of a face detection neural network1316and/or a CPU1320. The face detection neural network1316may be a neural network such as a CNN trained to detect faces in EBS data. While the present embodiment is a system for detecting and recognizing faces, it should be appreciated that the same principles may be applied to detecting and recognizing other types of objects.

RGB data1304may be input into an output circuit1312capable of processing RGB data1304for further analysis by a facial recognition neural network1328. The facial recognition neural network1328may be a neural network such as a CNN trained to recognize faces in RGB data. Specifically, it is trained to determine the identity of the face. While the present embodiment is a system for detecting and recognizing faces, it should be appreciated that the same principles may be applied to detecting and recognizing other types of objects.

Prior to the RGB data1304being input into the facial recognition neural network, the RGB data1304may be input into an on/off logic system1324which may be controlled by one or both of the face detection neural network and/or the CPU1320. For example, the face detection neural network1316may be trained to switch the on/off logic system1324upon detecting one or more faces in the input EBS data.

When the on/off logic system1324is switched to allow RGB data1304to be input into the facial recognition neural network1328, the facial recognition neural network1328may begin processing the RGB data1304to recognize faces in the RGB data1304. The facial recognition neural network1328may then output information relating to any recognized faces into an output circuit1332. For example, the neural network may be in communication with one or more online databases which may be used to gather names or other identifying information relating to the recognized faces. Such identifying information may be output into the output circuit1332.

Using a system as illustrated inFIG.13in accordance with the systems and methods described herein, an object recognition system may be implemented. Such an object recognition system may comprise a first sensor, such as a EBS sensor configured to detect a change of an amount of light intensity, a second sensor, such as an RGB sensor configured to detect an amount of light intensity, and a processor.

The processor may be configured to process an output from the first sensor and output an event signal. For example, the processor may be capable of processing EBS data to detect the presence of one or more faces or other objects in the EBS data. If a face or other object is detected, the processor may be configured to process an object detection based on the event signal. For example, the processor may be capable of detecting an object in EBS data and then recognize what type of object or the identity of the object that has been detected. In some embodiments, the processor may be trained to specifically detect facial data. The processor may be further configured to process an object recognition based on an output from the second sensor according to the object detection. For example, upon detecting a face in the EBS data, the processor may next analyze RGB data of the same or a similar scene and make an attempt to recognize any face in the RGB data.

In some embodiments, the RGB sensor may not be activated unless and until a face (or other type of object) has been detected using EBS data. In response to detecting a face (or other type of object) the processor may be configured to activate the second sensor. Furthermore, in response to processing the object detection based on the event signal, the processor may be configured to deactivate the first sensor.

In some embodiments, after processing the object recognition based on the output from the second sensor according to the object detection, the processor may reactivate the first sensor and deactivate the second sensor.

In the above descriptions, it should be appreciated that any of the various embodiments including separate EBS and RGB sensor or single sensors capable of both EBS and RGB data may be used to implement the various systems and methods described herein.

In some embodiments, switching from EBS to RGB may be triggered based on a detected direction of motion in EBS data. For example, a predefined object recognized by a convolutional neural network or other method of detecting objects may be monitored to detect a direction of motion of the object. Depending on the detected direction of motion of the object, the sensor may be switched to RGB mode or to a high-frames-per-second (“FPS”) RGB mode.

The switch of a sensor from EBS mode to RGB mode may be for a pre-determined amount of time. For example, after switching to RGB mode, the sensor may be switched back to EBS mode after a certain number of seconds or after a certain number of image frames. In some embodiments, the RGB mode may be analyzed to determine when an event has ended, at which time the sensor may be switched back to EBS mode.

Hereinbefore, embodiments of the present disclosure have been described, but the technical range of the present disclosure is not limited to the above-described embodiments, and various modifications can be made in a range not departing from the gist of the present disclosure. In addition, constituent elements in other embodiments and modification examples may be appropriately combined.

Disclosed herein is a combination EBS and RGB camera capable of utilizing advantages of both EBS and RGB modes. A sensor as described herein normally operates in a EBS mode and switches to RGB mode when an event warrants the switch. As used herein, RGB may refer to data relating to an amount of light intensity. An RGB sensor or a sensor operating in an RGB mode may be capable of or configured to detect an amount of light intensity.

As described herein, a EBS/RGB switchable sensor may be in a variety of forms. For example, in some embodiments, separate EBS and RGB sensors may be used. In such an embodiment, the separate EBS and RGB sensors may each comprise a plurality of pixels. The separate EBS and RGB sensors may be physically connected and may share a single lens.

In some embodiments, a single sensor with a mosaic of RGB and EBS pixels may be used. For example, a single sensor may comprise a grid of pixels. The grid of pixels may be a variety of non-switchable RGB pixels and EBS pixels. The pattern of pixels may be laid out in a random fashion or may be a particular pattern. In some embodiments, the EBS pixels may be in a small section of the grid of pixels or may be spread out evenly throughout the grid.

In some embodiments, a single sensor with switchable RGB and EBS pixels may be used. For example, a sensor may comprise a grid of pixels. Each pixel may be capable of detecting both EBS and the intensity of a color. For example, a first pixel may be switchable between collecting EBS data and red color data, while a second pixel may be switchable between collecting EBS data and green color data, and a third pixel may be switchable between collecting EBS data and blue color data.

Additional embodiments may include other combinations of switchable and non-switchable pixels and/or other color mosaic patterns.

As described herein, a EBS/RGB switchable sensor may be used in one or both of a EBS mode and an RGB mode. EBS sensors are advantageous in that EBS sensors are capable of capturing event data at high rates. EBS sensors also consume relatively lower power than RGB sensors.

In addition, the effects in the embodiments described in this specification are illustrative only, and other effect may exist without a limitation.

Furthermore, the present technology can include the following configurations:

(1)

An object recognition system comprising:

a first sensor configured to detect a change of an amount of light intensity;

a second sensor configured to detect an amount of light intensity; and

a processor configured to:process an output from the first sensor and output an event signal,process an object detection based on the event signal, andprocess an object recognition based on an output from the second sensor according to the object detection.

(2)

The object recognition system of (1), wherein in response to processing the object detection based on the event signal, the processor activates the second sensor.

(3)

The object recognition system of (2), wherein in response to processing the object detection based on the event signal, the processor deactivates the first sensor.

(4)

The object recognition system of (3), wherein after processing the object recognition based on the output from the second sensor according to the object detection, the processor reactivates the first sensor and deactivates the second sensor.

(5)

The object recognition system of (1), wherein processing the object detection based on the event signal comprises detecting an object in the output.

(6)

The object recognition system of (5), wherein processing the object recognition based on the output from the second sensor according to the object detection comprises recognizing the object.

(7)

The object recognition system of (1), wherein processing the object recognition based on the output from the second sensor according to the object detection comprising recognizing a face.

(8)

An imaging system, comprising:

a first sensor configured to detect a change of an amount of light intensity; and

a second sensor configured to detect an amount of light intensity, wherein an output from the first sensor is processed by a processor to output an event signal, a first object is detected by the processor based on the event signal, and an object recognition is processed by the processor based on an output from the second sensor according to the object detection.

(9)

The imaging system of (8), wherein in response to processing the object detection based on the event signal, the processor activates the second sensor.

(10)

The imaging system of (9), wherein in response to processing the object detection based on the event signal, the processor deactivates the first sensor.

(11)

The imaging system of (10), wherein after processing the object recognition based on the output from the second sensor according to the object detection, the processor reactivates the first sensor and deactivates the second sensor.

(12)

The imaging system of (8), wherein processing the object detection based on the event signal comprises detecting an object in the output.

(13)

The imaging system of (12), wherein processing the object recognition based on the output from the second sensor according to the object detection comprises recognizing the object.

(14)

The imaging system of (8), wherein processing the object recognition based on the output from the second sensor according to the object detection comprising recognizing a face.

(15)

A method of implementing object recognition, the method comprising performing functions as follows with a processor:

processing an output from a first sensor configured to detect a change of an amount of light intensity;

outputting an event signal based on the output from the first sensor,

processing an object detection based on the event signal, and

processing an object recognition according to the object detection based on an output from a second sensor configured to detect an amount of light intensity.

(16)

The method of (15), wherein in response to processing the object detection based on the event signal, the processor activates the second sensor.

(17)

The method of (16), wherein in response to processing the object detection based on the event signal, the processor deactivates the first sensor.

(18)

The method of (17), wherein after processing the object recognition based on the output from the second sensor according to the object detection, the processor reactivates the first sensor and deactivates the second sensor.

(19)

The method of (15), wherein processing the object detection based on the event signal comprises detecting an object in the output.

(20)

The method of (19), wherein processing the object recognition based on the output from the second sensor according to the object detection comprises recognizing the object.