SOLID-STATE IMAGING DEVICE AND RECOGNITION SYSTEM

More secure authentication is enabled. A solid-state imaging device according to an embodiment includes: an image processing unit that includes a plurality of first pixels arranged in a matrix on a first surface and generates image data on the basis of a light amount of incident light incident on each of the first pixels; and an event signal processing unit that includes a plurality of second pixels arranged in a matrix on a second surface parallel to the first surface and generates event data on the basis of a luminance change of incident light incident on each of the second pixels, in which the plurality of first pixels and the plurality of second pixels are arranged on a single chip.

FIELD

The present disclosure relates to a solid-state imaging device and a recognition system.

BACKGROUND

In recent years, with the spread of portable devices such as smartphones and tablet terminals, secure authentication systems have been required.

CITATION LIST

Patent Literature

Patent Literature 1: JP 2020-21855 A

Patent Literature 2: JP 2018-125848 A

SUMMARY

Technical Problem

However, conventionally, since an authentication system based on information acquired by one sensor is common, there is room for improvement in security against unauthorized access such as impersonation.

Therefore, the present disclosure proposes a solid-state imaging device and a recognition system that enable more secure authentication.

Solution to Problem

To solve the problems described above, a solid-state imaging device according to an embodiment of the present disclosure includes: an image processing unit including a plurality of first pixels arranged in a matrix on a first surface, the image processing unit generating image data based on a light amount of incident light incident on each of the first pixels; and an event signal processing unit including a plurality of second pixels arranged in a matrix on a second surface parallel to the first surface, the event signal processing unit generating event data based on a luminance change of incident light incident on each of the second pixels, wherein the plurality of first pixels and the plurality of second pixels are arranged on a single chip.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that, in each of the following embodiments, the same parts are denoted by the same reference signs, and redundant description will be omitted.

In addition, the present disclosure will be described according to the following item order.

1. First embodiment

1.1 Functional configuration example of recognition system

1.2 System configuration example

1.3 Configuration example of image sensor

1.4 Configuration example of unit pixel

1.5 Example of circuit configuration of unit pixel

1.5.1 Modification example of circuit configuration

1.6 Cross-sectional structure example of unit pixel

1.7 Organic material

1.8 Planar structure example

1.9 Wiring example of pixel drive line

1.10 Laminated structure example of image sensor

1.11 Recognition operation example

1.12 Example of circuit configuration of EVS pixel

1.12.1 First circuit configuration example

1.12.2 Second circuit configuration example

1.12.3 Third circuit configuration example

1.12.4 Fourth circuit configuration example

1.13 Synchronization control between laser light source and image sensor

1.13.1 First example

1.13.2 Second example

1.13.3 Third example

1.13.4 Fourth example

1.13.5 Fifth example

1.13.6 Sixth example

1.13.7 Seventh example

1.14 Working and effect

2. Second embodiment

2.1 Configuration example of unit pixel

2.2 Example of circuit configuration of unit pixel

2.3 Cross-sectional structure example of unit pixel

2.4 Planar structure example

2.5 Modification example of on-chip lens

2.6 Modification example of color filter array

2.7 Working and effect

3. Specific example of electronic device

4. Application example to mobile body

1. First Embodiment

First, a solid-state imaging device (Hereinafter, referred to as an image sensor.), an electronic device, and a recognition system according to a first embodiment will be described in detail with reference to the drawings. Note that, in the present embodiment, a case where a technique according to the present embodiment is applied to a complementary metal-oxide semiconductor (CMOS) image sensor will be exemplified, but the present invention is not limited thereto. For example, the technique according to the present embodiment can be applied to various sensors including a photoelectric conversion element, such as a charge-coupled device (CCD) image sensor and a time-of-flight (ToF) sensor.

1.1 Functional Configuration Example of Recognition System

FIG.1is a block diagram illustrating a functional configuration example of a recognition system according to a first embodiment. As illustrated inFIG.1, a recognition system1000includes two types of sensor units, an RGB sensor unit1001and an EVS sensor unit1003. Furthermore, the recognition system1000includes an RGB image processing unit1002, an event signal processing unit1004, a recognition processing unit1005, and an interface (I/F) unit1006. The RGB image processing unit1002may include the RGB sensor unit1001, and the event signal processing unit1004may include the EVS sensor unit1003.

The RGB sensor unit1001includes, for example, a plurality of pixels (Hereinafter, referred to as RGB pixels.) including a color filter that transmits wavelength components of each of the three primary colors of RGB, and generates a color image (Hereinafter, referred to as an RGB image.) including color components of the three primary colors of RGB. Note that, instead of the RGB sensor unit1001, a sensor unit or the like including a plurality of pixels including a color filter that transmits wavelength components of each of the three CMY primary colors may be used.

The EVS sensor unit1003includes, for example, a plurality of pixels (Hereinafter, referred to as an EVS pixels.) including an IR filter that transmits infrared (IR) light, and outputs event data (Also referred to as event information or a detection signal.) indicating a position (Hereinafter, referred to as an address.) of a pixel where an event has been detected on the basis of whether or not each EVS pixel has detected IR light (Hereinafter, referred to as an event.). Note that, in the present embodiment, the event may include an on-event indicating that IR light has come to be detected and an off-event indicating that IR light is desired to be detected.

The RGB image processing unit1002executes predetermined signal processing such as noise removal, white balance adjustment, and pixel interpolation on RGB image data input from the RGB sensor unit1001. Furthermore, the RGB image processing unit1002may execute recognition processing or the like using the RGB image data.

On the basis of event data input from the EVS sensor unit1003, the event signal processing unit1004generates image data (Hereinafter, referred to as EVS image data.) indicating pixels in which an event has been detected. For example, the event signal processing unit1004generates EVS image data indicating a pixel in which an on-event and/or an off-event is detected on the basis of event data input within a predetermined period. Note that the event signal processing unit1004may generate the EVS image data using an address of the pixel in which the event is detected, or may generate the EVS image data using a gradation signal (pixel signal) indicating the luminance of incident light read from the pixel in which the event is detected. Furthermore, the event signal processing unit1004may execute predetermined signal processing such as noise removal on the generated EVS image data.

Using the RGB image data input from the RGB image processing unit1002and/or the EVS image data input from the event signal processing unit1004, the recognition processing unit1005executes recognition processing of an object or the like existing within an angle of view of the RGB sensor unit1001and/or the EVS sensor unit1003. For the recognition processing by the recognition processing unit1005, recognition processing such as pattern recognition, recognition processing by artificial intelligence (AI), or the like may be used. For example, deep learning using a neural network such as convolution neural network (CNN) or recurrent neural network (RNN) may be applied to the recognition processing by AI. Furthermore, the recognition processing unit1005may execute part of the recognition processing and output a result (intermediate data or the like) thereof.

The interface unit1006outputs a recognition result (including intermediate data and the like) obtained by the recognition processing unit1005and image data acquired by the RGB sensor unit1001and/or the EVS sensor unit1003to an external application processor1100, for example.

Note that the event signal processing unit1004may execute region determination of an object on the EVS image data, and input information (Hereinafter, it is simply referred to as ROI information.) such as an address specifying a region of interest (ROI) obtained as a result to the RGB sensor unit1001and/or the RGB image processing unit1002. On the other hand, the RGB sensor unit1001may operate to acquire the RGB image data of the region corresponding to the ROI information input from the event signal processing unit1004. Alternatively, the RGB image processing unit1002may perform processing such as trimming of a region corresponding to the ROI information input from the event signal processing unit1004on the RGB image data input from the RGB sensor unit1001.

1.2 System Configuration Example

Next, a system configuration example of the recognition system according to the present embodiment will be described.FIG.2is a schematic diagram illustrating a schematic configuration example of an electronic device that implements the recognition system according to the first embodiment, andFIG.3is a block diagram illustrating a schematic configuration example of an electronic device that implements the recognition system according to the first embodiment.

As illustrated inFIG.2, an electronic device1according to the present embodiment includes a laser light source1010, an irradiation lens1030, an imaging lens1040, an image sensor100, and a system control unit1050.

As illustrated inFIG.3, the laser light source1010includes, for example, a vertical cavity surface emitting laser (VCSEL)1012and a light source drive unit1011that drives the VCSEL1012. However, the present invention is not limited to the VCSEL1012, and various light sources such as a light emitting diode (LED) may be used. Furthermore, the laser light source1010may be any of a point light source, a surface light source, and a line light source. In the case of a surface light source or a line light source, the laser light source1010may have, for example, a configuration in which a plurality of point light sources (for example, VCSELs) is arranged one-dimensionally or two-dimensionally. Note that, in the present embodiment, the laser light source1010may emit light of a wavelength band different from a wavelength band of visible light, such as infrared (IR) light, for example.

The irradiation lens1030is disposed on an emission surface side of the laser light source1010, and converts light emitted from the laser light source1010into irradiation light having a predetermined divergence angle.

The imaging lens1040is disposed on a light receiving surface side of the image sensor100, and forms an image by incident light on the light receiving surface of the image sensor100. The incident light can also include reflected light emitted from the laser light source1010and reflected by a subject901.

As will be described in detail later, as illustrated inFIG.3, the image sensor100includes, for example, a light receiving unit1022in which RGB pixels and EVS pixels are arranged in a two-dimensional lattice, and a sensor control unit1021that drives the light receiving unit1022to generate RGB image data and event data.

The system control unit1050includes, for example, a processor (CPU), and drives the VCSEL1012via the light source drive unit1011. Furthermore, the system control unit1050controls the image sensor100to acquire an RGB image, and controls the image sensor100in synchronization with the control on the laser light source1010to acquire event data detected according to light emission/extinction of the laser light source1010.

In such a configuration, the RGB sensor unit1001inFIG.1may be configured using the image sensor100and the system control unit1050, and the EVS sensor unit1003may be configured using the laser light source1010, the image sensor100, and the system control unit1050. Furthermore, the RGB image processing unit1002, the event signal processing unit1004, and the recognition processing unit1005inFIG.1may be configured using the image sensor100and/or the application processor1100, respectively.

For example, irradiation light emitted from the laser light source1010is projected onto the subject (also referred to as a measurement target or an object)901through the irradiation lens1030. The projected light is reflected by the subject901. Then, the light reflected by the subject901is incident on the image sensor100through the imaging lens1040. The EVS sensor unit1003in the image sensor100receives the reflected light reflected by the subject901to generate event data, and generates EVS image data on the basis of the generated event data. On the other hand, the RGB sensor unit1001in the image sensor100receives, for example, visible light in the incident light and generates RGB image data. The RGB image data and the EVS image data generated by the image sensor100are supplied to the application processor1100of the electronic device1. The application processor1100executes predetermined processing such as recognition processing on the RGB image data and the EVS image data input from the image sensor100.

1.3 Configuration Example of Image Sensor

FIG.4is a block diagram illustrating a schematic configuration example of the image sensor according to the first embodiment. As illustrated inFIG.4, the image sensor100according to the present embodiment includes, for example, a pixel array unit101, a vertical drive circuit102A, a horizontal drive circuit102B, an X arbiter104A, a Y arbiter104B, an RGB signal processing circuit103A, an EVS signal processing circuit103B, a system control circuit105, an RGB data processing unit108A, and an EVS data processing unit108B.

The pixel array unit101, the vertical drive circuit102A, the horizontal drive circuit102B, the RGB signal processing circuit103A, and the system control circuit105constitute, for example, the RGB sensor unit1001inFIG.1, and the pixel array unit101, the vertical drive circuit102A, the horizontal drive circuit102B, the X arbiter104A, the Y arbiter104B, the EVS signal processing circuit103B, the horizontal drive circuit102B, and the system control circuit105constitute, for example, the EVS sensor unit1003inFIG.1. Furthermore, the RGB signal processing circuit103A and the RGB data processing unit108A constitute, for example, the RGB image processing unit1002inFIG.1, and the EVS signal processing circuit103B and the EVS data processing unit108B constitute, for example, the event signal processing unit1004inFIG.1. Note that the recognition processing unit1005inFIG.1may be realized by the application processor1100alone, may be realized by causing the RGB data processing unit108A and the EVS data processing unit108B to cooperate with the application processor1100, or may be realized by causing the RGB data processing unit108A and the EVS data processing unit108B to cooperate with each other.

The pixel array unit101has a configuration in which unit pixels110are arranged in the row direction and the column direction, that is, in a two-dimensional lattice shape (also referred to as a matrix shape). Here, the row direction refers to an arrangement direction of pixels in a pixel row (lateral direction in drawings), and the column direction refers to an arrangement direction of pixels in a pixel column (longitudinal direction in drawings).

Each unit pixel110includes an RGB pixel10and an EVS pixel20. In the present description, the RGB pixel10and the EVS pixel20may be simply referred to as pixels, respectively. Although details of a specific circuit configuration and a pixel structure of the unit pixel110will be described later, the RGB pixel10includes a photoelectric conversion element that generates and accumulates charges according to the amount of received light, and generates a pixel signal of a voltage according to the amount of incident light. On the other hand, similarly to the RGB pixel10, the EVS pixel20includes a photoelectric conversion element that generates and accumulates a charge corresponding to the amount of received light, and when detecting incidence of light on the basis of the photocurrent flowing out of the photoelectric conversion element, outputs a request for requesting reading from itself to the X arbiter104A and the Y arbiter104B, and outputs a signal (also referred to as event data) indicating that an event has been detected according to arbitration by the X arbiter104A and the Y arbiter104B. A time stamp indicating the time when the event is detected may be added to the event data.

In the pixel array unit101, the pixel drive lines LD1and LD2are wired along the row direction for each pixel row, and the vertical signal lines VSL1and VSL2are wired along the column direction for each pixel column with respect to the matrix-like pixel array. For example, the pixel drive line LD1is connected to the RGB pixels10in each row, and the pixel drive line LD2is connected to the EVS pixels20in each row. On the other hand, for example, the vertical signal line VSL1is connected to the RGB pixels10of each column, and the vertical signal line VSL2is connected to the EVS pixels20of each column. However, the present invention is not limited thereto, and the pixel drive lines LD1and LD2may be wired so as to be orthogonal to each other. Similarly, the vertical signal lines VSL1and VSL2may be wired so as to be orthogonal to each other. For example, the pixel drive line LD1may be wired in the row direction, the pixel drive line LD2may be wired in the column direction, the vertical signal line VSL1may be wired in the column direction, and the vertical signal line VSL2may be wired in the row direction.

The pixel drive line LD1transmits a control signal for performing driving when a pixel signal is read from the RGB pixel10. The pixel drive line LD2transmits a control signal for bringing the EVS pixel20into an active state in which an event can be detected. InFIG.4, each of the pixel drive lines LD1and LD2is illustrated as one wiring, but the number is not limited to one. One end of each of the pixel drive line LD1and the pixel drive line LD2is connected to an output end corresponding to each row of the vertical drive circuit102A.

(Drive Configuration of RGB Pixels)

As will be described in detail later, each of the RGB pixels10includes a photoelectric conversion unit that photoelectrically converts incident light to generate a charge, and a pixel circuit that generates a pixel signal having a voltage value corresponding to the charge amount of the charge generated in the photoelectric conversion unit, and causes the pixel signal to appear in the vertical signal line VSL1under the control of the vertical drive circuit102A.

The vertical drive circuit102A includes a shift register, an address decoder, and the like, and drives the RGB pixels10of the pixel array unit101at the same time for all pixels or in units of rows. That is, the vertical drive circuit102A constitutes a drive unit that controls the operation of each of the RGB pixels10of the pixel array unit101together with the system control circuit105that controls the vertical drive circuit102A. Although a specific configuration of the vertical drive circuit102A is not illustrated, the vertical drive circuit generally includes two scanning systems of a reading scanning system and a sweeping scanning system.

The readout scanning system sequentially selects and scans each pixel of the pixel array unit101row by row in order to read out a signal from each pixel. The pixel signal read from each pixel is an analog signal. The sweep scanning system performs sweep scanning on a read row on which read scanning is performed by the read scanning system prior to the read scanning by an exposure time.

By the sweep scanning by the sweep scanning system, unnecessary charges are swept out from the photoelectric conversion element of each pixel of the read row, whereby the photoelectric conversion element is reset. Then, by sweeping out (resetting) unnecessary charges in the sweeping scanning system, a so-called electronic shutter operation is performed. Here, the electronic shutter operation refers to an operation of discarding charges of the photoelectric conversion element and newly starting exposure (starting accumulation of charges).

The signal read by the read operation by the read scanning system corresponds to the amount of light received after the immediately preceding read operation or electronic shutter operation. Then, a period from the read timing by the immediately preceding read operation or the sweep timing by the electronic shutter operation to the read timing by the current read operation is a charge accumulation period (also referred to as an exposure period) in each pixel.

The pixel signal output from each of the RGB pixels10of the pixel row selectively scanned by the vertical drive circuit102A is input to the RGB signal processing circuit103A through each of the vertical signal lines VSL1for each pixel column. The RGB signal processing circuit103A performs predetermined signal processing on the pixel signal output from each of the RGB pixels10of the selected row through the vertical signal line VSL1for each pixel column of the pixel array unit101, and temporarily holds the pixel signal after the signal processing.

Specifically, the RGB signal processing circuit103A performs at least noise removal processing such as correlated double sampling (CDS) processing or double data sampling (DDS) processing as signal processing. For example, the fixed pattern noise unique to the pixel such as the reset noise and the threshold variation of the amplification transistor in the pixel is removed by the CDS processing. The RGB signal processing circuit103A also has, for example, an analog-digital (AD) conversion function, converts an analog pixel signal read from the photoelectric conversion element into a digital signal, and outputs the digital signal.

The horizontal drive circuit102B includes a shift register, an address decoder, and the like, and sequentially selects a readout circuit (Hereinafter, referred to as a pixel circuit.) corresponding to a pixel column of the RGB signal processing circuit103A. By the selective scanning by the horizontal drive circuit102B, pixel signals subjected to signal processing for each pixel circuit in the RGB signal processing circuit103A are sequentially output.

Each EVS pixel20detects the presence or absence of an event based on whether or not a change exceeding a predetermined threshold has occurred in the photocurrent according to the luminance of the incident light. For example, each EVS pixel20detects that the luminance change exceeds or falls below a predetermined threshold as an event.

When detecting an event, the EVS pixel20outputs a request for requesting permission to output event data indicating the occurrence of the event to each of the X arbiter104A and the Y arbiter104B. Then, in a case where the EVS pixel20receives a response indicating the permission to output the event data from each of the X arbiter104A and the Y arbiter104B, the EVS pixel outputs the event data to the vertical drive circuit102A and the EVS signal processing circuit103B.

Furthermore, the EVS pixel20that has detected the event outputs an analog pixel signal generated by photoelectric conversion to the EVS signal processing circuit103B. That is, as a result of the arbitration by the X arbiter104A and the Y arbiter104B, the EVS pixel20permitted to read requests the vertical drive circuit102A to drive itself. On the other hand, the vertical drive circuit102A drives the EVS pixel20allowed to be read by arbitration, thereby causing the pixel signal to appear in the vertical signal line VSL2connected to the EVS pixel20.

The X arbiter104A arbitrates a request for requesting the output of the event data supplied from each of the plurality of EVS pixels20, and transmits a response based on the arbitration result (permission/non-permission of the output of the event data) and a reset signal for resetting the event detection to the EVS pixel20that has output the request.

The EVS signal processing circuit103B has an AD conversion function similarly to the RGB signal processing circuit103A, and converts an analog pixel signal read from the photoelectric conversion unit into a digital signal and outputs the digital signal. Furthermore, the EVS signal processing circuit103B may have a noise removal function such as CDS processing or DDS processing, for example, similarly to the RGB signal processing circuit103A.

In addition, the EVS signal processing circuit103B performs predetermined signal processing on the digital pixel signal obtained by the AD conversion and the event data input from the EVS pixel20, and outputs the event data and the pixel signal after the signal processing.

As described above, the change in the photocurrent generated in the EVS pixel20can also be regarded as a change in the amount of light (luminance change) incident on the photoelectric conversion unit of the EVS pixel20. Therefore, it can also be said that the event is a light amount change (luminance change) of the EVS pixel20exceeding the predetermined threshold. The event data indicating the occurrence of the event includes at least position information such as coordinates indicating the position of the EVS pixel20where the light amount change as the event has occurred. The event data can include the polarity of the light amount change in addition to the position information.

For a series of event data output at the timing when an event occurs from the EVS pixel20, as long as the interval between pieces of event data is maintained at the time when the event occurs, it can be said that the event data implicitly includes time information indicating a relative time when the event occurs.

However, when the interval between the pieces of event data is not maintained as it is at the time of occurrence of the event due to the event data being stored in the memory or the like, the time information implicitly included in the event data is lost. Therefore, before the interval between the pieces of event data is not maintained as it is at the time of occurrence of the event, the EVS signal processing circuit103B may include time information indicating a relative time at which the event such as a time stamp has occurred in the event data.

The system control circuit105includes a timing generator that generates various timing signals, and the like, and performs drive control of the vertical drive circuit102A, the horizontal drive circuit102B, the X arbiter104A, the Y arbiter104B, the RGB signal processing circuit103A, the EVS signal processing circuit103B, and the like on the basis of various timings generated by the timing generator.

Each of the RGB data processing unit108A and the EVS data processing unit108B has at least an arithmetic processing function, and performs various signal processing such as arithmetic processing on the image signal output from the RGB signal processing circuit103A or the EVS signal processing circuit103B.

The image data output from the RGB data processing unit108A or the EVS data processing unit108B may be subjected to predetermined processing in, for example, the application processor1100or the like in the electronic device1equipped with the image sensor100, or may be transmitted to the outside via a predetermined network.

Note that the image sensor100may include a storage unit for temporarily holding data necessary for signal processing in the RGB data processing unit108A and the EVS data processing unit108B, data processed by any one or more of the RGB signal processing circuit103A, the EVS signal processing circuit103B, the RGB data processing unit108A, and the EVS data processing unit108B, and the like.

1.4 Configuration Example of Unit Pixel

Next, a configuration example of the unit pixel110will be described. Note that, here, a case where the unit pixel110includes an RGB pixel10that acquires an RGB image of three primary colors of RGB and an EVS pixel20that detects an event will be described as an example. Note that, inFIG.5and the following description, when the color filters31r,31g,and31bthat transmit the light of the respective color components constituting the RGB three primary colors are not distinguished, the reference sign is31.

FIG.5is a schematic diagram illustrating a schematic configuration example of the pixel array unit according to the first embodiment. As illustrated inFIG.5, the pixel array unit101has a configuration in which the unit pixels110having a structure in which the unit pixels110including the RGB pixels10and the EVS pixels20are arranged along the incident direction of light are arranged in a two-dimensional lattice pattern. That is, in the present embodiment, the RGB pixel10and the EVS pixel20are positioned in the direction perpendicular to the arrangement direction (plane direction) of the unit pixels110, and the light transmitted through the RGB pixel10positioned on the upstream side in the optical path of the incident light is configured to be incident on the EVS pixel20positioned on the downstream side of the RGB pixel10. According to such a configuration, the photoelectric conversion unit PD2of the EVS pixel20is arranged on the surface side opposite to the incident surface of the incident light in the photoelectric conversion unit PD1of the RGB pixel10. Accordingly, in the present embodiment, the optical axes of the incident light of the RGB pixel10and the EVS pixel20arranged along the incident direction of the light coincide or substantially coincide with each other.

Note that, in the present embodiment, a case where the photoelectric conversion unit PD1constituting the RGB pixel10is made of an organic material and the photoelectric conversion unit PD2constituting the EVS pixel20is made of a semiconductor material such as silicon is exemplified, but the present invention is not limited thereto. For example, both the photoelectric conversion unit PD1and the photoelectric conversion unit PD2may be made of a semiconductor material, both the photoelectric conversion unit PD1and the photoelectric conversion unit PD2may be made of an organic material, or the photoelectric conversion unit PD1may be made of a semiconductor material, and the photoelectric conversion unit PD2may be made of an organic material. Alternatively, at least one of the photoelectric conversion unit PD1and the photoelectric conversion unit PD2may be made of a photoelectric conversion material different from the organic material and the semiconductor material.

1.5 Example of Circuit Configuration of Unit Pixel

Next, a circuit configuration example of the unit pixel110will be described.FIG.6is a circuit diagram illustrating a schematic configuration example of a unit pixel according to the first embodiment. As illustrated inFIG.6, the unit pixel110includes one RGB pixel10and one EVS pixel20.

The RGB pixel10includes, for example, a photoelectric conversion unit PD1, a transfer gate11, a floating diffusion region FD, a reset transistor12, an amplification transistor13, and a selection transistor14.

A selection control line included in the pixel drive line LD1is connected to a gate of the selection transistor14, a reset control line included in the pixel drive line LD1is connected to a gate of the reset transistor12, and a transfer control line included in the pixel drive line LD1is connected to a storage electrode (see a storage electrode37inFIG.8described later) described later of the transfer gate11. Furthermore, the vertical signal line VSL1having one end connected to the RGB signal processing circuit103A is connected to the drain of the amplification transistor13via the selection transistor14.

In the following description, the reset transistor12, the amplification transistor13, and the selection transistor14are also collectively referred to as a pixel circuit. The pixel circuit may include the floating diffusion region FD and/or the transfer gate11.

The photoelectric conversion unit PD1is made of, for example, an organic material, and photoelectrically converts incident light. The transfer gate11transfers the charge generated in the photoelectric conversion unit PD1. The floating diffusion region FD accumulates the charge transferred by the transfer gate11. The amplification transistor13causes a pixel signal having a voltage value corresponding to the charge accumulated in the floating diffusion region FD to appear in the vertical signal line VSL1. The reset transistor12releases the charge accumulated in the floating diffusion region FD. The selection transistor14selects the RGB pixel10to be read.

The anode of the photoelectric conversion unit PD1is grounded, and the cathode is connected to the transfer gate11. The transfer gate11will be described later in detail with reference toFIG.8, and includes, for example, a storage electrode37and a read electrode36. At the time of exposure, a voltage for collecting charges generated in the photoelectric conversion unit PD1to a semiconductor layer35in the vicinity of the storage electrode37is applied to the storage electrode37via the transfer control line. At the time of reading, a voltage for causing charges collected in the semiconductor layer35near the storage electrode37to flow out through the read electrode36is applied to the storage electrode37through the transfer control line.

The charge flowing out through the read electrode36is accumulated in the floating diffusion region FD configured by a wiring structure connecting the read electrode36, the source of the reset transistor12, and the gate of the amplification transistor13. Note that the drain of the reset transistor12may be connected to, for example, the power supply voltage VDD or a power supply line to which a reset voltage lower than the power supply voltage VDD is supplied.

The source of the amplification transistor13may be connected to a power supply line via, for example, a constant current circuit (not illustrated) or the like. The drain of the amplification transistor13is connected to the source of the selection transistor14, and the drain of the selection transistor14is connected to the vertical signal line VSL1.

The floating diffusion region FD converts the accumulated charge into a voltage of a voltage value corresponding to the charge amount. Note that the floating diffusion region FD may be, for example, a capacitance-to-ground. However, the floating diffusion region FD is not limited thereto, and may be a capacitance or the like added by intentionally connecting a capacitor or the like to a node where the drain of the transfer gate11, the source of the reset transistor12, and the gate of the amplification transistor13are connected.

The vertical signal line VSL1is connected to an analog-to-digital (AD) conversion circuit103aprovided for each column (that is, for each vertical signal line VSL1,) in the RGB signal processing circuit103A. The AD conversion circuit103aincludes, for example, a comparator and a counter, and converts an analog pixel signal into a digital pixel signal by comparing a reference voltage such as a single slope or a ramp shape input from an external reference voltage generation circuit (digital-to-analog converter (DAC)) with the pixel signal appearing in the vertical signal line VSL1. Note that the AD conversion circuit103amay include, for example, a correlated double sampling (CDS) circuit and the like, and may be configured to be able to reduce kTC noise and the like.

The EVS pixel20includes, for example, a photoelectric conversion unit PD2and an address event detection circuit210.

Similarly to the photoelectric conversion unit PD1, the photoelectric conversion unit PD2is made of, for example, a semiconductor material, and photoelectrically converts incident light. Although the detailed circuit configuration of the address event detection circuit210will be described later, as described above, the presence or absence of an event is detected on the basis of the change in the photocurrent flowing out of the photoelectric conversion unit PD2, and when the event is detected, a request for requesting permission to output event data indicating the occurrence of the event is output to each of the X arbiter104A and the Y arbiter104B. Then, the address event detection circuit210outputs the event data to the vertical drive circuit102A and the EVS signal processing circuit103B when receiving the response indicating the output permission of the event data from each of the X arbiter104A and the Y arbiter104B. At that time, the address event detection circuit210may include time information indicating a relative time at which the event such as a time stamp has occurred in the event data.

Similarly to the vertical signal line VSL1, the vertical signal line VSL2is connected to a signal processing circuit103bprovided for each column (that is, for each vertical signal line VSL2,) in the EVS signal processing circuit103B.

1.5.1 Modification Example of Circuit Configuration

FIG.7is a circuit diagram illustrating a schematic configuration example of a unit pixel according to a modification example of the first embodiment. As illustrated inFIG.7, an unit pixel110-1has a structure in which the RGB pixel10and the EVS pixel20are connected to a common vertical signal line VSL in the same configuration as the unit pixel110illustrated inFIG.6. The vertical signal line VSL is branched in a peripheral circuit, for example, and is connected to the AD conversion circuit103aof the RGB signal processing circuit103A or the signal processing circuit103bof the EVS signal processing circuit103B via a switch circuit131or132.

For example, the switch circuit131may be included in the RGB signal processing circuit103A or the EVS signal processing circuit103B. Furthermore, for example, the switch circuit131may be provided on the same semiconductor substrate as the pixel circuit of the RGB pixel10and/or the EVS pixel20, may be provided on a semiconductor substrate on which the signal processing circuit is arranged, or may be provided on a semiconductor substrate different from these. Furthermore, the control signal for controlling the switch circuit131may be supplied from the vertical drive circuit102A or the horizontal drive circuit102B, may be supplied from the sensor control unit1021(seeFIG.3), or may be supplied from another configuration.

According to such a configuration, it is possible to reduce the number of vertical signal lines VSL to be wired in the pixel array unit101, and thereby, it is possible to improve the quantum efficiency by increasing the light receiving area and to reduce the size and resolution of the image sensor100by improving the area efficiency.

1.6 Cross-sectional Structure Example of Unit Pixel

Next, an example of a cross-sectional structure of the image sensor100according to the first embodiment will be described with reference toFIG.8.FIG.8is a cross-sectional view illustrating a cross-sectional structure example of the image sensor according to the first embodiment. Here, a cross-sectional structure example will be described focusing on a semiconductor chip in which the photoelectric conversion units PD1and PD2in the unit pixel110are formed.

Furthermore, in the following description, a so-called back surface irradiation type cross-sectional structure in which the light incident surface is on the back surface side (opposite side to the element formation surface) of a semiconductor substrate50is exemplified, but the present invention is not limited thereto, and a so-called front surface irradiation type cross-sectional structure in which the light incident surface is on the front surface side (element formation surface side) of the semiconductor substrate50may be used. Furthermore, in the present description, a case where an organic material is used for the photoelectric conversion unit PD1of the RGB pixel10is exemplified, but as described above, one or both of an organic material and a semiconductor material (also referred to as an inorganic material) may be used as the photoelectric conversion material of each of the photoelectric conversion units PD1and PD2.

Note that, in a case where a semiconductor material is used for both the photoelectric conversion material of the photoelectric conversion unit PD1and the photoelectric conversion material of the photoelectric conversion unit PD2, the image sensor100may have a cross-sectional structure in which the photoelectric conversion unit PD1and the photoelectric conversion unit PD2are built in the same semiconductor substrate50, may have a cross-sectional structure in which a semiconductor substrate in which the photoelectric conversion unit PD1is built and a semiconductor substrate in which the photoelectric conversion unit PD2is built are bonded, or may have a cross-sectional structure in which one of the photoelectric conversion units PD1and PD2is built in the semiconductor substrate50and the other is built in a semiconductor layer formed on the back surface or the front surface of the semiconductor substrate50.

As illustrated inFIG.8, in the present embodiment, the photoelectric conversion unit PD2of the EVS pixel20is formed on the semiconductor substrate50, and the photoelectric conversion unit PD1of the RGB pixel10is provided on the back surface side (opposite side to the element formation surface) of the semiconductor substrate50. InFIG.8, for convenience of explanation, the back surface of the semiconductor substrate50is located on the upper side in the plane of drawing, and the front surface is located on the lower side.

For the semiconductor substrate50, for example, a semiconductor material such as silicon (Si) may be used. However, the semiconductor material is not limited thereto, and various semiconductor materials including compound semiconductors such as GaAs, InGaAs, InP, AlGaAs, InGaP, AlGaInP, and InGaAsP may be used.

The photoelectric conversion unit PD1of the RGB pixel10is provided on the back surface side of the semiconductor substrate50with an insulating layer53interposed therebetween. The photoelectric conversion unit PD1includes, for example, a photoelectric conversion film34made of an organic material, and a transparent electrode33and a semiconductor layer35disposed so as to sandwich the photoelectric conversion film34. The transparent electrode33provided on the upper side (Hereinafter, the upper side in the plane of drawing is an upper surface side, and the lower side is a lower surface side.) in the drawing with respect to the photoelectric conversion film34functions as, for example, an anode of the photoelectric conversion unit PD1, and the semiconductor layer35provided on the lower surface side functions as a cathode of the photoelectric conversion unit PD1.

The semiconductor layer35functioning as a cathode is electrically connected to the read electrode36formed in the insulating layer53. The read electrode36is electrically extended to the front surface (lower surface) side of the semiconductor substrate50by being connected to wirings61,62,63, and64penetrating the insulating layer53and the semiconductor substrate50. Although not illustrated inFIG.8, the wiring64is electrically connected to the floating diffusion region FD illustrated inFIG.6.

The storage electrode37is provided on the lower surface side of the semiconductor layer35functioning as a cathode with the insulating layer53interposed therebetween. Although not illustrated inFIG.8, the storage electrode37is connected to the transfer control line in the pixel drive line LD1, and as described above, at the time of exposure, a voltage for collecting charges generated in the photoelectric conversion unit PD1to the semiconductor layer35in the vicinity of the storage electrode37is applied, and at the time of readout, a voltage for causing charges collected in the semiconductor layer35in the vicinity of the storage electrode37to flow out via the read electrode36is applied.

Similarly to the transparent electrode33, the read electrode36and the storage electrode37may be transparent conductive films. For example, a transparent conductive film such as indium tin oxide (ITO) or zinc oxide (IZO) may be used for the transparent electrode33, the read electrode36, and the storage electrode37. However, the present invention is not limited thereto, and various conductive films may be used as long as the photoelectric conversion unit PD2is a conductive film capable of transmitting light in a wavelength band to be detected.

Furthermore, for the semiconductor layer35, for example, a transparent semiconductor layer such as IGZO may be used. However, the present invention is not limited thereto, and various semiconductor layers may be used as long as the photoelectric conversion unit PD2is a semiconductor layer capable of transmitting light in a wavelength band to be detected.

Moreover, as the insulating layer53, for example, an insulating film such as a silicon oxide film (SiO2) or a silicon nitride film (SiN) may be used. However, the present invention is not limited thereto, and various insulating films may be used as long as the photoelectric conversion unit PD2is an insulating film capable of transmitting light in a wavelength band to be detected.

A color filter31is provided on the upper surface side of the transparent electrode33functioning as an anode with a sealing film32interposed therebetween. The sealing film32is made of, for example, an insulating material such as silicon nitride (SiN), and may include atoms of aluminum (Al), titanium (Ti), and the like in order to prevent the atoms from diffusing from the transparent electrode33.

Although the arrangement of the color filters31will be described later, for example, a color filter31that selectively transmits light of a specific wavelength component is provided for one RGB pixel10. However, in a case where a monochrome pixel that acquires luminance information is provided instead of the RGB pixel10that acquires color information, the color filter31may be omitted.

The photoelectric conversion unit PD2of the EVS pixel20includes, for example, a p-type semiconductor region43formed in a p-well region42in the semiconductor substrate50and an n-type semiconductor region44formed near the center of the p-type semiconductor region43. The n-type semiconductor region44functions as, for example, a photoelectric conversion region that generates a charge according to the amount of incident light, and the p-type semiconductor region43functions as a region that forms a potential gradient for collecting the charge generated by photoelectric conversion into the n-type semiconductor region44.

For example, an IR filter41that selectively transmits IR light is disposed on the light incident surface side of the photoelectric conversion unit PD2. The IR filter41may be disposed, for example, in the insulating layer53provided on the back surface side of the semiconductor substrate50. By disposing the IR filter41on the light incident surface of the photoelectric conversion unit PD2, it is possible to suppress the incidence of visible light on the photoelectric conversion unit PD2, and thus, it is possible to improve the S/N ratio of IR light to visible light. This makes it possible to obtain a more accurate detection result of IR light.

For example, a fine uneven structure is provided on the light incident surface of the semiconductor substrate50in order to suppress reflection of incident light (IR light in this example). This uneven structure may be a structure called a moth-eye structure, or may be an uneven structure having a size and a pitch different from those of the moth-eye structure.

On the front surface (lower surface in the drawing) side of the semiconductor substrate50, that is, on the element formation surface side, there is provided a vertical transistor45that causes the charge generated in the photoelectric conversion unit PD2to flow out to the address event detection circuit210. The gate electrode of the vertical transistor45reaches the n-type semiconductor region44from the surface of the semiconductor substrate50, and is connected to the address event detection circuit210via the wirings65and66(part of the transfer control line of the pixel drive line LD2) formed in an interlayer insulating film56.

The semiconductor substrate50is provided with a pixel isolation part54that electrically isolates the plurality of unit pixels110from each other, and the photoelectric conversion unit PD2is provided in each region partitioned by the pixel isolation part54. For example, when the image sensor100is viewed from the back surface (upper surface in the drawing) side of the semiconductor substrate50, the pixel isolation part54has, for example, a lattice shape interposed between the plurality of unit pixels110, and each photoelectric conversion unit PD2is formed in each region partitioned by the pixel isolation part54.

For the pixel isolation part54, for example, a reflective film that reflects light such as tungsten (W) or aluminum (Al) may be used. As a result, the incident light entering the photoelectric conversion unit PD2can be reflected by the pixel isolation part54, so that the optical path length of the incident light in the photoelectric conversion unit PD2can be increased. In addition, since the pixel isolation part54has a light reflection structure, it is possible to reduce leakage of light to adjacent pixels, and thus, it is also possible to further improve image quality, distance measurement accuracy, and the like. Note that the configuration in which the pixel isolation part54has the light reflection structure is not limited to the configuration using the reflection film, and can be realized, for example, by using a material having a refractive index different from that of the semiconductor substrate50for the pixel isolation part54.

For example, a fixed charge film55is provided between the semiconductor substrate50and the pixel isolation part54. The fixed charge film55is formed using, for example, a high dielectric having a negative fixed charge so that a positive charge (hole) accumulation region is formed at an interface part with the semiconductor substrate50and generation of a dark current is suppressed. Since the fixed charge film55is formed to have a negative fixed charge, an electric field is applied to the interface with a semiconductor substrate138by the negative fixed charge, and a positive charge (hole) accumulation region is formed.

The fixed charge film55can be formed of, for example, a hafnium oxide film (HfO2film). In addition, the fixed charge film55can be formed to contain at least one of oxides such as hafnium, zirconium, aluminum, tantalum, titanium, magnesium, yttrium, and lanthanoid elements, for example.

Note thatFIG.8illustrates a case where the pixel isolation part54has a so-called full trench isolation (FTI) structure reaching from the front surface to the back surface of the semiconductor substrate50, but is not limited thereto. For example, various element isolation structures such as a so-called deep trench isolation (DTI) structure in which the pixel isolation part54is formed from the back surface or the front surface of the semiconductor substrate50to the vicinity of the middle of the semiconductor substrate50can be adopted.

A planarization film52made of a silicon oxide film, a silicon nitride film, or the like is provided on the upper surface of the color filter31. The upper surface of the planarization film52is planarized by, for example, chemical mechanical polishing (CMP), and an on-chip lens51for each unit pixel110is provided on the planarized upper surface. The on-chip lens51of each unit pixel110has such a curvature that incident light is collected in the photoelectric conversion units PD1and PD2. Note that a positional relationship among the on-chip lens51, the color filter31, the IR filter41, and the photoelectric conversion unit PD2in each unit pixel110may be adjusted according to, for example, the distance (image height) from the center of the pixel array unit101(pupil correction).

Furthermore, in the structure illustrated inFIG.8, a light shielding film for preventing obliquely incident light from leaking into the adjacent pixel may be provided. The light shielding film can be located above the pixel isolation part54provided inside the semiconductor substrate50(upstream in the optical path of the incident light). However, when pupil correction is performed, the position of the light shielding film may be adjusted according to, for example, the distance (image height) from the center of the pixel array unit101. Such a light shielding film may be provided, for example, in the sealing film32or the planarization film52. Furthermore, as a material of the light shielding film, for example, a light shielding material such as aluminum (Al) or tungsten (W) may be used.

1.7 Organic Material

In the first embodiment, when an organic semiconductor is used as the material of the photoelectric conversion film34, the layer structure of the photoelectric conversion film34can have the following structure. However, in the case of the laminated structure, the lamination order can be appropriately changed.

(1) Single-layer structure of p-type organic semiconductor

(2) Single-layer structure of n-type organic semiconductor

(3-1) Laminated structure of p-type organic semiconductor layer/n-type organic semiconductor layer

(3-2) Laminated structure of p-type organic semiconductor layer/mixed layer (bulk heterostructure) of p-type organic semiconductor and n-type organic semiconductor/n-type organic semiconductor layer

(3-3) Laminated structure of p-type organic semiconductor layer/mixed layer (bulk hetero structure) of p-type organic semiconductor and n-type organic semiconductor

(3-4) Laminated structure of n-type organic semiconductor layer/mixed layer (bulk heterostructure) of p-type organic semiconductor and n-type organic semiconductor

(4) Mixed layer of p-type organic semiconductor and p-type organic semiconductor (bulk heterostructure)

Here, examples of the p-type organic semiconductor include a naphthalene derivative, an anthracene derivative, a phenanthrene derivative, a pyrene derivative, a perylene derivative, a tetracene derivative, a pentacene derivative, a quinacridone derivative, a thiophene derivative, a thienothiophene derivative, a benzothiophene derivative, a benzothienobenzothiophene derivative, a triallylamine derivative, a carbazole derivative, a perylene derivative, a picene derivative, a chrysene derivative, a fluoranthene derivative, a phthalocyanine derivative, a subphthalocyanine derivative, a subporphyrazine derivative, a metal complex having a heterocyclic compound as a ligand, a polythiophene derivative, a polybenzothiadiazole derivative, a polyfluorene derivative, and the like.

Examples of the n-type organic semiconductor include fullerene and a fullerene derivative<for example, fullerene (higher fullerenes, endohedral fullerenes, etc.) such as C60, C70, or C74, or a fullerene derivative (for example, fullerene fluoride, PCBM fullerene compound, fullerene multimer, and the like)>, an organic semiconductor having a larger (deeper) HOMO and LUMO than a p-type organic semiconductor, and a transparent inorganic metal oxide.

The film thickness of the photoelectric conversion film34made of the organic material as described above is not limited to the following value, but may be, for example, 1×10−8m (meter) to 5×10−7m, preferably 2.5×10−8m to 3×10−7m, more preferably 2.5×10−8m to 2×10−7m, and still more preferably 1×10−7m to 1.8×10−7m. Note that the organic semiconductor is often classified into a p-type and an n-type, but the p-type means that holes are easily transported, and the n-type means that electrons are easily transported, and the organic semiconductor is not limited to the interpretation that it has holes or electrons as a majority carrier of thermal excitation like the inorganic semiconductor.

Examples of a material constituting the photoelectric conversion film34that photoelectrically converts light having a green wavelength include a rhodamine dye, a melacyanine dye, a quinacridone derivative, and a subphthalocyanine dye (subphthalocyanine derivative).

Furthermore, examples of a material constituting the photoelectric conversion film34that photoelectrically converts blue light include a coumaric acid dye, tris-8-hydroxyquinoline aluminum (Alq3), a melacyanine dye, and the like.

Moreover, examples of a material constituting the photoelectric conversion film34that photoelectrically converts red light include phthalocyanine dyes and subphthalocyanine dyes (subphthalocyanine derivatives).

Further, as the photoelectric conversion film34, a panchromatic photosensitive organic photoelectric conversion film that is sensitive to substantially all visible light from the ultraviolet region to the red region can be used.

1.8 Planar Structure Example

Next, a planar structure example of the pixel array unit according to the present embodiment will be described.FIG.9is a diagram illustrating a planar layout example of each layer of the pixel array unit according to the first embodiment, in which (A) illustrates a planar layout example of the on-chip lens51, (B) illustrates a planar layout example of the color filter31, (C) illustrates a planar layout example of the storage electrode37, and (D) illustrates a planar layout example of the photoelectric conversion unit PD2. Note that, inFIG.9, (A) to (D) illustrate planar layout examples of surfaces parallel to the element formation surface of the semiconductor substrate50. Furthermore, in the present description, a case where a 2×2 pixel Bayer array including a pixel (Hereinafter, referred to as an R pixel 10r.) that selectively detects a red (R) wavelength component, a pixel (Hereinafter, referred to as a G pixel 10g.) that selectively detects a green (G) wavelength component, and a pixel (Hereinafter, referred to as a B pixel 10b.) that selectively detects light of a blue (B) wavelength component is used as a unit array will be exemplified.

As illustrated in (A) to (D) ofFIG.9, in the present embodiment, one on-chip lens51, one color filter31, one storage electrode37, and one photoelectric conversion unit PD2are provided for one unit pixel110. Note that, in the present description, one storage electrode37corresponds to one RGB pixel10, and one photoelectric conversion unit PD2corresponds to one EVS pixel20.

As described above, in one unit pixel110, by arranging one RGB pixel10and one EVS pixel20along the traveling direction of the incident light, it is possible to improve coaxiality with respect to the incident light between the RGB pixel10and the EVS pixel20, and thus, it is possible to suppress spatial deviation occurring between the RGB image and the EVS image. Accordingly, it is possible to improve the accuracy of the results obtained by integrally processing the information (the RGB image and the EVS image) acquired by the different sensors.

1.9 Wiring Example of Pixel Drive Line

Next, wiring examples of the pixel drive line LD1connecting the RGB pixel10and the vertical drive circuit102A and the pixel drive line LD2connecting the EVS pixel20and the vertical drive circuit102A will be described.FIG.10is a plan view illustrating an example of wiring of a pixel drive line for the RGB pixel according to the first embodiment, andFIG.11is a plan view illustrating an example of wiring of a pixel drive line for the EVS pixel according to the first embodiment.

As illustrated inFIGS.10and11, for example, the pixel drive line LD1for driving the RGB pixel10and the pixel drive line LD2for driving the EVS pixel20may be wired so as to be orthogonal to each other. However, the present invention is not limited thereto, and the RGB drive line LD1and the IR drive line LD2may be wired in parallel. In this case, the pixel drive line LD1and the pixel drive line LD2may supply various control signals to the pixel array unit101from the same side or from different sides.

Furthermore,FIG.12is a plan view illustrating an example of wiring of signal lines for the EVS pixel according to the first embodiment. As illustrated inFIG.12, the X arbiter104A is connected to the EVS pixels20of each column via, for example, signal lines extending in the column direction, and the Y arbiter104B is connected to the EVS pixels20of each row via, for example, signal lines extending in the row direction.

1.10 Laminated Structure Example of Image Sensor

FIG.13is a diagram illustrating a laminated structure example of the image sensor according to the first embodiment. As illustrated inFIG.13, the image sensor100has a structure in which a pixel chip140and a circuit chip150are vertically laminated. The pixel chip140is, for example, a semiconductor chip including the pixel array unit101in which unit pixels110including an RGB pixel10and an EVS pixel20are arranged, and the circuit chip150is, for example, a semiconductor chip in which the pixel circuit and the address event detection circuit210illustrated inFIG.6are arranged.

For bonding the pixel chip140and the circuit chip150, for example, so-called direct bonding can be used, in which the bonding surfaces are flattened and the bonding surfaces are bonded to each other by an electronic force. However, the present invention is not limited thereto, and for example, so-called Cu—Cu bonding in which copper (Cu) electrode pads formed on the joint surfaces are bonded to each other, bump bonding, or the like can also be used.

Furthermore, the pixel chip140and the circuit chip150are electrically connected via a connection part such as a through-silicon via (TSV) penetrating the semiconductor substrate, for example. For the connection using the TSV, for example, a so-called twin TSV method in which two TSVs, that is, a TSV provided in the pixel chip140and a TSV provided from the pixel chip140to the circuit chip150are connected by an outer surface of the chip, a so-called shared TSV method in which both are connected by a TSV penetrating from the pixel chip140to the circuit chip150, or the like can be adopted.

However, when Cu—Cu bonding or bump bonding is used for bonding the pixel chip140and the circuit chip150, the two may be electrically connected via a Cu—Cu bonding part or a bump bonding part.

1.11 Recognition Operation Example

Next, an example of a recognition operation executed by the recognition system according to the present embodiment will be described. Note that, here, a recognition operation example of the recognition system1000described with reference toFIG.1will be described with reference to the electronic device1described with reference toFIGS.2and3. However, as described above, the recognition operation may be realized so as to be completed in the image sensor100, may be realized by processing image data acquired by the image sensor100in the application processor1100, or may be realized by executing a part of processing in the image sensor100on image data acquired by the image sensor100and executing the rest in the application processor1100.

FIG.14is a flowchart illustrating an example of a recognition operation according to the first embodiment. As illustrated inFIG.14, in the present operation, first, the system control unit1050drives the laser light source1010at a predetermined sampling period to cause the laser light source1010to emit irradiation light at a predetermined sampling period (Step S11), and drives the EVS sensor unit1003(seeFIG.1) in the image sensor100at a predetermined sampling period in synchronization with the driving of the laser light source1010to acquire EVS image data at a predetermined sampling period (Step S12).

Furthermore, the system control unit1050acquires RGB image data by driving the RGB sensor unit1001(seeFIG.1) in the image sensor100(Step S13).

Note that the acquisition of the RGB image data may be executed in parallel with the acquisition of the EVS image data, or may be executed in a period different from the acquisition period of the EVS image data. At this time, either the acquisition of the RGB image data or the acquisition of the EVS image data may be executed first. Furthermore, the RGB image data may be acquired once with respect to the acquisition of the EVS image data performed K times (K is an integer of 1 or more).

Among the RGB image data and the EVS image data acquired in this manner, the RGB image data is subjected to predetermined processing in the RGB image processing unit1002and then input to the recognition processing unit1005. Note that, in Step S11or S12, in a case where ROI information is input from the event signal processing unit1004to the RGB sensor unit1001or the RGB image processing unit1002inFIG.1, RGB image data and/or EVS image data of a region corresponding to the ROI information may be input to the recognition processing unit1005.

Next, the recognition processing unit1005executes recognition processing (first recognition processing) of an object existing within an angle of view of the image sensor100by using the input RGB image data (Step S14). As in the first embodiment, recognition processing such as pattern recognition, recognition processing by artificial intelligence, or the like may be used for the first recognition processing.

Next, the recognition processing unit1005executes recognition processing (second recognition processing) for more accurately recognizing an object existing within the angle of view using the result of the first recognition processing and the EVS image data (Step S15). For the second recognition processing, similarly to the first recognition processing, recognition processing such as pattern recognition, recognition processing by artificial intelligence, or the like may be used.

Next, the recognition processing unit1005outputs the result of the second recognition processing obtained in Step S15to the outside via the interface unit1006, for example (Step S16). Note that the recognition processing unit1005may execute a part of the first recognition processing and output the result (intermediate data or the like) to the outside, or may execute a part of the second recognition processing and output the result (intermediate data or the like).

Thereafter, a recognition system370determines whether or not to end the present operation (Step S17), and if not (NO in Step S17), returns to Step S11. On the other hand, when the processing is ended (YES in Step S17), the recognition system370ends the present operation.

1.12 Example of Circuit Configuration of EVS Pixel

Next, a specific circuit configuration of the EVS pixel20will be described with some examples. As described above, the EVS pixel20has an event detection function of detecting that the luminance change exceeds a predetermined threshold as an event.

The EVS pixel20detects whether or not an event has occurred based on whether or not the change amount of the photocurrent exceeds a predetermined threshold. The event includes, for example, an on-event indicating that the change amount of the photocurrent exceeds an upper limit threshold and an off-event indicating that the change amount falls below a lower limit threshold. Furthermore, the event data (event information) indicating the occurrence of an event includes, for example, one bit indicating a detection result of an on-event and one bit indicating a detection result of an off-event. Note that the EVS pixel20can be configured to have a function of detecting only an on-event, or can be configured to have a function of detecting only an off-event.

1.12.1 First Circuit Configuration Example

The address event detection circuit210of the EVS pixel20-1according to a first circuit configuration example has a configuration for detecting an on-event and detecting an off-event in a time division manner using one comparator.FIG.15illustrates a circuit diagram of the EVS pixel20according to the first circuit configuration example. The EVS pixel20according to the first circuit configuration example includes a photoelectric conversion unit PD2and an address event detection circuit210, and the address event detection circuit210has a circuit configuration including a light receiving circuit212, a memory capacity213, a comparator214, a reset circuit215, an inverter216, and an output circuit217. The EVS pixel20detects an on-event and an off-event under the control of the sensor control unit1021.

In the photoelectric conversion unit PD2, a first electrode (anode electrode) is connected to an input terminal of the light receiving circuit212, a second electrode (cathode electrode) is connected to a ground node which is a reference potential node, and the photoelectric conversion unit PD2photoelectrically converts incident light to generate a charge of a charge amount corresponding to intensity (light amount) of light. Furthermore, the photoelectric conversion unit PD2converts the generated charge into a photocurrent Iphoto.

The light receiving circuit212converts the photocurrent Iphotoaccording to the intensity (light amount) of light detected by the photoelectric conversion unit PD2into a voltage Vpr. Here, a relationship between the voltage Vprand the light intensity is usually a logarithmic relationship. That is, the light receiving circuit212converts the photocurrent Iphotocorresponding to the intensity of light applied to the light receiving surface of the photoelectric conversion unit PD2into a voltage Vprthat is a logarithmic function. However, the relationship between the photocurrent Iphotoand the voltage Vpris not limited to the logarithmic relationship.

The voltage Vpraccording to the photocurrent Iphotooutput from the light receiving circuit212passes through the memory capacity213and then becomes an inversion (−) input which is the first input of the comparator214as the voltage Vdiff. The comparator214is usually configured by a differential pair transistor. The comparator214uses the threshold voltage Vbprovided from the sensor control unit1021as a non-inverting (+) input that is the second input, and detects an on-event and an off-event in a time division manner. Furthermore, after the detection of the on-event/off-event, the reset circuit215resets the EVS pixel20.

The sensor control unit1021outputs the voltage Vonas the threshold voltage Vbin a time division manner at a stage of detecting an on-event, outputs the voltage Voffat a stage of detecting an off-event, and outputs the voltage Vresetat a stage of resetting. The voltage Vresetis set to a value between the voltage Vonand the voltage Voff, preferably an intermediate value between the voltage Vonand the voltage Voff. Here, the “intermediate value” means to include not only a case where the value is strictly an intermediate value but also a case where the value is substantially an intermediate value, and existence of various variations caused by design or manufacturing is allowed.

Furthermore, the sensor control unit1021outputs an ON selection signal to the EVS pixel20at a stage of detecting an on-event, outputs an OFF selection signal at a stage of detecting an off-event, and outputs a global reset signal (Global Reset) at a stage of performing reset. The ON selection signal is provided as a control signal to a selection switch SWonprovided between the inverter216and the output circuit217. The OFF selection signal is provided as a control signal to a selection switch SWoffprovided between the comparator214and the output circuit217.

In a stage of detecting an on-event, the comparator214compares the voltage Vonwith the voltage Vdiff, and when the voltage Vdiffexceeds the voltage Von, outputs on-event information On indicating that the change amount of the photocurrent Iphotoexceeds an upper limit threshold as a comparison result. The on-event information On is inverted by the inverter216and then supplied to the output circuit217through the selection switch SWon.

In the step of detecting the off-event, the comparator214compares the voltage Voffwith the voltage Vdiff, and when the voltage Vdiffbecomes lower than the voltage Voff, outputs off-event information Off indicating that the change amount of the photocurrent Iphotobecomes lower than a lower limit threshold as a comparison result. The off-event information Off is supplied to the output circuit217through the selection switch SWoff.

The reset circuit215includes a reset switch SWRS, a 2-input OR circuit2151, and a 2-input AND circuit2152. The reset switch SWRSis connected between the inverting (−) input terminal and the output terminal of the comparator214, and is turned on (closed) to selectively short-circuit between the inverting input terminal and the output terminal.

The OR circuit2151receives two inputs of the on-event information On via the selection switch SWonand the off-event information Off via the selection switch SWoff. The AND circuit2152uses the output signal of the OR circuit2151as one input, uses the global reset signal provided from the sensor control unit1021as the other input, and turns on (closes) the reset switch SWRSwhen either the on-event information On or the off-event information Off is detected and the global reset signal is in the active state.

As described above, when the output signal of the AND circuit2152becomes the active state, the reset switch SWRSshort-circuits between the inverting input terminal and the output terminal of the comparator214, and performs global reset on the EVS pixel20. As a result, the reset operation is performed only for the EVS pixel20in which the event is detected.

The output circuit217includes an off-event output transistor NM1, an on-event output transistor NM2, and a current source transistor NM3. The off-event output transistor NM1has a memory (not illustrated) for holding off-event information Off at a gate part thereof. This memory consists of the gate parasitic capacitance of the off-event output transistor NM1.

Similarly to the off-event output transistor NM1, the on-event output transistor NM2has a memory (not illustrated) for holding on-event information On at a gate part thereof. This memory consists of the gate parasitic capacitance of the on-event output transistor NM2.

In the readout stage, the off-event information Off held in the memory of the off-event output transistor NM1and the on-event information On held in the memory of the on-event output transistor NM2are transferred to a readout circuit130through the output line nRxOff and the output line nRxOn for each pixel row of the pixel array unit101when a row selection signal is provided from the sensor control unit1021to the gate electrode of the current source transistor NM3. The readout circuit130is, for example, a circuit provided in the EVS signal processing circuit103B (seeFIG.4).

As described above, the EVS pixel20according to the first circuit configuration example has an event detection function of detecting an on-event and detecting an off-event in a time-division manner under the control of the sensor control unit1021using one comparator214.

1.12.2 Second Circuit Configuration Example

The address event detection circuit210of the EVS pixel20-2according to a second circuit configuration example is an example in which detection of an on-event and detection of an off-event are performed in parallel (simultaneously) using two comparators.FIG.16illustrates a circuit diagram of the EVS pixel20according to the second circuit configuration example.

As illustrated inFIG.16, the address event detection circuit210according to the second circuit configuration example includes a comparator214A for detecting an on-event and a comparator214B for detecting an off-event. In this manner, by performing event detection using the two comparators214A and214B, the on-event detection operation and the off-event detection operation can be executed in parallel. As a result, a faster operation can be realized for the on-event and off-event detection operations.

The comparator214A for detecting an on-event is usually configured by a differential pair transistor. The comparator214A sets a voltage Vdiffcorresponding to the photocurrent Iphotoas a non-inverting (+) input which is a first input, sets a voltage Vonas a threshold voltage Vbas an inverting (−) input which is a second input, and outputs on-event information On as a comparison result between the two. The comparator214B for off-event detection is also usually configured by a differential pair transistor. The comparator214B sets a voltage Vdiffcorresponding to the photocurrent Iphotoas an inverting input which is a first input, sets a voltage Voffas a threshold voltage Vbas a non-inverting input which is a second input, and outputs off-event information Off as a comparison result between the two.

A selection switch SWonis connected between the output terminal of the comparator214A and the gate electrode of the on-event output transistor NM2of the output circuit217. A selection switch SWoffis connected between the output terminal of the comparator214B and the gate electrode of the off-event output transistor NM1of the output circuit217. On (close)/off (open) control of the selection switch SWonand the selection switch SWoffis performed by a sample signal output from the sensor control unit1021.

On-event information On that is the comparison result of the comparator214A is held in the memory of the gate part of the on-event output transistor NM2via the selection switch SWon. The memory for holding the on-event information On includes the gate parasitic capacitance of the on-event output transistor NM2. Off-event information Off that is a comparison result of the comparator214B is held in the memory of the gate part of the off-event output transistor NM1via the selection switch SWoff. The memory for holding the off-event information Off includes the gate parasitic capacitance of the off-event output transistor NM1.

The on-event information On held in the memory of the on-event output transistor NM2and the off-event information Off held in the memory of the off-event output transistor NM1are transferred to the readout circuit130through the output line nRxOn and the output line nRxOff for each pixel row of the pixel array unit101by applying a row selection signal from the sensor control unit1021to the gate electrode of the current source transistor NM3.

As described above, the EVS pixel20according to the second circuit configuration example has an event detection function of performing detection of an on-event and detection of an off-event in parallel (simultaneously) under the control of the sensor control unit1021using the two comparators214A and214B.

1.12.3 Third Circuit Configuration Example

The address event detection circuit210of the EVS pixel20-3according to a third circuit configuration example is an example of detecting only an on-event.FIG.17illustrates a circuit diagram of the EVS pixel20according to the third circuit configuration example.

The address event detection circuit210according to the third circuit configuration example includes one comparator214. The comparator214sets a voltage Vdiffcorresponding to the photocurrent Iphotoas an inverting (−) input which is a first input, and sets a voltage Vonprovided as a threshold voltage Vbfrom the sensor control unit1021as a non-inverting (+) input which is a second input, and compares both inputs to output the on-event information On as a comparison result. Here, by using an N-type transistor as a differential pair transistor constituting the comparator214, the inverter216used in the first circuit configuration example (seeFIG.17) can be made unnecessary.

On-event information On that is the comparison result of the comparator214is held in the memory of the gate part of the on-event output transistor NM2. The memory for holding the on-event information On includes the gate parasitic capacitance of the on-event output transistor NM2. The on-event information On held in the memory of the on-event output transistor NM2is transferred to the readout circuit130through the output line nRxOn for each pixel row of the pixel array unit101when a row selection signal is provided from the sensor control unit1021to the gate electrode of the current source transistor NM3.

As described above, the EVS pixel20according to the third circuit configuration example has an event detection function of detecting only the on-event information On under the control of the sensor control unit1021using one comparator214.

1.12.4 Fourth Circuit Configuration Example

The address event detection circuit210of the EVS pixel20-4according to a fourth circuit configuration example is an example of detecting only an off-event.FIG.18illustrates a circuit diagram of the EVS pixel20according to the fourth circuit configuration example.

The address event detection circuit210according to the fourth circuit configuration example includes one comparator214. The comparator214sets a voltage Vdiffcorresponding to the photocurrent Iphotoas an inverting (−) input which is a first input, and sets a voltage Voffprovided as a threshold voltage Vbfrom the sensor control unit1021as a non-inverting (+) input which is a second input, and compares both inputs to output off-event information Off as a comparison result. A P-type transistor can be used as the differential pair transistor constituting the comparator214.

Off-event information Off that is the comparison result of the comparator214is held in the memory of the gate part of the off-event output transistor NM1. The memory holding the off-event information Off includes the gate parasitic capacitance of the off-event output transistor NM1. The off-event information Off held in the memory of the off-event output transistor NM1is transferred to the readout circuit130through the output line nRxOff for each pixel row of the pixel array unit101when a row selection signal is provided from the sensor control unit1021to the gate electrode of the current source transistor NM3.

As described above, the EVS pixel20according to the fourth circuit configuration example has an event detection function of detecting only the off-event information Off under the control of the sensor control unit1021using one comparator214. Note that, in the circuit configuration ofFIG.18, the reset switch SWRSis controlled by the output signal of the AND circuit2152, but the reset switch SWRSmay be directly controlled by the global reset signal.

1.13 Synchronization Control Between Laser Light Source and Image Sensor

In the electronic device1using the image sensor100including the EVS pixel20according to the first circuit configuration example, the second circuit configuration example, the third circuit configuration example, or the fourth circuit configuration example, in the present embodiment, the laser light source1010and the image sensor100are controlled in synchronization under the control of the system control unit1050.

By synchronously controlling the laser light source1010and the image sensor100, it is possible to prevent other event information from being mixed and output in the event information caused by the motion of the subject. As the event information other than the event information caused by the motion of the subject, for example, event information caused by a change in a pattern projected on the subject or background light can be exemplified. By preventing the other event information from being output in a mixed manner among the event information caused by the motion of the subject, the event information caused by the motion of the subject can be more reliably acquired, and the process of separating the event information in the mixed state can be made unnecessary in the application processor that processes the event information.

Hereinafter, a specific example for synchronously controlling the laser light source1010and the image sensor100will be described. This synchronization control is performed by the light source drive unit1011and the sensor control unit1021under the control of the system control unit1050illustrated inFIGS.2and3.

1.13.1 First Example

A first example is an example of synchronization control in a case where the EVS pixel20is the first circuit configuration example (that is, an example in which detection of an on-event and an off-event is performed in a time division manner using one comparator.).FIG.19illustrates a flowchart of synchronization control processing according to the first example.

The sensor control unit1021globally resets a voltage Vdiffwhich is an inverting input of the comparator214, and sets a threshold voltage Vbwhich is a non-inverting input of the comparator214to a voltage Vonfor detecting an on-event (Step S101).

The global reset of the voltage Vdiffmay be performed after the event information is transferred to the readout circuit130. Furthermore, the global reset of the voltage Vdiffis performed by turning on (closing) the reset switch SWRSin the reset circuit215illustrated inFIG.15. These points are the same in each example described later.

Next, a subject (measurement target) is irradiated with light of a predetermined pattern from the laser light source1010which is a light source unit (Step S102). The laser light source1010is driven by the light source drive unit1011under the control of the system control unit1050. This point is the same in examples described later.

Next, the sensor control unit1021stores on-event information On, which is a comparison result of the comparator214, in the memory (Step S103). Here, the memory for storing the on-event information On is a gate parasitic capacitance of the on-event output transistor NM2in the output circuit217.

Next, the sensor control unit1021sets the threshold voltage Vbto the off-event detection voltage Voff(Step S104). Next, the light source drive unit1011ends the irradiation of the subject with light (Step S105). Next, the sensor control unit1021stores off-event information Off, which is a comparison result of the comparator214, in the memory (Step S106). Here, the memory for storing the off-event information Off is a gate parasitic capacitance of the off-event output transistor NM1in the output circuit217.

Thereafter, the sensor control unit1021sequentially transfers the on-event information On stored in the gate parasitic capacitance of the on-event output transistor NM2and the off-event information Off stored in the gate parasitic capacitance of the off-event output transistor NM1to the readout circuit130(Step S107).

Thereafter, the system control unit1050determines whether or not to end the present operation (Step S108). In a case where the present operation ends (YES in Step S108), the system control unit ends a series of processing for synchronization control. In a case where the present operation does not end (NO in Step S108), the system control unit returns to Step S101, and executes subsequent operations.

1.13.2 Second Example

A second example is a synchronization control example in a case where the EVS pixel20is the second circuit configuration example (that is, an example in which detection of an on-event and an off-event is performed in parallel using two comparators.).FIG.20illustrates a flowchart of synchronization control processing according to the second example.

The sensor control unit1021globally resets a voltage Vdiffwhich is an inverting input of the comparator214(Step S121). Next, the light source drive unit1011irradiates the subject with light of a predetermined pattern from the laser light source1010which is a light source unit (Step S122).

Next, the sensor control unit1021stores on-event information On, which is a comparison result of the comparator214, in the memory (Step S123). Here, the memory for storing the on-event information On is a gate parasitic capacitance of the on-event output transistor NM2in the output circuit217.

Next, the light source drive unit1011ends the irradiation of the subject with light (Step S124). Next, the sensor control unit1021stores off-event information Off, which is a comparison result of the comparator214, in the memory (Step S125). Here, the memory for storing the off-event information Off is a gate parasitic capacitance of the off-event output transistor NM1in the output circuit217.

Thereafter, the sensor control unit1021sequentially transfers the on-event information On stored in the gate parasitic capacitance of the on-event output transistor NM2and the off-event information Off stored in the gate parasitic capacitance of the off-event output transistor NM1to the readout circuit130(Step S126).

Thereafter, the system control unit1050determines whether or not to end the present operation (Step S127). In a case where the present operation ends (YES in Step S127), the system control unit ends a series of processing for synchronization control. In a case where the present operation does not end (NO in Step S127), the system control unit returns to Step S121, and executes subsequent operations.

1.13.3 Third Example

A third example is an example of synchronization control in a case where the EVS pixel20is the third circuit configuration example (that is, an example in which detection is performed only for an on-event by using one comparator.).FIG.21illustrates a flowchart of synchronization control processing according to the third example.

The sensor control unit1021globally resets a voltage Vdiffwhich is an inverting input of the comparator214(Step S141). Next, the light source drive unit1011irradiates the subject with light of a predetermined pattern from the laser light source1010which is a light source unit (Step S142).

Next, the sensor control unit1021stores on-event information On, which is a comparison result of the comparator214, in the memory (Step S143). Here, the memory for storing the on-event information On is a gate parasitic capacitance of the on-event output transistor NM2in the output circuit217. Thereafter, the sensor control unit1021sequentially transfers the on-event information On stored in the gate parasitic capacitance of the on-event output transistor NM2to the readout circuit130(Step S144).

Thereafter, the system control unit1050determines whether or not to end the present operation (Step S145). In a case where the present operation ends (YES in Step S145), the system control unit ends a series of processing for synchronization control. In a case where the present operation does not end (NO in Step S145), the system control unit returns to Step S141, and executes subsequent operations.

1.13.4 Fourth Example

A fourth example is an example of synchronization control in a case where the EVS pixel20is the fourth circuit configuration example (that is, an example in which only an off-event is detected using one comparator).FIG.22illustrates a flowchart of synchronization control processing according to the fourth example.

The sensor control unit1021globally resets a voltage Vdiffwhich is an inverting input of the comparator214(Step S161). Next, the light source drive unit1011irradiates the subject with light of a predetermined pattern from the laser light source1010which is a light source unit (Step S162).

Next, the sensor control unit1021turns on the reset switch SWRS(Step S163). Next, the light source drive unit1011ends the irradiation of the subject with light (Step S164). Next, the sensor control unit1021stores off-event information Off, which is a comparison result of the comparator214, in the memory (Step S165). Here, the memory for storing the off-event information Off is a gate parasitic capacitance of the off-event output transistor NM1in the output circuit217.

Thereafter, the sensor control unit1021sequentially transfers the off-event information Off stored in the gate parasitic capacitance of the off-event output transistor NM1to the readout circuit130(Step S166).

Thereafter, the system control unit1050determines whether or not to end the present operation (Step S167). In a case where the present operation ends (YES in Step S167), the system control unit ends a series of processing for synchronization control. In a case where the present operation does not end (NO in Step S167), the system control unit returns to Step S161, and executes subsequent operations.

1.13.5 Fifth Example

Here, as a fifth example, a pixel arrangement example in a case where ON pixels20aand OFF pixels20bare mixed in the pixel array unit101will be described. In the present description, the “ON pixel20a” is the EVS pixel20according to the third circuit configuration example, that is, a first pixel having a function of detecting only an on-event. Furthermore, the “OFF pixel20b” is the EVS pixel20according to the fourth circuit configuration example, that is, a second pixel having a function of detecting only an off-event.

FIGS.23and24illustrate pixel arrangement examples (part 1) of the ON pixel20aand the OFF pixel20baccording to the fifth example, andFIGS.25and26illustrate pixel arrangement examples (part 2) thereof. Here, in order to simplify the drawing, a pixel arrangement (pixel array) of a total of 16 pixels of four pixels in the X direction (row direction/horizontal direction) x four pixels in the Y direction (column direction/vertical direction) is illustrated. The arrangement of the EVS pixels20in the pixel array unit101may be, for example, repetition of the pixel arrangement illustrated inFIGS.23to26.

The pixel arrangement example illustrated inFIG.23has a configuration in which ON pixels20aand OFF pixels20bare alternately arranged in both the X direction and the Y direction. The pixel arrangement example illustrated inFIG.24has a configuration in which a total of four pixels of two pixels in the X direction x two pixels in the Y direction are set as blocks (units), and blocks of ON pixels20aand blocks of OFF pixels20bare alternately arranged in both the X direction and the Y direction.

The pixel arrangement example illustrated inFIG.25has an arrangement configuration in which, among a total of 16 pixels, the middle four pixels are OFF pixels20b,and the surrounding12pixels are ON pixels20a.The pixel arrangement example illustrated inFIG.26has an arrangement configuration in which, in the pixel arrangement of 16 pixels in total, the pixels in the odd column and the even row are ON pixels20a,and the remaining pixels are OFF pixels20b.

Note that the pixel arrangement of the ON pixel20aand the OFF pixel20bexemplified here is an example, and the pixel arrangement is not limited thereto.

1.13.6 Sixth Example

A sixth example is a synchronization control example (part 1) in the case of the fifth example, that is, a synchronization control example (part 1) in the case of pixel arrangement in which the ON pixels20aand the OFF pixels20bare mixed in the pixel array unit101.FIG.27illustrates a flowchart of synchronization control processing according to the sixth example.

First, the sensor control unit1021globally resets all pixels including the ON pixels20aand the OFF pixels20b(Step S201). Next, the light source drive unit1011irradiates the subject with light of a predetermined pattern from the laser light source1010which is a light source unit (Step S202). Next, the sensor control unit1021stores the on-event information On detected by the ON pixel20ain the memory (Step S203). Here, the memory for storing the on-event information On is a gate parasitic capacitance of the on-event output transistor NM2in the output circuit217.

Thereafter, the sensor control unit1021sequentially transfers the on-event information On and the off-event information Off to the readout circuit130(Step S207), and then globally resets the voltage Vdiff, which is the inverting input of the comparator214, for the pixel for which the event detection has been performed (Step S208).

Thereafter, the system control unit1050determines whether or not to end the present operation (Step S209). In a case where the present operation ends (YES in Step S209), the system control unit ends a series of processing for synchronization control. In a case where the present operation does not end (NO in Step S209), the system control unit returns to Step S202, and executes subsequent operations.

1.13.7 Seventh Example

A seventh example is a synchronization control example (part 2) in the case of the fifth example, that is, a synchronization control example (part 2) in the case of pixel arrangement in which the ON pixels20aand the OFF pixels20bare mixed in the pixel array unit101.FIG.28illustrates a flowchart of synchronization control processing according to the seventh example.

First, the sensor control unit1021globally resets all pixels including the ON pixel20aand the OFF pixel20b(Step S221). Next, the light source drive unit1011irradiates the subject with light of a predetermined pattern from the laser light source1010which is a light source unit (Step S222). Next, the sensor control unit1021stores the on-event information On detected by the ON pixel20ain the memory (Step S223). Here, the memory for storing the on-event information On is a gate parasitic capacitance of the on-event output transistor NM2in the output circuit217.

Next, the sensor control unit1021sequentially transfers the on-event information On stored in the gate parasitic capacitance of the on-event output transistor NM2in the output circuit217to the readout circuit130(Step S224), and then, turns on the reset switch SWRSof the OFF pixel20b(Step S225).

Next, the light source drive unit1011ends the irradiation of the subject with light (Step S226). Next, the sensor control unit1021stores the off-event information Off detected by the OFF pixel20bin the memory (Step S227). Here, the memory for storing the off-event information Off is a gate parasitic capacitance of the off-event output transistor NM1in the output circuit217.

Thereafter, the sensor control unit1021sequentially transfers the on-event information On and the off-event information Off to the readout circuit130(Step S228), and then globally resets the voltage Vdiff, which is the inverting input of the comparator214, for the pixel for which the event detection has been performed (Step S229).

Thereafter, the system control unit1050determines whether or not to end the present operation (Step S230). In a case where the present operation ends (YES in Step S230), the system control unit ends a series of processing for synchronization control. In a case where the present operation does not end (NO in Step S230), the system control unit returns to Step S222, and executes subsequent operations.

1.14 Working and Effect

As described above, according to the first embodiment, since it is possible to acquire a plurality of pieces of sensor information of the RGB image acquired by the RGB pixel10and the EVS image acquired by the EVS pixel20, it is possible to improve the accuracy of recognition processing using these pieces of sensor information. For example, as described above, by acquiring EVS image data in addition to RGB image data, it is possible to more accurately determine unauthorized access such as impersonation using a photograph in face authentication. Accordingly, it is possible to realize a solid-state imaging device and a recognition system that enable more secure authentication.

Furthermore, in the present embodiment, it is also possible to further improve the accuracy of the recognition processing by executing multi-stage recognition processing using a plurality of pieces of sensor information. As a result, it is possible to realize a solid-state imaging device and a recognition system that enable more secure authentication.

2. Second Embodiment

Next, a second embodiment will be described in detail with reference to the drawings. Note that, in the following description, the same configurations as those of the above-described embodiment are cited, and redundant description is omitted.

In the first embodiment described above, a case where one EVS pixel20is associated with one RGB pixel10has been described as an example. On the other hand, in the second embodiment, a case where a plurality of RGB pixels10is associated with one EVS pixel20will be described as an example.

2.1 Configuration Example of Unit Pixel

First, a configuration example of a unit pixel110A according to the present embodiment will be described. Note that, here, as in the first embodiment, a case where the unit pixel110A includes RGB pixels for acquiring an RGB image of three primary colors of RGB and an EVS pixel for acquiring an EVS image of infrared (IR) light will be described as an example. Furthermore, the RGB pixels10are arranged according to, for example, the Bayer array.

FIG.29is a schematic diagram illustrating a schematic configuration example of a unit pixel according to the second embodiment. As illustrated inFIG.29, the unit pixel110A has a structure in which one EVS pixel20is arranged in the light incident direction with respect to four RGB pixels10arranged in two rows and two columns. That is, in the present embodiment, one EVS pixel20is positioned in the direction perpendicular to the arrangement direction (planar direction) of the unit pixel110A for the four RGB pixels10, and the light transmitted through the four RGB pixels10positioned on the upstream side in the optical path of the incident light is configured to be incident on one EVS pixel20positioned on the downstream side of the four RGB pixels10. Therefore, in the present embodiment, the optical axes of the incident light of the unit array of the Bayer array including the four RGB pixels10and the EVS pixel20coincide or substantially coincide with each other.

2.2 Example of Circuit Configuration of Unit Pixel

FIG.30is a circuit diagram illustrating a schematic configuration example of a unit pixel according to the second embodiment. Note thatFIG.30is based on the unit pixel110according to the second modification described in the first embodiment with reference toFIG.6, but is not limited thereto, and may be based on an unit pixel110-3illustrated inFIG.7.

As illustrated inFIG.30, the unit pixel110A includes a plurality of RGB pixels10-1to10-N (InFIG.30, N is 4.) and one EVS pixel20. As described above, in a case where one unit pixel110A includes the plurality of RGB pixels10, one pixel circuit (reset transistor12, floating diffusion region FD, amplification transistor13, and selection transistor14) can be shared by the plurality of RGB pixels10(pixel sharing). Therefore, in the present embodiment, the plurality of RGB pixels10-1to10-N share a pixel circuit including the reset transistor12, the floating diffusion region FD, the amplification transistor13, and the selection transistor14. That is, in the present embodiment, a plurality of the photoelectric conversion units PD1and the transfer gate11are connected to the common floating diffusion region FD.

2.3 Cross-sectional Structure Example of Unit Pixel

FIG.31is a cross-sectional view illustrating a cross-sectional structure example of the image sensor according to the second embodiment. Note that, in the present description, a case where each unit pixel110A includes the four RGB pixels10arranged in two rows and two columns and one EVS pixel20will be described as an example, similarly toFIG.29. Furthermore, in the following description, an example of a cross-sectional structure thereof will be described focusing on a semiconductor chip in which the photoelectric conversion units PD1and PD2are formed in the unit pixel110A, similarly toFIG.8. Moreover, in the following description, structures similar to the cross-sectional structure of the image sensor100described with reference toFIG.8in the first embodiment are cited, and redundant description is omitted.

As illustrated inFIG.31, in the present embodiment, in a cross-sectional structure similar to the cross-sectional structure illustrated inFIG.8, the on-chip lens51, the color filter31, and the storage electrode37are divided into four in two rows and two columns (However, two out of the four are illustrated inFIG.31.), thereby configuring the four RGB pixels10. Note that the four RGB pixels10in each unit pixel110A may constitute a basic array of the Bayer array.

2.4 Planar Structure Example

FIG.32is a diagram illustrating a planar layout example of each layer of the pixel array unit according to the second embodiment, in which (A) illustrates a planar layout example of the on-chip lens51, (B) illustrates a planar layout example of the color filter31, (C) illustrates a planar layout example of the storage electrode37, and (D) illustrates a planar layout example of the photoelectric conversion unit PD2. Note that, inFIG.32, (A) to (D) illustrate planar layout examples of surfaces parallel to the element formation surface of the semiconductor substrate50.

As illustrated in (A) to (D) ofFIG.32, in the present embodiment, four on-chip lenses51, four color filters31, four storage electrodes37, and one photoelectric conversion unit PD2are provided for one unit pixel110A. Note that, in the present description, one storage electrode37corresponds to one RGB pixel10, and one photoelectric conversion unit PD2corresponds to one EVS pixel20.

As described above, in one unit pixel110, by arranging the basic array of the Bayer array including the four RGB pixels10and one EVS pixel20along the traveling direction of the incident light, it is possible to improve the coaxiality with respect to the incident light between each of the RGB pixels10and the EVS pixel20, and thus, it is possible to suppress the spatial deviation occurring between the RGB image and the EVS image. Accordingly, it is possible to improve the accuracy of the results obtained by integrally processing the information (the RGB image and the EVS image) acquired by the different sensors.

2.5 Modification Example of On-chip Lens

In the second embodiment described above, the case where one on-chip lens51is provided for one RGB pixel10has been exemplified, but the present invention is not limited thereto, and one on-chip lens may be provided for a plurality of the RGB pixels10.FIG.33is a diagram illustrating a planar layout example of each layer of a pixel array unit according to a modification example of the on-chip lens of the second embodiment, and similarly toFIG.32, (A) illustrates a planar layout example of the on-chip lens51, (B) illustrates a planar layout example of the color filter31, (C) illustrates a planar layout example of the storage electrode37, and (D) illustrates a planar layout example of the photoelectric conversion unit PD2.

In the modification example of the on-chip lens illustrated inFIG.33, as illustrated in (A), the two on-chip lenses51arrayed in the row direction in some of the unit pixels110A among the plurality of unit pixels110A are replaced with one on-chip lens251of 2×1 pixels straddling the two RGB pixels10. Furthermore, as illustrated in (B) ofFIG.33, in the two RGB pixels10sharing the on-chip lens251, the color filters31that selectively transmit the same wavelength component are provided. In the example illustrated in (B) ofFIG.33, in the upper left unit pixel110A, the color filter31bthat selectively transmits the blue (B) wavelength component originally in the Bayer array is replaced with the color filter31gthat selectively transmits the green (G) wavelength component, whereby the color filters31of the two RGB pixels10sharing the on-chip lens251are unified to the color filters31g.

Note that, for the RGB pixels10in which the color filters31are replaced in this manner, the pixel values of the wavelength components to be originally detected according to the Bayer array may be interpolated from, for example, the pixel values of surrounding pixels. For this pixel interpolation, various methods such as linear interpolation may be used.

Furthermore, in the modification example of the on-chip lens, a case where the two on-chip lenses51arranged in the row direction are made common is exemplified, but the present invention is not limited thereto, and various modifications such as a configuration in which the two on-chip lenses51arranged in the column direction are made common, a configuration in which all of the four on-chip lenses51included in one unit pixel110A are replaced with one on-chip lens, and the like can be made. In that case, the color filters31that selectively transmit the same wavelength component may be used as the color filters31of the RGB pixels10that share the on-chip lens.

Moreover, the sharing of the on-chip lens51between the adjacent RGB pixels10is not limited to the second embodiment, and can also be applied to the first embodiment.

2.6 Modification Example of Color Filter Array

Furthermore, in the above-described embodiment and the modification example thereof, the Bayer array has been exemplified as the filter array of the color filters31, but the present invention is not limited thereto. For example, various filter arrays such as a 3×3 pixel color filter array adopted in an X-Trans (registered trademark) CMOS sensor, a 4×4 pixel quad Bayer array (also referred to as a quadra array), and a 4×4 pixel color filter array (also referred to as a white RGB array) in which a white RGB color filter is combined with a Bayer array may be used.

FIG.34is a diagram illustrating a planar layout example of each layer of a pixel array unit according to a modification example of the color filter array of the second embodiment, and similarly toFIGS.32and33, (A) illustrates a planar layout example of the on-chip lens51, (B) illustrates a planar layout example of the color filter31, (C) illustrates a planar layout example of the storage electrode37, and (D) illustrates a planar layout example of the photoelectric conversion unit PD2.

In the modification example of the color filter array illustrated inFIG.34, as illustrated in (B), a quadra array of 4×4 pixels in total in which each color filter31in the Bayer array of 2×2 pixels is divided into 2×2 pixels is illustrated as the color filter array. In such a quadra array, as illustrated in (A) ofFIG.34, even in a case where the on-chip lens51is shared by two adjacent RGB pixels10, since the color filters31in these RGB pixels10are originally aligned as illustrated in (B), it is not necessary to change the array of the color filters31, and thus, there is no need to perform pixel interpolation.

2.7 Working and Effect

As described above, according to the second embodiment, the four photoelectric conversion units PD1of the four RGB pixels10and one photoelectric conversion unit PD2of one EVS pixel20are arranged in the light incident direction. Even in such a configuration, similarly to the first embodiment, it is possible to acquire a plurality of pieces of sensor information of the RGB image EVS image, and thus, it is possible to improve the accuracy of recognition processing using these pieces of sensor information. Accordingly, it is possible to realize a solid-state imaging device and a recognition system that enable more secure authentication.

Furthermore, similarly to the first embodiment, by executing multi-stage recognition processing using a plurality of pieces of sensor information, accuracy of the recognition processing can be further improved, so that it is possible to realize a solid-state imaging device and a recognition system that enable more secure authentication.

Other configurations, operations, and effects may be similar to those of the above-described embodiment, and thus detailed description thereof is omitted here.

3. Specific Example of Electronic Device

Here, a smartphone is exemplified as a specific example of an electronic device to which the recognition system of the present disclosure can be applied.FIG.35illustrates an external view of a smartphone according to a specific example of the electronic device of the present disclosure as viewed from the front side.

A smartphone300according to the present specific example includes a display unit320on a front side of a housing310. Furthermore, the smartphone300includes a light emitting unit330and a light receiving unit340in an upper part on the front side of the housing310. Note that an arrangement example of the light emitting unit330and the light receiving unit340illustrated inFIG.35is an example, and is not limited to this arrangement example.

In the smartphone300which is an example of the mobile device having the above-described configuration, the laser light source1010(VCSEL1012) in the electronic device1according to the above-described embodiment can be used as the light emitting unit330, and the image sensor100can be used as the light receiving unit340. That is, the smartphone300according to the present specific example is manufactured by using the electronic device1according to the above-described embodiment as the three-dimensional image acquisition system.

The electronic device1according to the above-described embodiment can increase the resolution of a range image without increasing the number of light sources in the array dot arrangement of the light sources. Therefore, the smartphone300according to the present specific example can have a highly accurate face recognition function (face authentication function) by using the electronic device1according to the above-described embodiment as the three-dimensional image acquisition system (face authentication system).

4. Application Example to Mobile Body

The technique according to the present disclosure can be further applied to various products. For example, the technique according to the present disclosure may be realized as a device mounted on any type of mobile body such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility, an airplane, a drone, a ship, and a robot.

An example of the vehicle control system to which the technique according to the present disclosure can be applied has been described above. The technique according to the present disclosure can be applied to the imaging section12031and the like among the configurations described above. Specifically, the imaging sections12101,12102,12103,12104,12105, and the like illustrated inFIG.37may be mounted on the vehicle12100. By applying the technique according to the present disclosure to the imaging sections12101,12102,12103,12104,12105, and the like, it is possible to improve the accuracy of results obtained by integrally processing information (for example, a color image and a monochrome image) acquired by different sensors.

Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments as it is, and various modifications can be made without departing from the gist of the present disclosure. Furthermore, components of different embodiments and modification examples may be appropriately combined.

Furthermore, the effects of each embodiment described in the present specification are merely examples and are not limited, and other effects may be provided.

Note that the present technique can also have the following configurations.

A solid-state imaging device including:an image processing unit including a plurality of first pixels arranged in a matrix on a first surface, the image processing unit generating image data based on a light amount of incident light incident on each of the first pixels; andan event signal processing unit including a plurality of second pixels arranged in a matrix on a second surface parallel to the first surface, the event signal processing unit generating event data based on a luminance change of incident light incident on each of the second pixels,wherein the plurality of first pixels and the plurality of second pixels are arranged on a single chip.
(2)

The solid-state imaging device according to (1), wherein the plurality of first pixels include an organic photoelectric conversion film.

The solid-state imaging device according to (1) or (2), wherein at least a part of the plurality of first pixels overlaps the plurality of second pixels in a first direction.

The solid-state imaging device according to (3), wherein the first direction is a direction perpendicular to a plane on which the first pixels are arranged.

The solid-state imaging device according to any one of (1) to (4), whereineach of the first pixels includes a first photoelectric conversion unit that photoelectrically converts the incident light,each of the second pixels includes a second photoelectric conversion unit that photoelectrically converts the incident light, andthe second photoelectric conversion unit is disposed on a surface side opposite to an incident surface of the incident light in the first photoelectric conversion unit.
(6)

The solid-state imaging device according to any one of (1) to (5), including:a first chip including the first pixels and the second pixels; anda second chip including a driving unit that drives the first pixels and the second pixels, and a reading unit that reads a pixel signal from the first pixels and the second pixels,wherein the first chip and the second chip are bonded to each other to constitute the single chip including a laminated structure.
(7)

The solid-state imaging device according to any one of (1) to (6), whereinthe image processing unit generates the image data based on a light amount of light of two or more wavelength components, andthe event signal processing unit generates the event data indicating a position of a second pixel in which luminance of the incident light has changed.
(8)

The solid-state imaging device according to any one of (1) to (7), whereineach of the first pixels detects the light amount of visible light included in the incident light, andeach of the second pixels detects a luminance change of infrared light included in the incident light.
(9)

The solid-state imaging device according to any one of (1) to (8), whereineach of the second pixels detects at least one of a state where the luminance of the incident light exceeds a predetermined threshold and a state where the luminance of the incident light falls below a predetermined threshold.
(10)

The solid-state imaging device according to any one of (1) to (8), whereinat least one of the plurality of second pixels detects that the luminance of the incident light exceeds a predetermined threshold, and another one of the plurality of second pixels detects that the luminance of the incident light falls below a predetermined threshold.
(11)

The solid-state imaging device according to any one of (1) to (10), whereinthe event signal processing unit includes a plurality of the first pixels for one of the second pixels in the image processing unit.
(12)

A recognition system including:the solid-state imaging device according to any one of (1) to (11); anda recognition processing unit that executes recognition processing based on the image data acquired by the image processing unit in the solid-state imaging device and the event data acquired by the event signal processing unit.
(13)

The recognition system according to (12), further including:a light source that emits light of a predetermined wavelength band; anda control unit that controls the light source and the solid-state imaging device,wherein each of the second pixels includes a wavelength selection filter that selectively transmits light of the predetermined wavelength band,the event signal processing unit generates the event data based on the luminance change of light of the predetermined wavelength band in the incident light, andthe control unit performs control to synchronize a light emission timing of the light source with a drive timing of the event signal processing unit in the solid-state imaging device.
(14)

The recognition system according to (12) or (13), whereinthe recognition processing unit executes:first recognition processing based on one of the image data and the event data; andsecond recognition processing based on a result of the first recognition processing and another one of the image data and the event data.

REFERENCE SIGNS LIST

34PHOTOELECTRIC CONVERSION FILM

43p-TYPE SEMICONDUCTOR REGION

44n-TYPE SEMICONDUCTOR REGION

54PIXEL SEPARATION UNIT

55FIXED CHARGE FILM

101PIXEL ARRAY UNIT

102A VERTICAL DRIVE CIRCUIT

102B HORIZONTAL DRIVE CIRCUIT

103A RGB SIGNAL PROCESSING CIRCUIT

103aAC CONVERSION CIRCUIT

103B EVS SIGNAL PROCESSING CIRCUIT

103bSIGNAL PROCESSING UNIT

105SYSTEM CONTROL CIRCUIT

108A RGB DATA PROCESSING UNIT

108B EVS DATA PROCESSING UNIT

210ADDRESS EVENT DETECTION CIRCUIT

212LIGHT RECEIVING CIRCUIT

21512INPUT OR CIRCUIT

21522INPUT AND CIRCUIT

340LIGHT RECEIVING UNIT

1001RGB SENSOR UNIT

1002RGB IMAGE PROCESSING UNIT

1003EVS SENSOR UNIT

1004EVENT SIGNAL PROCESSING UNIT

1005RECOGNITION PROCESSING UNIT

1010LASER LIGHT SOURCE

1011LIGHT SOURCE DRIVE UNIT

1021SENSOR CONTROL UNIT

1022LIGHT RECEIVING UNIT

1050SYSTEM CONTROL UNIT

NM3CURRENT SOURCE TRANSISTOR