Patent ID: 12219272

DESCRIPTION OF THE EMBODIMENTS

Hereinbelow, preferred embodiments of the technique of the present disclosure will be described with reference to the drawings. Note that the individual drawings are merely drawn for the purpose of explaining structures and configurations, and the dimensions of individual members shown in the drawings do not necessarily reflect actual dimensions. In addition, in the individual drawings, the same reference numerals denote the same members or components and, hereinbelow, the description of duplicate contents will be omitted.

First Embodiment

(Configuration) A photoelectric conversion device1of a first embodiment will be described with reference toFIGS.1to8.FIG.1is a block diagram showing an example of the configuration of the photoelectric conversion device1. The photoelectric conversion device1includes a lens unit11, a dual band pass filter (DBPF) unit12, an image sensor unit13, a processor unit14, an external computer unit15, an image display unit16, and an image recording unit17.

The lens unit11has a transmittance which allows the passage of light from a wavelength band of visible light to a wavelength band of infrared light. Note that the lens unit11is preferably subjected to chromatic aberration correction in ranges of the wavelength bands of the passage. The DBPF unit12is an optical filter which allows the passage of light in part of the wavelength band of visible light and light in part of the wavelength band of infrared light. Though described later, the DBPF unit12blocks light in a wavelength band positioned between visible light and near-infrared light in order to separate light having passed through the lens unit11into a visible light component and an IR component with high accuracy. With this, the DBPF unit12separates light having passed through the lens unit11into visible light and infrared light, and causes the visible light and the infrared light to be incident on a pixel area of the image sensor unit13. As shown inFIG.2, the image sensor unit13, which serves as a photoelectric conversion element, has a pixel array131and a controller132. The processor unit14can communicate with the controller132, and controls the pixel array131with the controller132and acquires image data of a subject (not shown). The processor unit14, which serves as an image generation unit, generates a visible light image and an infrared light image by using a signal output from the photoelectric conversion element. The detail of processing executed by the processor unit14will be described later.

The processor unit14processes the image data of the subject acquired from the image sensor unit13, and transmits the processed image data to each of the external computer unit15, the image display unit16, and the image recording unit17. In addition, the processor unit14has an image data separation unit141which separates the image data of the subject into a visible light image and an infrared light image. The detail of processing executed by the processor unit14will be described later. The processor unit14may be, e.g., an integrated circuit or a device capable of programming individual functions (e.g., a programmable logic device (PLD) such as a field programmable gate array (FPGA)). Alternatively, the processor unit14may also be an arithmetic unit such as a micro processing unit (MPU) or a digital signal processor (DSP) for implementing the individual functions. Alternatively, the processor unit14may be a dedicated integrated circuit (an application specific integrated circuit (ASIC) or the like). Alternatively, the processor unit14may include a CPU and a memory, and the individual functions may be implemented on software. That is, the functions of the processor unit14are implemented by hardware and/or software.

The external computer unit15can communicate with the processor unit14and the image recording unit17, and acquires the image data of the subject or the image data which is separated into the visible light image and the infrared light image of the subject and performs image processing. The image display unit16can communicate with the processor unit14and the external computer unit15, and displays RAW data of the image data of the subject, the image data which is separated into the visible light image and the infrared light image of the subject, and image data after adjustment in which the image quality of each image data mentioned above is adjusted. The image recording unit17records the image data received from the processor unit14or the external computer unit15.

FIG.2shows an example of the configuration of the image sensor unit13in the first embodiment. The image sensor unit13has a control unit133and a signal reading unit134in addition to the pixel array131and the controller132. The pixel array131includes a plurality of pixels PX arranged in a matrix (so as to form a pixel area including a plurality of rows and a plurality of columns). In the present embodiment, a color filter array shown as an example inFIG.4is provided on the pixels PX. In the present embodiment, the control unit133is a vertical scanning circuit constituted by a decoder and a shift register, and drives the plurality of pixels PX for each row. The signal reading unit134includes a signal amplification circuit1341and a sampling circuit1342disposed in each column, a multiplexer1343, and a horizontal scanning circuit1344constituted by a decoder and a shift register.

With this configuration, the signal reading unit134performs signal reading for each column from the plurality of pixels PX driven by the control unit133. In sampling by the sampling circuit1382, correlated double sampling (CDS) processing is used. The controller132includes a timing generator, and performs synchronization control of the pixels PX, the control unit133, and the signal reading unit134.

FIG.3shows a transmittance characteristic of each wavelength of the DBPF unit12. In an example shown inFIG.3, the transmittance characteristic is a characteristic which allows the passage of part of visible light (light having a wavelength of 400 nm to 700 nm) and the passage of part of infrared light (light having a wavelength of 850 nm to 1000 nm). The transmittance characteristic shown in the drawing for separating visible light and infrared light condensed by the lens unit11into lights in wavelength bands which are different from each other with the DBPF unit12is only an example, and the transmittance characteristic may be any transmittance characteristic which can satisfy an arithmetic expression which allows separation between a visible light component and an infrared light component. The detail of the arithmetic expression will be described later.

FIG.4schematically shows the color filter array of the image sensor unit13in the first embodiment. As shown inFIG.4, when a pixel array of two columns×two rows is assumed to be one unit of a pixel area including a plurality of rows and a plurality of columns, an R+IR pixel41and a B+IR pixel44are included in the unit, and the R+IR pixel41and the B+IR pixel44are disposed so as to be positioned diagonally. In addition, a W+IR pixel42and a W+2IR pixel43are included in the unit, and the W+IR pixel42and the W+2IR pixel43are disposed so as to be positioned diagonally. Note that the W+2IR pixel43is provided as an example of a first pixel, and the W+IR pixel42is provided as an example of a second pixel.

FIG.5shows the transmittance characteristic of each wavelength of each color filter of the image sensor unit13in the first embodiment. As shown inFIG.5, the B+IR pixel44has the transmittance characteristic which allows the passage of light having a wavelength of 400 nm to 550 nm and light having a wavelength of 700 nm to 1000 nm. In addition, the R+IR pixel41has the transmittance characteristic which allows the passage of light having a wavelength of 550 nm to 1000 nm. Further, each of the W+IR pixel42and the W+2IR pixel43has the transmittance characteristic which allows the passage of light of 400 nm to 1000 nm which includes the wavelength band of visible light serving as a first wavelength band and the wavelength band of infrared light serving as a second wavelength band. An example of visible light in the first wavelength band is white light.

FIGS.6A and6Bshow schematic views of cross-sectional structures of four pixels which are the W+IR pixel42and the W+2IR pixel43(FIG.6A), and the R+IR pixel41and the B+IR pixel44(FIG.6B) which are included in the image sensor unit13. Note that a microlens (ML) unit139ais an optical system for efficiently performing light condensing to the W+IR pixel42. A W+2IR filter unit1310ais a filter which allows the passage of, out of light condensed in the ML unit139a, light in a wavelength band from visible light to infrared light shown inFIG.5. Note that the W+2IR filter unit1310ais a second filter which allows the passage of visible light in the first wavelength band and infrared light in the second wavelength band. An infrared neutral density filter unit1311ais a filter having a transmittance which reduces infrared light having passed through the W+2IR filter unit1310a, and reduces sensitivity to infrared light by half. Note that the infrared neutral density filter unit1311ais an example of a first light reduction unit which reduces infrared light having passed through the second filter. A photodiode (PD) unit1312aconverts light having passed through the infrared neutral density filter unit1311ainto electrical charge.

In addition, an ML unit139bis an optical system for efficiently performing light condensing to the W+2IR pixel43. A W+2IR filter unit1310bis a filter which allows the passage of, out of light condensed in the ML unit139b, light in the wavelength band from visible light to infrared light shown inFIG.5. Note that the W+2IR filter unit1310bis a first filter which allows the passage of visible light in the first wavelength band and infrared light in the second wavelength band. A PD unit1312bconverts light having passed through the W+2IR filter unit1310binto electrical charge. With the structure described above, in one unit of the pixel area, the W+2IR pixel43has IR sensitivity which is twice as high as that of the W+IR pixel42.

In addition, ML units139cand139dare optical systems for efficiently performing light condensing to the R+IR pixel41and the B+IR pixel44, respectively. An R+2IR filter unit1310cand a B+2IR filter unit1310dare filters which allow the passage of, out of light condensed in the ML units139cand139d, light in the wavelength band from visible light to infrared light shown inFIG.5. Note that each of the R+2IR filter unit1310cand the B+2IR filter unit1310dis a third filter which allows the passage of visible light in the first wavelength band and infrared light in the second wavelength band. Infrared neutral density filter units1311cand1311dare filters having the transmittances which reduce infrared lights having passed through the R+2IR filter unit1310cand the B+2IR filter unit1310d, and reduce sensitivity to infrared light by half. Note that each of the infrared neutral density filter units1311cand1311dis an example of a second light reduction unit which reduces infrared light having passed through the third filter. Thus, the infrared neutral density filter units1311cand1311dare provided also in the R+IR pixel41and the B+IR pixel44other than the W+IR pixel42and the W+2IR pixel43. In addition, the light reduction rate of each of the infrared neutral density filter units1311cand1311dserving as the light reduction units provided in the R+IR pixel41and the B+IR pixel44is equal to the light reduction rate of the infrared neutral density filter unit1311a. PD units1312cand1312dconvert lights having passed through the infrared neutral density filter units1311cand1311dinto electrical charge.

Note that, in the present embodiment, as shown inFIGS.6A and6B, a structure in which the infrared neutral density filter is stacked on the PD as means for providing a difference in infrared sensitivity is adopted, but other structures may also be adopted. That is, for example, a structure in which, by providing a difference in the depth of impurity implantation for photoelectric conversion of the PD, a desired difference is provided in the sensitivity to infrared light may also be adopted. For example, the W+2IR pixel43is provided as an example of a third pixel, and the W+IR pixel42is provided as an example of a fourth pixel. In addition, the W+2IR filter unit1310ais used as a fourth filter which allows the passage of visible light in a third wavelength band and infrared light in a fourth wavelength band. Further, the W+2IR filter unit1310bis used as a fifth filter which allows the passage of visible light in the third wavelength band and infrared light in the fourth wavelength band. In addition, the PD unit1312bis used as a first photoelectric conversion unit, and the PD unit1312ais used as a second photoelectric conversion unit. At this point, it is possible to provide a difference in infrared sensitivity between the PD units by providing a difference between the depth of impurity implantation in the PD unit1312band the depth of impurity implantation in the PD unit1312a. Further, each of the R+2IR filter unit1310cand the B+2IR filter unit1310dserves as a sixth filter which allows the passage of visible light in the third wavelength band and infrared light in the fourth wavelength band.

FIG.7shows the transmittance characteristic of each wavelength of each of the infrared neutral density filters stacked on the R+IR pixel41, the W+IR pixel42, and the B+IR pixel44of the image sensor unit13. In addition, though the depiction thereof is omitted, the transmittance of light reaching each of the individual PD units1312a,1312c, and1312dis a product of the transmittance shown inFIG.3, the transmittance shown inFIG.5, and the transmittance shown inFIG.7.

(Arithmetic Calculation) Image data generated by the image sensor unit13including the R+IR pixel41, the W+IR pixel42, the W+2IR pixel43, and the B+IR pixel44described above is transmitted to the processor unit14. The processor unit14includes at least the image data separation unit141which performs separation between the visible light image and the infrared light image. In addition, in the present embodiment, the image data separation unit141is included in the processor unit14, but may also be included in the image sensor unit13.

Hereinbelow, a description will be given of an example of processing executed by the image data separation unit141in the first embodiment. As shown inFIG.8, pixel values of pixels after the separation between the visible light image and the infrared light image are denoted by R′, G21′, G12′, B′, and IR′. By performing addition and subtraction by using the following expressions (1-1) to (1-6), it is possible to separate the pixel values of the individual pixels into the pixel values R′, G21′, G12′, B′, and IR′.
G=W−(R+B)  (1-1)
IR′=(W+2IR)−(W+IR)  (1-2)
R′=(R+IR)−IR′(1-3)
B′=(B+IR)−IR′(1-4)
G12′=(W+IR)−IR′(1-5)
G21′=(W+2IR)−{(R+IR)+(B+IR)}  (1-6)

Herein, it is assumed that the transmittance characteristics shown inFIGS.3,5, and7are set such that the above expression (1-1) is satisfied.

(Effect) The photoelectric conversion element according to the present embodiment is different from a conventional photoelectric conversion element which is configured to acquire the visible light image and the infrared light image concurrently in that the IR pixel and the G pixel are replaced with the W pixels. With this configuration, in the photoelectric conversion element according to the present embodiment, information which can be used for the generation of the visible light image is increased as compared with the conventional photoelectric conversion element, i.e., it is possible to achieve increases in the resolution and the sensitivity of a captured image.

In addition, according to the photoelectric conversion element of the present embodiment, as compared with a conventional color filter array, it is possible to implement the separation between the visible light image and the infrared light image more easily, and higher color reproducibility is obtained when a color image is generated from the visible light image. In addition, in the photoelectric conversion element of the present embodiment, the IR components are included in the pixel values obtained from all pixels, and hence, even in the case where the color image is generated without separating visible light and infrared light from each other, an increase in sensitivity can be expected to be achieved. For example, in the photoelectric conversion element of the present embodiment, edge detection performance can be expected to be improved in a low-illuminance photographing environment in night-time photographing or the like.

Second Embodiment

(Configuration) Next, the photoelectric conversion device1according to a second embodiment will be described with reference toFIGS.1,3,5,7,9,10A and10B. An example of the configuration of the photoelectric conversion device1according to the second embodiment is the same as that of the photoelectric conversion device1according to the first embodiment shown inFIG.1. In addition, the transmittance characteristic of each wavelength of the DBPF unit12of the photoelectric conversion device1according to the second embodiment is the transmittance characteristic shown inFIG.3. Further, the circuit diagram showing an example of the configuration of the image sensor unit13of the photoelectric conversion device1according to the second embodiment and the transmittance characteristic of each color filter stacked on the pixel are the same as those shown inFIGS.2and5. In addition, in the photoelectric conversion device1according to the second embodiment, the transmittance characteristic of the infrared neutral density filter stacked on the pixel of the image sensor unit13serving as the photoelectric conversion element is the transmittance characteristic shown inFIG.7.

In the photoelectric conversion device1according to the second embodiment, the configurations of the color filter and the infrared neutral density filter which are stacked on the pixels of the image sensor unit13are different from those in the first embodiment. In addition, the arithmetic expressions used by the image data separation unit141of the photoelectric conversion device1according to the second embodiment are different from the arithmetic expressions in the first embodiment.

FIG.9shows the color filter array of the image sensor unit13in the second embodiment. When the pixel array of two columns×two rows is assumed to be one unit of the pixel area, an R+2IR pixel91and a B+2IR pixel94are included in the unit, and the R+2IR pixel91and the B+2IR pixel94are disposed so as to be positioned diagonally. In addition, a W+2IR pixel92and a W+IR pixel93are included in the unit, and the W+2IR pixel92and the W+IR pixel93are disposed so as to be positioned diagonally.

FIGS.10A and10Bshow schematic views of cross-sectional structures of the W+2IR pixel92and the W+IR pixel93(FIG.10A), and the R+2IR pixel91and the B+2IR pixel94(FIG.10B) which are included in the image sensor unit13. An ML unit139eis an optical system for efficiently performing light condensing to the W+IR pixel93. A W+2IR filter unit1310eis a filter which allows the passage of, out of light condensed in the ML unit139e, light in the wavelength band from visible light to infrared light shown inFIG.5. An infrared neutral density filter unit1311eis a filter having the transmittance which reduces infrared light having passed through the W+2IR filter unit1310e, and reduces sensitivity to infrared light by half. A PD unit1312econverts light having passed through the infrared neutral density filter unit1311einto electrical charge.

In addition, an ML unit139fis an optical system for efficiently performing light condensing to the W+2IR pixel92. A W+2IR filter unit1310fis a filter which allows the passage of, out of light condensed in the ML unit139f, light in the wavelength band from visible light to infrared light shown inFIG.5. A PD unit1312fconverts light having passed through the W+2IR filter unit1310finto electrical charge. With the structure described above, in one unit of the pixel area, the W+2IR pixel92has the IR sensitivity which is twice as high as that of the W+IR pixel93.

In addition, an ML unit139gis an optical system for efficiently performing light condensing to the R+2IR pixel91. An R+2IR filter unit1310gis a filter which allows the passage of, out of light condensed in the ML unit139g, light in the wavelength band from visible light to infrared light shown inFIG.5. A PD unit1312gconverts light having passed through the R+2IR filter unit1310ginto electrical charge. In addition, an ML unit139his an optical system for efficiently performing light condensing to the B+2IR pixel94. A B+2IR filter unit1310his a filter which allows the passage of, out of light condensed in the ML unit139h, light in the wavelength band from visible light to infrared light shown inFIG.5. A PD unit1312hconverts light having passed through the B+2IR filter unit1310hinto electrical charge. Consequently, similarly to the W+2IR pixel92, each of the R+2IR pixel91and the B+2IR pixel94has a structure in which the infrared neutral density filter is “not” stacked.

The transmittance characteristic of each wavelength of the infrared neutral density filter stacked on the W+IR pixel93of the image sensor unit13in the present embodiment is the same as that in the first embodiment, as shown inFIG.7. Note that the transmittance of light reaching the PD unit1312eis a product of the transmittance shown inFIG.3, the transmittance shown inFIG.5, and the transmittance shown inFIG.7.

(Arithmetic Calculation) Image data generated by the image sensor unit13including the R+2IR pixel91, the W+2IR pixel92, the W+IR pixel93, and the B+2IR pixel94described above is transmitted to the processor unit14. The processor unit14includes at least the image data separation unit141which performs separation between the visible light image and the infrared light image. In addition, in the present embodiment, the image data separation unit141is included in the processor unit14, but may also be included in the image sensor unit13.

Hereinbelow, a description will be given of an example of processing executed by the image data separation unit141in the second embodiment. As shown inFIG.11, the pixel values of the pixels after the separation between the visible light image and the infrared light image are denoted by R′, G21′, G12′, B′, and IR′. By performing addition and subtraction by using the expression (1-1) and the following expressions (2-1) to (2-5), it is possible to separate the pixel values of the individual pixels into the pixel values R′, G21′, G12′, B′, and IR′.
IR′=(W+2IR)−(W+IR)  (2-1)
R′=(R+2IR)−2IR′(2-2)
B′=(B+2IR)−2IR′(2-3)
G12′=(W+2IR)−2IR′(2-4)
G21′=(W+2IR)−2IR′(2-5)

Herein, it is assumed that the transmittance characteristics shown inFIGS.3,5, and7are set such that the above expression (1-1) is satisfied.

(Effect) The photoelectric conversion element according to the present embodiment is different from the conventional photoelectric conversion element which is configured to acquire the visible light image and the infrared light image concurrently in that the IR pixel and the G pixel are replaced with the W pixels. With this configuration, in the photoelectric conversion element according to the present embodiment, information which can be used for the generation of the visible light image is increased as compared with the conventional photoelectric conversion element, i.e., it is possible to achieve increases in the resolution and the sensitivity of the captured image. Further, in the photoelectric conversion element according to the present embodiment, the IR sensitivity in one unit is improved as compared with the first embodiment, and hence the infrared light image having higher sensitivity can be expected to be generated.

Third Embodiment

(Configuration) Next, the photoelectric conversion device1according to a third embodiment will be described with reference toFIGS.1,3,5,12,13,14A,14B,15, and16. An example of the configuration of the photoelectric conversion device1according to the third embodiment is the same as that of the photoelectric conversion device1according to the first embodiment shown inFIG.1. In addition, the transmittance characteristic of each wavelength of the DBPF unit12of the photoelectric conversion device1according to the third embodiment is the transmittance characteristic shown inFIG.3. Further, in the photoelectric conversion device1according to the third embodiment, the circuit diagram showing an example of the configuration of the image sensor unit13serving as the photoelectric conversion element and the transmittance characteristic of each color filter stacked on the pixel are the same as those shown inFIGS.2and5.

In the photoelectric conversion device1according to the third embodiment, the configuration of the infrared neutral density filter stacked on the pixel of the image sensor unit13is different from that in the first embodiment. In addition, the configurations of the color filter and the infrared neutral density filter which are stacked on the pixels of the image sensor unit13of the photoelectric conversion device1according to the third embodiment are different from those in the first embodiment. Further, the arithmetic expressions used by the image data separation unit141of the photoelectric conversion device1according to the third embodiment are different from the arithmetic expressions in the first embodiment.

FIG.12shows the color filter array of the image sensor unit13in the third embodiment. When the pixel array of two columns×two rows is assumed to be one unit of the pixel area, an R+2IR pixel1201and a B+2IR pixel1204are included in the unit, and the R+2IR pixel1201and the B+2IR pixel1204are disposed so as to be positioned diagonally. In addition, a W+IR pixel1202and a W+4IR pixel1203are included in the unit, and the W+IR pixel1202and the W+4IR pixel1203are disposed so as to be positioned diagonally.

FIG.13shows the transmittance characteristic of each wavelength of each color filter of the image sensor unit13in the third embodiment. As shown inFIG.13, the R+2IR pixel1201has the transmittance characteristic which allows the passage of light having a wavelength of 550 nm to 1000 nm. In addition, the B+2IR pixel1204has the transmittance characteristic which allows the passage of light having a wavelength of 400 nm to 550 nm and light having a wavelength of 750 nm to 1000 nm. Further, each of the W+IR pixel1202and the W+4IR pixel1203has the transmittance characteristic which allows the passage of light having a wavelength of 400 nm to 1000 nm. The IR sensitivity in the third embodiment is twice as high as that in the first embodiment.

FIGS.14A and14Bshow schematic views of cross-sectional structures of the W+IR pixel1202and the W+4IR pixel1203(FIG.14A), and the R+2IR pixel1201and the B+2IR pixel1204(FIG.14B) which are included in the image sensor unit13. An ML unit139iis an optical system for efficiently performing light condensing to the W+IR pixel1202. A W+4IR filter unit1310iis a filter which allows the passage of, out of light condensed in the ML unit139i, light in a wavelength band from visible light to infrared light shown inFIG.13. An infrared neutral density filter unit1311iis a filter having the transmittance which reduces infrared light having passed through the W+4IR filter unit1310i, and reduces sensitivity to infrared light to ¼. A PD unit1312iconverts light having passed through the infrared neutral density filter unit1311iinto electrical charge.

In addition, an ML unit139jis an optical system for efficiently performing light condensing to the W+4IR pixel1203. A W+4IR filter unit1310jis a filter which allows the passage of, out of light condensed in the ML unit139j, light in the wavelength band from visible light to infrared light shown inFIG.13. A PD unit1312jconverts light having passed through the W+4IR filter unit1310jinto electrical charge. With the structure described above, in one unit of the pixel area, the W+4IR pixel1203has the IR sensitivity which is four times as high as that of the W+IR pixel1202.

In addition, an ML unit139kis an optical system for efficiently performing light condensing to the R+2IR pixel1201. An R+4IR filter unit1310kis a filter which allows the passage of, out of light condensed in the ML unit139k, light in the wavelength band from visible light to infrared light shown inFIG.13. An infrared neutral density filter unit1311kis a filter having the transmittance which reduces infrared light having passed through the R+4IR filter unit1310k, and reduces sensitivity to infrared light by half. A PD unit1312kconverts light having passed through the infrared neutral density filter unit1311kinto electrical charge.

In addition, an ML unit139mis an optical system for efficiently performing light condensing to the B+2IR pixel1204. A B+4IR filter unit1310mis a filter which allows the passage of, out of light condensed in the ML unit139m, light in the wavelength band from visible light to infrared light shown inFIG.13. An infrared neutral density filter unit1311mis a filter having the transmittance which reduces infrared light having passed through the B+4IR filter unit1310m, and reduces sensitivity to infrared light by half. Consequently, the light reduction rate of each of the infrared neutral density filter units1311kand1311mserving as the light reduction units provided in the R+2IR pixel1201and the B+2IR pixel1204is lower than the light reduction rate of the infrared neutral density filter unit1311i. A PD unit1312mconverts light having passed through the infrared neutral density filter unit1311minto electrical charge.

FIG.15shows the transmittance characteristic of each wavelength of each of the infrared neutral density filters stacked on the R+2IR pixel1201, the W+IR pixel1202, and the B+2IR pixel1204of the image sensor unit13in the present embodiment. In addition, the transmittance of light reaching each of the PD units1312i,1312k, and1312mis a product of the transmittance shown inFIG.3, the transmittance shown inFIG.13, and the transmittance shown inFIG.15.

(Arithmetic Calculation) Image data generated by the image sensor unit13including the R+2IR pixel1201, the W+IR pixel1202, the W+4IR pixel1203, and the B+2IR pixel1204described above is transmitted to the processor unit14. The processor unit14includes at least the image data separation unit141which performs separation between the visible light image and the infrared light image. In addition, in the present embodiment, the image data separation unit141is included in the processor unit14, but may also be included in the image sensor unit13.

Hereinbelow, a description will be given of an example of processing executed by the image data separation unit141in the third embodiment. As shown inFIG.16, the pixel values of the pixels after the separation between the visible light image and the infrared light image are denoted by R′, G21′, G12′, B′, and IR′. By performing addition and subtraction by using the expression (1-1) and the following expressions (3-1) to (3-5), it is possible to separate the pixel values of the individual pixels into the pixel values R′, G21′, G12′, B′, and IR′.
IR′={(W+4IR)−(W+IR)}/3  (3-1)
R′=(R+2IR)−2IR′(3-2)
B′=(B+2IR)−2IR′(3-3)
G12′=(W+IR)−IR′(3-4)
G21′=(W+4IR)−{(R+2IR)+(B+2IR)}  (3-5)

Herein, it is assumed that the transmittance characteristics shown inFIGS.3,13, and15are set such that the above expression (1-1) is satisfied.

(Effect) Similarly to the first embodiment, the photoelectric conversion element according to the present embodiment is different from the conventional photoelectric conversion element which is configured to acquire the visible light image and the infrared light image concurrently in that the IR pixel and the G pixel are replaced with the W pixels. With this configuration, in the photoelectric conversion element according to the present embodiment, information which can be used for the generation of the visible light image is increased as compared with the conventional photoelectric conversion element, i.e., it is possible to achieve increases in the resolution and the sensitivity of the captured image. Further, in the photoelectric conversion element according to the present embodiment, the IR sensitivity in one unit is improved as compared with the first and second embodiments, and hence the infrared light image having higher sensitivity can be expected to be generated. Further, in the photoelectric conversion element according to the present embodiment, the dynamic range of the IR sensitivity can also be expected to be extended by using a difference in the IR sensitivity.

Fourth Embodiment

(Configuration) Next, the photoelectric conversion device1according to a fourth embodiment will be described with reference toFIGS.1,3,5,7, and17to20. An example of the configuration of the photoelectric conversion device1according to the fourth embodiment is the same as that of the photoelectric conversion device1according to the first embodiment shown inFIG.1. The transmittance characteristic of each wavelength of the DBPF unit12of the photoelectric conversion device1according to the fourth embodiment is the transmittance characteristic shown inFIG.3. In addition, in the photoelectric conversion device1according to the fourth embodiment, the circuit diagram showing an example of the configuration of the image sensor unit13serving as the photoelectric conversion element and the transmittance characteristic of each color filter stacked on the pixel are the same as those shown inFIGS.2and5.

In the photoelectric conversion device1according to the fourth embodiment, the configuration of each color filter stacked on the pixel of the image sensor unit13is different from that in the first embodiment. In addition, the configurations of the color filter and the infrared neutral density filter which are stacked on the pixels of the image sensor unit13of the photoelectric conversion device1according to the fourth embodiment are different from those in the first embodiment. Further, the arithmetic expressions used by the image data separation unit141of the photoelectric conversion device1according to the fourth embodiment are different from the arithmetic expressions in the first embodiment.

FIG.17shows the color filter array of the image sensor unit13in the fourth embodiment. When the pixel array of two columns×two rows is assumed to be one unit of the pixel area, three W+IR pixels1701,1702, and1704and one W+2IR pixel1703are included in the unit.

FIG.18shows the transmittance characteristic of each wavelength of each color filter of the image sensor unit13in the fourth embodiment. As shown inFIG.18, each of the W+IR pixels1701,1702, and1704, and the W+2IR pixel1703has the transmittance characteristic which allows the passage of light having a wavelength of 400 nm to 1000 nm.

FIGS.19A and19Bshow schematic views of cross-sectional structures of the W+IR pixels1701,1702, and1704, and the W+2IR pixel1703which are included in the image sensor unit13in the fourth embodiment. ML units139n,139q, and139rare optical systems for efficiently performing light condensing to the W+IR pixels1701,1702, and1704. W+2IR filter units1310n,1310q, and1310rare filters which allow the passage of, out of light condensed in the ML units139n,139q, and139r, light in a wavelength band from visible light to infrared light shown inFIG.18. Infrared neutral density filter units1311n,1311q, and1311rare filters having the transmittances which reduce infrared light having passed through the W+2IR filter units1310n,1310q, and1310r, and reduce sensitivity to infrared light by half. PD units1312n,1312q, and1312rconvert light having passed through the infrared neutral density filter units1311n,1311q, and1311rinto electrical charge.

In addition, an ML unit139pis an optical system for efficiently performing light condensing to the W+2IR pixel1703. A W+2IR filter unit1310pis a filter which allows the passage of, out of light condensed in the ML unit139p, light in the wavelength band from visible light to infrared light shown inFIG.18. A PD unit1312pis a unit which converts light having passed through the W+2IR filter unit1310pinto electrical charge. With the structure described above, in one unit of the pixel area, the W+2IR pixel1703has the IR sensitivity which is twice as high as that of each of the W+IR pixels1701,1702, and1704.

The transmittance characteristic of each wavelength of each of the infrared neutral density filters stacked on the W+IR pixels1701,1702, and1704of the image sensor unit13in the present embodiment is the transmittance characteristic shown inFIG.7. In addition, the transmittance of light reaching each of the PD units1312n,1312q, and1312ris a product of the transmittance shown inFIG.3, the transmittance shown inFIG.7, and the transmittance shown inFIG.18.

(Arithmetic Calculation) Image data generated by the image sensor unit13including the W+IR pixels1701,1702, and1704, and the W+2IR pixel1703described above is transmitted to the processor unit14. The processor unit14includes at least the image data separation unit141which performs separation between the visible light image and the infrared light image. In addition, in the present embodiment, the image data separation unit141is included in the processor unit14, but may also be included in the image sensor unit13.

Hereinbelow, a description will be given of an example of processing executed by the image data separation unit141in the fourth embodiment. As shown inFIG.20, the pixel values of the pixels after the separation between the visible light image and the infrared light image are denoted by W11′, W21′, W12′, W22′, and IR′. By performing addition and subtraction by using the following expressions (4-1) to (4-5), it is possible to separate the pixel values of the individual pixels into the pixel values W11′, W21′, W12′, W22′, and IR′.
IR′=(W21+IR)−{(W11+IR)+(W12+IR)+(W22+IR)}/3  (4-1)
W11′=(W11+IR)−IR′(4-2)
W12′=(W12+IR)−IR′(4-3)
W22′=(W22+IR)−IR′(4-4)
W21′=(W21+2IR)−2IR′(4-5)

(Effect) The photoelectric conversion element according to the present embodiment is different from the conventional photoelectric conversion element which is constituted by the pixel array of two columns×two rows of, e.g., W, W, W, and IR in that the IR pixel is replaced with the W pixel. With this configuration, in the photoelectric conversion element according to the present embodiment, information which can be used for the generation of the visible light image is increased as compared with the conventional photoelectric conversion element, i.e., it is possible to achieve increases in the resolution and the sensitivity of the captured image.

Fifth Embodiment

(Configuration) Next, the photoelectric conversion device1according to a fifth embodiment will be described with reference toFIGS.1,21, and22. An example of the configuration of the photoelectric conversion device1according to the fifth embodiment is the same as that of the photoelectric conversion device1according to the first embodiment shown inFIG.1. In addition, in the photoelectric conversion device1according to the fifth embodiment, as the array and the transmittance characteristic of the color filter stacked on the pixel of the image sensor unit13, the cross-sectional view of the pixel, and the transmittance characteristic of the infrared neutral density filter, the arrays and the transmittance characteristics in the first to fourth embodiments may be appropriately combined and adopted.

FIG.21shows an example of application of the photoelectric conversion device1according to the fifth embodiment. As shown inFIG.21, an infrared irradiation device52applies infrared rays to an area in the angle of view which is imaged by the photoelectric conversion device1. A subject53is a suspicious person imaged by the photoelectric conversion device1during, e.g., night-time monitoring. With regard to the material of sunglasses54worn by the subject53, a common material which reflects visible light but allows the passage of infrared light is assumed to be used. In addition, with regard to clothes55worn by the subject53, a complicated design (“ABC” in the drawing) which requires image generation with high resolution in the photoelectric conversion device1is assumed to be provided in the clothes55.

(Processing Flow)FIG.22is a flowchart showing an example of a control method at the time of imaging of the subject53which is executed by the processor unit14of the photoelectric conversion device1. First, in Step S100(hereinafter simply described as “S100”. The same applies to other steps), the image sensor unit13serving as the photoelectric conversion element receives an instruction to start imaging by an operation from, e.g., a user of the photoelectric conversion device1, and starts imaging. Next, in S110, the image data separation unit141separates image data acquired by the imaging of the image sensor unit13into a visible light image and an infrared light image. Next, in S120, the processor unit14executes color development processing of the visible light image. Note that this processing may be executed by the external computer unit15. Next, in S130, the processor unit14executes image recognition processing on the subject53by using the visible light image and the infrared light image. Note that this processing may be executed by the external computer unit15.

Next, in S140, the processor unit14determines whether or not the subject53satisfies a predetermined condition related to the suspicious person with the image recognition processing in S130. Herein, examples of the predetermined condition include abnormal behavior of the subject53and abnormal clothes (a design of clothes) of the subject. For example, the predetermined condition includes that the subject53wears the sunglasses54as shown inFIG.21or that the subject53repeats back-and-forth movement in the angle of view. In the case where the subject53satisfies the predetermined condition (S140: YES), the processor unit14advances the processing to S150. On the other hand, in the case where the subject53does not satisfy the predetermined condition (S140: NO), the processor unit14ends the processing of the present flowchart. Note that the processor unit14may perform the determination processing by using a plurality of the predetermined conditions in S140, and may also be configured to advance to S150in the case where at least one predetermined condition is satisfied. Alternatively, the processor unit14may also be configured to advance to S150in the case where two or more or all predetermined conditions are satisfied. In S150, the processor unit14records the image of the subject53having served as a determination target in S140in the image recording unit17, and ends the processing of the present flowchart.

(Effect) According to the photoelectric conversion device according to the present embodiment, in the case where a suspicious person is found during, e.g., night-time photographing, it is possible to perform image generation having high sensitivity and high resolution, and hence it is possible to detect the subject determined to be the suspicious person with high accuracy and record the image thereof. For example, in the case where the subject53is determined to be the suspicious person inFIG.21, it is possible to resolve the detail of the design of the clothes55in the visible light image of the subject53, and reproduce the color of the clothes55with high accuracy. Further, in the infrared light image of the subject53, it is possible to image eyes of the subject53through the sunglasses54, and hence it becomes possible to discern expressions of the subject53from the infrared light image. With this, the image generated by the photoelectric conversion device1can be expected to be useful for preventing crime.

Sixth Embodiment

(Configuration) Next, the photoelectric conversion device1according to a sixth embodiment will be described with reference toFIGS.1,23, and24. An example of the configuration of the photoelectric conversion device1according to the sixth embodiment is the same as that of the photoelectric conversion device1according to the first embodiment shown inFIG.1. In addition, in the photoelectric conversion device1according to the sixth embodiment, as the array and the transmittance characteristic of the color filter stacked on the pixel of the image sensor unit13, the cross-sectional view of the pixel, and the transmittance characteristic of the infrared neutral density filter, the arrays and the transmittance characteristics in the first to fourth embodiments may be appropriately combined and adopted.

FIG.23shows an example of application of the photoelectric conversion device1according to the sixth embodiment. As shown inFIG.23, an infrared irradiation device62applies infrared rays to an area in the angle of view which is imaged by the photoelectric conversion device1with low illuminance. An inspection object63is an object to be inspected which is imaged by the photoelectric conversion device1in this low-illuminance environment. A belt conveyer64conveys the inspection object63and moves the inspection object63in one direction.

(Flow)FIG.24is a flowchart showing an example of a control method at the time of imaging of the inspection object63which is executed by the processor unit14of the photoelectric conversion device1. First, in S200, the image sensor unit13serving as the photoelectric conversion element receives an instruction to start imaging by an operation from, e.g., a user of the photoelectric conversion device1, and starts imaging. Next, the image data separation unit141separates image data acquired by the imaging of the image sensor unit13into a visible light image and an infrared light image. Next, in S220, the processor unit14executes color development processing of the visible light image. Note that this processing may also be executed by the external computer unit15. Next, in S230, the processor unit14executes color inspection of the inspection object63by using the visible light image with image recognition processing, and executes contamination inspection by using the infrared light image. Note that the color inspection and the contamination inspection can be implemented by using known techniques, and hence, herein, the detailed description thereof will be omitted.

Next, in S240, the processor unit14determines whether or not the inspection object63satisfies predetermined conditions related to appearance abnormality and contamination based on results of various inspections in S230. Herein, with regard to examples of the predetermined condition, examples of the predetermined condition in S240include the occurrence of color unevenness in an image of the inspection object63in the visible light image, and the presence of a foreign matter which reflects infrared light from the inspection object63in the infrared light image. In the case where the inspection object63satisfies the predetermined condition (S240: YES), the processor unit14advances the processing to S250. On the other hand, in the case where the inspection object63does not satisfy the predetermined condition (S240: NO), the processor unit14ends the processing of the present flowchart. Note that the processor unit14may perform the determination processing by using a plurality of the predetermined conditions in S240, and may also be configured to advance to S250in the case where at least one predetermined condition is satisfied. Alternatively, the processor unit14may also be configured to advance to S250in the case where two or more or all predetermined conditions are satisfied. In S250, the processor unit14records the image of the inspection object63having served as the determination target in S240in the image recording unit17, and ends the processing of the present flowchart.

(Effect) According to the photoelectric conversion device according to the present embodiment, in the inspection of the inspection object in an environment in which an illumination condition is limited such as, e.g., a low-illuminance environment, it is possible to perform image generation having high sensitivity and high resolution, and hence it is possible to detect the inspection object having abnormality with high accuracy and record the image thereof. In addition, it is possible to execute inspection processing on both of the visible light image and the infrared light image, and hence it is possible to perform both of the appearance inspection and the contamination inspection of the inspection object. With this, for example, the appearance inspection and the contamination inspection can be expected to be performed on each of a plurality of inspection objects which are conveyed successively at high speed by the belt conveyer with high accuracy even when turnaround time (TAT) is reduced.

While the foregoing is the description related to the present embodiments, the configurations and the processing of the photoelectric conversion element and the photoelectric conversion device described above are not limited to the above-described embodiments, and various modifications may be made within the range which does not lose identity with the technical idea of the present invention. For example, the placement of the individual pixels of two columns×two rows described above is not limited to those shown in the drawings, and the pixels may be appropriately interchanged. In addition, the transmittance characteristic of each filter described above is not limited to those shown in the drawings, and the wavelength band of light which passes through the filter may be appropriately adjusted. Further, the same effects as those of the embodiments described above can be expected to be achieved also by arranging the pixels in the same manner as in each of the above embodiments in a pixel area in which at least one of the number of columns and the number of rows is larger than two.

In addition, in the embodiments described above, a pixel provided with a filter which allows the passage of light in a wavelength band of green (G) may also be adopted instead of the pixel provided with the filter which allows the passage of white light. For example, as shown inFIG.25, as a modification of the first embodiment, it is possible to constitute one unit of the pixel area with the G+IR pixel45and G+2IR pixel46instead of the W+IR pixel42and the W+2IR pixel43. In this case, the transmittance characteristic of each color filter corresponds to the transmittance characteristic shown inFIG.26instead of the transmittance characteristic shown inFIG.5. Note that the transmittance characteristic shown inFIG.26is only an example, and the transmittance may be appropriately adjusted. In addition, by causing the configuration of the G+IR pixel45and the G+2IR pixel46to correspond to the configuration of the W+IR pixel42and the W+2IR pixel43described above, it is possible to achieve increases in the resolution and the sensitivity of each of the visible light image and the infrared light image.

According to the technique of the present disclosure, there are provided a photoelectric conversion element and a photoelectric conversion device which adequately perform separation between a visible component and an infrared component for the purpose of acquiring both of a visible light image and an infrared light image, and allow the visible light image to have high sensitivity and high resolution while maintaining high color separation.

(Other Embodiments) While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

This application claims the benefit of Japanese Patent Application No. 2021-188610, filed on Nov. 19, 2021, which is hereby incorporated by reference herein in its entirety.