Solid-state imaging device and electronic equipment

A solid-state imaging device capable of acquiring an RGB image, a CMY image, and luminance information through one imaging process. The solid-state imaging device includes a pixel array portion in which a plurality of pixel unit groups are arrayed, the pixel unit group including pixel units disposed in a 2×2 matrix, the pixel unit including pixels disposed in an 2×2 matrix, and the pixels including a photoelectric conversion unit and a color filter. Each of the pixel unit groups is configured such that an R filter and a C filter are included as the color filters in a first pixel unit among four pixel units constituting the pixel unit group, a G filter and an M filter are included as the color filters in each of second and third pixel units, and a B filter and a Y filter are included as the color filters in a fourth pixel unit.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a national stage application under 35 U.S.C. 371 and claims the benefit of PCT Application No. PCT/JP2021/000380, having an international filing date of 7 Jan. 2021, which designated the United States, which PCT application claimed the benefit of Japanese Patent Application No. 2020-030458, filed 26 Feb. 2020, the entire disclosures of each of which are incorporated herein by reference.

TECHNICAL FIELD

The present technology relates to a solid-state imaging device and electronic equipment.

BACKGROUND ART

In the related art, a solid-state imaging device having a configuration in which one pixel of a Bayer array is divided into a plurality of pixels has been proposed (see, for example, PTL 1). In the solid-state imaging device disclosed in PTL 1, it is possible to obtain a high-resolution captured image by performing full-resolution demosaic processing (a series of processing for performing demosaic processing after remosaic processing), it is possible to obtain a captured image having an excellent SN ratio by performing binning processing, and it is possible to obtain a captured image of a high dynamic range (HDR) by changing exposure conditions in each of a plurality of pixels.

CITATION LIST

Patent Literature

SUMMARY

Technical Problem

However, in the solid-state imaging device disclosed in PTL 1, it is not possible to acquire all of an RGB image, a CMY image, and luminance information of a subject through one imaging process.

An object of the present disclosure is to provide a solid-state imaging device capable of acquiring an RGB image, a CMY image, and luminance information through one imaging process, and electronic equipment.

Solution to Problem

A solid-state imaging device of the present disclosure includes (a) a pixel array portion in which a plurality of pixel unit groups are arrayed, the pixel unit group being constituted by pixel units disposed in a 2×2 matrix, the pixel unit being constituted by pixels disposed in an m×m (m is a natural number of 2 or greater) matrix, and the pixel including a photoelectric conversion unit and a color filter formed to correspond to the photoelectric conversion unit, and (b) each of the pixel unit groups includes an R filter transmitting red light and a C filter transmitting cyan light having a complementary color relation with the red light as the color filters in a first pixel unit among four pixel units constituting the pixel unit group, includes a G filter transmitting green light and an M filter transmitting magenta light having a complementary color relation with the green light as the color filters in each of second and third pixel units, and includes a B filter transmitting blue light and a Y filter transmitting yellow light having a complementary color relation with the blue light as the color filters in a fourth pixel unit.

Electronic equipment of the present disclosure includes (a) a solid-state imaging device that includes a pixel array portion in which a plurality of pixel unit groups are arrayed, the pixel unit group being constituted by pixel units disposed in a 2×2 matrix, the pixel unit being constituted by pixels disposed in an m×m (m is a natural number of 2 or greater) matrix, and the pixel including a photoelectric conversion unit and a color filter formed to correspond to the photoelectric conversion unit, in which each of the pixel unit groups includes an R filter transmitting red light and a C filter transmitting cyan light having a complementary color relation with the red light as the color filters in a first pixel unit among four pixel units constituting the pixel unit group, includes a G filter transmitting green light and an M filter transmitting magenta light having a complementary color relation with the green light as the color filters in each of second and third pixel units, and includes a B filter transmitting blue light and a Y filter transmitting yellow light having a complementary color relation with the blue light as the color filters in a fourth pixel unit, (b) an optical lens that forms image light from a subject into an image on an imaging surface of the solid-state imaging device, and (c) a signal processing circuit that performs signal processing on a signal output from the solid-state imaging device.

DESCRIPTION OF EMBODIMENTS

Hereinafter, examples of a solid-state imaging device1and electronic equipment according to an embodiment of the present disclosure will be described with reference toFIGS.1to28. The embodiments of the present disclosure will be described in the following order. Note that the present disclosure is not limited to the following examples. In addition, the effects described in this specification are exemplary and not limiting, and other effects may be provided.1. First embodiment: electronic equipment1-1 Overall configuration of electronic equipment1-2 Configurations of main parts1-3 Image generation processing2. Second embodiment: electronic equipment2-1 Configurations of main parts2-2 Modification example3. Example of application to moving body4. Example of application to endoscopic operation system

1. First Embodiment: Electronic Equipment

1-1 Overall Configuration of Electronic Equipment

Electronic equipment100according to a first embodiment of the present disclosure will be described. As the electronic equipment100, various electronic equipment such as imaging devices, for example, a digital still camera and a digital video camera, a mobile phone having an imaging function, or other equipment having an imaging function can be adopted.FIG.1is a schematic view illustrating the entire electronic equipment100according to the first embodiment of the present disclosure.

As illustrated inFIG.1, the electronic equipment100includes a solid-state imaging device101(hereinafter, also referred to as a “solid-state imaging device1”), an optical lens102, a shutter device103, a drive circuit104, and a signal processing circuit105. The signal processing circuit105includes a mosaic image generation unit106, a white balance adjustment unit107, a mode determination unit108, a remosaic processing unit109, a binning processing unit110, a luminance value calculation unit111, and a luminance synthesis unit112. In the electronic equipment100, the optical lens102forms an image of image light (incident light113) received from a subject into an image on an imaging surface of the solid-state imaging device101, the solid-state imaging device101converts the amount of incident light113into an electrical signal in pixel units and outputs a pixel signal, and the signal processing circuit105performs signal processing on the pixel signal which is output from the solid-state imaging device101. At this time, the shutter device103controls a light irradiation period and a light shielding period for the solid-state imaging device101. In addition, the drive circuit104supplies a driving signal for controlling a transfer operation of the pixel signal and a shutter operation of the shutter device103.

FIG.2is a schematic configuration diagram illustrating the solid-state imaging device1. The solid-state imaging device1inFIG.2is a backside irradiation type complementary metal oxide semiconductor (CMOS) image sensor.

As illustrated inFIG.2, the solid-state imaging device1includes a substrate2, a pixel array portion3, a vertical drive circuit4, a column signal processing circuit5, a horizontal drive circuit6, an output circuit7, and a control circuit8. The pixel array portion3includes a plurality of pixels9that are arrayed in a matrix on the substrate2. Each of the pixels9includes a photoelectric conversion unit24, a color filter19formed to correspond to the photoelectric conversion unit24, and a microlens20as illustrated inFIGS.3A and3B. Regarding the pixels9, four pixels9arrayed in a 2×2 matrix constitute one pixel unit10. In addition, regarding the pixel unit10, four pixel units10arrayed in a 2×2 matrix constitute one pixel unit group11. That is, the pixel array portion3is configured such that a plurality of pixel unit groups11are arrayed in a matrix.

Note that, although an example in which one pixel unit10is constituted by the pixels9arrayed in a 2×2 matrix has been described in the first embodiment, other configurations can also be adopted. For example, as illustrated inFIG.4, the pixel unit may be constituted by the pixels9arrayed in an m×m (m is a natural number of 2 or greater) matrix. InFIG.4, a case where m is 5 or greater is illustrated.

The vertical drive circuit4, which is constituted by, for example, a shift register, selects a desired pixel drive wiring12, supplies a pulse for driving the pixels9to the selected pixel drive wiring12, and drives the pixels9in units of rows. That is, the vertical drive circuit4sequentially performs selection scanning on the pixels9in the pixel array portion3in the vertical direction in units of rows, and supplies a pixel signal based on signal charge generated in accordance with the amount of light received in the photoelectric conversion unit24of each of the pixels9to the column signal processing circuits5through vertical signal lines13.

The column signal processing circuit5is disposed, for example, for each column of the pixels9, and performs signal processing such as noise removal for each pixel column on a signal which is output from the pixels9corresponding to one row. For example, the column signal processing circuit5performs signal processing such as correlated double sampling (CDS) and analog digital (AD) conversion for removing pixel-specific fixed pattern noise.

The horizontal drive circuit6, which is constituted by, for example, a shift register, sequentially outputs a horizontal scanning pulse to the column signal processing circuits5to select each of the column signal processing circuits5in order, and outputs a pixel signal (hereinafter, also referred to as a “pixel value”) having been subjected to signal processing to the horizontal signal line14from each of the column signal processing circuits5.

The output circuit7performs signal processing on sequentially supplied pixel signals (pixel values) and outputs the pixel signals through the horizontal signal line14from each of the column signal processing circuits5. Examples of the signal processing which may be adopted include buffering, black level adjustment, column variation correction, and various types of digital signal processing. The control circuit8generates a clock signal or a control signal as a reference for operations of the vertical drive circuit4, the column signal processing circuit5, the horizontal drive circuit6, and the like on the basis of a vertical synchronization signal, a horizontal synchronization signal, and a master clock signal. In addition, the control circuit8outputs the generated clock signal or control signal to the vertical drive circuit4, the column signal processing circuit5, the horizontal drive circuit6, and the like.

1-2 Configurations of Main Parts

Next, a detailed configuration of the solid-state imaging device1inFIG.1will be described.FIG.3Ais a diagram illustrating a cross-sectional configuration of the pixel array portion3of the solid-state imaging device1.FIG.3Bis a diagram illustrating a minimum unit array of the color filter19along a line B-B inFIG.3A. InFIGS.3A and3B, a backside irradiation type CMOS image sensor is used as the solid-state imaging device1.

As illustrated inFIGS.3A and3B, the solid-state imaging device1according to the first embodiment includes a light receiving layer18in which the substrate2, an insulating film15, a light shielding film16, and a flattening film17are laminated in that order. In addition, a light condensing layer21in which the color filter19and the microlens20(an on-chip lens) are laminated in that order is formed on a surface of the light receiving layer18on the insulating film15side (hereinafter, also referred to as a “rear surface S1”). Further, a wiring layer22and a support substrate23are laminated in this order on a surface of the light receiving layer18on the substrate2side (hereinafter, also referred to as a “surface S2”). Note that the rear surface S1of the light receiving layer18and the rear surface of the flattening film17are the same surface, and thus the rear surface of the flattening film17will be referred to as a “rear surface S1” in the following description. In addition, the surface S2of the light receiving layer18and the surface of the substrate2are the same surface, and thus the surface of the substrate2will be referred to as a “surface S2” in the following description.

The substrate2is constituted by a semiconductor substrate formed of, for example, silicon (Si), and forms the pixel array portion3illustrated inFIG.1. In the pixel array portion3, a plurality of photoelectric conversion units24formed on the substrate2are disposed in a matrix. In the photoelectric conversion unit24, signal charge corresponding to the amount of incident light113is generated and accumulated. In addition, a pixel separation unit25is disposed between the photoelectric conversion units24that are adjacent to each other so that light having passed through other photoelectric conversion units24does not infiltrate. The insulating film15continuously covers the entire substrate2on the rear surface S1side (the entirety on a light receiving surface side). In addition, the light shielding film16is formed in a lattice shape in a portion of the insulating film15on a rear surface S3side (a portion on a light receiving surface side) so that a light receiving surface of each of the plurality of photoelectric conversion units24is open.

The color filter19is formed to correspond to each of the photoelectric conversion units24on the rear surface S1side (light receiving surface side) of the insulating film15. That is, one color filter19is formed for one photoelectric conversion unit24(pixel9). Thereby, the color filters19form color filter arrays26that are regularly arranged in a matrix. Each of the color filters19is configured to transmit light having a specific wavelength of the incident light113(red light, green light, blue light, cyan light, magenta light, or yellow light) and make the transmitted light incident on the photoelectric conversion unit24.

As the color filters19, an R filter19Rthat transmits red light, a G filter19Gthat transmits green light, a B filter19Bthat transmits blue light, a C filter19Cthat transmits cyan light having a complementary color relation with red light transmitted by the R filter19R, an M filter19Mthat transmits magenta light having a complementary color relation with green light transmitted by the G filter19G, and a Y filter19Ythat transmits yellow light having a complementary color relation with blue light transmitted by the B filter19Bare used. In other words, the C filter19Cis a color filter having a transmittance decreasing at a wavelength at which the transmittance of the R filter19Rincreases and having a transmittance increasing at a wavelength at which the transmittance of the R filter19Rdecreases in a spectral distribution diagram as illustrated inFIG.5. The M filter19Mis a color filter having a transmittance decreasing at a wavelength at which the transmittance of the G filter19Gincreases and having a transmittance increasing at a wavelength at which the transmittance of the G filter19Gdecreases in a spectral distribution diagram as illustrated inFIG.6. In addition, the Y filter19Yis a color filter having a transmittance decreasing at a wavelength at which the transmittance of the B filter19Bincreases and having a transmittance increasing at a wavelength at which the transmittance of the B filter19Bdecreases in a spectral distribution diagram as illustrated inFIG.7.

InFIGS.3A and3B, a sign R indicates the R filter19R, and similarly hereinafter, a sign G indicates the G filter19G, a sign B indicates the B filter19B, a sign C indicates the C filter19C, a sign M indicates the M filter19M, and a sign Y indicates the Y filter19Y. Further, in the following description, a pixel9including the R filter19Rwill be represented as a red pixel9R, a pixel9including the G filter19Gwill be represented as a green pixel9G, a pixel9including the B filter19Bwill be represented as a blue pixel9B, a pixel9including the C filter19Cwill be represented as a cyan pixel9C, a pixel9including the M filter19Mwill be represented as a magenta pixel9M, and a pixel9including the Y filter19Ywill be represented as a yellow pixel9Y.

In addition, an array pattern of the color filters19(the R filter19R, the G filter19G, the B filter19B, the C filter19C, the M filter19M, the Y filter19Y) is configured such that an array of the color filters19disposed in a 4×4 matrix as illustrated inFIG.3Bis set as a minimum unit of an array of the color filters19(hereinafter, also referred to as a “minimum unit array”), and the minimum unit array is disposed in all of the pixel unit groups11of the pixel array portion3as illustrated inFIG.8. The minimum unit array of the color filters19is an array in which a portion of a four-division Bayer array is corrected, the four-division Bayer array being configured such that the R filter19Ris disposed in the pixel unit10on an upper right side (hereinafter, also referred to as a “first pixel unit101”) among the four pixel units10constituting the pixel unit group11, the G filter19Gis disposed in the pixel unit10on an upper left side (hereinafter, also referred to as a “second pixel unit102”) and the pixel unit10on a lower right side (hereinafter, also referred to as a “third pixel unit103”), and the B filter19Bis disposed in the pixel unit10on a lower left side (hereinafter, also referred to as a “fourth pixel unit104”) as illustrated inFIG.3B. Specifically, the R filters19Rin the pixels9on the upper right side and the lower left side among the pixels9in a 2×2 array constituting the first pixel unit101of the four-division Bayer array are replaced with the C filters19C. In addition, the G filters19Gin the pixels9on the upper right side and the lower left side among the pixels9in a 2×2 array constituting each of the second pixel unit102and the third pixel unit103are replaced with the M filters19M. In addition, the B filters19Bin the pixels9on the upper right side and the lower left side among the pixels9in a 2×2 array constituting the fourth pixel unit104are replaced with the Y filters19Y.

That is, in the minimum unit array of the color filter19, the R filters19Rare included in one pixel pair out of pixel pairs (two pixels9constituting one set) which are disposed at diagonal positions in the first pixel unit101, and the C filters19Care included in the other pixel pair. Similarly hereinafter, the G filters19Gare included in one pixel pair out of pixel pairs disposed at diagonal positions in the second pixel unit102, and the M filters19Mare included in the other pixel pair. In addition, the G filters19Gare included in one pixel pair out of pixel pairs disposed at diagonal positions in the third pixel unit103, and the M filters19Mare included in the other pixel pair. In addition, the B filters19Bare included in one pixel pair out of pixel pairs disposed at diagonal positions in the fourth pixel unit104, and the Y filters19Yare included in the other pixel pair.

With this arrangement of the color filters19in the minimum unit array, in all of the color filters19included in the first pixel unit101, the R filter19Rand the C filter19Chaving a transmittance increasing at a wavelength at which the transmittance of the R filter19Rdecreases are combined with each other as illustrated inFIG.5, and thus a transmittance is substantially flat for all light from a short wavelength side to a long wavelength side. Similarly hereinafter, in all of the color filters19included in the second pixel unit102, the G filter19Gand the M filter19Mhaving a transmittance increasing at a wavelength at which the transmittance of the G filter19Gdecreases are combined with each other as illustrated inFIG.6, and thus a transmittance is substantially flat for all light from a short wavelength side to a long wavelength side. Further, in all of the color filters19included in the fourth pixel unit104, the B filter19Band the Y filter19Yhaving a transmittance increasing at a wavelength at which the transmittance of the B filter19Bdecreases are combined with each other as illustrated inFIG.7, and thus a transmittance is substantially flat for all light from a short wavelength side to a long wavelength side.

Note that, as illustrated inFIG.4, in a case where the pixel unit10is constituted by the pixels9disposed in an m×m matrix, and m is a natural number of 3 or greater (FIG.4illustrates a case where m is 5 or greater), the pixels9mutually disposed in a checkered pattern may be configured to include the R filter19R, the G filter19G, and the B filter19B. Specifically, the first pixel unit101is configured such that some pixels9mutually positioned in a checkered pattern among the plurality of pixels9disposed in an m×m matrix include the R filter19R, and the remaining pixels9include the C filter19C. Similarly hereinafter, the second pixel unit102is configured such that some pixels9mutually positioned in a checkered pattern among the plurality of pixels9disposed in an m×m matrix include the G filter19G, and the remaining pixels9include the M filter19M. In addition, the third pixel unit103is configured such that some pixels9mutually positioned in a checkered pattern among the plurality of pixels9disposed in an m×m matrix include the G filter19G, and the remaining pixels9include the M filter19M. In addition, the fourth pixel unit104is configured such that some pixels9mutually positioned in a checkered pattern among the plurality of pixels9disposed in an m×m matrix include the B filter19B, and the remaining pixels9include the Y filter19Y.

In this manner, the solid-state imaging device1according to the first embodiment is configured such that the color filters19also include the C filter19C, the M filter19M, and the Y filter19Y, in addition to the R filter19R, the G filter19G, and the B filter19B. For this reason, it is possible to generate an RGB image (an RGB mosaic image38inFIG.14) by using pixel signals of the red pixel9Rincluding the R filter19R, the green pixel9Gincluding the G filter19G, and the blue pixel9Bincluding the B filter19B. In addition, it is possible to generate a CMY image (a CMY mosaic image39inFIG.15) by using pixel signals of the cyan pixel9Cincluding the C filter19C, the magenta pixel9Mincluding the M filter19M, and the yellow pixel9Yincluding the Y filter19Y. Further, it is possible to generate luminance information (a luminance image41inFIG.16) by adding pixel values of the pixels9in the same pixel unit10. For this reason, it is possible to acquire the RGB image, the CMY image, and the luminance information through one imaging process. The RGB image is susceptible to imaging in a dark place because the sensitivity of the color filter19is low, but color reproducibility is good because the overlap of spectral sensitivities of the color filters19is small. Although the CMY image has poor color reproducibility due to a large overlap of spectral sensitivities of the color filters19, the sensitivity of the color filter19is low, and thus the CMY image is more susceptible to imaging in a dark place than the RGB image. In addition, it is possible to further improve color reproducibility by using six colors of a combination of the RGB image and the CMY image.

Further, in a case where the solid-state imaging device1according to the first embodiment is configured such that the R filters19Rare included in one pixel pair out of the pixel pairs disposed at diagonal positions in the first pixel unit101, and the C filters19Care included in the other pixel pair, the center of gravity of an image pixel31Rhaving only color information of the color red in the mosaic image30corresponding to the array of the color filters19and the center of gravity of an image pixel37Rgenerated by binning processing for the image pixel31Rhaving only the color information of the color red become equal to each other as illustrated inFIG.14, and thus it is not necessary to correct the center of gravity of the image pixel31Rof the color red in the binning processing. In addition, as illustrated inFIGS.14and15, this is also the same for an image pixel31Ghaving only color information of the color green, an image pixel31Rhaving only color information of the color blue, an image pixel31Chaving only color information of the color cyan, an image pixel31Mhaving only color information of the color magenta, and an image pixel31Yhaving only color information of the color yellow. For this reason, it is possible to acquire a more appropriate RGB image and CMY image. InFIGS.14and15, a sign R indicates the image pixel31Rhaving only the color information of the color red (hereinafter, also referred to as a “red image pixel”), and similarly hereinafter, a sign G indicates the image pixel31Ghaving only the color information of the color green (hereinafter, also referred to as a “green image pixel”), a sign B indicates the image pixel31Bhaving only the color information of the color blue (hereinafter, also referred to as a “blue image pixel”), a sign C indicates the image pixel31Chaving only the color information of the color cyan (hereinafter, also referred to as a “cyan image pixel”), a sign M indicates the image pixel31Mhaving only the color information of the color magenta (hereinafter, also referred to as a “magenta image pixel”), and a sign Y indicates the image pixel31Yhaving only the color information of the color yellow (hereinafter, also referred to as a “yellow image pixel”).

Note that, although an example in which the pixel units10on the upper left side and the lower right side are set as the second pixel unit102and the third pixel unit103has been described in the first embodiment, other configurations can also be adopted. For example, it is also possible to adopt a configuration in which the second pixel unit102and the third pixel unit103are disposed in the pixel unit10on the upper right side and the pixel unit10on the lower left side, a configuration in which the second pixel unit102and the third pixel unit103are disposed in the pixel unit10on the upper left side and the lower left side, or a configuration in which the second pixel unit102and the third pixel unit103are disposed in the pixel unit10on the upper right side and the lower right side. In addition, for example, it is also possible to adopt a configuration in which the first pixel unit101is disposed on the lower side, and the fourth pixel unit104is disposed on the upper side. That is, each of the pixel unit groups11may be only required to be configured such that the R filter19Rand the C filter19Care included as the color filters19in the first pixel unit101among the four pixel units10constituting the pixel unit group11, the G filter19Gand the M filter19Mare included as the color filters19in each of the second pixel unit102and the third pixel unit103, and the B filter19Band the Y filter19Yare included as the color filters19in the fourth pixel unit104.

The microlens20is formed to correspond to each of the photoelectric conversion units24on the rear surface S4side (light receiving surface side) of the color filter19. That is, one microlens20is formed for one photoelectric conversion unit24(pixel9). Thereby, the microlenses20form microlens arrays27that are regularly arranged in a matrix. Each of the microlenses20is configured to collect image light (incident light113) from a subject and guide the collected incident light113to the vicinity of the rear surface (light receiving surface) of the photoelectric conversion unit24through the color filter19.

Note that, although an example in which one microlens20is formed for one photoelectric conversion unit24has been described in the first embodiment, other configurations can also be adopted. For example, in a case where the green pixel9Gis used as a phase difference pixel, a configuration may be adopted in which two green pixels9Garrayed in a 1×2 matrix are disposed as illustrated inFIG.9, and one microlens20is formed for the disposed two green pixels9G(phase difference pixels). According to such a configuration, it is possible to detect a phase difference of a captured image between two green pixels9G(phase difference pixels) that share one microlens20.

The wiring layer22is formed on the surface S2side of the substrate2, and is configured to include an interlayer insulating film28and wirings29laminated as a plurality of layers with the interlayer insulating film28interposed therebetween. The wiring layer22drives a pixel transistor constituting the pixel9through the plurality of layers of wirings29.

The support substrate23is formed on a surface of the wiring layer22opposite to a side facing the substrate2. The support substrate23is a substrate for securing the strength of the substrate2at a manufacturing stage of the solid-state imaging device1. As a material of the support substrate23, for example, silicon (Si) can be Used.

1-3 Image Generation Processing

Next, image generation processing executed by the signal processing circuit105(the mosaic image generation unit106, the white balance adjustment unit107, the mode determination unit108, the remosaic processing unit109, the binning processing unit110, the luminance value calculation unit111, the luminance synthesis unit112) will be described.

When the image generation processing is executed, the mosaic image generation unit106first generates the mosaic image30corresponding to an array of the color filters19as illustrated inFIG.11based on pixel signals (pixel values) that are output from the red pixel9R, the green pixel9G, the blue pixel9B, the cyan pixel9C, the magenta pixel9M, and the yellow pixel9Yin step S101as illustrated inFIG.10.

Subsequently, the processing proceeds to step S102, and the white balance adjustment unit107estimates color temperature of a light source based on the pixel values of the image pixels31of the mosaic image30which are generated in step S101, and adjusts the white balance of the mosaic image30based on the estimated color temperature.

Subsequently, the processing proceeds to step S103, and the mode determination unit108determines which one of a high resolution mode and a high SN ratio mode an imaging mode selected by a user of the electronic equipment100is. In addition, in a case where a determination result is a high resolution mode, the processing proceeds to step S104. On the other hand, in a case where a determination result is a high SN ratio mode, the processing proceeds to step S105.

In step S104, the remosaic processing unit109performs remosaic processing on the mosaic image30for which white balance has been corrected in step S102. In the remosaic processing, a pixel array of the red, green, and blue image pixels31R,31G, and31Bconstituting the mosaic image30is converted into a Bayer array as illustrated inFIG.12, thereby generating an RGB mosaic image33constituted by red, green, and blue image pixels32R,32G, and32B of the Bayer array. In addition, as illustrated inFIG.13, a pixel array of the cyan, magenta, and yellow image pixels31C,31M, and31Yis converted into a predetermined array determined in advance, thereby generating a CMY mosaic image35constituted by cyan, magenta, and yellow image pixels34C,34M, and34Yof the predetermined array.FIG.13illustrates a case where a pixel array formed by replacing the red, green, and blue image pixels32R,32G, and32B of the Bayer array with the cyan, magenta, and yellow image pixels34C,34M, and34Yis used as a predetermined array.FIG.12illustrates enlarged parts of the mosaic image30and the RGB mosaic image33.

In addition,FIG.13illustrates enlarged parts of the mosaic image30and the CMY mosaic image35.

On the other hand, in step S105, the binning processing unit110performs binning processing on the mosaic image30for which white balance has been corrected in step S102, and then the processing proceeds to step S106. In the binning processing, as illustrated inFIG.14, the pixel values of the red image pixels31Rare added in each of pixel groups361corresponding to the first pixel unit101in the mosaic image30to form a pixel value of one red image pixel37R. Similarly hereinafter, pixel values of the green image pixels31Gare added in each of pixel groups362corresponding to the second pixel unit102to form a pixel value of one green image pixel37G. In addition, pixel values of the green image pixels31Gare added in each of pixel groups363corresponding to the third pixel unit103to form a pixel value of one green image pixel37G. In addition, pixel values of the blue image pixels31Bare added in each of pixel groups364corresponding to the fourth pixel unit104to form a pixel value of one blue image pixel37B. Thereby, an RGB mosaic image38constituted by the red image pixel37R(image pixel37Rhaving only color information of the color red), the green image pixel37G(the image pixel37Ghaving only color information of the color green), and the blue image pixel37G(the image pixel37B having only color information of the color blue) is generated. The number of pixels of the RGB mosaic image38is a quarter of the number of pixels of the mosaic image30.FIG.14illustrates enlarged parts of the mosaic image30and the RGB mosaic image38.

Further, in the binning processing, as illustrated inFIG.15, the pixel values of the cyan image pixels31Care added in each of the pixel groups361corresponding to the first pixel unit101in the mosaic image30for which white balance has been corrected in step S102to form a pixel value of one cyan image pixel38C. Similarly hereinafter, pixel values of the magenta image pixels31Mare added in each of the pixel groups362corresponding to the second pixel unit102to form a pixel value of one magenta image pixel38M. In addition, pixel values of the magenta image pixels31Mare added in each of pixel groups363corresponding to the third pixel unit103to form a pixel value of one magenta image pixel38M. In addition, pixel values of the yellow image pixels31Yare added in each of the pixel groups364corresponding to the fourth pixel unit104to form a pixel value of one yellow image pixel38Y. Thereby, a CMY mosaic image39constituted by the cyan image pixel38C(the image pixel38Chaving only color information of the color cyan), the magenta image pixel38M(the image pixel38Mhaving only color information of the color magenta), and the yellow image pixel38Y(the image pixel38Yhaving only color information of the color yellow) is generated. The number of pixels of the CMY mosaic image39is a quarter of the number of pixels of the mosaic image30.FIG.15illustrates enlarged parts of the mosaic image30and the CMY mosaic image39.

Further, in the binning processing, the luminance value calculation unit111adds the pixel values of the image pixels31Rand31Chaving color information of red and cyan colors in each of the pixel groups361corresponding to the first pixel unit101in the mosaic image30for which white balance has been corrected in step S102to calculate a luminance value of one image pixel40as illustrated inFIG.16. Similarly hereinafter, the pixel values of the image pixels31Gand31Mhaving color information of green and magenta colors in each of the pixel groups362corresponding to the second pixel unit102to calculate a luminance value of one image pixel40. In addition, the pixel values of the image pixels31Gand31Mhaving color information of green and magenta colors are added in each of the pixel groups363corresponding to the third pixel unit103to calculate a luminance value of one image pixel40. In addition, the pixel values of the image pixels31Band31Mhaving color information of blue and yellow colors are added in each of the pixel groups364corresponding to the fourth pixel unit104to calculate a luminance value of one image pixel40. Thereby, the luminance image41representing only luminance values of the image pixels40is generated. The number of pixels of the luminance image41is a quarter of the number of pixels of the mosaic image30.FIG.16illustrates enlarged parts of the mosaic image30and the luminance image41.

Note that a configuration may be adopted in which a numerical value obtained by multiplying a pixel value of the mosaic image30by a correction coefficient is used as a pixel value used to calculate a luminance value, the mosaic image being an image for which white balance has been corrected in step S102. Specifically, luminance values Br1, Br2, and Br3of the image pixels40of the luminance image41are calculated in accordance with the following Formula (1) based on pixel values R, G, B, Cy, Mg, and Ye of the image pixels31of the mosaic image30and correction coefficients a11, a21, a12, a22, a13, and a23determined in advance.

In the above-described Formula (1), R is a pixel value of the red image pixel31R, G is a pixel value of the green image pixel31G, B is a pixel value of the blue image pixel31B, Cy is a pixel value of the cyan image pixel31C, Mg is a pixel value of the magenta image pixel31M, and Ye is a pixel value of the yellow image pixel31Y. In addition, Br1is a luminance value obtained from the pixel values R and Cy of the red and cyan image pixels31Rand31C, Br2is a luminance value obtained from the pixel values G and Mg of the green and magenta image pixels31Gand31M, and Br3is a luminance value obtained from the pixel values B and Ye of the blue and yellow image pixels31Band31Y. As the correction coefficients all, a21, a12, a22, a13, and a23, a numerical value for which a numerical value different for each color temperature of a light source is set is used so that a relation of Br1=Br2=Br3is established. Thereby, as illustrated inFIGS.5,6, and7, it is possible to reduce a difference between the luminance values Br1, Br2, and Br3.

Subsequently, the processing proceeds to step S106, the luminance synthesis unit112performs luminance synthesis processing on the RGB mosaic image38generated in step S105or the CMY mosaic image39. In the luminance synthesis processing, first, it is determined whether a subject is bright based on the pixel values of the image pixels31of the mosaic image30generated in step S101. Further, in a case where it is determined that the subject is bright, the luminance value of the luminance image41is synthesized with the RGB mosaic image38to generate a synthesis image42as illustrated inFIG.17. It is possible to achieve both a high color reproducibility and a high resolution by synthesizing the RGB mosaic image38and the luminance image41. In addition, parallax correction is not necessary unlike, for example, a compound-eye camera separately including a solid-state imaging device for obtaining the RGB mosaic image38and a solid-state imaging device for obtaining the luminance image41.

In the synthesis of the RGB mosaic image38and the luminance image41, first, a Y1signal, a Cb signal, and a Cr signal of pixels are calculated in accordance with the following Formula (2) based on the pixel values R, G, and B of the pixels37of the RGB mosaic image38and the coefficients b11, b21, b31, b12, b22, b32, a13, a23, and a33determined in advance. Subsequently, luminance signals Y of the pixels43of the synthesis image42are calculated in accordance with the following Formula (3) based on the calculated Y1signal and luminance values (Y2signals) of the image pixels40of the luminance image41. That is, the Y1signal is corrected using the Y2signal to be set as a luminance signal Y. In this case, the Cb signal and the Cr signal calculated in the above-described Formula (2) are used as color information of the pixels43.

Here, the signal value YiCbCr is obtained by complement processing from an adjacent pixel, and thus there is a tendency for a resolution (equivalent to the Y1signal) to deteriorate. On the other hand, the resolution of the Y2signal hardly deteriorates. For this reason, according to the above-described Formula (3), it is possible to obtain a high-resolution image.

On the other hand, in a case where it is determined that the subject is dark, the CMY mosaic image39is converted into an RGB mosaic image44using a color conversion matrix as illustrated inFIG.18. Subsequently, the luminance image41is synthesized with the converted RGB mosaic image44to generate a synthesis image45. It is possible to achieve both a high SN ratio and a high resolution by synthesizing the luminance image41with the RGB mosaic image44. In addition, parallax correction is not necessary unlike, for example, the above-described compound-eye camera.

As described above, in the solid-state imaging device1according to the first embodiment of the present disclosure, each of the pixel unit groups11is configured such that the R filter19Rand the C filter19Care included as the color filters19in the first pixel unit101among the four pixel units10constituting the pixel unit group11, the G filter19Gand the M filter19Mare included as the color filters19in each of the second pixel unit102and the third pixel unit103, and the B filter19Rand the Y filter19Yare included as the color filters19in the fourth pixel unit104. For this reason, an RGB image can be generated by using the pixel signals of the red pixel9R, the green pixel9G, and the blue pixel9B. In addition, a CMY image can be generated by using the pixel signals of the cyan pixel9C, the magenta pixel9M, and the yellow pixel9Y. Further, luminance information can be generated by adding pixel values of the pixels9in the same pixel unit10. For this reason, it is possible to provide the solid-state imaging device1capable of acquiring an RGB image, a CMY image, and luminance information through one imaging process.

2. Second Embodiment: Electronic Equipment

2-1 Configurations of Main Parts

Next, electronic equipment100according to a second embodiment of the present disclosure will be described.FIG.19is a diagram illustrating the overall configuration of the electronic equipment100according to the second embodiment. In addition,FIG.20is a flowchart illustrating image generation processing of the first embodiment. InFIGS.19and20, parts corresponding to those inFIGS.1and10are given the same reference numerals and signs, and repeated descriptions thereof will not be given.

The electronic equipment100according to the second embodiment is different from that in the first embodiment in that the signal processing circuit105includes an HDR image generation unit114as illustrated inFIG.19, and image generation processing includes step S201instead of step S106inFIG.10as illustrated inFIG.20.

In step S201, the HDR image generation unit114performs HDR image generation processing based on the RGB mosaic image38generated in step S105and the CMY mosaic image39(seeFIGS.14and15). In the HDR image generation processing, the CMY mosaic image39is converted into an RGB mosaic image46using a color conversion matrix as illustrated inFIGS.21and22. Subsequently, the converted RGB mosaic image46and the RGB mosaic image38generated in step S105are synthesized to generate an HDR image47. As a method of synthesizing the RGB mosaic image46obtained from the CMY mosaic image39and the RGB mosaic image38generated in step S105, for example, a method of adding the pixel value of the RGB mosaic image46and the pixel value of the RGB mosaic image38for each image pixel48can be adopted.

As described above, the electronic equipment100according to the second embodiment of the present disclosure is configured to convert the CMY mosaic image39into the RGB mosaic image46and synthesize the converted RGB mosaic image46and the RGB mosaic image38generated by the binning processing unit110to generate the HDR image47(high dynamic range image). For this reason, it is possible to generate the HDR image47in addition to an RGB image, a CMY image, and luminance information through one imaging process.

2-2 Modification Example

Note that, in the electronic equipment100according to the first and second embodiments of the present disclosure, an example in which binning processing is digitally performed by the signal processing circuit105provided outside the solid-state imaging device1has been described, but other configurations can also be adopted. For example, a configuration may be adopted in which binning processing is performed in an analog manner at the time of reading a pixel signal from the pixel9of the solid-state imaging device1. Specifically, as illustrated inFIG.23, it is possible to adopt a configuration in which signal lines50connected to floating diffusions49of the pixels9of the same color are electrically connected to each other, among signal lines50connected to floating diffusions49of the pixels9constituting the pixel unit10, and pixel signals of the pixels9of the same color are added and output to a CDS circuit51. Thereby, it is possible to obtain a result of the addition of the pixel values of the pixels9of the same color.

In addition, as illustrated inFIG.24, it is possible to adopt a configuration in which signal lines50connected to the floating diffusions49of the pixels9constituting the pixel unit10are electrically bonded to each other, and pixel signals of the plurality of pixels9are added and output to the CDS circuit51. Thereby, it is possible to obtain luminance values (luminance information) of the image pixels40of the luminance image41illustrated inFIG.16.

3. Example of Application to Moving Body

The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technique according to the present disclosure may be realized as a device mounted on any type of moving body such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility, an airplane, a drone, a ship, and a robot.

FIG.25is a block diagram illustrating an example of a schematic configuration of a vehicle control system which is an example of a moving body control system to which the technology of the present disclosure can be applied.

A vehicle control system12000includes a plurality of electronic control units connected to each other via a communication network12001. In the example illustrated inFIG.25, the vehicle control system12000includes a drive system control unit12010, a body system control unit12020, a vehicle exterior information detection unit12030, a vehicle interior information detection unit12040, and an integrated control unit12050. In addition, as functional configurations of the integrated control unit12050, a microcomputer12051, a sound image output unit12052, and an in-vehicle network interface (I/F)12053are shown.

The drive system control unit12010controls operations of devices related to the drive system of the vehicle in accordance with various programs. For example, the drive system control unit12010functions as a driving force generation device for generating a driving force of a vehicle such as an internal combustion engine or a drive motor, a driving force transmission mechanism for transmitting a driving force to wheels, a steering mechanism for adjusting a steering angle of a vehicle, and a control device such as a braking device that generates a braking force of a vehicle.

The body system control unit12020controls operations of various devices equipped in a vehicle body in accordance with various programs. For example, the body system control unit12020functions as a control device of a keyless entry system, a smart key system, a power window device, or various lamps such as a head lamp, a back lamp, a brake lamp, and a turn signal or fog lamp. In this case, radio waves transmitted from a portable device that substitutes for a key or signals of various switches may be input to the body system control unit12020. The body system control unit12020receives inputs of these radio waves or signals and controls a door lock device, a power window device, a lamp, and the like of a vehicle.

The vehicle exterior information detection unit12030detects information outside the vehicle in which the vehicle control system12000is mounted. For example, an imaging unit12031is connected to the vehicle exterior information detection unit12030. The vehicle exterior information detection unit12030causes the imaging unit12031to capture an image of the outside of the vehicle and receives the captured image. The vehicle exterior information detection unit12030may perform object detection processing or distance detection processing for peoples, cars, obstacles, signs, and letters on the road based on the received image.

The imaging unit12031is an optical sensor that receives light and outputs an electrical signal according to the amount of received light. The imaging unit12031can also output the electrical signal as an image and ranging information. In addition, the light received by the imaging unit12031may be visible light or invisible light such as infrared rays.

The vehicle interior information detection unit12040detects information inside the vehicle. For example, a driver state detection unit12041that detects a state of a driver is connected to the vehicle interior information detection unit12040. The driver state detection unit12041includes, for example, a camera that captures an image of the driver, and the vehicle interior information detection unit12040may calculate a degree of fatigue or concentration of the driver or may determine whether or not the driver is dozing on the basis of detection information input from the driver state detection unit12041.

The microcomputer12051can calculate a control target value of the driving force generation device, the steering mechanism, or the braking device on the basis of information inside and outside the vehicle acquired by the vehicle exterior information detection unit12030or the vehicle interior information detection unit12040, and output a control command to the drive system control unit12010. For example, the microcomputer12051can perform cooperative control for the purpose of realizing functions of an advanced driver assistance system (ADAS) including vehicle collision avoidance, impact mitigation, following traveling based on an inter-vehicle distance, vehicle speed maintenance driving, vehicle collision warning, vehicle lane deviation warning, and the like.

Further, by controlling the driving force generation device, the steering mechanism, the braking device, and the like on the basis of information regarding the vicinity of the vehicle acquired by the vehicle exterior information detection unit12030or the vehicle interior information detection unit12040, the microcomputer12051can perform cooperative control for the purpose of automated driving or the like in which autonomous travel is performed without depending on an operation of the driver.

In addition, the microcomputer12051can output a control command to the body system control unit12020based on the information outside the vehicle acquired by the vehicle exterior information detection unit12030. For example, the microcomputer12051can perform coordinated control for antiglare such as switching a high beam to a low beam by controlling a headlamp according to a position of a preceding vehicle or an oncoming vehicle detected by the vehicle exterior information detection unit12030.

The sound image output unit12052transmits an output signal of at least one of a sound and an image to an output device capable of visually or audibly notifying an occupant of a vehicle or the outside of the vehicle of information. In the example illustrated inFIG.25, an audio speaker12061, a display unit12062, and an instrument panel12063are illustrated as output devices. The display unit12062may include, for example, at least one of an onboard display and a head-up display.

FIG.26is a diagram illustrating an example of positions at which the imaging unit12031is installed.

The imaging units12101,12102,12103,12104, and12105are provided at positions such as a front nose, side-view mirrors, a rear bumper, a back door, and an upper portion of a windshield in a vehicle interior of the vehicle12100, for example. The imaging unit12101provided on the front nose and the imaging unit12105provided in the upper portion of the windshield in the vehicle interior mainly acquire images in front of the vehicle12100. The imaging units12102and12103provided on the side mirrors mainly acquire images on a lateral side of the vehicle12100. The imaging unit12104provided on the rear bumper or the back door mainly acquires images behind the vehicle12100. Front view images acquired by the imaging units12101and12105are mainly used for detection of preceding vehicles, pedestrians, obstacles, traffic signals, traffic signs, lanes, and the like.

Note that,FIG.26illustrates an example of imaging ranges of the imaging units12101to12104. An imaging range12111indicates an imaging range of the imaging unit12101provided at the front nose, imaging ranges12112and12113respectively indicate imaging ranges of the imaging units12102and12103provided at the side mirrors, and an imaging range12114indicates an imaging range of the imaging unit12104provided at the rear bumper or the back door. For example, a bird's-eye view image of the vehicle12100as viewed from above can be obtained by superimposing image data captured by the imaging units12101to12104.

For example, the microcomputer12051can extract, particularly, a closest three-dimensional object on a path through which the vehicle12100is traveling, which is a three-dimensional object traveling at a predetermined speed (for example, 0 km/h or higher) in the substantially same direction as the vehicle12100, as a preceding vehicle by acquiring a distance to each of three-dimensional objects in the imaging ranges12111to12114and temporal change in the distance (a relative speed with respect to the vehicle12100) on the basis of distance information obtained from the imaging units12101to12104. Further, the microcomputer12051can set an inter-vehicle distance which should be guaranteed in advance in front of a preceding vehicle and can perform automated brake control (also including following stop control) or automated acceleration control (also including following start control). In this manner, it is possible to perform cooperative control for the purpose of automated driving or the like in which autonomous travel is performed without depending on an operation of the driver.

For example, the microcomputer12051can classify and extract three-dimensional object data regarding three-dimensional objects into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, and other three-dimensional objects such as utility poles on the basis of the distance information obtained from the imaging units12101to12104and use the three-dimensional object data for automatic avoidance of obstacles. For example, the microcomputer12051identifies obstacles in the vicinity of the vehicle12100into obstacles that can be visually recognized by the driver of the vehicle12100and obstacles that are difficult to be visually recognized. In addition, the microcomputer12051can determine a collision risk indicating a degree of risk of collision with each obstacle, and when the collision risk has a value equal to or greater than a set value and there is a possibility of collision, outputs a warning to the driver via the audio speaker12061or the display unit12062and performs forced deceleration or avoidance steering via the drive system control unit12010, so that it can perform driving assistance for collision avoidance.

At least one of the imaging units12101to12104may be an infrared camera that detects infrared rays. For example, the microcomputer12051can recognize a pedestrian by determining whether there is a pedestrian in the captured image of the imaging units12101to12104. Such pedestrian recognition is performed by, for example, a procedure in which feature points in the captured images of the imaging units12101to12104as infrared cameras are extracted and a procedure in which pattern matching processing is performed on a series of feature points indicating the outline of the object and it is determined whether the object is a pedestrian. When the microcomputer12051determines that there is a pedestrian in the captured images of the imaging units12101to12104, and the pedestrian is recognized, the sound image output unit12052controls the display unit12062so that the recognized pedestrian is superimposed and displayed with a square contour line for emphasis. In addition, the sound image output unit12052may control the display unit12062so that an icon indicating a pedestrian or the like is displayed at a desired position.

An example of the vehicle control system to which the technique according to the present disclosure may be applied has been described above. The technology of the present disclosure can be applied to the imaging unit12031and the like in the above-described configuration. Specifically, the solid-state imaging devices101and1inFIGS.1and2and the signal processing circuit105inFIG.1can be applied to the imaging unit12031. By applying the technique according to the present disclosure to the imaging unit12031, a clearer captured image can be obtained, and thus it is possible to reduce a driver's fatigue.

4. Example of Application to Endoscopic Operation System

The technology according to the present disclosure (the present technology) may be applied to, for example, an endoscopic operation system.

FIG.27is a diagram illustrating an example of a schematic configuration of an endoscopic operation system to which the technology according to the present disclosure (the present technology) can be applied.

FIG.27illustrates a state in which an operator (doctor)11131is performing operator on a patient11132on a patient bed11133using the endoscopic operation system11000. As illustrated, the endoscopic operation system11000is configured of an endoscope11100, other surgical instruments11110such as a pneumoperitoneum tube11111and an energized treatment tool11112, a support arm device11120that supports the endoscope11100, and a cart11200equipped with various devices for endoscopic operations.

The endoscope11100is configured of a lens barrel11101, a region of which having a predetermined length from a tip is inserted into a body cavity of the patient11132, and a camera head11102connected to a base end of the lens barrel11101. Although the endoscope11100configured as a so-called rigid mirror having the rigid lens barrel11101is illustrated in the illustrated example, the endoscope11100may be configured as a so-called flexible mirror having a flexible lens barrel.

An opening portion into which an objective lens is fitted is provided at the tip of the lens barrel11101. A light source device11203is connected to the endoscope11100, and light generated by the light source device11203is guided to the tip of the lens barrel by a light guide provided to extend inside the lens barrel11101and is radiated toward an observation target in the body cavity of the patient11132via the objective lens. Note that the endoscope11100may be a forward-viewing endoscope, a perspective-viewing endoscope, or a side-viewing endoscope.

An optical system and an imaging element are provided inside the camera head11102, and the reflected light (observation light) from the observation target converges on the imaging element by the optical system. The observation light is photoelectrically converted by the imaging element, and an electrical signal corresponding to the observation light, that is, an image signal corresponding to an observation image is generated. The image signal is transmitted to a camera control unit (CCU)11201as RAW data.

The CCU11201is configured of a central processing unit (CPU), a graphics processing unit (GPU) or the like, and comprehensively controls operations of the endoscope11100and a display device11202. Further, the CCU11201receives the image signal from the camera head11102and performs various types of image processing for displaying an image based on the image signal, for example, development processing (demosaic processing) and the like, on the image signal.

The display device11202displays an image based on an image signal having been subjected to image processing by the CCU11201under the control of the CCU11201.

The light source device11203is constituted by, for example, a light source such as a light emitting diode (LED), and supplies irradiation light at the time of imaging a surgical part or the like to the endoscope11100.

An input device11204is an input interface for the endoscopic operation system11000. The user can input various types of information or instructions to the endoscopic operation system11000via the input device11204. For example, the user inputs an instruction to change imaging conditions (a type of irradiation light, a magnification, a focal length, or the like) of the endoscope11100.

A treatment tool control device11205controls the driving of an energized treatment tool11112for cauterizing or incising tissue, sealing a blood vessel, or the like. In order to secure a field of view of the endoscope11100and secure an operation space of an operator, a pneumoperitoneum device11206sends gas into the body cavity of the patient11132via the pneumoperitoneum tube11111in order to inflate the body cavity. A recorder11207is a device that can record various types of information related to operator. A printer11208is a device that can print various types of information related to operator in various formats such as text, images and graphs.

The light source device11203that supplies the endoscope11100with the radiation light for imaging the surgical part can be configured of, for example, an LED, a laser light source, or a white light source configured of a combination thereof. When a white light source is formed by a combination of RGB laser light sources, it is possible to control an output intensity and an output timing of each color (each wavelength) with high accuracy and thus, the light source device11203adjusts white balance of the captured image. Further, in this case, laser light from each of the respective RGB laser light sources is radiated to the observation target in a time-division manner, and driving of the imaging element of the camera head11102is controlled in synchronization with a radiation timing, so that it is also possible to capture images corresponding to each of RGB in a time-division manner. According to this method, it is possible to obtain a color image without providing a color filter to the imaging element.

Further, the driving of the light source device11203may be controlled to change the intensity of output light at predetermined time intervals. The driving of the imaging element of the camera head11102is controlled in synchronization with the timing of the change in the light intensity to acquire an image in a time-division manner, and the image is synthesized, whereby it is possible to generate a so-called image in a high dynamic range without underexposure or overexposure.

In addition, the light source device11203may have a configuration in which light in a predetermined wavelength band corresponding to special light observation can be supplied. In the special light observation, for example, by emitting light in a band narrower than that of irradiation light (that is, white light) during normal observation using wavelength dependence of light absorption in a body tissue, so-called narrow band light observation (narrow band imaging) in which a predetermined tissue such as a blood vessel in the mucous membrane surface layer is imaged with a high contrast is performed. Alternatively, in the special light observation, fluorescence observation in which an image is obtained by fluorescence generated by emitting excitation light may be performed. The fluorescence observation can be performed by emitting excitation light to a body tissue and observing fluorescence from the body tissue (autofluorescence observation), or locally injecting a reagent such as indocyanine green (ICG) to a body tissue and emitting excitation light corresponding to a fluorescence wavelength of the reagent to the body tissue to obtain a fluorescence image. The light source device11203may be configured to be able to supply narrow band light and/or excitation light corresponding to such special light observation.

FIG.28is a block diagram illustrating an example of a functional configuration of the camera head11102and CCU11201illustrated inFIG.27.

The camera head11102has a lens unit11401, an imaging unit11402, a drive unit11403, a communication unit11404, and a camera head control unit11405. The CCU11201has a communication unit11411, an image processing unit11412, and a control unit11413. The camera head11102and the CCU11201are connected to each other via a transmission cable11400so that they can communicate with each other.

The lens unit11401is an optical system provided at a portion for connection to the lens barrel11101. The observation light taken in from the tip of the lens barrel11101is guided to the camera head11102and incident on the lens unit11401. The lens unit11401is configured of a combination of a plurality of lenses including a zoom lens and a focus lens.

The imaging unit11402is constituted by an imaging element. The imaging element constituting the imaging unit11402may be one element (so-called single plate type) or a plurality of elements (so-called multi-plate type). When the imaging unit11402is configured as a multi-plate type, for example, image signals corresponding to RGB are generated by the imaging elements, and a color image may be obtained by synthesizing the image signals. Alternatively, the imaging unit11402may be configured to include a pair of imaging elements for acquiring image signals for the right eye and the left eye corresponding to three-dimensional (3D) display. When 3D display is performed, the operator11131can ascertain the depth of biological tissues in the surgical part more accurately. Here, when the imaging unit11402is configured as a multi-plate type, a plurality of lens units11401may be provided according to the imaging elements.

Further, the imaging unit11402may not necessarily be provided in the camera head11102. For example, the imaging unit11402may be provided immediately after the objective lens inside the lens barrel11101.

The drive unit11403is constituted by an actuator, and moves the zoom lens and the focus lens of the lens unit11401by a predetermined distance along an optical axis under the control of the camera head control unit11405. Thereby, the magnification and the focus of the image captured by the imaging unit11402can be appropriately adjusted.

The communication unit11404is configured of a communication device for transmitting or receiving various information to or from the CCU11201. The communication unit11404transmits the image signal obtained from the imaging unit11402as RAW data to the CCU11201via the transmission cable11400.

Further, the communication unit11404receives a control signal for controlling the driving of the camera head11102from the CCU11201and supplies the control signal to the camera head control unit11405. The control signal includes, for example, information on imaging conditions such as information for designating a frame rate of a captured image, information for designating an exposure value at the time of imaging, and/or information for designating a magnification and a focus of the captured image.

Also, the imaging conditions such as the frame rate, the exposure value, the magnification, and the focus described above may be appropriately designated by the user or may be automatically set by the control unit11413of the CCU11201on the basis of the acquired image signal. In the latter case, a so-called auto exposure (AE) function, auto focus (AF) function and auto white balance (AWB) function are provided in the endoscope11100.

The camera head control unit11405controls the driving of the camera head11102on the basis of the control signal from the CCU11201received via the communication unit11404.

The communication unit11411is configured of a communication device for transmitting and receiving various types of information to and from the camera head11102. The communication unit11411receives an image signal transmitted from the camera head11102via the transmission cable11400.

In addition, the communication unit11411transmits a control signal for controlling the driving of the camera head11102to the camera head11102. The image signal or the control signal can be transmitted through electric communication, optical communication, or the like.

The image processing unit11412performs various image processing on the image signal which is the RAW data transmitted from the camera head11102.

The control unit11413performs various kinds of control regarding imaging of the surgical part or the like using the endoscope11100and display of a captured image obtained by imaging the surgical part or the like. For example, the control unit11413generates the control signal for controlling the driving of the camera head11102.

Further, the control unit11413causes the display device11202to display the captured image obtained by imaging the surgical part or the like on the basis of the image signal that has been subjected to the image processing by the image processing unit11412. In this case, the control unit11413may recognize various objects in the captured image using various image recognition techniques. For example, the control unit11413can recognize surgical tools such as forceps, specific biological parts, bleeding, mist when the energized treatment tool11112is used, and the like by detecting an edge shape, a color, and the like of an object included in the captured image. When the control unit11413causes the display device11202to display the captured image, it may cause various types of surgical support information to be superimposed and displayed with the image of the surgical part using the recognition result. When the surgical support information is superimposed and displayed, and presented to the operator11131, it is possible to reduce the burden on the operator11131and the operator11131can reliably proceed the operation.

The transmission cable11400connecting the camera head11102and the CCU11201to each other is an electric signal cable that supports electric signal communication, an optical fiber that supports optical communication, or a composite cable thereof.

Here, in the example shown in the drawing, communication is performed in a wired manner using the transmission cable11400, but communication between the camera head11102and the CCU11201may be performed in a wireless manner.

An example of the endoscopic operation system to which the technique according to the present disclosure can be applied has been described above. The technique according to the present disclosure may be applied to, for example, the imaging unit11402of the camera head11102, the image processing unit11412of the CCU11201, and the like among the configurations described above. Specifically, the solid-state imaging devices101and1inFIGS.1and2can be applied to the imaging unit10402, and the signal processing circuit105inFIG.1can be applied to the image processing unit11412. By applying the technology according to the present disclosure to the imaging unit10402and the image processing unit11412, it is possible to obtain a clearer image of the surgical part and thus, the surgeon can reliably confirm the surgical part.

Here, although the endoscopic operation system has been described as an example, the technology according to the present disclosure may be applied to other, for example, a microscopic operation system.

The present technology can also take on the following configurations.

(1) A solid-state imaging device including:a pixel array portion in which a plurality of pixel unit groups are arrayed, the pixel unit group being constituted by pixel units disposed in a 2×2 matrix, the pixel unit being constituted by pixels disposed in an m×m (m is a natural number of 2 or greater) matrix, and the pixel including a photoelectric conversion unit and a color filter formed to correspond to the photoelectric conversion unit,wherein each of the pixel unit groups includes an R filter transmitting red light and a C filter transmitting cyan light having a complementary color relation with the red light as the color filters in a first pixel unit among four pixel units constituting the pixel unit group, includes a G filter transmitting green light and an M filter transmitting magenta light having a complementary color relation with the green light as the color filters in each of second and third pixel units, and includes a B filter transmitting blue light and a Y filter transmitting yellow light having a complementary color relation with the blue light as the color filters in a fourth pixel unit.

(2) The solid-state imaging device according to (1),wherein m is a natural number of 3 or greater,the first pixel unit is configured such that some of the pixels mutually positioned in a checkered pattern among the plurality of pixels disposed in an m×m matrix include the R filter, and the remaining pixels include the C filter,the second pixel unit is configured such that some of the pixels mutually positioned in a checkered pattern among the plurality of pixels disposed in an m×m matrix include the G filter, and the remaining pixels include the M filter, andthe third pixel unit is configured such that some of the pixels mutually positioned in a checkered pattern among the plurality of pixels disposed in an m×m matrix include the B filter, and the remaining pixels include the Y filter.

(3) The solid-state imaging device according to (1),wherein m=2,the first pixel unit is configured such that one pixel pair out of pixel pairs disposed at diagonal positions includes the R filters, and the other pixel pair includes the C filters,the second pixel unit is configured such that one pixel pair out of pixel pairs disposed at diagonal positions includes the G filters, and the other pixel pair includes the M filters, andthe third pixel unit is configured such that one pixel pair out of pixel pairs disposed at diagonal positions includes the B filters, and the other pixel pair includes the M filters.

(4) Electronic equipment including:a solid-state imaging device that includes a pixel array portion in which a plurality of pixel unit groups are arrayed, the pixel unit group being constituted by pixel units disposed in a 2×2 matrix, the pixel unit being constituted by pixels disposed in an m×m (m is a natural number of 2 or greater) matrix, and the pixel including a photoelectric conversion unit and a color filter formed to correspond to the photoelectric conversion unit, in which each of the pixel unit groups includes an R filter transmitting red light and a C filter transmitting cyan light having a complementary color relation with the red light as the color filters in a first pixel unit among four pixel units constituting the pixel unit group, includes a G filter transmitting green light and an M filter transmitting magenta light having a complementary color relation with the green light as the color filters in each of second and third pixel units, and includes a B filter transmitting blue light and a Y filter transmitting yellow light having a complementary color relation with the blue light as the color filters in a fourth pixel unit;an optical lens that forms image light from a subject into an image on an imaging surface of the solid-state imaging device; anda signal processing circuit that performs signal processing on a signal output from the solid-state imaging device.

(5) The electronic equipment according to (4),wherein the signal processing circuit includes a binning processing unit that generates an RGB mosaic image by adding pixel values of image pixels included in each of pixel groups corresponding to the first pixel unit in a mosaic image corresponding to an array of the color filters obtained from the signal to set the pixel values as a pixel value of one image pixel having only color information of the color red, adding pixel values of image pixels having only color information of the color green in each of pixel groups corresponding to the second and third pixel units to set the pixel values as a pixel value of one image pixel having only color information of the color green, and adding pixel values of image pixels having only color information of the color blue in each of pixel groups corresponding to the fourth pixel unit to set the pixel values as a pixel value of one image pixel having only color information of the color blue.

(6) The electronic equipment according to (4) or (5),wherein the signal processing circuit includes a binning processing unit that generates a CMY mosaic image by adding pixel values of image pixels having only color information of the color cyan in each of pixel groups corresponding to the first pixel unit in a mosaic image corresponding to an array of the color filters obtained from the signal to set the pixel values as a pixel value of one image pixel having only color information of the color cyan, adding pixel values of image pixels having only color information of the color magenta in each of pixel groups corresponding to the second and third pixel units to set the pixel values as a pixel value of one image pixel having only color information of the color magenta, and adding pixel values of image pixels having only color information of the color yellow in each of pixel groups corresponding to the fourth pixel unit to set the pixel values as a pixel value of one image pixel having only color information of the color yellow.

(7) The electronic equipment according to any one of (4) to (6),wherein the signal processing circuit includes a binning processing unit that adds pixel values of image pixels having color information of red and cyan colors in each of pixel groups corresponding to the first pixel unit in a mosaic image corresponding to an array of the color filters obtained from the signal to calculate a luminance value of one image pixel, adds pixel values of image pixels having color information of green and magenta colors in each of pixel groups corresponding to the second and third pixel units to calculate a luminance value of one image pixel, and adds pixel values of image pixels having color information of blue and yellow colors in each of pixel groups corresponding to the fourth pixel unit to calculate a luminance value of one image pixel.

(8) The electronic equipment according to (7),wherein the luminance value calculation unit uses a numerical value obtained by multiplying the pixel value of the mosaic image by a correction coefficient which is set for each color temperature of a light source, as a pixel value to be used to calculate the luminance value.

(9) The electronic equipment according to (7) or (8),wherein the signal processing circuit includesa binning processing unit that generates an RGB mosaic image by adding pixel values of image pixels having only color information of the color red in each of pixel groups corresponding to the first pixel unit in a mosaic image corresponding to an array of the color filters obtained from the signal to set the pixel values as a pixel value of one image pixel having only color information of the color red, adding pixel values of image pixels having only color information of the color green in each of pixel groups corresponding to the second and third pixel units to set the pixel values as a pixel value of one image pixel having only color information of the color green, and adding pixel values of image pixels having only color information of the color blue in each of pixel groups corresponding to the fourth pixel unit to set the pixel values as a pixel value of one image pixel having only color information of the color blue, anda luminance synthesis unit that synthesizes the RGB mosaic image and the luminance value.

(10) The electronic equipment according to (7) or (8),wherein the signal processing circuit includesa binning processing unit that generates a CMY mosaic image by adding pixel values of image pixels having only color information of the color cyan in each of pixel groups corresponding to the first pixel unit in a mosaic image corresponding to an array of the color filters obtained from the signal to set the pixel values as a pixel value of one image pixel having only color information of the color cyan, adding pixel values of image pixels having only color information of the color magenta in each of pixel groups corresponding to the second and third pixel units to set the pixel values as a pixel value of one image pixel having only color information of the color magenta, and adding pixel values of image pixels having only color information of the color yellow in each of pixel groups corresponding to the fourth pixel unit to set the pixel values as a pixel value of one image pixel having only color information of the color yellow, anda luminance synthesis unit that converts the CMY mosaic image into an RGB mosaic image and then synthesizes the converted RGB mosaic image and the luminance value.

(11) The electronic equipment according to (5),wherein the binning processing unit generates the RGB mosaic image and generates a CMY mosaic image by adding pixel values of image pixels having only color information of the color cyan in each of pixel groups corresponding to the first pixel unit in a mosaic image corresponding to an array of the color filters obtained from the signal to set the pixel values as a pixel value of one image pixel having only color information of the color cyan, adding pixel values of image pixels having only color information of the color magenta in each of pixel groups corresponding to the second and third pixel units to set the pixel values as a pixel value of one image pixel having only color information of the color magenta, and adding pixel values of image pixels having only color information of the color yellow in each of pixel groups corresponding to the fourth pixel unit to set the pixel values as a pixel value of one image pixel having only color information of the color yellow, andthe signal processing circuit includes an HDR image generation unit that converts the CMY mosaic image into an RGB mosaic image and synthesizes the converted RGB mosaic image and the RGB mosaic image generated by the binning processing unit to generate a high dynamic range image.

REFERENCE SIGNS LIST