Patent Description:
In recent electronic apparatuses such as a smartphone, a mobile phone, and a personal computer (PC), various sensors such as cameras are mounted on a frame (bezel) of a display unit. On the other hand, there is a demand for making the external size of the electronic apparatuses as compact as possible without affecting a screen size, and a bezel width tends to be decreased. In view of such a background, a technique has been proposed in which a camera module is disposed immediately below a display unit, and an image of object light having passed through the display unit is captured by a camera. <CIT> relates to a display device that includes a display part, an imaging part, a defective area determining part <NUM>, and an image processing part <NUM>. <CIT> relates to an image processing apparatus comprising an acquiring unit configured to acquire pixel signals in which light fluxes from a subject are stored, each pixel signal being such that a pixel has stored a pupil-divided light flux obtained by performing pupil division on the light flux, a determination unit configured to determine, with respect to pixels of the pixel signals that are associated with a predetermined region, whether or not a pixel value of each pixel satisfies a predetermined relationship to a pixel value of a pixel having stored another light flux from the same subject, and a detection unit configured to detect that there is an external factor that varies the pixel value in the predetermined region if the pixels of the pixel signals include a predetermined number or more of pixels that are determined to satisfy the predetermined relationship.

A touch sensor is built in a display unit of a smartphone or the like, and a fingerprint, hand oil, or the like often adheres to a display surface. Furthermore, there may be scratches on the display surface or a film for protecting the display surface, or bubbles may be generated between the display surface and the film.

As described above, when an image is captured through the display unit by a camera disposed immediately below the display unit in a state in which there is an abnormality on the display surface, there is a possibility that a foreign substance is shown in the captured image or the foreign substance is in focus, and the image cannot be captured as intended.

The present disclosure provides an electronic apparatus capable of capturing an image without being affected by an abnormality on a display surface.

The following aspects may include some but not all features as literally defined in the claims and are present for illustration purposes only.

In order to solve the above problems, according to an aspect of the present disclosure, there is provided an electronic apparatus including:.

The display control unit may cause the display unit to display information urging removal of the abnormality.

The information may include information corresponding to a type of the abnormality.

The display control unit may cause the display unit to display an indicator indicating the position where the abnormality occurs.

According to another aspect of the present disclosure, there is provided an electronic apparatus including:.

The abnormality detection unit may detect the abnormality on the display surface on the basis of a plurality of images captured by the imaging unit in a state in which a region overlapping with an angle of view of the imaging unit on the display surface is caused to emit light at each of the plurality of light emission wavelengths.

According to still another aspect of the present disclosure, there is provided an electronic apparatus including:.

The correction processing unit may correct the image captured by the imaging unit on the basis of information regarding at least one of a type of the abnormality, a color of the abnormality, a size of the abnormality, a position of the abnormality, or the number of the abnormalities.

An abnormality determination unit that determines a type of the abnormality may be further provided and
the correction processing unit may perform correction processing according to the type of the abnormality determined by the abnormality determination unit on the image captured by the imaging unit.

The correction processing may include at least one of edge emphasis processing, distortion correction processing, or scratch correction processing.

A model generation unit that generates a model correcting the image captured by the imaging unit on the basis of information regarding the abnormality may be further provided, the model being trained on the basis of at least one piece of information regarding a type of the abnormality, a color of the abnormality, a size of the abnormality, a position of the abnormality, or the number of the abnormalities, and an image before and after the correction processing unit performs correction, and
the correction processing unit may correct the image captured by the imaging unit by giving the image captured by the imaging unit and the information regarding the abnormality to the trained model.

The imaging unit may include a plurality of cameras that captures images at different angles of view, and
the correction processing unit may remove the abnormality from the image captured by the imaging unit on the basis of a plurality of the images captured by the plurality of cameras.

The plurality of polarization elements may include a plurality of types of polarization elements that detects different polarization states.

A distance detection unit that detects a distance to an object imaged by the imaging unit may be further provided, and the abnormality detection unit may detect the abnormality on the basis of the distance detected by the distance detection unit.

The imaging unit may include a plurality of cameras that captures images at different angles of view, and
the distance detection unit may detect the distance on the basis of the captured images of the plurality of cameras.

There may be further provided a fingerprint detection unit that detects a fingerprint of a finger brought into contact with the display surface on the basis of an image captured by the imaging unit.

Hereinafter, embodiments of an electronic apparatus will be described with reference to the drawings. Main components of the electronic apparatus will be mainly described below, but the electronic apparatus may have components and functions that are not illustrated or described. The following description does not exclude the components and functions that are not illustrated or described. In the following, the subject-matter of the independent claim <NUM> relates to features of the sixth embodiment and the first to the fifth embodiment and the seventh to the tenth embodiment comprise some but not all features of the claims and are present for highlighting specific aspects of the claimed subject-matter.

<FIG> are schematic cross-sectional views of an electronic apparatus <NUM> according to the first embodiment. The electronic apparatus <NUM> of <FIG> is any electronic apparatus <NUM> having both a display function and an imaging function, such as a smartphone, a mobile phone, a tablet, or a PC. The electronic apparatus <NUM> of <FIG> includes a camera module (imaging unit) <NUM> disposed on the opposite side of a display surface 1a of a display unit <NUM>. As described above, in the electronic apparatus <NUM> of <FIG>, the camera module is provided on the back side of the display surface 1a of the display unit <NUM>. Therefore, the camera module captures an image through the display unit <NUM>.

<FIG> is a schematic external view of the electronic apparatus <NUM> of <FIG>, and <FIG> is a cross-sectional view taken along line A-A of <FIG>. In the example of <FIG>, the display surface 1a extends to the vicinity of the external size of the electronic apparatus <NUM>, and a width of a bezel 1b around the display surface 1a is set to several mm or less. Normally, a front camera is often mounted on the bezel 1b, but in <FIG>, as indicated by a broken line, a camera module <NUM> functioning as the front camera is disposed on a back surface side of a substantially central portion of the display surface 1a. By providing the front camera on the back surface side of the display surface 1a in this manner, it is not necessary to dispose the front camera on the bezel 1b, and the width of the bezel 1b can be decreased.

Note that, in <FIG>, the camera module <NUM> is disposed on the back surface side of the substantially central portion of the display surface 1a, but in the present embodiment, the camera module <NUM> is only required to be disposed on the back surface side of the display surface 1a, for example, the camera module <NUM> may be disposed on the back surface side near the peripheral edge portion of the display surface 1a. As described above, the camera module <NUM> in the present embodiment is disposed at an arbitrary position on the back surface side overlapping with the display surface 1a.

As illustrated in <FIG>, the display unit <NUM> is a stacked body obtained by stacking a protective film 2b, a polyimide substrate 2c, a display layer 2d, a barrier layer 2e, a touch sensor layer 2f, an adhesive layer <NUM>, a circular polarization plate <NUM>, an optical clear adhesive (OCA) 2i, and a cover glass 2j in this order. The display layer 2d may be, for example, an organic light emitting device (OLED) display layer, a liquid crystal display layer, a MicroLED, or a display layer based on other display principles. The display layer 2d may include a plurality of layers. For example, the display layer 2d may include a color filter layer, a backlight layer, and the like. The display unit <NUM> performs display using light in a visible light wavelength range, but the light displayed in the display unit <NUM> may include an infrared light component.

The barrier layer 2e is a layer that prevents oxygen and moisture from entering the display layer 2d. A touch sensor is incorporated in the touch sensor layer 2f. There are various types of touch sensors, such as a capacitive type and a resistive film type, but any type may be adopted. Furthermore, the touch sensor layer 2f and the display layer 2d may be integrated.

The adhesive layer <NUM> is provided for bonding the circular polarization plate <NUM> and the touch sensor layer 2f. A material having high visible light transmittance is used for the adhesive layer <NUM>. The circular polarization plate <NUM> is provided to reduce glare and increase visibility of the display surface 1a even in a bright environment. The optical clear adhesive 2i is provided to increase adhesion between the circular polarization plate <NUM> and the cover glass 2j. A material having high visible light transmittance is used as the optical clear adhesive 2i. The cover glass 2j is provided to protect the display layer 2d and the like. Note that the layer configuration of the display unit <NUM> is not necessarily limited to that illustrated in <FIG> and <FIG>.

The camera module <NUM> is disposed on the opposite side to the display surface 1a of the display unit <NUM>, that is, on a back side of the display unit <NUM>. The camera module <NUM> includes an imaging unit <NUM> and an optical system <NUM>. The optical system <NUM> is disposed on a light incident surface side of the imaging unit <NUM>, that is, on a side close to the display unit <NUM>, and condenses the light passing through the display unit <NUM> on the imaging unit <NUM>. The optical system <NUM> usually includes a plurality of lenses. As described later, there may be a case where a plurality of the camera modules <NUM> is disposed on the opposite side to the display surface 1a of the display unit <NUM>. In this case, a focal length of the optical system <NUM> of each of the camera modules <NUM> may be different from each other such that imaging with a different angle of view such as a wide-angle or a telephoto angle can be performed.

The imaging unit <NUM> includes a photoelectric conversion unit 4a. The photoelectric conversion unit 4a photoelectrically converts light incident through the display unit <NUM>. The photoelectric conversion unit 4a may be a complementary metal oxide semiconductor (CMOS) sensor or a charge coupled device (CCD) sensor. Furthermore, the photoelectric conversion unit 4a may be a photodiode or an organic photoelectric conversion film.

The photoelectric conversion unit 4a includes a photoelectric conversion element such as a CMOS sensor or the like for each pixel. The pixels can be arranged in any manner. Specifically, the arrangement form of pixels may be a Bayer arrangement, an interline arrangement, a check pattern arrangement, a stripe arrangement, or other arrangements.

As illustrated in <FIG> and <FIG>, the electronic apparatus <NUM> according to the present embodiment is disposed such that the display unit <NUM> and the camera module <NUM> overlap with each other in a front and back direction. Therefore, the camera module <NUM> captures an image of the object light passing through the display unit <NUM>. As illustrated in <FIG>, the display unit <NUM> includes a plurality of layers, and there is no problem as long as each layer has a high transmittance of light in a wavelength band in which the imaging unit <NUM> has sufficient sensitivity, but there is a possibility that transmittance of some layers are low in practice. For example, the visible light transmittance of the polyimide substrate 2c is not so high. Therefore, as illustrated in <FIG>, one or more openings <NUM> may be formed in a layer having not so high transmittance among a plurality of layers constituting the display unit <NUM>, and the camera module <NUM> may capture an image of the light passing through the opening <NUM>. In the example of <FIG>, a plurality of the openings <NUM> penetrating a layer other than the cover glass 2j among a plurality of the layers constituting the display unit <NUM> is provided. These openings <NUM> are provided at positions overlapping with the camera module <NUM> in a plan view of the display unit <NUM>. The diameter and the number of each opening <NUM> are set to appropriate values in consideration of the display of the display unit <NUM> and the image quality of the image captured by the imaging unit <NUM>.

Note that in a case where a highly transparent substrate is used instead of the polyimide substrate 2c, the opening <NUM> is not necessarily provided.

<FIG> is a cross-sectional view illustrating an example of a cross-sectional structure of the imaging unit <NUM>. The imaging unit <NUM> of <FIG> includes the photoelectric conversion unit 4a formed in a semiconductor substrate <NUM>, and the photoelectric conversion unit 4a is divided by an element separation layer <NUM> for each pixel. A flattening layer <NUM> is disposed on the photoelectric conversion unit 4a, and a color filter layer <NUM> is disposed on the flattening layer <NUM>. The color filter layer <NUM> may include a filter layer of three colors of RGB, or may include a filter layer of cyan, magenta, and yellow which are complementary colors of RGB. Alternatively, a filter layer that transmits colors other than visible light such as infrared light may be provided, a filter layer having a multispectral characteristic may be provided, or a filter layer of a decreased color such as white may be provided. By transmitting light other than visible light such as infrared light, sensing information such as depth information can be detected. An on-chip lens <NUM> is disposed on the color filter layer <NUM>. The light enters through the on-chip lens <NUM>. In the present specification, the side on which the on-chip lens <NUM> is disposed is referred to as a back surface side of the imaging unit <NUM>.

On a front surface side of the imaging unit <NUM>, a readout circuit <NUM> is formed on the semiconductor substrate <NUM>, and the periphery of the readout circuit <NUM> is covered by an interlayer insulation film <NUM>. The readout circuit <NUM> includes a transfer transistor, a reset transistor, an amplification transistor, a selection transistor, and the like. Note that a cross-sectional structure of the imaging unit <NUM> is not limited to the cross-sectional structure illustrated in <FIG>.

<FIG> is a block diagram illustrating an internal configuration of the electronic apparatus <NUM> according to the first embodiment. As illustrated in <FIG>, the electronic apparatus <NUM> includes an imaging device <NUM>, an application processor <NUM>, an image signal generation unit <NUM>, an A/D conversion unit <NUM>, a display control unit <NUM>, and the display unit <NUM>.

The imaging device <NUM> can include one or a plurality of semiconductor devices, and includes an A/D conversion unit <NUM>, a signal processing unit <NUM>, an imaging control unit <NUM>, an exposure adjustment unit <NUM>, an abnormality detection unit <NUM>, and an output unit <NUM> in addition to the imaging unit <NUM> included in the camera module <NUM>, the optical system <NUM>, and an infrared ray (IR) cut filter <NUM>.

The A/D conversion unit <NUM> converts an analog pixel signal captured by the imaging unit <NUM> into digital pixel data.

For example, as illustrated in detail in <FIG>, the signal processing unit <NUM> includes a clamp unit 32a, a color output unit 32b, a defect correction unit 32c, a linear matrix unit 32d, a gamma correction unit 32e, a luminance/chroma signal generation unit 32f, a noise reduction unit <NUM>, and an edge emphasis unit <NUM>.

The clamp unit 32a performs processing of defining a black level. More specifically, the clamp unit 32a performs processing of subtracting black level data from the digital pixel data. The color output unit 32b outputs, for example, pixel data for each color of RGB. The defect correction unit 32c performs processing of correcting the imaged data of the specific pixel that cannot be correctly read for some reason in accordance with the imaged data of surrounding pixels. The linear matrix unit 32d performs matrix operation on color information for RGB to perform more correct color reproduction. The gamma correction unit 32e performs gamma correction so as to enable display excellent in visibility in accordance with a display characteristic of the display unit <NUM>. For example, the gamma correction unit 32e performs conversion from ten bits to eight bits while changing a gradient. The luminance/chroma signal generation unit 32f generates a luminance/chroma signal for the display unit <NUM> to display on the basis of output data of the gamma correction unit 32e. The noise reduction unit <NUM> performs processing of reducing noise included in the luminance/chroma signal. The edge emphasis unit <NUM> performs processing of emphasizing an edge of an object image on the basis of the luminance/chroma signal. The noise reduction processing of the noise reduction unit <NUM> and the edge emphasis processing of the edge emphasis unit <NUM> may be performed only in a case where a predetermined condition is satisfied. The output unit <NUM> outputs the luminance/chroma signal subjected to the noise reduction processing.

The imaging control unit <NUM> sets a frame rate at which the imaging unit <NUM> performs imaging on the basis of the luminance/chroma signal. The exposure adjustment unit <NUM> adjusts an exposure time according to the frame rate set by the imaging control unit <NUM>. The photoelectric conversion unit 4a performs imaging for each pixel according to the exposure time adjusted by the exposure adjustment unit <NUM>.

The abnormality detection unit <NUM> detects an abnormality on the display surface of the display unit <NUM>. The abnormality includes various deposits on the display surface, scratches on the display surface, deposits and scratches on a protective material such as a film for protecting the display surface, bubbles between the display surface and the protective material, and the like. The deposits include stains, hand oil, solidified substances, liquefied substances, and the like. The scratches include breaks, cracks, chips, and the like. In this manner, the abnormality detection unit <NUM> detects a plurality of types of abnormalities. In the present specification, the abnormality may be collectively referred to as a foreign substance.

The abnormality detection unit <NUM> transmits detected abnormality information to the application processor <NUM>. Here, the abnormality information is, for example, information on the type and position of the abnormality. Furthermore, the abnormality detection unit <NUM> transmits captured image data including the abnormality information to the output unit <NUM>. The output unit <NUM> outputs the captured image data together with the abnormality information. The output unit <NUM> may send, to the image signal generation unit <NUM>, the captured image data showing the abnormality.

The application processor <NUM> is a semiconductor device separate from the camera module <NUM>, and is mounted on the same substrate as or a different substrate from the camera module <NUM>. The application processor <NUM> includes a central processing unit (CPU) and the like therein, and executes an operating system or programs such as various application software. The application processor <NUM> may have a function of performing image processing of a graphics processing unit (GPU), signal processing of a baseband processor, or the like. The application processor <NUM> executes various processing on the input image data and calculation result as necessary, performs control to display an image on the display unit <NUM> of the electronic apparatus <NUM>, or transmits the image to an external cloud server via a predetermined network.

When receiving the abnormality information sent from the abnormality detection unit <NUM>, the application processor <NUM> sends the information to the display control unit <NUM>. The display control unit <NUM> highlights a position where the abnormality detected by the abnormality detection unit <NUM> occurs on the display unit <NUM>. A specific example of highlighting will be described later.

The image signal generation unit <NUM> generates an image signal for displaying the image on the display unit <NUM>. The A/D conversion unit <NUM> converts the image signal into digital pixel data. The display control unit <NUM> performs control to display the digital pixel data on the display unit <NUM>. At this time, as described above, the position where the abnormality occurs is highlighted on the basis of the abnormality information detected by the abnormality detection unit <NUM>.

<FIG> is a flowchart illustrating processing operation of the electronic apparatus <NUM> according to the first embodiment, and illustrates, for example, processing operation of the application processor <NUM>, processing operation of the imaging control unit <NUM>, and processing operation of the display control unit <NUM>. This flowchart is continuously repeated while the electronic apparatus <NUM> is powered on.

First, it is determined whether or not a user of the electronic apparatus <NUM> starts the camera module <NUM> (Step S1). When the user does not start the camera module <NUM>, the image generated by the image signal generation unit <NUM> is displayed on the display unit <NUM> on the basis of an instruction from the application processor <NUM> (Step S2).

In a case where the user starts the camera module <NUM>, exposure is performed by the imaging unit <NUM> during a display-off period of the display unit <NUM> (Step S3). The display on the display unit <NUM> is updated at a frequency of <NUM> to <NUM> frames per second. There is a display-off period in which display is not performed between two consecutive frames. Furthermore, in one frame, the display is performed while scanning each horizontal line, and there is the display-off period in which the display is not performed between the horizontal lines. In Step S3, exposure is performed by the imaging unit <NUM> within these display-off periods.

The reason why the exposure is performed within the display-off period is that there is a possibility that the captured image is adversely affected when the camera module <NUM> performs imaging in a state in which the image is displayed on the display unit <NUM>. However, as will be described later, there may be a case where imaging is performed in a state in which the display unit <NUM> is intentionally caused to emit light in a predetermined emission color in order to detect the abnormality. Therefore, more precisely, the display-off period in Step S3 refers to a period in which the image display, which is displayed on the display unit <NUM> before the camera module <NUM> is started, is not performed.

Next, the signal processing unit <NUM> performs various signal processing on the captured image obtained by the exposure processing of the imaging unit <NUM> (Step S4).

Next, the abnormality detection unit <NUM> detects whether or not there is an abnormality on the display surface on the basis of the captured image data after the signal processing (Step S5). Various methods are conceivable as an abnormality detection method, and specific examples thereof will be described later. In a case where the abnormality is not detected, the captured image data output from the signal processing unit <NUM> is sent to the output unit <NUM> (Step S6).

When the abnormality is detected in Step S5, the abnormality detection unit <NUM> sends the abnormality information including the position where the abnormality occurs to the application processor <NUM>, and sends the abnormality information and the captured image data to the output unit <NUM> (Step S7).

The application processor <NUM> sends the abnormality information sent from the abnormality detection unit <NUM> to the display control unit <NUM>. When displaying the image based on the image signal generated by the image signal generation unit <NUM> on the display surface, the display control unit <NUM> highlights the position where the abnormality occurs on the display surface (Step S8).

<FIG>, and <FIG> are diagrams illustrating a specific example of highlighting a position where an abnormality 2p occurs. <FIG> illustrates an example of displaying an annular indicator 2q surrounding the position where the abnormality 2p occurs. The indicator 2q of <FIG> is desirably displayed in a display color different from the display color of the background image in a noticeable manner. Furthermore, a line width of the indicator 2q may be increased.

<FIG> illustrates an example of displaying an arrow indicator 2q indicating the position where the abnormality 2p occurs. Furthermore, a message "Please wipe" may be written as character information at an arrow head of the arrow.

<FIG> illustrates an example in which the abnormality 2p included in the captured image data is enlarged and displayed on the display surface in order to notify the user of a type of abnormality 2p. <FIG> illustrates an example in which the abnormality 2p is a fingerprint, and the user can grasp the type of the abnormality by using the enlarged image.

<FIG> illustrates an example in which the abnormality 2p is damage to a film 2r stuck to the display surface. In this case, a message "Please replace a film" or the like may be written as character information.

As described above, in the first embodiment, when the abnormality 2p on the display surface is detected at the time of imaging by the imaging unit <NUM> disposed on the opposite side to the display surface of the display unit <NUM>, the position where the abnormality 2p occurs is highlighted on the display unit <NUM>. Therefore, it is possible to notify the user in advance that there is a problem in imaging by the imaging unit <NUM>, and to urge the user to remove the abnormality 2p. An image quality of the captured image can be improved when the user performs the imaging by the imaging unit <NUM> after performing a work of eliminating the abnormality 2p, such as wiping the display surface, at the time of highlight of the display unit <NUM>.

In the second embodiment, the abnormality 2p is detected on the basis of a plurality of captured images captured in a state of emitting light at a plurality of emission wavelengths.

<FIG>, <FIG> are diagrams schematically illustrating an example in which exposure is performed by the imaging unit <NUM> in a state in which a display region <NUM>, which is a part of the display surface 1a and in an angle-of-view range of the camera module <NUM>, is caused to emit light at different emission wavelengths. For example, <FIG> illustrate examples in which light is emitted at emission wavelengths of red, green, and blue to perform exposure, respectively.

As described above, the imaging by the camera module <NUM> is performed in the display-off period such as between frames or between the horizontal lines, but the light emissions in <FIG> are for detecting the foreign substances and is performed using a part of the display period.

The electronic apparatus <NUM> according to the second embodiment has an internal configuration similar to that in <FIG> and <FIG>, but performs processing operation partially different from that in <FIG>.

<FIG> is a flowchart illustrating processing operation of the electronic apparatus <NUM> according to the second embodiment. Hereinafter, the processing operation different from that in <FIG> will be mainly described. In a case where the user does not start the camera module <NUM>, the same operation as in Steps S1 and S2 of <FIG> is performed (Steps S11 and S12).

In a case where the user starts the camera module <NUM>, as illustrated in <FIG>, the partial display region <NUM>, which is within the angle-of-view range of the camera module <NUM>, is caused to emit light at a first emission wavelength (Step S13). Since an angle-of-view range varies depending on the focal length of the lens included in the camera module <NUM>, the size of the display region <NUM> caused to emit the light at the first emission wavelength also needs to be adjusted according to the focal length of the lens. Alternatively, the entire region of the display surface 1a may emit the light at the first emission wavelength.

While the light is emitted in Step S13, the imaging unit <NUM> performs exposure (Step S14), and the signal processing unit <NUM> performs various signal processing to generate captured image data (Step S15).

Next, the same operation as in Steps S13 to S15 is performed in a state in which the display region <NUM> is caused to emit light at a second emission wavelength as illustrated in <FIG> (Steps S16 to S18). Subsequently, the same operation as in Steps S13 to S15 is performed in a state in which the display region <NUM> is caused to emit light at a third emission wavelength as illustrated in <FIG> (Steps S19 to S21).

The abnormality detection unit <NUM> detects the presence or absence of the abnormality 2p on the basis of the three pieces of captured image data obtained in Steps S15, S18, and S21 (Steps S22 to S24).

As described above, in the second embodiment, the imaging unit <NUM> performs the imaging in a state in which the display region <NUM>, which is at least a part of the display unit <NUM>, is caused to emit light at a plurality of types of emission wavelengths, and the abnormality detection unit <NUM> detects the abnormality 2p on the basis of a plurality of pieces of the captured image data obtained. According to the type of abnormality 2p, only light of a specific emission wavelength is reflected or transmitted. Therefore, by performing the imaging in a state in which the display unit <NUM> emits light at a plurality of types of emission wavelengths, the presence or absence of the abnormality 2p and the type of abnormality 2p can be easily specified.

In the third embodiment, when the abnormality detection unit <NUM> detects the abnormality 2p, the correction processing for the captured image data is performed.

<FIG> is a block diagram illustrating an internal configuration of the electronic apparatus <NUM> according to the third embodiment. The electronic apparatus <NUM> of <FIG> is obtained by adding a correction processing unit <NUM> to the electronic apparatus <NUM> of <FIG>. The correction processing unit <NUM> corrects the image displayed on the display unit <NUM> so that the abnormality 2p is not noticeable. More specifically, the correction processing unit <NUM> performs correction processing on the captured image data output from the signal processing unit <NUM> on the basis of information regarding the abnormality 2p detected by the abnormality detection unit <NUM>, and outputs the image data after the correction processing. The image data output from the correction processing unit <NUM> is sent to the output unit <NUM>. The correction processing performed by the correction processing unit <NUM> may include, for example, at least one of edge emphasis processing, distortion correction processing, and scratch correction processing.

The abnormality detection unit <NUM> in the electronic apparatus <NUM> according to the third embodiment may cause the display surface 1a to emit light at a plurality of emission wavelengths to detect the abnormality 2p of the display surface 1a as in the second embodiment. The abnormality detection unit <NUM> according to the present embodiment can detect the type of abnormality 2p, and an absorption characteristic and a scattering characteristic for a specific emission wavelength change in accordance with the type of abnormality 2p. The abnormality detection unit <NUM> can relatively accurately estimate the type of abnormality 2p from a correspondence relationship between the emission wavelength and the absorption characteristic or scattering characteristic. When the abnormality detection unit <NUM> can detect the type of abnormality 2p, the correction processing unit <NUM> can perform appropriate correction processing according to the type of abnormality 2p.

<FIG> is a flowchart illustrating a processing procedure of the correction processing performed by the correction processing unit <NUM>. The correction processing of <FIG> is performed instead of Steps S7 and S8 of <FIG> or Steps S24 and S25 of <FIG> in a case where it is determined that there is an abnormality on the display surface 1a in Step S5 of <FIG> or Step S22 of <FIG>.

First, the type of abnormality 2p is determined on the basis of the information regarding the abnormality 2p detected by the abnormality detection unit <NUM> (Step S31).

Next, it is determined whether or not the correction processing unit <NUM> of the electronic apparatus <NUM> can perform effective correction processing (Step S32). For example, in a case where the electronic apparatus <NUM> is a portable apparatus such as a smartphone, hardware performance is inferior to that of a PC, a server, or the like, and it is difficult to perform advanced correction processing. Therefore, in a case where the correction processing satisfying the user cannot be performed within several seconds, it is determined that the effective correction processing cannot be performed. The determination processing in Step S32 is performed by the correction processing unit <NUM>, for example.

When it is determined that the effective correction processing can be performed in Step S32, the correction processing unit <NUM> performs the correction processing according to the type of abnormality 2p (Step S33). For example, when the abnormality detection unit <NUM> detects that the abnormality 2p is hand oil adhering on the display surface 1a, the edge of the captured image captured via the hand oil tends to be blurred. Therefore, the edge emphasis processing is performed. In this case, the edge emphasis processing may be performed inside the correction processing unit <NUM>, or the correction processing unit <NUM> may instruct the edge emphasis unit <NUM> of the signal processing unit <NUM> to enhance the edge emphasis processing. Furthermore, when the abnormality detection unit <NUM> detects that the abnormality 2p is a water droplet adhering to the display surface 1a, the object light is refracted by the water droplet, and thus distorted captured image data is obtained. Therefore, the correction processing unit <NUM> may perform distortion correction processing, and when the abnormality detection unit <NUM> detects that the abnormality 2p is a scratch on the display surface 1a, the correction processing unit <NUM> may perform correction processing of removing the scratch. The image data after the correction processing is sent to the output unit <NUM> (Step S34).

On the other hand, when it is determined that the effective correction processing cannot be performed in Step S32, the captured image data output from the signal processing unit <NUM> and the information regarding the abnormality 2p detected by the abnormality detection unit <NUM> are transmitted to a specific information processing apparatus that performs advanced correction processing (Step S35). The information processing apparatus may be, for example, a server apparatus connected to a network or a high-performance PC. Thereafter, in the information processing apparatus, the correction processing is performed on the captured image data and the correction processing unit <NUM> receives the image data after the correction processing (Step S36). The received image data after the correction processing is sent to the output unit <NUM>.

<FIG> is a block diagram illustrating an example of a connection form between the electronic apparatus <NUM> and an information processing apparatus <NUM>. In the example of <FIG>, the electronic apparatus <NUM> and the information processing apparatus <NUM> transmit and receive information via a network <NUM>. The information processing apparatus <NUM> may be provided in an unspecified place separated from the electronic apparatus <NUM>, that is, in a cloud environment, or may be provided in a place (for example, a base station or the like) relatively close to the electronic apparatus <NUM> in a similar manner to a mobile edge computing (MEC).

A database apparatus <NUM> is connected to the information processing apparatus <NUM>, and the database apparatus <NUM> stores information regarding a correspondence relationship between the type of abnormality 2p and the correction processing and information regarding a model that performs the correction processing according to the type of abnormality 2p. The information processing apparatus <NUM> learns a model by using the type of abnormality 2p and the captured image data as input data for the model such that the captured image data subjected to appropriate correction processing is output. In the process of learning, weight information of a neural network is variously changed to update the model. The information processing apparatus <NUM> accumulates various types of abnormality 2p and a result of the correction processing in the database apparatus <NUM>, repeats learning of the model, and thus a highly reliable model can be generated. By learning such a model in advance, it is possible to perform highly reliable correction processing in a short time on the type of abnormality 2p and the captured image data, which are sent from the electronic apparatus <NUM> and return the image data after the correction processing to the electronic apparatus <NUM>.

Note that, in the flowchart of <FIG>, an example has been described in which the correction processing is performed by the external information processing apparatus <NUM> in a case where it is determined that the effective correction processing cannot be performed inside the electronic apparatus <NUM>. However, in a case where the external information processing apparatus <NUM> does not exist or in a case where communication with the external information processing apparatus <NUM> cannot be performed for some reason, the fact that the effective correction processing cannot be performed may be displayed on the display unit <NUM>.

As described above, in the third embodiment, when the abnormality 2p is detected by the abnormality detection unit <NUM>, the correction processing for the captured image data is performed such that the abnormality 2p is not noticeable. Therefore, high-quality captured image data can be obtained without the user performing the work of removing the abnormality 2p on the display surface 1a.

In the fourth embodiment, a plurality of the camera modules <NUM> is disposed on the opposite side to the display surface 1a of the display unit <NUM>.

<FIG> is a plan view of the electronic apparatus <NUM> according to the fourth embodiment. In the electronic apparatus <NUM> of <FIG>, three camera modules <NUM> are disposed on the opposite side to the display surface 1a of the display unit <NUM>. Note that the number of camera modules <NUM> is not limited to three. A focal length of each camera module <NUM> may be the same or different.

<FIG> is a cross-sectional view taken along line A-A of <FIG>. The imaging range of each camera module <NUM> is indicated by a broken line. In the present embodiment, at least two camera modules <NUM> can image an arbitrary place on the display surface 1a of the display unit <NUM>. Therefore, the abnormality 2p occurring at an arbitrary place on the display surface 1a can be imaged by a plurality of the camera modules <NUM>.

The type of abnormality 2p can be easily specified by imaging an abnormal place on the display surface 1a in different directions. The abnormality detection unit <NUM> detects the type of abnormality 2p with reference to a plurality of pieces of captured image data captured by a plurality of the camera modules <NUM>. At that time, similarly to the second embodiment, the imaging may be performed in a state in which the display surface 1a is caused to emit light at a plurality of different emission wavelengths.

The correction processing unit <NUM> performs correction processing for removing the abnormality 2p on the basis of a plurality of pieces of captured image data captured by a plurality of the camera modules <NUM>. It is possible to perform effective correction processing when performing the correction processing on the basis of a plurality of captured image data captured in a plurality of directions as compared with when performing the correction processing on the basis of captured image data captured in one direction. Note that, in a case where the correction processing unit <NUM> determines that the effective correction processing cannot be performed, a plurality of pieces of the captured image data captured by a plurality of the camera modules <NUM> may be transmitted to the external information processing apparatus <NUM>, and the information processing apparatus <NUM> may perform the correction processing.

As described above, in the fourth embodiment, by providing a plurality of the camera modules <NUM>, the abnormality 2p in an arbitrary place on the display surface 1a can be imaged by at least two camera modules <NUM>. Therefore, it is possible to more accurately detect the presence or absence of the abnormality 2p and the type of the abnormality 2p, and perform the correction processing for removing the abnormality 2p.

The fifth embodiment relates to a specific detection method of the abnormality 2p, and can be applied to the abnormality detection unit <NUM> according to the first to fourth embodiments.

<FIG> is a plan view illustrating an example of a pixel arrangement of the imaging unit <NUM> according to the fifth embodiment. In the example of <FIG>, pixels for three colors of RGB are disposed in a two-dimensional direction. Each of the pixels includes a photoelectric conversion unit 4a, and converts incident light into an electric signal corresponding to a light amount. Some of a plurality of the pixels are pixels for phase difference detection, and a pixel region is divided into two. Each of the divided pixels 10a and 10b in the pixel for the phase difference detection individually performs photoelectric conversion. A difference between the electrical signals photoelectrically converted by two divided pixels 10a and 10b is phase difference information, and a defocus amount can be detected on the basis of the phase difference information. The defocus amount can be used to adjust focus. In the example of <FIG>, the example is described in which the pixels for phase difference detection are disposed at four corners, but where and how many the pixels for phase difference detection are disposed are arbitrary.

The presence or absence of the abnormality 2p can be detected by using the defocus amount obtained by the two divided pixels 10a and 10b constituting the pixel for the phase difference detection. That is, in a case where there is a foreign substance on the display surface 1a, the pixel for the phase difference detection recognizes the foreign substance as a part of the object and detects the defocus amount. The foreign substance can be detected by using the detected defocus amount. For example, in a case where a plurality of the camera modules <NUM> is disposed as illustrated in <FIG>, at least two camera modules <NUM> detect the defocus amount caused by the foreign substance, and thus it is possible to specify that the detected defocus amount is not caused by an originally intended object but caused by the foreign substance on the display surface 1a.

As described above, in the fifth embodiment, by providing the pixels for phase difference detection in a part of the pixel arrangement for imaging, it is possible to accurately detect the abnormality 2p on the display surface 1a.

The sixth embodiment detects the abnormality 2p on the display surface 1a by a method different from that of the fifth embodiment. Processing of detecting the abnormality 2p according to the sixth embodiment can be applied to the abnormality detection unit <NUM> according to the first to fourth embodiments.

<FIG> is a plan view illustrating an example of a pixel arrangement of the imaging unit <NUM> according to the sixth embodiment. In the example of <FIG>, similarly to <FIG>, pixels for three colors of RGB are disposed in a two-dimensional direction. Each of the pixels includes a photoelectric conversion unit 4a. In some pixels, a polarization element 8b is disposed on a light incident surface side of the photoelectric conversion unit 4a. Hereinafter, the pixel in which the polarization element 8b is disposed is referred to as a polarization pixel 10c, and the pixel in which the polarization element 8b is disposed is referred to as a non-polarization pixel 10d.

The polarization element 8b polarizes light that is incident through the display unit <NUM>. The light polarized by the polarization element 8b is incident on the corresponding photoelectric conversion unit 4a and is photoelectrically converted.

<FIG> illustrates an example in which <NUM> pixels are the non-polarization pixels 10d and four pixels are the polarization pixels 10c among <NUM> × <NUM> = <NUM> pixels, but a ratio between the non-polarization pixel 10d and the polarization pixel 10c and a location of the polarization pixel 10c are arbitrary.

<FIG> is a cross-sectional view of the imaging unit <NUM> according to the sixth embodiment. In <FIG>, the same components as those in <FIG> are denoted by the same reference numerals, and differences will be mainly described below. In the imaging unit <NUM> of <FIG>, a light shielding layer <NUM> and an underlying insulation layer <NUM> are disposed on the flattening layer <NUM>, and a plurality of the polarization elements 8b is disposed on the underlying insulation layer <NUM> in a state of being separated from each other. The polarization element 8b is, for example, a wire grid polarization element 8b having a line-and-space structure disposed on a part of the underlying insulation layer <NUM>.

<FIG> is a perspective view illustrating an example of a detailed structure of each polarization element 8b. As illustrated in <FIG>, each of a plurality of the polarization elements 8b includes a plurality of line portions 8d having a protruding shape extending in one direction and a space portion 8e between the line portions 8d. There is a plurality of types of polarization elements 8b in which extending directions of the line portions 8d are different from each other. More specifically, there are three or more types of polarization elements 8b, and for example, three types of angles formed by the arrangement direction of the photoelectric conversion unit 4a and the extending direction of each of the line portions 8d may be zero degrees, <NUM> degrees, and <NUM> degrees. Alternatively, the angle between the arrangement direction of the photoelectric conversion unit 4a and the extending direction of the line portion 8d may be four types of angles of zero degrees, <NUM> degrees, <NUM> degrees, and <NUM> degrees, or may be other angles. Alternatively, a plurality of the polarization elements 8b may polarize light only in a single direction. A material of a plurality of the polarization elements 8b may be a metal material such as aluminum or tungsten, or an organic photoelectric conversion film.

As illustrated in <FIG>, in a case where the imaging unit <NUM> includes four types of polarization elements 8b, each of the polarization elements 8b passes light each having a different polarization state, and the corresponding photoelectric conversion unit 4a performs the photoelectric conversion. In a case where there is the abnormality 2p on the display surface 1a, the types of light passing through four types of polarization elements 8b change in accordance with the type of abnormality 2p. Therefore, the type of abnormality 2p can be specified by analyzing electric signals photoelectrically converted by four types of polarization pixels 10c.

Furthermore, the polarization pixel 10c can also be used for extracting and removing components of flare and diffraction light which are incident on the display surface 1a.

<FIG> is a block diagram illustrating an internal configuration of a signal processing unit 32A of the electronic apparatus <NUM> having a function of suppressing an influence of the flare light and the diffraction light. A configuration other than the signal processing unit 32A of the electronic apparatus <NUM> is similar to that in <FIG>.

In <FIG>, the same components as those in <FIG> are denoted by the same reference numerals, and differences will be mainly described below. The signal processing unit 32A of <FIG> includes a polarization output unit 32j, a flare extraction unit <NUM>, and a flare correction signal generation unit <NUM> in addition to the internal configuration of the signal processing unit <NUM> of <FIG>.

The polarization output unit 32j outputs polarization information data. The flare extraction unit <NUM> extracts at least one of the flare component or the diffraction light component from the polarization information data. At least one of the flare component and the diffraction light component extracted by the flare extraction unit <NUM> is a correction amount. The flare correction signal generation unit <NUM> corrects digital pixel data by subtracting the correction amount extracted by the flare extraction unit <NUM> from the digital pixel data output from the color output unit 32b. The output data of the flare correction signal generation unit <NUM> is digital pixel data from which at least one of the flare component or the diffraction light component is removed. As described above, the flare correction signal generation unit <NUM> functions as a correction unit that corrects the captured image photoelectrically converted by a plurality of the non-polarization pixels 10d on the basis of the polarization information.

A signal level of the digital pixel data at the pixel position of the polarization pixel 10c is lower by an amount of light passing through the polarization element 8b. Therefore, the defect correction unit 32c regards the polarization pixel 10c as a defect and performs predetermined defect correction processing.

As illustrated in <FIG>, by acquiring the polarization information with a plurality of the polarization element 8b, it is possible to generate captured image in a state in which the flare component and diffraction light component included in the light, which is repeatedly reflected in the display unit <NUM> and incident on a plurality of the non-polarization pixels 10d, are simply and reliably removed.

As described above, in the sixth embodiment, since the polarization pixel 10c using the polarization element 8b is provided in a part of the pixel arrangement for imaging, the abnormality 2p on the display surface 1a can be detected using the polarization pixel 10c. Furthermore, in the polarization pixel 10c, the flare and diffraction light incident on the display surface 1a can be extracted and removed.

In the seventh embodiment, the abnormality 2p on the display surface 1a is detected using a sensor that optically detects a distance to the object.

<FIG> is a block diagram illustrating a schematic configuration of the electronic apparatus <NUM> according to the seventh embodiment. The electronic apparatus <NUM> of <FIG> includes a distance measurement unit <NUM> in addition to the block configuration of the electronic apparatus <NUM> of <FIG>. The distance measurement unit <NUM> includes a light projection unit 27a and a light reception unit 27b. The light projection unit 27a intermittently emits pulsed light having a predetermined wavelength, and the light reception unit 27b receives the light emitted by the light projection unit 27a and reflected by the object. The distance measurement unit <NUM> measures a distance to the object on the basis of a difference between a time when light is emitted by the light projection unit 27a and a time when the light is received by the light reception unit 27b. The light projection unit 27a may perform scanning in the light projecting direction within a predetermined angle range. Therefore, the light from the light projection unit 27a can be emitted to the entire region of the display surface 1a.

In a case where there is the abnormality 2p on the display surface 1a of the display unit <NUM>, when the abnormality 2p is a foreign substance, the light from the light projection unit 27a is reflected. Therefore, since a distance from the distance measurement unit <NUM> to the display surface 1a is known in advance, the abnormality detection unit <NUM> can accurately detect whether or not the object is a foreign substance on the display surface 1a by using the distance measured by the distance measurement unit <NUM>.

As described above, in the seventh embodiment, since the distance measurement unit <NUM> that optically detects the distance to the object is provided, it is possible to simply and accurately detect whether or not there is the abnormality 2p on the display surface 1a.

Note that, instead of measuring the distance to the object, by using the time until the light reception unit 27b receives the light after the light projection unit 27a projects light, a plurality of cameras that captures images at different angles of view may be provided, and the distance to the object may be measured on the basis of the captured images captured by a plurality of the cameras.

In the first to seventh embodiments described above, an example has been described in which the imaging unit <NUM> of the electronic apparatus <NUM> is used to image the object, but the imaging unit <NUM> can be used for biometric authentication applications such as fingerprint detection, iris detection, retina detection, vein detection, and biometric detection. Since an appropriate focal length varies depending on the detection target of the living body, the electronic apparatus <NUM> of the first to seventh embodiments can be applied to detection of various pieces of biological information by providing a lens having a focal length matching the detection target and optimizing a prism, a light source, and the like.

In the electronic apparatus <NUM> according to the ninth embodiment, the optical system <NUM> of the camera module <NUM> is different from that of the first to eighth embodiments.

<FIG> is a diagram illustrating a cross-sectional structure of the imaging unit <NUM> of the camera module <NUM> mounted on an electronic apparatus <NUM> according to the ninth embodiment. The imaging unit <NUM> of <FIG> includes a microlens array <NUM> instead of a single lens or a lens group in which single lenses are arranged in an optical axis direction.

More specifically, the imaging unit <NUM> of <FIG> includes a photoelectric conversion unit 4a disposed along a bottom surface of a housing <NUM>, the microlens array <NUM> disposed above the photoelectric conversion unit 4a, a plurality of light shielding bodies <NUM> disposed between adjacent microlenses <NUM>, and a light guide plate <NUM> disposed above the microlens array <NUM>. The imaging unit <NUM> of <FIG> is applicable to any of the first to eighth embodiments described above.

Various electronic apparatuses can be considered as specific candidates of the electronic apparatus <NUM> having the configuration described in the first to ninth embodiments. For example, <FIG> is a plan view in a case where the electronic apparatus <NUM> according to the first to ninth embodiments is applied to a capsule endoscope <NUM>. The capsule endoscope <NUM> of <FIG> includes, for example, a camera (ultra-small camera) <NUM> for capturing an image in a body cavity, a memory <NUM> for recording image data captured by the camera <NUM>, and a wireless transmitter <NUM> for transmitting the recorded image data to the outside via an antenna <NUM> after the capsule endoscope <NUM> is discharged to the outside of a subject, in a housing <NUM> having both end surfaces hemispherical and a cylindrical central portion.

Furthermore, in the housing <NUM>, a central processing unit (CPU) <NUM> and a coil (magnetic force/current conversion coil) <NUM> are provided. The CPU <NUM> controls imaging by the camera <NUM> and data accumulation operation in the memory <NUM>, and controls data transmission of the wireless transmitter <NUM> from the memory <NUM> to a data reception device (not illustrated) outside the housing <NUM>. The coil <NUM> supplies power to the camera <NUM>, the memory <NUM>, the wireless transmitter <NUM>, the antenna <NUM>, and a light source 52b to be described later.

Moreover, the housing <NUM> is provided with a magnetic (reed) switch <NUM> for detecting a state in which the capsule endoscope <NUM> is set in the data reception device. The CPU <NUM> supplies power from the coil <NUM> to the wireless transmitter <NUM> when the reed switch <NUM> detects the state of being set to the data reception device and data transmission becomes possible.

The camera <NUM> includes, for example, an image sensor 52a including an objective optical system for capturing an image in the body cavity, and a plurality of the light sources 52b for illuminating the body cavity. Specifically, the camera <NUM> includes, as the light source 52b, for example, a complementary metal oxide semiconductor (CMOS) sensor or a charge coupled device (CCD), which includes a light emitting diode (LED).

The display unit <NUM> of the electronic apparatus <NUM> according to the first to ninth embodiments is a display unit including a light emitting body such as the light source 52b of <FIG>. The capsule endoscope <NUM> of <FIG> includes, for example, two light sources 52b, but these light sources 52b can be configured by a display panel having a plurality of light source units or an LED module having a plurality of LEDs. In this case, by disposing the imaging unit <NUM> of the camera <NUM> below the display panel or the LED module, restrictions on the layout arrangement of the camera <NUM> are reduced, and the capsule endoscope <NUM> having a smaller size can be realized.

Furthermore, <FIG> is a rear view in a case where the electronic apparatus <NUM> according to the first to ninth embodiments is applied to a digital single-lens reflex camera <NUM>. The digital single-lens reflex camera <NUM> and a compact camera include the display unit <NUM> that displays a preview screen on a back surface on an opposite side to the lens. The camera module <NUM> may be disposed on the opposite side to the display surface of the display unit <NUM> such that a face image of a photographer can be displayed on the display surface 1a of the display unit <NUM>. In the electronic apparatus <NUM> according to the first to ninth embodiments, since the camera module <NUM> can be disposed in a region overlapping the display unit <NUM>, it is not necessary to provide the camera module <NUM> in a frame portion of the display unit <NUM>, and the size of the display unit <NUM> can be increased as much as possible.

<FIG> is a plan view illustrating an example in which the electronic apparatus <NUM> according to the first to ninth embodiments is applied to a head mounted display (HMD) <NUM>. The HMD <NUM> of <FIG> is used for virtual reality (VR), augmented reality (AR), mixed reality (MR), substituional reality (SR), and the like. As illustrated in <FIG>, in the current HMD, a camera <NUM> is mounted on the outer surface, and the wearer of the HMD can visually recognize the surrounding image, but there is a problem that the surrounding people cannot recognize the expression of eyes or a face of the wearer of the HMD.

Therefore, in <FIG>, the display surface of the display unit <NUM> is provided on the outer surface of the HMD <NUM>, and the camera module <NUM> is provided on the opposite side to the display surface of the display unit <NUM>. Therefore, the expression of the face of the wearer, which is imaged by the camera module <NUM>, can be displayed on the display surface of the display unit <NUM>, and the people around the wearer can grasp the expression of the face of the wearer and the movement of the eyes of the wearer in real time.

In the case of <FIG>, since the camera module <NUM> is provided on the back surface side of the display unit <NUM>, there is no restriction on the installation location of the camera module <NUM>, and the degree of freedom in design of the HMD <NUM> can be increased. Furthermore, since the camera can be disposed at an optimum position, it is possible to prevent problems such as misalignment of the eyes of the wearer displayed on the display surface.

As described above, in the tenth embodiment, the electronic apparatus <NUM> according to the first to ninth embodiments can be used for various applications, and a utility value can be increased.

Claim 1:
An electronic apparatus (<NUM>) comprising:
a display unit (<NUM>);
an imaging unit (<NUM>) that is disposed on an opposite side to a display surface (1a) of the display unit (<NUM>), wherein the imaging unit (<NUM>) is disposed below the display unit (<NUM>) and captures an image through the display unit (<NUM>);
an abnormality detection unit (<NUM>) that detects an abnormality (2p) on the display surface (1a); and
a display control unit (<NUM>) that highlights a position where the abnormality (2p) detected by the abnormality detection unit (<NUM>) occurs on the display unit (<NUM>),
wherein the imaging unit (<NUM>) includes:
a plurality of photoelectric conversion units (4a) that photoelectrically converts light incident through the display unit (<NUM>); and
characterized in that the the imaging unit further includes:
a plurality of polarization elements (8b) that is disposed on a light incident side of at least one of the plurality of photoelectric conversion units (4a),
wherein the abnormality detection unit (<NUM>) detects the abnormality (2p) on a basis of polarization information polarized by the plurality of polarization elements (8b) and photoelectrically converted by the photoelectric conversion units (4a), and
wherein the plurality of polarization elements (8b) includes a plurality of types of polarization elements that detects different polarization states.