An image-pickup unit includes an image sensor and a color filter. Each filter segment of the color filter corresponds to one of a plurality of pixels of the image sensor, and the plurality of filter segments include first to Z-th (2L−1≦Z≦2L, where L is an integer equal to or larger than four) filter segments having spectral transmittances for transmitted wavelength bands different from each other among light from an object. Each pixel of the image sensor receives light of a plurality of wavelength bands. The plurality of filter segments are irregularly disposed.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to production of a multiband image made of spectral images of four or more different wavelength bands.

Description of the Related Art

A multiband image is effective in identifying the kind of an imaged object and in examining the state of the object, and is exploited in various fields such as remote sensing, biotechnology, medical science, and machine vision. The multiband image is also exploited in the field of computer graphics so as to precisely depict colors of the object. The multiband image has a three-dimensional structure of two spatial dimensions and one wavelength dimension. Different methods for acquiring the multiband image yield different spatial resolutions, wavelength resolutions, and temporal resolutions for the image.

U.S. Patent Application Publication No. 2010/0140461 discloses a method of acquiring a multiband image through an array of color filters that are disposed on an image sensor and have spectral transmittances different from each other. U.S. Patent Application Publication No. 2007/0296965 discloses a method of acquiring a multiband image through combination of image estimation processing and a compression sensing optical system including an encoding mask and a dispersive element. These methods disclosed in U.S. Patent Application Publication No. 2010/0140461 and U.S. Patent Application Publication No. 2007/0296965 can acquire a multiband image with single image capturing and thus yield a high temporal resolution.

In the method disclosed in U.S. Patent Application Publication No. 2010/0140461, light incident on a pixel has a single wavelength band. Thus, an increase in the wavelength resolution results in a decreased incident light amount and a decreased image capturing sensitivity. The method disclosed in U.S. Patent Application Publication No. 2007/0296965 requires a multi-imaging optical system and thus an increased size of the system.

SUMMARY OF THE INVENTION

The present invention provides an image-pickup unit, an image-pickup apparatus, and an image processing system that each have a small configuration and can acquire a highly sensitive multiband image through single image capturing.

An image-pickup unit as an aspect of the present invention includes an image sensor including M×N pixels and configured to photoelectrically convert an object image, and a color filter disposed on an object side of the image sensor, and including a plurality of filter segments. Each filter segment corresponds to one of the pixels of the image sensor. The plurality of filter segments include first to Z-th filter segments having spectral transmittances different from each other. The image-pickup unit satisfies predetermined conditions.

DESCRIPTION OF THE EMBODIMENTS

FIG. 1is a schematic diagram of an image-pickup unit110of a multiband image acquisition system (image processing system) according to an embodiment of the present invention. The multiband image acquisition system acquires a multiband image made of spectral images of four or more different wavelength bands. The image-pickup unit110includes an image sensor111and a color filter130disposed on an object side of the image sensor111. The image sensor111includes a plurality (M×N) of pixels configured to perform image pickup of an object.

FIG. 2is a schematic diagram of a configuration of the color filter130. As illustrated inFIG. 2, the color filter130includes a plurality of filter segments131each configured to receive light from the object, serving as a color filter. The plurality of filter segments131are disposed two-dimensionally (in a matrix array). Each filter segment131corresponds to one of the pixels of the image sensor111. The plurality of filter segments131include first to Z-th filter segments132having spectral transmittances for different wavelength bands of the light from the object. For example, the plurality of filter segments131may include a plurality of sets of the first to Z-th filter segments132. In this case, the first to Z-th filter segments132are arranged differently between the sets (irregularly arranged).

FIGS. 3A to 3Ceach illustrate wavelength bands obtainable by appropriately selecting the spectral characteristic of the color filter130. InFIGS. 3A to 3C, a horizontal axis represents a wavelength λ, and a vertical axis represents a spectral transmittance T.FIG. 3Aillustrates an example of acquiring a multiband image of L different wavelength bands in a measured wavelength range.FIG. 3Billustrates an example of acquiring a multiband image of L different wavelength bands having intervals between them.FIG. 3Cillustrates an example of acquiring a multiband image of L different wavelength bands overlapping between neighbors.

FIG. 4illustrates an exemplary spectral transmittance characteristic with the number of wavelength bands L=4 and the number of filter segments Z=16. InFIG. 4, a horizontal axis represents the wavelength λ, and a vertical axis represents the spectral transmittance T. First to sixteenth filter segments have spectral transmittance characteristics different from each other, including a filter segment that only transmits light of a single wavelength band, and a filter segment that transmits light of a plurality of wavelength bands. The filter segment that only transmits light of a single wavelength band functions as a primary color filter. The filter segment that transmits light of a plurality of wavelength bands functions as a complementary color filter. Employing a plurality of complementary color filters increases the amount of incident light to be taken in.

The phrase “different spectral transmittance characteristics” means that transmitted wavelength bands of the light from the object are different from each other. For example, when a filter segment has the same characteristic as that of the first filter segment inFIG. 4except for a different spectral transmittance in the same wavelength band, such as a lower spectral transmittance T in a wavelength band k1, this filter segment is regarded as having the same spectral transmittance characteristic as that of the first filter segment. Once a target wavelength band is determined, the “different spectral transmittance characteristics” exclude characteristics of having transmitted wavelength bands different from each other within the same wavelength band (for example, shorter and longer transmitted wavelength ranges within the determined wavelength band k1).

When equal numbers of filter segments having the spectral characteristics illustrated inFIG. 4are disposed on the color filter130, 50% of an incident light amount is received for each wavelength band. This value is higher than the value of 25% obtained with primary color filters disposed in a matrix array and the same number (or four) of wavelength bands as disclosed in U.S. Patent Application Publication No. 2010/0140461. This indicates that the image-pickup unit110allows a large amount of incident light to be taken in.

In this embodiment, a condition below allows a large amount of incident light to be taken in even when the number of wavelength bands is increased:

0.4≤1L⁢∑k=1L⁢⁢PkM⁢⁢N≤0.9(1)2L-1≤Z≤2L(2)
where L is an integer equal to or larger than four, and Pkis the number of pixels that receive light of a wavelength band k (k=1, 2, . . . , L).

Since the color filter130includes a plurality of complementary color filters, the image sensor111includes a large number of pixels each having color information of mixed bands. Thus, each wavelength band information needs to be taken out of the pixels having mixed color information so as to obtain a multiband image of the number of wavelength bands L.

This embodiment employs image estimation processing described later to pick up each wavelength band from the pixels having mixed color information. Expressions (1) and (2) are conditional expressions to be satisfied so as to take in a large amount of incident light and facilitate the separation of each wavelength band from the pixels receiving mixed light in the image estimation processing.

Expression (1) indicates that a larger number of filter segments than normal are included. A value exceeding the lower limit of Expression (1) facilitates the separation of each wavelength band but reduces the amount of incident light to be taken in. A value exceeding the upper limit increases the amount of incident light to be taken but makes difficult the separation of each wavelength band.

Expression (2) indicates that one pixel receives light of a plurality of wavelength bands. Color filters (filter segments) having such a number of spectral characteristics that Expression (2) with the number of wavelength bands L is satisfied need to be provided. Otherwise, it is difficult to achieve both the intake of a large amount of incident light and the facilitation of the separation of each wavelength band obtained from the pixels having mixed color information in the image estimation processing.

In addition, the first to Z-th filter segments132need to be irregularly disposed to facilitate the separation of each wavelength band obtained from the pixels having mixed color information in the image estimation processing.

FIGS. 5A to 5Hillustrate a relationship between a wavelength band width and the spectral characteristic of each filter segment131. InFIGS. 5A to 5H, a horizontal axis represents the wavelength λ, and a vertical axis represents the spectral transmittance T. As illustrated inFIGS. 5A to 5H, the amount of incident light in each wavelength band depends on the spectral transmittance of the filter segment131. A pixel that receives light of the wavelength band k is configured to have such an average transmittance at the wavelength band k that a condition below is satisfied so as to achieve a highly sensitive image capturing with an increased amount of incident light.

Here, λk-1represents the shortest wavelength in the wavelength band k, λkrepresent the longest wavelength therein, and T(λ) represents the spectral transmittance of the filter segment131corresponding to the wavelength band k.

When Expression (3) is satisfied for wavelength band1, the pixel receives light of wavelength band1. When Expression (3) is not satisfied, the pixel does not receive the light of wavelength band1. This also applies to neighboring wavelength bands. With the spectral transmittance of the filter segment illustrated inFIG. 5H, the pixel receives light of wavelength bands1and2.

FIG. 6three-dimensionally illustrates the spectral characteristic structure of the color filter130. The structure inFIG. 6has a distribution Fk(i,j) (where i=1, 2, . . . , M, and j=1, 2, . . . , N) of the transmittance characteristic of the color filter corresponding to the wavelength band k (k=1, 2, . . . , L) for the number of wavelength bands L. Fk(i,j) represents the average transmittance calculated on the left-hand side of Expression (3). With this structure, the color filter130satisfies a condition below.

Here, Rk1,k2represents a cross-correlation between distributions Fk1and Fk2of spectral transmittances in wavelength bands k1and k2, that is, represents an average value of the cross-correlation between the wavelength bands in Expressions (4) and (5). Rk1,k2is 0.5 when no cross-correlation is present between the wavelength bands k1and k2, and becomes closer to 1.0 for a stronger positive cross-correlation and to 0.0 for a stronger negative cross-correlation.

No cross-correlation is necessary between the wavelength bands in separating each wavelength band obtained from the pixels having mixed color information in the image estimation processing, and Expressions (4) and (5) specify an appropriate range for this. Expressions (4) and (5) also represent the irregular disposition of the first to Z-th filter segments132.

A condition (4′) below may be satisfied in place of Expression (4).

FIG. 7illustrates an exemplary spectral transmittance characteristic of the color filter130when the number of pixels is 64×64 and the number of wavelength bands is 4. InFIG. 7, a white part represents transmission of light, and a black part represents non-transmission of light. In this example, the transmittance distribution Fk(i,j) is generated with random numbers. Expression (4) has a value of 0.5011 in this case, indicating almost no cross-correlation between the wavelength bands.

FIGS. 8A to 8Dillustrate variations of the distribution of the transmittance characteristic of the color filter130when the number of pixels is 64×64 and the wavelength band is k.FIGS. 8A to 8Deach illustrate, on the right side, a sectional profile of the transmittance taken along a straight line on the left side diagram. The minimum value of Fk(i,j) may be zero as illustrated inFIG. 8A, or may be non-zero as illustrated inFIGS. 8B and 8C. In addition, Fk(i,j) may be a transmittance distribution of a plurality of gradations as illustrated inFIG. 8D. Although the transmittance distributions illustrated inFIGS. 8B to 8Dallow a larger amount of light to be taken in, this increase in the amount of light makes it difficult to separate each wavelength band obtained from the pixels having mixed color information in the image estimation processing. A ratio of min(Fk(i,j)) as the minimum value of Fk(i,j) and max(Fk(i,j)) as the maximum value thereof may satisfy a condition below.

FIGS. 9A to 9Cillustrate variations of the distribution of the transmittance characteristic of the filter segment131when the number of pixels is 64×64 and the wavelength band is k. When Pminrepresents the number of pixels having the minimum transmittance of Fk(i,j), and Pmaxrepresents the number of pixels having the maximum transmittance of Fk(i,j), the distribution inFIG. 9Ahas Pmax/Pmin=0.25, the distribution inFIG. 9Bhas Pmax/Pmin=1, and the distribution inFIG. 9Chas Pmax/Pmin=4. Although a larger ratio Pmax/Pminallows a larger amount of light to be taken in, this increase in the amount of light makes it difficult to separate each wavelength band obtained from the pixels having mixed color information in the image estimation processing. The ratio Pmax/Pminmay satisfy a condition below.

FIG. 10illustrates an exemplary configuration of the color filter130. The color filter130includes a stack of color filter layers133corresponding to respective wavelength bands.

FIGS. 11A to 11Eare schematic diagrams for describing a process of manufacturing the color filter layer133. First, the manufacturing process applies a transparent substrate134illustrated inFIG. 11Awith a photoresist135illustrated inFIG. 11B, and then forms a negative resist pattern illustrated inFIG. 11Cfor the color filter130through a patterning process and a liftoff process. The manufacturing process then forms, on this negative resist pattern, an optical element (notch filter, absorption material that absorbs light of a particular wavelength band, and plasmonic filter, for example)136that reflects or absorbs light of the wavelength band k as illustrated inFIG. 11D. Finally, as illustrated inFIG. 11E, the manufacturing method removes the negative resist pattern to produce the color filter layer133.

InFIG. 11E, a part where the optical element136is formed does not transmit light of the wavelength band k, and a part where the optical element136is not formed transmits light of all wavelength bands. The color filter130is obtained by stacking the color filter layers133corresponding to respective wavelength bands. An antireflection film may be provided to reduce a reflection loss at the color filter layer133. The process illustrated inFIGS. 10 and 11A to 11Eis merely an exemplary process of manufacturing the color filter130.

The use of the color filter130allows a highly sensitive compressed multiband image to be acquired by a single image capturing through a single-time imaging system.

In this embodiment, pixels for respective wavelength bands are irregularly disposed as illustrated inFIG. 7so as to take in a large amount of incident light and to facilitate the separation of each wavelength band obtained from the pixels having mixed color information in the image estimation processing. Thus, color moire caused when pixels are regularly disposed like a Bayer array is less likely to be produced in principle. This allows the image-pickup unit110to maintain its performance without an optical low-pass filter in some cases.

Next follows, with reference toFIGS. 12A to 12CandFIG. 13, a description of the image estimation processing that estimates a multiband image f of M×N pixels×L different wavelength bands based on a captured image g as a compressed multiband image of M×N pixels.

FIG. 12Ais a block diagram of a multiband image acquisition system100according to Embodiment 1 of the present invention.FIG. 12Bis an external perspective diagram of an example of the multiband image acquisition system100.FIG. 12Cillustrates an example of an image acquirer101.

The multiband image acquisition system100includes the image acquirer101, an A/D converter102, an image processing unit103, a storage unit104, a recording medium105, a display unit106, a system controller107, a drive controller108, and a state detector109. The multiband image acquisition system100may be a lens-integrated image-pickup apparatus illustrated inFIG. 12B, or an image-pickup system including an interchangeable lens and an image-pickup apparatus body.

The image acquirer101includes, as illustrated inFIG. 12C, a nonillustrated single-time imaging optical system (image-pickup optical system)120that forms an object image, and the image-pickup unit110illustrated inFIG. 1that photoelectrically converts the object image. The A/D converter102converts an analog signal from the image sensor111into a digital signal. The image processing unit (image processor)103provides the digital signal with various pieces of image processing and performs the image estimation processing. The storage unit (storage)104includes various memories such as ROM, RAM, and the like, and stores information of the spectral transmittance of the filter segment131, various pieces of control information, variables, temporally stored values, programs, and image signals processed by the image processing unit103. The recording medium105includes a non-transitory computer-readable medium such as DVD-ROM. The display unit106includes, for example, a liquid crystal display, and displays an image to be captured, captured images stored in the storage unit104and the recording medium105, and various pieces of control information. The system controller107is a controller that controls an operation of each component and includes a micro computer. The drive controller108controls drive of a focus lens, a zoom lens, an aperture stop, an image stabilization lens, and the like of the imaging optical system. The state detector109is a detection unit that detects various states including an image capturing condition of the multiband image acquisition system100.

FIG. 13is a flowchart of a process of the system controller107to obtain a multiband image from a captured image, in which “s” represents a step. The flowchart illustrated inFIG. 13may be realized as a program that causes a computer to execute each step.

First, at S101, a captured image g digitalized through the image acquirer101and the A/D converter102is acquired. Next, at S102, the image processing unit103provides an initial value of an estimated image f. Next, at S103, the image processing unit103calculates an evaluation function. Next, at S104, the image processing unit103determines whether the evaluation function calculated at S103is smaller or larger than a threshold. When the evaluation function is smaller than the threshold, the image processing unit103determines that the estimated image f is appropriate, and outputs the estimated image f before ending the process. When the evaluation function is larger than the threshold, the image processing unit103determines that the estimation of the estimated image f is insufficient, and the process proceeds to S105, where the image processing unit103counts the number of iterations of the estimation processing and ends the processing when a set number of times is reached. When the number of iterations has not reached the set number of times, the process proceeds to S106, where the image processing unit103updates the estimated image f. Then, the process repeats S103to S106until the processing ends eventually. The estimated image f obtained at the end is recorded on the recording medium105, or displayed on the display unit106.

The image estimation processing in Embodiment 1 employs the evaluation function using the transmittance characteristic Fk(i,j) of the color filter130. The transmittance characteristic Fk(i,j) is known, and its information is stored in the storage unit104. When Fk(i,j) represents the estimated image of the wavelength band k, an image g′ captured through the color filter130is expressed by Expression (8) below.

Here, η is a noise term. Then, whether the estimated image f is appropriate can be evaluated by Expression (9) below.

In this manner, the appropriateness of the estimated image Fk(i,j) can be checked by using the transmittance characteristic Fk(i,j) of the color filter130and the captured image g. The evaluation function is not limited to an L2-norm such as Expression (9), and may be any function using the transmittance characteristic Fk(i,j) of the color filter130, and may be, for example, an L1-norm such as Expression (9′) below.

However, the estimation of Fk(i,j) that minimizes Expressions (9) and (9′) is insufficient. This is because the estimation with Expressions (9) and (9′) is invalid in a part where the transmittance characteristic Fk(i,j) is zero. This is likely to cause a pixel defect (lack of image information) in a part where Fk(i,j)=0 in the image Fk(i,j) estimated with Expressions (9) and (9′).

Thus, a process to interpolate a pixel defect part may be provided at the same time. Specifically, Expression (10) below is employed as the evaluation function.

Here, φ(f) is called a regularization term, which is used to interpolate the pixel defect part. γ is a parameter for adjusting the effect of the regularization term, and needs to be adjusted in accordance with the kind of the captured image g as appropriate.

The regularization term φ(f) is, for example, a total variation (TV) norm regularization term in Expression (11).

However, the regularization term φ(f) is not limited thereto.

The series of image estimation processes illustrated inFIG. 13may be performed by using a fast image estimation processing algorithm such as TwIST algorithm disclosed by J. M. Bioucas-Dias and M. A. T. Figueiredo, “A new TwIST: two-step iterative shrinkage/thresholding algorithms for image restoration, IEEE Trans.on Image Processing, vol. 16, Dec. 2007.

FIGS. 14A to 14Dillustrate a captured image acquired by the multiband image acquisition system100and simulation results of multiband images produced from the captured image. The images used in the simulation are of 256×256 pixels and 31 wavelength bands, and the TwIST algorithm was used in the image estimation processing.

FIG. 14Aillustrates parts (k=1, 16, 31) of the transmittance characteristic Fk(i,j) of the filter segment131.FIG. 14Billustrates the captured image g.FIG. 14Cillustrates an RGB image produced from a multiband image of a correct image.FIG. 14Dillustrates an RGB image produced from a multiband image of the estimated image Fk(i,j). As described above, a display image may be produced from a multiband image.

FIG. 15Acompares spectral transmittances at corresponding points A inFIGS. 14C and 14D, andFIG. 15Bcompares spectral transmittances at corresponding points B inFIGS. 14C and 14D. InFIGS. 15A and 15B, a horizontal axis represents the wavelength (nm), and a vertical axis represents the spectral transmittance. InFIGS. 15A and 15B, a dotted line represents a correct (original) spectral transmittance, and a solid line represents a spectral transmittance calculated by the image estimating processing in Embodiment 1. The comparison indicates that the image estimating processing in this embodiment accurately estimates the spectral transmittance.

Embodiment 1 allows, for example, the small compact digital camera illustrated inFIG. 12Bto acquire a highly sensitive multiband image.

In Embodiment 1, the image processing unit of the image-pickup apparatus performs processing for producing a multiband image, but a personal computer (PC) or a dedicated apparatus may perform sharpening as an image processing apparatus.FIG. 16Ais a block diagram of a multiband image acquisition system according to Embodiment 2.FIG. 16Bis a system configuration diagram thereof. Embodiment 2 differs from Embodiment 1 in that an image-pickup apparatus201and an image processing apparatus203are separately provided in Embodiment 2.

The image-pickup apparatus201such as a single-lens reflex camera is used to acquire the captured image g. Then, the captured image g is recorded on a non-transitory computer-readable recording medium202such as an SD card202-1and a hard disk202-2. The captured image g thus recorded is stored on a storage unit205in the image processing apparatus203through a USB cable and various networks (LAN and the Internet), and a communication unit204. The image processing apparatus203may be a PC203-1and cloud computing203-2. Then, an image processing unit206provides the image estimation processing and calculates the estimated image f. The estimated image f thus calculated is stored on the storage unit205, displayed on a display apparatus207such as a liquid crystal display, or output to an output apparatus208such as a printer.

In Embodiment 2, the image estimation processing, which is heavily-loaded, is performed by using the PC203-1and the cloud computing203-2, which allows a user to easily obtain a highly sensitive multiband image.

Other Embodiments

This application claims the benefit of Japanese Patent Application No. 2014-235305, filed on Nov. 20, 2014, which is hereby incorporated by reference herein in its entirety.