Image processing apparatus, program, method and image capturing system

An image processing apparatus with extended dynamic range includes a first gradation-conversion-characteristics calculating unit that calculates first gradation conversion characteristics for each of a plurality of areas of input image signals; and a gradation conversion unit that performs gradation conversion on each of the areas of the image signals by using the first gradation conversion characteristics. A filter processing and reduction processing unit generates a plurality of band signals having mutually different frequency bands by multi-resolution conversion processing from the image signals subjected to the gradation conversion. A second gradation-conversion-characteristics calculating unit calculates second gradation conversion characteristics associated with the individual band signals from the first gradation conversion characteristics An NR processing unit performs processing for noise reduction on the individual band signals by using the second gradation conversion characteristics; and an adding unit combines the band signals subjected to the processing for noise reduction.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus, a program, a method, and an image capturing system that perform processing for noise reduction on input image signals.

This application is based on Japanese Patent Application No. 2009-022886, the content of which is incorporated herein by reference.

2. Description of Related Art

There has hitherto been known what is called space-variant gradation conversion processing, in which in order to increase the dynamic range of an image scene, gradation conversion processing is performed while varying the gradation conversion characteristics for each area of the image scene (e.g., see the Publication of Japanese Patent No. 3465226).

By performing the space-variant gradation conversion processing described above, the effect of brightening dark areas of the image scene is achieved.

BRIEF SUMMARY OF THE INVENTION

The present invention provides an image processing apparatus, a program, a method, and an image capturing system with which it is possible to extend the dynamic range by performing gradation conversion processing and to effectively reduce noise in image signals subjected to the gradation conversion processing.

A first aspect of the present invention is an image processing apparatus including a first gradation-conversion-characteristics calculating unit that calculates gradation conversion characteristics for each of a plurality of areas of input image signals; a gradation conversion unit that performs gradation conversion on each of the areas of the image signals by using the first gradation conversion characteristics calculated by the first gradation-conversion-characteristics calculating unit; a band-signal generating unit that generates a plurality of band signals having mutually different frequency bands by multi-resolution conversion processing from the image signals subjected to the gradation conversion by the gradation conversion unit; a second gradation-conversion-characteristics calculating unit that calculates second gradation conversion characteristics associated with the individual band signals generated by the band-signal generating unit from the first gradation conversion characteristics calculated by the first gradation-conversion-characteristics calculating unit; a noise reducing unit that performs processing for noise reduction on the individual band signals by using the second gradation conversion characteristics calculated by the second gradation-conversion-characteristics calculating unit; and an image combining unit that combines the band signals subjected to the processing for noise reduction by the noise reducing unit.

“Area” includes an area containing only a single pixel as well as an area containing a plurality of pixels.

According to the above aspect, upon input of image signals, the first gradation-conversion-characteristics calculating unit calculates first gradation conversion characteristics suitable for each area of the image signals, and the gradation conversion unit performs gradation conversion processing on each area. The image signals subjected to the gradation conversion processing are divided into a plurality of band signals by the band-signal generating unit, and the second gradation-conversion-characteristics calculating unit calculates second gradation conversion characteristics associated with the individual band signals. The noise reducing unit performs processing for noise reduction on the individual band signals by using the second gradation conversion characteristics, and the plurality of band signals subjected to the processing for noise reduction are combined by the image combining unit.

Accordingly, since it is possible to perform processing for noise reduction on individual band signals on the basis of the second gradation conversion characteristics calculated for the individual band signals, it is possible to effectively reduce noise in the image signals subjected to the gradation conversion processing.

A second aspect of the present invention is an image capturing system including an image capturing unit that captures an image of an object; and the above-described image processing apparatus that processes image signals acquired by the image capturing unit.

A third aspect of the present invention is a recording medium having recorded thereon an image processing program for causing a computer to execute first gradation-conversion-characteristics calculation processing for calculating first gradation conversion characteristics for each of a plurality of areas of input image signals; gradation conversion processing for performing gradation conversion on each of the areas of the image signals by using the first gradation conversion characteristics calculated in the first gradation-conversion-characteristics calculation processing; band-signal generation processing for generating a plurality of band signals having mutually different frequency bands by multi-resolution conversion processing from the image signals subjected to the gradation conversion in the gradation conversion processing; second gradation-conversion-characteristics calculation processing for calculating second gradation conversion characteristics associated with the individual band signals generated in the band-signal generation processing from the first gradation conversion characteristics calculated in the first gradation-conversion-characteristics calculation processing; noise reduction processing for performing processing for noise reduction on the individual band signals by using the second gradation conversion characteristics calculated in the second gradation-conversion-characteristics calculation processing; and image combination processing for combining the plurality of band signals subjected to the processing for noise reduction in the noise reduction processing.

A fourth aspect of the present invention is an image processing method comprising a first gradation-conversion-characteristics calculating step of calculating first gradation conversion characteristics for each of a plurality of areas of input image signals; a gradation conversion step of performing gradation conversion on each of the areas of the image signals by using the first gradation conversion characteristics calculated in the first gradation-conversion-characteristics calculating step; a band-signal generating step of generating a plurality of band signals having mutually different frequency bands by multi-resolution conversion processing from the image signals subjected to the gradation conversion in the gradation conversion step; a second gradation-conversion-characteristics calculating step of calculating second gradation conversion characteristics associated with the individual band signals generated in the band-signal generating step from the first gradation conversion characteristics calculated in the first gradation-conversion-characteristics calculating step; a noise reducing step of performing processing for noise reduction on the individual band signals by using the second gradation conversion characteristics calculated in the second gradation-conversion-characteristics calculating step; and an image combining step of combining the plurality of band signals subjected to the processing for noise reduction in the noise reducing step.

DETAILED DESCRIPTION OF THE INVENTION

Hereinafter, an image processing apparatus and an image capturing system according to a first embodiment of the present invention will be described with reference to the drawings. Here, description will be given in the context of an example where an image processing apparatus according to the present invention is applied to an image capturing system.

FIG. 1is a block diagram schematically showing the configuration of an image capturing system1including an image processing apparatus according to this embodiment. As shown inFIG. 1, the image capturing system1includes an optical block11, such as a lens, an image capturing block (image capturing unit)13, such as a CCD, that acquires an image of an object formed by the optical block11in the form of image signals, an image processing unit15that processes the image signals acquired by the image capturing block13, a first gradation-conversion-characteristics calculating unit16that calculates space-variant first gradation conversion characteristics from the image signals processed by the image processing unit15, a gradation conversion unit17that performs space-variant gradation conversion processing on the image signals processed by the image processing unit15, by using the first gradation conversion characteristics calculated by the first gradation-conversion-characteristics calculating unit16, a second gradation-conversion-characteristics calculating unit19that calculates second gradation conversion characteristics from the first gradation conversion characteristics calculated by the gradation-conversion-characteristics calculating unit16, a noise reducing unit21that performs processing for noise reduction on the image signals subjected to the gradation conversion processing by the gradation conversion unit17, by using the second gradation conversion characteristics calculated by the second gradation-conversion-characteristics calculating unit19, an image compressing unit23that compresses the image signals subjected to the processing for noise reduction by the noise reducing unit21, and a recording medium25for recording thereon the image signals compressed by the image compressing unit23.

The operation of the image capturing system1configured as described above will be described below.

An image of an object formed on the surface of the image capturing block13by the optical block11is converted into electrical signals by the image capturing block13, and CDS (Correlated Double Sampling)/differential sampling, adjustment of the analog gain, etc. are performed. Then, the result is converted into digital signals by an A/D conversion unit (not shown), and the image processing unit15performs various types of processing for improving the image quality, such as γ correction, color separation, and chrominance matrix correction. The image signals processed by the image processing unit15are output to the first gradation-conversion-characteristics calculating unit16and the gradation conversion unit17.

In the first gradation-conversion-characteristics calculating unit16, first gradation conversion characteristics are calculated by using the image signals processed by the image processing unit15, and the first gradation conversion characteristics are output to the gradation conversion unit17and the second gradation-conversion-characteristics calculating unit19. Here, the first gradation conversion characteristics are obtained by dividing the pixel values of the image signals after the gradation conversion processing by the pixel values of the image signals before the gradation conversion processing, and represent the amplification factors of the individual pixel values.

In the gradation conversion unit17, space-variant gradation conversion processing is performed on the image signals processed by the image processing unit15, by using the first gradation conversion characteristics calculated by the first gradation-conversion-characteristics calculating unit16. The image signals subjected to the gradation conversion processing by the gradation conversion unit17are output to the noise reducing unit21.

In the noise reducing unit21, processing for noise reduction is performed on the image signals subjected to the gradation conversion processing by the gradation conversion unit17. The processing for noise reduction will be described later in detail. The image signals subjected to the processing for noise reduction by the noise reducing unit.21are output to the image compressing unit23, compressed by the image compressing unit23, and recorded on the recording medium25.

Hereinafter, the processing for noise reduction will be described in detail by usingFIG. 2.

FIG. 2is a block diagram showing detailed functions of the second gradation-conversion-characteristics calculating unit19and the noise reducing unit21. As shown inFIG. 2, the second gradation-conversion-characteristics calculating unit19includes N−1 (N is an integer greater than or equal to 1) stages of filter processing and reduction processing units101and111. This embodiment will be described in the context of an example where N=3.

In the second gradation-conversion-characteristics calculating unit19, the first gradation conversion characteristics calculated by the first gradation-conversion-characteristics calculating unit16are subjected to low-pass filter processing and then reduced in the filter processing and reduction processing unit101at the first stage. Through this processing, the size of the first gradation conversion characteristics is reduced to, for example, half both in the horizontal direction and the vertical direction. The reduced first gradation conversion characteristics are supplied to the filter processing and reduction processing unit111at the second stage and further reduced. That the filter processing and reduction processing units101and111reduce the gradation conversion characteristics supplied from the filter processing and reduction processing units at the previous stages (the first gradation-conversion-characteristics calculating unit16in the case of the filter processing and reduction processing unit101) to the same sizes as the associated band signals.

Now, the processing for reducing the gradation conversion characteristics will be described.

In multi-resolution conversion processing, images of different resolutions (sizes) are generated for individual bands. Since the gradation conversion characteristics are amplification factors of the pixel values of individual pixels, the gradation conversion characteristics have the same size as an image. In order to apply the gradation conversion characteristics to an image of a certain band, the gradation conversion characteristics have to be converted into the same size as the image of that band. As the size conversion method, an arbitrary resizing method, such as the nearest neighbor method, is used. With the image and the gradation conversion characteristics of each band having the same size, it is possible to perform processing on each corresponding pair of pixels.

The noise reducing unit21is configured to perform multi-resolution conversion by a Laplacian pyramid, and it includes N stages of filter processing and reduction processing units201,211, and221, and also includes enlargement processing units202,212, and222, subtracting units (band-signal generating units)203,213, and223, NR processing units (noise reducing units)204,214, and224, enlargement processing units205,215, and225, and adding units (image combining units)206,216, and226, provided respectively in association with the filter processing and reduction processing units201,211, and221. As described earlier, N is an integer greater than or equal to 1, and description will be given here in the context of an example where N=3.

First, processing for dividing input image signals200into a plurality of band signals and reducing noise in the individual band signals will be described.

The filter processing and reduction processing unit201at the first stage performs low-pass filter processing on the input image signals200and then reduces the result to a predetermined size. Thus, the image size of the input image signals200is reduced to, for example, half both in the horizontal direction and the vertical direction. The reduced image generated by the filter processing and reduction processing unit201is supplied to the enlargement processing unit202and the filter processing and reduction processing unit211at the subsequent stage.

The enlargement processing unit202enlarges the reduced image supplied from the filter processing and reduction before the reduction processing by the filter processing and reduction processing unit201. That is, here, the reduced image is enlarged to the size of the input image signals200, and the result is output to the subtracting unit203. The subtracting unit203calculates difference signals between the image signals input from the enlargement processing unit202and the image signals input to the filter processing and reduction processing unit201, i.e., the input image signals200, thereby generating band signals, and outputs the band signals to the NR processing unit204. Furthermore, the NR processing unit204receives the associated second gradation conversion characteristics, i.e., the gradation conversion characteristics input to the filter processing and reduction processing unit101at the first stage, as second gradation conversion characteristics from the second gradation-conversion-characteristics calculating unit19. The NR processing unit204performs predetermined noise reduction processing on the input band signals by using the second gradation conversion characteristics and outputs the result to the adding unit206.

Here, as a specific method of noise reduction processing, for example, coring processing is conceivable. Coring processing is processing in which input signals with absolute values less than or equal to a threshold are all changed to zero, whereas input signals with absolute values greater than the threshold are changed to signals obtained by decreasing the absolute values by the threshold. That the NR processing unit204multiplies space-variant second gradation conversion characteristics by a preset threshold and performs the coring processing described above, thereby performing processing for noise reduction on the band signals input from the subtracting unit203.

At the second stage and the third stage, processing similar to the processing in the first stage is performed on image signals output from the filter processing and reduction processing unit at the previous stage. That is, at the second stage and the third stage, band signals are generated from image signals output from the filter processing and reduction processing units201and211, respectively, and processing for noise reduction is performed on the generated band signals by using the associated second gradation conversion characteristics. The band signals subjected to the processing for noise reduction as described above are output to the adding units216and226.

Next, processing for combining the band signals subjected to the processing for noise reduction and outputting the result as output image signals250will be described.

The enlargement processing unit225at the third stage performs enlargement processing on the reduced image from the filter processing and reduction processing unit221and outputs the result to the adding unit226. The adding unit226adds together the noise-reduced band signals supplied from the NR processing unit224and the reduced image supplied from the enlargement processing unit225and outputs the result to the enlargement processing unit215.

At the second stage and the first stage, processing similar to the processing described above is performed, so that the adding units216and206add together the noise-reduced band signals supplied from the NR processing units and the reduced images supplied from the enlargement processing units at the subsequent stages and output the results to the enlargement processing units at the previous stages (the image compressing unit23in the case of the adding unit206). In this way, noise-reduced band signals are recombined sequentially from the low-frequency side, and noise-reduced output image signals250finally resized back to the predetermined size are output from the adding unit206.

As described above, with the image capturing system1according to this embodiment, regarding a plurality of areas in image signals acquired by the image capturing block13, the first gradation-conversion-characteristics calculating unit16calculates first gradation conversion characteristics suitable for each of the areas, and the gradation conversion unit17performs gradation conversion processing on each of the areas. The image signals subjected to the gradation conversion processing are divided into a plurality of band signals by the filter processing and reduction processing units201,211, and221, and the second gradation-conversion-characteristics calculating unit19calculates second gradation conversion characteristics associated with the individual band signals. The NR processing units204,214, and224perform processing for noise reduction on the individual band signals by using the second gradation conversion characteristics, and the plurality of band signals subjected to the processing for noise reduction are combined by the adding units206,216, and226.

Accordingly, since it is possible to perform processing for noise reduction on individual band signals on the basis of second gradation conversion characteristics calculated for the individual band signals, it is possible to effectively reduce noise in image signals subjected to gradation conversion processing.

Next, a second embodiment of the present invention will be described by usingFIG. 3.

An image capturing system2according to this embodiment differs from the first embodiment in that image signals are separated into luminance signals and chrominance signals and processing for noise reduction is performed on the individual signals. Hereinafter, regarding the image capturing system2according to this embodiment, a description of commonalities with the first embodiment will be omitted, and differences will be mainly described.

As shown inFIG. 3, instead of the noise reducing unit21shown inFIG. 1, the image capturing system2includes an image-signal separating unit (not shown) that separates input image signals into luminance signals and chrominance signals, a luminance-signal noise reducing unit31that performs processing for noise reduction on the luminance signals, and a chrominance-signal noise reducing unit41that performs processing for noise reduction on the chrominance signals.

The luminance-signal noise reducing unit31is configured to perform multi-resolution conversion by a Laplacian pyramid, and it includes N stages of filter processing and reduction processing units301,311, and321, and also includes enlargement processing units302,312, and322, subtracting units303,313, and323, correcting units308,318, and328, NR processing units304,314, and324, enlargement processing units305,315, and325, and adding units306,316, and326, provided respectively in association with the individual filter processing and reduction processing units301,311, and321.

The chrominance-signal noise reducing unit41is configured to perform multi-resolution conversion by a Laplacian pyramid, and it includes N stages of filter processing and reduction processing units401,411, and421, and also includes enlargement processing units402,412, and422, subtracting units403,413, and423, NR processing units404,414, and424, enlargement processing units405,415, and425, and adding units406,416, and426, provided respectively in association with the individual filter processing and reduction processing units401,411, and421.

N is an integer greater than or equal to 1, and description will be given here in the context of an example where N=3.

The correcting units308,318, and328correct second gradation conversion characteristics for input luminance signals300and input chrominance signals400individually as second gradation conversion characteristics for luminance signals and for chrominance signals and output the corrected second gradation conversion characteristics to the NR processing units304,314, and324of the luminance-signal noise reducing unit31and the NR processing units404,414, and424of the chrominance-signal noise reducing unit41. In other components of the luminance-signal noise reducing unit31and the chrominance-signal noise reducing unit41, processing similar to the processing in the first embodiment described earlier is performed on the luminance signals or the chrominance signals.

The operation of the thus-configured image capturing system2will be described below.

Image signals acquired by the image capturing block13are separated into luminance signals and chrominance signals by the image-signal separating unit. The luminance signals and chrominance signals are individually subjected to filter processing and reduction processing, enlargement processing, and subtraction processing in individual bands, similarly to the first embodiment described earlier.

Space-variant second gradation conversion characteristics are corrected in individual bands by the correcting units308,318, and328into gradation conversion characteristics for luminance signals and gradation conversion characteristics for chrominance signals. As a specific correcting method, for example, correction is performed by using equation (1) below, where G denotes gradation conversion characteristics at an arbitrary pixel before correction, G′ denotes corrected gradation conversion characteristics, and α (0≦α) denotes a correcting parameter.
G′=α*G+1−α  (1)

Here, when the corrected gradation conversion characteristic G′ is a value greater than 1, noise reduction processing is performed aggressively. By making it possible to adjust α, thereby setting different values as gradation conversion characteristics for luminance signals and chrominance signals, setting is possible such that a large value is chosen as gradation conversion characteristics for luminance signals in order to remove luminance noise aggressively, whereas a small value is chosen as gradation conversion characteristics for chrominance signals in order to maintain chroma in noise-reduced image signals.

Furthermore, when G is a value greater than 1, the value of G′ approaches 1 as a approaches 0, and the degree of noise reduction is weakened.

The thus-corrected gradation conversion characteristics for luminance signals and chrominance signals are input to the NR processing units304,314, and324of the luminance-signal noise reducing unit31and the NR processing units404,414, and424of the chrominance-signal noise reducing unit41, and the NR processing units perform processing similar to the processing in the first embodiment described earlier on the luminance signals and chrominance signals Then, the luminance signals and chrominance signals in individual bands are subjected to enlargement processing and adding processing, and the results are output as noise-reduced output luminance signals350and output chrominance signals450resized back to the predetermined size.

As described above, with the image capturing system2according to this embodiment, it is possible to separate input image signals into luminance signals and chrominance signals by an image-signal separating unit, correct space-variant second gradation conversion characteristics individually for the luminance signals and the chrominance signals, and perform processing for noise reduction on the luminance signals and chrominance signals in individual hands by using the corrected gradation conversion characteristics. Accordingly, it is possible to perform noise reduction processing adaptively on luminance noise and chrominance noise included in luminance signals and chrominance signals.

Although it has been described that the correcting units308,318, and328correct second gradation conversion characteristics for luminance signals and for chrominance signals in this embodiment, it is possible to correct second gradation conversion characteristics for either luminance signals or chrominance signals alone.

Furthermore, although it has been described that image signals subjected to gradation conversion by the gradation conversion unit17are separated into luminance signals and chrominance signals and a plurality of band signals are generated individually for the luminance signals and the chrominance signals in this embodiment, alternatively, it is possible to generate a plurality of band signals subjected to gradation conversion and separate the band signals into luminance signals and chrominance signals.

Although the embodiments of the present invention have been described above in detail with reference to the drawings, specific configurations are not limited to the embodiments, and design modifications or the like not departing from the spirit of the present invention are encompassed.

For example, the embodiments described above are examples of applying image processing apparatuses according to the present invention to image capturing systems. Specific image capturing systems may be products such as floor-type cameras for broadcasting, ENG cameras, consumer handy cameras, or digital cameras, image-signal correcting programs (CG programs) that deal with moving pictures, or image editing apparatuses.

Furthermore, although processing by hardware is assumed with an image processing apparatus in the embodiments described above, limitation to such a configuration is not necessary. For example, a configuration in which processing is performed by separate software is possible. In this case, an image processing apparatus has a CPU, a main storage device such as a RAM, and a computer-readable recording medium having recorded thereon a program for implementing the entirety or part of the processing described above. The CPU reads out the program recorded on the storage medium and executes information manipulation and calculation processing, whereby processing similar to the processing by the image processing apparatuses described above is implemented. Here, the computer-readable recording medium refers to a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, or the like.

Furthermore, although coring processing has been described as an example of the method of processing for noise reduction, the method is not limited as long as it is possible to vary the degree of noise reduction processing on the basis of space-variant second gradation conversion characteristics.

Furthermore, although it has been described that first gradation conversion characteristics are reduced after low-pass filter processing by the filter processing and reduction processing units of the first gradation-conversion-characteristics calculating unit16, it is possible to perform only reduction processing.

Furthermore, although it has been described that the second gradation-conversion-characteristics calculating unit19calculates second gradation conversion characteristics associated with the individual band signals, the second gradation-conversion-characteristics calculating unit19may calculate second gradation conversion characteristics associated with some of the band signals.

Furthermore, although it has been described that NR processing units perform processing for noise reduction on Individual hand signals by using the second gradation conversion characteristics, processing for noise reduction may be performed only on some of the band signals by using the second gradation conversion characteristics.