Image processing unit, image processing method, and image processing program

The present invention provides an image processing unit capable of differentiating two images with a difference which the user expects to see, an imaging processing method thereof, and an image processing program thereof. The image processing unit of the present invention includes: a histogram creation unit for creating a color histogram of each image obtained by an imaging sensor; a correlation calculation unit for comparing the color histogram of a processing object image to be subjected to predetermined processing with the color histogram of a comparison object image obtained prior to the processing object image, to calculate the color correlation between these images; and a decision unit for deciding a parameter to be used for the predetermined processing according to the calculated correlation.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing unit to be mounted on or connected to a device such as an electronic camera. Further, the present invention relates to an image processing method and an image processing program which are applied to the image processing unit.

2. Description of the Related Art

An image processing circuit inside the conventional electronic camera automatically conducts white balance processing, color transformation processing, gray-scale transformation processing, and so on, on individual images.

Parameters, which are used for these processings, are decided for each of the processings according to color temperature, intensity, and the like of the individual images.

Therefore, if there exists even a slight difference between two images, even though they are obtained by photographing the same scene, different parameters are set for the above processings. This results in the individual images' having different finishes.

In general, however, when a user obtains two images by photographing the same scene twice consecutively, probably with slight framing changes, the two images will mostly have a little difference. In such a case, the user often hopes to have the two images with the same photographic atmosphere (particularly, color balance).

The aforesaid conventional electronic camera, however, may recognize a large difference between the two images, event though the user can see no difference therebetween, and as a result, these two images may have quite different photographic atmospheres.

SUMMARY OF THE INVENTION

Hence, it is an object of the present invention to provide an image processing unit, an image processing method, and an image processing program which realize creation of two images with a difference which the user expects to see.

An image processing unit of the present invention comprises a histogram creation unit for creating a color histogram of each image obtained by an imaging sensor; a correlation calculation unit for calculate color correlation between a processing object image and a comparison object image by comparing the color histograms of the two object images, the processing object image being subjected to a predetermined processing and the comparison object image being obtained prior to the processing object image; and a decision unit for deciding a parameter to be used for the predetermined processing according to the calculated correlation. In general, the human eyes easily recognize differences in colors of images when comparing a plurality of images. Deciding the parameter according to the color histogram makes it possible to differentiate the plurality of images with a difference which the user expects to see.

It is preferable that the histogram creation unit evaluates, when creating the color histogram, a color at each position on the image according to a hue area to which this color belongs. By way of this evaluation a plurality of images can be differentiated with a difference which the user expects to see.

The histogram creation unit preferably sets respective dimensions of the hue areas unequally. This realizes such image processing that a subtle difference in colors which appears a large difference to the human eyes is discriminated, on the other hand, a large difference in colors which appears a small difference to the human eyes is judged as equivalent.

The histogram creation unit preferably evaluates, when creating the color histogram, a color at each position on the image according to a hue area and a chroma area to which this color belongs.

The histogram creation unit preferably sets respective dimensions of the hue areas and/or chroma areas unequally. This realizes such image processing that a subtle difference in colors which appears a large difference to the human eyes is discriminated, on the other hand, a large difference in colors which appears a small difference to the human eyes is judged as equivalent.

The histogram creation unit preferably evaluates, when creating the color histogram, a frequency at which each color appears on the image. The evaluation is made for every area of the image consisting of a plurality of pixels. This shortens the time required for creation of the color histogram.

It is preferable that the image processing unit of the invention further comprises a tentative decision unit for tentatively deciding the parameter suitable for each of the images obtained by the imaging sensor, and the decision unit decide, according to the correlation, the parameter to be used by applying weighted average to the tentatively decided parameter for the processing object image and the tentatively decided parameter for the comparison object image. This makes it possible to reflect the colors of the processing object image and of the comparison object image, and the correlation between the two images in the parameter.

The correlation calculation unit preferably calculates, as an index of the correlation, a Euclidean distance between a vector of the frequency at which each color appears in the color histogram of the processing object image and a vector of the frequency at which each color appears in the color histogram of the comparison object image. This realizes reliable calculation of the correlation.

Moreover, the histogram creation unit preferably excludes an area of the image from objects of color evaluation for creation of the color histogram when the area has a color being outside a predetermined color area. This can prevent a distinctive color from having influence on the parameter even in a case only one of the processing object image and the comparison object image includes the distinctive color.

An image processing method of the present invention comprises: a histogram creation step of creating a color histogram of each image obtained by an imaging sensor; a correlation calculation step of calculating color correlation between a processing object image and a comparison object image by comparing the color histograms of the two object images, the processing object image being subjected to a predetermined processing and the comparison object image being obtained prior to the processing object image for comparison; and a decision step of deciding a parameter to be used for the predetermined processing according to the calculated correlation. The human eyes are generally sensitive to differences in colors of images when comparing a plurality of images. This way of deciding the parameter based on the color histogram makes it possible to differentiate the plurality of images with a difference which the user expects to see.

Preferably, in the histogram creation step, a color at each position on the image is evaluated for the creation of the color histogram, according to a hue area to which this color belongs.

In the histogram creation step, respective dimensions of the hue areas are preferably set unequally. This realizes such image processing that a subtle difference in colors which appear a large difference to the human eyes is discriminated, on the other hand, a large difference in colors which appears a small difference to the human eyes is judged as equivalent.

In the histogram creation step, when the color histogram is created, a color at each position on the image is preferably evaluated based on a hue area and a chroma area to which this color belongs.

In the histogram creation step, respective dimensions of the hue areas and/or of chroma areas are preferably set unequally. This realizes such image processing that a subtle difference in colors which appears a large difference to the human eyes is discriminated, on the other hand, a large difference in colors which appears a small difference to the human eyes is judged as equivalent.

In the histogram creation step, when the color histogram is created, a frequency at which each color appears on the image is preferably evaluated for each area consisting of a plurality of pixels. This shortens the time required for creating the color histogram.

It is preferable that the image processing method further comprises a tentative decision step of tentatively deciding the parameter suitable for each of the images obtained by the imaging sensor. In the decision step, the parameter to be used is decided according to the correlation by applying weighted average to the tentatively decided parameter for the processing object image and the tentatively decided parameter for the comparison object image. This makes it possible to properly reflect the colors of the processing object image and of the comparison object image, and the correlation between the two images in the parameter.

Preferably in the correlation calculation step, as an index of the correlation calculated is a Euclidean distance between a vector of the frequency at which each color appears in the color histogram of the processing object image and a vector of the frequency at which each color appears in the color histogram of the comparison object image. This realizes reliable calculation of the correlation.

In the histogram creation step, an area on the image is preferably excluded from objects of color evaluation for creation of the color histogram when the area has a color outside a predetermined color area. This can prevent a distinctive color from affecting the parameter even in a case only one of the processing object image and the comparison object image includes the distinctive color.

An image processing program of the present invention causes an image processing unit to execute the steps of: creating a color histogram of each image obtained by an imaging sensor; calculating color correlation between a processing object image and a comparison object image by comparing the color histograms of the two images, the processing object image being subjected to a predetermined processing and the comparison object image being obtained prior to the processing object image for comparison; deciding a parameter to be used for the predetermined processing according to the calculated correlation. This image processing program is capable of causing the image processing unit to execute processing by the image processing method of the present invention.

As described hitherto, the present invention realizes an image processing unit which can differentiate a plurality of images with a difference which the user expects to see, and an image processing method and an image processing program thereof.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, an embodiment of the present invention will be explained with reference to the drawings.

FIG. 1is a view showing a schematic configuration of an electronic camera of this embodiment.

A camera body1is provided with an imaging sensor11, an A/D converter12, a white balance processing circuit14, a color interpolation circuit15, a gray-scale transformation circuit16, a setting circuit13, and so on.

An image of a subject is formed through a lens2on the imaging sensor11. An output signal of the imaging sensor11is inputted to the A/D converter12via a not-shown analog gain controller to be converted to a digital signal. The digital signal undergoes white balance processing in the white balance processing circuit14, a color interpolation processing in the color interpolation circuit15, and a gray-scale transformation processing in the gray-scale transformation circuit16, respectively.

Here, the setting circuit13sets for each image a gain (hereinafter, defined as Gr, Gb) by which the white balance processing circuit14multiplies the digital signals (hereinafter, defined as R/G, B/G). This setting circuit13corresponds to what characterizes the present invention.

Note that the reference symbol13ainFIG. 1denotes an operating circuit in the setting circuit13, the reference symbol13bdenotes a RAM used for processing thereof, and the reference symbol13cdenotes a ROM used for this processing.

In the ROM13c, various kinds of the gains (Gr, Gb) to be set for the white balance processing circuit14are stored in advance.

Various kinds of gains (Gr1, Gb1), (Gr2, Gb2), . . . are, for example, stored in advance in the ROM13cin the form of color temperature-gain table such as (13c-1) inFIG. 1, in correspondence with color temperature of an image.

In the ROM13cof this embodiment, an ‘alienation-weight table’ (to be described later) such as (13c-2) inFIG. 1is also stored in advance.

In the conventional processing, however, the color temperature of each image is obtained, thereafter its corresponding gain (Gr, Gb) for each image obtained in reference with the color temperature-gain table is only sent to the white balance processing circuit14for the setting.

FIG. 2is an operational flow chart of the setting circuit13.

Note that only processing for the jth (j≦2) obtained image Ijis shown in this operational flow chart.

The gain (Gr, Gb) is set for the first image I1similarly to the conventional art. Further, the gain (Gr1, Gb1) thereof and a color histogram H1(refer to Steps S1, S3described below for the obtaining method thereof) of the image I1are stored in the RAM13bas necessary information for the processing on a subsequent second image I2.

In this embodiment, as shown inFIG. 3, the gain (Grj, Gbj) for the image Ijis decided not only according to this image Ij(B inFIG. 3) but also according to the correlation between the image Ijand an image Ij-1, (A inFIG. 3).

In Step S1inFIG. 2, each color (Ri, Gi, Bi) of small areas ei(i=1 to n, and in n=35 inFIG. 3) in the image Ijis obtained. Here, since each of the small areas eiconsists of a plurality of pixels, the color (Ri, Gi, Bi) thereof can be defined as the total value (or the average value) of the respective pixel values in the small area ei.

Further, in Step S2, the color temperature of the image Ijis obtained in a fashion similar to the conventional technique, and the gain (Grj, Gbj) is tentatively decided based on the color temperature-gain table in the ROM13c.

Further, in Step S3, the color histogram Hjof the image Ijis created based on the color (Ri, Gi, Bi) (i=1 to 35) of each of the small areas eiobtained in Step S1. In this embodiment, the color histogram Hjis used for judging the correlation between the image Ijand the Imagej-1.

At comparing a plurality of images, in general, the human eyes are sensitive to differences in colors, so that the judgment made based on the color histogram is close to that of the human eyes.

For creation of the color histogram, which color area of the color diagram each color (Ri, Gi, Bi) of the respective small areas ejbelongs to is determined, and the frequency at which each color appears is counted for every color area.

Here, for example, the color areas are differently positioned in a hue direction of the color diagram (namely, a circumferential direction of a circle with (R/G, B/G)=(1, 1) being its center).

The human eyes are also sensitive to a chroma direction to some degree, so that it is preferable to provide several color areas in the chroma direction (namely, a diameter direction of the circle with (R/G, B/G)=(1, 1) being its center), as shown inFIG. 4.

In addition, it is preferable that the respective dimensions of the color areas in the hue direction and in the chroma direction are set unequally. This is because it is preferable to discriminate with reliability colors which have a subtle difference from each other but appears quite different to the human eyes, while colors having a large difference from each other but appearing slightly different to the human are preferably judged as equivalent.

In the example shown inFIG. 4, in the chroma direction provided are a plurality of color areas close to a green color (lower left of the color diagram) (consequently, each of the dimensions thereof in the chroma direction is relatively narrow).

This enables green with a low chroma (such as artificial color of a fluorescent lamp) and green with a high chroma (such as natural color of a plant) to be included in different color areas. As a result, the artificial green and the natural green are discriminated.

FIG. 5Bis the color histogram Hjof the image Ijcreated in the manner described above.

Note that at this instant the RAM13balready has the color histogram Hj-1of the previous image Ij-1and the gain (Grj-1, Gbj-1) tentatively decided for this image Ij-1stored therein.

FIG. 5Ais the color histogram Hj-1of the image Ij-1.

In Step S4inFIG. 2obtained is the correlation between the color histogram Hj(FIG. 5B) of the image Ijcreated in Step S3and the color histogram Hj-1(FIG. 5A) of the image Ij-1stored in the RAM13b.

An index of the correlation is, for example, a Euclidean distance (hereinafter, referred to as ‘alienation’) D between a vector of each frequency at which each color appears in the color histogram Hjand a vector of each frequency at which each color appears in the color histogram Hj-1(Formula (1)).

Here, “c” in the formula signifies the number (1 to 10) of the color areas in the color diagram.

The smaller this alienation D is, the higher the correlation between the color of the image Ijand the color of the previous image Ij-1is.

Then, in Step S5inFIG. 2, the gain (Gr, Gb) to be used for the image Ijis decided according to this alienation D, the gain (Grj, Gbj) tentatively decided for the image Ijin Step S2, and the gain (Grj-1, Gbj-1) tentatively decided for the previous image Ij-1.

At this time, the weighted average is applied to the gain (Grj, Gbj) and the gain (Grj-1, Gbj-1) according to the alienation D with reference to the ‘alienation-weight table’ shown inFIG. 1(13c-2). This table stores weights αj-1, αjby which the gain (Grj-1, Gbj-1) and the gain (Grj, Gbj) are to be multiplied, in correspondence with the respective alienations D.

Here, the contents of this table are determined in advance such that the smaller the alienation D becomes, the closer to the gain (Grj-1, Gbj-1) the gain (Gr, Gb) after the averaging becomes, and that the larger the alienation D becomes, the closer to the gain (Grj, Gbj) the gain (Gr, Gb) after the averaging becomes.

The setting circuit13decides the value of the gain (Gr, Gb) by applying the weighted average by the formula (2) (this corresponds to Step S5) and sets this value for the white balance processing circuit14(Step S6).

Note that at the time the processing hitherto is finished, the color histogram Hjand the gain (Grj, Gbj) of the image Ijare stored in the RAM13bto replace the color histogram Hj-1and the gain (Grj-1, Gbj-1) of the image Ij-1.

Thereafter, the white balance processing circuit14inFIG. 1performs white balance processing on the image Ii, using the gain (Gr, Gb).

In this embodiment, as described above, the correlation between the image Ijand the image Ij-1is determined according to the color histogram, and a parameter (gain (Gr, Gb)) for the white balance processing on the image Ijis decided according to this correlation.

As described above, since the human eyes are sensitive to color differences, this way of deciding based on the color histogram can differentiate the image Ij-1and the image Ijin white balance in a similar way that the human eyes do.

Incidentally, the evaluation of the color of each of the small areas eiin the aforesaid Step S1in this embodiment may also be made in the following manner.

First, when there exists a pixel having a distinctive color (for example, a primary color) on the image, the pixel is excluded from objects of color evaluation.

Specifically, the color of the small area eiincluding the pixel having the distinctive color is recognized as the total value (or the average value) of pixels except the pixel having the distinctive color.

In this way, the correlation will not be lower (the alienation D will not be higher) even when only one of the image Ijand the image Ij-1includes the distinctive color, since the distinctive color is ignored.

In a case where a part of one of the image Ijand the image Ij-1contains the distinctive color but most of the areas of the image Ijand the image Ij-1are similar to each other, the human eyes usually recognizes the two images as equivalent. Therefore, it can be said that the correlation is determined in a similar way that the human eyes do.

Incidentally, in the above-described embodiment, the number of the combinations of the weights (αj-1, αj) is five, namely, (10, 0), (8, 2), . . . (0,10), in accordance with the levels of the alienation D as shown inFIG. 1(13c-2), but it may be a different number (for example, 2). For reference, when the number is two and the alienation D has a value smaller than a predetermined value, (αj-1, αj), will be (1, 0), and when the alienation D has a value equal to or larger than the predetermined value, (αj-1, αj) will be (0, 1).

Further, in the above-described embodiment, the objects of the weighted average are the gain (Grj-1, Gbj-1) of the image Ij-1and the gain (Grj, Gbj) of the image Ij, but they may be the color temperatures of the image Ij-1and of the image Ij.

In this case, the gain (Gr, Gb) is stored in the ‘color temperature-gain table’ in advance, in correspondence with the color temperature after the weight averaging. The gain (Gr, Gb) is set for the white balance processing circuit14.

Moreover, in the above-described embodiment, the parameter set according to the correlation is the parameter (gain (Gr, Gb)) for the white balance processing, but it may be a parameter for the gray-scale transformation processing (gray-scale transformation curve) and a parameter for color transformation processing (color transformation matrix).

Further, in the above-described embodiment, the gain tentatively decided for a certain image is used for deciding the gain for its subsequent image but an actually set gain may be used.

In this case, especially when the electronic camera is in a continuous exposure mode, it is highly likely that the correlation between continuously obtained images is sufficiently high. Namely, the weights (αj-1, αj) are likely to be (1, 0) (in other words, the value of the tentatively decided gain (Grj, Gbj) is not required for the operation of the weighted average). Hence, a part of the operational flow chart inFIG. 2may be modified such that Step S2(the step of tentatively deciding (Grj, Gbj)) inFIG. 2is executed only when necessary.

Moreover, the above-described embodiment describes a case where the present invention is applied to the electronic camera, but the present invention is applicable to other devices on which an image processing circuit is mounted, for example, a scanner.

It is also possible to constitute an image processing program comprising the steps (FIG. 2) described in the above embodiment (a program executable by an image processing device such as a computer).