An image processing apparatus includes an acquisition unit for acquiring image data, a determination unit for determining a specific area from the acquired image data, a first calculation unit for calculating a first white balance correction value in accordance with white pixels of an area including an area other than the specific area, a second calculation unit for calculating a second white balance correction value based on a color evaluation value of the specific area, a third calculation unit for calculating a third white balance correction value by performing weighted addition of the first white balance correction value and the second white balance correction value based on a distribution of the color evaluation value of the specific area.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to a white balance adjustment technique in an image processing apparatus.

Description of the Related Art

In recent years, a so-called TTL (Through The Lens) method has been widely used in an automatic white balance processing performed in an image capturing device or the like. In the automatic white balance processing by the TTL method, the color of the light source is estimated according to the image obtained as a result of the image capturing. Then, in order to accurately calculate the white balance (hereinafter WB) correction value, it is necessary to distinguish the light source color and the subject color in the image.

In addition, an image processing apparatus which uses a neural network to determine the similarity between each preset reference scene and a shooting scene, and specify color space coordinates corresponding to the shooting scene is known (see Japanese Patent Laid-Open No. 2013-168723).

However, in the image processing apparatus described in the above-mentioned Japanese Patent Laid-Open No. 2013-168723, there is a possibility that an inappropriate white balance correction value is calculated when the determination is erroneous.

SUMMARY OF THE INVENTION

The present invention has been made in consideration with the above-mentioned problems, and makes it possible to perform white balance adjustment with high accuracy.

According to a first aspect of the present invention, there is provided an image processing apparatus comprising: at least one processor or circuit configured to function as: an acquisition unit configured to acquire image data; a determination unit configured to determine a specific area from the acquired image data; a first calculation unit configured to calculate a first white balance correction value in accordance with white pixels of an area including an area other than the specific area; a second calculation unit configured to calculate a second white balance correction value based on a color evaluation value of the specific area; and a third calculation unit configured to calculate a third white balance correction value by performing weighted addition of the first white balance correction value and the second white balance correction value based on a distribution of the color evaluation value of the specific area.

According to a second aspect of the present invention, there is provided an image processing method comprising: acquiring an image data; determining a specific area from the acquired image data, performing a first calculation for calculating a first white balance correction value in accordance with white pixels of an area including an area other than the specific area, performing a second calculation for calculating a second white balance correction value based on a color evaluation value of the specific area, performing a third calculation for calculating a third white balance correction value by performing weighted addition of the first white balance correction value and the second white balance correction value based on a distribution of the color evaluation value of the specific area.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the claimed invention. Although a plurality of features are described in the embodiments, not all of the plurality of features are essential to the present invention, and the plurality of features may be arbitrarily combined. Furthermore, in the accompanying drawings, the same reference numerals are assigned to the same or similar components, and a repetitive description thereof is omitted.

First Embodiment

FIG. 1is a block diagram showing a configuration example of a digital camera100which is an embodiment of the image processing apparatus of the present invention.

InFIG. 1, a lens group101is a zoom lens including a focus lens. A shutter102is provided with an aperture function and exposes the image sensor included in the imaging unit103in accordance with the control of the system control unit150. The imaging unit103includes an image sensor such as a CCD/CMOS image sensor, and converts the optical image obtained through the lens group101into an electric signal by photoelectric conversion. An A/D converter104converts the analog signal read from the imaging unit103into a digital signal and outputs the image data to the image processing unit105.

The image processing unit105performs various image processing such as white balance adjustment and y correction on the image data output from the A/D conversion unit104or the image data output from the memory control unit107. Further, the image processing unit105performs predetermined evaluation value calculation processing using the face detection result of the face detection unit113and the captured image data, and the system control unit150performs exposure control and focus adjustment control based on the obtained evaluation value. In this way, AF (automatic focus) processing, AE (automatic exposure) processing, AWB (automatic white balance) processing, etc. by the TTL (through the lens) method are carried out.

The memory106temporarily stores image data when the image processing unit105performs various image processing, and stores image data read from the recording medium112through the recording medium interface (I/F)111and image data for display on the display unit109. The memory control unit107controls reading and writing of the memory106. The D/A converter108converts the input digital signal into an analog signal. For example, the image display data stored in the memory106is converted into an analog signal and the analog signal is output to the display unit109.

The display unit109includes a display device such as an LCD, and displays a captured image, an image read out from the recording medium112, a live view image, and the like, as well as a user interface for performing an operation. The codec unit110compresses and encodes or decodes the image data. The codec unit110encodes or decodes the image data stored in the memory106in a format conforming to a standard such as a MPEG.

The recording medium I/F111mechanically and electrically connects a removable recording medium112such as a semiconductor memory card or a card-type hard disk to the digital camera100. The face detection unit113analyzes the image data to detect an area in which a face appears in the image.

The system control unit150includes a CPU or an MPU, executes a program stored in the nonvolatile memory121by expanding the program in a work area of the system memory122, and controls each function of the entire digital camera100.

The operation unit120includes a touch panel for displaying the interface described above, buttons, and switches, and notifies an operation by the user to the system control unit150.

The non-volatile memory121includes, as an auxiliary storage device, a non-volatile solid state memory such as a EEPROM for storing programs, parameters, and the like. The system memory122expands a program or the like read from the nonvolatile memory121and stores constants and variables for the operation of the system control unit150, as a main storage device.

Next, the configuration of the image processing unit105and the processing of each unit will be described with reference toFIG. 2.

InFIG. 2, the image data output from the A/D converter104is input to the image signal generation unit201. The image signal generation unit201performs simultaneous processing on the input RGB image data composed of the bayer array to generate image signals R, G, and B having a plurality of colors per pixel. The image signal generation unit201outputs the generated image signal to the detection unit202and the WB control unit203.

The detection unit202detects a specific area such as a green area of grass, leaves, turf, or the like, a skin color area of a person, or the like from the image signal output from the image signal generation unit201. As a technique for detecting a target area from images, there is a semantic domain division method (Semantic Segmentation) using machine-learning. The semantic domain division method is an image recognition method in which a category of an area occupied by a recognition target in an image is recognized from a feature amount or the like of the area.FIGS. 3A and 3Bshow an example of input and output in the semantic domain division method. A user defines area category to be recognized, such as Person, Sky, Grass, and the like, with respect toFIG. 3A, and the area category is output on a pixel-by-pixel basis as shown inFIG. 3B. In the present embodiment, the area category is output on a pixel-by-pixel basis, and the reliability indicating the likelihood of the category is output on a pixel-by-pixel basis or a specific block-by-block basis. For example, in the case ofFIG. 3B, higher reliability is output for pixels or blocks included in the area of “Grass” for the category “grass”. The reliability can be expressed, for example, in percentage units of 0 to 100. Generally, as a method for achieving this recognition and division, a method based on supervised learning, in which data that is ideally divided into regions and is tagged with categories for each region is used as a correct image, etc. is used.

The WB control unit203calculates the WB correction value based on the image signal output by the image signal generation unit201and the specific area information output (determined) by the area detection circuit in the detection unit202. Then, the WB control unit203corrects the white balance of the image data using the calculated WB correction value. The detailed configuration of the WB control unit203and the method of calculating the WB correction value will be described later.

The color conversion matrix (MTX) circuit204multiplies the color gain so that the image data corrected by the WB control unit203is reproduced in the optimum color, and converts the image data into two color difference signal data R-Y and B-Y. The low-pass filter (LPF) circuit205limits the band of the color difference signal data R-Y, B-Y. The CSUP(Chroma Suppress) circuit206suppresses the false color signal of the saturated portion of the color difference signal data that is band-limited by the LPF circuit205.

On the other hand, the image data corrected by the WB control unit203is also supplied to the luminance signal (Y) generating circuit211, and the luminance signal data Y is generated by the luminance signal generating circuit211. The edge enhancement circuit212applies edge enhancement processing to the generated luminance signal data Y.

The color difference signal data R-Y and B-Y output from CSUP circuit206and the luminance signal data Y output from the edge enhancement circuit212are converted into RGB signal data in the RGB converter circuit207.

The gamma (γ) correction circuit208applies a gradation correction to RGB signal data in accordance with a predetermined γ characteristic. The gamma-corrected RGB signal data, after being converted into YUV signal data by the color brightness converting circuit209, is compressed and encoded in JPEG compressing circuit210and recorded as an image data file on the recording medium112.

Next, the calculation process of the WB correction value performed in the WB control unit203will be described with reference to a flowchart ofFIG. 4.

In step S401, the WB control unit203calculates the first WB correction value for the white pixel of the area including the area other than the specific area. The calculation of the first WB correction value will be described with reference to the flowchart ofFIG. 5.

In step S501, the WB control unit203reads out the image signal stored in the memory106, and divides the image signal into arbitrary m blocks.

In step S502, the WB control unit203calculates a color average value (R[i], G[i] B[i]) by averaging pixel values for each color for each block i (integers of i=1 to m), and calculates a color evaluation value (Cx[i], Cy[i]) using the following equation.

In step S503, the WB control unit203performs white detection using a graph having a coordinate axes as shown inFIG. 6A. The negative direction of x coordinate (Cx) represents the color evaluation value when the white of the high color temperature subject is photographed, and the positive direction represents the color evaluation value when the white of the low color temperature subject is photographed. The y-coordinate (Cy) indicates the degree of green component of the light source and Green component increases as the negative direction, i.e. it indicates that the fluorescent lamp. More specifically, in step S503, the WB control unit203determines whether or not the color evaluation value (Cx[i], Cy[i]) of the i-th block calculated in step S502is included in the white detection range601set in advance as shown inFIGS. 6A-6C.

The white detection range601is obtained by photographing white under different light sources in advance and plotting the calculated color evaluation value. The white detection range can be set separately depending on the shooting mode. In step S503, when it is determined that the calculated color evaluation value (Cx [i], Cy [i]) is included in the white detection range, the WB control unit203advances the process to step S504. On the other hand, if it is determined that this is not the case, the process proceeds to step S505.

In step S504, the WB control unit203determines that the block corresponding to the color evaluation value is white, and integrates the color average value (R[i], G[i], B[i]) of the block. The processes of these step S503and step S504can be represented by the following equations.

Here, when the color evaluation value (Cx[i], Cy[i]) is included in the white detection range, Sw[i] is set to1, and when it is not included in the white detection range, Sw[i] is set to0, thereby the process of whether adding or not adding the color evaluation value ((R[i], G[i], B[i]) according to the determination of step S503is substantially performed.

In step S505, the WB control unit203determines whether or not the above-described process has been performed on all the blocks. When it is determined that the process has been performed on all the blocks, the WB control unit203advances the process to step S506, and if it is determined that not, returns the process to step S502, and repeats the process for the unprocessed block.

In step S506, the WB control unit203calculates the first white balance correction value WBCo1(WBCo1_R,WBCo1_G,WBCo1_B) from the integrated value (SumR, SumG, SumB) of the obtained color evaluation values using the following equation.

The process returns to step S402inFIG. 4, and in step S402, the WB control unit203calculates a second WB correction value from the evaluation value of the specific area. The calculation of the second WB correction value will be described with reference to the flowchart ofFIG. 7.

In step S701, the WB control unit203acquires the reliability of the specific area output from the detection unit202. Here, as described above, the reliability A of the natural greenness of grass, turf, leaves, etc. and the skin-likeness of a person output for each block is acquired. The reason for extracting green and skin is that natural green and human skin exist within a specific color evaluation value range, and an appropriate white balance correction value can be inferred by judging the evaluation value.

In step S702, the WB control unit203multiplies the acquired reliability A of each block by the RGB signal value of the corresponding block to newly obtain (R′ [i], G′ [i], and B′ [i]), and then calculates the color evaluation value (Cx[i], Cy[i]) of the specific area for each block in the same manner as inFIG. 5. Furthermore, as shown in the following equation, the reliability of each block and the signal value for each RGB are integrated to calculate the color evaluation value of a specific color in the entire image. A[i]×R[i], A[i]×G[i], and A[i]×B[i] in the following equations correspond to the above-described [i], G′ [i], and B′ [i], respectively.

Alternatively, in the same manner as the white detection range601described above, it may be determined whether or not the color evaluation value is included in the detection range of the green area (refer toFIG. 6B) or the detection range of the skin area (refer toFIG. 6C) set in advance, and integration may be performed.

The color evaluation value (Cx′, Cy′) of the specific area is calculated from the integrated value (SumR′, SumG′, SumB′) calculated above. Here, when the integrated value is 0 or the reliability A of the specific area is 0, the first WB correction value already obtained is used.

In step S703, the WB control unit203estimates the color temperature of the environmental light source from the color evaluation value (Cx′, Cy′) calculated in step S702.FIG. 8is a diagram showing the distribution of evaluation value of various grass, turf, and the like. Since the green distribution is different for each light source color temperature, it is possible to estimate the color temperature of the environmental light source from its distribution. That is, the color temperature is estimated based on which green distribution the color evaluation value calculated in step S702exists in. For example, when the color estimation values (Cx′, Cy′) exist in green distribution around 8000K, the color temperature of the environmental light source is estimated to be 8000K, and 8000K on the black body radiation axis is set as the second white balance correction value (WBCo2).

AlthoughFIG. 8shows the distribution of the evaluation value of grass, turf, and the like, the distribution of the skin of a person differs depending on the color temperature of the environmental light source, so that the color temperature can be estimated similarly to the green distribution, and the second WB correction value may be determined from the skin area of the person.

Returning to step S403ofFIG. 4, in step S403, the WB control unit203calculates the reliability of the second WB correction value from the color evaluation value distribution for each block of the area determined to be the specific area. Here, the reliability is calculated to detect that the output result from the detection unit202is incorrect, and to detect the fact that the color temperature of the environmental light source is in a situation that it is difficult to estimate, and the like.

For example, when the distribution of the color evaluation value for each of the blocks in the specific area is divided and distributed as shown inFIG. 9A, there is a possibility that one of the color evaluation values may be erroneously detected. In this case, it is determined that the color temperature of the environmental light source is in a situation in which it is difficult to estimate, and it is determined that the reliability of the second WB correction value obtained by integrating the block is low In addition, when the variation of the color evaluation value for each block is large as shown inFIG. 9B, it is determined that the color temperature estimation accuracy of the environmental light source is low, and it is determined that the reliability of the second WB correction value is low

As a method of calculating the reliability, there is a method of calculating the variance value of the color evaluation value of the specific area for each block extracted by the detection unit202. The variance value is calculated using the equation shown below, and when the variance value is equal to or larger than a certain threshold value, it is determined that the reliability of the second WB correction value is low.

VarCx=1n⁢∑j=1n⁢(Cx⁡[j]-AveCx)2VarCy=1n⁢∑j=1n⁢(Cy⁡[j]-AveCy)2AveCx: Average Color Evaluation Value of Specific Area (Cx)AveCy: Average Color Evaluation Value of Specified Area (Cy)

As shown inFIGS. 10A and 10B, GainA and GainB are obtained from the variance value of the color evaluation value, and the reliability of the second WB correction value by the variance value is calculated.

In the above description, when the variance is large, it is determined that the reliability of the second WB correction value is low, but when the color evaluation values of various natural greens such as grass and turf are plotted as shown inFIG. 8, the distribution shape has a certain spread under the same color temperature environment. From this characteristic, the distribution correlation under the same color temperature environment is calculated and if the correlation is high, it can be determined that the reliability of the second WB correction value is high even when the variance value is large for the color evaluation value of the green area.

As a method of calculating the correlation, the correlation (linear correlation value) of the linear approximation formula is calculated from the color evaluation value of the green area, and the reliability of the second WB correction value is calculated from the correlation. The correlation calculation formula is as follows.

This correlation coefficient takes a value of 0 to 1, and when it is close to 1, it is determined that there is a correlation, that is, the shape of the green distribution.

Gain D is calculated as shown inFIG. 10Cby using r2calculated above as the correlation coefficient. When calculating the second WB correction value in the green area, the reliability shall be determined by the maximum value of Gain C and Gain D.When the second WB correction value is calculated in the skin area, the reliability may be determined only by Gain C.

Further, when the area ratio of the specific area is low, there is a possibility that the calculation accuracy of the second WB correction value is low, and therefore, the reliability of the second WB correction value is calculated also based on the area ratio. As shown inFIG. 10D, GainE is calculated in accordance with the area ratio AR (the area of the specific arealthe area including the area other than the specific area). Therefore, the reliability β of the second WB correction value is as follows.

The reliability β calculated above is defined as a mixing ratio for weighted addition of the first WB correction value and the second WB correction value.

In step S404, the first WB correction value and the second WB correction value are weighted and added to each other by the mixing ratio calculated above, thereby the final white balance correction value (WBCo) is calculated.

In the above embodiment, the image processing apparatus of the present invention is exemplified by a digital camera, but the present invention is not limited thereto. For example, the present invention can be applied to a digital video camera or an information processing apparatus including a digital camera, for example, a personal computer, a mobile terminal, and the like.

Other Embodiments

This application claims the benefit of Japanese Patent Application No. 2020-033749, filed Feb. 28, 2020 which is hereby incorporated by reference herein in its entirety.