IMAGE PROCESSING APPARATUS

An image processing apparatus which captures a group of combining frames different in exposures and generates one frame by combining frames includes an image capturing unit configured to capture four frames including a proper frame every other frame and an under frame in one frame and an over frame in a remaining one frame therebetween as a combining frame group, a development unit configured to develop each frame in the combining frame group, a combining unit configured to select at least one frame in the combining frame group to combine based on the developed combining frame group, and a processing unit configured to apply an effect based on the combined combining frame.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to image processing for obtaining a moving image in frames subjected to high dynamic range combination from images in a plurality of frames especially in digitalized moving image signals.

Description of the Related Art

There is a conventional technique of image processing for obtaining a high dynamic range image (hereinbelow, referred to as HDR) by combining images in a plurality of frames which are captured in different exposure amounts. According to the technique, an image without overexposure or underexposure can be obtained by combining signals of proper exposure in each image. According to the technique, the same can be applied to a moving image by repeating combining of a plurality of captured frames into one frame in time series. However, when the technique is applied to a moving image, there is an issue that a frame rate of a combined moving image is generally reduced lower than an image capturing frame rate, and thus a movement of a moving object portion looks unnatural. In contrast, a method for making a movement look natural is known in which a moving object portion is detected, and a multiplex combining unit and the like combines images according to a detected result and adds an effect of giving a blurring appearance to the object.

On the other hand, when an effect is individually applied to each image before combining, the effect can be further greatly exerted. In this case, the effect becomes greater when the number of frames to be captured is larger and an exposure difference is larger. However, when assuming a case in which a moving object is detected in frames having the same exposure, as the number of frames to be captured and the exposure difference are larger, a distance becomes larger between the frames having the same exposure, and accuracy of moving object detection is deteriorated. In addition, when information such as auto focus (AF), auto exposure (AE), and auto white balance (AWB), (hereinbelow, referred to as an auto correction system) is obtained from an image, it is desirable to obtain the information from a properly exposed frame, and correction is performed by applying the information to a frame other than the properly exposed frame. In this case, when a distance is large between the properly exposed frame and the other frame, correction accuracy is deteriorated.

Japanese Patent Application Laid-Open No. 2011-199787 describes a technique in which when one image is generated by combining a plurality of images, the number of combined images is determined based on a noise amount and a required contrast amount.

Japanese Patent Application Laid-Open No. 2006-5681 describes a technique which can store images capturing an object in different exposure times in one moving image data and reproduce the moving image data in a plurality of reproduction modes, and in other words, a technique which can reproduce two types of images having different atmospheres by storing two types of images in one stream and selecting frames when reproducing.

According to the technique described in Japanese Patent Application Laid-Open No. 2011-199787, when combination of moving images is assumed, a frame rate needs to be changed accordingly in a case where the number of combined frames is variable. In addition, when the number of combined images is increased, a temporal distance becomes larger between frames having the same exposure, and accuracy of the moving object detection is significantly deteriorated. Further, when the information of the auto correction system is obtained from a certain frame, a temporal distance becomes larger between the frame from which the information is obtained and a frame to be corrected, and correction accuracy is deteriorated.

According to the technique described in Japanese Patent Application Laid-Open No. 2006-5681, a configuration of an image capturing method is similar to that of HDR in a moving image, however, it is difficult to exert an HDR effect since images are not combined.

In other words, the conventional techniques are not sufficient to exert the HDR effect when increasing the number of combined frames while maintaining accuracy of the moving object detection and correction accuracy of the auto correction system.

SUMMARY OF THE INVENTION

One exemplary embodiment of the present invention is an image processing apparatus which captures a group of combining frames different in exposures and generates one frame by combining frames and includes an image capturing unit configured to capture four frames including a proper frame every other frame and an under frame in one frame and an over frame in a remaining one frame therebetween as a combining frame group, a development unit configured to develop each frame in the combining frame group, a combining unit configured to select at least one frame in the combining frame group to combine based on the developed combining frame group, and a processing unit configured to apply an effect based on the combined combining frame.

DESCRIPTION OF THE EMBODIMENTS

Various exemplary embodiments of the present invention will be described below.

According to a first exemplary embodiment, a configuration is described in which a digital camera for capturing a moving image captures four frames including a proper frame, an under frame, a proper frame, and an over frame in time series, performs HDR composition thereon, and applies a painterly effect to the generated HDR image. Each frame is developed using a gamma for matching brightness of the frame with each other. Thus, moving object detection which is essential in a moving image can be performed by calculating a difference value between adjacent frames.

In addition, a method is described which calculates a white balance (WB) coefficient using each proper frame and uses the WB coefficient for WB correction of an under frame and an over frame captured immediately after each proper frame. Further, as an effect to be applied to a moving image, processing (a painterly effect) is described as an example which generates a halo (a white or black halo) at an edge portion having a large density difference in addition to darkening a bright portion and brightening a dark portion by correcting a tone.

As described above, the present exemplary embodiment is directed to compatibility between maintaining of the painterly effect by performing HDR composition using three types of exposures and applying the effect and continuity of the WB correction in a moving image by calculating a WB coefficient using a proper frame to be periodically inserted.

FIG. 1is a block diagram illustrating an example of a camera system according to the first exemplary embodiment.

The camera system illustrated inFIG. 1includes an image capturing system1as an image capturing unit, a signal processing unit2, an HDR painterly processing unit3, a signal processing unit4, an encoding processing unit5, an output unit6, a user interface (UI) unit7, and a bus8.

FIG. 2is a block diagram illustrating an example of the HDR painterly processing unit3.

The HDR painterly processing unit3illustrated inFIG. 2includes input terminals301,302,303and304, WB coefficient calculation units305and306, development units307,308and309, combining units310and311, a tone correction unit312, a local contrast correction unit313, and an output terminal314.

A processing outline according to the present apparatus including the above-described configuration is described below.

The image capturing system1photoelectrically converts light passing through an iris, a lens, and the like by an imaging element including a complementary metal-oxide semiconductor (CMOS) and a charge-coupled d (CCD) and supplies the photoelectrically converted image data to the signal processing unit2. The imaging element includes a Bayer array. The signal processing unit2performs analog-to-digital (A/D) conversion, gain control, and the like on the photoelectrically converted image data and supplies the processing result to the HDR painterly processing unit3as a digital image signal. The UI unit7performs imaging settings such as selection of moving image/still image modes, an HDR painterly mode, an ISO sensitivity, and a shutter speed, and information of these settings is supplied to the image capturing system1, the signal processing unit2, the HDR painterly processing unit3, the signal processing unit4, the encoding processing unit5, and the output unit6via the bus8.

The signal processing unit2inputs a proper frame101, an under frame102, a proper frame103, and an over frame104to the HDR painterly processing unit3via the input terminals301,302,303, and304. The proper frame101is supplied to the WB coefficient calculation unit305. The under frame102is supplied to the development unit307. The proper frame103is supplied to the WB coefficient calculation unit306and the development unit308. The over frame104is supplied to the development unit309. The WB coefficient calculation unit305calculates a WB coefficient based on the input proper frame101and supplies the WB coefficient to the development unit307. Similarly, the WB coefficient calculation unit306calculates a WB coefficient based on the input proper frame103and supplies the WB coefficient to the development units308and309.

The development unit307develops the under frame102based on the input WB coefficient and supplies the developed frame to the combining unit310. The development unit308develops the proper frame103based on the input WB coefficient and supplies the developed frame to the combining unit310. The development unit309develops the over frame104based on the input WB coefficient and supplies the developed frame to the combining unit311.

The combining unit310combines the developed under frame102and the developed proper frame103and supplies the combined frame as a first combined frame to the combining unit311. The combining unit311combines the first combined frame and the developed over frame104and supplies the combined frame as a second combined frame to the tone correction unit312.

The tone correction unit312performs tone correction on the second combined frame and supplies the result as a tone curve corrected image to the local contrast correction unit313. The local contrast correction unit313performs local contrast correction on the image data and output the result as an output image to the output terminal314.

Processing by the HDR painterly processing unit3in the camera system configured as described above is described in more detail below.

FIG. 3illustrates an image capturing order according to the present exemplary embodiment.FIG. 3illustrates that the proper frame101, the under frame102, the proper frame103, and the over frame104are captured in this order in time series, and one frame is generated by combining these four frames as a combining frame group. Further, a proper frame105, an under frame106, a proper frame107, and an over frame108are successively input as a combining frame group to generate next one frame. Here, a case is described as an example in which the proper frame101, the under frame102, the proper frame103, and the over frame104are input.

The proper frame101, the under frame102, the proper frame103, and the over frame104are respectively input from the input terminals301,302,303, and304in a successive manner by frame unit. For the sake of description, an exposure difference of the proper frame and the under frame and an exposure difference of the proper frame and the over frame are respectively, for example, two stages based on a difference in the ISO sensitivity.

The WB coefficient calculation unit305performs processing for whitening white, specifically, calculates a gain (equivalent to the WB coefficient) for making signal values of red, green and blue (R, G, and B) in an area to be white the same using information of the input proper frame101. The WB coefficient calculation unit306performs processing equivalent to that of the WB coefficient calculation unit305using information of the proper frame103.

Generally, it is desirable to calculate the WB coefficient using a frame near a proper exposure.

Therefore, according to the present exemplary embodiment, a proper frame is captured every other frame, a WB coefficient is calculated using the information of the proper frame, and the WB coefficient is applied to the proper frame used for calculating the WB coefficient and another frame than the immediately before proper frame. The proper frame is thus periodically inserted, and accordingly a temporal distance between a frame used for calculating the WB coefficient and a frame being applied with the WB coefficient can be closer, and white balance processing can be highly accurately performed. In addition, the proper frame is periodically inserted, and thus the WB coefficient can be continuously calculated, so that continuity of the processing which is essential for a moving image can be secured.

The development units307,308, and309respectively perform development processing on the under frame102, the proper frame103, and the over frame104.FIG. 4is a block diagram illustrating the development units307,308, and309. Regarding the outline of processing, most parts of the processing are common in the development units307,308, and309, so that the common parts are described using the development unit307.

A white balance unit3071performs processing for whitening white using the input WB coefficient. A noise reduction (NR) processing unit3072reduces noise which is not derived from an object image in an input image but caused by the sensor and the like. A color interpolation unit3073generates a color image in which all pixels completely include R, G, and B color information pieces by interpolating a color mosaic image. The generated color image is processed via a matrix transformation unit3074and a gamma conversion unit3075, and accordingly a basic color image is generated. Subsequently, a color adjustment unit3076performs processing namely image correction such as, chroma enhancement, hue correction, and edge enhancement on the color image for improving a visual quality of the image.

According to the present exemplary embodiment, gains are applied to image signals captured in different exposures in advance to equalize luminance levels therebetween. A gain is required to be set so as not to cause overexposure or underexposure, thus not a uniform gain but a gamma corresponding to an exposure value as illustrated inFIG. 5is used. InFIG. 5, a solid line, a dotted line, and a thick line respectively represent examples of a gamma for a proper frame, a gamma for an under frame, and a gamma for an over frame. These gammas are used, and the gamma conversion unit3075applies the gamma for the under frame, a gamma conversion unit3085applies the gamma for the proper frame, and a gamma conversion unit3095applies the gamma for the over frame.

As can be seen from gamma characteristics illustrated inFIG. 5, the under frame is applied with the gain larger than the gain applied to the proper frame, and thus there is a concern that noise is increased in the under frame after development compared to the proper frame. In addition, the proper frame is applied with the gain larger than the gain applied to the over frame, and thus there is a concern that noise is increased in the proper frame after development compared to the over frame. Thus, the NR processing unit3072performs NR stronger than that of a NR processing unit3082, and the NR processing unit3082performs NR stronger than that of a NR processing unit3092, so that noise amounts of the proper frame, the under frame, and the over frame after development are equalized. Thus, the NR processing can reduce a feeling of strangeness caused by differences among images of the proper frame, the under frame, and the over frame after the HDR composition. As specific methods for noise reduction, there are various methods including a general method such as smoothing process using an appropriate kernel size and a method using a filter such as an ϵ filter and an edge-preserving bilateral filter, however, an appropriate method may be used in consideration of a balance in a processing speed of the system and a resource such as a memory.

Regarding the above-described development processing, the development units307,308, and309are described, however, block configurations are the same, so that one single development unit may be used in common by switching parameters used in the development according to an input of the proper frame, the under frame, or the over frame.

The combining unit310calculates a composition ratio of the under frame102using a luminance composition ratio to be calculated in response to luminance of the under frame102and a luminance difference composition ratio to be calculated in response to a difference value between the under frame102and the proper frame103. Further, the combining unit310combines the under frame102and the proper frame103based on the calculated composition ratio and outputs the combined frame as the first combined frame.

FIG. 6is a block diagram illustrating the combining unit310. The under frame102is input from an input terminal3101and supplied to a luminance composition ratio calculation unit3103, a luminance difference composition ratio calculation unit3104, and a combining processing unit3106. The proper frame103is input from an input terminal3102and supplied to the luminance difference composition ratio calculation unit3104and the combining processing unit3106.

The luminance composition ratio calculation unit3103calculates the luminance composition ratio in response to the luminance of the under frame102and supplies the luminance composition ratio to a composition ratio calculation unit3105. The luminance difference composition ratio calculation unit3104calculates the luminance difference composition ratio in response to a luminance difference between the under frame102and the proper frame103and supplies the luminance difference composition ratio to the composition ratio calculation unit3105. The composition ratio calculation unit3105supplies, for each area, one having a larger value of the luminance composition ratio and the luminance difference composition ratio as a final composition ratio to the combining processing unit3106. The combining processing unit3106combines the proper frame103and the under frame102based on the final composition ratio and outputs the first combined frame from an output terminal3107.

The processing is described in more detail below.

The luminance composition ratio calculation unit3103calculates the luminance composition ratio of the under frame102with respect to the luminance of the under frame102.FIG. 7is an example of a graph for calculating the luminance composition ratio. The processing is described below with reference toFIG. 7. The luminance composition ratio is calculated for each area in response to the luminance of the under frame102.FIG. 7represents that the proper frame103is used in an area darker than a luminance composition threshold value Y1, and the under frame102is used in an area brighter than a luminance composition threshold value Y2to obtain an HDR composition image. In addition, in an intermediate area in boundaries Y1to Y2near the luminance composition threshold values, the composition ratio is gradually changed to smooth switching of images.

Next, the luminance difference composition ratio calculation unit3104is described.

The luminance difference composition ratio calculation unit3104calculates the luminance difference composition ratio of the under frame102with respect to the luminance difference between the under frame102and the proper frame103.FIG. 8is an example of a graph for calculating the luminance difference composition ratio. The processing is described below with reference toFIG. 8. The luminance difference composition ratio is calculated for each area in response to the luminance difference between the under frame102and the proper frame103.FIG. 8represents that the proper frame103is used in an area of which the luminance difference is smaller than a luminance difference composition threshold value d1, and the under frame102is used in an area of which the luminance difference is larger than a luminance difference composition threshold value d2. In addition, in an intermediate area in boundaries d1to d2near the luminance difference composition threshold values, the composition ratio is gradually changed to smooth switching of images.

Next, the composition ratio calculation unit3105is described.

The composition ratio calculation unit3105calculates the final composition ratio using the luminance composition ratio and the luminance difference composition ratio. In this regard, one having a larger value of the luminance composition ratio and the luminance difference composition ratio is calculated as the final composition ratio for each pixel.

Finally, the combining processing unit3106calculates combined image data of the first combined frame using the calculated final composition ratio based on a following formula.

Each term in the formula is as follows.
fg1: a composition ratio
FI1: image data of the first combined frame
UI1: image data of the under frame102
MI1: image data of the proper frame103

The combining unit311calculates the composition ratio of the over frame104using the luminance composition ratio calculated in response to luminance of the over frame104and the luminance difference composition ratio calculated in response to a difference value between the over frame104and the first combined frame. Further, the combining unit311combines the over frame104and the first combined frame based on the calculated composition ratio and outputs the combined frame as the second combined frame.

FIG. 9is a block diagram illustrating the combining unit311. The over frame104is input from an input terminal3111and supplied to a luminance composition ratio calculation unit3113, a luminance difference composition ratio calculation unit3114, and a combining processing unit3116. The first combined frame is input from an input terminal3112and supplied to the luminance difference composition ratio calculation unit3114and the combining processing unit3116.

The luminance composition ratio calculation unit3113calculates the luminance composition ratio in response to the luminance of the over frame104and supplies the luminance composition ratio to a composition ratio calculation unit3115. The luminance difference composition ratio calculation unit3114calculates the luminance difference composition ratio in response to the luminance difference between the over frame104and the first combined frame and supplies the luminance difference composition ratio to the composition ratio calculation unit3115. The composition ratio calculation unit3115supplies, for each area, one having a larger value of the luminance composition ratio and the luminance difference composition ratio as the final composition ratio to the combining processing unit3116. The combining processing unit3116combines the first combined frame and the over frame104based on the final composition ratio and outputs the combined frame as the combined image data (the second combined frame) from an output terminal3117.

The processing is described in more detail below.

The luminance composition ratio calculation unit3113calculates the luminance composition ratio of the over frame104with respect to the luminance of the over frame104.FIG. 10is an example of a graph for calculating the luminance composition ratio. The processing is described below with reference toFIG. 10. The luminance composition ratio is calculated for each area in response to the luminance of the over frame104.FIG. 10represents that the over frame104is used in an area darker than a luminance composition threshold value Y3, and the first combined frame is used in an area brighter than a luminance composition threshold value Y4to obtain the HDR composition image. In addition, in an intermediate area in boundaries Y3to Y4near the luminance composition threshold values, the composition ratio is gradually changed to smooth switching of images.

Next, the luminance difference composition ratio calculation unit3114is described.

The luminance difference composition ratio calculation unit3114calculates the luminance difference composition ratio of the over frame104with respect to the luminance difference between the over frame104and the first combined frame.FIG. 11is an example of a graph for calculating the luminance difference composition ratio. The processing is described below with reference toFIG. 11. The luminance difference composition ratio is calculated for each area in response to the luminance difference between the over frame104and the first combined frame.FIG. 11represents that the first combined frame is used in an area of which the luminance difference is smaller than a luminance difference composition threshold value d3, and the over frame104is used in an area of which the luminance difference is larger than a luminance difference composition threshold value d4. In addition, in an intermediate area in boundaries d3to d4near the luminance difference composition threshold values, the composition ratio is gradually changed to smooth switching of images.

Next, the composition ratio calculation unit3115is described.

The composition ratio calculation unit3115calculates the final composition ratio using the luminance composition ratio and the luminance difference composition ratio. In this regard, one having a larger value of the luminance composition ratio and the luminance difference composition ratio is calculated as the final composition ratio.

Finally, the combining processing unit3116calculates combined image data of the second combined frame using the calculated final composition ratio based on a following formula.

Each term in the formula is as follows.
fg2: a composition ratio
FI2: image data of the second combined frame (the combined image data)
OI1: image data of the over frame104
FI1: image data of the first combined frame

The tone correction unit312corrects a tone curve using a lookup table (LUT) with respect to the combined image data.FIG. 12illustrates an example of a tone curve of an LUT, in which an abscissa axis represents input luminance, and an ordinate axis represents output luminance. As seen inFIG. 12, contrast is enhanced in a dark portion and a bright portion and reduced in an intermediate luminance portion, and thus an effect of looking the image as a painting can be exerted. The image thus subjected to the combined image data tone curve correction is output as a tone curve corrected image to the local contrast correction unit313.

Regarding the tone curve correction, processing for enhancing the contrast of the dark portion and the bright portion is performed as illustrated inFIG. 12. However, when no gradation remains in the dark portion and the bright portion, an enhancement effect is less how much the contrast enhancement processing is added. According to the present exemplary embodiment, the tone curve correction is performed on an image which is obtained by the HDR composition using the three types of exposures and in which gradation remains in the dark portion and the bright portion, so that the contrast enhancement effect can be more greatly exerted compared to, for example, an image obtained by the HDR composition using two types of exposures.

The local contrast correction unit313performs processing for generating a halo near an edge having a large difference in brightness and darkness.FIG. 13is a block diagram illustrating the local contrast correction unit313.

The tone curve corrected image113is input from an input terminal3131and supplied to an area information generation unit3132and a luminance correction unit3135. The area information generation unit3132divides the image into areas in block unit, calculates an average value of each area, and supplies the average value as a representative value of each area to an area information substitution unit3133. The area information substitution unit3133converts the representative value of each area into a gain value using a gain table and supplies the gain value to a gain value calculation unit3134. The gain value calculation unit3134converts the gain value of each area into a gain value of each pixel and supplies the converted gain value to the luminance correction unit3135. The luminance correction unit3135calculates a luminance corrected image114(not illustrated) based on the tone curve corrected image113and the gain value of each pixel and outputs the luminance corrected image114from an output terminal3136.

Next, the local contrast correction unit313is described in more detail.

The area information generation unit3132divides the input tone curve corrected image113into areas. Fig. illustrates an example when an image is divided into nine in a horizontal direction and into six in a vertical direction. According to the present exemplary embodiment, an image is divided into rectangular shapes, however, an image can be divided into arbitrary shapes including polygonal shapes such as triangle shapes and hexagonal shapes. Further, an average value of luminance values of all pixels included in the area is calculated for each divided area as a representative luminance value of the area.FIG. 15illustrates an example of a representative luminance value of each area corresponding toFIG. 14. According to the present exemplary embodiment, the representative value of the area is the average value of the luminance, however, an average value of any of the RGB values may be regarded as the representative value of the area.

The area information substitution unit3133replaces the representative luminance value of each area with the gain value. For example, the representative luminance value can be replaced with the gain value by referring to a gain table stored in advance.FIG. 16illustrates an example of a gain table characteristic according to the present exemplary embodiment. The gain table characteristic is adjusted, and thus intensity of the halo generated in the image output from the luminance correction unit3135can be changed. For example, when the halo to be generated is strengthened, a difference may be increased between a gain to an area of which the luminance average value is low and a gain to an area of which the luminance average value is high.

The gain value calculation unit3134calculates the gain value of each pixel using the gain value of each area as an input. For example, the gain of each pixel is calculated based on a principle described below. First, a distance from a pixel for calculating a gain (a target pixel) to a center or a gravity center of a plurality of areas near an area including the target pixel is calculated, and four areas are selected in order of shortness of the distance. Subsequently, the gain value of each pixel is calculated by performing two-dimensional linear interpolation so that the gain value of each of the four selected areas is applied with a larger weight as the distance is smaller between the target pixel and the center/gravity center of the area. There is no limitation on methods for calculating the gain value of each pixel based on the gain value of each area, and it is needless to say that other methods may be used.

If an original image having 1920*1080 pixels is divided into blocks of 192 vertical pixels by 108 horizontal pixels, 10*10 pixels (an image constituted of the representative luminance value) are output from the area information generation unit3132. When an image in which each pixel value of the image (the representative luminance value) is replaced with the gain value by the area information substitution unit3133is enlarged by the linear interpolation to the number of pixels of the original image, each pixel value after the enlargement will be the gain value of the pixel corresponding to the original image.

The luminance correction unit3135performs luminance correction on each pixel by applying the gain value calculated for each pixel to the tone curve corrected image113and outputs the result to the output terminal3136. The luminance correction is realized by following formulae.

Rout, Gout, and Bout respectively represent the RGB pixel values after the luminance correction, Rin, Gin, and Bin respectively represent the RGB pixel values of the tone curve corrected image113, and Gain represents the gain value of each pixel.

The local contrast correction is processing for applying a larger gain to a dark portion in an image as illustrated in a gain table inFIG. 16. In this case, if a gradation is not remained in the dark portion, an effect of applying the gain cannot be exerted, and also noise is generated in the dark portion which may greatly deteriorate the image quality in some cases. According to the present exemplary embodiment, the over frame is used in the HDR composition, so that the noise of the dark portion can be suppressed compared to a case when the over frame is not used, and the gradation in the dark portion can be maintained.

Finally, the luminance corrected image is output from the output terminal314.

As described above, according to the present exemplary embodiment, the HDR image is generated using images of the three types of exposures, namely the proper frame, the under frame, and the over frame, and accordingly an image can be generated in which gradations are remained in a dark portion and a bright portion. The tone curve correction and the local contrast correction for exerting the painterly effect are applied to the thus generated image, so that the higher painterly effect can be realized than a case, for example, when the HDR image is generated using only the proper frame and the under frame. In addition, the WB coefficient is calculated using the proper frame which is inserted every other frame while capturing images of the three types of exposures, so that a time lag in the WB correction other than the proper frame can be reduced, and the WB correction can be realized which is more highly accurate and maintains the continuity essential for a moving image.

The present exemplary embodiment is described using the WB coefficient as the example, however, it is needless to say that the present exemplary embodiment can be applied to processing for performing detection in a proper frame and applying the detection result to an under frame and an over frame, such as processing for performing gradation correction based on a histogram of an image.

According to a second exemplary embodiment, development is performed without matching brightness between frames different in exposures so as to increase a painterly effect more than that in the first exemplary embodiment. According to the first exemplary embodiment, brightness between different exposures is equalized using the gamma, and thus moving object detection can be performed by calculating a difference between different exposures, however, the same cannot be applied to the present exemplary embodiment. Thus, according to the present exemplary embodiment, a method for detecting a moving object is described in which a difference value is calculated between proper frames of which brightness is matched with each other in five frames including a proper frame, an under frame, a proper frame, an over frame, and further a proper frame captured after the over frame in time series for the moving object detection. In addition, an effect by tone correction after the HDR composition and processing for generating a halo at an edge portion having a large density difference are described as an example of the painterly effect in addition to a tone correction effect by the gamma.

FIG. 1is a block diagram illustrating an example of a camera system according to the second exemplary embodiment. The configuration and outline of processing inFIG. 1are the same as those according to the first exemplary embodiment, and thus the descriptions thereof are omitted.

FIG. 17is a block diagram illustrating an example of the HDR painterly processing unit3.

The HDR painterly processing unit3illustrated inFIG. 17includes input terminals321,322,323,324, and325, development units326,327,328,329, and330, movement detection units331and332, combining units333and334, a tone correction unit335, a local contrast correction unit336, and an output terminal337.

A processing outline of the HDR painterly processing unit3including the above-described configuration is described below.

The proper frame105, the proper frame101, the proper frame103, the under frame102, and the over frame104are input from the signal processing unit2via the input terminals321,322,323,324, and325and respectively supplied to the development units326,327,328,329, and330.

The development units326,327,328,329, and330perform development processing on the respective input images. The development unit326supplies the developed image to the movement detection unit331. The development unit327supplies the developed image to the movement detection unit332. The development unit328supplies the developed image to the movement detection units331and332and the combining unit333. The development unit329supplies the developed image to the combining unit333. The development unit330supplies the developed image to the combining unit334. The movement detection unit331performs movement detection based on the developed proper frames105and103and supplied the movement detection result to the combining unit334.

The movement detection unit332performs movement detection based on the developed proper frames101and103and supplies the movement detection result to the combining unit333. The combining unit333combines the developed proper frame103and the developed under frame102based on the movement detection result and supplies the combined frame as a third combined frame to the combining unit334. The combining unit334combines the developed over frame104and the third combined frame based on the movement detection result and supplies the combined frame as a fourth combined frame to the tone correction unit335.

The tone correction unit335performs tone correction processing on the fourth combined frame and supplies the result as a tone corrected frame to the local contrast correction unit336. The local contrast correction unit336performs local contrast correction processing on the tone corrected frame and outputs the result as a local contrast correction frame from the output terminal337.

Processing by the HDR painterly processing unit3in the camera system configured as described above is described in more detail below.

FIG. 3illustrates an image capturing order according to the present exemplary embodiment. The image capturing order isFIG. 3is the same as that according to the first exemplary embodiment, and thus the description thereof is omitted. However, according to the present exemplary embodiment, a case is described as an example in which the proper frame105, the proper frame101, the proper frame103, the under frame102, and the over frame104are input as described above.

The proper frame105, the proper frame101, the proper frame103, the under frame102, and the over frame104are respectively input from the input terminals321,322,323,324, and325by frame unit. For the sake of description, an exposure difference of the proper frame and the under frame and an exposure difference of the proper frame and the over frame are respectively, for example, two stages based on a difference in the ISO sensitivity.

The development units326,327,328,329and330respectively perform the development processing on the proper frame105, the proper frame101, the proper frame103, the under frame102, and the over frame104. The block configurations are common in the development units326,327,328,329, and330, and thus the common portion is described using the development unit326as a representative, and different portions of processing are described using the respective blocks.FIG. 18is a block diagram illustrating the development unit326.

A white balance unit3261performs processing for whitening white, specifically, calculates a gain (equivalent to the WB coefficient) for making signal values of R, G, and B in an area to be white the same using respectively input frame information. A NR processing unit3262reduces noise which is not derived from an object image in an input image but caused by the sensor and the like. A color interpolation unit3263generates a color image in which all pixels completely include R, G, and B color information pieces by interpolating a color mosaic image. The generated color image is processed via a matrix transformation unit3264and a gamma conversion unit3265, and accordingly a basic color image is generated. Subsequently, a color adjustment unit3266performs processing namely image correction such as, chroma enhancement, hue correction, and edge enhancement on the color image for improving a visual quality of the image.

According to the present exemplary embodiment, the same gain is applied to image signals captured in different exposures in advance. A gain is required to be set so as not to cause overexposure or underexposure, thus not a uniform gain but a gamma as illustrated inFIG. 19is used. According to the present exemplary embodiment, processing for brightening a dark portion and darkening a bright portion is performed to more enhance the painterly effect. As a final target, it is desirable to combine images so that the under frame, the over frame, and the proper frame are respectively assigned to a bright portion, a dark portion, and an intermediate portion. Thus, when brightness of the proper frame is used as a reference, the under frame is applied with a gamma for darkening compared to the proper frame, and the over frame is applied with a gamma for brightening compared to the proper frame. In other words, the same gamma is applied to each frame, and accordingly an effect of brightening the dark portion and darkening the bright portion in the final combined image can be exerted. The gamma conversion units3265,3275,3285,3295, and3305perform the above-described gamma processing on the respective frames.

The development processing is described above on the assumption that the development units326,327,328,329, and330physically exist, however, the block configurations are the same, so that one single development unit may be used in common by switching parameters used in the development according to an input of the proper frame, the under frame, or the over frame.

The movement detection unit331detects a movement between the developed proper frames105and103. Specifically, the movement detection unit331calculates a difference value of each pixel in the proper frame105and in the proper frame103and outputs the difference value of each pixel as a first movement detection frame. Similarly, the movement detection unit332calculates a difference value of each pixel in the proper frame101and that in the proper frame103to detect a movement between the developed proper frames101and103and outputs the difference value of each pixel as a second movement detection frame.

The combining unit333calculates a composition ratio of the under frame102using a luminance composition ratio to be calculated in response to luminance of the developed under frame102and a luminance difference composition ratio to be calculated in response to the first movement detection frame, Further, the combining unit333combines the under frame102and the proper frame103based on the calculated composition ratio and outputs the combined frame as the third combined frame.

FIG. 20is a block diagram illustrating the combining unit333. The developed under frame102is input from an input terminal3331and supplied to a combining processing unit3337. The second movement detection frame is input from an input terminal3332and supplied to a luminance difference composition ratio calculation unit3335. The developed proper frame103is input from an input terminal3333and supplied to a luminance composition ratio calculation unit3334and the combining processing unit3337.

The luminance composition ratio calculation unit3334calculates the luminance composition ratio based on luminance of the developed proper frame103and supplies the luminance composition ratio to a composition ratio calculation unit3336. The luminance difference composition ratio calculation unit3335calculates the luminance difference composition ratio based on the second movement detection frame and outputs the luminance difference composition ratio to the composition ratio calculation unit3336. The composition ratio calculation unit3336supplies, for each area, one having a larger value of the luminance composition ratio and the luminance difference composition ratio as a final composition ratio to the combining processing unit3337. The combining processing unit3337combines the proper frame103and the under frame102based on the final composition ratio and outputs the combined frame as a third combined frame from an output terminal3338.

The processing is described in more detail below.

The luminance composition ratio calculation unit3334calculates the luminance composition ratio of the under frame102using the luminance of the proper frame103.FIG. 21is an example of a graph for calculating the luminance composition ratio. The processing is described below with reference toFIG. 21. The luminance composition ratio is calculated for each area in response to the luminance of the proper frame103.FIG. 21represents that the proper frame103is used in an area darker than a luminance composition threshold value Y5, and the under frame102is used in an area brighter than a luminance composition threshold value Y6to obtain an HDR composition image. In addition, in an intermediate area in boundaries Y5to Y6near the luminance composition threshold values, the composition ratio is gradually changed to smooth switching of images.

Next, the luminance difference composition ratio calculation unit3335is described.

The luminance difference composition ratio calculation unit3335calculates the luminance difference composition ratio of the under frame102using the second movement detection frame.FIG. 22is an example of a graph for calculating the luminance difference composition ratio. The processing is described below with reference toFIG. 22. The luminance difference composition ratio is calculated for each area using the second movement detection frame.FIG. 22represents that the proper frame103is used in an area in which a value of the movement detection frame is less than a luminance difference composition threshold value d5, and the under frame102is used in an area in which a value of the movement detection frame is greater than a luminance difference composition threshold value d6. In addition, in an intermediate area in boundaries d5to d6near the luminance difference composition threshold values, the composition ratio is gradually changed to smooth switching of images.

Next, the composition ratio calculation unit3336is described.

The composition ratio calculation unit3336calculates the final composition ratio using the luminance composition ratio and the luminance difference composition ratio. In this regard, one having a larger value of the luminance composition ratio and the luminance difference composition ratio is calculated as a final composition ratio fg3 for each pixel.

Finally, the combining processing unit3337calculates combined image data of the third combined frame121using the calculated final composition ratio based on a following formula.

Each term in the formula is as follows.
fg3: a composition ratio
FI3: image data of the third combined frame
MI3: image data of the proper frame103
UI3: image data of the under frame102

The combining unit334calculates the composition ratio of the over frame104using the luminance composition ratio to be calculated in response to luminance of the third combined frame and the luminance difference composition ratio to be calculated in response to the first movement detection frame. Further, the combining unit334combines the third combined frame and the over frame104based on the calculated composition ratio and outputs the combined frame as a fourth combined frame.

FIG. 23is a block diagram illustrating the combining unit334. The developed over frame104is input from an input terminal3341and supplied to a combining processing unit3347. The first movement detection frame is input from an input terminal3342and supplied to a luminance difference composition ratio calculation unit3345. The third combined frame is input from an input terminal3343and supplied to a luminance composition ratio calculation unit3344and the combining processing unit3347.

The luminance composition ratio calculation unit3344calculates the luminance composition ratio based on the luminance of the third combined frame and supplies the luminance composition ratio to a composition ratio calculation unit3346. The luminance difference composition ratio calculation unit3345calculates the luminance difference composition ratio based on the first movement detection frame and outputs the luminance difference composition ratio to the composition ratio calculation unit3346. The composition ratio calculation unit3346supplies, for each area, one having a larger value of the luminance composition ratio and the luminance difference composition ratio as the final composition ratio to the combining processing unit3347. The combining processing unit3347combines the third combined frame and the over frame104based on the final composition ratio and outputs the combined frame as the fourth combined frame from an output terminal3348.

The processing is described in more detail below.

The luminance composition ratio calculation unit3344calculates the luminance composition ratio of the over frame104using the luminance of the third combined frame.FIG. 24is an example of a graph for calculating the luminance composition ratio. The processing is described below with reference toFIG. 24. The luminance composition ratio is calculated for each area in response to the luminance of the over frame104.

FIG. 24represents that the over frame104is used in an area darker than a luminance composition threshold value Y7, and the third combined frame is used in an area brighter than a luminance composition threshold value Y8to obtain the HDR composition image. In addition, in an intermediate area in boundaries Y7to Y8near the luminance composition threshold values, the composition ratio is gradually changed to smooth switching of images.

Next, the luminance difference composition ratio calculation unit3345is described.

The luminance difference composition ratio calculation unit3345calculates the luminance difference composition ratio of the over frame104with respect to the first movement detection frame.FIG. 25is an example of a graph for calculating the luminance difference composition ratio. The processing is described below with reference toFIG. 25.FIG. 25represents that the third combined frame is used in an area in which a value of the movement detection frame is less than a luminance difference composition threshold value d7, and the over frame104is used in an area in which a value of the movement detection frame is greater than a luminance difference composition threshold value d8. In addition, in an intermediate area in boundaries d7to d8near the luminance difference composition threshold values, the composition ratio is gradually changed to smooth switching of images.

Next, the composition ratio calculation unit3346is described.

The composition ratio calculation unit3346calculates the final composition ratio using the luminance composition ratio and the luminance difference composition ratio. In this regard, one having a larger value of the luminance composition ratio and the luminance difference composition ratio is calculated as a final composition ratio fg4 for each pixel.

Finally, the combining processing unit3347calculates combined image data of the fourth combined frame using the calculated final composition ratio based on a following formula.

Each term in the formula is as follows.
fg4: a composition ratio
FI4: image data of the fourth combined frame
OI4: image data of the over frame104
F13: image data of the third combined frame

The tone correction unit335corrects a tone curve using a LUT with respect to the third combined frame. Regarding the tone curve described here, the tone curve used inFIG. 12according to the first exemplary embodiment may be used to obtain the painterly effect on an image by enhancing contrast of a dark portion and a bright portion and reducing contrast of an intermediate luminance portion than that of the first exemplary embodiment for an effect of the gamma, and the tone curve may be linearized to obtain only an effect of the gamma. The configuration of the tone correction unit335is similar to that of the tone correction unit312according to the first exemplary embodiment, so that the description thereof is omitted.

Finally, the local contrast correction unit336performs processing for generating a halo near an edge having a large difference in brightness and darkness. The processing is also similar to that of the local contrast correction unit313according to the first exemplary embodiment, so that the description thereof is omitted.

As described above, the present exemplary embodiment can more enhance the painterly effect by applying the gamma which does not equalize brightness to each frame before the HDR composition in addition to the first exemplary embodiment. Further, the present exemplary embodiment can realize the moving object detection highly accurately by performing the movement detection between the proper frames of which brightness are matched with each other while realizing generation of the image as described above. The present exemplary embodiment is described using the movement detection as the example, however, it is needless to say that the present exemplary embodiment can be realized by processing which can be realized by only frames of which brightness are matched with each other, for example, position alignment processing between frames.

In the method described according to the first and the second exemplary embodiments, a frame rate is reduced lower than an image capturing frame rate, however, the processing can be realized without reducing the frame rate as much as possible by partly overlapping frames to be combined as illustrated inFIG. 26.

The image processing apparatus and the control method thereof according to the above-described first and second exemplary embodiments may be realized by a general-purpose information processing apparatus such as a personal computer and a computer program executed by the information processing apparatus.

FIG. 27is a block configuration diagram illustrating an information processing apparatus according to a third exemplary embodiment.

InFIG. 27, a central processing unit (CPU)900performs control of an entire apparatus and various types of processing. A memory901is constituted of a read-only memory (ROM) storing a basic input output system (BIOS) and a boot program and a random access memory (RAM) used by the CPU900as a work area. An instruction input unit903is constituted of a keyboard, a pointing device such as a mouse, and various switches. An external storage device904(for example, a hard disk device) provides an operating system (OS) necessary for the control of the present apparatus, a computer program according to the first exemplary embodiment, and a storage area necessary for calculation. A storage device905accesses a portable storage medium (for example, a Blu-ray disk read-only memory (BD-ROM) and a digital versatile disk read-only memory (DVD-ROM) disk) for storing moving image data. A bus902is used to exchange image data between a computer and an external interface.

A digital camera906captures an image and also obtains each speed which is an output from each speed sensor. A display907outputs a processing result, and a communication circuit909is constituted of a local area network (LAN), a public circuit, a wireless circuit, and an airwave. A communication interface908transmits and receives image data via the communication circuit909.

The information processing apparatus including the above-described configuration is described.

When a power source is input to the apparatus by the instruction input unit903before processing, the CPU900causes the external storage device904to load the OS to the memory901(RAM) according to the boot program (stored in the ROM) in the memory901. Further, an application program is loaded from the external storage device904to the memory901according to an instruction from a user, and thus the present apparatus functions as the image processing apparatus.FIG. 28illustrates a storage condition of the memory when the application program is loaded to the memory901.

The memory901stores the OS for controlling the entire apparatus and various types of software and video processing software for performing HDR composition and adding the painterly effect. The memory901further stores image input software for controlling the camera906to capture a proper frame, an under frame, a proper frame, and an over frame in this order and to input (capture) a frame one by one as a moving image. In addition, the memory901includes an image area for storing image data and a working area for storing various parameters.

FIG. 29is a flowchart illustrating video processing by an application executed by the CPU900.

In step S1, initialization is performed on each unit. In step S2, it is determined whether the program is terminated. The termination is determined based on whether a user inputs a termination instruction from the instruction input unit903.

In step S3, an image is input to the image area of the memory901by frame unit. In step S4, the HDR composition and the painterly effect addition are performed as the image processing, and the processing returns to step S2.

The image processing in step S4is described in detail using a flowchart inFIG. 30.

In step S401, a proper frame, an under frame, a proper frame, and an over frame which are at least temporally continuous in images stored in the storage device905and various parameters are stored in the memory901. In step S402, the WB coefficient is calculated using the proper frame. In step S403, the proper frame is developed using the WB coefficient calculated using the proper frame itself, and the other frames are developed using the WB coefficient calculated using the proper frame temporally previous thereto. In step S404, the luminance composition ratios are respectively calculated using the under frame and the over frame. In step S405, the luminance difference composition ratios are respectively calculated using the proper frame and the under frame, and the proper frame and the over frame. In step S406, the proper, under, and over frames are combined using the luminance composition ratio and the luminance difference composition ratio. In step S407, the tone correction is performed on the combined frame. Finally, in step S408, the local contrast correction processing is performed, and the calculated image frame is stored in the memory901.

As described above, the present exemplary embodiment can obtain an effect similar to that of the first exemplary embodiment in an image quality. The computer program is normally stored in a computer-readable storage medium, and the computer program can be executed by setting the computer-readable storage medium in a reading apparatus included in a computer and copying or installing to a system. Accordingly, it is obvious that the above-described computer-readable storage medium is included in the scope of the present invention.

Other Embodiments

This application claims the benefit of Japanese Patent Application No. 2017-073927, filed Apr. 3, 2017, which is hereby incorporated by reference herein in its entirety.