Patent Publication Number: US-8982232-B2

Title: Image processing apparatus and image processing method

Description:
BACKGROUND OF THE INVENTION 
     1. Field of the Invention 
     The present invention relates to a technology for expanding dynamic range by combining images. 
     2. Description of the Related Art 
     Recently, a large number of imaging apparatuses, such as digital cameras, capable of recording a combined image by combining a plurality of images have been developed as products. Some of these imaging apparatuses have a function for ameliorating overexposure or dark portion noise by combining an image captured at a higher exposure with an image captured at a lower exposure than the exposure at which the brightness of an object is appropriate. This function is called a high dynamic range (HDR) function. Generally, the dynamic range of an image sensor is narrower than the dynamic range in the natural world. For example, when capturing a backlit scene, overexposure can occur. 
     When capturing an image at a lower exposure, since the overall image is dark, although overexposure can be suppressed, the impression of dark portion noise deteriorates. In contrast, when capturing an image at a higher exposure, since the overall image is bright, although the overexposure occurs, the impression of dark portion noise is ameliorated. With the high dynamic function, the overexposure and the impression of dark portion noise can be ameliorated by combining a plurality of images having low exposures and high exposures, with weighting given to the underexposed images for bright regions (overexposure etc.), and conversely, weighting given to overexposed images for dark regions. 
     When the object moves while capturing the plurality of images, the object can turn into a double image due to combining processing. Consequently, a technology has been discussed for suppressing double images in regions in which object movement has been detected by prohibiting combining processing. Generally, a region in which object movement was detected outputs a low exposure image. This is because for images captured at a high exposure, a moving object can exhibit overexposure. 
     Japanese Patent Application Laid-Open No. 2002-190983 discusses a technology that, for a region in which object movement was detected, outputs an underexposed image and weakens the strength of contour enhancement to suppress the object from turning into a double image and suppress amplification of the impression of noise in the underexposed image. 
     As described above, when combining a plurality of images, double images can be suppressed by prohibiting the combining processing for regions in which movement was detected and outputting the underexposed image as is. However, there is the problem that for a region in which movement was detected, when an underexposed image is output, the impression of noise in the region in which movement was detected deteriorates. 
     Although further deterioration in the impression of noise due to noise amplification in a region in which object movement was detected can be prevented by outputting an underexposed image and weakening the strength of contour enhancement like in Japanese Patent Application Laid-Open No. 2002-190983, it is difficult to ameliorate the impression of noise in the underexposed image that was originally there. 
     SUMMARY OF THE INVENTION 
     The present invention is directed to, when combining a plurality of images, improving image quality by reducing noise in a region in which movement was detected. 
     According to an aspect of the present invention, an image processing apparatus capable of generating a combined image by combining a plurality of images, includes an acquisition unit configured to acquire a plurality of images including a first image having a first exposure, a second image having a second exposure that is higher than the first exposure, and at least one low-noise image having a lower noise than the first image, a moving region detection unit configured to detect moving regions in which a difference between the first image and the second image is greater than a predetermined value, a brightness detection unit configured to detect a brightness of the moving regions, a replacement region setting unit configured to set a replacement region from among the moving regions based on the brightness detected by the brightness detection unit, and a combining unit configured to combine the plurality of images, wherein the combining unit is configured to generate a combined image by combining the first image and the second image, and replace the replacement region in the combined image with an image of a region corresponding to the low-noise image. 
     Further features and aspects of the present invention will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the invention and, together with the description, serve to explain the principles of the invention. 
         FIG. 1  illustrates a configuration example according to a first exemplary embodiment. 
         FIG. 2  illustrates a configuration example of a moving region information generation unit. 
         FIGS. 3A ,  3 B, and  3 C illustrate moving region information. 
         FIG. 4  illustrates a configuration example of a replacement region information generation unit. 
         FIG. 5  illustrates a configuration example of an image combining unit. 
         FIG. 6  illustrates an operation flow for generating replacement region information. 
         FIGS. 7A to 7F  illustrate replacement region information. 
         FIG. 8  illustrates an operation flow for replacing with a low-noise image. 
         FIGS. 9A to 9E  illustrate replacement processing. 
     
    
    
     DESCRIPTION OF THE EMBODIMENTS 
     Various exemplary embodiments, features, and aspects of the invention will be described in detail below with reference to the drawings. 
       FIG. 1  is a block diagram illustrating a configuration example when an image processing apparatus according to the present exemplary embodiment is applied in an imaging apparatus. The configuration example according to the present exemplary embodiment will now be described with reference to  FIG. 1 . 
     The imaging apparatus according to the present exemplary embodiment is configured from a lens  101 , an imaging unit  102 , an analog/digital (A/D) conversion unit  103 , a camera signal processing unit  104 , a system control unit  105  (an underexposure calculation unit and an overexposure calculation unit), and a moving region information generation unit  106  (a moving region detection unit). In addition, the imaging apparatus has a replacement region information generation unit  107  (a replacement region setting unit), an image combining unit  108  (a combination standard image generation unit), a memory  109 , a tone mapping unit  110 , an image recording unit  111 , and an image display unit  112 . Next, operation of the imaging apparatus according to the present exemplary embodiment will be described. 
     First, the lens  101  guides object light to the imaging unit  102 . The imaging unit  102  captures an image of the object using an image sensor such as a charged-couple device (CCD) or a complementary metal-oxide-semiconductor (CMOS). The obtained object light is converted into an electric signal and output to the A/D conversion unit  103 . The A/D conversion unit  103  amplifies the analog signal based on a predetermined gain, and converts the amplified analog signal into a digital signal. 
     The camera signal processing unit  104  performs image processing, such as white balance and contour enhancement, on the digital signal output from the A/D conversion unit  103 , and outputs the resultant image data. In the present exemplary embodiment, since below-described combined image data having a wide dynamic range is generated based on post-development-processing luminance (Y) and color difference (UV) signals, the image processing, such as white balance and contour enhancement, is performed by the camera signal processing unit  104 . 
     However, the method for generating the combined image data is not limited to this. Specifically, a pre-development-processing RAW image format may be employed at the stage when the data is input into the image combining unit  108 . In this case, the camera signal processing unit  104  only performs pre-development processing such as pixel interpolation processing. 
     The system control unit  105  transmits instructions to the respective blocks in the camera, controls each block, and performs arithmetic processing. For example, the system control unit  105  controls exposure based on shutter speed and aperture according to the brightness of the image data output from the camera signal processing unit  104 . In the present exemplary embodiment, the imaging apparatus has a HDR imaging mode that generates image data having a wider dynamic range than single image data by combining image data obtained by imaging an object a plurality of times at different exposures based on control by the system control unit  105 . 
     The moving region information generation unit  106  detects a moving region and outputs moving region information by evaluating the absolute value of a pixel difference between image data output from the camera signal processing unit  104  and image data delayed by the memory  109 . 
     The replacement region information generation unit  107  determines a region to be replaced with a low-noise image in a below-described combination standard image, and outputs the replacement region information. The method for generating the replacement region information will be described in detail below. 
     The image combining unit  108  combines image data output from the camera signal processing unit  104  and image data delayed by the memory  109 . During this process, the image combining unit  108  determines a combining ratio based on the brightness of the image data output from the camera signal processing unit  104  and the moving region information generated by the moving region information generation unit  106 , combines the image data, and outputs the combined image data as a combination standard image to the memory  109 . Further, the image combining unit  108  combines the image data output from the camera signal processing unit  104  and the combination standard image delayed by the memory  109  based on replacement region information generated by the replacement region information generation unit  107  and moving region information generated by the moving region information generation unit  106 . The combined image data is output to the below-described tone mapping unit  110 . 
     The memory  109  is a memory for recording images, such as the combination standard image, output from the image combining unit  108 . 
     The tone mapping unit  110  performs tonal compression of the combined image data based on, for example, a respective characteristic curve for display and recording, by nonlinear processing such as gamma processing. 
     The image recording unit  111  compresses and codes the image that was tone-mapped by the tone mapping unit  110  based on a predetermined compression and coding method, and records the generated data in a recording medium. The image display unit  112  is configured from, for example, a liquid crystal panel. The image display unit  112  resizes the image that was tone-mapped by the tone mapping unit  110  to an image size for display, and displays the resized image. 
     Next, operation of the moving region information generation unit  106 , the replacement region information generation unit  107 , and the image combining unit  108  will be described in more detail. 
     (Moving Region Information Generation Unit  106 ) 
     The moving region information generation unit  106  detects a moving region and outputs moving region information by evaluating the absolute value of a pixel difference between image data output from the camera signal processing unit  104  and image data delayed by the memory  109 . Below, first an example of capturing an underexposed image will be described, and then an example of capturing an overexposed image will be described. The underexposed image is recorded in the memory  109  as is without subjecting it to image combining processing by the image combining unit  108 . The imaging order is not limited to capturing an underexposed image and then an overexposed image. Instead, the overexposed image may be captured first. 
       FIG. 2  illustrates a configuration example of the moving region information generation unit  106 . 
     A pixel level matching unit  201  equalizes (normalizes) the brightness of the underexposed image output from the memory  109  and the brightness of the overexposed image output from the camera signal processing unit  104 . Specifically, the pixel level matching unit  201  matches the brightness of the underexposed image with the brightness of the overexposed image either by amplifying a pixel value of the underexposed image or by reducing a pixel value of the overexposed image based on an exposure difference. 
     A pixel difference absolute value calculation unit  202  calculates the absolute value of a pixel difference for each individual pixel between the underexposed image and the overexposed image whose pixel levels have been matched by the pixel level matching unit  201 . In a moving region, an absolute value of this pixel difference increases. Conversely, in a still region, the absolute value of the pixel difference decreases. 
     A moving region grouping unit  203  forms groups of pixel difference absolute value among the pixel difference absolute values for the individual pixels calculated by the pixel difference absolute value calculation unit  202 , determines whether there is movement in each region, and outputs the resultant moving region information. 
       FIG. 3  schematically illustrates the processing that is performed by operation of the moving region information generation unit  106 .  FIG. 3A  illustrates an image captured at a low exposure,  FIG. 3B  illustrates an image captured at a high exposure, and  FIG. 3C  illustrates moving region information. Further,  FIGS. 3A and 3B  illustrate images having pixel levels that were matched by the pixel level matching unit  201 . In the moving region information illustrated in  FIG. 3C , the blackened portions have been determined to be moving regions. 
       FIG. 3  illustrates an imaging scene in which a bright rectangular object moves from left to right across a dark background. Consequently, the rectangular object that was on the left for the underexposed image in  FIG. 3A  moves toward the center for the overexposed image in  FIG. 3B . 
     The pixel difference absolute value calculation unit  202  calculates the pixel difference absolute value between the pixel value of the image obtained by normalizing the underexposed image in  FIG. 3A  and the pixel value of the overexposed image in  FIG. 3B  for each individual pixel. Since the rectangular object is moving, the pixel difference absolute value at that portion increases. Further, since the background portion, which is the portion other than the rectangular object, is not moving, the pixel difference absolute value decreases. 
     Moving region information illustrated in  FIG. 3C  is generated by the grouping of pixel difference absolute values by the moving region grouping unit  203 . The moving region information in  FIG. 3C  is determined to be a moving region because a region a 302 , which is the movement destination, and a region a 301 , which is the movement origin of the rectangular object, have a pixel difference of a larger absolute value. 
     Based on the above operations, the moving region information generation unit  106  detects a moving region and outputs moving region information. 
     (Replacement Region Information Generation Unit  107 ) 
     Next, the replacement region information generation unit  107  will be described in more detail with reference to  FIG. 4 . 
     A moving region brightness calculation unit  401  (brightness calculation unit) calculates the brightness in the moving region detected by the moving region information generation unit  106  and in a moving region peripheral region in an image output from the memory  109 . This brightness is, for example, the average luminance value of the individual pixels in each region. 
     A moving region picture determination unit  402  (picture evaluation unit) evaluates a picture in the moving region indicated by the moving region information and in a moving region peripheral region in an image output from the memory  109 . Specifically, for example, the moving region picture determination unit  402  calculates the variance of the luminance values of the moving region or the moving region peripheral region. If the variance is large, the moving region picture determination unit  402  determines that the picture is complex (has texture). If the variance is smaller than a predetermined value, the moving region picture determination unit  402  determines that the picture is smooth. 
     A replacement region determination unit  403  determines a region to be replaced with a low-noise image in the combination standard image, based on the brightness in the moving region and the moving region peripheral region calculated by the moving region brightness calculation unit  401 , and pictures in the moving region and the moving region peripheral region determined by the moving region picture determination unit  402 . The determined replacement region in the combined image is output as replacement region information. The method for determining the replacement region will be described in detail below. 
     (Image Combining Unit  108 ) 
     Next, the image combining unit  108  will be described in more detail with reference to  FIG. 5 . 
       FIG. 5  illustrates a configuration example of the image combining unit  108 . 
     A brightness calculation unit  501  calculates the brightness of an image output from the memory  109  for each pixel. This brightness is, for example, a luminance value. A combining ratio calculation unit  502  calculates a combining ratio for each region based on the brightness calculated by the brightness calculation unit  501  and the moving region information generated by the moving region information generation unit  106 . 
     Specifically, for bright regions, the combining ratio is set higher for an underexposed image than for an overexposed image, since an overexposed image is likely to have overexposure. For dark regions, the combining ratio is set higher for an overexposed image than for an underexposed image, since an underexposed image is likely to show a loss of shadow detail. Further, in a moving region, if a plurality of images is combined, the moving object can turn into a double image in the combined image. Consequently, the combining ratio of an underexposed image is set higher for a moving region than for a non-moving region. 
     The reason for setting the combining ratio of an underexposed image higher than for an overexposed image is because a moving object in an overexposed image may suffer from overexposure. A combining unit  503  combines for each region the image data output from the camera signal processing unit  104  and the image data output from the memory  109  based on the combining ratio calculated by the combining ratio calculation unit  502 . However, when replacement region information is output from the replacement region information generation unit  107 , the combining unit  503  only performs combining processing on the replacement region as the combination target region. Regions other than the replacement region are output as is without combining with the image output from the memory  109 . Such a combined image in which a replacement region remains is referred to here as a “combination standard image”. 
     Next, an operation flow in which the replacement region information generation unit  107  generates replacement region information will be described with reference to  FIGS. 6 and 7 . The operations in  FIGS. 6 and 7  are performed by the respective blocks based on instructions from the system control unit  105 . 
     Processing starts from step S 601  when the system control unit  105  detects that the imaging apparatus has entered an HDR imaging mode for generating an image having a wider dynamic range than apiece of captured image data. In step S 601 , the imaging apparatus is in a standby state waiting for a half press operation of a release switch that instructs the start and holding of exposure control, focus control and the like. When the release switch is half pressed, in step S 602 , the system control unit  105  determines the imaging conditions, including the determination of a plurality of different exposure values (a first exposure, a second exposure, and a low-noise image exposure) to be used for the proper imaging. 
     In step S 603 , the imaging unit  102  captures an underexposed image (a first image) at a low exposure value. This underexposed image is subjected to signal processing by the camera signal processing unit  104 , and is then temporarily recorded in the memory  109 . In step S 604 , the imaging unit  102  captures overexposed images (a second image and a low-noise image) at a higher exposure value than the exposure of the low exposure value. In step S 605 , the moving region information generation unit  106  inputs an overexposed image output from the camera signal processing unit  104  and the underexposed image output from the memory  109 , and as described above, generates moving region information based on the pixel difference absolute values. 
     In step S 606 , the image combining unit  108  combines an overexposed image output from the camera signal processing unit  104  and the underexposed image output from the memory  109  based on the moving region information and image brightness, outputs a combination standard image, and records the output combination standard image in the memory  109 . In step S 607 , the replacement region information generation unit  107  determines the region to be replaced with the low-noise image, and outputs the generated replacement region information. 
     The method for determining the replacement region will now be described in detail with reference to  FIG. 7 . Similar to  FIG. 3 ,  FIG. 7  illustrates an imaging scene in which a bright rectangular object moves from left to right across a dark background. Since  FIGS. 7A ,  7 B, and  7 D are the same as  FIGS. 3A ,  3 B, and  3 D, a description thereof will be omitted here. 
       FIG. 7C  illustrates a combination standard image in which the underexposed image of  FIG. 7A  and the overexposed image of  FIG. 7B  have been combined by the image combining unit  108 . The combination standard image in  FIG. 7C  has been combined such that, regarding the regions determined as being a moving region based on the moving region information in  FIG. 7D  (region a 301  and region a 302 ), the underexposed image of  FIG. 7A  has a higher combining ratio. On the other hand, regarding the regions that were not determined as being a moving region based on the moving region information in  FIG. 7D  (regions other than region a 301  and region a 302 ), the combining ratio of the overexposed image is set higher since the background region is dark in both the underexposed image of  FIG. 7A  and the overexposed image of  FIG. 7B . 
     The thus-combined combination standard image of  FIG. 7C  mixes regions having a high combining ratio of an overexposed image and regions having a high combining ratio of an underexposed image. Since an underexposed image has a worse signal-to-noise ratio (S/N) than an overexposed image, in the combination standard image of  FIG. 7C , a difference in the impression of noise tends to be especially noticeable between regions a 301  and a 302  and their peripheral regions. 
     The moving region brightness calculation unit  401  and the moving region picture determination unit  402  perform brightness calculation and picture evaluation of the moving regions (region a 701  and region a 702 ) and the moving region peripheral regions (region a 703  and region a 704 ) indicated by the moving region information of  FIG. 7D  in the replacement determination region in  FIG. 7E . 
     The replacement region determination unit  403  sets a moving region having a moving region brightness equal to or less than a predetermined value, and a difference between the brightness of the moving region and the moving region peripheral region equal to or less than a predetermined value, as a replacement region. 
     Alternatively, the replacement region determination unit  403  sets a moving region having a moving region brightness equal to or less than a predetermined value, and where a picture of the moving region and the moving region peripheral region is determined to be smooth, as a replacement region. 
     This is because poor S/N tends to be visually more noticeable in dark regions than in bright regions. Further, poor S/N tends to be visually more noticeable in smooth picture regions than in complex picture regions. In the example illustrated in  FIG. 7 , when the brightness of the combination standard image of the moving regions a 701  and a 702  is compared with a predetermined value (threshold), the brightness of the region a 702  is less (the region is darker) than the predetermined value. In addition, the brightness difference in the combination standard image of the moving region a 702  and the moving region peripheral region a 704  is smaller than the predetermined value. In such a case, as illustrated in  FIG. 7F , the region a 705  (region that was a 702 ) is determined to be a replacement region. 
     Similarly, in  FIG. 7 , the combination standard image picture in the moving region a 702  and the moving region peripheral region a 704  are both smooth, and the moving region a 701  and the moving region peripheral region a 703  exceed the standard value (are not smooth). In this case too, as illustrated in  FIG. 7F , the region a 705  is determined to be a replacement region. 
     When emphasis is placed on a image quality, all moving regions can be determined to be replacement regions without narrowing down the moving regions to determine replacement regions from among all the moving regions indicated by the moving region information. However, the more replacement regions there are, the greater the number of times image capturing has to be performed to acquire a low-noise image of the replacement region. Therefore, when emphasis is placed on a reduction in the overall imaging processing time, it is desirable to narrow down the replacement regions based on the two determination methods described above. 
     Based on such an operation flow, the replacement region information generation unit  107  generates replacement region information. 
     Next, an operation flow in which the image combining unit  108  replaces the combination standard image with a low-noise image based on replacement region information will be described with reference to  FIGS. 8 and 9 . 
     In step S 801 , the imaging unit  102  captures a low-noise image. This low-noise image is an image that at least has less noise or higher signal-to-noise ratio than the underexposed image captured to generate the combination standard image. Examples include an image captured at a higher exposure than the underexposed image generated by changing an exposure time or F number and an image captured at a lower amplification rate than the amplification rate (ISO sensitivity) of the A/D conversion unit  103  when the underexposed image is captured. 
     In step S 802 , the moving region information generation unit  106  inputs the low-noise image output from the camera signal processing unit  104  and the combination standard image output from the memory  109 , and generates moving region information. In step S 803 , the image combining unit  108  combines the combination standard image and the low-noise image based on the moving region information and the replacement region information. 
     The combining method performed by the image combining unit  108  will be described with reference to  FIG. 9 . 
       FIG. 9  is a continuation of the imaging scenes of  FIGS. 3 and 7 , in which a bright rectangular object moves from left to right across a dark background.  FIGS. 9A and 9E  illustrate the same combination standard image and replacement region information as  FIGS. 7C and 7F .  FIG. 9B  illustrates a low-noise image.  FIG. 9C  illustrates a combination image that combines the combination standard image and the low-noise image. Further,  FIG. 9D  illustrates moving region information generated by the moving region information generation unit  106  in step S 802 , in which region a 901  and region a 902  having a pixel difference of a large absolute value between the combination standard image of  FIG. 9A  and the low-noise image of  FIG. 9B  are determined to be moving regions. 
     The image combining unit  108  sets the replacement region a 903  indicated by the replacement region information in  FIG. 9E  generated by the replacement region information generation unit  107  as a combination target region, and the combination standard image of  FIG. 9A  is output as is without performing combining processing on the regions other than the replacement region a 903 . As indicated by the moving region information of  FIG. 9D , the replacement region a 903  indicated by the replacement region information of  FIG. 9E  includes a part of the moving region a 902  and the region a 904 , which is not a moving region. 
     Of the replacement region a 903 , the image combining unit  108  combines the region a 904 , which is not a moving region, with the low-noise image set at a high combining ratio, and the moving region is output with the combination standard image of  FIG. 9A  as is. Specifically, since a combined image is output in which, in the combination standard image, a region corresponding to the replacement region and a region that is not a moving region are replaced with a low-noise image, the combined image has a better S/N than the combination standard image has. 
     In calculating the brightness with the brightness calculation unit  501 , when the low-noise image replacement region and the region that is not a moving region is overexposed or underexposed, the combination standard image does not have to be replaced with the low-noise image. 
     Returning to  FIG. 8 , in step S 804 , the replacement region information is updated by removing the region replaced with the low-noise image in step S 803  from the replacement region information. 
     A combined image having an improved combination standard image S/N can be generated by repeating steps S 801  to S 804 . This repeated processing finishes when it is determined in step S 805  that any of the following ending conditions has been met.
         Replacement of all replacement regions is completed.   A predetermined time has elapsed since the start of replacement processing.   Instruction is issued to stop replacement processing from the user.       

     During the period from starting the imaging to finishing the replacement processing, the image does not have to be displayed on the image display unit  112 . This is because by not displaying the image, the user realizes that imaging is not completed, so that the user continues imaging while still holding the imaging apparatus. 
     In step S 806 , the tone mapping unit  110  performs tone mapping on the combined image. In step S 807 , the tone-mapped combined image is recorded in the image recording unit  111 . 
     In the present exemplary embodiment, although an example was described in which a low-noise image is captured after generating a combination standard image, the low-noise image can be captured before generating the combination standard image. In such a case, the low-noise image is recorded in the memory  109 , and is used by the image combining unit  108  for image combination. 
     Other Embodiments 
     Aspects of the present invention can also be realized by a computer of a system or apparatus (or devices such as a CPU or MPU) that reads out and executes a program recorded on a memory device to perform the functions of the above-described embodiment (s), and by a method, the steps of which are performed by a computer of a system or apparatus by, for example, reading out and executing a program recorded on a memory device to perform the functions of the above-described embodiment(s). For this purpose, the program is provided to the computer for example via a network or from a recording medium of various types serving as the memory device (e.g., computer-readable medium). 
     While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all modifications, equivalent structures, and functions. 
     This application claims priority from Japanese Patent Application No. 2011-152959 filed Jul. 11, 2011, which is hereby incorporated by reference herein in its entirety.