Patent Description:
Conventionally, a technique for high dynamic range compositing processing for compositing a plurality of images captured with different exposure amounts is known. According to this technique, it is possible to obtain an image without overexposure or underexposure by joining appropriate exposure signals of each image. Weighted addition ratios for when compositing a plurality of images of different exposures are often determined based on luminance values. For example, <NUM>% of an underexposed image without overexposure is used for a bright portion, <NUM>% of an overexposed image without underexposure is used for a dark portion, and <NUM>% of an appropriate image is used for portions near an appropriate exposure in a central portion of luminance. In an intermediate region of a range between a dark portion and an appropriately exposed portion, a usage ratio between an overexposed image and an appropriate image are determined and the images are composited. It is similar for an intermediate region of a range and the like between an appropriately exposed portion and a bright portion. A mix table in which a usage ratio for each image is thus defined for each brightness is held in advance, and the images are composited using the table.

In recent years, with improvement in sensor performance and the like, an image capturing device (Dual Gain Output (DGO)) including two column circuits for an output signal from a unit pixel, separately holding gains of amplifiers in the column circuits, and capable of outputting images of different gains has been used. The image capturing device can output two images of different gains (high gain and low gain images) with a single exposure. Compositing two images according to DGO has advantages, such as processing for aligning the two images being unnecessary and good compositing being possible for a moving subject. Accordingly, DGO is well suited to high dynamic range (HDR) compositing for obtaining an image with an expanded dynamic range.

An image capturing device includes floating diffusions (FDs), each serving a role of a capacitor for storing electric charge. The FD changes in capacitance for electric charge that it can hold depending on the setting. When the capacitance is increased, the FD can handle more light, and so, at low sensitivity, the FD is used with an increased capacitance. However, when the capacitance is increased, noise increases, and the effect of the noise appears in a composite image. In response to this, <CIT>, discloses a method in which when compositing RAW image data, noise is reduced according to an ISO sensitivity and compositing ratios of the images.

However, in the prior art disclosed in the aforementioned document, compositing coefficients are outputted based on the ratios at which the images are used when compositing. That is, the document basically targets images of different sensitivities but of the same exposure and does not mention anything about reducing noise occurring when compositing images of different brightnesses.

With the above advantages, compositing according to DGO allows known image processing to be performed by compositing RAW images; however, in a case of exposure according to DGO, when a high gain image and a low gain image are generated, noise is present due to an exposure difference and the noise worsens as the exposure difference increases; therefore, the effect of the noise appears in image compositing.

Document <CIT> discloses an image pickup apparatus that enables amplifying a signal based on output of a photoelectric conversion unit, with a plurality of gains to obtain an image having a better S/N ratio in accordance with gain settings in. An image pickup element provided in an image pickup apparatus has a photoelectric conversion unit in which unit pixels are arranged in a matrix and a signal voltage is generating by photoelectric conversion. A column amplifier of the image pickup element can amplify the signal voltage photoelectrically converted with a plurality of gains. An image pickup element control unit performs drive control of the image pickup element to perform gain settings. An image composition unit performs image composition by using a plurality of image signals having different gains. When the image pickup element changes an amplification factor of the column amplifier to output image signals with a plurality of gains, the image pickup element control unit changes a value of a second gain that is smaller than that of a first gain among the gains and performs control so as not to change the value of the first gain.

Document <CIT> discloses a camera capable of easily performing high-quality photographing, without using strobe light emission. The camera calculates exposure index, corresponding to the luminance of a subject measured by a photometer section and exerts exposure control based upon the exposure index. When the camera has a film sensitivity equal to or higher than the sensitivity indicated by ISO <NUM>, the exposure control is exerted using an exposure index obtained by making prescribed correction to a regular exposure index.

The present invention has been made in view of such problems and provides a technique for high dynamic range compositing processing in which noise is reduced in image compositing for when exposure is largely different.

The present invention in its first aspect provides an image processing apparatus as specified in claims <NUM> to <NUM>.

The present invention in its second aspect provides a control method of an image processing apparatus as specified in claim <NUM>.

The present invention in its third aspect provides a computer program as specified in claim <NUM>.

According to the present invention, it is possible to generate, from two images having an exposure difference, a composite image in which a noise amount is reduced.

<FIG> is a block diagram for when an image processing apparatus in a first embodiment is applied to an image capturing apparatus typified by a digital camera or the like.

The image capturing apparatus includes an optical lens <NUM>, an image capturing device <NUM>, an image acquiring unit <NUM>, an image compositing unit <NUM>, a signal processing unit <NUM>, an exposure control unit <NUM>, an image capturing device control unit <NUM>, a control unit <NUM>, and a UI <NUM>. The control unit <NUM> is responsible for controlling the entire apparatus and is configured by a processor, a ROM for storing programs to be executed by the processor and various parameters, and a RAM to be used as a working area. The UI <NUM> is configured by various buttons and keys typified by a shutter button, a display unit, and a touch panel provided in front of the display unit and functions as a user interface between a user and the image capturing apparatus.

The optical lens <NUM> captures light of a subject and forms an image of the subject on an image capturing surface of the image capturing device <NUM>. The image capturing device <NUM> converts the optical image formed on the image capturing surface into electric signals and outputs the electric signals. Typical examples include a charge coupled device (CCD) image sensor, a CMOS image sensor, and the like. Some image capturing devices directly output analog video signals while others perform analog-digital conversion processing and output digital data, such as low voltage differential signaling (LVDS). In addition, the image capturing device <NUM> in the embodiment allows output of image data of two different gains by amplifying each of the electric signals obtained by a single exposure with two types of gains.

Here, a circuit configuration of the image capturing device <NUM> used in the embodiment is illustrated in <FIG>.

A timing pulse control unit <NUM> controls the operation of the image capturing device <NUM> by supplying an operation clock signal CLK for each block of the image capturing device <NUM> and by supplying a timing signal to each block.

A vertical scanning circuit <NUM> performs timing control for sequentially reading out, in one frame, voltages represented by pixel signals accumulated by a pixel unit <NUM>. Generally, video signals are sequentially read out line by line from a top line to a bottom line in one frame.

The pixel unit <NUM> includes a plurality of photoelectric conversion devices arranged in a matrix. Each of the photoelectric conversion devices generates an electrical signal corresponding to an amount of incident light and outputs the electric signal. In the present embodiment, the pixel unit <NUM> converts captured light into charge and accumulates the charge in capacitor floating diffusions (FDs). A capacitance of an FD can be changed between large and small, and signal noise is improved by changing the capacitance according to ISO sensitivity. Basically, the FD is used with the capacitance being set to large for a low ISO sensitivity and to small for a high ISO sensitivity. When images of two different gains are outputted, which will be described later, the capacitance for accumulating charge is common to the two gains. In addition, the capacitance is not limited to two types, large and small, and a setting of three or more levels may be possible.

A column amplifier <NUM> is used for electrically amplifying signals read out from the pixel unit <NUM>. By amplifying the signals in the column amplifier <NUM>, signal levels of pixels are amplified with respect to the noise to be outputted by a subsequent column ADC <NUM>, thereby equally improving signal noise. Further, a structure is such that a gain of the column amplifier <NUM> can be changed by the timing pulse control unit <NUM>. The image capturing device <NUM> includes two input memories in the column amplifier unit <NUM> for generating a high dynamic range (HDR) image and can output two types of gains by changing the gain of the column amplifier. By including two input memories, it is possible to apply two gains to signals of a certain time, read out from the FDs, and output the signals, and so, although the amount of data is increased, it is possible to obtain simultaneous images with two different gains. Although it is assumed that the present invention has two types of output of the image capturing device <NUM> of the embodiment, there is no particular limitation on the number of types of simultaneous output.

The column ADC <NUM> converts signals from the column amplifier <NUM> from analog to digital. Then, the column ADC <NUM> supplies digitalized signals (digital image data) to a horizontal transfer circuit <NUM>. Horizontal transfer circuit <NUM> outputs line-by-line digital image data to the signal processing circuit <NUM>. The signal processing circuit <NUM> is a circuit for digitally performing signal processing and can add an offset value of a certain amount in digital processing as well as easily perform gain calculation by performing shift calculation and multiplication. The signal processing circuit <NUM> passes the processed image data to an external output circuit <NUM>. The external output circuit <NUM> supplies the image data as image data to an external device (image acquiring unit <NUM>).

The description of the configuration of <FIG> is returned to. The image acquiring unit <NUM> captures image data outputted from the image capturing device <NUM>. When the image capturing device <NUM> does not perform AD conversion and outputs analog signals, the above-described analog-to-digital conversion is performed in the image acquiring unit <NUM>.

The image compositing unit <NUM> composites image data (image data with two types of gains) for generating an HDR image, outputted from the image capturing device <NUM>, into an HDR image by using a predetermined compositing method. An example is a compositing method in which a high gain image (hereinafter, referred to as an H image) is used for a dark portion, a low gain image (hereinafter, referred to as an L image) is used for a bright portion, and a usage ratio of each image for weighted addition of the H image and the L image is determined for an intermediate region between, for example, a dark portion and a bright portion. In the present embodiment, a compositing algorithm is not limited so long as it is a technique for compositing two image of different gains.

The image compositing unit <NUM> generates compositing ratio information allowing a compositing ratio between the H image and the L image for each pixel at the time of compositing processing to be identified and attaches the generated compositing ratio information to a composite image. In the present embodiment, a compositing ratio is generated for each pixel; however, information may be information holding compositing information of only a mixed region, and the compositing information is not limited so long as the compositing ratio can be determined.

The signal processing unit <NUM> performs a pixel addition function as a typical image processing function of the image capturing apparatus and performs various kinds of image processing, such as noise reduction, gamma correction, knee correction, digital gain, defect correction, and the like. The image acquiring unit <NUM> and the signal processing unit <NUM> also include a storage circuit for storing setting values necessary for respective corrections and image processing, not explicitly illustrated in the block diagram.

A signal storage unit <NUM> stores an HDR composite image and video signals received from the image compositing unit <NUM> and the signal processing unit <NUM> in a storage medium (not illustrated). The storage medium may be a storage apparatus, such as an HDD; a non-volatile memory card, such as an SD card; or the like, and a type thereof does not matter.

The exposure control unit <NUM> can calculate an exposure amount from video signal information received from the image acquiring unit <NUM>. The exposure control unit <NUM> in the present embodiment acquires the exposure amount with a method of calculating the exposure amount from the video signal information but may acquire the exposure amount from a control exposure at the time of image capturing. The exposure control unit <NUM> determines the operation of the image capturing device control unit <NUM> based on information related to the calculated exposure amount and transmits a parameter thereof to the image capturing device control unit <NUM>.

Next, the operation of the image capturing device and the image compositing unit <NUM> at the time of HDR image generation will be described.

<FIG> is a block diagram of the image compositing unit <NUM> and illustrates a configuration until compositing processing in which the H image and the L image for which two types of gains have been applied, outputted from the image acquiring unit <NUM>, are used. The processing equivalent to the illustration may be implemented by hardware or by a processor executing a control program.

An exposure correcting unit <NUM> causes the gains of the inputted H image and L image to be the same. This is to cause output for an input signal to be linear after the images have been composited into a single image. <FIG> are graphs in which an amount of input light of the H image and an amount of input light of the L image are on a horizontal axis, and an output code after AD conversion is on a vertical axis. Candela (cd) is mainly used for units of the amount of light of the horizontal axis.

<FIG> illustrates a graph of the H image (bold line) and the L image (fine line). In this state, the brightnesses of the two images are different, and the images cannot be composited as they are.

Therefore, as illustrated in <FIG>, the exposure correcting unit <NUM> applies a gain to the L image for the brightnesses of the H image and the L image to match. <FIG> is an example in which the brightness of the L image has been adjusted to that of the H image. An exposure-corrected image for which the brightness has been adjusted to that of the H image by exposure correction on the L image is hereinafter referred to as a low gain <NUM> image (hereinafter referred to as an L2 image). A gain for obtaining the L2 image from the L image depends on the exposure difference between the H image and the L image. The exposure difference depends on exposure parameters (such as a shutter speed and an aperture) at the time of image capturing for obtaining an HDR composite image.

<FIG> illustrates an example of table information (assumed to be stored in the ROM in the control unit <NUM>) in which a relationship between the exposure difference and a shift amount of the correction is defined. The exposure difference in the illustrated table uses the H image as a reference and indicates a difference in exposure of the L image from that of the H image. As illustrated in the drawing, the larger the exposure difference, the larger the shift amount of the correction. The exposure correcting unit <NUM> generates the L2 image by correcting the L image based on a shift amount determined by referring to the table such that the exposure is the same as that of the H image. For example, when the exposure difference between the H image and the L image at the time of image capturing by the image capturing device <NUM> is "-<NUM> EV", the exposure correcting unit <NUM> generates the L2 image by performing exposure correction according to a gain shift amount of "+<NUM> levels" for the L image.

It should be noted that at the time of L2 image generation, when the exposure difference between the H image and the L image is large, the gain correction amount will also be large, and simultaneously, the amount of noise will increase. When the amount of noise increases, the image quality of the generated HDR images naturally deteriorates.

Therefore, in the present embodiment, when the exposure difference is a preset threshold or more or when the gain shift amount is a threshold or more, a noise correcting unit <NUM> performs correction for noise reduction on the L2 image. For example, the noise correcting unit <NUM> performs filter processing in which an average value (or a weighted average value) of a pixel of interest to be corrected (one of R, G, and B pixels of a Bayer array) in the L2 image and four pixels closest to the pixel of interest and of the same color is used as a corrected pixel value of the pixel of interest. Consequently, when a luminance value of the pixel of interest is extremely high (or low) with respect to its surrounding due to a lot of noise being superimposed on the pixel of interest, a value thereof can be made a natural value, thereby realizing noise reduction. A size of a filter may be appropriately set or may be selected by the user.

Now, the compositing processing unit <NUM> generates HDR image data by performing compositing processing using the H image and the L2 image received via the noise correcting unit <NUM> and outputs the HDR image data.

Here, detailed processing of the compositing processing unit <NUM> will be described below.

First, a value of a pixel at coordinates (x, y) of the HDR image to be outputted is expressed as PHDR(x, y), a value of a pixel at coordinates (x, y) of the H image is expressed as PH(x, y), and a value of a pixel at coordinates (x, y) of the L2 image is expressed as PL2(x, y).

A threshold for determining a highlight region is defined as T_H, and a threshold for determining a dark region is defined as T_L.

When a value of a pixel is larger than the threshold T_H, that pixel is determined to be a pixel of the highlight region. When a value of a pixel is smaller than the threshold T_L, that pixel is determined to be a pixel of the dark region. When a value of a pixel is the threshold T_L or more and the threshold T_H or less, that pixel is determined to be that of the intermediate region.

The compositing processing unit <NUM> determines a value of each pixel of the HDR image according to the following.

In the present embodiment, a method is such that brightness adjustment and noise correction are performed and compositing is performed; however, when the processing order is reversed, similar processing is possible by using for a noise correction amount a correction amount in which a gain of brightness adjustment is considered, and so, the order is not limited.

A second embodiment will be described. In the second embodiment, a case in which a photographer acquires two images in a single instance of image capturing by using the image capturing device <NUM> capable of outputting a plurality of images on which different gains have been applied and composites the images will be described. A configuration in the second embodiment is the same as the configuration of the first embodiment illustrated in <FIG>.

In the following, the processing of the control unit <NUM> will be described with reference to a flowchart of <FIG>. Assume that the processing illustrated in <FIG> is executed when the user selects an HDR compositing mode by operating the UI <NUM> and presses a shutter button.

In step S601, the control unit <NUM> determines an ISO sensitivity of the H image and the L image at the time of image capturing in the image capturing device <NUM>, based on the current image capturing parameters. A method of determining the ISO sensitivity may be such that the ISO sensitivity is set manually by the photographer or determined automatically by the camera. In the present embodiment, a method is such that a gain is set according to the ISO sensitivity; however, the gain may be set according to exposure time. Next, in step S602, the control unit <NUM> causes the image capturing device control unit <NUM> to perform image capturing and generate two images (the H image and the L image) with different gains by controlling the image capturing device control unit <NUM>.

Next, in step S603, the control unit <NUM> obtains a gain shift amount from an ISO sensitivity difference (exposure difference) between the H image and the L image. The control unit <NUM> controls the exposure correcting unit <NUM> to apply the obtained gain shift amount to the L image and generate the L2 image in which the brightness has been matched to that of the H image.

Next, in step S604, the control unit <NUM> determines whether the gain shift amount based on the exposure difference acquired in step S603 is a threshold (such as two levels) or more. When it is determined that the gain shift amount based on the exposure difference acquired in step S603 is the threshold (such as two levels) or more, the control unit <NUM> advances the processing to step S605, and when it is determined that the amount is less than the threshold, the control unit <NUM> advances the processing to step S606.

In step S605, the control unit <NUM> controls the noise correcting unit <NUM> to execute noise correction processing on the L2 image.

In step S606, the control unit <NUM> controls the compositing processing unit <NUM> to composite the H image and the L2 image (when the processing has gone through step S605, the L2 image on which noise correction processing has been performed) and generate an HDR composite image. Then, in step S606, the control unit <NUM> generates compositing information and adds it to the HDR image.

Claim 1:
An image processing apparatus (<NUM>-<NUM>, <NUM>, <NUM>) operable to process first image data and second image data obtained from image capturing means (<NUM>) that is capable of amplifying, with a first gain and a second gain being lower than the first gain, a signal obtained by photoelectric conversion to obtain the first image data and the second image data, respectively, the apparatus comprising:
exposure correcting means (<NUM>) for applying a gain to the second image data so that the brightness of the second image matches the brightness of the first image; whereby said gain applied to the second image data is based on the exposure difference between first and second image data;
noise correcting means (<NUM>) for applying a noise reduction process to the second image data of which brightness having been corrected; and
generating means (<NUM>) for generating composite image data by compositing the second image data having been applied the noise reduction process by the noise correcting means and the first image data.