IMAGE PROCESSING DEVICE, IMAGE PROCESSING METHOD, AND SOLID-STATE IMAGING DEVICE

According to an embodiment, a high dynamic range synthesizing unit synthesizes first image signal and second image signal. A main control exposure value calculating unit calculates a main control exposure value based on a signal designated as a main control signal between the first image signal and the second image signal. A sub-control exposure value calculating unit multiplies the main control exposure value by a high dynamic range magnification and sets the multiplication result as a sub-control exposure value for a sub-control signal. The sub-control signal causes the main control signal to follow lightness adjustment.

DETAILED DESCRIPTION

In general, according to an embodiment, an image processing device includes a high dynamic range synthesizing unit, an exposure value calculating unit, and a control amount converting unit. The high dynamic range synthesizing unit generates a synthesized image by synthesizing first image signal and second image signal. The first image signal is an image signal in accordance with an amount of light incident on a first pixel during a first charge accumulation period. The second image signal is an image signal in accordance with an amount of light incident on a second pixel during a second charge accumulation period. The second charge accumulation period is shorter than the first charge accumulation period. The exposure value calculating unit calculates an exposure value to which a lightness adjustment amount is reflected. The lightness adjustment amount is an adjustment amount used to adjust the lightness of the synthesized image in accordance with illuminance at the time of shooting. The control amount converting unit converts the exposure value into control amounts for an electronic shutter time, an analog gain, and a digital gain. The exposure value calculating unit includes a main control exposure value calculating unit and a sub-control exposure value calculating unit. The main control exposure value calculating unit calculates a main control exposure value based on a signal designated as a main control signal between the first image signal and second image signal. The main control exposure value is the exposure value for the main control signal. The sub-control exposure value calculating unit calculates a sub-control exposure value. The sub-control exposure value is the exposure value for a sub-control signal. The sub-control signal is one of the first image signal and second image signal other than the main control signal. The sub-control signal causes the main control signal to follow lightness adjustment. The sub-control exposure value calculating unit multiplies the main control exposure value calculated by the main control exposure value calculating unit by a high dynamic range magnification and sets the multiplication result as the sub-control exposure value. The high dynamic range magnification is set in advance as a ratio of the first charge accumulation period to the second charge accumulation period.

Hereinafter, an image processing device, an image processing method, and a solid-state imaging device according to embodiments will be described in detail with reference to the attached drawings. Further, the invention is not limited to the embodiments.

FIG. 1is a block diagram illustrating an overall configuration of a solid-state imaging device according to a first embodiment.FIG. 2is a block diagram illustrating an overall configuration of a digital camera including the solid-state imaging device illustrated inFIG. 1.

A digital camera1includes a camera module2and a post stage processing unit3. The camera module2includes an imaging optical system4and a solid-state imaging device5. The post stage processing unit3includes an image signal processor (ISP)6, a storage unit7, and a display unit8. The camera module2is applied not only to the digital camera1but also to, for example, an electronic device such as a camera-attached portable terminal.

The imaging optical system4acquires light from a subject and forms a subject image. The solid-state imaging device5captures the subject image. The ISP6performs signal processing on an image signal obtained through the imaging performed by the solid-state imaging device5. The storage unit7stores an image subjected to the signal processing by the ISP6. The storage unit7outputs the image signal to the display unit8in response to a user's operation or the like. The display unit8displays the image according to the image signal input from the ISP6or the storage unit7. The display unit8is, for example, a liquid crystal display.

The solid-state imaging device5is, for example, a complementary metal oxide semiconductor (CMOS) image sensor. The solid-state imaging device5may be a charge coupled device (CCD) as well as the CMOS image sensor. The solid-state imaging device5includes a pixel array10, a preprocessing unit11, an imaging processing circuit12, an interface (I/F)14, a timing generator15, and an auto exposure (AE) control circuit16.

In the pixel array10, the light acquired by the imaging optical system4is converted into a signal charge by a photodiode to capture a subject image. For example, the pixel array10generates an analog image signal by acquiring signal values of respective color components of red (R), green (G), and blue (B) in the order corresponding to a Bayer array.

The preprocessing unit11performs correlated double sampling, an analog gain (AG) and a digital gain (DG) amplification, analog-to-digital conversion (AD conversion) on the image signal from the pixel array10.

The imaging processing circuit12performs various kinds of signal processing on the digital image signal input from the preprocessing unit11. The imaging processing circuit12includes a high dynamic range (HDR) synthesizing unit13. The HDR synthesizing unit13performs HDR synthesis on the digital image signal input to the imaging processing circuit12to generate a synthesized image. The imaging processing circuit12performs not only the HDR synthesis by the HDR synthesizing unit13but also signal processing such as defect correction, noise reduction, shading correction, and white balance adjustment.

The I/F14outputs the image signal subjected to the signal processing by the imaging processing circuit12. The I/F14performs a process of transmitting the image signal to an external device, for example, appropriately performs conversion from serial data to a parallel output or conversion from an parallel input to serial data.

The AE control circuit16controls the AE operation of the digital camera1according to lightness at the time of shooting. The AE control circuit16transmits data of the AG and the DG to the preprocessing unit11. The AE control circuit16transmits data of an electronic shutter time (ES) to the timing generator15. The imaging processing circuit12and the AE control circuit16function as an image processing device. The timing generator15outputs a pulse used to drive the pixel array10.

FIG. 3is a diagram illustrating the array of pixels in a pixel array. The pixel array10is installed in as a Bayer array of four Gr, R, Gb, and B pixels. The R pixel detects red light. The B pixel detects blue light. The Gr and Gb pixels detect green light. The Gr pixel is parallel to the R pixel in a horizontal line. The Gb pixel is parallel to the B pixel in a horizontal line.

In the pixel array10, charge accumulation periods are set to be alternately different for each line area including two horizontal lines of a Gr/R line and a B/Gb line. A first charge accumulation period which is a charge accumulation period of a long-time exposure line area (first line area)17is longer than a second charge accumulation period which is a charge accumulation period of a short-time exposure line area (second line area)18.

The long-time exposure line area17includes two horizontal lines formed by long-time exposure pixels which are first pixels. The short-time exposure line area18includes two horizontal lines formed by short-time exposure pixels which are second pixels. The long-time exposure line area17and the short-time exposure line area18are alternately disposed in the vertical direction.

The long-time exposure pixel detects the amount of incident light during the first charge accumulation period. The short-time exposure pixel detects the amount of incident light during the second charge accumulation period. The pixel array10outputs a long-time exposure image signal (a first image signal) according to the amount of incident light on the long-time exposure pixels during the first charge accumulation period and a short-time exposure image signal (a second image signal) according to the amount of incident light on the short-time exposure pixels during the second charge accumulation period. The HDR synthesizing circuit13synthesizes the long-time exposure image signal and the short-time exposure image signal input to the imaging processing circuit12.

FIG. 4is a diagram illustrating output characteristics of the long-time exposure pixel and the short-time exposure pixel and synthesis of the image signals by the HDR synthesizing circuit. In the long-time exposure pixel, when the amount of incident light is higher than a predetermined saturated light amount I0, a signal charge generated through photoelectric conversion reaches an accumulation capacitance of a photodiode.

When the amount of incident light is equal to or less than the saturated light amount I0, the signal level of a long-time exposure image signal S1increases in proportion to an increase in the amount of incident light. When the amount of incident light is greater than the saturated light amount I0, the signal level of the long-time exposure image signal S1is constant. Even when the amount of incident light is greater than the saturated light amount I0, the signal level of the short-time exposure image signal S2increases in proportion to an increase in the amount of incident light.

The HDR synthesizing unit13multiplies the short-time exposure image signal S2by a predetermined HDR magnification to cause the output level of the long-time exposure pixel to coincide with the output level of the short-time exposure pixel. The HDR magnification corresponds to an exposure ratio which is a ratio of the first charge accumulation period of the long-time exposure pixel to the second charge accumulation period of the short-time exposure pixel. The HDR synthesizing unit13generates a synthesized image signal S through an interpolation process using the long-time exposure image signal S1and the short-time exposure image signal S2multiplied by the HDR magnification.

FIG. 5is a diagram illustrating control of the AE operation performed by the AE control circuit. The vertical axis of an illustrated graph represents an adjustment amount of a signal level with respect to incident light. The AE control circuit16causes the adjustment amount of the signal level to be variable through adjustment of the amount of charge accumulated according to the ES and an amplification ratio of the signal level according to the AG and the DG.

The horizontal axis of the illustrated graph represents illuminance. The illuminance is assumed to be lowered from the left to the right of the horizontal axis direction. The AE control circuit16increases the adjustment amount of the signal level because a signal level increases as the illuminance is lowered at the time of shooting. In the drawing, a portion indicated by a tone represents an adjustment amount of the signal level according to the ES, a portion indicated by a diagonal line represents an adjustment amount of the signal level according to the DG, and a portion indicated by a hatching represents an adjustment amount of the signal level according to the AG.

In the camera module2, so-called flicker in which lightness and darkness of an image is changed due to a power frequency of a fluorescent lamp supplying illumination light may occur at the time of indoor shooting. The camera module2can suppress the flicker by adjusting the ES using a double period of the period of the flicker as a unit. For example, when the power frequency of the fluorescent lamp is 60 Hz, the camera module2can suppress the flicker by adjusting the ES by 1/120 seconds.

For example, when a frame rate of a synthesized image is assumed to be 60 fps (frame per second), the camera module2sets the ES to one of 2/120 seconds and 1/120 seconds to suppress the flicker with 60 Hz. When the illuminance is high, the camera module2adjusts the ES within a range equal to or less than 1/120 seconds in order to prioritize the suppression of saturation of an output charge with respect to the amount of incident light than the suppression of the flicker.

In this example, the AE control circuit16divides an illuminance range with which shooting sensitivity is correlated by the camera module2into three stages and switches the control (lightness adjustment) of the AE operation according to the ES, the AG, and the DG. The AE control circuit16fixes the ES to 2/120 seconds within a low illuminance range b3and adjusts only the AG. The AE control circuit16fixes the ES to 1/120 seconds within an illuminance range b2which is a higher illuminance range than the illuminance range b3and adjusts only the DG.

The AE control circuit16adjusts the ES to be shorter step by step with an increase in the illuminance within an illuminance range b1which is a higher illuminance range than the illuminance range b2. The AE control circuit16adjusts a change amount of the illuminance corresponding to a unit less than a quantization unit of the ES according to the DG. Further, when the quantization unit of the ES is equal to or less than the resolution of the illuminance, the AE control circuit16may not perform the adjustment according to the DG.

The form of the control of the AE operation by the AE control circuit16can be appropriately changed. For example, after determining the ES according to the illuminance, the AE control circuit16may adjust one of the AG and the DG or may adjust both the AG and the DG.

The AE control circuit16may appropriately change the setting of the ES according to the frame rate of a synthesized image or the period of flicker. When the frame rate of a synthesized image is set to 30 fps with respect to the flicker with a frequency of 60 Hz, the AE control circuit16can adjust the ES to 4/120 seconds maximally. When the frequency of the flicker is 50 Hz, the AE control circuit16adjusts the ES by 1/100 seconds.

FIG. 6is a diagram illustrating calculation of control amounts of the ES, the AG, and the DG by the AE control circuit. The AE control circuit16performs the control of the AE operation on one of the long-time exposure image signal and the short-time exposure image signal designated as a main control signal. The AE control circuit16causes the AE operation for a sub-control signal to follow the AE operation for the main control signal. The sub-control signal is one of the long-time exposure image signal and the short-time exposure image signal other than the main control signal.

For example, it is assumed that the long-time exposure image signal is designated as the main control signal. The AE control circuit16calculates proper exposure L1for the long-time exposure pixel based on the long-time exposure image signal and calculates a control amount according to the proper exposure L1. For example, when the proper exposure L1falls within the illuminance range b3, the AE control circuit16calculates a control amount ES1(for example, 2/120 seconds) for the ES and a control amount AG1(for example, six times) for the AG.

The AE control circuit16calculates proper exposure L2for the short-time exposure pixel by multiplying the proper exposure L1by an HDR magnification M. For example, when the HDR magnification M is set to four times, the AE control circuit16multiples the proper exposure L1by 4 to calculate the proper exposure L2.

The AE control circuit16calculates a control amount according to the proper exposure L2. For example, when the proper exposure L2falls within the illuminance range b3, the AE control circuit16calculates a control amount ES2(for example, 2/120 seconds) for the ES and a control amount AG2(for example, 1.5 times) for the AG.

InFIG. 6, a gap between L1and L2in the horizontal axis direction corresponds to a difference in the illuminance according to the HDR magnification M. The AE control of causing the AE operation of the sub-control signal to follow the AE operation of the main control signal can be expressed as an operation of referring to an adjustment amount of the vertical axis by moving L1and L2in the horizontal axis direction with the gap between L1and L2maintained in the horizontal axis direction inFIG. 6.

FIG. 7is a block diagram illustrating the configuration of the AE control circuit. The AE control circuit16includes a main control signal switching unit20, a brightness signal generating unit21, a brightness average value calculating unit22, a brightness target value comparing unit23, an EV calculating unit24, a control amount converting unit25, a flicker detection integration unit26, and a flicker period estimating unit27.

The long-time exposure image signal S1and the short-time exposure image signal S2from the imaging processing circuit12(seeFIG. 1) are input to the AE control circuit16. The main control signal switching unit20outputs, as the main control signal, one of the long-time exposure image signal S1and the short-time exposure image signal S2input to the AE control circuit16. The main control signal switching unit20switches the output as the main control signal between the long-time exposure image signal S1and the short-time exposure image signal S2according to a change instruction signal33used to give an instruction to change the main control signal.

For example, the change instruction signal33is set as a signal generated in response to a user's setting operation. For example, when the image quality of a dark portion of an image is considered to be important, the camera module2may select the long-time exposure image signal S1from the long-time exposure pixel as the main control signal.

The brightness signal generating unit21generates a brightness signal35from the main control signal from the main control signal switching unit20. The brightness signal35is, for example, a signal for information corresponding to a brightness component of a YUV color space. For example, the brightness signal generating unit21extracts brightness information on a G component from RAW image data which is the main control signal and sets the extracted brightness information as the brightness signal35. The brightness signal generating unit21sets the brightness values of G components detected with a Gr pixel and a Gb pixel as the brightness signal35.

The brightness signal generating unit21uses, as the brightness signal35, the brightness value of a G component from which the most information on the brightness can be obtained among the R, G, and B components. The embodiment is not limited to the case in which the brightness signal generating unit21generates the brightness signal35only from the brightness value of the G component. For example, the brightness signal generating unit21may generate the brightness signal35using the brightness values of the R, G, and B components. The brightness signal35may be, for example, a signal obtained by adding the brightness values of the R, G, and B components by a predetermined ratio.

The brightness average value calculating unit22integrates and averages the brightness signals35of the entire screen and calculates a brightness average value36. The brightness average value calculating unit22may calculate the brightness average value36after weighting the brightness signals35for each area set in a screen.

The brightness target value comparing unit23compares the brightness average value36from the brightness average value calculating unit22to a preset brightness target value and calculates a difference. The brightness target value comparing unit23outputs a difference between the brightness average value36and the brightness target value as a lightness adjustment amount37used to adjust the lightness of a synthesized image according to illuminance at the time of shooting. For example, an adjustment amount of an exposure value (EV) for exposure correction is set as the lightness adjustment amount37.

The EV calculating unit24includes a main control EV calculating unit31and a sub-control EV calculating unit32. The main control EV calculating unit31calculates a main control EV41. The main control EV41is an EV for the main control signal. The main control EV calculating unit31calculates the main control EV41by performing calculation to reflect the lightness adjustment amount37from the brightness target value comparing unit23to the lightness of an image by the main control signal. The main control EV41corresponds to the proper exposure L1for the main control signal.

The sub-control EV calculating unit32calculates a sub-control EV42. The sub-control EV42is an EV for the sub-control signal. The sub-control EV calculating unit32multiplies the main control EV41calculated by the main control EV calculating unit31by the HDR magnification M and sets the multiplication result as the sub-control EV42. The sub-control EV42corresponds to the proper exposure L2for the sub-control signal.

The sub-control EV calculating unit32determines whether one of the long-time exposure image signal S1and the short-time exposure image signal S2is the sub-control signal based on the change instruction signal33. For example, when the HDR magnification M is set to four times and the long-time exposure image signal S1is designated as the main control signal, the sub-control EV calculating unit32multiplies the main control EV41by ¼ and sets the multiplication result as the sub-control EV42. On the other hand, when the HDR magnification M is set to four times and the short-time exposure image signal S2is designated as the main control signal, the sub-control EV calculating unit32multiplies the main control EV41by four and sets the multiplication result as the sub-control EV42. Further, the HDR magnification M is set to, for example, a fixed value set in advance.

The flicker detection integration unit26performs integration for the flicker detection on the main control signal from the main control signal switching unit20and outputs an integration result34. The flicker period estimating unit27estimates the period of the flicker based on the integration result34from the flicker detection integration unit26, and outputs an estimation result38.

FIG. 8is a diagram illustrating occurrence of flicker. The illumination of a fluorescent lamp blinks at the frequency which is the double of the power frequency. By sequentially reading signal charges of each line, exposure start times by an electronic shutter are different depending on the positions of the read lines. Thus, in a frame, uneven brightness caused due to the blinking of the fluorescent lamp is shown as bright and dark stripes.

The flicker period is 1/100 s or 1/120 s with respect to the power frequency of 50 Hz or 60 Hz. For example, when the frame period is 1/30 s with respect to the power frequency of 50 Hz, stripes with of a period of 1/100 s occur in which a line corresponding to a horizontal synchronization period T1in which the amount of light is the maximum is a bright portion and a line corresponding to a horizontal synchronization period T2in which the amount of light is the minimum is a dark portion. The horizontal synchronization periods T1and T2are set to, for example, two ms.

The flicker detection integration unit26performs integration of the main control signal for which a line is used as a unit in several portions in the screen. The flicker period estimating unit27estimates the flicker period from a difference between the integration results34of respective portions in the screen. For example, the flicker period estimating unit27estimates one of the flicker period of 1/100 s and the flicker period of 1/120 s by comparing the difference between the integration results34at the intervals of 1/100 s to the difference between the integration results34at the intervals of 1/120 s.

When the frame period is an integer multiple of 1/100 s with respect to the power frequency of 50 Hz and the frame period is an integer multiple of 1/120 s with respect to the power frequency of 60 Hz, the amount of exposure becomes constant irrespective of an exposure timing, and thus no flicker occurs. The flicker period estimating unit27estimates the period of flicker occurring when the frame period is not an integer multiple of the period of the blinking of the fluorescent lamp.

The control amount converting unit25converts the main control EV41from the main control EV calculating unit31into the control amount ES1for the ES, the control amount AG1for the AG, and a control amount DG1for the DG. The control amount converting unit25determines the control amount ES1according to the main control EV41as the proper exposure L1which falls within a given illuminance range (for example, b1to b3illustrated inFIG. 6).

In the example illustrated inFIG. 6, when the main control EV41falls within one of the illuminance ranges b2and b3, the control amount converting unit25determines the control amount ES1according to the flicker period output as an estimation result38. The control amount converting unit25sets a value of an integer multiple of the estimated flicker period as the control amount ES1.

When the main control EV41falls within the illuminance range b3, the control amount converting unit25determines the control amount AG1according to the main control EV41based on a linear relation between the main control EV41and the control amount AG1. When the main control EV41falls within the illuminance range b2, the control amount converting unit25determines the control amount DG1according to the main control EV41based on a linear relation between the main control EV41and the control amount DG1.

When the main control EV41falls within the illuminance range b1, the control amount converting unit25determines the control amount ES1corresponding to the main control EV41from the ES set step by step according to the illuminance. In this case, the control amount converting unit25determines the control amount ES1irrespective of the flicker period which is the estimation result38. The control amount converting unit25determines the control amount DG1according to the main control EV41based on a linear relation between the main control EV41and the control amount DG1in the quantization unit of the ES.

The control amount converting unit25converts the sub-control EV42from the sub-control EV calculating unit32into the control amount ES2for the ES, the control amount AG2for the AG, and a control amount DG2for the DG. The control amount converting unit25determines the control amount ES2according to the sub-control EV42as the proper exposure L2which falls within a given illuminance range (for example, b1to b3illustrated inFIG. 6). As in the conversion of the main control EV41into the control amounts ES1, AG1, and DG1, the control amount converting unit25converts the sub-control EV42into the control amounts ES2, AG2, and DG2.

The control amount converting unit25performs the conversion of the main control EV41calculated by the main control EV calculating unit31into each control amount and the conversion of the sub-control EV42calculated by the sub-control EV calculating unit32into each control amount in a time division manner. The AE control circuit16uses the control amount converting unit25common to the generation of each control amount in regard to the main control signal and the sub-control signal, and thus the circuit size can be reduced.

For example, when the long-time exposure image signal S1is designated as the main control signal, the timing generator15illustrated inFIG. 1outputs a pulse suitable for the control amount ES1to the long-time exposure pixel in the pixel array10. The timing generator15outputs a pulse suitable for the control amount ES2to the short-time exposure pixel in the pixel array10.

Further, when the long-time exposure image signal S1is designated as the main control signal, the preprocessing unit11illustrated inFIG. 1amplifies the long-time exposure image signal S1using the control amounts AG1and DG1. The preprocessing unit11amplifies the short-time exposure image signal S2using the control amounts AG2and DG2.

The EV calculating unit24calculates the main control EV41and the sub-control EV42falling within the illuminance range in which the camera module2has shooting sensitivity. The main control EV calculating unit31sets a limitation on the main control EV41so that not only the main control EV41but also the sub-control EV42calculated by the sub-control EV calculating unit32are included in the illuminance range.

For example, when the long-time exposure image signal S1is designated as the main control signal, the sub-control EV42for the short-time exposure image signal S2have a value which is larger than the main control EV41by the HDR magnification M. The main control EV calculating unit31limits the maximum value of the main control EV41so that the sub-control EV42is included in a range equal to or less than the maximum illuminance in which the camera module2has shooting sensitivity. Referring to the graph ofFIG. 6, adjustment of L1is limited such that L1can be moved to the left side (high illuminance side) of the graph within the limit of the left end of the graph reached by L2located to the left side from L1by M.

For example, when the short-time exposure image signal S2is designated as the main control signal, the sub-control EV42for the long-time exposure image signal S1have a value which is smaller than the main control EV41by the HDR magnification M. The main control EV calculating unit31limits the minimum value of the main control EV41so that the sub-control EV42is included in a range equal to or greater than the minimum illuminance in which the camera module2has shooting sensitivity. Referring to the graph ofFIG. 6, adjustment of L2is limited such that L2can be moved to the right side (low illuminance side) of the graph within the limit of the right end of the graph reached by L1located to the right side from L2by the HDR magnification M.

The EV calculating unit24can acquire the main control EV41and the sub-control EV42according to the shooting sensitivity of the camera module2by providing such limitations on the main control EV41. The AE control circuit16can perform the control of the AE operation according to the shooting sensitivity of the camera module2on both the long-time exposure image signal S1and the short-time exposure image signal S2.

For example, the camera module2is assumed to switch between an HDR shooting mode in which the HDR synthesis is performed and a normal shooting mode in which the HDR synthesis is not performed. In the HDR shooting mode, the AE control circuit16calculates each control amount for the long-time exposure image signal S1and the short-time exposure image signal S2.

FIG. 9is a block diagram illustrating elements used for the control of the AE operation in the normal shooting mode in the AE control circuit illustrated inFIG. 7. In the normal shooting mode, the solid-state imaging device5applies the same charge accumulation period to the respective pixels classified into the long-time exposure pixel and the short-time exposure pixel in the HDR shooting mode.

An image signal SO from the imaging processing circuit12is input to the AE control circuit16. The brightness signal generating unit21generates the brightness signal35from the image signal S0. The brightness average value calculating unit22integrates and averages the brightness signals35and calculates the brightness average value36. The brightness target value comparing unit23outputs a difference between the brightness average value36and the brightness target value as a lightness adjustment amount37used to adjust the lightness of an image according to illuminance at the time of shooting. The EV calculating unit24calculates the EV43by performing calculation to reflect the lightness adjustment amount37from the brightness target value comparing unit23to the lightness of an image by the image signal S0.

The flicker detection integration unit26performs integration for the flicker detection on the image signal S0and outputs an integration result34. The flicker period estimating unit27estimates the period of the flicker based on the integration result34from the flicker detection integration unit26, and outputs an estimation result38. The control amount converting unit25converts the EV43from the EV calculating unit24into a control amount ES0for the ES, a control amount AG0for the AG, and a control amount DG0for the DG. In the normal shooting mode, the control amount converting unit25calculates each control amount for the image signal S0obtained by applying the same charge accumulation period on each pixel.

The timing generator15illustrated inFIG. 1outputs a pulse suitable for the control amount ES0to the pixel array10. The preprocessing unit11amplifies the image signal S0based on the control amounts AG0and DG0.

The solid-state imaging device5according to the first embodiment performs the AE control through a simple calculation process, compared to a case in which a continuously adjusted HDR magnification M is applied, by calculating the control amounts for the sub-control signal based on the fixed HDR magnification M. The solid-state imaging device5can control the AE operation according to the lightness at the time of shooting through the simple calculation process in relation to the long-time exposure image signal and the short-time exposure image signal.

The AE control circuit16can reduce the circuit size and shorten the processing time by simplifying the calculation process. The solid-state imaging device5can realize the control of the AE operation in the HDR shooting mode by adding a circuit with a relatively small size such as the sub-control EV calculating unit32to the circuit configuration in which the HDR synthesis is not performed. The solid-state imaging device5can be configured by a small and simple circuit.

Each circuit configuration described in this embodiment may realize the function described in this embodiment and may be appropriately modified.

FIG. 10is a block diagram illustrating the configuration of an AE control circuit included in an image processing device according to a second embodiment. An AE control circuit50according to this embodiment can be applied to the solid-state imaging device5(seeFIG. 1) according to the first embodiment. The same reference numerals are given to the same units as those of the first embodiment and the description thereof will not be repeated.

The AE control circuit50includes a first control amount converting unit (control amount converting unit for main control)51and a second control amount converting unit (control amount converting unit for sub-control)52which are control amount converting units, instead of the control amount converting unit25illustrated inFIG. 7.

The first control amount converting unit51converts a main control EV41calculated by a main control EV calculating unit31into control amounts ES1, AG1, and DG1. The second control amount converting unit52converts a sub-control EV42calculated by a sub-control EV calculating unit32into control amounts ES2, AG2, and DG2.

The solid-state imaging device5according to the second embodiment can be configured by a small and simple circuit, as in the first embodiment. The AE control circuit50performs conversion of the main control EV41into each control amount by the first control amount converting unit51and conversion of the sub-control EV42into each control amount by the second control amount converting unit52in parallel. The AE control circuit50causes the AE operation to be performed faster by acquiring the control amounts in regard to the main control signal and the sub-control signal in parallel.

FIG. 11is a block diagram illustrating the configuration of an AE control circuit included in an image processing device according to a third embodiment. The AE control circuit60according to this embodiment can be applied to the solid-state imaging device5(seeFIG. 1) according to the first embodiment. The same reference numerals are given to the same units as those of the first and second embodiments and the description thereof will not be repeated.

The first control amount converting unit51outputs the control amount ES1calculated from a main control EV41to the second control amount converting unit52. The first control amount converting unit51and the second control amount converting unit52which are control amount converting units applies the same control amount ES1for the ES to the main control signal and the sub-control signal. The second control amount converting unit52determines the control amounts AG2and DG2according to the control amount ES1from the first control amount converting unit51and the sub-control EV42from the sub-control EV calculating unit32. The first control amount converting unit51and the second control amount converting unit52set different control amounts for at least one of the AG and the DG in regard to the main control signal and the sub-control signal.

The timing generator15illustrated inFIG. 1outputs a pulse suitable for the control amount ES1to both the long-time exposure pixel and the short-time exposure pixel of the pixel array10. The AE control circuit60performs conversion of the main control EV41into each control amount by the first control amount converting unit51and conversion of the sub-control EV42into each control amount by the second control amount converting unit52in parallel. The AE control circuit60causes the AE operation to be performed faster by acquiring the control amounts in regard to the main control signal and the sub-control signal in parallel.

The AE control circuit60may include a control amount converting unit25(seeFIG. 7) of the first embodiment instead of the first control amount converting unit51and the second control amount converting unit52. In this case, the control amount converting unit25performs conversion of the main control EV41calculated by the main control EV calculating unit31into each control amount and conversion of the sub-control EV42calculated by the sub-control EV calculating unit32into each control amount in a time division manner. The AE control circuit60uses the control amount converting unit25common to the generation of each control amount in regard to the main control signal and the sub-control signal, and thus the circuit size can be reduced.

FIG. 12is a diagram illustrating calculation of control amounts of ES, AG, and DG by the AE control circuit. The AE control circuit60performs the control of the AE operation on one of the long-time exposure image signal S1and the short-time exposure image signal S2designated as a main control signal. The AE control circuit60causes the AE operation on the sub-control signal which is one of the ling-time exposure image signal S1and the short-time exposure image signal S2other than the main control signal to follow the AE operation in regard to the main control signal.

For example, it is assumed that the long-time exposure image signal S1is designated as the main control signal. The first control amount converting unit51calculates proper exposure L1for the long-time exposure pixel based on the long-time exposure image signal S1. The first control amount converting unit51calculates the control amounts such as the ES1and the AG1according to the proper exposure L1.

The AE control circuit60calculates proper exposure L2for the short-time exposure pixel by multiplying the proper exposure L1by an HDR magnification M. The second control amount converting unit52calculates a control amount according to the proper exposure L2. The second control amount converting unit52uses the control amount ES1for the long-time exposure image signal S1as the control amount for the ES without change. Further, the second control amount converting unit52calculates the control amount such as the AG2other than the ES according to the property exposure L2.

InFIG. 12, a straight line AGL represents a relation between the control amount for the AG in regard to the long-time exposure image signal S1and the illuminance. A straight line AGS represents a relation between the control amount for the AG in regard to the short-time exposure image signal S2and the illuminance. The gap between the straight line AGL and the straight line AGS in the vertical axis direction corresponds to the difference in the AG according to the HDR magnification M.

For example, when the proper exposure L2falls within the illuminance range b1, the proper exposure L1falls within the illuminance range b2or b3(seeFIG. 6in regard to the illuminance ranges b1, b2, and b3), and the illuminance width of the proper exposures L1and L2cover the plurality of illuminance ranges, the AE operation is different between the long-time exposure pixel and the short-time exposure pixel. When different control amounts for the ES are applied to the long-time exposure pixel and the short-time exposure pixel, flicker may occur only in the short-time exposure pixel for which the proper exposure L2is a high illuminance side. The flicker in the short-time exposure pixel occurs more easily, as an illuminance width between the proper exposure L1and the proper exposure L2is larger, that is, as the HDR magnification M is larger.

The AE control circuit60according to the third embodiment can prevent the flicker from occurring in the short-time exposure pixel by applying the same control amount for the ES to the long-time exposure pixel and the short-time exposure pixel. The AE control circuit60according to this embodiment is appropriate when the prevention of the flicker is desirable.

For example, when the camera module2mounted on a drive recorder takes a picture of a traffic light in which an LED is used for display, the LED is not turned on and an image is recorded from a deviation between a frame rate and a period in which the LED is turned on and off. In this case, when the camera module2prevents the flicker from occurring by applying the AE control circuit60according to this embodiment, signal display of the LED can be accurately recorded.

The solid-state imaging device5according to the third embodiment can be configured by a small and simple circuit, as in the first embodiment. Further, when the AE control circuit60applies the control amount ES1determined in regard to the main control signal to the sub-control signal without change, a process of separately calculating the control amount for the ES in regard to the sub-control signal may not be provided. Thus, the solid-state imaging device5can cause the AE operation to be performed faster.