Device and method for producing an enhanced color image using a flash of infrared light

A device and method for producing an enhanced color image of a scene of interest captures a grayscale image of the scene of interest using a flash of infrared light and a color image of the scene of interest without using any flash of infrared light. The grayscale information from the grayscale image and the visible color information from the color image are combined to produce the enhanced color image.

BACKGROUND OF THE INVENTION

Digital cameras have become ubiquitous consumer products. In fact, digital cameras have now been incorporated into other consumer products, such as cellular phones and personal digital assistance (PDA) devices. Some of the reasons for this wide use of digital cameras include lower operating cost (no films required), instant review of captured images, ease of digital image processing and ease of image distribution via the Internet. However, digital cameras have challenges similar to conventional film cameras, such as capturing image in low lighting conditions.

There are a number of techniques available when capturing an image of a scene of interest in a low lighting condition using a digital camera. One technique is to use an electronic flash to brighten the scene of interest. This technique works well when taking a picture of a nearly subject, such as a person in close proximity to the camera. However, the use of a flash of light is intrusive and inappropriate in certain situations.

Another technique to capture an image of a scene of interest in a low lighting condition is to use a long exposure time to capture the image so that enough light can be received by the camera to produce a proper image of a scene of interest. However, the long exposure time subjects the camera to movements caused by the unsteadiness of the person taking the picture. Thus, it is common for the resulting image to appear blurry, which significantly degrades the quality of the captured image.

Another technique to capture an image of a scene of interest in a low lighting condition is to use an infrared flash to illuminate the scene of interest with a flash of infrared light. Using a flash of infrared light eliminates the need for a long exposure time to capture an image in a low lighting condition. Thus, using a flash of infrared light can significantly improve the image quality of the captured image. Furthermore, since infrared light cannot be seen, using a flash of infrared light is not intrusive as a flash of visible light. However, these advantages come with a price since the resulting image using a flash of infrared light will be monochromatic.

In view of these concerns, what is needed is a device and method for capturing a high-quality color image of a scene of interest in a low lighting condition without using an intrusive flash of visible light.

SUMMARY OF THE INVENTION

A device and method for producing an enhanced color image of a scene of interest captures a grayscale image of the scene of interest using a flash of infrared light and a color image of the scene of interest without using any flash of infrared light. The grayscale information from the grayscale image and the visible color information from the color image are combined to produce the enhanced color image. The device and method allows a user to capture a high-quality color image even under a low lighting condition.

A device for producing an enhanced color image of a scene of interest in accordance with an embodiment of the invention comprises an infrared flash configured to produce a flash of infrared light, a color image sensor configured to electronically capture images, and a processor operatively coupled to the infrared flash and the color image sensor to control the infrared flash and the color image sensor to capture a grayscale image of the scene of interest using the flash of infrared light and a color image of the scene of interest without using any flash of infrared light. The processor is further configured to combine grayscale information from the grayscale image with visible color information from the color image to produce the enhanced color image of the scene of interest.

A method for producing an enhanced color image of a scene of interest in accordance with an embodiment of the invention comprises emitting a flash of infrared light at the scene of interest, electronically capturing a grayscale image of the scene of interest using the flash of infrared light, electronically capturing a color image of the scene of interest without using any flash of infrared light, and combining grayscale information from the grayscale image with visible color information from the color image to produce the enhanced color image of the scene of interest.

DETAILED DESCRIPTION

With reference toFIG. 1, a digital imaging device10in accordance with an embodiment of the invention is described. The digital imaging device10may be any type of a digital imaging device, such as a digital camera, a digital video camera or a camera phone. As described in more detail below, the digital imaging device10is designed to produce high-quality color images, even when the images are captured under low lighting conditions.

As shown inFIG. 1, the imaging device10includes a user input interface14, an infrared (IR) flash16, a lens18, an IR filter19, a focusing mechanism20, a color image sensor12, an analog-to-digital converter (ADC)22, a processor24and a storage device26. The user input interface14allows a user to input commands and/or selections into the imaging device10. The user input interface14may include any type of input devices, such as buttons, dials, levers, switches and a touch screen display with graphical controls. The IR flash16operates to produce flashes of infrared light to provide illumination during low lighting conditions. The IR flash16may be an integrated component of the imaging device10. Alternatively, the IR flash16may be an external device that can be attached to the imaging device10when needed. The IR flash16can be set using the user input interface14to be automatically activated by the processor24when lighting conditions warrant the use of the IR flash.

The lens18is used to focus a scene of interest onto the color image sensor12to capture an image of that scene. The focusing mechanism20operates to move the lens18to focus the lens with respect to the scene of interest. The focusing mechanism20can be controlled manually using the user input interface14or automatically by the processor24.

The IR filter19is used to filter out IR light so that IR light does not reach the color image sensor12. The IR filter19is connected to a mechanism (not shown) to selectively position the IR filter in front of the color image sensor12when IR filtering is desired. When IR filtering is not desired, the IR filter19is moved out of the way to allow IR light to reach the color image sensor12. In an alternative embodiment, the IR filter19is implemented as a lens cap to be manually placed in front of the lens by the user when IR filtering is desired.

The color image sensor12is configured to electronically capture the focused image by generating image signals in the form of accumulated electrical charges in response to impinging light at different photosensitive locations on the image sensor. As shown inFIG. 2, which is an exploded view of the color image sensor12, the image sensor includes an imaging array30of photosensitive elements32and a color filtering array (CFA)34. Each photosensitive element32of the imaging array30accumulates an electrical charge when light is impinging upon that element, thereby producing an analog image signal. Thus, the photosensitive elements32of the imaging array30can be considered to be photosensitive locations or pixels of the image sensor12. As an example, the imaging array30may be a charged coupled device (CCD) array or a complementary metal-oxide semiconductor (CMOS) array.

The CFA34of the color image sensor12includes color pass filters36that are arranged in a predefined pattern. In the illustrated embodiment, the CFA34includes red (R), green (G) and blue (B) pass filters36that are arranged in a Bayer pattern. The CFA34is positioned over the imaging array30and faces the lens18to selectively transmit certain color lights to the photosensitive elements32of the imaging array through the color pass filters36. Thus, the light that impinges upon each photosensitive element32of the imaging array30depends on the color pass filter36of CFA34that is positioned over that photosensitive element. Since the incident light on each photosensitive element32of the imaging array30depends on the color pass filter36over that photosensitive element, each photosensitive element will sometimes be referred to herein as R, G or B photosensitive element, depending on the color pass filter positioned over that photosensitive element.

The R pass filters36of the CFA34allow red color light to be transmitted. Similarly, the G and B pass filters36of the CFA34allow green and blue lights, respectively, to be transmitted. In addition to the respective color light, each of these color pass filters36also allows infrared light, if any, to be transmitted. Thus, if the IR flash16is used and the IR filter19is not positioned in front of the color image sensor12, the light received by the R photosensitive elements32includes both red and infrared light components, which are reflected in the analog image signals (“R0”) generated by the R photosensitive elements. Similarly, if the IR flash16is used and the IR filter19is not positioned in front of the color image sensor12, the light received by the G photosensitive elements32includes both green and infrared components, which are reflected in the analog image signals (“G0”) generated by the G photosensitive elements, and the light received by the B photosensitive elements32includes both blue and infrared light components, which are reflected in the analog image signals (“B0”) generated by the B photosensitive elements. Thus, when the IR flash16is used and the IR filter19is not positioned in front of the color image sensor12, the R0, G0and B0analog image signals generated by the R, G and B photosensitive elements32of the imaging sensor30can be mathematically expressed as follows:
R0=R+IR, G0=G+IR, andB0=B+IR,
where R, G, B and IR represent red, green, blue and infrared light components, respectively. For a single exposure period using the IR flash16without the IR filter19, the R, G and B photosensitive elements32of the imaging sensor30produces a frame of R0, G0and B0analog image signals, which is equivalent to a single captured image of a scene of interest.

However, when the IR flash16is not used and the IR filter19is positioned in front of the color image sensor12, the R, G and B pass filters36of the CFA34allow only red, green and blue color lights, respectively, to be transmitted since no infrared light will be received by the CFA. As a result, all of the light received at each photosensitive element32of the imaging array30is either red, green or blue color light, depending on the color pass filter36of the CFA34positioned over that photosensitive element. Thus, if the IR flash16is not used and the IR filter19is positioned in front of the color image sensor12, the light received by the R photosensitive elements32of the imaging array30include only red light components, which are reflected in the analog image signals (“R0”) generated by the R photosensitive elements. Similarly, if the IR flash16is not used and the IR filter19is positioned in front of the color image sensor12, the light received by the G photosensitive elements32includes only green light components, which are reflected in the analog image signals (“G1”) generated by the G photosensitive elements, and the light received by the B photosensitive elements32includes only blue light components, which are reflected in the analog image signals (“B1”) generated by the B photosensitive elements. Thus, when the IR flash16is not used and the IR filter19is positioned in front of the color image sensor12, the R1, G1and B1, analog image signals generated by the R, G and B photosensitive elements32of the imaging sensor30can be mathematically expressed as follows:
R1=R, G1=G, andB1=B
For a single exposure period without using the IR flash16and with the IR filter19being positioned in front of the color image sensor12, the R, G and B photosensitive elements32of the imaging sensor30produces a frame of R1, G1and B1, analog image signals, which is equivalent to a single captured image of a scene of interest.

Turning back toFIG. 1, the analog image signals generated by the photosensitive elements32of the image sensor12in the form of accumulated electrical charges are converted to corresponding digital signals by the ADC22. The digital image signals are then transmitted to the processor24for signal processing.

The processor24of the imaging device10processes the digital image signals from the ADC22to produce a digital image of the captured scene of interest. As described in more detail below, the processor24processes at least two frames of digital image signals of the same scene of interest to produce a high-quality color image even under a low lighting condition. The processor24may also perform other tasks, such as demosaicing, image enhancements and compression. The resulting high-quality digital color image is stored in the storage device26, which may include a removable memory card. The processor24also controls various active components of the imaging device10, such as the IR flash16, the focusing mechanism20, the image sensor12and the ADC22. The processor24also performs operations commanded by a user through the user input interface14.

The digital imaging device10includes other components that are commonly found in conventional digital cameras, which are not shown or described herein so that the inventive features of the imaging device are not obscured.

An image enhancing operation of the digital imaging device10in accordance with an embodiment of the invention is now described with reference to a flow diagram ofFIG. 3. The image enhancing operation produces a high-quality color image of a scene of interest even when captured under a low lighting condition. At block302, the mode of operation for the digital imaging device10is set to the image enhancing mode. The setting of the operational mode to the image enhancing mode can be executed in response to a user input made using the user input interface14. Alternatively, the setting of the operational mode to the image enhancing mode can be executed automatically by the digital imaging device10by sensing the ambient lighting condition using a sensor (not shown).

Next, at block304, a grayscale image of a scene of interest is captured using a flash of infrared light, which is produced by the IR flash16, during a first exposure period without the IR filter19being positioned in front of the color image sensor12. This grayscale image is captured as a frame of R0, G0and B0analog image signals, which are generated by the R, G and B photosensitive elements32of the color image sensor12. The R0, G0and B0analog image signals are then converted to R0, G0and B0digital image signals by the ADC22. Since each of the R0, G0and B0digital image signals includes both red, green or blue color component and infrared component, each of the R0, G0and B0digital image signals includes grayscale information, which is derived from the respective color and infrared components. The frame of R0, G0and B0digital image signals is converted into the grayscale image and may be temporarily stored in the storage device26.

Next, at block306, a color image of the same scene of interest is captured without using a flash of infrared light during a second exposure period with the IR filter19being positioned in front of the color image sensor12. The color image is rich in visible color information of the scene of interest. However, the quality of the color image with respect to sharpness and clarity is significantly lower than the grayscale image. As illustrated inFIG. 4, there are different techniques to capture a color rich image under a low lighting condition. Although only two techniques to capture a color rich image under a low lighting condition are described herein, other techniques are possible. In accordance with one technique, at block410, the color image is captured using a long exposure period. This long exposure period is significantly longer than the first exposure period used to capture the grayscale image. As an example, the long exposure period may be ten times longer (e.g., ⅓ second) than the first exposure period (e.g., 1/30 second) used to capture the grayscale image. In accordance with an alternative technique, at block412, an underexposed color image of the same scene of interest is captured using a shorter exposure period, which can be same as the first exposure period for the grayscale image, and then, at block414, the image signals of the underexposed color image are amplified to increase the brightness of the underexposed image to produce the color image. The same effect can be achieved by setting the ISO parameter of the digital imaging device10to a high value. This alternative technique, however, introduces significant amount of noise into the resulting color image. The noise can be reduced, however, by limiting how much the signal can vary from one image pixel to the next. Alternatively, the noise can be reduced by averaging the signal of each image pixel with the signals from neighboring image pixels.

Next, at block308, the color image is interpolated into a demosaiced color image such that each image pixel of the resulting color image contains red, green and blue components, representing red, green and blue intensity values of that pixel. Any color interpolation process can be used to produce the demosaiced color image. As an example, the missing color components of an image pixel of the color image can be obtained by averaging the signals from surrounding image pixels containing the missing color information. The demosaiced color image may be temporarily stored in the storage device26.

Next, at block310, the grayscale image is converted into a high-quality output color image using the visible color information of the demosaiced color image. That is, the grayscale image is “colorized” using the visible color information of the demosaiced color image. This process can be compared to a process of “colorizing” an old black-and-white image. As illustrated inFIG. 5, there are different techniques to convert the grayscale image into the high-quality output color image. Although only two techniques to convert the infrared image into the high-quality output color image are described herein, other techniques are possible. In accordance with one technique, at block520, the red, green and blue values of each pixel of the demosaiced color image are modified such that the highest red, green or blue value equals the grayscale value of the corresponding pixel of the grayscale image while maintaining the original red, green and blue color ratio. These modified red, green and blue values will then be used in a corresponding pixel of the high-quality color image. As an example, if the grayscale value at a particular pixel of the grayscale image is 152 out of 255 steps and the red, green and blue color ratio at a corresponding pixel of the demosaiced color image is 1:3:2, then the modified red, green and blue values would be 51, 152 and 101, respectively. In accordance with an alternative technique, at block522, the red, green and blue values of each pixel of the demosaiced color image are modified so that the sum of the modified red, green and blue values equals the grayscale value of the corresponding pixel of the grayscale image while maintaining the original red, green and blue color ratio. These modified red, green and blue values will then be used in a corresponding pixel of the high-quality color image. As an example, if the grayscale value at a particular pixel of the grayscale image is 152 out of 255 steps and the red, green and blue color ratio at a corresponding pixel of the demosaiced color image is 1:3:2, then the modified red, green and blue values would be 25.3, 76 and 50.6, respectively, since the sum of the modified red, green and blue values (25.3+76+50.6=152) equals the grayscale value (152). In either technique, the grayscale information from the grayscale image and the visible color information from the demosaiced color image are combined to produce the high-quality output color image.

Next, at block316, the high-quality output color image is stored in the storage device126.

A method for producing an enhanced color image of a scene of interest in accordance with an embodiment of the invention is described with reference to the flow diagram ofFIG. 6. At block602, a flash of infrared light is emitted at the scene of interest. Next, at block604, a grayscale image of the scene of interest is electronically captured using the flash of infrared light. Each pixel of the grayscale image includes grayscale information for a particular location of the scene of interest. The grayscale information is derived from both infrared light and visible color light. Next, at block606, a color image of the same scene of interest is electronically captured without using any flash of infrared light. Each pixel of the color image includes visible color information for a particular location of the scene of interest. Next, at block608, the grayscale information from the grayscale image and the visible color information from the color image are combined to produce the enhanced color image of the scene of interest.