Joint automatic demosaicking and white balancing

Joint automatic demosaicking and white balancing. In one example embodiment, a digital image processing method includes several acts. First, directional color correlation signals, global gains, and orientations of edges are calculated in a CFA input image. Next, missing luminance components in CFA chrominance locations are demosaicked along edges in the input image using CFA chrominance components and the directional color correlation signals. Then, the CFA chrominance components are white-balanced using the demosaicked luminance components, the CFA chrominance components, and white-balancing gains expressed as a function of the global gains and local gains calculated directly from a pixel under consideration. Next, missing chrominance components in CFA chrominance locations in the input image are demosaicked. Finally, missing chrominance components in CFA luminance locations in the input image are demosaicked. Performance of these acts results in the transformation of the CFA input image into a full-color white-balanced output image.

BACKGROUND

1. Technical Field

Embodiments of the present invention relate to digital image processing. More specifically, embodiments of the invention relate to methods for joint automatic demosaicking and white balancing of a color filter array (CFA) single-sensor image.

2. Related Art

A digital image is a representation of a two-dimensional analog image as a finite set of pixels. Digital images can be created by a variety of devices, such as digital cameras, scanners, and various other computing devices. Digital image processing is the use of computer algorithms to perform image processing on digital images. Image processing operations include, for example, demosaicking, white balancing, color to grayscale conversion, color adjustment, intensity adjustment, scene analysis, and object recognition. Demosaicking refers to the process of interpolating a full-color digital image from mosaic data received from a CFA internal to many digital cameras. White balancing refers to the process of adjusting the color in a digital image in order to compensate for color shifts due to scene illumination.

In common single-sensor digital camera imaging pipelines, automatic white-balancing solutions are used before or after the demosaicking step to adjust the coloration of the captured visual data in order to compensate for the scene illuminant. Since demosaicking restores the full-color information from the gray-scale CFA sensor readings, the white balancing after demosaicking approach is significantly slower than the white balancing before demosaicking approach when using the same white-balancing solution. On the other hand, white-balancing solutions usually adjust chrominance components (i.e., R and B components of color images in a common Red-Green-Blue format) using luminance components (i.e., G components of color images in a common Red-Green-Blue format). Since in CFA images the pixel locations with available chrominance samples lack the luminance component, which thus has to be estimated prior to adjusting the chrominance component, the white balancing before demosaicking approach may not be as accurate as the white balancing after demosaicking approach when using the same powerful demosaicking solution.

SUMMARY

In general, example embodiments relate to methods for joint automatic demosaicking and white balancing of a digital image.

In a first example embodiment, a digital image processing method includes several acts. First, directional color correlation signals, global gains, and orientations of edges are calculated in a CFA input image. Next, missing luminance components in CFA chrominance locations are demosaicked along edges in the input image using CFA chrominance components and the directional color correlation signals. Then, the CFA chrominance components are white-balanced using the demosaicked luminance components, the CFA chrominance components, and white-balancing gains expressed as a function of the global gains and local gains calculated directly from a pixel under consideration. Next, missing chrominance components in CFA chrominance locations in the input image are demosaicked. Finally, missing chrominance components in CFA luminance locations in the input image are demosaicked. Performance of these acts results in the transformation of the CFA input image into a full-color white-balanced output image.

In a second example embodiment, a digital image processing apparatus includes a calculation module, a luminance component demosaicking module, means for jointly white balancing and demosaicking, and a chrominance-luminance component demosaicking module. The calculation module is configured to calculate directional color correlation signals, global gains, and orientations of edges in a CFA input image. The luminance component demosaicking module is configured to demosaick missing luminance components in CFA chrominance locations along edges in the input image using CFA chrominance components and the directional color correlation signals. The means for means for jointly white balancing and demosaicking is configured white balance the CFA chrominance components and demosaick missing chrominance components in CFA chrominance locations in the input image. The chrominance-luminance component demosaicking module is configured to demosaick missing chrominance components in CFA luminance locations in the input image. The calculation module, the luminance component demosaicking module, the means for jointly white balancing and demosaicking, and the chrominance-luminance component demosaicking module are configured to transform the CFA input image into a full-color white-balanced output image.

In a third example embodiment, one or more computer-readable media have computer-readable instructions thereon which, when executed, implement a digital image processing method, for example, of the sort discussed above in connection with the first example embodiment.

In a fourth example embodiment, a digital image processing apparatus includes a calculation module, a luminance component demosaicking module, a chrominance component white-balancing module, a chrominance-chrominance component demosaicking module, and a chrominance-luminance component demosaicking module. The calculation module is configured to calculate directional color correlation signals, global gains, and orientations of edges in a CFA input image having a Bayer pattern with GRGR phase in odd rows and BGBG phase in even rows. The luminance component demosaicking module is configured to demosaick missing G components in CFA R and B locations along edges in the input image. The chrominance component white-balancing module is configured to white-balance the CFA R and B components using the demosaicked G components, the CFA R and B components, and white-balancing gains expressed as a function of the global gains and local gains calculated directly from a pixel under consideration. The chrominance-chrominance component demosaicking module is configured to demosaick missing R components in CFA B locations in the input image and missing B components in CFA R locations in the input image. The chrominance-luminance component demosaicking module is configured to demosaick missing R and B components in CFA luminance locations in the input image. The calculation, luminance component demosaicking, chrominance component white-balancing, chrominance-chrominance component demosaicking, and chrominance-luminance component demosaicking modules are configured to transform the CFA input image having only one of an R, G, or B component in each pixel into a full-color white-balanced output image having an R, G, and B component in each pixel.

DETAILED DESCRIPTION OF SOME EXAMPLE EMBODIMENTS

In general, example embodiments relate to methods for joint automatic demosaicking and white balancing of a digital image. Example embodiments can be used to automatically produce a visually pleasing full-color white-balanced image from a gray-scale mosaic image received from a CFA internal to a digital camera. Performing the demosaicking and white-balancing operations simultaneously on the CFA data reduces processing errors and increases computational efficiency. In addition, performing these operations simultaneously allows for high-quality white-balanced full-color red-green-blue (RGB) images, making the proposed framework an attractive solution for both digital color imaging devices and image processing software for digital cameras.

I. Example Environment

Computer-executable instructions comprise, for example, instructions and data which cause a processor of a general purpose computer or a special purpose computer to perform a certain function or group of functions. Although the subject matter is described herein in language specific to methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific acts described herein. Rather, the specific acts described herein are disclosed as example forms of implementing the claims.

Examples of special purpose computers include image processing apparatuses such as digital cameras (an example of which includes, but is not limited to, the Epson R-D1 digital camera manufactured by Seiko Epson Corporation headquartered in Owa, Suwa, Nagano, Japan), digital camcorders, projectors, printers, scanners, check scanners, copiers, portable photo viewers (examples of which include, but are not limited to, the Epson P-3000, P-6000, or P-7000 portable photo viewers manufactured by Seiko Epson Corporation), or portable movie players, or some combination thereof, such as a printer/scanner/copier combination, printer/check scanner combination, or a or a digital camera/camcorder combination. An image processing apparatus may include joint automatic demosaicking and white-balancing capability, for example, to automatically demosaick and white balance a CFA image in order to automatically generate a full-color white-balanced image. For example, a camera with this joint automatic demosaicking and white-balancing capability may include one or more computer-readable media that implement the example methods disclosed herein, or a computer connected to the camera may include one or more computer-readable media that implement the example methods disclosed herein.

A schematic representation of an example camera100is disclosed inFIG. 1. The example camera100exchanges data with a host computer150by way of an intervening interface102. Application programs and a camera driver may also be stored for access on the host computer150. When an image retrieve command is received from the application program, for example, the camera driver controls conversion of the command data to a format suitable for the camera100and sends the converted command data to the camera100. The driver also receives and interprets various signals and data from the camera100, and provides necessary information to the user by way of the host computer150.

When data is sent by the host computer150, the interface102receives the data and stores it in a receive buffer forming part of a RAM104. The RAM104can be divided into a number of sections, for example through addressing, and allocated as different buffers, such as a receive buffer or a send buffer. Data, such as digital image data, can also be obtained by the camera100from the capture mechanism(s)112. For example, the capture mechanism(s)112may be a CFA-equipped sensor capable of generating a mosaic-like image of color visual data corresponding to a real-world scene. This so-called CFA image can then be stored in the receive buffer or the send buffer of the RAM104.

A processor106uses computer-executable instructions stored on a ROM108or on a flash EEPROM110, for example, to perform a certain function or group of functions, such as the example methods for joint automatic demosaicking and white balancing disclosed herein. Where the data in the receive buffer of the RAM104is a CFA image, for example, the processor106can implement the methodological acts of the example methods for joint automatic demosaicking and white balancing on the CFA image in order to automatically generate a full-color white-balanced image. Further processing in an imaging pipeline may then be performed on the full-color white-balanced image before being displayed by the camera100on a display114, such as an LCD display for example, or transferred to the host computer150, for example.

It is understood that CFA images may be received by the camera100from sources other than the computer150and the capture mechanism(s)112, including, but not limited to, the flash EEPROM110or the ROM108. Example embodiments of the camera100include, but are not limited to, the Epson R-D1 Digital Camera manufactured by Seiko Epson Corporation.

II. Example Method

FIG. 2is a flowchart of an example method200for joint automatic demosaicking and white balancing of a digital image. The example method200will now be discussed in connection withFIG. 2. Prior to performing the method200, an input color filter array (CFA) image can be targeted for demosaicking and white balancing in order to produce a full-color white-balanced output image.

At202, directional color correlation signals, global gains, and orientations of edges are calculated in the input image. At204, missing luminance components in CFA chrominance locations are demosaicked along edges in the input image using CFA chrominance components and the directional color correlation signals. At206, the CFA chrominance components are white-balanced using the demosaicked luminance components, the CFA chrominance components, and white-balancing gains expressed as a function of the global gains and local gains calculated directly from a pixel under consideration. At208, missing chrominance components in CFA chrominance locations in the input image are demosaicked. Then, at210, missing chrominance components in CFA luminance locations in the input image are demosaicked.

Performing the demosaicking and white-balancing operations simultaneously on the CFA image reduces processing errors and increases computational efficiency. In addition, it allows for high-quality white-balanced full-color RGB images, making the example method200an attractive solution for both digital color imaging devices and image processing software for digital cameras.

III. Example Implementation of the Example Method

FIG. 3is an example implementation300of the example method200ofFIG. 2. The example implementation300of the example method200will now be disclosed in connection withFIG. 3. As disclosed inFIG. 3, the example implementation300receives as input a CFA image302and generates a full-color white-balanced output image304. Although the input image302and the output image304each include only 9 rows and 9 columns of pixels, it is understood that the example method200can be performed on an input image having a greater or lesser number of rows and/or columns, resulting in an output image having a greater or lesser number of rows and/or columns.

As disclosed inFIG. 3, the example implementation300operates directly on the CFA image302before this image has been demosaicked into the full-color image or adjusted to generate its white-balanced version. Since a conventional image sensor is essentially a monochrome device, each sensor cell has its own color filter to acquire visual information. In typical CFA sensor configurations, each sensor cell or pixel location of the acquired image corresponds to the red (R), green (G), or blue (B) color filter in the CFA. The CFA image302thus constitutes a gray-scale mosaic-like image z with pixels Z(r,s). The arrangement of color filters in CFA images varies among camera manufacturers. One example arrangement is the Bayer pattern with GRGR phase in odd rows and BGBG phase in even rows. In the CFA image302with K1rows and K2columns where r=1, 2, . . . , K1and s=1, 2, . . . , K2denote, respectively, the image row and column, z(r,s)corresponds to the R components for (odd r, even s), the G components for (odd r, odd s) and (even r, even s), and the B components for (even r, odd s). Note that the G components are often referred to as luminance signals, whereas the R and B components are often referred to as chrominance signals. Although the following description of the example implementation300is disclosed in connection with the Bayer CFA presented above, the extension of the example method200to other CFA arrangements is straightforward and contemplated.

With continued reference toFIGS. 2 and 3, at202, the directional color correlation signals, global gains, and the orientation of edges are calculated in the input image302. The purpose of the preprocessing calculations at202is to obtain information for guiding the subsequent joint demosaicking and white-balancing process.

The performance of demosaicking algorithms critically depends on their ability to follow both the spectral (i.e. color) and structural (i.e. edge) characteristics of the captured image in order to avoid color artifacts and edge blur. This ability usually corresponds to the formation of the directional color correlation signals δ(z) from the CFA luminance and chrominance values along the dominant edge direction. The δ(z) signal is the function of the CFA image z and consists of the samples δ(r,s). More specifically, the directional color correlation signal δ(r,s)may be obtained according to the following formula:
δ(r,s)=ƒΛΩ(z,(r,s))
where:Λ denotes the spectral correlation operator characterizing the relation between the luminance and chrominance components, andΩ denotes the edge orientation operator.
In some example embodiments, the relation between the luminance and chrominance components denoted by Λ is subtraction or division. For example, in case of subtraction Λ(a1, a2)=a1−a2. In most demosaicking solutions, Ω-based edge orientation estimates are obtained by maximizing certain local similarity criterion defined on CFA samples in some spatial direction (usually vertical, horizontal, and two diagonal directions) and Λ-based color-correlation signals are usually formed as the luminance-chrominance differences; as both these approaches are relatively easy to implement.

A simple way to generate the directional color correlation signal δ(z) with samples δ(r,s)is as follows:

Automatic white-balancing solutions rely on the spectral characteristics to set the parameters (gains) of a white-balancing process in order to produce images with visually pleasing colors. Due to the mosaic nature of the CFA image, the gain values α(r,s)can be obtained using the global spectral characteristics of the CFA image z as follows:

After the values ofR,GandBare achieved using one of the procedures listed below Equation (3), then following the standard formulation of the white-balancing problem the global gain values can be expressed as α(r,s)=G/Rfor (odd r, even s), α(r,s)=G/Bfor (even r, odd s), and α(r,s)=1 for (odd r, odd s) and (even r, even s).

After the preprocessing calculations of act202are completed, the framework can perform the actual demosaicking and white-balancing operations in order to transform the CFA image302into a demosaicked, full-color white-balanced output image304. The pixels x(r,s)=[x(r,s)1,x(r,s)2,x(r,s)3] of this demosaicked, white-balanced full-color image consist of x(r,s)1, x(r,s)2, and x(r,s)3components which denote the R, G, and B component, respectively. In practice, automatic white-balancing solutions usually use luminance components to adjust chrominance components while keeping luminance components unchanged. For this reason, at204, the luminance components in CFA chrominance locations (R and B locations of the input image302) are demosaicked along edges in the input image302using the CFA chrominance components (R and B locations of the input image302) and the directional color correlation signals δ(z). Since pixels of the CFA data lack the luminance component in chrominance locations, joint demosaicking and white-balancing operations of acts206and208should be realized as follows:

ƒ(•) denotes the function of the global gain α(r,s)from Equation (2), and the local gain β(r,s)defined using the demosaicked luminance component x(r,s)2from Equation (4) as follows:

Performing Equation (4) in all CFA chrominance locations (odd r, even s) and (even r, odd s) produces demosaicked white-balanced full-color pixels x(r,s). However, this is not the case in the CFA luminance locations (odd r, odd s) and (even r, even s), where only the luminance component is available. Therefore, to complete the joint demosaicking and white-balancing process, the procedure of act210should continue as follows:

More specifically, since in this embodiment Equation (2) forms the color correlation signals δ(r,s)as differences between CFA luminance and chrominance components and the global gains are calculated as ratios of the luminance and chrominance values, x(r,s)2may be demosaicked and then used to obtain the white-balanced chrominance samples x(r,s)1and x(r,s)3as follows:
x(r,s)2=z(r,s)+δ(r,s)for (oddr, evens) and (evenr, odds)
x(r,s)1=z(r,s)(w1x(r,s)2/z(r,s)+w2G/R)/(w1+w2) for (oddr, evens)
x(r,s)3=z(r,s)(w1x(r,s)2/z(r,s)+w2G/R)/(w1+w2) for (evenr, odds)  (7)
In Equation (7), δ(r,s)and ƒ(•) from Equation (4) are expressed as β(r,s)=x(r,s)2/z(r,s)and ƒ(α(r,s),β(r,s))=(w1β(r,s)+w2α(r,s))/(w1+w2) where the terms w1and w2are the weights used to control the contribution of the local and global white-balancing terms to the total gain. It should be emphasized here that using the actual demosaicked luminance component to adjust the acquired chrominance components in the white-balancing process in Equation (7) greatly minimizes the processing errors and enhances the efficiency of the joint white-balancing and demosaicking process.

Using the last expression from Equation (4), the white balanced chrominance (R and B) components can be obtained in chrominance (B and R, respectively) CFA locations as follows:

x(r,s)⁢k=x(r,s)⁢2+1ζ⁢∑(i,j)∈ζ⁢(x(i,j)⁢k-x(i,j)⁢2)(8)
where ζ={(r−1,s−1),(r−1,s+1),(r+1,s−1),(r+1,s+1)}, thus implying |ζ|=4. The value of k should be set as k=1 for (even r, odd s) and k=3 for (odd r, even s).

Since available R and B components are now located in the CFA chrominance locations (odd r, even s) and (even r, odd s), they can be used to produce demosaicked white balanced chrominance components in CFA luminance locations (odd r, odd s) and (even r, even s) using Equation (8) with ζ={(r−1,s),(r+1,s)} for |z(r−1,s)−z(r+1,s)|≦|z(r,s−1)−z(r,s+1)| and ζ={(r,s−1),(r,s+1)} for |z(r 1,s)−z(r+1,s)|<|z(r,s 1)−z(r,s+1)|, thus implying in both cases |ζ|=2. The comparison of |z(r−1,s)−z(r+1,s)| and |z(r,s−1)−z(r,s+1)| constitutes the edge-orientation operator from Equation (2) which is now employed to ensure consistency in luminance and chrominance estimates. Performing Equation (8) with the above edge-sensing procedure in all CFA luminance locations completes the joint demosaicking and white-balancing process, thus producing the restored full-color output image304with an adjusted coloration.