Systems and methods for camera image correction and white balancing

Systems and methods are provided for calibrating a digital camera. First pixel values of a first calibration image illuminated by white light are captured using a color sensor. A color sensor adjustment matrix is determined based on a comparison of the first pixel values and baseline pixel values associated with the calibration image. Second pixel values of a second calibration image illuminated by a first color temperature light are captured. The second pixel values are adjusted based on the color sensor adjustment matrix, and a first color temperature balance factor is determined based on the adjusted second pixel values, where a white balance adjustment is performed on captured image data based on the first color temperature balance factor.

FIELD

The technology described herein relates generally to photographic image capture and more particularly to photographic image capture processing.

BACKGROUND

When a picture is taken using a digital camera, typically red, green, and blue (RGB) pixel values are captured by respective sensors. Depending on a light source, coloring of certain objects may appear off. For example, when a white or gray object is illuminated with a low color temperature light source, the object in the captured image may appear reddish in color. Similarly, when a white or gray object is illuminated with a high color temperature light so the object in the captured image may appear bluish in color. A white balancing process, as described herein, is used to improve the quality of the captured image.

SUMMARY

Examples of systems and methods are provided for calibrating a digital camera. First pixel values of a first calibration image illuminated by white light are captured using a color sensor. A color sensor adjustment matrix is determined based on a comparison of the first pixel values and baseline pixel values associated with the calibration image. Second pixel values of a second calibration image illuminated by a first color temperature light are captured. The second pixel values are adjusted based on the color sensor adjustment matrix, and a first color temperature balance factor is determined based on the adjusted second pixel values, where a white balance adjustment is performed on captured image data based on the first color temperature balance factor.

As another example, a digital camera includes a raw age capture module configured to capture image data based on a command. The camera further includes a memory, where the memory contains a color sensor adjustment matrix, where the color sensor adjustment matrix contains data that is based on a comparison of captured first pixel values of a first calibration image illuminated by white light using a color sensor and baseline pixel values associated with the calibration image. The memory further contains a first color temperature balance factor, where the first color temperature balance factor is based on captured second pixel values of a second calibration image illuminated by a first color temperature light that is adjusted based on the color sensor adjustment matrix. The camera further includes a data processor configured to adjust image data captured by the raw image capture module based on the first color temperature balance factor.

As a further example, a method of adjusting a captured image includes receiving captured image data captured by a digital camera. A color sensor adjustment of the captured image data is performed using a color sensor adjustment matrix, wherein the color sensor adjustment matrix contains data that is based on a comparison of previously captured first pixel values of a first calibration image illuminated by white light using a color sensor and baseline pixel values associated with the calibration image. A white balance adjustment of the captured image data is performed using a color temperature balance factor, wherein the color temperature balance factor is based on previously captured second pixel values of a second calibration image illuminated by a given color temperature light that is adjusted based on the color sensor adjustment matrix. The adjusted captured image data is saved in a computer-readable memory.

DETAILED DESCRIPTION

FIG. 1is a block diagram depicting a digital camera having parameters adjusted by a calibration module. A digital camera102having a lens104and a raw image capture module106“takes pictures” to capture raw image data associated with one or more calibration images108. That raw image data is stored as a plurality of pixel values in a memory110. Typically, a pixel value includes one or more of a red value, a green value, and a blue value for each pixel location in an array of pixel locations that make up the picture. A data processor112is responsive to the memory110for accessing the raw image data.

Oftentimes, the quality of a picture produced from the raw image data alone is sub-optimal for a variety of reasons. For example, imperfections in the raw image capture module106can have an adverse affect on a generated picture. As another example, environmental factors, such as the type of light illuminating a subject of a picture can also have an effect on picture quality. In a production mode, the data processor112is configured to adjust the raw image data with the intent to output an adjusted image that is pleasing to a viewer. In one example, the data processor112is configured perform a color adjustment and a white balance adjustment to the raw image data to generate the adjusted image. That adjusted image is saved in the memory110, where the adjusted image can be subsequently accessed or exported by a user.

In a calibration mode, the data processor112works in conjunction with a calibration module114to set certain image adjustment parameters, such as through image captures of calibration images108having known characteristics. Such image adjustment parameters are stored in the memory110and are utilized by the data processor112in adjusting raw image data when the camera is in the production mode. In one example, the adjustment parameters includes a color sensor adjustment matrix that is used by the data processor112in performing a color adjustment, and the adjustment parameters further include one or more color temperature balance factors used by the data processor in performing a white balance adjustment.

Calibration of the digital camera102can be performed in a variety of ways. In an environment where a camera manufacturer is confident of the consistency of certain camera102hardware (e.g., lens104, raw image capture module106) across a plurality of manufactured cameras, calibration can be performed once or a sample of times for a number of cameras (e.g., a production batch, a number of cameras having hardware from a common production batch). In such an environment, the adjustment parameters are determined by performing calibration for one camera or a sample of cameras, and those adjustment parameters are loaded into the memories110of a larger number of manufactured cameras. In that environment, an external calibration module114may be used because calibration is not performed for many of the individual manufactured digital cameras. In another example, calibration is performed for each digital camera to determine adjustment parameters that are specific to that digital camera. In that case, an external calibration module114may be used or each camera may include an internal calibration module, as depicted inFIG. 6.

FIG. 2is a flow diagram depicting an example method of calibrating a digital camera, such as through the use of a calibration module. The method begins by determining parameters of a color sensor adjustment matrix. At202, first pixel values of a first calibration image are captured using a color sensor, where the first calibration image is illuminated by white light (e.g., a D50 light having a color temperature of 5000K°). By illuminating the first calibration in white light, the color temperature of the illuminating light has no effect on the captured color of the first calibration image. Such illuminating results in a control case, where any color imperfection in the first pixel values is due to imperfections in camera hardware, such as a raw image capture module.

The calibration image has known properties.FIG. 3is a depiction of an example calibration image. The calibration image302includes alignment aids304at each of the corners. The calibration image further includes four rows of color panels, where each of the panels includes a different color. In the example ofFIG. 3, the top three rows306of panels depict different colors, while the bottom row308of panels depicts varying degrees of grayscale. Certain parameters of the calibration image are known. In one example, correct, baseline red, green, and blue pixel values are known for each of the panels of the calibration image. These known calibration image baseline parameters are used in determining adjustment parameters for cameras.

With reference back toFIG. 2, having captured the calibration image illuminated in white light, any differences between the captured first pixel values and the baseline pixel values associated with the calibration image are caused by imperfections in the color sensor. At204, a color sensor adjustment matrix is determined based on a comparison of the captured first pixel values and the baseline pixel values associated with the calibration image. In one example, red, green, and blue values are extracted from the captured first pixel values for each of a plurality of panels of a calibration image, such as the calibration image ofFIG. 3. Those measured red, green, and blue values are used to populate a matrix:

Having determined the color sensor adjustment matrix, the method ofFIG. 2continues its calibration operations to determine one or more color temperature balance factors for use in performing white balance adjustments. At206, a second calibration image is illuminated by a first color temperature light of a known temperature. The second calibration image may be the same image as the first calibration image or the second calibration image may be a different image. In one example, the second calibration includes only the grayscale panels of the fourth row308of the calibration image302depicted inFIG. 3.

At208, the second pixel values are adjusted based on the color sensor adjustment matrix, such as according to:
S=PS×A,
where S is a matrix containing the adjusted second pixel values, PSis a matrix containing the captured second pixel values for each of the one or more utilized panels of the second calibration image, and A is the color sensor adjustment matrix. This operation adjusts the second pixel values for any imperfections in the color sensors of the camera. The adjusted second pixel values have then been controlled for those imperfections in the color sensors, and variances between the adjusted second pixel values and baseline values for the second calibration image panel pixel values are due to the illumination by the first color temperature light. At210, a first color temperature balance factor for the first color temperature light is determined based on the adjusted second pixel values. In one example, only grayscale panels in the second calibration image are utilized. Grayscale images are expected to have equal red, green, and blue pixel values. In that example, the first color temperature balance factor can be determined by calculating an amount of adjustment necessary to best equalize the red, green, and blue pixel values across the one or more grayscale panels of the second calibration image. In another example, the first color temperature balance factor is determined by comparing the adjusted second pixel values and the baseline red, green, and blue values for panels of the second calibration image. The first color temperature balance factor that is associated with the first color temperature light used to illuminate the second calibration image is stored for utilization in white balancing adjustments of pictures taken by a camera in production mode. For example, the red, green, and blue values of a captured image of a subject illuminated by the first color temperature light can be adjusted according to the first color temperature balance factor to improve the captured image.

Operations206,208, and210can be repeated multiple times to generate additional color temperature balance factors, where each of the color temperature balance factors is associated with a different calibration color temperature light used to illuminate the second calibration image at206. The multiple color temperature balance factors (e.g., one for each of one or more of a Hor light source, an A light source, a CW light source, a D50 light source, and a D65 light source) are then used to adjust images captured by the camera in production mode. In one example, the camera is configured to detect a color temperature illuminating a subject of a captured image. The camera selects a color temperature balance factor associated with a calibration color temperature light closest to the detected color temperature. In another example, the camera selects two color temperature balance factors associated with two calibration color temperature lights closest to the detected color temperature. A data processor of the camera interpolates between the two selected color temperature balance factors to generate a composite color temperature balance factor that is used to perform a white balance adjustment on the captured image.

FIG. 4is a block diagram depicting a camera and calibration module calibrating a digital camera by determining a color sensor adjustment matrix. The camera402includes a raw image capture module404. The raw image capture module404captures first pixel values of a first calibration image406that is illuminated by white light408. The first pixel values are stored in a memory410. A calibration module412, alone or in conjunction with a data processor414, accesses the first pixel values for processing. The calibration module412further receives baseline pixel values associated with the first calibration image406, such as from a calibration data store416. The calibration module412, which may take the form of a physical module or a software code module running on the camera402or an external computer system, processes the first pixel values and the baseline pixel values associated with the calibration image, such as through one of the processes described with respect toFIG. 2, to generate a color sensor adjustment matrix. That color sensor adjustment matrix is stored in the calibration data store416for subsequent uses, such as in additional calibration operations as well as image adjustment operations on a camera running in production mode.

FIG. 5is a block diagram depicting a camera and calibration module calibrating a digital camera by determining a color temperature balance factor using pixel values adjusted using a color sensor adjustment matrix. The camera502includes a raw image capture module504. The raw image capture module504captures second pixel values of a second calibration image506that is illuminated by first color temperature light508. The second pixel values are stored in a memory510. A calibration module512, alone or in conjunction with a data processor514, accesses the second pixel values for processing. The second pixel values are adjusted based on the color sensor adjustment matrix, such as the color sensor adjustment matrix determined in the example ofFIG. 4, which is accessed, for example from a calibration data store516. After adjusting the second pixel values using the color sensor adjustment matrix, the calibration module512determines a first color temperature balance factor based on the adjusted second pixel values.

The color sensor adjustment matrix and/or first color temperature balance factor are provided for storage on the memory of the camera510, and in some implementations, other cameras, and are used in providing color sensor adjustments and white balance adjustments of captured images in a production mode. In one example, a camera captures image data and detects that the subject of the image was illuminated by light of the first color temperature. The camera adjusts the captured image data based on the color sensor adjustment matrix and then further adjusts the captured image data based on the first color temperature balance factor to provide an improved picture.

FIG. 6is a block diagram depicting a digital camera having an internal calibration module. In the example ofFIG. 6, each camera includes a calibration mode where either or both of a color sensor adjustment matrix and a color temperature balance factor are determined specifically for the camera on which the calibration module is included. Implementations where individual digital cameras are calibrated enable adjustment parameters to be tailored to the specific hardware in each digital camera.

FIG. 7is a flow diagram depicting an example method of calibrating a digital camera. At702, a raw image of a color chart calibration image that illuminated in D50 is captured. At704, a sensor correction operation is performed. There, samples of RGB values for each of 24 panels on the color chart are taken and compared to baseline RGB values for those panels to determine a color sensor adjustment matrix according to:
A=(PT×P)−1×PT×O.
Having determined the color sensor adjustment matrix, captured images, both calibration and production, can be adjusted to compensate for camera hardware error according to S=P×A.

A white balancing operation for a captured image is illustrated at706. The example white balancing operation begins with a color temperature detection. Upon detecting a color temperature of the captured image, the white balancing operation706accesses a lookup table containing white balance factor records, where each of the white balance factor records is associated with a color temperature. The white balancing operation706identifies which two white balance factor records are associated with color temperatures closest to the detected color temperature of the captured image. The white balancing operation706interpolates between those two white balance factor records to determine white balancing factors (Kr, Kg, Kb) specific to the captured image's detected color temperature. The interpolated color balance factors are applied to the Red, Green, and Blue values of pixels of the captured image to generate an image that is color corrected according to S=P×A and white balanced according to the process at706.

FIG. 8is a block diagram depicting a digital camera. A digital camera802includes a raw image capture module804that captures an image of an object806through a lens808and stores the captured image in a memory810. A data processor812is configured to interact with the memory to adjust the captured image. The data processor812is configured access a color sensor adjustment matrix stored in a memory810of the digital camera802to perform a color adjustment transformation on the captured image. The data processor812is further configured to access one or more color temperature balance factors stored in a memory of the digital camera802and to use those balance factors to perform a white balancing adjustment on the captured image. The adjusted image data is stored to a memory810of the digital camera810, where an adjusted image814is accessible to a user (e.g., for viewing on a display of the digital camera802, for viewing on an external computer system, for printing, and for otherwise exporting).

This application uses examples to illustrate the invention. The patentable scope of the invention includes other examples.