Abstract:
A method of correcting a digital image comprises the steps of removing unrepresentative pixels, measuring the mean levels of the remaining pixels for each channel, determining an aim mean for each channel from the measured aim mean, determining a correction level, calculating a power law correction for each channel from the aim mean and the correction level and applying the power law correction to the input image to give a corrected output image.

Description:
CROSS REFERENCE TO RELATED APPLICATIONS  
       [0001]    This is a U.S. original patent application which claims priority on Great Britain patent application No. 0211486.6 filed May 18, 2002. 
     
    
     
       FIELD OF THE INVENTION  
         [0002]    This invention relates to the manipulation of digital images, in particular to the color balancing of digital images.  
         BACKGROUND OF THE INVENTION  
         [0003]    The proportion of digital images requiring printing or display is growing with the increasing use of digital images in the home, office and on the internet. The originating equipment is typically a digital camera or scanner. In the case of a digital camera, the scene illumination may be less than ideal, resulting in a color balance error on the final image. In the case of a print scanner the color cast in the original print could give an unacceptable scanned version of the print. Even if the originating equipment provides an accurate ‘well balanced’ image any subsequent processing may introduce color casts that need correction.  
           [0004]    There are a large number of white balance patents for digital and video cameras but they have the advantage of sampling the color of the light before any non-linear processing and subsequent clipping. U.S. Pat. No. 4,883,360 and U.S. Pat. No. 4,954,884 are typical for digital cameras and U.S. Pat. No. 4,638,350 is typical for video cameras.  
           [0005]    There are also a large number of patents relating to color balance of photographic printers where they have to accommodate the large range of exposures on color negative film. Many make use of tone scale statistics within the image such as that taught in EP 1014695 and other more complex methods such as taught in U.S. Pat. No. 4,677,465 and U.S. Pat. No. 4,729,016. The consequence is that they have to be more aggressive than is needed for digital cameras resulting in the possibility of a greater number of color balance failures than would be the case with a restricted color balance range. All of these rely on being able to process accurate unclipped image data which is not available from digital images submitted for printing.  
           [0006]    The problem with color balancing digital cameras is that clipped highlights and shadows remain neutral even when the mid-range image data requires color balancing. This invention aims to be able to take any digital image and correct it for color balance whilst maintaining the neutrality of any highlights or shadows.  
         SUMMARY OF THE INVENTION  
         [0007]    According to the present invention there is provided a method of correcting a digital image comprising a plurality of channels, the method comprising the steps of eliminating unrepresentative pixels, measuring the mean level of the remaining pixels for each channel, determining an aim mean for each channel from the measured mean levels, determining a correction level, calculating a power law correction for each channel from the aim mean level and the correction level and applying the power law correction to the input image data to give a corrected output image.  
           [0008]    Preferably the unrepresentative pixels are eliminated by means of at least one mask.  
           [0009]    The invention allows any digital image to be color balanced whilst maintaining neutral shadows and highlights. The images can be color balanced without knowing the source of the image. No operator intervention is required. 
       
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0010]    The invention will now be described, by way of example, with reference to the accompanying drawings, in which:  
         [0011]    [0011]FIG. 1 is a block diagram illustrating the method of the invention;  
         [0012]    [0012]FIG. 2 is a block diagram illustrating the steps taking place in block  1  of FIG. 1;  
         [0013]    [0013]FIG. 3 is a block diagram illustrating the steps taking place in block  9  of FIG. 2;  
         [0014]    [0014]FIG. 4 is a block diagram illustrating the steps taking place in blocks  2  and  3  of FIG. 1;  
         [0015]    [0015]FIG. 5 is a block diagram illustrating the steps taking place in block  5  of FIG. 1;  
         [0016]    [0016]FIG. 6 is a block diagram illustrating the steps taking place in block  6  of FIG. 1; and  
         [0017]    [0017]FIG. 7 is a perspective diagram of a computer system for implementing the present invention. 
     
    
     DETAILED DESCRIPTION OF THE INVENTION  
       [0018]    In the following description, a preferred embodiment of the present invention will be described in terms that would ordinarily be implemented as a software program. Those skilled in the art will readily recognize that the equivalent of such software may also be constructed in hardware. Because image manipulation algorithms and systems are well known, the present description will be directed in particular to algorithms and systems forming part of, or cooperating more directly with, the system and method in accordance with the present invention. Other aspects of such algorithms and systems, and hardware and/or software for producing and otherwise processing the image signals involved therewith, not specifically shown or described herein, may be selected from such systems, algorithms, components and elements known in the art. Given the system as described according to the invention in the following materials, software not specifically shown, suggested or described herein that is useful for implementation of the invention is conventional and within the ordinary skill in such arts.  
         [0019]    Still further, as used herein, the computer program may be stored in a computer readable storage medium, which may comprise, for example; magnetic storage media such as a magnetic disk (such as a hard drive or a floppy disk) or magnetic tape; optical storage media such as an optical disc, optical tape, or machine readable bar code; solid state electronic storage devices such as random access memory (RAM), or read only memory (ROM); or any other physical device or medium employed to store a computer program.  
         [0020]    Before describing the present invention, it facilitates understanding to note that the present invention is preferably utilized on any known computer system, such a personal computer. Consequently, the computer system will not be discussed in detail herein. It is also instructive to note that the images are either directly input into the computer system (for example by a digital camera) or digitized before input into the computer system (for example by scanning an original, such as a silver halide film).  
         [0021]    [0021]FIG. 1 illustrates the method of the invention.  
         [0022]    The method can be divided into three stages. The first stage, the analysis of the image data to extract color balance information, takes place in blocks  1 ,  2 ,  3  and  4  in FIG. 1. The second stage, a calculation stage using the extracted information to give a corrected figure, takes place in block  5 . The third stage, the final application of the correction on the image data, takes place in block  6 . Each of these stages will be described in more detail below.  
         [0023]    In block  1  the analysis process uses a number of masks to remove unrepresentative pixels from the color analysis. Typically the masks eliminate pixels which are clipped or which are highly saturated. This is described in more detail below, with respect to FIG. 2. Block  1  additionally measures the extent of the neutral highlights and shadows and passes the information on to block  6 . In block  2  the mean level of the pixels remaining after the masking of block  1  is measured to obtain a measure of the color balance of the image. In block  4  a weighting generator generates two spatial weighting functions. These weighting functions are used to weight mean level measurements of the input image toward the centre of the image or toward the periphery of the image. For the color balance measurement, measured in block  2 , the weighting is toward the centre of the image to give a greater bias toward the likely position of the subject of the image. These weighting functions, although preferred, are not essential to the invention.  
         [0024]    In block  3  two mean levels of the input image are measured, one with a centre weighting and one with a peripheral weighting. The ratio of the two measurements is output from block  3  and used to set the correction level used in block  5 . In calculation block  5  the measurement of the color balance from block  2  is combined with the measurements from block  3  to generate a new aim mean level for each color or channel. A power law correction, based on the required shift in mean level, is calculated for each color or channel and sent to the output implementation block, block  6 . In block  6  the input data is normalized and the neutral levels masked out. Each color separation of the remaining data is passed through the corresponding power law correction, to move the mean level of each separation toward its new aim mean level. The corrected data is then re-scaled and added back to the neutral highlight and shadow data to give a color corrected output.  
         [0025]    [0025]FIG. 2 is an expansion of block I in FIG. 1. In the embodiment illustrated the input data for analysis is first sub-sampled, in block  7 , to reduce the computing time and then low-pass filtered, in block  8 , to remove noise. However these are not essential steps to the invention. The conditioned data, i.e. the sub-sampled and filtered data, is then passed to the various masks, blocks  9 ,  10 ,  11  and  12 , before the mean level analysis, to determine the color balance of the image.  
         [0026]    An edge inclusion mask is generated, in block  9 , in order to measure the mean level of just the edge pixels, i.e. where detail is found in the image. This eliminates uniform areas of the image from the measurement. FIG. 3 is an expansion of block  9 . The output from block  8  is further low pass filtered in block  18 . In block  19  a low pass filtered version of the analysis data is subtracted from itself and generates a high pass filtered signal. The standard deviation of the high pass filtered signal is measured in block  20 . This standard deviation is used to set an adaptive threshold for the generation of the edge detail mask in block  21 . A scaling factor, ‘Scale’ in equation 1, for the standard deviation can be set to optimize the resulting mask. This could, for example, be 0.5.  
         EdgeMask=1 when |Detail|&gt;( Std *Scale)  (1)  
         [0027]    A highlight exclusion mask is generated in block  10 . This highlight exclusion mask eliminates triply clipped and near triply clipped pixels from the analysis. These pixels will always be neutral and can only detract from the color measurement. The mask is true if all three channels are above the white threshold. The minimum code value for which the highlights are neutral is set empirically. Generally it will be maximum digital data value, and this value will be used in the implementation block, block  6 . Peculiar to digital camera images is the possibility that the camera processing has already made the shadows neutral. This is not true of all digital cameras but if the shadows are already neutral the color balancing process of the present invention should not make them colored.  
         [0028]    A shadow exclusion mask is generated in block  11 . This shadow exclusion mask detects the neutral shadows and eliminates those pixels from the analysis. The maximum code value for which the shadows are neutral is determined, generally it will be the data minimum value. This value is typically zero but may go up to 20, for example, in an 8 bit system. This value will be used later in the implementation block, block  6 . Together, blocks  10  and  11  determine the neutral levels.  
         [0029]    A third exclusion mask, a saturation mask, is generated in block  12 . This is not essential to the invention but is a preferred feature. This mask eliminates highly saturated areas of the image from the analysis. The saturation threshold is adaptive. Equations 2, 3 and 4 below are shown as an example set of equations for deriving the saturation mask. In this example the mask is derived by taking the sum of the absolute differences between an equal weight neutral and each color, as shown in equations 2 and 3. The maximum level of the saturation from equation 3 is used to set an adaptive threshold for the saturation mask generation, equation 4.  
         Neutral=(Red+Green+Blue)/3  (2)  
         Saturation=(|Neutral−Red|+|Neutral−Green|+|Neutral−Blue|)  (3)  
         [0030]    Saturation mask=1 when  
         (|Neu−Red|)&gt;thresh)AND(|Neu−Green|)&gt;thresh)AND(|Neu−Blue|)&gt;thresh)  (4)  
         [0031]    Where Neu is the Neutral from equation 2  
         [0032]    Thresh=saturation scale factor * maximum saturation from equation 3  
         [0033]    The ‘saturation scale factor’ is optimized during development. A typical value could, for example, be 0.4.  
         [0034]    The three exclusion masks, from blocks  10 ,  11  and  12 , are combined into a composite exclusion mask in block  17 . This mask is then applied to the image data in blocks  13  and  14 . In block  13  the mean level of the whole image is measured. In block  14  the mean level of the edge, i.e. detail, of the image is measured. The outputs from blocks  13  and  14  are passed to the centre weighted mean level measurement block, block  2  in FIG. 1, to generate two sets of mean level data, representative of the color balance of the image. The image data for the color correction level can be optionally masked, by highlight and shadow clip masks, in block  32  before being passed to block  24 , for the centre weighted mean level measurement, and block  25  for the peripheral mean level measurement. Blocks  24  and  25  are explained further with respect to FIG. 4.  
         [0035]    [0035]FIG. 4 is an expansion of blocks  2  and  3  from FIG. 1.  
         [0036]    The weighting functions generated in block  4  are used to generate the weighting plane, using an offset gausian or cosine function. An offset level is used to set the peripheral weighting, i.e. the weighting for the periphery of the image, and a scaling factor is used to set the central weighting, i.e. the weighting for the centre of the image. The scale could be set to clip the central area to give it a uniform weighting.  
         [0037]    In block  22  the means for each color for the edge, i.e. detail, masked image are determined using a centre weighted histogram. In block  23  the means for each color for the whole masked image are determined using a centre weighed histogram. Block  23  is preferable but not essential to the invention. The means are a measure of the color balance of the image and are used to determine the aim mean in the calculation block, block  5 . The spatial weighting in the histogram varies the equivalent value of a pixel with respect to its position in the image, i.e. a pixel of a particular data value will always contribute to the same histogram bin, but unless it is at the centre of the picture it will not be worth a full pixel. In blocks  24  and  25  the means of the filtered image data, from block  32 , are determined using centre weighted histograms (in block  24 ) and peripheral weighted histograms (in block  25 ). These means, i.e. the outputs of blocks  24  and  25  are used to determine the color correction level of the aim means calculated from the outputs of blocks  22  and  23  in the calculation block, block  5 .  
         [0038]    [0038]FIG. 5 is an expansion of block  5  in FIG. 1. In block  26  the aim mean level for each color is determined, using the outputs of blocks  22  and  23  and any preferred color balance set by parameters, e.g. Red pref  and Blue pref . In block  27  the color correction level is determined using the outputs of blocks  24  and  25 . The aim mean level and the color correction level are combined in block  28  to give the power law corrections for the implementation block, block  6 , in  10  FIG. 1.  
         [0039]    In block  26  the mean level for each color channel is comprised of a combination of the edge pixel mean (edgemean) and the masked mean of the whole image (maskedMean). Equation 5 is an example of the calculation used for the red channel. It will be understood that other color channels would be similarly calculated.  
           R   mean   =A*R   edgemean   +B*R   maskedMean   (5)  
         [0040]    Where ‘A’ and ‘B’ are each between 0 and 1 but also where A+B=1  
         [0041]    An average of the color mean levels is calculated and used as the neutral aim level, Aim average . The neutral aim level may not necessarily be an equal weight neutral. An example is shown in equation 6 below.  
         Aim Average =( R   mean   +G   mean   +G   mean   +B   mean )/4  (6)  
         [0042]    At this point the aim mean level for each color is the same in order, ultimately, to give a neutral color balance to the image. However it may be preferred to have a warmer balance to the image. In this case Red pref  and Blue pref  could be used.  
         [0043]    To modify the red channel the following equation would be used:  
         Red aim =Aim Average *Red pref   (7)  
         [0044]    A similar equation would be used to determine Blue aim .  
         [0045]    Red pref  could for example have a value of 1.05 and Blue pref  could have a value of 0.95.  
         [0046]    The color correction level is calculated in block  27 . This sets the level to which the color means calculated in equation 5 are moved toward the Aim Average  from equations 6 or 7. It is set by the difference in mean level between the centre weighted means and the peripheral weighted means for each color. The difference can be measured in a number of ways. Equation 8 below gives an example based on normalized difference. The level of color correction is set by the normalized difference between the centre and peripheral means of the image data obtained from blocks  24  and  25 . The bigger the difference the lower the correction. The equation is weighted toward green, but it will be understood that any combination could be used to suit the range of images being balanced.  
                   Difference   =         (          RcentreMean   -   RedgeMean          )       (     RcentreMean   +   RedgeMean     )       +     2   *                                    (          GcentreMean   -   GedgeMean          )       (     GcentreMean   +   GedgeMean     )       +                                (          BcentreMean   -   BedgeMean          )       (     BcentreMean   +   BedgeMean     )                     (   8   )                               
 
         [0047]    The ‘difference’ factor may then be scaled and raised to a power, when optimizing the algorithm to the image population being processed, to give the final color correction factor, C correction .  
         e.g.  C   correction .=(Difference*scale) power   (9)  
         [0048]    In block  28  the aim mean levels and the color correction level are combined to give the corresponding differential power law corrections. At this point the level of correction for each color can also be varied to suit the population images being processed. The power law correction is calculated from normalized data and an example is shown in equation 10.  
           G   correction =(log 10  Aim Average )/(Log 10 (( G   mean   +C   correction *Aim Average )/(1+C correction )))  (10)  
         [0049]    A similar correction is made for the red and blue channels. FIG. 6 is an expansion of block  6  of FIG. 1.  
         [0050]    In block  6  the power law correction obtained from block  28  is applied to the input data after removal of the neutral levels and the appropriate color mask in block  29 . The color mask is generated in block  31 . Equation 11 given below, is used to generate the mask. Green is used as the example.  
         G mask =1 when (G in &lt;nlev)  (11)  
         [0051]    where nlev is the neutral level and G in  is the green input data  
         [0052]    In block  29  the neutral level data is subtracted and the remaining data power law corrected.  
           G   corrected =(( G   in   −nlev )* G{overscore ( mask ){circumflex over ( )})}   G   correction   (12)  
         [0053]    Finally in block  30  the corrected data, the masked green data and the neutral data is recombined to give the color corrected output.  
           G   out   =G   corrected   +G   mask   *G   in   +G   m {overscore ( ask   *n )} lev   (13)  
         [0054]    Where G in  is the green input data, nlev is the neutral level and G mask  is the green mask.  
         [0055]    It will be understood that similar corrections are made to the red and blue channels to give color corrected red and blue outputs.  
         [0056]    The correction data may be optionally low pass filtered in block  33 , to reduce any noise addition or missing codes due to the non-linear nature of the correction. If the correction data is being low pass filtered, then G out , equation  13 , is subtracted from the filtered G in  to give just the correction data which is then added onto the full band image data in block  30 .  
         [0057]    The present invention, either as a stand alone algorithm or a component part of a broader algorithm may be included in a range of applications in which digital images are input, images captured digitally, or film images scanned; digital image processing performed; and digital images output, stored or printed. These applications include digital processing laboratories (mini and wholesale), kiosks, internet fulfillment, and commercial or home computer systems with appropriate peripherals, digital cameras and so on.  
         [0058]    Referring to FIG. 7, there is illustrated a computer system  110  for implementing the present invention. Although the computer system  110  is shown for the purpose of illustrating a preferred embodiment, the present invention is not limited to the computer system  110  shown, but may be used on any electronic processing system. The computer system  110  includes a microprocessor-based unit  112  for receiving and processing software programs and for performing other processing functions. A display  114  is electrically connected to the microprocessor-based unit  112  for displaying user-related information associated with the software, e.g., by means of a graphical user interface. A keyboard  116  is also connected to the microprocessor based unit  112  for permitting a user to input information to the software. As an alternative to using the keyboard  116  for input, a mouse  118  may be used for moving a selector  120  on the display  114  and for selecting an item on which the selector  120  overlays, as is well known in the art.  
         [0059]    A compact disk-read only memory (CD-ROM)  122  is connected to the microprocessor based unit  112  for receiving software programs and for providing a means of inputting the software programs and other information to the microprocessor based unit  112  via a compact disk  124 , which typically includes a software program. In addition, a floppy disk  126  may also include a software program, and is inserted into the microprocessor-based unit  112  for inputting the software program. Still further, the microprocessor-based unit  112  may be programmed, as is well known in the art, for storing the software program internally. The microprocessor-based unit  112  may also have a network connection  127 , such as a telephone line, to an external network, such as a local area network or the Internet. A printer  128  is connected to the microprocessor-based unit  112  for printing a hardcopy of the output of the computer system  110 .  
         [0060]    Images may also be displayed on the display  114  via a personal computer card (PC card)  130 , such as, as it was formerly known, a PCMCIA card (based on the specifications of the Personal Computer Memory Card International Association) which contains digitized images electronically embodied in the card  130 . The PC card  130  is ultimately inserted into the microprocessor based unit  112  for permitting visual display of the image on the display  114 . Images may also be input via the compact disk  124 , the floppy disk  126 , or the network connection  127 . Any images stored in the PC card  130 , the floppy disk  126  or the compact disk  124 , or input through the network connection  127 , may have been obtained from a variety of sources, such as a digital camera (not shown) or a scanner (not shown). In accordance with the invention, the algorithm may be stored in any of the storage devices heretofore mentioned and applied to images in order to color balance the images.  
         [0061]    It is to be understood that various modifications and changes may be made without departing from the present invention, the present invention being defined by the following claims.