# Patent

## Title
White balancing that uses values of white-balanced colors on the visible gamut's boundary

## Abstract

White balancing is performed using a mapping that maps the visible gamut's boundary into itself. A predefined color Win is mapped into a color Wout which is a white color or a color perceived as white under some viewing conditions. If Sin is some other color, a corresponding color Pin is determined on the visible gamut's boundary. Pin can be on the intersection of the visible gamut's boundary with a plane containing Sin and Win in a linear color coordinate system. Sin is mapped into a color Sout obtained from Wout and a value of the mapping on Pin.

## Background
BACKGROUND OF THE INVENTION
The present invention relates to digital image processing, and more particularly to color transformations of the kind typically associated with white balancing.
Images are affected by image viewing conditions, e.g. by background and surrounding colors and illumination. However, humans can adapt to the viewing conditions, and it may be desirable to adjust the digital representation of the image in accordance with human adaptation. For example, if a scene is illuminated by a red lamp, the scene may acquire a reddish tint. An object that would reflect white light under white illumination may reflect reddish light under red illumination and may look reddish. However, humans adapt to the red illumination and may stop noticing the reddish tint. If the image is faithfully captured by a digital camera, the reddish tint would be captured, and would be reproduced when the image is displayed with a printer or a computer monitor. If the reproduced image is viewed under white illumination, the viewer would notice the reddish tint because the white illumination would not allow the viewer to adapt to the tint. Therefore, it may desirable to remove the reddish tint from the reproduced image so that the viewer's perception of the reproduced image would approach the viewer's perception of the original image.
Human adaptation can be modeled by a “white balancing” process which transforms image colors using a linear transformation chosen based on human adaptation to a white color. Suppose for example that the scene illumination or some other viewing conditions change a white color W to some color Wnew (Wnew would be reddish in the red-lamp example above). Supposedly, humans would adapt to the viewing conditions and would perceive the color Wnew as W. Therefore, a linear transformation is chosen that maps the color Wnew into W. Then another linear transformation may be performed that would change W into a color that would be perceived as W under the viewing conditions of the reproduced image.
SUMMARY
This section summarizes some features of the invention. Other features are described in the subsequent sections. The invention is defined by the appended claims, which are incorporated into this section by reference.
The inventor has observed that a linear transformation may map a color into a region outside of the visible gamut. Also, the inventor has observed that human adaptation is not linear. For example, humans do not seem to adapt to at least some of the colors located on the visible gamut's boundary (some of such colors can be generated by monochromatic radiation; the boundary also includes other colors, e.g. purple).
In some embodiments of the present invention, a white balancing process maps the visible gamut boundary into itself. (The terms “mapping” and “transformation” are used interchangeably herein.) For example, each color on the visible gamut's boundary can be mapped into itself. In addition, as in prior art, some chosen color Wnew is mapped into a white color W (alternatively W may be a non-white color perceived as white under some viewing conditions; Wnew may or may not be white). Any multiple αWnew of the color Wnew is mapped into a multiple of W. The remaining colors (i.e. colors that are not on the visible gamut's boundary and are not multiples of Wnew) are mapped using some interpolation based on the mapping of the colors on the visible gamut's boundary and Wnew.
To illustrate some embodiments, we will use the subscript “in” for the starting colors (the input colors of the white-balancing process), and the subscript “out” for the output colors. Accordingly, we will denote Win=Wnew, and Wout=W The process maps Win into Wout, and maps each color Pin on the visible gamut's boundary into a color Pout on the visible gamut's boundary. (In some embodiments, Pout=Pin.) For the remaining colors Sin, some interpolation is used to determine the corresponding colors Sout. More particularly, suppose that Sin can be represented as some function (e.g. a linear combination) of the color Win and some color Pin on the visible gamut's boundary. Then Sout is generated as some function (e.g. a linear combination) of Wout and the respective color Pout (i.e. the value of the mapping on Pin). In some embodiments, the functions are chosen so that Sout is in the visible gamut.
The invention is not limited to the features and advantages described above. Other features are described below. The invention is defined by the appended claims.



BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 illustrates color vectors for some embodiments of the present invention.
FIG. 2 shows a flowchart of a white-balancing process according to some embodiments of the present invention, and a block diagram of a computer system executing the white balancing process according to some embodiments of the present invention.
FIGS. 3 and 4 illustrate color vectors and color coordinate systems used in some embodiments of the present invention.
FIG. 5 shows a flowchart of a white-balancing process according to some embodiments of the present invention.



DESCRIPTION OF SOME EMBODIMENTS
The embodiments described in this section illustrate but do not limit the invention. The invention is defined by the appended claims.
FIGS. 1 and 2 illustrate an exemplary white balancing process. Colors are shown as vectors in some color coordinate system. Visible gamut 110 is shown as a non-circular cone with a vertex O, but can be some other shape depending on the color coordinate system. Color Win is mapped into a white color Wout (the term “white” includes gray colors) or into some multiple of Wout (in a linear color coordinate system), i.e. into some color cWout where c is a positive constant. The color Wout can be, for example, a standard white point (e.g. CIE D50 or D65) with some predefined physical luminance, but the invention is not limited to any particular Wout definition. Non-white colors are also possible. For example, if the white balancing is performed to transform an image captured under red illumination to an image to be viewed under blue illumination, then Wout may be bluish white.
Any given input color Sin is mapped into a color Sout as shown in FIG. 2. In the example of FIG. 2, the image is captured by an image capture device 201 (e.g. a digital camera, an image synthesis system in a computer, or some other device). The image is white balanced by a software-programmed computer system including one or more computer processors 202 executing computer instructions stored in a memory 204 and/or received over a network (not shown), and/or by hardwired circuitry. The white-balanced image may then be reproduced by a device 205 (e.g. a computer monitor, a printer, or some other device).
The white balancing operation includes steps 206-230. If the input color Sin is a multiple of Win (in a linear color coordinate system), i.e. Sin=αWin for some non-negative number α, then Sout is set to αWout. See step 206. The white balancing for the color Sin can now be terminated, i.e. the subsequent steps are skipped.
If Sin lies on the visible gamut's boundary (step 208), then Sout is generated to be on the visible gamut's boundary, and the subsequent steps are skipped. Exemplary Sout computations include:

Sout=Sin  (1)

or

Sout=Sin·|Wout|/|Win|  (2)

where |•| denotes some measure of the magnitude, luminance or brightness, as discussed in more detail below.

If Sin is neither a multiple of Win nor on the visible gamut's boundary, then at step 210 a color Pin is determined on the visual gamut's boundary such that Pin corresponds to Sin in some predefined sense. For example, Pin can be such that Sin is a function of Pin and Win (and possibly other parameters). In some embodiments, the function is a linear combination, i.e.:

Sin=αPin+βWin  (3)

for some coefficients α, β determined at step 210.

The color Pin can be chosen in many ways. In the case of equation (3), in a linear color coordinate system, the color Pin lies in the plane containing the vectors Win and Sin, and hence on the intersection of this plane with the visible gamut's boundary. If the boundary is a conical surface, the intersection may be two rays originating from the origin O. Pin can be chosen on any of these rays. In some embodiments Pin is chosen so that Sin lies between Pin and Win (as in FIG. 1). In particular, in some embodiments, both α and β are positive. The magnitude of Pin is unimportant since αPin can be adjusted by the choice of α.
Suppose that the white balancing process maps the color Pin into some color Pout (determined at step 220, or possibly determined at step 208 performed when white-balancing the color Pin). At step 230, Sout is obtained as some function of Pout and Wout (and possibly of some other parameters). In some embodiments, the function is a linear combination:

Sout=α′Pout+β′Wout  (4)

where α′ and β′ are some coefficients such that Sout is in visible gamut 110. Of note, Sout will be in the visible gamut if α′ and β′ are non-negative and the visible gamut is a cone with an aperture of at most 180°.

The coefficients α′ and β′ can be chosen in different ways. In some embodiments, each of these coefficients is a continuous function of one or both of the coefficients α and β. For example, α=α′ and β=β′. In some embodiments, α′ and β′ are chosen so that Sin and Sout will have the same brightness. Brightness can be defined in different ways, e.g. as the physical luminance, the lightness in CIELUV, or the color vector's length in an orthonormal color coordinate system such as DEF. See U.S. patent application Ser. No. 11/321,443 filed Dec. 28, 2005 by S. N. Bezryadin, published as no. 2007/0146745 A1 on Jun. 28, 2007, incorporated herein by reference. In some embodiments, α′ and β′ are chosen to preserve the brightness ratio B(Sout)/B(Sin), where B(•) is the brightness function. Thus,

B(Sout)/B(Sin)=B(Wout)/B(Win)=const.  (5)

In some embodiments, the brightness is preserved on the visible gamut's boundary, but is increased or decreased inside the visible gamut:

B(Pout)=B(Pin)

B(Sout) increases or decreases when Sin moves from the visible gamut's boundary to Win.
Some embodiments preserve color saturation. Saturation represents the amount of white in the color (greater saturation means less white). For input colors Sin, saturation may represent the amount of color Win in Sin. For output colors Sout, saturation may represent the amount of Wout. Saturation can be defined in different ways. For example, saturation of Sin can be represented as the angle γ1 (FIG. 1) between Sin and Win. Saturation of Sout can be represented as the angle γ2 between Sout and Wout. In some embodiments, Sout is chosen so that

γ2=γ1  (6)

In particular, the color Sout can be determined as follows:

1. At step 210, determine Pin as some non-zero vector which, in a linear color coordinate system, lies in the intersection of (i) the plane containing the vectors Win, Sin and (ii) the boundary of visible gamut 110.
2. At step 220, determine Pout as described above.
3. At step 230, in the plane containing Pout and Wout, select the ray in the visible gamut such that the angle γ2 between the ray and Sout is equal to the angle γ1 between Sin and Win. This ray defines the direction of Sout. The ray is chosen to lie between Pout and Wout.
4. Finally, at step 230, determine the magnitude |Sout| of the vector Sout in some suitable way, for example to preserve the magnitude ratio, i.e.:

|Sout|=|Sin|·|Wout|/|Win|  (7)

If the magnitude is the brightness, then the brightness ratio will be preserved as in (5).

In some embodiments, equation (6) is replaced with preserving the saturation ratio:

C(Sin)/C(Pin)=C(Sout)/C(Pout)  (8)

where C(•) is saturation (which can be defined with reference to Win for input colors and with reference to Wout for output colors). Denote φ1=<(Sin,Pin), i.e. the angle between Sin and Pin, and φ2=<(Sout,Pout). If the saturation is defined using angles as for (6), then (8) can be implemented as:

γ1/(γ1+φ1)=γ2/(γ2+φ2)  (9)

Equation (9) uniquely defines γ2 because the sum γ2+φ2=<(Pout,Wout) is uniquely defined by Pout and Wout Thus, at step 230, γ2 is determined as:

γ2=γ1/(γ1+φ1)·(γ2+φ2)  (10)

Some embodiments use the equations (8)-(10) for small values of γ1 (e.g. γ1<0/1 radians), but use some other equation (e.g. a polynomial in γ1 of a degree greater than 1) for larger values of γ1. In some embodiments, γ2 is a continuous and/or monotonic function of γ1.

Computational complexity of the white balancing process may depend on the color representation. Some representations use a linear color coordinate system such that Wout lies on one of the coordinate axes. For example, if Wout is a D65 white point, a suitable color coordinate system is the orthonormal system DEF described in the aforementioned U.S. patent application Ser. No. 11/321,443 and illustrated in FIG. 3. Briefly, DEF is a linear transformation of the 1931 CIE XYZ color coordinate system for the 2° field:




  
    
      
        
          [
          
            
              
                D
              
            
            
              
                E
              
            
            
              
                F
              
            
          
          ]
        
        =
        
          
            A
            
              XYZ
              -
              DEF
            
          
          ⁡
          
            [
            
              
                
                  
                    
                      
                        X
                      
                    
                    
                      
                        Y
                      
                    
                  
                
              
              
                
                  Z
                
              
            
            ]
          
        
      
    
    
      
        (
        11
        )
      
    
  
  
    
      
        where
        ⁢
        
          :
        
      
    
    
      
        
      
    
  
  
    
      
        
          A
          
            XYZ
            -
            DEF
          
        
        =
        
          [
          
            
              
                0.205306
              
              
                0.712507
              
              
                0.467031
              
            
            
              
                1.853667
              
              
                
                  -
                  1.279659
                
              
              
                
                  -
                  0.442859
                
              
            
            
              
                
                  -
                  0.365451
                
              
              
                1.011998
              
              
                
                  -
                  0.610425
                
              
            
          
          ]
        
      
    
    
      
        (
        12
        )
      
    
  




The DEF system is orthonormal in the sense that its color matching functions d(λ), ē(λ), f(λ) form an orthonormal system in L2:

∫ d(λ)ē(λ)dλ=∫ d(λ) f(λ)dλ=∫ē(λ) f(λ)dλ=0  (13)

∫[ d(λ)]2dλ=∫[ē(λ)]2dλ=∫[ f(λ)]2dλ=K 

where K is a positive constant defined by the measurement units. The integrals are taken over the visible range of wavelengths λ.

The D axis (D>0 and E=F=0) corresponds to the CIE D65 white color standard. (D65 defines the chromaticity coordinates only.) For red colors produced by a monochromatic radiation with λ=700 nm, the coordinate F=0 and E>0. The visible spectrum is a non-circular cone extending upward from the vertex at the origin O (the origin is the black color).
Some embodiments use the spherical coordinate system BCH (Brightness, Chroma, Hue) based on DEF and described in the aforementioned U.S. patent application Ser. No. 11/321,443. For a color S with DEF coordinates (D,E,F), the BCH coordinates are:

B=|S|=√{square root over (D2+E2+F2)}  (14)

C (“chroma”) is the angle between the color S and the D axis;
H (“hue”) is the angle between (i) the orthogonal projection SEF of the vector S on the EF plane and (ii) the E axis.
For illustration, suppose that white balancing needs to be performed in the BCH coordinates so as to preserve the saturation ratio as in (9) and also preserve either the brightness B defined by (14) or the brightness ratio |Sout|/|Sin|. Suppose further that Pout=Pin at step 208 (FIG. 2). Denote the BCH coordinates of Sin as (Bin,Cin,Hin), and the BCH coordinates of Sout as (Bout,Cout,Hout). Denote further P=Pin, and the BCH coordinates of P as (BP,CP,HP). At step 230, Bout, Cout, Hout are computed as follows:

BoutBin  (15)

to preserve the brightness, or

Bout=Bin·|Wout|/|Win|  (16)

to preserve the brightness ratio. Further (see (10)):

Cout=CP·γ1/(γ1+φ1)  (17)

Hout=HP  (18)

In some embodiments, Hout is set to Hin.
FIGS. 4, 5 illustrate an embodiment which uses two possibly orthonormal coordinate systems: D′E′F′ for the input colors and D′′E′′F′′ for the output colors. D′E′F′ is chosen so that the color Win lies on one of the coordinate axes—the D′ axis. D′′E′′F′′ is chosen so that the color Wout lies on one of the coordinate axes—the D′′ axis. In some embodiments, D′′E′′F′′ is a linear transform (e.g. a rotation) of D′E′F′. In some embodiments, each of the two coordinate systems is a linear transform (e.g. a rotation) of DEF. One of D′E′F′, D′′E′′F′′ may be DEF.
The white balancing uses the spherical coordinate systems B′C′H′ and B′′C′′H′′ defined for D′E′F′ and D′′E′′F′′ respectively as in (14).
At step 510 (FIG. 5), the white-balancing system (e.g. a computer system as in FIG. 2) obtains the vectors Win, Wout in some color coordinate system, and selects the coordinate systems D′E′F′, D′′E′′F′′ as described above. At step 520, the input colors Sin are converted into the B′C′H′ coordinates if needed (i.e. if the input data are initially in some other coordinate system). At step 530, the white-balancing steps 206-230 of FIG. 2 are performed, for example, as follows.
Let us denote Pin=(BPin,CPin,HPin) in B′C′H′, and Pout=(BPout,CPout,HPout) in B′′C′′H′′. As seen in the D′E′F′ diagram in FIG. 4, if Pin is in the plane containing Sin and Win, then HPin=Hin, so determining Pin is facilitated at step 210. In some embodiments, the computer system sets CPin=C′max(Hin), defined as the maximum C′ value in the visible gamut for the fixed hue H′=Hin. The value BPin is unimportant, and can be determined in any suitable way, for example as BPin=1, or BPin=Bin, or BPin=|Win|.
At step 220, if Pout=Pin, then the Pout coordinates (BPout,CPout,HPout) in B′′C′′H′′ can be computed as a linear transform of (BPin,CPin,HPin). Full linear transform computation can be omitted. Indeed, BPout can be set to any suitable value as described above, e.g. BPout=BPin. Also, once HPout has been computed (as a linear transform or in some other way), then CPout can be computed as C′′max(HPout). In some embodiments, HPout is set to HPin to simplify the computations. Other computational algorithms are also possible.
At step 230, Sout can be computed using any of the techniques described above in connection with FIGS. 1-3, or using some other techniques. For example, Hout can be set to HPout. Bout can be computed using any of the techniques described above, e.g. Bout=Bin·|Wout|/|Win|. Cout can also be computed using any of the techniques described above, for example,

Cout=CPout·Cin/CPin  (19.1)

or

Cout=Cin  (19.2)

or using the equation (19.2) for small Cin values (e.g. Cin<0.1 radians) but using some other equation (e.g. polynomial of a degree greater than one) for larger Cin values.

The invention includes a machine-implemented (e.g. computer-implemented) method for white-balancing a digital image using a mapping that maps digital colors into digital colors, the method comprising obtaining information on a value Wout of said mapping on each of one or more reference colors Win. Each color Win is in a visible gamut but not on the visible gamut's boundary.
For at least one color Sin in the digital image such that Sin is not a multiple of any one of said one or more reference colors Win and is not on the visible gamut's boundary, the method comprises determining a value Sout of said mapping on the color Sin, wherein determining Sout comprises: determining one or more colors Pin on the visible gamut's boundary from the color Sin; and determining the value Sout from (i) one or more of said values of the mapping on the one or more reference colors, and (ii) a value of said mapping on each of the one or more colors Pin.
In some embodiments, the mapping maps the visible gamut's boundary into itself. Thus, each Pout is in the visible gamut's boundary. This is not necessary however. In some embodiments, each color on the visible gamut's boundary is mapped into itself under said mapping (i.e. the mapping is the identity mapping on the visible gamut's boundary).
In some embodiments, for each non-black color Sin, a ratio of a length of Sout to a length of Sin is independent of Sin, the length being a vector length in a predefined color coordinate system. In some embodiments, the predefined color coordinate system is 10%-orthonormal, or orthonormal.
In some embodiments, each reference color Win is perceived as white by a human being under predefined viewing conditions. In some embodiments, each color Wout is perceived as white by a human being under predefined viewing conditions. For example, if the viewing conditions include a blue lamp, Wout could be bluish white assuming that humans would adapt to the blue lamp and would perceive bluish white as white.
The invention includes computer systems programmed to perform the white balancing methods discussed above. The invention includes computer-readable media (disks, tapes, semiconductor memories, and possibly others) comprising a computer program for causing a computer system to perform the white balancing methods described above. The invention covers downloading such computer programs over a network and in particular covers network transmission methods comprising transmission of such computer programs over a network.
The invention is not limited to the embodiments described above, and in particular to any computations or color coordinate systems. Other embodiments and variations are within the scope of the invention, as defined by the appended claims.

## Claims

1. A machine-implemented method for white-balancing a digital image using a mapping that maps digital colors into digital colors, the method comprising:
obtaining, by the machine, information on a value Wout of the mapping on a reference digital color Win lying in a visible gamut but not on the visible gamut's boundary, the mapping of Win into Wout being to adapt Win to viewing conditions different from those in the image, wherein the mapping maps digital colors into digital colors and therefore each value of the mapping (including the value Wout) is a digital color;
for at least one digital color Sin in the digital image such that Sin is not a multiple of the reference digital color Win and is not on the visible gamut's boundary, determining, by the machine, a value Sout of said mapping on the digital color Sin, wherein determining Sout comprises:
determining, by the machine, a digital color Pin on the visible gamut's boundary from the digital color Sin; and
determining, by the machine, the value Sout from said information on the value Wout and from a value Pout of said mapping on the digital color Pin.

2. The method of claim 1 wherein said mapping maps the visible gamut's boundary into itself.

3. The method of claim 1 wherein each color on the visible gamut's boundary is mapped into itself under said mapping.

4. The method of claim 1 wherein the color Pin is a linear combination of the color Sin and the reference color Win.

5. The method of claim 1 wherein in a linear color coordinate system, the color Pin lies in a plane containing the reference color Win and the color Sin.

6. The method of claim 1 wherein the color Sout is a linear combination of the color Wout and the value Pout.

7. The method of claim 1 wherein in a linear color coordinate system, the color Sout lies in a plane containing the color Wout and the value Pout.

8. The method of claim 1 wherein for each non-black color, a ratio of a length of a value of said mapping on the non-black color to a length of the non-black color is independent of the non-black color, the length being a vector length in a predefined color coordinate system.

9. The method of claim 8 wherein the predefined color coordinate system is orthonormal.

10. The method of claim 1 wherein the reference color Win is perceived as white by a human being under predefined viewing conditions.

11. The method of claim 1 wherein Wout is perceived as white by a human being under predefined viewing conditions.

12. The method of claim 1 wherein the mapping maps coordinates of Sin in a first color coordinate system into coordinates of Sout in a second color coordinate system different from the first color coordinate system.

13. A computer system programmed to perform the method of claim 1.

14. Non-transitory computer-readable media comprising a computer program for causing a computer system to perform the method of claim 1.

15. A network transmission method comprising transmitting a computer program comprising one or more computer instructions over a network for downloading each instruction into a non-transitory storage of a computer system, the computer program being for causing the computer system to perform the method of claim 1.

16. Non-transitory computer-readable media comprising a computer program for causing a computer system to perform the method of claim 2.

17. A network transmission method comprising transmitting a computer program comprising one or more computer instructions over a network for downloading each instruction into a non-transitory storage of a computer system, the computer program being for causing the computer system to perform the method of claim 2.

18. Non-transitory computer-readable media comprising a computer program for causing a computer system to perform the method of claim 3.

19. A network transmission method comprising transmitting a computer program comprising one or more computer instructions over a network for downloading each instruction into a non-transitory storage of a computer system, the computer program being for causing the computer system to perform the method of claim 3.

20. Non-transitory computer-readable media comprising a computer program for causing a computer system to perform the method of claim 4.

21. A machine-implemented method for white-balancing a digital image using a mapping that maps digital colors into digital colors to adapt at least one digital color in the digital image to viewing conditions different from those in the digital image, the method comprising, for each digital color Sin in the digital image, determining, by a machine, a value of said mapping on the digital color Sin, wherein:
said mapping maps each digital color Sin on a visible gamut's boundary into the visible gamut's boundary;
said mapping maps at least one digital color Sin which is not on the visible gamut's boundary into a digital color different from Sin and not on the visible gamut's boundary to adapt the digital color Sin to said viewing conditions.

22. The method of claim 21 wherein said mapping maps each color Sin on the visible gamut's boundary into itself.

23. The method of claim 21 wherein said mapping maps each color Sin in the visible gamut but not on the visible gamut's boundary into a color in the visible gamut but not on the visible gamut's boundary.

24. A computer system programmed to perform the method of claim 21.

25. Non-transitory computer-readable media comprising a computer program for causing a computer system to perform the method of claim 21.

26. A network transmission method comprising transmitting a computer program comprising one or more computer instructions over a network for downloading each instruction into a non-transitory storage of a computer system, the computer program being for causing the computer system to perform the method of claim 21.

27. A computer system programmed to perform the method of claim 22.

28. Non-transitory computer-readable media comprising a computer program for causing a computer system to perform the method of claim 23.

29. A network transmission method comprising transmitting a computer program comprising one or more computer instructions over a network for downloading each instruction into a non-transitory storage of a computer system, the computer program being for causing the computer system to perform the method of claim 23.