Method for automatic partial white balance correction

A method for automatic partial correction of color images for non-white illumination. The images are processed in a manner similar to that by which the visual system processes signals related to color vision to achieve "color constancy".

FIELD AND BACKGROUND OF THE INVENTION 
The present invention relates to image processing and, more particularly, 
to a method for partially correcting color images for colored illumination 
without knowledge of either the color or the intensity of the 
illumination. 
The perceived colors of visible objects are determined, both by the physics 
of light reflection and by the way in which the visual system processes 
the reflected light that it receives. With regard to the physics, the 
physical color of the light reflected to the visual system by visible 
objects is determined, both by the reflectivity spectra of their surfaces, 
and by the color spectrum of the illuminating light. For an illuminated 
object to not appear dark, the illuminating light must include colors that 
the object reflects. For example, a red object looks red in light, such as 
red light or white light, whose spectrum includes a substantial red 
component, but looks dark in light, such as blue light, whose spectrum 
lacks a red component. 
Through a physiological phenomenon called "color constancy", the visual 
system is capable of partially correcting perceived colors to compensate 
for colored illumination. For example, white sheets of paper look 
substantially white both at noon, when daylight is predominantly bluish, 
and at sunset, when daylight is predominantly reddish. 
Photographs of scenes, including both still pictures and motion pictures, 
whether recorded by analog means (photographic film) or digital means 
(video cameras), normally are perceived differently from the way the 
scenes themselves would be perceived by direct vision. At least two means 
are known in the art for achieving "white balance", i.e., correcting for 
departures from whiteness of the illuminating light. Video cameras 
typically have manual means for achieving white balance. These means 
require that the video camera be aimed manually at a reference surface 
that is assumed to be white under white illumination, to record parameters 
related to the spectrum of the illumination so that the subsequently 
recorded pictures may be corrected for the non-whiteness of the 
illumination. Furthermore, the illumination spectrum may change suddenly, 
for example, if a cloud passes in front of the sun, or if the object being 
photographed moves from sunlight to shade. These changes in illumination 
degrade the accuracy of the white balance correction. More advanced video 
cameras often include automatic white balance mechanisms, but these are 
not entirely satisfactory. 
There is thus a widely recognized need for, and it would be highly 
advantageous to have, a more satisfactory method for performing at least a 
partial white balance correction, either automatically or interactively, 
without knowing the illumination spectrum. 
SUMMARY OF THE INVENTION 
According to the present invention there is provided a method for partially 
correcting a scene for illumination color, the scene including an 
intensity spectrum at each of a plurality of pixels arranged in a 
rectangular grid, the method comprising the steps of: at each pixel: (a) 
multiplying the intensity spectrum by a spectral response function of a 
red photoreceptor, thereby providing a red spectral product; (b) 
multiplying the intensity spectrum by a spectral response function of a 
green photoreceptor, thereby providing a green spectral product; (c) 
multiplying the intensity spectrum by a spectral response function of a 
blue photoreceptor, thereby providing a blue spectral product; (d) 
integrating the red spectral product; (e) integrating the green spectral 
product; and (f) integrating the blue spectral product; thereby providing 
a red image, a green image, and a blue image, each image having a pixel 
value at each of the plurality of pixels. 
The underlying concept of the present invention is to process color 
pictures in a manner similar to that in which the neurons of the visual 
system process signals related to color vision to achieve color constancy. 
Ideally, the input to the present invention is the intensity spectrum of a 
scene as a function of wavelength, measured at each pixel in a rectangular 
array of pixels. This intensity spectrum is multiplied by the spectral 
response function of each of the types of photoreceptor cells of the 
retina (red cones, green cones, and blue cones) to incident light, and 
integrated with respect to wavelength, thereby providing, at each pixel, a 
red intensity value, a green intensity value, and a blue intensity value. 
Collectively, the red values, the green values, and the blue values are 
examples of what is referred to herein as "images": rectangular arrays of 
values, one value per pixel. These values then are processed according to 
the algorithm of the present invention to provide images corrected for 
non-white illumination. 
This ideal input rarely is attainable. Therefore, the scope of the present 
invention includes the processing of images obtained by other means. For 
example, the three input images may be in the form of analog signals from 
transducers whose spectral responses are similar to the spectral responses 
of cone cells, in which case the intensity values are electrical signals, 
typically voltage levels. These analog signals may be processed directly, 
using an embodiment of the present invention in analog hardware. 
Alternatively, these analog signals may be digitized, and processed 
digitally according to the present invention. Usually, however, the input 
to the present invention consists of digital images, such as are acquired 
by video cameras, that come ultimately from transducers whose spectral 
responses does not match the responses of cone cells. In that case, the 
digital pixel intensity values must be transformed to photoreceptor 
response coordinates, or "fundamentals", corresponding to the spectral 
responses of the three types of cones. 
The most common color coordinate system for digital color images is the 
so-called red-green-blue, r-g-b, or chromaticity, coordinates. Digital 
images in other three-color schemes, such as yellow-cyan-magenta, may be 
transformed mathematically to r-g-b. The transformation from r-g-b 
coordinates, or from CIE x-y-z coordinates, to photoreceptor coordinates 
may be found, for example, in G. Wyszecki and W. S. Styles, "Color 
Science" (Wiley, 1982), pages 139 and 615. In what follows, all references 
to "red", "green" and "blue" will be to photoreceptor response 
coordinates, and not to chromaticity coordinates. 
The present invention includes an algorithm in the spirit of that presented 
by Ronen Dahari and Hedva Spitzer in an article titled "Spatiotemporal 
adaptation model for retinal ganglion cells", published in the Journal of 
the Optical Society of America Series A, Volume 13 Number 3 (March 1996), 
which article is incorporated by reference for all purposes as if fully 
set forth herein. The paper by Dahari and Spitzer presents a model for the 
adaptation of visual perception to changing intensity of illumination. It 
has been conjectured that color constancy works by an analogous mechanism, 
with modifications as described herein. 
FIG. 1 is a schematic cross section of the human retina, showing that the 
retina consists of five layers of cells, receptors 1, horizontal cells 2, 
bipolar cells 3, amacrine cells 4, and retinal ganglion cells 5. Receptors 
1 shown here are rod cells, which respond to light intensity, rather than 
color. The mechanism of adaptation to changing intensity modeled by Dahari 
and Spitzer operates in achromatic retinal ganglion cells 5. The receptive 
field of achromatic ganglion cell 5 includes both receptors 6 of the 
center receptive field area and receptors 7 of the surround receptive 
field area. The responses of center area receptors 6 and surround area 
receptors 7 are combined in achromatic ganglion cell 5, in one of two 
different ways, depending on the type of achromatic ganglion cell 5. An 
"on-center" cell responds positively to signals of increased light 
intensity from the center area of the receptive field and negatively to 
signals of increased light intensity from the surround area of the 
receptive field. An "off-center" cell responds negatively to signals of 
increased light intensity from the center area of the receptive field and 
positively to signals of increased light intensity from the surround area 
of the receptive field. 
The mechanism of color constancy is analogous, with two main modifications. 
The first is that color must be considered. There are six main groups of 
retinal ganglion cells 5 involved in color perception, corresponding to 
the three kinds of cone cells that respond to color analogously to the 
response of rod cells 1 to intensity. FIG. 2 shows schematically the 
receptive field processed by an on-center achromatic retinal ganglion cell 
5, showing that the intensity response 9 from surround area rod cells 7 is 
subtracted from the intensity response 8 from center area rod cells 6. 
Similarly, a first group of on-center red-processing ganglion cells 
modifies the cell response of red light by subtracting green surround 
responses from red center responses; a second group of on-center 
green-processing ganglion cells modifies the cell response of green light 
by subtracting red surround responses from green center responses; and a 
third group of on-center blue-processing ganglion cells modifies the cell 
response of blue light by subtracting yellow surround responses, i.e., a 
combination of red surround responses and green surround responses, from 
blue center responses. 
The second modification is that the perception of color is further modified 
by responses from "remote" areas of the receptive field that are even 
farther than the "surround" areas from the "center" areas. This is 
believed also to occur at the ganglion cell level. 
The present invention imitates this mechanism of color perception to 
provide a partial automatic white balance correction for a color picture, 
under the assumption that the average scene imaged in the picture is 
"gray", i.e., that all three colors are of roughly equal intensities in 
the scene as a whole when viewed in white light. Corrections corresponding 
to those performed in the retinal ganglion layer are performed 
computationally to the three intensity images that constitute the picture.

DESCRIPTION OF THE PREFERRED EMBODIMENTS 
The present invention is of a method of emulating the color constancy 
mechanism of visual perception. Specifically, the present invention can be 
used to perform a partial automatic "white balance" correction of color 
pictures without knowledge of the illumination spectrum. 
The principles and operation of a "white balance" correction method 
according to the present invention may be better understood with reference 
to the drawings and the accompanying description. 
The present invention attempts to treat the input red, green, and blue 
intensity values as though they were the responses of red, green and blue 
cone cells. The method transforms those values into a "response image" in 
a manner similar to that by which an on-center retinal ganglion cell 
responds to inputs from cone cells, infers what that response image would 
be if the average scene were gray, and inverts the transformation to 
produce corrected images. It is to be understood that references to an 
"image" herein are references to values at pixels, or "pixel values", 
treated collectively, as an array. Thus, the term "image" as used herein 
includes purely mathematical objects, and does not necessarily correspond 
to a physical image, although the original input images certainly do 
correspond to physical images. The forward transformation is performed 
analogously to the top path of FIG. 1 of Dahari and Spitzer, to provide, 
at each pixel, a red center response, a green center response, and a blue 
center response, with an additional consideration of a remote correction, 
as described herein. These responses are corrected for the non-whiteness 
of the input images, and then inverted by following the top path of FIG. 1 
of Dahari and Spitzer backwards. In some preferred embodiments of the 
present invention, a forward transformation analogous to the bottom path 
of FIG. 1 of Dahari and Spitzer is performed on the red and green images, 
to provide, at each pixel, a red surround response and a green surround 
response, again, with an additional consideration of a remote correction, 
as described herein. These surround responses are subtracted from the 
center responses, in emulation of the action of "on-center" retinal 
ganglion cells, as follows: the green surround response is subtracted from 
the red center response; the red surround response is subtracted from the 
green center response; and the red surround response and the green 
surround response are averaged to form a yellow response, which is 
subtracted from the blue center response. The resulting modified center 
responses are inverted in the same way as in embodiments in which this 
surround correction is not performed. 
This procedure for combining center responses with surround responses tends 
to blur the edges between regions of strongly contrasting color. This 
blurring may be removed by further processing, in emulation of the action 
of the "double opponent" cells of the visual cortex. These cells combine 
the responses of the on-center and off-center retinal ganglion cells in a 
manner that resembles a mathematical second derivative. For example, one 
type of double opponent cell gets its (new) center response from the first 
group of on-center ganglion cells (the group that subtracts green surround 
responses from red center responses), and gets its (new) surround response 
from a corresponding group of off-center cells, the group that subtracts 
red center responses from green surround responses. To emulate the action 
of these double opponent cells, it is necessary first to transform the 
input images to response images corresponding to the actions of both 
on-center retinal ganglion cells and off-center retinal ganglion cells. 
This can be done in more than one way. For example, the action of the type 
of double opponent cell described above can be emulated by assigning the 
emulated response of the first group of on-center ganglion cells to the 
new center response, and assigning the emulated response of a 
corresponding group of off-center cells to the new surround response. 
Alternatively, the second part of this action can be emulated by assigning 
the negative of the emulated response of the first group of on-center 
ganglion cells to the new surround response. 
As in Dahari and Spitzer, the first step of the present invention is the 
transformation of each input image to an "output function" G. For each 
color, there is a center output function G.sub.c and a surround output 
function G.sub.s. Each of the three center output functions G.sub.c is 
computed by convolving the corresponding color image with a center local 
spatial Gaussian filter, as shown in equations 2 and 3 of Dahari and 
Spitzer, thereby producing a center smoothed image whose pixel values are 
the required G.sub.c 's. Similarly, each of the three surround output 
functions G.sub.s is computed by convolving the corresponding color image 
with a surround local spatial Gaussian filter, thereby producing a 
surround smoothed image whose pixel values are the required G.sub.s 's. 
Typical values of the radii p of the Gaussian filters are 0.5 pixels for 
the center filter and 1.5 pixels for the surround filter. 
Optionally, the pixel values may be normalized before the spatial filtering 
with the Gaussian filters: 
EQU p :=p/(p+p.sub.0) 
wherein p represents a pixel value, p.sub.0 is a normalization constant, 
and ":=" represents replacement. 
Also as in Dahari and Spitzer, the present invention computes a response R 
at each pixel from the corresponding output function, using a 
Naka-Rushton-like equation. Unlike Dahari and Spitzer, the Naka-Rushton 
equation of the present invention has two semisaturation terms in the 
denominator. For each color, the center response in terms of the center 
output function G.sub.c is 
EQU R.sub.c =G.sub.c /(G.sub.c +.sigma..sub.c,1 =.sigma..sub.c,r) 
and the surround response in terms of the surround output function G.sub.s 
is 
EQU R.sub.s =G.sub.s /(G.sub.s +.sigma..sub.s,1 =.sigma..sub.s,r) 
.sigma..sub.c,1 is a center local semisaturation term, similar to the 
semisaturation term defined in Dahari and Spitzer, equation 9. 
.sigma..sub.c,r is a color center remote semisaturation term that embodies 
the modification of the "center" response due to the "remote" response. 
Similarly, .sigma..sub.s,1 is a surround local semisaturation term, and 
.sigma..sub.s,r is a color surround remote semisaturation term that 
embodies the modification of the "surround" response due to the "remote" 
response. 
In the preferred embodiments of the present invention in which surround 
responses are subtracted from center responses, this subtraction is done 
at this point. Specifically, 
R.sub.c,red :=R.sub.c,red -R.sub.s green 
R.sub.c,green :=R.sub.c,green -R.sub.s,red 
R.sub.c,blue :=R.sub.c,blue -(R.sub.s,red +R.sub.s,green)12 
Then, whether or not the center responses are adjusted by this subtraction, 
the correction for the non-whiteness of the scene is accomplished by 
substituting a white remote semisaturation term .sigma..sub.w for 
.sigma..sub.c,r and solving for a corrected output function H.sub.c in 
terms of R.sub.c : 
EQU H.sub.c =R.sub.c (.sigma..sub.c,1 +.sigma..sub.w)/(1-R.sub.c) 
Preferably, the same .sigma..sub.w is used at all pixels of all three 
colors. This common .sigma..sub.w is computed, either by averaging all the 
.sigma..sub.c,r 'S of all three colors, or through the CIE standard 
scotopic observer (Wyszecki and Styles, page 256, table 1(4.3.2)). 
The forward transformation is inverted by deconvolving H.sub.c with respect 
to the center spatial Gaussian filter that is used to produce G.sub.c. If 
the optional step of normalizing the pixel values before spatial filtering 
was taken, it must be undone at this point: 
EQU p :=pp.sub.0 /(1-p) 
The form of the method of the present invention is simpler when applied to 
still photography, because time variation may be ignored. The method for 
computing the output functions and the semisaturation terms for still 
photography now will be described. 
The center local semisaturation term is computed as 
EQU .sigma..sub.c,1 =.alpha..sub.c G.sub.c +.beta..sub.c 
where .alpha..sub.c and .beta..sub.c are constant parameters. Similarly, 
the surround local semisaturation term is computed as 
EQU .sigma..sub.s,1 =.alpha..sub.s G.sub.s +.beta..sub.s 
where .alpha..sub.s and .beta..sub.s are constant parameters. A typical 
value of both .alpha..sub.c and .alpha..sub.s, is between 1and 2. A 
typical value of both .beta..sub.c and .beta..sub.s is between 0.01 and 
0.2. 
The remote semisaturation terms are computed by convolving the 
corresponding output functions with spatial exponential filters, thereby 
producing "remote images" whose pixel values are the required remote 
semisaturation terms. Specifically, at a pixel with spatial coordinates 
(x,y), the center remote semisaturation term is obtained from a 
convolution of G.sub.c with a center remote spatial exponential filter: 
EQU .sigma..sub.c,r =.gamma..sub.c .smallcircle..smallcircle.G.sub.c (x',y') 
exp (-r(x-x',y-y')/.lambda..sub.c) dx'dy' 
and the surround remote semisaturation term is obtained from a convolution 
of G.sub.s with a surround remote spatial exponential filter: 
EQU .sigma..sub.s,r =.lambda..sub.s .smallcircle..smallcircle.G.sub.s (x',y') 
exp (-r(x-x',y-y')/.lambda..sub.2) dx'dy' 
where, in both spatial exponential filters, r(x,y) is Euclidean distance: 
EQU r(x,y)=sqrt(x.sup.2 +y.sup.2) 
A typical value of the constant parameters .gamma..sub.c and .gamma..sub.s 
is between 1 and 3. A typical value of the radii .lambda..sub.c and 
.lambda..sub.s is two-thirds of a linear dimension of the image, measured 
in numbers of pixels. 
As can be understood from the subscripts "c" (center), "s" (surround), and 
"r" (remote), the various spatial convolutions extend over different 
numbers of pixels. The convolution for computing G.sub.c typically extends 
over one pixel, i.e., that "convolution" actually is a multiplication. 
Typical domains for the other convolutions are shown in FIGS. 3A and 3B. 
The domain of the convolution for computing G.sub.s is shown in FIG. 3A. 
That convolution typically extends over eight pixels 12 immediately 
surrounding a target pixel 11, as shown in FIG. 3A, but may include the 16 
pixels immediately surrounding those eight pixels, thereby extending over 
a total of 24 pixels. The domain of the convolutions for computing the 
remote semisaturation terms .sigma..sub.c,r and as .sigma..sub.s,r 
typically extend over about half the image, but may extend over as few as 
the 24 pixels that are the maximum for the "surround" convolution. For 
example, a typical remote semisaturation domain for a 30.times.30 (900 
pixel total) image is shown in FIG. 3B: 440 pixels 13 surrounding target 
pixel 11. At the boundaries of the images, all convolutions use periodic 
boundary conditions. 
The form of the method of the present invention that is applied to digital 
video photography takes time variation into account, in a manner similar 
to that of the model of Dahari and Spitzer. The output functions G.sub.s 
and G.sub.s now are functions of time t, because the images they are 
created from vary in time. In addition to the spatial filters defined 
above, which are applied to entire images at a single value of t, temporal 
filters, as defined below, are applied pixel-wise to these functions of t. 
Two kinds of temporal filters are used: temporal low pass filters, which 
are independent of the functions to which they are applied, and adaptive 
temporal filters, which depend, at any given time t, on the prior history 
of the functions to which they are applied. G.sub.c (t) and G.sub.s (t) 
are convolved with corresponding low-pass temporal filters as shown in 
equations 4 and of Dahari and Spitzer. A typical values of the low-pass 
temporal filter time constants, .tau..sub.c for the center low-pass 
temporal filter convolved with G.sub.c, and .tau..sub.s for the surround 
low-pass temporal filter convolved with G.sub.s, is 20 milliseconds. 
Second, center and surround adaptive functions G.sub.c,b (t) and G.sub.s,b 
(t), analogous to Dahari and Spitzer's adapting component G.sub.b (t), are 
used in the computation of the semisaturation terms .sigma..sub.c,1 and 
.sigma..sub.s,1 which now also are functions of t: 
EQU .sigma..sub.c,1 (t)=.alpha..sub.c G.sub.c,b (t)+.beta..sub.c 
EQU .sigma..sub.s,1 (t)=.alpha..sub.s G.sub.s,b (t)+.beta..sub.s 
These adaptive functions are computed by convolving the corresponding 
output functions with corresponding adaptive temporal filters as shown in 
equations 7 and 8 of Dahari and Spitzer, and in the boxes labeled 
"dynamical adaptive filter" in FIG. 1 of Dahari and Spitzer. What makes 
these filters adaptive is that the associated time "constants" actually 
are functions of both time and the prior histories of G.sub.c and G.sub.s. 
Suppressing the subscripts "c" and "s" for clarity, the most preferred 
form of the function .tau..sub.b that describes the decay of the adaptive 
filter is: 
EQU .tau..sub.b (t)=.tau..sub.m /(1+abs(G(t)+G.sub.b (t))/G.sub.n) 
In this expression, .tau..sub.m is the maximum expected value of 
.tau..sub.b (t), G(t) is the output function, G.sub.c or G.sub.s, after 
convolution with the corresponding low-pass temporal filter; G.sub.b (t) 
is the corresponding adaptive function, G.sub.c,b (t) or G.sub.s,b (t), 
i.e., the output of the convolution, at times prior to the time at which 
the convolution presently is being computed; and G.sub.n is a 
normalization constant. Because the adaptive filter is causal, it is 
well-defined despite being defined in terms of its own output. 
Time variation is taken into account while inverting the transformation, by 
deconvolving H.sub.c with respect to the center low-pass temporal filter 
with which G.sub.s is convolved, before deconvolving H.sub.c with respect 
to the center spatial Gaussian filter. 
While the invention has been described with respect to a limited number of 
embodiments, it will be appreciated that many variations, modifications 
and other applications of the invention may be made.