Abstract:
A method for automatic partial correction of color images for non-white illumination. The images are processed in a manner similar to that by which the visual system processes signals related to color vision to achieve &#34;color constancy&#34;.

Description:
FIELD AND BACKGROUND OF THE INVENTION 
     The present invention relates to image processing and, more particularly, to a method for partially correcting color images for colored illumination without knowledge of either the color or the intensity of the illumination. 
     The perceived colors of visible objects are determined, both by the physics of light reflection and by the way in which the visual system processes the reflected light that it receives. With regard to the physics, the physical color of the light reflected to the visual system by visible objects is determined, both by the reflectivity spectra of their surfaces, and by the color spectrum of the illuminating light. For an illuminated object to not appear dark, the illuminating light must include colors that the object reflects. For example, a red object looks red in light, such as red light or white light, whose spectrum includes a substantial red component, but looks dark in light, such as blue light, whose spectrum lacks a red component. 
     Through a physiological phenomenon called &#34;color constancy&#34;, the visual system is capable of partially correcting perceived colors to compensate for colored illumination. For example, white sheets of paper look substantially white both at noon, when daylight is predominantly bluish, and at sunset, when daylight is predominantly reddish. 
     Photographs of scenes, including both still pictures and motion pictures, whether recorded by analog means (photographic film) or digital means (video cameras), normally are perceived differently from the way the scenes themselves would be perceived by direct vision. At least two means are known in the art for achieving &#34;white balance&#34;, i.e., correcting for departures from whiteness of the illuminating light. Video cameras typically have manual means for achieving white balance. These means require that the video camera be aimed manually at a reference surface that is assumed to be white under white illumination, to record parameters related to the spectrum of the illumination so that the subsequently recorded pictures may be corrected for the non-whiteness of the illumination. Furthermore, the illumination spectrum may change suddenly, for example, if a cloud passes in front of the sun, or if the object being photographed moves from sunlight to shade. These changes in illumination degrade the accuracy of the white balance correction. More advanced video cameras often include automatic white balance mechanisms, but these are not entirely satisfactory. 
     There is thus a widely recognized need for, and it would be highly advantageous to have, a more satisfactory method for performing at least a partial white balance correction, either automatically or interactively, without knowing the illumination spectrum. 
     SUMMARY OF THE INVENTION 
     According to the present invention there is provided a method for partially correcting a scene for illumination color, the scene including an intensity spectrum at each of a plurality of pixels arranged in a rectangular grid, the method comprising the steps of: at each pixel: (a) multiplying the intensity spectrum by a spectral response function of a red photoreceptor, thereby providing a red spectral product; (b) multiplying the intensity spectrum by a spectral response function of a green photoreceptor, thereby providing a green spectral product; (c) multiplying the intensity spectrum by a spectral response function of a blue photoreceptor, thereby providing a blue spectral product; (d) integrating the red spectral product; (e) integrating the green spectral product; and (f) integrating the blue spectral product; thereby providing a red image, a green image, and a blue image, each image having a pixel value at each of the plurality of pixels. 
     The underlying concept of the present invention is to process color pictures in a manner similar to that in which the neurons of the visual system process signals related to color vision to achieve color constancy. Ideally, the input to the present invention is the intensity spectrum of a scene as a function of wavelength, measured at each pixel in a rectangular array of pixels. This intensity spectrum is multiplied by the spectral response function of each of the types of photoreceptor cells of the retina (red cones, green cones, and blue cones) to incident light, and integrated with respect to wavelength, thereby providing, at each pixel, a red intensity value, a green intensity value, and a blue intensity value. Collectively, the red values, the green values, and the blue values are examples of what is referred to herein as &#34;images&#34;: rectangular arrays of values, one value per pixel. These values then are processed according to the algorithm of the present invention to provide images corrected for non-white illumination. 
     This ideal input rarely is attainable. Therefore, the scope of the present invention includes the processing of images obtained by other means. For example, the three input images may be in the form of analog signals from transducers whose spectral responses are similar to the spectral responses of cone cells, in which case the intensity values are electrical signals, typically voltage levels. These analog signals may be processed directly, using an embodiment of the present invention in analog hardware. Alternatively, these analog signals may be digitized, and processed digitally according to the present invention. Usually, however, the input to the present invention consists of digital images, such as are acquired by video cameras, that come ultimately from transducers whose spectral responses does not match the responses of cone cells. In that case, the digital pixel intensity values must be transformed to photoreceptor response coordinates, or &#34;fundamentals&#34;, corresponding to the spectral responses of the three types of cones. 
     The most common color coordinate system for digital color images is the so-called red-green-blue, r-g-b, or chromaticity, coordinates. Digital images in other three-color schemes, such as yellow-cyan-magenta, may be transformed mathematically to r-g-b. The transformation from r-g-b coordinates, or from CIE x-y-z coordinates, to photoreceptor coordinates may be found, for example, in G. Wyszecki and W. S. Styles, &#34;Color Science&#34; (Wiley, 1982), pages 139 and 615. In what follows, all references to &#34;red&#34;, &#34;green&#34; and &#34;blue&#34; will be to photoreceptor response coordinates, and not to chromaticity coordinates. 
     The present invention includes an algorithm in the spirit of that presented by Ronen Dahari and Hedva Spitzer in an article titled &#34;Spatiotemporal adaptation model for retinal ganglion cells&#34;, published in the Journal of the Optical Society of America Series A, Volume 13 Number 3 (March 1996), which article is incorporated by reference for all purposes as if fully set forth herein. The paper by Dahari and Spitzer presents a model for the adaptation of visual perception to changing intensity of illumination. It has been conjectured that color constancy works by an analogous mechanism, with modifications as described herein. 
     FIG. 1 is a schematic cross section of the human retina, showing that the retina consists of five layers of cells, receptors 1, horizontal cells 2, bipolar cells 3, amacrine cells 4, and retinal ganglion cells 5. Receptors 1 shown here are rod cells, which respond to light intensity, rather than color. The mechanism of adaptation to changing intensity modeled by Dahari and Spitzer operates in achromatic retinal ganglion cells 5. The receptive field of achromatic ganglion cell 5 includes both receptors 6 of the center receptive field area and receptors 7 of the surround receptive field area. The responses of center area receptors 6 and surround area receptors 7 are combined in achromatic ganglion cell 5, in one of two different ways, depending on the type of achromatic ganglion cell 5. An &#34;on-center&#34; cell responds positively to signals of increased light intensity from the center area of the receptive field and negatively to signals of increased light intensity from the surround area of the receptive field. An &#34;off-center&#34; cell responds negatively to signals of increased light intensity from the center area of the receptive field and positively to signals of increased light intensity from the surround area of the receptive field. 
     The mechanism of color constancy is analogous, with two main modifications. The first is that color must be considered. There are six main groups of retinal ganglion cells 5 involved in color perception, corresponding to the three kinds of cone cells that respond to color analogously to the response of rod cells 1 to intensity. FIG. 2 shows schematically the receptive field processed by an on-center achromatic retinal ganglion cell 5, showing that the intensity response 9 from surround area rod cells 7 is subtracted from the intensity response 8 from center area rod cells 6. Similarly, a first group of on-center red-processing ganglion cells modifies the cell response of red light by subtracting green surround responses from red center responses; a second group of on-center green-processing ganglion cells modifies the cell response of green light by subtracting red surround responses from green center responses; and a third group of on-center blue-processing ganglion cells modifies the cell response of blue light by subtracting yellow surround responses, i.e., a combination of red surround responses and green surround responses, from blue center responses. 
     The second modification is that the perception of color is further modified by responses from &#34;remote&#34; areas of the receptive field that are even farther than the &#34;surround&#34; areas from the &#34;center&#34; areas. This is believed also to occur at the ganglion cell level. 
     The present invention imitates this mechanism of color perception to provide a partial automatic white balance correction for a color picture, under the assumption that the average scene imaged in the picture is &#34;gray&#34;, i.e., that all three colors are of roughly equal intensities in the scene as a whole when viewed in white light. Corrections corresponding to those performed in the retinal ganglion layer are performed computationally to the three intensity images that constitute the picture. 
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS 
     The invention is herein described, by way of example only, with reference to the accompanying drawings, wherein: 
     FIG. 1 is a schematic cross section of a human retina; 
     FIG. 2 is a schematic diagram of the receptive field of one achromatic retinal ganglion cell; 
     FIG. 3A shows a typical range of pixels included in the &#34;surround&#34; convolutions; 
     FIG. 3B shows a typical range of pixels included in the &#34;remote&#34; convolutions. 
    
    
     DESCRIPTION OF THE PREFERRED EMBODIMENTS 
     The present invention is of a method of emulating the color constancy mechanism of visual perception. Specifically, the present invention can be used to perform a partial automatic &#34;white balance&#34; correction of color pictures without knowledge of the illumination spectrum. 
     The principles and operation of a &#34;white balance&#34; correction method according to the present invention may be better understood with reference to the drawings and the accompanying description. 
     The present invention attempts to treat the input red, green, and blue intensity values as though they were the responses of red, green and blue cone cells. The method transforms those values into a &#34;response image&#34; in a manner similar to that by which an on-center retinal ganglion cell responds to inputs from cone cells, infers what that response image would be if the average scene were gray, and inverts the transformation to produce corrected images. It is to be understood that references to an &#34;image&#34; herein are references to values at pixels, or &#34;pixel values&#34;, treated collectively, as an array. Thus, the term &#34;image&#34; as used herein includes purely mathematical objects, and does not necessarily correspond to a physical image, although the original input images certainly do correspond to physical images. The forward transformation is performed analogously to the top path of FIG. 1 of Dahari and Spitzer, to provide, at each pixel, a red center response, a green center response, and a blue center response, with an additional consideration of a remote correction, as described herein. These responses are corrected for the non-whiteness of the input images, and then inverted by following the top path of FIG. 1 of Dahari and Spitzer backwards. In some preferred embodiments of the present invention, a forward transformation analogous to the bottom path of FIG. 1 of Dahari and Spitzer is performed on the red and green images, to provide, at each pixel, a red surround response and a green surround response, again, with an additional consideration of a remote correction, as described herein. These surround responses are subtracted from the center responses, in emulation of the action of &#34;on-center&#34; retinal ganglion cells, as follows: the green surround response is subtracted from the red center response; the red surround response is subtracted from the green center response; and the red surround response and the green surround response are averaged to form a yellow response, which is subtracted from the blue center response. The resulting modified center responses are inverted in the same way as in embodiments in which this surround correction is not performed. 
     This procedure for combining center responses with surround responses tends to blur the edges between regions of strongly contrasting color. This blurring may be removed by further processing, in emulation of the action of the &#34;double opponent&#34; cells of the visual cortex. These cells combine the responses of the on-center and off-center retinal ganglion cells in a manner that resembles a mathematical second derivative. For example, one type of double opponent cell gets its (new) center response from the first group of on-center ganglion cells (the group that subtracts green surround responses from red center responses), and gets its (new) surround response from a corresponding group of off-center cells, the group that subtracts red center responses from green surround responses. To emulate the action of these double opponent cells, it is necessary first to transform the input images to response images corresponding to the actions of both on-center retinal ganglion cells and off-center retinal ganglion cells. 
     This can be done in more than one way. For example, the action of the type of double opponent cell described above can be emulated by assigning the emulated response of the first group of on-center ganglion cells to the new center response, and assigning the emulated response of a corresponding group of off-center cells to the new surround response. Alternatively, the second part of this action can be emulated by assigning the negative of the emulated response of the first group of on-center ganglion cells to the new surround response. 
     As in Dahari and Spitzer, the first step of the present invention is the transformation of each input image to an &#34;output function&#34; G. For each color, there is a center output function G c  and a surround output function G s . Each of the three center output functions G c  is computed by convolving the corresponding color image with a center local spatial Gaussian filter, as shown in equations 2 and 3 of Dahari and Spitzer, thereby producing a center smoothed image whose pixel values are the required G c  &#39;s. Similarly, each of the three surround output functions G s  is computed by convolving the corresponding color image with a surround local spatial Gaussian filter, thereby producing a surround smoothed image whose pixel values are the required G s  &#39;s. Typical values of the radii p of the Gaussian filters are 0.5 pixels for the center filter and 1.5 pixels for the surround filter. 
     Optionally, the pixel values may be normalized before the spatial filtering with the Gaussian filters: 
     
         p :=p/(p+p.sub.0) 
    
     wherein p represents a pixel value, p 0  is a normalization constant, and &#34;:=&#34; represents replacement. 
     Also as in Dahari and Spitzer, the present invention computes a response R at each pixel from the corresponding output function, using a Naka-Rushton-like equation. Unlike Dahari and Spitzer, the Naka-Rushton equation of the present invention has two semisaturation terms in the denominator. For each color, the center response in terms of the center output function G c  is 
     
         R.sub.c =G.sub.c /(G.sub.c +σ.sub.c,1 =σ.sub.c,r) 
    
     and the surround response in terms of the surround output function G s  is 
     
         R.sub.s =G.sub.s /(G.sub.s +σ.sub.s,1 =σ.sub.s,r) 
    
     σ c ,1 is a center local semisaturation term, similar to the semisaturation term defined in Dahari and Spitzer, equation 9. σ c ,r is a color center remote semisaturation term that embodies the modification of the &#34;center&#34; response due to the &#34;remote&#34; response. Similarly, σ s ,1 is a surround local semisaturation term, and σ s ,r is a color surround remote semisaturation term that embodies the modification of the &#34;surround&#34; response due to the &#34;remote&#34; response. 
     In the preferred embodiments of the present invention in which surround responses are subtracted from center responses, this subtraction is done at this point. Specifically, 
     R c ,red :=R c ,red -R s  green 
     R c ,green :=R c ,green -R s ,red 
     R c ,blue :=R c ,blue -(R s ,red +R s ,green)12 
     Then, whether or not the center responses are adjusted by this subtraction, the correction for the non-whiteness of the scene is accomplished by substituting a white remote semisaturation term σ w  for σ c ,r and solving for a corrected output function H c  in terms of R c  : 
     
         H.sub.c =R.sub.c (σ.sub.c,1 +σ.sub.w)/(1-R.sub.c) 
    
     Preferably, the same σ w  is used at all pixels of all three colors. This common σ w  is computed, either by averaging all the σ c ,r &#39;S of all three colors, or through the CIE standard scotopic observer (Wyszecki and Styles, page 256, table 1(4.3.2)). 
     The forward transformation is inverted by deconvolving H c  with respect to the center spatial Gaussian filter that is used to produce G c . If the optional step of normalizing the pixel values before spatial filtering was taken, it must be undone at this point: 
     
         p :=pp.sub.0 /(1-p) 
    
     The form of the method of the present invention is simpler when applied to still photography, because time variation may be ignored. The method for computing the output functions and the semisaturation terms for still photography now will be described. 
     The center local semisaturation term is computed as 
     
         σ.sub.c,1 =α.sub.c G.sub.c +β.sub.c 
    
     where α c  and β c  are constant parameters. Similarly, the surround local semisaturation term is computed as 
     
         σ.sub.s,1 =α.sub.s G.sub.s +β.sub.s 
    
     where α s  and β s  are constant parameters. A typical value of both α c  and α s , is between 1and 2. A typical value of both β c  and β s  is between 0.01 and 0.2. 
     The remote semisaturation terms are computed by convolving the corresponding output functions with spatial exponential filters, thereby producing &#34;remote images&#34; whose pixel values are the required remote semisaturation terms. Specifically, at a pixel with spatial coordinates (x,y), the center remote semisaturation term is obtained from a convolution of G c  with a center remote spatial exponential filter: 
     
         σ.sub.c,r =γ.sub.c ∘∘G.sub.c (x&#39;,y&#39;) exp (-r(x-x&#39;,y-y&#39;)/λ.sub.c) dx&#39;dy&#39; 
    
     and the surround remote semisaturation term is obtained from a convolution of G s  with a surround remote spatial exponential filter: 
     
         σ.sub.s,r =λ.sub.s ∘∘G.sub.s (x&#39;,y&#39;) exp (-r(x-x&#39;,y-y&#39;)/λ.sub.2) dx&#39;dy&#39; 
    
     where, in both spatial exponential filters, r(x,y) is Euclidean distance: 
     
         r(x,y)=sqrt(x.sup.2 +y.sup.2) 
    
     A typical value of the constant parameters γ c  and γ s  is between 1 and 3. A typical value of the radii λ c  and λ s  is two-thirds of a linear dimension of the image, measured in numbers of pixels. 
     As can be understood from the subscripts &#34;c&#34; (center), &#34;s&#34; (surround), and &#34;r&#34; (remote), the various spatial convolutions extend over different numbers of pixels. The convolution for computing G c  typically extends over one pixel, i.e., that &#34;convolution&#34; actually is a multiplication. Typical domains for the other convolutions are shown in FIGS. 3A and 3B. The domain of the convolution for computing G s  is shown in FIG. 3A. That convolution typically extends over eight pixels 12 immediately surrounding a target pixel 11, as shown in FIG. 3A, but may include the 16 pixels immediately surrounding those eight pixels, thereby extending over a total of 24 pixels. The domain of the convolutions for computing the remote semisaturation terms σ c ,r and as σ s ,r typically extend over about half the image, but may extend over as few as the 24 pixels that are the maximum for the &#34;surround&#34; convolution. For example, a typical remote semisaturation domain for a 30×30 (900 pixel total) image is shown in FIG. 3B: 440 pixels 13 surrounding target pixel 11. At the boundaries of the images, all convolutions use periodic boundary conditions. 
     The form of the method of the present invention that is applied to digital video photography takes time variation into account, in a manner similar to that of the model of Dahari and Spitzer. The output functions G s  and G s  now are functions of time t, because the images they are created from vary in time. In addition to the spatial filters defined above, which are applied to entire images at a single value of t, temporal filters, as defined below, are applied pixel-wise to these functions of t. Two kinds of temporal filters are used: temporal low pass filters, which are independent of the functions to which they are applied, and adaptive temporal filters, which depend, at any given time t, on the prior history of the functions to which they are applied. G c  (t) and G s  (t) are convolved with corresponding low-pass temporal filters as shown in equations 4 and of Dahari and Spitzer. A typical values of the low-pass temporal filter time constants, τ c  for the center low-pass temporal filter convolved with G c , and τ s  for the surround low-pass temporal filter convolved with G s , is 20 milliseconds. Second, center and surround adaptive functions G c ,b (t) and G s ,b (t), analogous to Dahari and Spitzer&#39;s adapting component G b  (t), are used in the computation of the semisaturation terms σ c ,1 and σ s ,1 which now also are functions of t: 
     
         σ.sub.c,1 (t)=α.sub.c G.sub.c,b (t)+β.sub.c 
    
     
         σ.sub.s,1 (t)=α.sub.s G.sub.s,b (t)+β.sub.s 
    
     These adaptive functions are computed by convolving the corresponding output functions with corresponding adaptive temporal filters as shown in equations 7 and 8 of Dahari and Spitzer, and in the boxes labeled &#34;dynamical adaptive filter&#34; in FIG. 1 of Dahari and Spitzer. What makes these filters adaptive is that the associated time &#34;constants&#34; actually are functions of both time and the prior histories of G c  and G s . Suppressing the subscripts &#34;c&#34; and &#34;s&#34; for clarity, the most preferred form of the function τ b  that describes the decay of the adaptive filter is: 
     
         τ.sub.b (t)=τ.sub.m /(1+abs(G(t)+G.sub.b (t))/G.sub.n) 
    
     In this expression, τ m  is the maximum expected value of τ b  (t), G(t) is the output function, G c  or G s , after convolution with the corresponding low-pass temporal filter; G b  (t) is the corresponding adaptive function, G c ,b (t) or G s ,b (t), i.e., the output of the convolution, at times prior to the time at which the convolution presently is being computed; and G n  is a normalization constant. Because the adaptive filter is causal, it is well-defined despite being defined in terms of its own output. 
     Time variation is taken into account while inverting the transformation, by deconvolving H c  with respect to the center low-pass temporal filter with which G s  is convolved, before deconvolving H c  with respect to the center spatial Gaussian filter. 
     While the invention has been described with respect to a limited number of embodiments, it will be appreciated that many variations, modifications and other applications of the invention may be made.