Patent Publication Number: US-9838660-B1

Title: Methods and systems for white balance

Description:
BACKGROUND 
     In general, automatic white balancing refers to an imaging technique of reproducing a white object as a white image according to a light source illuminating the white object, such as sunlight, a fluorescent lamp, an incandescent room lamp, or the like. 
     Since human beings are adapted to colors, they can properly distinguish color differences of an object depending on the change in the lighting and location. However, since cameras are not adapted to colors, color differences of an object are reproduced according to the lighting and location. Accordingly, in order to control a phenomenon that changes colors of an object according to light sources of a camera, an automatic white balance technique is employed. 
     In an automatic white balance technique, areas of an input image are searched and gains of various channels are adjusted using an average of red, green and blue (RGB) values or chrominance signals (R-Y (luma)), (B-Y) of image data corresponding to the searched gray areas. 
     SUMMARY 
     At least some example embodiments use properties of natural images for automatic white/gray balance gains computation, or, alternatively, for estimation of illuminant properties. The properties of natural images include: (i) natural images are images of objects; (ii) object surfaces often consist of homogenous materials, and have homogeneous reflection properties. “Homogenous reflection properties” means that in an image, pixels, corresponding to such surfaces have a similar hue, if an illuminant is perfectly white/gray. This property is referred to as “Hue Constancy.” Additional properties of natural images include (iii) if an illuminant is not white/gray, these pixels will not be of the same hue anymore. 
     According to an example embodiment, in a first stage, pixels may be sub-divided into groups according to hue. Next, the white balance gains as well as other parameters are estimated, which enables to perform a new sub-division of the pixels. 
     At least one example embodiment discloses an apparatus including a processor configured to execute computer-readable instructions to receive image data from a plurality of pixels, determine a first white point based on the image data and a threshold percentage of a histogram of the image data, determine a second white point based on the image data, determine a third white point based on groups of the image data corresponding to a same hue or desaturation, the processor configured to divide the image data into the groups and generate an image based on at least the first white point, the second white point and the third white point. 
     At least another example embodiment discloses a method including receiving image data from a plurality of pixels, determining a first white point based on the image data and a threshold percentage of a histogram of the image data, determining a second white point based on the image data, determining a third white point based on groups of image data corresponding to a same hue or desaturation, the processor configured to divide the image data into the groups and generating an image based on at least the first white point, the second white point and the third white point. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       Example embodiments will become more appreciable through the description of the drawings in which: 
         FIG. 1  is a block diagram illustrating an example embodiment of an image sensor. 
         FIG. 2  illustrates a white balancing circuit of  FIG. 1 , according to an example embodiment. 
         FIG. 3A  illustrates a method of determining white balancing gains for a single hue, according to an example embodiment. 
         FIG. 3B  illustrates a method of determining white balancing gains for multiple hues, according to an example embodiment. 
         FIG. 3C  illustrates another method of determining white balancing gains, taking into account pixels that have a saturation of zero or close to zero, according to an example embodiment. 
         FIG. 4  illustrates an iterative method of determining gains for channels of an image based on the methods shown in  FIGS. 3A-3C , according to an example embodiment. 
         FIG. 5  illustrates a method of determining gains for channels of an image according to an example embodiment. 
         FIG. 6  illustrates a method of determining a UTP value according to an example embodiment. 
         FIG. 7  is a block diagram illustrating an electronic imaging system according to an example embodiment. 
     
    
    
     DETAILED DESCRIPTION 
     Example embodiments will now be described more fully with reference to the accompanying drawings. Many alternate forms may be embodied and example embodiments should not be construed as limited to example embodiments set forth herein. In the drawings, like reference numerals refer to like elements. 
     Unless specifically stated otherwise, or as is apparent from the discussion, terms such as “processing” or “computing” or “calculating” or “determining” or “displaying” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical, electronic quantities within the computer system&#39;s registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices. 
     In the following description, illustrative embodiments will be described with reference to acts and symbolic representations of operations (e.g., in the form of flow charts, flow diagrams, data flow diagrams, structure diagrams, block diagrams, etc.) that may be implemented as program modules or functional processes include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types and may be implemented using existing hardware in existing electronic systems (e.g., digital single lens reflex (DSLR) cameras, digital point-and-shoot cameras, personal digital assistants (PDAs), smartphones, tablet personal computers (PCs), laptop computers, etc.). Such existing hardware may include one or more Central Processing Units (CPUs), digital signal processors (DSPs), application-specific-integrated-circuits, field programmable gate arrays (FPGAs) computers or the like. 
     As disclosed herein, the term “storage medium”, “computer readable storage medium” or “non-transitory computer readable storage medium” may represent one or more devices for storing data, including read only memory (ROM), random access memory (RAM), magnetic RAM, core memory, magnetic disk storage mediums, optical storage mediums, flash memory devices and/or other tangible machine readable mediums for storing information. The term “computer-readable medium” may include, but is not limited to, portable or fixed storage devices, optical storage devices, and various other mediums capable of storing, containing or carrying instruction(s) and/or data. 
     As a result, example embodiments provide methods and systems for white balancing. 
       FIG. 1  is a block diagram of an image sensor  1000  according to an example embodiment. In the example shown in  FIG. 1 , the image sensor  1000  is a complementary-metal-oxide-semiconductor (CMOS) image sensor. The image sensor  1000  may be embodied in a portable electronic device such as a digital camera, a mobile phone, a smart phone, a tablet personal computer (PC), a personal digital assistant (PDA), a mobile internet device (MID), or a wearable computer or another electronic device (e.g., laptop computer, etc.) including, associated with or connected to a camera. However, example embodiments should not be limited to this example. 
     Referring to  FIG. 1 , a timing circuit  106  controls a line driver  102  through one or more control lines CL. In one example, the timing circuit  106  causes the line driver  102  to generate a plurality of transfer pulses (e.g., reset/shutter, sampling, readout, and/or selection). The line driver  102  outputs the transfer pulses to a pixel array  100  over a plurality of read and reset lines RRL. The read and reset lines RRL may include transfer lines, sampling lines, reset lines, and selection lines. 
     The pixel array  100  includes a plurality of pixels arranged in an array of rows ROW_0, . . . , ROW_i, . . . , ROW_N−1 and columns COL_0, . . . , COL_i, . . . , COL_N−1. As discussed herein, rows and columns may be collectively referred to as lines. Each of the plurality of read and reset lines RRL corresponds to a line of pixels in the pixel array  100  having a Bayer color pattern. In the example embodiment shown in  FIG. 1 , each pixel is an active-pixel sensor (APS), and the pixel array  100  is an APS array. 
     As is known, in the Bayer color pattern, ‘R’ represents a pixel for sensing read color light, and ‘B’ represents a pixel for sensing blue color light. ‘G’ represents a pixel for sensing green color light. 
     Still referring to  FIG. 1 , the analog-to-digital converter (ADC)  104  converts the output pixel data (e.g., voltages) from the i-th line ROW_i of readout pixels into a digital signal (also referred to herein as image data). The ADC  104  then outputs the image data D OUT  to an image processing circuit  108 . The image processing circuit  108  performs further processing so as to generate an image to be displayed on a display device (e.g., monitor, etc.) and/or stored in a memory (not shown). 
     As shown, the image processing circuit  108  includes a white balancing circuit  1200  and further image processing circuits  1400 . The white balance circuit generates a white balanced image Image WB  and outputs it to the further image processing circuits  1400  such as a color inconsistency information extraction circuit. 
     The white balancing circuit  1200  and the further image processing circuits  1400  and/or any components thereof may be hardware, firmware, hardware executing software or any combination thereof. When the color correction circuit  1200  the further image processing circuits  1400  is hardware, such hardware may include one or more Central Processing Units (CPUs), system-on-chips (SOCs), digital signal processors (DSPs), application-specific-integrated-circuits (ASICs), field programmable gate arrays (FPGAs) computers or the like configured as special purpose machines to perform the functions of the color correction circuit  1200  or the further image processing circuits  1400 . CPUs, SOCs, DSPs, ASICs and FPGAs may generally be referred to as processors and/or microprocessors. 
       FIG. 2  illustrates a white balancing circuit according to an example embodiment. 
     As shown in  FIG. 2 , the white balancing circuit  1200  includes an Under Threshold Percentage (UTP) computation circuit  1215 , a white balance (WB) computation circuit  1210 , a joint segmentation and AWB circuit  1205 , a joint WB and confidence computation circuit  1220 , a gray surface recognition circuit  1230 , a selector  1235  and a white balance image generator  1240 . 
     Each of the UTP computation circuit  1215 , the white balance computation circuit  1210 , the joint segmentation and AWB circuit  1205 , the joint WB and confidence computation circuit  1220 , the gray surface recognition circuit  1230 , the selector  1235  and the white balance image generator  1240  may be implemented in hardware, a processor configured to execute software, firmware, or any combination thereof, for example. When at least one of the UTP computation circuit  1215 , the white balance computation circuit  1210 , the joint segmentation and AWB circuit  1205 , the joint WB and confidence computation circuit  1220 , the gray surface recognition circuit  1230 , the selector  1235  and the white balance image generator  1240  is hardware, such existing hardware may include one or more Central Processing Units (CPUs), digital signal processors (DSPs), application-specific-integrated-circuits (ASICs), field programmable gate arrays (FPGAs) computers or the like configured as special purpose machines to perform the functions of the at least one of the UTP computation circuit  1215 , the white balance computation circuit  1210 , the joint segmentation and AWB circuit  1205 , the joint WB and confidence computation circuit  1220 , the gray surface recognition circuit  1230 , the selector  1235  and the white balance image generator  1240 . CPUs, DSPs, ASICs and FPGAs may generally be referred to as processors and/or microprocessors. 
     In the event where at least one of the UTP computation circuit  1215 , the white balance computation circuit  1210 , the joint segmentation and AWB circuit  1205 , the joint WB and confidence computation circuit  1220 , the gray surface recognition circuit  1230 , the selector  1235  and the white balance image generator  1240  is a processor executing software, the processor is configured as a special purpose machine to execute the software, stored in a storage medium (e.g., memory  140 ), to perform the functions of the at least one of the UTP computation circuit  1215 , the white balance computation circuit  1210 , the joint segmentation and AWB circuit  1205 , the joint WB and confidence computation circuit  1220 , the gray surface recognition circuit  1230 , the selector  1235  and the white balance image generator  1240 . In such an embodiment, the processor may include one or more Central Processing Units (CPUs), digital signal processors (DSPs), application-specific-integrated-circuits (ASICs), field programmable gate arrays (FPGAs) computers. 
     The UTP computation circuit  1215 , the white balance computation circuit  1210  and the joint segmentation and AWB circuit  1205  process the input image data D OUT  and output data to the joint WB and confidence computation circuit  1220 . More specifically, the WB computation circuit  1210  generates white balance values (e.g., normalized blue and red values) WB CC  using any conventional method. While white balance values are referred to as normalized blue and red values, it should be understood that normalized green and red values may be used. The UTP computation circuit  1215  generates white balance values WB UTP  based on the input image D OUT  as will be described with respect to  FIG. 6 , and the joint segmentation and AWB (automatic white balancing) circuit  1205  performs white balancing in accordance with the methods described in  FIGS. 3A-4  to generate the white balance values WB JS . Each of joint segmentation and AWB circuit  1205 , WB computation circuit  1210  and UTP computation circuit  1215  determines gains for red and blue channels and applies the gains to the red, green and blue white point values such that the sum of the white point values modified by the gains equals 1. 
       FIGS. 3A-4  illustrate methods of determining gains for channels of an image. More specifically,  FIGS. 3A-3C  illustrate a general initial method for determining white/balance gains for a number of hues.  FIG. 4  illustrates an iterative method of determining gains for channels of an image based on the methods shown in  FIGS. 3A-3C . The methods shown in  FIGS. 3A-4  are performed by the joint segmentation and AWB circuit  1205 . 
     During image capturing, a camera obtains rays of light reflected and refracted by surfaces of scene objects. “Object” may refer to something that exists in a scene including natural and artificial objects such as buildings, cars, people, animals, etc., and amorphous objects such as sky, grass and ocean. The colors of the captured objects in the image depend on the original colors of the objects as well as illumination conditions of the scene. For example, an image of some scene in day light illumination condition will look different than the same scene during a sunset or a dawn. This difference is caused by difference in temperature of illuminants: it is colder in the middle of the day and warmer on the sunset. 
     Captured images are images of objects. Each object consists of a small set of materials. The surface of each object has subsets with homogeneous reflection properties. Thus, the objects have sub-surfaces of same hues, e.g., grass of the tennis court. 
     If an illuminant is white or gray, hues of a sky or walls in the office do not vary. However, this is not the case, if an illuminant is not white or gray. Misbalancing gains affect hues of image pixels. 
     The inventors have discovered white/gray balance gains may be determined based on estimating which image pixels are of a same hue. 
     To minimize a distortion S, the joint segmentation and AWB circuit  1205  may use the following optimization model for a single hue: 
                     S   ⁡     (     Gr   ,   Gb   ,   φ     )       =         ∑     j   =   1     N     ⁢       (         U   ⁡     (       Gr   ·     r   j       ,     g   j     ,     Gb   ·     b   j         )       ·     sin   ⁡     (   φ   )         -       V   ⁡     (       Gr   ·     r   j       ,     g   j     ,     Gb   ·     b   j         )       ·     cos   ⁡     (   φ   )           )     2       -&gt;   min             (   1   )               
where Gr,Gb,φ are gains for red and blue channels and the single hue, respectively, U(r j ,g j ,b j ) and V(r j ,g j ,b j ) are linear functions of a j-th image pixel for computing the Chroma where r, g and b are red, green and blue intensity levels, respectively.
 
       FIG. 3A  illustrates a method of determining white balancing gains using equation (1) for a single hue, according to an example embodiment. At S 300 , the joint segmentation and AWB circuit  1205  extracts pixels which hypothetically belong to a single hue component. The pixels are not necessarily of the same exact hue value, since there may be some noise. In addition, the assessment of the pixel belonging to the component does not have to be 100% correct, some pixels may be totally unrelated with a hue component of the pixel majority. The joint segmentation and AWB circuit  1205  extracts the pixels of a same hue by performing an E-step of an expectation-maximization algorithm. Thus, the joint segmentation and AWB circuit  1205  determines a likelihood of each pixel belonging to the single hue. If the likelihood of a pixel exceeds a threshold value, the pixel is determined to be part of a set of N pixels belonging to the same single hue. At S 305 , the joint segmentation and AWB circuit  1205  performs an optimization step on the set of N pixels belonging to the same single hue. More specifically, the joint segmentation and AWB circuit  1205  performs an M-step of the expectation-maximization algorithm to maximize the likelihood of each of the N pixels belonging to the same single hue. The output generates optimized white-balance gains Gr and Gb. 
     Equation (1) can be modified to account for multiple object surfaces with each object surface being represented by pixels with its own hue. Thus, to minimize the distortion S, the joint segmentation and AWB circuit  1205  may use the following optimization model for K hues: 
                     S   ⁡     (     Gr   ,   Gb   ,     φ   1     ,     φ   2     ,   …   ⁢           ,     φ   K       )       =       ∑     k   =   1     K     ⁢       ∑     j   =   1       N   k       ⁢     (           U   ⁡     (       Gr   ·     r     j   ,   k         ,     g     j   ,   k       ,     Gb   ·     b     j   ,   k           )       ·     sin   ⁡     (     φ   k     )         -               V   ⁡     (       Gr   ·     r     j   ,   k         ,     g     j   ,   k       ,     Gb   ·     b     j   ,   k           )       ·     cos   ⁡     (     φ   k     )         )     2       -&gt;   min                   (   2   )               
where (r j,k ,g j,k ,b j,k ) is a j-th pixel that belongs to the k-th hue component.
 
       FIG. 3B  illustrates a method of determining white balancing gains using equation (2) for K hues. Pixels that belong to some hue component have a Chroma error (a distance between pixel representations in UV space and the UV line representing their hue component), which is modeled by a symmetric exponential distribution with the first moment equal to 0. 
     At S 310   1 - 310   k , the joint segmentation and AWB circuit  1205  extracts pixels that are one of the K hues and groups the pixels according to the associated hue. For example, the white balancing circuit generates a set of N1 pixels having a first hue and performs the same for each of the K hues. Similar to  FIG. 3A , the joint segmentation and AWB circuit  1205  extracts the pixels of a same hue by performing an E-step of an expectation-maximization algorithm. At S 315 , the joint segmentation and AWB circuit  1205  performs an optimization step on the sets of N1-NK pixels. The output generates optimized white-balance gains Gr and Gb. 
       FIG. 3C  illustrates another extension of a method of determining white balancing gains, taking into account pixels that have a saturation of zero or close to zero. Pixels having a saturation of zero or close to zero means that the pixels are gray or almost gray. These pixels may affect the optimization, since they weakly constraint their corresponding angles. Thus, the method of  FIG. 3C  models the pixels that are gray or almost gray separately. 
     Equation (2) can be modified to account also for pixels that are gray or almost gray. Thus, to minimize the distortion S, the white balancing circuit may use the following optimization model for the K hues: 
     
       
         
           
             
               
                 
                   
                     S 
                     ⁡ 
                     
                       ( 
                       
                         Gr 
                         , 
                         Gb 
                         , 
                         
                           φ 
                           1 
                         
                         , 
                         
                           φ 
                           2 
                         
                         , 
                         … 
                         ⁢ 
                         
                             
                         
                         , 
                         
                           φ 
                           K 
                         
                       
                       ) 
                     
                   
                   = 
                   
                     
                       ∑ 
                       
                         k 
                         = 
                         1 
                       
                       K 
                     
                     ⁢ 
                     
                       
                         ∑ 
                         
                           j 
                           = 
                           1 
                         
                         
                           N 
                           k 
                         
                       
                       ⁢ 
                       
                         ( 
                         
                           
                             
                               
                                 U 
                                 ⁡ 
                                 
                                   ( 
                                   
                                     
                                       Gr 
                                       · 
                                       
                                         r 
                                         
                                           j 
                                           , 
                                           k 
                                         
                                       
                                     
                                     , 
                                     
                                       g 
                                       
                                         j 
                                         , 
                                         k 
                                       
                                     
                                     , 
                                     
                                       Gb 
                                       · 
                                       
                                         b 
                                         
                                           j 
                                           , 
                                           k 
                                         
                                       
                                     
                                   
                                   ) 
                                 
                               
                               · 
                               
                                 sin 
                                 ⁡ 
                                 
                                   ( 
                                   
                                     φ 
                                     k 
                                   
                                   ) 
                                 
                               
                             
                             - 
                             
                               
                                 
                                   
                                       
                                     
                                       
                                         V 
                                         ⁡ 
                                         
                                           ( 
                                           
                                             
                                               Gr 
                                               · 
                                               
                                                 r 
                                                 
                                                   j 
                                                   , 
                                                   k 
                                                 
                                               
                                             
                                             , 
                                             
                                               g 
                                               
                                                 j 
                                                 , 
                                                 k 
                                               
                                             
                                             , 
                                             
                                               Gb 
                                               · 
                                               
                                                 b 
                                                 
                                                   j 
                                                   , 
                                                   k 
                                                 
                                               
                                             
                                           
                                           ) 
                                         
                                       
                                       · 
                                       
                                         cos 
                                         ⁡ 
                                         
                                           ( 
                                           
                                             φ 
                                             k 
                                           
                                           ) 
                                         
                                       
                                     
                                     ) 
                                   
                                   2 
                                 
                                 ++ 
                               
                               ⁢ 
                               
                                 
                                   ∑ 
                                   
                                     m 
                                     = 
                                     1 
                                   
                                   M 
                                 
                                 ⁢ 
                                 
                                   [ 
                                   
                                     
                                       
                                         U 
                                         ⁡ 
                                         
                                           ( 
                                           
                                             
                                               Gr 
                                               · 
                                               
                                                 r 
                                                 m 
                                               
                                             
                                             , 
                                             
                                               g 
                                               m 
                                             
                                             , 
                                             
                                               Gb 
                                               · 
                                               
                                                 b 
                                                 m 
                                               
                                             
                                           
                                           ) 
                                         
                                       
                                       2 
                                     
                                     + 
                                     
                                       
                                         V 
                                         ⁡ 
                                         
                                           ( 
                                           
                                             
                                               Gr 
                                               · 
                                               
                                                 r 
                                                 m 
                                               
                                             
                                             , 
                                             
                                               g 
                                               m 
                                             
                                             , 
                                             
                                               Gb 
                                               · 
                                               
                                                 b 
                                                 m 
                                               
                                             
                                           
                                           ) 
                                         
                                       
                                       2 
                                     
                                   
                                   ] 
                                 
                               
                             
                           
                           -&gt; 
                           min 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   3 
                   ) 
                 
               
             
           
         
       
     
       FIG. 3C  illustrates an initial method using equation (3) for K hues. As shown, at S 320 , the joint segmentation and AWB circuit  1205  performs K+1 extraction steps. More specifically, the joint segmentation and AWB circuit  1205  extracts pixels of a same hue for each of the K hues to generate sets of N1-NK pixels. Moreover, the joint segmentation and AWB circuit  1205  extracts pixels that are gray or “close to gray”, which forms a set of M pixels. 
     At S 325 , the joint segmentation and AWB circuit  1205  performs an optimization step on the sets of N1-NK pixels and M gray pixels. 
     The model of equation (3) can be extended to model images which do not belong to one of the K hue components or to the close to gray pixels. Thus, to minimize the distortion S, the white balancing circuit may use the following optimization model for the K hues: 
                     S   ⁡     (     Gr   ,   Gb   ,     φ   1     ,     φ   2     ,   …   ⁢           ,     φ   K       )       =       ∑     k   =   1     K     ⁢       ∑     j   =   1       N   k       ⁢     (           U   ⁡     (       Gr   ·     r     j   ,   k         ,     g     j   ,   k       ,     Gb   ·     b     j   ,   k           )       ·     sin   ⁡     (     φ   k     )         -                   V   ⁡     (       Gr   ·     r     j   ,   k         ,     g     j   ,   k       ,     Gb   ·     b     j   ,   k           )       ·     cos   ⁡     (     φ   k     )         )     2     ++     ⁢       ∑     m   =   1     M     ⁢     [         U   ⁡     (       Gr   ·     r   m       ,     g   m     ,     Gb   ·     b   m         )       2     +       V   ⁡     (       Gr   ·     r   m       ,     g   m     ,     Gb   ·     b   m         )       2       ]         +       #   ⁢     (   garbagepixels   )         #   ⁢   pixels         -&gt;   min                   (   4   )               
where E m=1   M [U(Gr·r m ,g m ,Gb·b m ) 2 +V(Gr·r m ,g m ,Gb·b m ) 2 ]→min is the model used by the joint segmentation and AWB circuit  1205  for the pixels that are gray or “close to gray”.
 
     The garbage pixel component allows the garbage pixels to have Chromas distributed uniformly. The formalization of this model component is done by counting the relative number of such pixels. 
     As described above with reference to  FIGS. 3A-3C , image pixels are subdivided into groups, which are assumed to be of the same hues (up to noise). 
       FIG. 4  illustrates how the image pixels are subdivided. More specifically,  FIG. 4  illustrates an iterative method. At a first stage, model parameters are initialized using default values and, for each pixel, a likelihood to belong to each of the model components is computed. Using this likelihood, the pixel division into model components is performed. Next, the model parameters are re-estimated again, and the iterative process continues in this manner until convergence. 
     As shown in  FIG. 4 , the joint segmentation and AWB circuit  1205  initializes the model parameters. The model parameters are subjects for optimization, as will be described below. The model parameters include white balance gains (white balance values) for red and blue channels (red and blue pixels) Gr and Gb, respectively, a standard deviation of the Chroma error for each hue and a standard deviation of the close to gray component. The Chroma error is a distance of pixel representations in the UV space from the UV line representing their hue component. The Chroma error is distributed according to an exponential symmetric one dimensional distribution with the first moment being zero. 
     In at least one example embodiment, the white balance gain for the green channel is one. However, example embodiments are not limited thereto. 
     As shown in  FIG. 4 , S 410 , S 415 , S 420  and S 425  is an iterative process that is performed until convergence is reached. 
     At S 410 , the joint segmentation and AWB circuit  1205  determines a likelihood for each pixel belonging to a hue. The white balance circuit  1200  performs this process for each pixel. The likelihood of a pixel with respect to some component is computed by using a formula for a likelihood computation, for example, equation (5) below, in case of the hue component, and equation (6), in case of the grey component. 
     For example, a Gaussian distribution of an exponential family distribution is: 
     
       
         
           
             
               
                 
                   
                     P 
                     ⁡ 
                     
                       ( 
                       
                         
                           
                             p 
                             j 
                           
                           | 
                           Gr 
                         
                         , 
                         Gb 
                         , 
                         
                           φ 
                           k 
                         
                         , 
                         
                           σ 
                           k 
                         
                       
                       ) 
                     
                   
                   = 
                   
                     
                       1 
                       
                         
                           
                             2 
                             ⁢ 
                             π 
                           
                         
                         ⁢ 
                         
                           σ 
                           k 
                         
                       
                     
                     ⁢ 
                     
                       e 
                       
                         
                           - 
                           
                             1 
                             
                               2 
                               ⁢ 
                               
                                 σ 
                                 k 
                                 2 
                               
                             
                           
                         
                         ⁢ 
                         
                           
                             ( 
                             
                               
                                 
                                   U 
                                   ⁡ 
                                   
                                     ( 
                                     
                                       
                                         Gr 
                                         · 
                                         
                                           r 
                                           j 
                                         
                                       
                                       , 
                                       
                                         g 
                                         j 
                                       
                                       , 
                                       
                                         Gb 
                                         · 
                                         
                                           b 
                                           j 
                                         
                                       
                                     
                                     ) 
                                   
                                 
                                 · 
                                 
                                   sin 
                                   ⁡ 
                                   
                                     ( 
                                     φ 
                                     ) 
                                   
                                 
                               
                               - 
                               
                                 
                                   V 
                                   ⁡ 
                                   
                                     ( 
                                     
                                       
                                         Gr 
                                         · 
                                         
                                           r 
                                           j 
                                         
                                       
                                       , 
                                       
                                         g 
                                         j 
                                       
                                       , 
                                       
                                         Gb 
                                         · 
                                         
                                           b 
                                           j 
                                         
                                       
                                     
                                     ) 
                                   
                                 
                                 · 
                                 
                                   cos 
                                   ⁡ 
                                   
                                     ( 
                                     φ 
                                     ) 
                                   
                                 
                               
                             
                             ) 
                           
                           2 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   5 
                   ) 
                 
               
             
           
         
       
     
     where σ k  stands for a soft tolerance for proximity of a pixel Chroma to a line in UV space corresponding to a k-th hue component. In other words, σ k  is a standard deviation of the chroma error. A pixel, which is far from the mean of the component in terms of that sigma, will have a very low likelihood. 
     For close-to-gray pixels, another Gaussian distribution is used for modeling, namely: 
                     P   ⁡     (         p   j     |   Gr     ,   Gb   ,     σ   gray       )       =       1         2   ⁢   π       ⁢     σ   gray         ⁢     e       -     1     2   ⁢     σ   gray   2           ⁢     (         U   ⁡     (       Gr   ·     r   j       ,     g   j     ,     Gb   ·     b   j         )       2     +       V   ⁡     (       Gr   ·     r   j       ,     g   j     ,     Gb   ·     b   j         )       2       )                   (   6   )               
where σ gray  stands for a soft tolerance for proximity of a pixel Chroma to a line in UV space corresponding to a close-to-gray component.
 
     The modeling of the “garbage” component is almost the same as was given in equation (4). The only difference is that the garbage component is multiplied by a tuning parameter θ garbage , which determines the amount of penalty brought by the garbage component: 
     
       
         
           
             
               θ 
               garbage 
             
             ⁢ 
             
               
                 
                   # 
                   ⁢ 
                   garbage 
                   ⁢ 
                   
                       
                   
                   ⁢ 
                   pixels 
                 
                 
                   # 
                   ⁢ 
                   pixels 
                 
               
               . 
             
           
         
       
     
     Based on the likelihoods determined at S 410 , the joint segmentation and AWB circuit  1205  divides the pixels among the hue, close to gray and garbage components (i.e., the model components) at S 415 . The joint segmentation and AWB circuit  1205  may divide the pixels among the components using an expectation step (E-step) of an expectation-maximization algorithm, such as described in Dempster, et al.,  Maximum Likelihood from Incomplete Data via the EM Algorithm , Journal of the Royal Statistical Society, Vol. 39, No. 1, pp. 1-38 (1977), the entire contents of which are hereby incorporated by reference. 
     As a result, latent (unobserved) variables Y={y j } j=1   # pixels  are introduced, where each y j  associates the j-th pixel to one of the model components (hue components, close-to gray component and garbage component). 
     The division performed by the joint segmentation and AWB circuit  1205  may be performed using a hard way or soft way. In the hard way of modeling, the joint segmentation and AWB circuit  1205  determines y j  to have a value from 1 to K+2 (K hue components, the close-to-gray component and the garbage component). The value of y j  is the number 1 to K+2 corresponding to the component associated with the highest likelihood. In the soft modeling, the joint segmentation and AWB circuit  1205  determines y j  to be a vector of likelihoods with respect to all the model components. 
     As a result, incomplete and complete log-likelihoods are determined using the EM method. 
     At S 420 , the joint segmentation and AWB circuit  1205  optimizes the model parameters. More specifically, the joint segmentation and AWB circuit  1205  determines the red and blue channels gains (red and blue pixels) Gr and Gb, respectively, a Chroma error for each hue and a standard deviation of the close to gray component that maximizes the expected log-likelihoods using the M-step of the expectation-maximization algorithm. 
     At S 425 , the white balance circuit determines whether the model parameters have converged. The convergence is determined by examining the change of the complete likelihood that occurs between two subsequent iterations of the EM. If that change was small enough, i.e. it was less than a threshold, which is a tuning parameter of the method, the optimization is converged. Alternatively, the optimization is stopped, if the number of iterations reached its maximum, which is also a tuning parameter. 
     If the model parameters have not converged, the method returns to S 410  using the model parameters previously determined at S 420 . 
     If the model parameters have converged at S 430 , the converged model parameters are the white balance values WB JS . In other words, the converged model parameters may be referred to the white balance values WB JS . The white balance values WB JS  are output to the joint WB and confidence computation circuit  1220 . 
     The joint WB and confidence computation circuit  1220  uses the outputs from the joint segmentation and AWB circuit  1205 , the WB computation circuit  1210  and the UTP computation circuit  1215  as will be described in greater detail below. 
       FIG. 5  illustrates a hardware flowchart of the method of determining gains for channels of an image according to an example embodiment, as described in  FIGS. 3A-4 . 
     As shown in  FIG. 5 , the input image is subjected to demozaicing using a known algorithm such as a nearest neighbors based algorithm at S 505 . At S 510 , the image is divided into slices. At S 515 , the method of  FIG. 4  is then applied to each slice. At S 520 , the resulting gains from S 515  are clustered. For example, the resulting gains may be clustered using a mean shift algorithm. 
     Referring back to  FIG. 2 , the white balance circuit  1200  includes the UTP computation circuit  1215 , the WB computation circuit  1210  and the joint segmentation and AWB circuit  1205 . 
     The UTP computation circuit  1215  extracts a color histogram for the input image and calculates a UTP value for the red and blue channels. 
     A conventional method for recognizing gray reflecting surfaces in an image is to apply a known illumination detection method and then look for large surfaces with a color similar to the detected illumination color. However, some images have large non-gray reflecting surfaces and the color of these surfaces may be similar enough to the detected illumination color. This is even more severe in scenes where the detected illumination was wrong to begin with, making the error even greater. 
     As a result, the inventors have developed a method to validate the illumination color and employ the gray surface detection method when a confidence of the illumination is above a threshold. 
     The example embodiment described below discloses a ratio between the red and blue light intensities. However, example embodiments are not limited thereto and ratios between the red or the blue light intensity and that of the green light, thus receiving the full red (R), green (G), and blue (B) ratios, may be determined. 
       FIG. 6  illustrates a method of determining UTP values according to an example embodiment. 
     At S 600 , the UTP computation circuit  1215  extracts a color histogram from the input image D OUT . At S 605 , the UTP computation circuit  1215  determines a UTP threshold based on the color histogram. The UTP computation circuit  1215  determines a second percentile of pixels of both the blue and red channels. The UTP computation circuit  1215  defines the higher value between the two second percentiles as the UTP threshold. At S 610 , the UTP computation circuit  1215  then determines the UTP value for each channel based on the UTP threshold. More specifically, the UTP computation circuit  1215  determines the percentile for each of the blue and red channels at/or below the UTP threshold and defines the percentiles as the UTP value. Thus, one UTP value will be two and the other will be two or higher. 
     The inventors have discovered that the log of the ratio between the UTP values correlates with a ratio between blue and red light intensities. Thus: 
                     0.27   *     ln   ⁡     (       UTP   ⁡     (   r   )         UTP   ⁡     (   b   )         )         +     1.2   ~       whitePoint   ⁡     (   b   )         whitePoint   ⁡     (   r   )                   (   7   )               
where whitepoint(b) is the white point value for the blue channel and whitepoint(r) is the white point value for the red channel.
 
     The UTP computation circuit  1215  determines the UTP values for red UTP(r) and the UTP values for blue UTP(b) using any known method. The UTP computation circuit  1215  then uses equation (7) to determine the ratio between normalized white balance values whitePoint(b) and whitePoint(r). 
     At S 615 , the UTP computation circuit  1215  determines the white balance values WB UTP . More specifically, the UTP computation circuit  1215  then determines the normalized white balance values white point whitePoint(b) and a white point corresponding to green whitePoint(g) as follows: 
     
       
         
           
             
               
                 
                   
                     
                       whitePoint 
                       ⁡ 
                       
                         ( 
                         b 
                         ) 
                       
                     
                     
                       whitePoint 
                       ⁡ 
                       
                         ( 
                         r 
                         ) 
                       
                     
                   
                   = 
                   k 
                 
               
               
                 
                   ( 
                   8 
                   ) 
                 
               
             
           
         
       
     
     
       
         
           
             
               
                 
                   
                     
                       whitePoint 
                       ⁡ 
                       
                         ( 
                         g 
                         ) 
                       
                     
                     
                       whitePoint 
                       ⁡ 
                       
                         ( 
                         r 
                         ) 
                       
                     
                   
                   = 
                   j 
                 
               
               
                 
                   ( 
                   9 
                   ) 
                 
               
             
           
         
       
     
     Using equations (8) and (9), the UTP computation circuit  1215  determines a relationship of the white points as:
 
whitePoint( b )+whitePoint( g )=( k+j )*whitePoint( r )  (10)
 
Thus,
 
whitePoint( b )+whitePoint( g )+whitePoint( r )=( k+j +1)*whitePoint( r )  (11)
 
Since
 
whitePoint( b )+whitePoint( g )+whitePoint( r )=1  (12)
 
Then,
 
1=( k+j +1)*whitePoint( r )  (13)
 
whitePoint( r )=1/( K+j +1)  (14)
 
     Therefore, the UTP computation circuit  1215  determines the normalized white balance values whitePoint(b) and whitePoint(g) as follows:
 
whitePoint( b )= k /( K+j +1)  (15)
 
whitePoint( g )= j /( K+j +1)  (16)
 
     The normalized white balance values whitePoint(b), whitePoint(g) and whitepoint(r) constitute the WB values WB UTP  that are sent from the UTP Computation circuit  1215  to the joint WB and confidence computation circuit  1220 . 
     The joint WB and confidence computation circuit  1220  takes the white balance values for WB computation from the joint segmentation and AWB circuit  1205  (WB JS ), the WB computation circuit  1210  (WB CC ) and UTP computation circuit  1215  (WB UTP ) and mixes them into a single result, this mix can be either be a simple average or a smarter weighted averaging. Each of the white balance values WB JS , WB CC  and WB UTP  forms a white point (red, blue and green values for a white point). 
     The joint WB and confidence computation circuit  1220  determines how to combine the WB data WB JS , WB CC  and WB UTP  into a result with a highest possible confidence. 
     For each of the 3 white points from the white balance values WB UTP , WB CC  and WB JS , the joint WB and confidence computation circuit  1220  determines a Euclidean distance over the Rn,Bn space to the nearest other point (nearesNeighborDist). 
     The joint WB and confidence computation circuit  1220  assigns a weight to each of the three white points. The joint WB and confidence computation circuit  1220  determines the weight as 1/(nearesNeighborDist+1). 
     The joint WB and confidence computation circuit  1220  generates a mixed value WB JOINT  by determining a weighted average of the 3 white points using the determined weights. 
     The joint WB and confidence computation circuit  1220  determines a confidence CONF of the mixed value WB JOINT  by averaging the equation (1/(distanceToMergedPoint+1)) over the three white points, where distanceToMergedPoint is the distance between the white point and the mixed value WB JOINT . Thus, the joint WB and confidence computation circuit  1220  determines a result of (1/(distanceToMergedPoint+1)) three times (one for each white point) and averages the three results. 
     The joint WB and confidence computation circuit  1220  outputs the mixed value WB JOINT  to the gray surface recognition circuit  1230  and the selector  1235 . The joint WB and confidence computation circuit  1220  outputs the confidence CONF to the selector  1235 . 
     The gray surface recognition circuit  1230  receives the mixed value WB JOINT  and replaces the mixed value WB JOINT  with the color of a gray wall WB GRAY . More specifically, the gray surface recognition circuit  1230  compares the mixed value WB JOINT  to at least one surface (e.g., an area covering more than a threshold number of pixels) whose color is close to the mixed value WB JOINT  (i.e., “close-to-gray pixels”). If the mixed value WB JOINT  and the color of the at least one surface is within a threshold difference, the gray surface recognition circuit  1230  averages the color of the pixels in the surface and replaces the mixed value WB JOINT  with the averaged value as a white point WB GRAY . 
     The selector  1235  selects the output from the joint WB and confidence computation circuit  1220  or the gray surface recognition circuit  1230  based on the confidence. If the confidence CONF is equal to or higher than a threshold confidence value, the selector  1235  selects the color of the gray wall WB GRAY  to be used as the white point for the white color WB FINAL . The threshold confidence value may be set by a user and determined based on empirical data. If cases where gray walls are expected a relatively lower threshold confidence value may be used and in cases to be outdoor (where gray walls are not expected), a relatively higher threshold confidence value may be used. If the confidence CONF is lower than the threshold confidence value, the selector  1235  selects the mixed value WB JOINT  to be used for white balancing, WB FINAL . 
     The white balance image generator  1240  generates an image Image WB  using the white point WB FINAL  using conventional methods of applying gains to the image data D OUT . 
     The UTP illumination detection method described herein may be used in combination with other illumination detection techniques to create an improved illumination detector. 
       FIG. 7  is a block diagram illustrating an electronic imaging system according to an example embodiment. 
     Referring to  FIG. 7 , the electronic imaging system includes: an image sensor  700 ; an image signal processor (ISP)  702 ; a display  704 ; and a memory  708 . The image sensor  700 , the ISP  702 , the display  704  and the memory  708  communicate with one another via a bus  706 . 
     The image sensor  700  may be an image sensor according to example embodiments described herein. The image sensor  700  is configured to capture image data by converting optical images into electrical signals. The electrical signals are output to the ISP  702 . 
     The ISP  702  processes the captured image data for storage in the memory  708  and/or display by the display  704 . In more detail, the ISP  702  is configured to: receive digital image data from the image sensor  700 ; perform image processing operations on the digital image data; and output a processed image or processed image data. The ISP  702  may be or include the image processing circuit  108  shown in  FIG. 1 . 
     The ISP  702  may also be configured to execute a program and control the electronic imaging system. The program code to be executed by the ISP  702  may be stored in the memory  708 . The memory  708  may also store the image data and/or images acquired by the image sensor and processed by the ISP  702 . The memory  708  may be any suitable volatile or non-volatile memory. 
     The memory  708  may be a two-dimensional (2D) or three dimensional (3D) memory array. A 3D memory array is monolithically formed in physical levels of arrays of memory cells having an active area disposed above a silicon substrate and circuitry associated with the operation of those memory cells, whether such associated circuitry is above or within such substrate. The term “monolithic” means that layers of each level of the array are directly deposited on the layers of each underlying level of the array. 
     The 3D memory array includes vertical NAND strings that are vertically oriented such that at least one memory cell is located over another memory cell. The at least one memory cell may comprise a charge trap layer. 
     The following patent documents, which are hereby incorporated by reference, describe suitable configurations for three-dimensional memory arrays, in which the three-dimensional memory array is configured as a plurality of levels, with word lines and/or bit lines shared between levels: U.S. Pat. Nos. 7,679,133; 8,553,466; 8,654,587; 8,559,235; and U.S. Patent Application Publication No. 2011/0233648. 
     The electronic imaging system shown in  FIG. 7  may be connected to an external device (e.g., a personal computer or a network) through an input/output device (not shown) and may exchange data with the external device. 
     The electronic imaging system shown in  FIG. 7  may embody various electronic control systems including an image sensor, such as a digital still camera. Moreover, the electronic imaging system may be used in, for example, mobile phones, personal digital assistants (PDAs), laptop computers, netbooks, MP3 players, navigation devices, household appliances, or any other device utilizing an image sensor or similar device. 
     The foregoing description of example embodiments has been provided for purposes of illustration and description. It is not intended to be exhaustive or limiting. Individual elements or features of a particular example embodiment are generally not limited to that particular example embodiment. Rather, where applicable, individual elements or features are interchangeable and may be used in a selected embodiment, even if not specifically shown or described. The same may also be varied in many ways. All such modifications are intended to be included within the scope of this disclosure.