Abstract:
An automatic white balancing method and system. A modification of “color in perspective” white balancing technique is implemented. Rather than a default projection plane used by the typical white balancing technique, the projection plane for a RGB space is automatically selected to allows the projected reference gamuts in the projection plane to be as disjoined as possible, thereby making them as uncorrelated as possible. In turn, automatic white balancing is performed using the color temperature associated with the projected reference gamut most correlated with the projected gamut of an image sensor.

Description:
FIELD OF THE INVENTION 
     The invention relates to image white balancing, particularly to image white balancing that uses statistical analysis. 
     BACKGROUND 
     A digital camera is often required to estimate or detect the illumination condition while a user is taking a picture. Then the digital camera needs to perform automatic white balancing in order to compensate for the color deviation caused by the extreme illumination. For example, a white paper under yellow light will be image captured as yellow, even though the desired result is to show the white paper as white. As such, the digital camera should perform white balancing in order to make the white paper look “white” even under the yellow light. 
     Several auto white balancing techniques have been proposed in the past. Examples include the widely used gray-world algorithm, the RGB max algorithm, neural network approaches, and “color in perspective” approaches. 
     Among these auto white balancing techniques, “color in perspective” technique is reported to have superior performance. An image sensor&#39;s responses are recorded for various color temperatures (light sources), with each color temperature tested with a plurality of different surfaces. 
     Specifically, as each color temperature is shone on a variety of surfaces, the resulting pixel values sensed by the image sensor can be plotted in a 3-dimensional color space (e.g., a RGB space). As the sensor&#39;s response to a particular color temperature, these pixel values can be seen as forming a “gamut” or a cluster in this color space. With a different color temperature, the sensor&#39;s response to this color temperature is represented as another gamut (i.e., a cluster) formed in this color space. In turn, the sensor&#39;s responses to respectively n color temperatures are represented respectively as n different gamuts in the color space. These resulting “reference” gamuts then serve as references with which to perform white balancing on sensed images. 
     In theory, as the image sensor captures an image, the pixel values of the captured image also form a gamut for that image in the color space. Out of the n reference gamuts, the reference gamut having the highest correlation with the image&#39;s gamut is selected. The color temperature associated with this selected reference gamut is used as the source light for white balancing the captured image. 
     In practice, before the gamut selection is to take place, buffer limitation necessitates 2-dimensional representations of the n reference gamuts. Also, buffer limitation necessitates a 2-dimensional representation of the gamut that corresponds to the pixel values from the captured image. Information is inevitably lost from this “3D to 2D” projection. Therefore, to select a right direction of the projection is crucial to performing white balancing. 
     The conventional techniques project the 3D gamuts in the color space onto a 2D “chroma” surface. In other words, luminance information is discarded. However, due to the nature of digital imaging sensors and the characteristics of optical filters, the luminance dimension actually contains considerable information for distinguishing different illuminations (light sources). 
     For example, one research group discovered that if the gamuts are projected onto the sensor&#39;s raw red and blue surface (RB plane) rather than pure chroma surface (ab plane in Lab color space, or uv plane Yuv color space), the color separability is actually improved. However, the RB plane might not be the near-optimal 2D surface onto which to project the reference gamuts. As such, in the present invention, a system and method are proposed to systematically select the near-optimal 2D surface onto which to project the reference gamuts. 
    
    
     
       BRIEF DESCRIPTION OF THE FIGURES 
       The accompanying drawings which are incorporated in and form a part of this specification, illustrate embodiments of the invention and together with the description, serve to explain the principles of the invention: 
         FIG. 1  shows an image sensor chip adapted to perform automatic white balancing in accordance with one embodiment of the invention. 
         FIG. 2  shows examples of image sensor responses as the reference gamuts corresponding to different color temperatures in accordance with one embodiment of the invention. 
         FIG. 3  is a flow chart that outlines steps for performing automatic white balancing on an image in accordance with one embodiment of the invention. 
     
    
    
     DETAILED DESCRIPTION 
     Reference is made in detail to embodiments of the invention. While the invention is described in conjunction with the embodiments, the invention is not intended to be limited by these embodiments. On the contrary, the invention is intended to cover alternatives, modifications and equivalents, which may be included within the spirit and scope of the invention as defined by the appended claims. Furthermore, in the following detailed description of the invention, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, as is obvious to one ordinarily skilled in the art, the invention may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so that aspects of the invention will not be obscured. 
       FIG. 1  shows an image sensor chip  100  adapted to perform automatic white balancing in accordance with one embodiment of the invention. 
     Image sensor chip  100  comprises an image sensor array  110  and an image processing component  150 . Image sensor array  110  is adapted for capturing and digitizing images to be processed by image processing component  150 . Image sensor chip  100  is typically used within an image capturing device that could be, but is not limited to, a digital camcorder, a digital still camera, a video phone, a video conferencing equipment, a PC camera, a cell phone, or a security monitor. 
     Image sensor array  110  comprises an image sensor  115 , and an analog-to-digital converter (ADC)  120 . Image processing component  150  comprises a color processing component  155 , a compression engine  160  and a transceiver  165 . 
     Images are captured by sensor  115 , then digitized by ADC  120  into pixel values to be transmitted to image processing component  150 . The pixel values are color processed by color processing component  155 . Color processing component  155  typically performs digital image processing that could include, but is not limited to, auto exposure control, auto white balancing and image sharpening. In turn, the color processed pixel values undergo compression performed by compression engine  160 . The compressed image data is then transmitted out of image sensor chip by transceiver  165 . 
     Referring still to  FIG. 1 , automatic white balancing is performed using a circuit component  157  within said color processing component  155 . Upon receiving captured image data to be white balanced, circuit component  157  determines the correlation between a target gamut with each of n reference gamuts. Specifically, the target gamut refers to the set of pixel values representing the image data. The n reference gamuts refer to n set of pixel values that are associated respectively with n color temperatures (i.e., n lighting sources). 
     In turn, circuitry  157  performs auto white balancing on the received image data by using the color temperature associated with the set of pixel values that is most correlated with the target gamut (i.e., the set of pixel values representing the captured image data). The relationship among color temperatures and their associated sets of pixel values can be stored in a lookup table in memory of image sensor chip  100 . 
     More specifically, the set of pixel values representing the image data are represented as a “target” set (where this set is also called a gamut, or a cluster) of points within a 3-D RGB space. Likewise, each of the n sets of pixel values is represented as a reference set (where this set is also called a gamut, or a cluster) of points within the RGB space. Also, these reference n gamuts of points are associated respectively with n color temperatures. 
     Each of the n reference sets of points (in the RGB space) can be made up of k subsets of points. For example, a subset of points in reference G(T) (one of the n reference gamuts) can be considered as the response of image sensor  115  to T (one of the n color temperatures coming off a particular surface). Another subset of points within reference gamut G((T) represents the response of image sensor  115  to color temperature T coming off a different surface. As such, if k surfaces are used, then a reference gamut has k subsets of points that together represent the response of image sensor  115  to color temperature T reflecting off these k surfaces. On the other hand, points of the target gamut associated with the image data represent image sensor&#39;s response to a scene illuminated by the current color temperature. 
     In practice, sizes of the above n+1 gamuts are reduced in order to conserve buffer capacity of image sensor chip  100 . The reduction is carried out by projecting the n+1 gamuts onto a projection plane, resulting in n+1 projected 2-D gamuts. The n reference gamuts are projected on the projection plane, resulting in n projected reference gamuts. The target gamut is also projected on the projection plane, resulting in a projected target gamut. 
     In turn, circuit  157  performs auto white balancing on the received image data by using the color temperature associated with the projected reference gamut that is the most correlated with the projected target gamut. That is, white balancing is performed using the color temperature associated with the projected reference gamut having the highest correlation with the projected target gamut. But if high correlation exists among the n reference gamuts, quantifying correlation between the projected target gamut with each of the n projected reference gamuts will not be clear-cut. Thus, in order to correlate as accurately as possible, the projection plane is selected to allow the n projected gamuts to be as uncorrelated as possible. In so doing, quantifying correlation between the projected target gamut with each of the n projected reference gamuts becomes easier and more accurate. 
     As understood herein, image sensor  115  could be, but is not limited to, a CMOS sensor. Also, conventional compression technique such as JPEG can be performed by compression engine  160 . Furthermore, transceiver  165  could be, but is not limited to, a USB transceiver or a wireless transceiver. 
     Also, as understood herein, the present invention does not need to be implemented with a single chip design above, where the auto white balancing circuit is integrated with the sensor array on a single chip. For example, in another embodiment of the invention, an auto white balancing circuit and a sensor array reside respectively in different chips. In yet another embodiment, an auto white balancing circuit does not even reside within an image capturing device containing a sensor array. 
       FIG. 2  shows an example of image sensor responses as the n reference gamuts corresponding to respectively n different color temperatures in accordance with one embodiment of the invention. For illustrative purpose, n=3. And the color space is a RGB space. However, as understood herein, n can be different from 3 in other embodiments of the invention. The color space can be different from the RGB space in other embodiments of the invention. 
     In a RGB space  210 , n=3 reference gamuts  211 – 213  are shown. Reference gamuts  211 – 213  represent respectively 3 color temperatures T 1 , T 2  and T 3  (i.e., 3 light sources) that could be used for white balancing the captured image. The points in reference gamut  211  can be thought of as image sensor response to color temperature T 1  reflecting from m surfaces. The points in reference gamut  212  can be thought of as image sensor response to color temperature T 2  reflecting from the same m surfaces. The points in reference gamut  213  can be thought of as image sensor response to color temperature T 3  reflecting from the same m surfaces. 
     Three projected 2-D reference gamuts  251 – 253  are formed by projecting reference gamuts  211 – 213  onto a projection plane  250 . Projection plane  250  is selected to achieve the least correlation among projected reference gamuts  251 – 253 . In turn, each of projected reference gamuts  251 – 253  is compared to a projected 2-D target gamut  270  on projection plane  250 . 
     Specifically, projected 2-D target gamut  270  is formed by projecting onto projection plane  250  a 3-D target gamut  230  (in the RGB space  210 ) that represents a image sensor response to an image to be white balanced. Next, each of projected reference gamuts  251 – 253  is compared to 2-D target gamut  270 . In turn, the color temperature of the projected reference gamut that is the most correlated with 2-D target gamut  270  is used for white balancing the capture image. For example, out of projected reference gamuts  251 – 253 , if say projected reference gamut  253  is the most correlated with projected target 2-D gamut  270 , then the color temperature of projected reference gamut  253  is chosen for white balancing the capture image. 
     As understood herein, because projection plane  250  is selected to achieve the least correlation among projected reference gamuts  251 – 253 , quantifying correlation between projected target gamut  270  with each of projected reference gamuts  251 – 253  is easier than an approach that uses a different projection plane or a fixed projection plane. 
     In contrast, referring still to  FIG. 2 , if reference gamuts  211 – 213  are projected onto RB plane  255  as in conventional approaches, then resulting projected reference gamuts  281 – 283  overlap extensively. As such, quantifying correlation between a projected target gamut  290  with each of projected reference gamuts  281 – 283  becomes difficult as compared to the above approach that uses optimal or near-optimal projection plane  250 . 
     In general, determining which reference gamut is most correlated with the target gamut can be viewed as a pattern classification problem. Moreover, several techniques can be applied to reduce the dimensionality of the color space such as RGB space  210  to a appropriate projection plane. For example, principle component analysis or neural network can be used. 
     Specifically, one approach uses the so-called Fisher&#39;s linear discriminant method. For a two-class classification problem, the input data vector x (that for example indicates a point of RGB space  210 ) can be projected to a one-dimensional vector y using
 
 y=W   T   x, W=S   w   −1 ( m   1   −m   2 ).
 
As understood herein, W is the projection matrix, SW is “within-class” scatter matrix, and (m 1 −m 2 ) is the difference vector between the means (i.e., m 1  and m 2 ) respectively of the two classes are applied. The two classes here can be two of the n reference gamuts.
 
     In performing automatic white balancing, due to the fact that color temperatures follow the Planck&#39;s blackbody spectral radiation law. The n centers respectively of the n 3-D reference gamuts can be plotted to fit into a trajectory in the 3D color space (e.g., RGB space). The next step is then to use regression analysis to find a 2-D plane that contains most of these n centers. 
     Once this “trajectory plane” is available, within-class scatter matrix is used to fine-tune the orientation of the “trajectory plane” to obtain a projection plane so that the resulting n projected 2-D reference gamuts are as disjointed (uncorrelated) as possible. Indeed, experiments show that when compared to the chroma plane, the projection plane obtained in the manner above allows better separability between the n projected reference gamuts. 
     Another method of obtaining a projection plane uses the span of two vectors (m L –m M ) and (m H –m M ) as the projection plane, wherein mL is the mean of a reference gamut associated with a low color temperature, m M  is the mean of a reference gamut associated with a mid-range color temperature, and m H  is the mean of a reference gamut associated with a high color temperature. Again, the projection plane obtained is adapted to allow the resulting n projected 2-D reference gamuts to be as disjointed (uncorrelated) as possible. 
       FIG. 3  is a flow chart  300  that outlines steps for automatic white balancing a captured image of an image sensor in accordance with one embodiment of the invention. 
     In step  305 , n reference gamuts are formed in a 3-dimensional color space (e.g., a RGB space). Specifically, for the first gamut, m reflections of a first color temperature as reflected from m different surfaces are used in obtaining respectively m subsets of points in the first reference gamut. The first reference gamut comprises these first m subsets of points. For the second reference gamut, m reflections of a second color temperature as reflected from the same m surfaces are used in obtaining m subsets of points in the second reference gamut. The second reference gamut comprises these second m subsets of points. Similarly, for the j-th reference gamut, n reflections of a j-th color temperature as reflected from the same m surfaces are used in obtaining m subsets of points in the j-th reference gamut. The j-th reference gamut comprises these j-th m subsets of points. As such, for k=1 to n, for the k-th reference gamut, n reflections of a k-th color temperature as reflected from the same m different surfaces are used in obtaining the k-th m subsets of points in the k-th reference gamut. 
     In step  310 , a projection plane is selected onto which to project the n reference gamuts and obtain their respective 2-D projected gamuts. The projection plane can be different from a fixed projection plane used in conventional approaches. The projection plane is selected so that the n 2-D reference gamut projections are as uncorrelated as possible. As such, if the color space is a RGB space for example, the selected projection plane need not be RB plane if the RB plane does not permit the resulting n reference gamut projections to be as uncorrelated as possible. Moreover, different approaches can be used for finding the projection plane, as long as the projection plane allows the n 2-D reference gamut projections to be as maximally uncorrelated, or close to being maximally uncorrelated within an acceptable error bound. 
     In step  315 , n 2-D reference gamut projections are created by projecting the n 3-D reference gamuts onto the selected projection plane. As described above, these n 2-D reference gamut projections are as uncorrelated as possible (or near to being maximally uncorrelated within acceptable error bound) because the projection plane has been determined specifically with the purpose of minimizing their correlation. 
     In step  320 , through image capturing, a “target” gamut associated with the captured image is formed in the 3-D color space. 
     In step  325 , a 2-D target gamut projection is formed on the selected projection plane by projecting the target gamut onto the selected projection plane. 
     In step  330 , the target gamut projection is compared with each of the n reference gamut projections to quantify the n correlations of the target gamut projection respectively with the n reference gamut projections. 
     In step  335 , the color temperature with which to white balance the captured image is selected. Specifically, the selected color temperature is associated with the reference gamut projection that among the n reference gamut projections correlates the most with the target gamut projection. 
     As understood herein, throughout the detailed description, RGB space can be replaced with another type of color space. Moreover, the foregoing descriptions of specific embodiments of the invention have been presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to explain the principles and the application of the invention, thereby enabling others skilled in the art to utilize the invention in its various embodiments and modifications according to the particular purpose contemplated. The scope of the invention is intended to be defined by the claims appended hereto and their equivalents.