Face Color Constancy : Towards Illumination Invariant Face Recognition

We propose a novel technique to compensate for variations in face images due to changes in the illumination conditions. Other than color of the faces, shadows and specular reflections on the faces also change with respect to lighting environment. These variations result in false recognitions even in the best face recognition algorithms. We tackle this problem by processing the faces using a color constancy model specific to face images, which we named face color constancy. The face color constancy algorithm is trained with face images in different lighting environments. The color changes observed are used to learn the color mapping from one lighting environment to the other. These mappings are represented in a low dimensional subspace to obtain basis vector fields. Using these basis vector fields we can model the nonlinear color changes to transform a new probe face to a reference face in the gallery. Using face color constancy preprocessing, we show that for fixed pose and expression we require a single training image per subject to perform accurate face recognition. Introduction The appearance of a face with fixed expression and pose is determined by the illumination conditions. Here we define illumination conditions as including all those parameters which affect the final RGB values recorded by the camera given that pose and expression are fixed. These parameters are determined by the intensity of the light, frequency of the light, position of the light with respect to the face, shadows over the face, reflectance of the regions in the neighborhood of the face, and camera characteristics. The appearance of the same face with fixed pose and expression under different illumination conditions can change significantly. The accuracy of a face recognition algorithm is going to be severely affected due to these large variations. In general, we know that appearance-based face recognition methods like Eigenfaces [2] need a number of training images for each subject in order to cope with illumination variations. We aim to deal with these variations by automatically transforming the faces images towards a reference lighting condition. When the pose and expression are fixed we require only a single training image per subject for accurate face recognition. We test our approach on PIE database [4] since the illumination variations are more representative of the real world situations than other standard face databases currently available. The illumination variations in PIE database were generated due to presence or absence of ambient light combined with changing position of a point source with respect to the face. The changes in colors, shadows and specular reflections on faces due to the changing illuminations are highly complex. Also we do not have any physical parameters of the lighting, surface reflectances or camera sensitivities. Hence our face color constancy (FCC) approach needs to be completely unsupervised. We realize this by using the color changes observed on a face for different illuminations to drive the model. By training on few faces of different ethnicities, our model is able to generalize the color mapping for any unseen ethnicity from the PIE database. After the FCC algorithm is trained we obtain basis vector fields, which can be used to explain the observed color changes. Using these basis vectors we can transform a face image towards a reference illumination condition. In a face recognition system with a single image per subject, the basis vectors can be used to transform a probe face towards its reference in the gallery. Previous work Many interesting approaches have been used to address the problem of illumination invariant face recognition. Most of the work has been done in building models that try to explicitly model varying illumination. The human face is treated as a Lambertian surface and mathematical models are created which can describe all images it can produce under all possible illuminations. In [1], using spherical harmonics it has been shown that the set of images of a convex Lambertian object obtained under a wide variety of lighting conditions can be approximated accurately by a 9 dimensional linear subspace. A shape from shading method was attempted in [9], but suffered from drawbacks of shape form shading. Another well known model that is widely used is the illumination cone model [3]. It was also shown that illumination cones of human faces can be approximated well by lowdimensional linear subspaces [8]. These appearance-based methods need training images of subjects under a number of different illumination conditions. This drawback is overcome in [5] and [6] by reconstructing 3D face information for each subject in the training set. Color constancy algorithms are very appealing to use for compensating face image variations taken under different illumination conditions. A common approach in the color constancy literature is to use linear models of reflectance and illuminant spectra to estimate surface reflectances and various physical parameters. Most of the methods make strong assumptions about the distribution of reflectances. Gray world algorithms [11] assume that the average reflectance of all the surfaces in a scene is gray. While the white world algorithms [12] assume that the brightest pixel corresponds to a scene point with maximal reflectance. The assumptions make these algorithms impractical for use on real world images. Some of the algorithms require prior knowledge of the illuminant spectra, surface reflectances and camera sensitivities during training to achieve color constancy [10, 15]. There exist very few color constancy algorithms [13, 14] that work on real images, but only for limited conditions. Hence for the purpose of FCC on real world images we develop a data driven approach that can be applied directly to real world images. Our approach is influenced by the color eigenflows method [7], which was used to “flow” an image to a target image using the eigenflows. Their method was trained on a wide range of colors while our method is specific to faces hence more appropriate for FCC. Our approach of using color constancy to achieve illumination invariant face recognition is unique in many ways compared to previous illumination invariant face recognition methods. We can deal with real world illumination environments where many light sources are active simultaneously, the position of the light sources are changing and the neighborhood scene is changing. Our model is also capable of FCC when camera processing such as auto-gain and camera color balancing functions are active. Given a few faces under different illumination environments in these real word situations, our model can learn the basis vectors needed to compensate for these variations in any new face image. Our model can also compensate for strong shadows and specular reflections to some extent. Learning the joint color changes We perform training on face images from the PIE database [4] to learn the color changes on faces for different lighting conditions. We use only frontal faces with normal expression. The database has two sets of illumination variations. In set 1 the ambient lights are on and the point source is changing its state and position. In set 2 the ambient lights are off and again the point source is changing its state and position. We use set 1 to learn the joint color changes. Let the RGB color space be defined as, C = {(r, g, b) ∈ 3 R : 0 ≤ r ≤ 255, 0 ≤ g ≤ 255, 0 ≤ b ≤ 255}. This space defines all the possible color vectors observable in images. The color vector of an image pixel p is denoted as c(p) ∈ C. Let 1 ≤ i ≤ N, where N is the number of subjects and let 1 ≤ j ≤ M, where M is the number of illumination conditions under which each subject’s image is taken. Also let 1 ≤ k ≤ P, where P is the number of pixels in each face image. The mapping of colors under different illumination conditions is represented by difference of two corresponding pixels: 0 0 d( , ) c( ) c( ) k k k k i ij i ij I I I I = − (1) This difference vector tells us how a particular pixel’s value changed from illumination condition of Ii0 to illumination condition of Iij. This difference vector is computed for each of the P pair of pixels to obtain a vector field that is defined at all points in C for which there are colors in image Iij. The vector field is constructed by placing each vector difference at the point c( k ij I ) in the color space C. The vector field Φ’ over C is defined as: 0 '( ( )) ( , ), 1 k k k ij i ij c I d I I k P Φ = ≤ ≤ (2) This vector field is only defined at particular color points in C that happen to be in image k ij I , hence Φ’ is called a partially observed color flow. We wish to obtain a full color flow from the partially observed color flow. A simple approach to obtain the full color flow is to follow an interpolation scheme as proposed in [7]. The color flow at a color point (r, g, b) is obtained by a weighted proximity based average of nearby observed color flow vectors.