Patent Document

FIELD OF THE INVENTION 
   The invention relates generally to image processing, and more particularly to photometrically normalizing images of objects. 
   BACKGROUND OF THE INVENTION 
   In many computer vision systems, and specifically in face identification systems, it is very difficult to design systems that are invariant to arbitrary lighting. Indeed, large independent U.S. government tests have concluded that the identification of faces in images acquired with arbitrary lighting fail to achieve the success rate of faces in images acquired with controlled lighting, see Phillips, “Face Recognition Vendor Test (FRVT) 2002 report,” Technical report, National Institute of Standards and Technology, March 2003, and Phillips et al., “The FERET evaluation methodology for face-recognition algorithms,” IEEE Trans. on Pattern Analysis and Machine Intelligence, 22(10), pp. 1090-1104, 2000. 
   The present invention provides a solution for the difficult but routine problem of facial identification as applied to access control and surveillance applications. 
   In such applications, an image is acquired of a face of a possibly unknown individual in an arbitrary scene under arbitrary illumination. A camera at a fixed pose, e.g., frontal, acquires the image. The camera is uncalibrated and has unknown intrinsic parameters. The image can obtained from a video, archival photography, web imagery, family photo albums, identification photograph, and the like. 
   Without any 3D measurement of the individual or the scene, the problem is to match the face in the single image to images of known individuals stored in a database. The stored images were acquired under fixed lighting, e.g., diffuse or frontal. 
   To solve this problem, all images need to be normalized geometrically and photometrically to provide a single fixed illumination template suitable for robust pattern matching and illumination invariant face identification. Naturally, the canonical choice of illumination would include non-directional or diffuse, or at least frontal, lighting that maximizes visibility of all key facial features. 
   Because the focus is on illumination-invariance, it is assumed that the geometric normalization is performed in a preprocessing step. The preprocessing can include detecting the location of the face in the image, detecting facial features, such as the eyes, rigid transforms, i.e., scale, rotation and translation, to align the detected features. It is also assumed that some simple photometric normalization may have already taken place, e.g., a non-spatial global transform, which is only a function of intensity, e.g., gain, contrast, and brightness. 
   Much of the prior art on modeling lighting has focused on finding a compact low-dimensional subspace to model all lighting variations. Under theoretical Lambertian assumption, the image set of an object under all possible lighting conditions forms a polyhedral ‘illumination cone’ in the image space, Belhumeur et al., “What is the set of images of an object under all possible lighting conditions,” Int&#39;l J. Computer Vision, volume 28, pp. 245-260, 1998. 
   Subsequent work that applies the above theory to face recognition is described by Basri et al., “Lambertian reflectance and linear subspaces,” Int&#39;l Conf. on Computer Vision, volume 2, pages 383-390, 2001. Basri et al. represent lighting using a spherical harmonic basis wherein the low dimensional linear subspace is shown to be effective for face recognition. 
   One method analytically determines the low dimensional subspace with spherical harmonics, Ramamoorthi, “Analytic PCA construction for theoretical analysis of lighting variability in images of a Lambertian object,” IEEE Trans. on Pattern Analysis and Machine Intelligence, 24, Oct. 2002. Another method arranges lighting to best generate equivalent basis images for recognition, Lee et al., “Nine points of light: Acquiring subspaces for face recognition under variable lighting,” Proc. IEEE Conf. on Computer Vision &amp; Pattern Recognition, pages 519-526, 2001. 
   A complementary approach is to generate a lighting invariant ‘signature’ image. Although that technique cannot deal with large illumination changes, it does have the advantage that only one image per object is required in the database. 
   Other prior art normalization techniques generate invariant templates by using histogram equalization or linear ramp subtraction, Rowley et al., “Neural network-based face detection,” IEEE Trans. on Pattern Analysis and Machine Intelligence, 20(1), pp. 23-38, 1998. 
   It is known that the image gradient is illumination-insensitive and can be used in a probabilistic framework to determine the likelihood that two images were acquired from the same object, Chen et al., “In search of illumination invariants,” Proc. IEEE Conf. on Computer Vision &amp; Pattern Recognition, pages 1-8, 2000. 
   The near symmetry of faces can be used to determine an illumination invariant prototype image for an individual without recovering albedos, Zhao et al., “Symmetric shape-from-shading using self-ratio image,” Int&#39;l J. Computer Vision, 45(1), pp., 55-75, 2001. 
   Another method assumes that different faces have a common shape but different texture and determines an albedo ratio as an illumination-invariant signature, Shashua et al., “The quotient image: Class-based rerendering and recognition with varying illuminations” IEEE Trans. on Pattern Analysis and Machine Intelligence, 23(2), pp. 129-139, 2001. 
   Object relighting methods have also been described for computer graphic applications. One application uses corneal imaging for embedding realistic virtual objects, e.g., faces, into a scene, resulting in synthetic faces that are properly ‘relit’ in accordance with estimated environmental lighting, Nishino et al., “Eyes for relighting,” Proceedings of SIGGRAPH, 2004. 
   Another method uses a radiance environment map, Wen et al., “Face relighting with radiance environment maps,” Proc. IEEE Conf. on Computer Vision &amp; Pattern Recognition, 2003. That method renders relatively high quality faces using the spherical harmonics, Rammamoorthi et al., “A signal processing framework for inverse rendering,” Proceedings of SIGGRAPH, 2001. 
   However, for face identification there is no need for high-quality rendering or photorealism. In fact, most known 2D face identification systems operate at low to moderate resolutions, e.g., ˜100 pixels across the face. 
   SUMMARY OF THE INVENTION 
   The invention provides a method for estimating directional lighting in uncalibrated images of objects, e.g., faces. This inverse problem is solved using constrained least-squares and class-specific priors on shape and reflectance. 
   For simplicity, the principal directional illuminant is modeled as a combination of Lambertian and ambient components. By using a ‘generic’ 3D shape for the object and an average 2D albedo, the method can efficiently estimate, in real-time, the incident directional lighting with high accuracy, with or without shadows. 
   The estimate of the directional lighting is used in a forward rendering step to “relight” arbitrarily lit input images of objects to a canonical diffuse form as needed for illumination-invariant object identification. 

   
     BRIEF DESCRIPTION OF THE DRAWINGS 
       FIG. 1  is a diagram of a shape model and an illumination model according to the invention; 
       FIG. 2  is a flow diagram of a method for determining a direction of a principal light source in an image according to the invention; and 
       FIG. 3  is a flow diagram for photometrically normalizing images of an objects according to the invention. 
   

   DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
   Lighting Estimation 
   As shown in  FIG. 1 , my invention uses a generic 3D shape model  101  and a Lambertian or diffuse reflectance illumination model  102  for photometrically normalizing images of objects, e.g., faces. In the illumination model  102  diffuse reflectance has a constant bi-directional reflectance distribution function (BRDF). These models are used for object identification. The example application used to describe my invention is face identification and/or verification. There, the problem is to match an unknown face image to images in a database of known face images. 
   A face can have some specular reflection, due to secretion of sebum oil by sebaceous glands in the skin. However, the specular reflection is not always consistent. Therefore, the specular reflection is of little use in face identification. Hence, my illumination model  102  includes only Lambertian and ambient components. 
   As shown in  FIG. 2 , let I(x, y) be the intensity at a pixel (x, y) in an input image  201  corresponding to a point on a surface of a convex object, e.g., a face or the equivalent 3D shape model  101  with the Lambertian surface reflectance  102 . The point is illuminated by a mixture of ambient light and a single principal light source  103  at infinity in a direction sε             3 , with intensity |s|.
   I designate a unit surface normal n=s/|s| as a direction from the point to the principal light source, i.e., pointing out. This direction, e.g., in azimuth/elevation angles, is my main estimand of interest. The magnitude of the light source is of little consequence for our method because the magnitude can be absorbed by the imaging system parameters that model gain and exposure. 
   Let ρ(x, y) be the albedo  221  of the skin surface, which is either known or is otherwise estimated. Albedo is the fraction of incident light that is reflected by the surface, and for faces, albedo represents diffuse skin texture. Therefore albedo-map and texture-map are synonymous. 
   Let n(x, y)  231  be the unit surface normal of the point on the facial surface that projects onto the pixel I(x, y) in the image, under orthography. 
   Under the Lambertian model with a constant BRDF, a monochrome intensity of the pixel is given by
 
 I ( x,y )=α{ρ( x,y )[ max( n ( x,y ) T   s,  0)+ c]}+β,   (1)
 
where α and β represent intrinsic camera system parameters, i.e., lens aperture and gain. In my analysis, the parameters α and β are essentially nuisance parameters, which only effect the dynamic range or (gain) and offset (exposure bias) of pixel intensity but not the lighting direction. Therefore, I can set (α, β) to their default values of (1, 0) with proper normalization. The parameter c represents a relative intensity of the ambient illumination, as described below, and can be set to zero, if necessary. The term max(n(x, y) T s sets negative values of the Lambertian cosine factor to zero for surface points that are in a shadow.
 
   For simplicity, I assume that only the single principal light source  103  is responsible for the majority of the observed directional lighting in the image, i.e., diffuse attenuation and/or shadowing. Any other ambient light sources present in the scene, e.g., diffuse or directional, are non-dominant. Hence, the overall contribution of the other ambient light sources is represented by a global ambient component with relative intensity c in Equation (1). 
   Nearly all 2D view-based face identification systems are adversely affected by directional lighting, but to a much lesser extent by subtle ambient lighting effects, see Phillips et al. above. Therefore, in most cases, the direction to the principal lighting source is more important than any other lighting phenomena, especially when the other light sources are non-dominant. 
   Therefore, the invention reverses the effect of the principal illumination. This improves the performance of identifying objects that are illuminated arbitrarily. 
   The direction  251  to the principal lighting source is estimated by a least-squares formulation with simplifying assumptions based on the illumination model  102  as expressed by Equation (1). More important, I solve this problem efficiently in a closed form with elementary matrix operations and dot-products. 
   Estimating Light Source Direction 
   Specifically, as shown in  FIG. 2 , I construct  210  a column intensity vector {right arrow over (I)}  211  of pixel intensities by ‘stacking’ all the non-zero values an input image I(x, y)  201 . If I assume that the object is lit only by the principal light source  103 , i.e., there is no ambient light, then zero-intensity pixels are most likely in a shadow. Therefore, these pixels cannot indicate the direction to the principal light source, unless ray-casting is used locate the light source. In practical applications, there always is some amount of ambient light. Therefore, I can use a predetermined non-zero threshold or a predetermined mask for selecting pixels to stack in the intensity vector {right arrow over (I)}. 
   Similarly, I construct  220  an albedo vector {right arrow over (ρ)}  222  to be the corresponding vectorized albedo map or diffuse texture  221 . 
   I generate  230  a 3-column shape matrix N  231  by row-wise stacking of the corresponding surface normals of the shape model  101 . Then, I construct  240  a shape-albedo matrix Aε             p×3 , where each row α in the matrix A  241  is a product of the albedo and the unit surface normal in the corresponding rows of the albedo vector {right arrow over (ρ)}  222  and the shape matrix N  231 . This corresponds to the element-wise Hadamard matrix product operator o:
 
 A =({right arrow over (ρ)}1 1×3 ) o N. 

   To determine  250  the unknown direction s* 251  to the principal light source, I use a matrix equation for least-squares minimization of an approximation error in Equation (1) in the vectorized form 
                   arg   ⁢           ⁢       min   s     ⁢            I   →     -     α   ⁢           ⁢   c   ⁢           ⁢     ρ   →       -   As              ,           (   2   )               
which yields the solution
   s *=( A   T   A ) −1   A   T ( {right arrow over (I)}−αc{right arrow over (ρ)}−As ),  (3) 
where  T  denotes the transpose operator.
 
   Note that I am only interested in the estimated unit light source vector s*/|s*| for its direction and not the magnitude. The magnitude depends on specific camera gain and exposure. This estimation problem is ‘well-behaved’ because it is heavily over-constrained. That is, the number of non-zero elements in {right arrow over (I)} ‘observations’ is on the order of O(10 3 ) as compared to the three unknowns in s*. In fact, because I only use the direction to the principle light source, there are only two angular estimands: azimuth and elevation. The estimate of the principal lighting direction is therefore quite stable with respect to noise and small variations in the input {right arrow over (I)}. 
   Note that the albedo-shape matrix A  241  comes from the generic shape model  101  and albedo  221 . Hence, the shape-albedo matrix A  241  represents the entire class of objects, e.g., all frontal faces. 
   Assuming that the model  101  is adequately representative, there is no need to measure the exact shape or even exact albedo of an individual as long as all shapes and albedos are roughly equal to a first order as far as lighting direction is concerned. 
   Furthermore, the pseudo-inverse (A T A) −1  in Equation (3) is directly proportional to the error covariance of the least-squares estimate s* under Gaussian noise. 
   If I define a matrix P=A(A T A) −1 , of dimensions p×3, then I see that the only on-line computation in Equation (3) is the projection of the intensity vector {right arrow over (I)}  211  on the three columns of the matrix P, which are linearly independent. In fact, the three columns are basic functions for the illumination subspace of my generic face model. 
   Moreover, I can always find an equivalent orthogonal basis for this subspace using a QR-factorization: P=QR, where the unitary matrix Q has three orthonormal columns spanning the same subspace as the matrix P. The 3×3 upper triangular matrix R defines the quality of the estimates because R −1  is a Cholesky factor, i.e., a matrix square root, of the error covariance. The QR-factorization aids the interpretation and analysis of the estimation in terms of pixels and bases because the input image is directly projected onto the orthonormal basis Q to estimate the direction  251  to the principal light source  103 . The QR decomposition also saves computation in larger problems. 
   Because the matrices P and Q are independent of the input data, the matrices can be predetermined and stored for later use. Also, the computational cost of using Equation (3) minimal. The computation requires only three image-sized dot-products. The subsequent relighting, described below, only requires a single dot-product. Therefore, the lighting normalization according to the invention is practical for real-time implementation. 
   Face Relighting 
   As shown in  FIG. 3 , given the estimate s* 251  of the directional lighting in the input image  201 , I can approximately ‘undo’ the lighting” by estimating  310  the albedo  311  or diffuse skin texture of the face, and then relight  320  this specific albedo, combined with the generic shape model  101 , under any desired illumination, e.g., frontal or pure diffuse. 
   Whereas both generic shape and albedo were used in the inverse problem of estimating the directional lighting, only the generic shape  101  is needed in the forward problem of relighting the input image  201 , as the input image  201  itself provides the albedo information. The basic assumption here is that all objects have almost the same 3D geometry as defined by the generic shape model  101 . 
   I find that moderate violations of this basic assumption are not critical because what is actually relighted to generate an illumination invariant template image is the texture as seen in the input image  201 . This texture carries most of the information for 2D object identification. In fact, it is not possible to drastically alter the albedo of the input image by using a slightly different 3D face shape. Therefore, for faces, despite small variations in geometry for different individuals, an individual&#39;s identity is substantially preserved, as long as the face texture is retained. 
   Referring back to Equation (1), after I have a lighting estimate s* 251  and my ‘plug-in’ shape, i.e., surface normals n  231  of the generic face model  101 , I can solve directly for albedo using 
                     ρ   *     =       I   -   β       α   ⁡     (         n   T     ⁢     s   *       +   c     )           ,           (   4   )               
where for clarity the spatial indices (x, y) are not expressed for all 2D-arrays (I, ρ, n). Here, it is assumed that the intensities are non-zero, and that n T s* is greater than zero. Notice that the estimated albedo ρ* 311  at a point (x, y) depends only on the corresponding pixel intensity I(x, y) of the input image  201  and the surface normal n(x, y)  231 . Thus, if a point on an object is in shadow, and there is no ambient illumination, then I is zero and n T s* is negative. In this case, the corresponding albedo cannot be estimated with Equation (4), and a default average albedo is substituted in for the pixel corresponding to that point.
 
   The estimated albedo  311  is then used to generate  320  our invariant (fixed-illumination) image I o    322 
 
 I   o =α o {ρ*[max( n   T   s   o ,0)+ c   o ]}+β o .  (5)
 
   In equation (5) the variable s o    321  denotes the invariant direction to the desired source of principal illumination. The default direction is directly in front of the object and aligned with a horizontal axis through the object, i.e., on-axis frontal lighting, and c o  is the ambient component of the output image  322 . Similarly α o  and β o  designate the format parameters of an output display device. 
   Ambient Illumination 
   It is also possible to model arbitrary ambient illumination as represented by the parameter c. By using a representative set of N training images, I can estimate numerically components of the ambient illumination using optimality criteria 
                     c   *     =     arg   ⁢           ⁢       min   s     ⁢       ∑     I   =   1     N     ⁢                ρ   i     ⁡     (   c   )       -       1   N     ⁢       ∑     i   =   1     N     ⁢       ρ   i     ⁡     (   c   )                  2             ,           (   6   )               
where ρ i (c) denotes an albedo of the i th  training image estimated with a relative ambient intensity c as defined in Equation (3).
 
   EFFECT OF THE INVENTION 
   The invention provides a simple and practical method for estimating a direction to a principal light source in a photometrically uncalibrated input image of an object such as a face. The exact shape and albedo (surface texture) of the object is unknown, yet the generic shape and albedo of the object class is known. Furthermore, the method photometrically normalizes the input image for illumination-invariant template matching and object identification. The necessary computations require less than five dot-products for each pixel in the input image. 
   The method has better performance for datasets of realistic access-control imagery, which exhibits complex real-world illumination environments. The performance enhancement is directly due to a tighter clustering of an individual&#39;s images in image space, which will help sophisticated image matching and identification systems to achieve illumination invariance. 
   Results indicate that the estimation of lighting direction is relatively robust and the subsequent relighting normalization is feasible in real-time, with only a few simple dot product operations. The lighting normalization according to the invention is a viable and superior alternative to linear ramp and histogram equalization techniques of the prior art. 
   Although the invention has been described by way of examples of preferred embodiments, it is to be understood that various other adaptations and modifications may be made within the spirit and scope of the invention. Therefore, it is the object of the appended claims to cover all such variations and modifications as come within the true spirit and scope of the invention.

Technology Category: 3