Abstract:
The invention discloses a face recognition method that reconstructs a 3D face model from a single face image, synthesizes a set of face images under different conditions (such as pose, light . . . ) via the 3D face model, feeds the set of face images under different conditions to the face recognition classifier for training, and making intermediate decisions whether to-be identified individual from a series of video frames is a legal system user by the face recognition classifier. Moreover, the method not only recognizes legal system users, but also rejects imposters, a function inspired by the idea of LLE. Finally, better reliability can be achieved by fusing temporal intermediate decisions.

Description:
BACKGROUND OF THE INVENTION 
     1. Field of the Invention 
     The present invention relates to a face recognition method, particularly reconstructing a 3D morphable face model from a single face image, and recognizing a series of face video frames from a video receiving apparatus by fusing similarity probability of each video frame. 
     2. Description of the Related Art 
     Face recognition is one of the most important research areas in computer vision and biometric recognition. For 2D face recognition, the main challenge is the variations of face images for the same individual under different capturing conditions such as angles, light source, etc. To overcome the problem, most developed algorithms rely on a large number of training examples under different conditions for each individual. However, it is difficult to collect 2D face images under a precise condition with the variation in real-world applications. 
     Recently, 3D face modeling from images is very popular for many applications, such as facial animation and face recognition. Model-based statistical techniques have been widely used for robust face modeling. Most of the previous 3D face reconstruction techniques require more than one 2D face image to achieve satisfactory 3D face modeling. Although another approach for 3D face reconstruction requires only a single image, the problem is restricted by requiring a statistical head model known as a priori. Besides, most of the previous face recognition method cannot differentiate imposters who have not been present before. For the increasingly popular face recognition system, simplifying sampling complexity and training difficulties, and increasing recognition accuracy are problems need to be solved. 
     SUMMARY OF THE INVENTION 
     To solve aforementioned problem, one objective of the present invention is to propose a face recognition method, which constructs a 3D face model using a single 2D face image, and simulate face image variations under different conditions. The synthesized face images are used to train a face recognition classifier. Besides, the present invention employs a locally linear embedding concept, which not only can recognize face images of legal system users, but also can reject imposters. Furthermore, the present invention temporally integrates single-frame decisions by a probabilistic approach to increase the reliability of face recognition. 
     To achieve aforementioned objective, one embodiment of the present invention proposes a face recognition method including obtaining an original face image from a database, reconstructing a 3D face model for the original face image using a 3D morphable face model, providing synthesized face images from the 3D face model for training a face recognition classifier, receiving a plurality of video frames of a face, obtaining a similarity probability for each video frame from the face recognition classifier, and fusing the temporal similarity probability for each video frame by accumulation to obtain a face recognition confidence value, and if the face recognition confidence value is larger than a threshold, the face is recognized to be the same as the face in the original face image. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a flowchart of the face recognition method of one embodiment of the present invention; 
         FIG. 2   a - FIG. 2   d  are diagrams showing the 3D morphable model reconstructed from an original face image according to one embodiment of the present invention; 
         FIG. 3  shows the synthesized images according to one embodiment of the present invention; and 
         FIG. 4  is a diagram showing the conceptual framework of fusing temporal information for face recognition of one embodiment of the present invention. 
     
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     The face recognition method of the present invention reconstructs a 3D face model from a single 2D face image using a 3D morphable model, and synthesizes a large set of 2D face images under different head poses for training the face recognition model. Also, this method temporally integrates single-frame recognition decisions via a probabilistic approach to increase recognition accuracy, and rejects non-system allowed imposters based on a LLE (locally linear embedding) method. 
     The flowchart of one embodiment of the present invention is shown in  FIG. 1 . This embodiment uses face recognition as an example, but the method can also be applied to recognizing diagrams or other images. In step S 01  of this embodiment, an original face image is obtained from a database, and a 3D face model is reconstructed from the original face image using a 3D morphable face model. The 3D morphable face model provides prior knowledge of geometry and texture information of a normal face. A 3D morphable face model can be represented by a 3D geometry data set, S=(x 1 , y 1 , z 1 , x 2 , y 2 , z 2 , . . . , x N , y N , z N )ε            , and 3D texture data set, T=(r 1 , g 1 , b 1 , r 2 , g 2 , b 2 , . . . , r N , g N , b N )ε          , where N is the number of sampling points for the curved surface of a 3D face. Then the 3D geometry data set and 3D texture data set of the 3D face is approximated by a linear combination of a plurality of eigenhead basis vectors, shown as follows:
                     S   ⁡     (   α   )       =       S   _     +       ∑     i   =   1     m     ⁢       α   i     ⁢     s   i                   (   1   )                 T   ⁡     (   β   )       =       T   _     +       ∑     i   =   1     m     ⁢       β   i     ⁢     t   i                   (   2   )               
wherein  S  is the mean shape vector,  T  is the mean texture vector, s i  is the i th  eigen-shape basis vector, t i  is the i th  eigen-texture basis vector, α=[α 1 , . . . , α m ] and β=[β 1 , . . . , β m ] are the shape and texture coefficients, respectively, to represent the 3D face.
 
     The embodiment uses 3D scans and image database of BU-3DFE (3D facial expression database) as the data source for producing eigen head basis vectors. Please referring to  FIG. 2   a  to  FIG. 2   d , they are conceptual diagrams of a 3D morphable face model in one embodiment of the present invention.  FIG. 2   a  shows the generic face model labeled with a plurality of feature points;  FIG. 2   b  is the original face image from a database;  FIG. 2   c  shows the reconstructed 3D face model of  FIG. 2   b  after registration, re-sampling and smoothing; and  FIG. 2   d  shows the triangulation detail of  FIG. 2   c.    
     The abovementioned 3D face reconstruction is initialized by the feature points in  FIG. 2   a , and then a shape data set, α, a texture data set, β, and a plurality of 3D characteristic parameters are obtained respectively by photometric error minimization to achieve a 3D morphable face model best represent the input face. The 3D characteristic parameters are found from minimizing a cost function of the following form:
 
∥ I   input   −I   model (α,β, P   ortho   ,f,R,t   2d )∥  (3)
 
wherein I input  is the original face image, P ortho  is an orthographic projection matrix, f is the scaling factor, R is the 3D rotation matrix, and t 2d  is the translation vector. The minimization of function (3) can be solved by using the Levenberg-Marquart (LM) optimization.
 
     In step S 02 , possible variations of the image are simulated by the reconstructed 3D face model, and used to train a face recognition classifier for the original face image.  FIG. 3  shows the 2D face images synthesized with pre-defined characteristic parameters, based on the original 2D face image, by using the reconstructed 3D face model. The image in the middle is the original face image, and others are the synthesized face images under different poses. 
     In step S 03 , a plurality of video frames of a face are received, and a similarity probability is obtained for each video frame from the face recognition classifier. In step S 04  the probability of each video frame is fused temporally to obtain a confidence value. If the confidence value is greater than a threshold, the received face video frames are recognized to have the same identity as the individual in the original face image. The method for fusing previous recognition results is shown in expression (4): 
                       P   i   t     =       (       ω   ×     P   i     t   -   1         +       (     1   -   ω     )     ×     P   i     t   ,   SVC           )         ∑   i     ⁢     (       ω   ×     P   i     t   -   1         +       (     1   -   ω     )     ×     P   i     t   ,   SVC           )           ,     i   =   1     ,   2   ,   …   ⁢           ,   n           (   4   )               
wherein ω is a forgetting factor, P i   t,SVC  is a posterior probability denoting the probability that this user belongs to class i decided by a single video frame at time t. In one embodiment, we merge P i   t,SVC  with the previously accumulated probability P i   t−1  and obtain a final similarity probability P i   t  accumulated up to time t.
 
     On the other hand, when it comes to an applicable face recognition system, it must have the ability to reject imposters who have not been present before and only accept legal system users. Borrowing the idea from LLE that nonlinear data is locally linear on a manifold, if a to-be recognized image cannot be well-reconstructed linearly via the training images of the same person, the image is likely to be from an imposter. Therefore, a recognized face I t  at frame t is supposed to be reconstructed well by its K nearest neighbors in the predicted class c. The similarity error ε t  of an input face I t  is defined to be: 
                       ɛ   t     ⁡     (   w   )       =            I   t     -       ∑     j   =   1     K     ⁢       w   j     c   ,   t       ⁢     I   j   c                        (   5   )               
wherein w j   c,t  denotes the weights of K nearest neighbors in class c that best approximate said video frame I t , and
 
                   ∑     j   =   1     K     ⁢     w   j     c   ,   t         =   1     ;       {     I   j   c     }       j   =   1     K           
is the set of K nearest neighbors of the same class. Then a sigmoid function is applied to transform ε t  to the corresponding rejection probability P rej   t  with a recursive definition:
 
                       P   rej   t     =     1   -     exp   ⁡     (       -     ɛ   t         ɛ   threshold   c       )           ,         P   ^     rej   t     =       ω   ×       P   ^     rej     t   -   1         +       (     1   -   ω     )     ×     P   rej   t                   (   6   )               
wherein ε threshold   c  is the parameter controlling the rejection rate. It turns out that face I t  cannot belong to a system user if it has a large rejection probability {circumflex over (P)} rej   t . Please referring to  FIG. 4 , it is a diagram showing the conceptual framework of fusing temporal information for face recognition. Finally, the similarity probability P i   t  is summed up with the rejection probability {circumflex over (P)} rej   t , and normalized to obtain the posterior probability {circumflex over (P)} i   t  of the video frame belonging to class c up to time t, which can be expressed by expression (7):
 
                         P   ^     i   t     =         β   t     ×     (       ω   ×     P   i     t   -   1         +       (     1   -   ω     )     ×     P   i     t   ,   SVC           )                 (       ω   ×       P   ^     rej     t   -   1         +       (     1   -   ω     )     ×     P   rej   t         )     +       β   t     ×                   ∑   i     ⁢     (       ω   ×     P   i     t   -   1         +       (     1   -   ω     )     ×     P   i     t   ,   SVC           )                 ,           (   7   )               
wherein i=1, 2, . . . , n, and β t =1−{circumflex over (P)} rej   t  is the acceptance rate at time t, and {circumflex over (P)} t =[{circumflex over (P)} 1   t , {circumflex over (P)} 2   t , . . . , {circumflex over (P)} rej   t ] T  is the posterior probability distribution. If {circumflex over (P)} i   t  exceeds a threshold P th , the present invention can decide the identity of the face in video frames. If
 
               arg   ⁢           ⁢       max   i     ⁢       P   ^     i   t         =     n   +   1           
(class n+1 for the imposter class), then the to-be recognized face is rejected; otherwise, the face is determined to be of a legal system user and his/her identity is further recognized, i.e.
 
     
       
         
           
             
               c 
               = 
               
                 arg 
                 ⁢ 
                 
                     
                 
                 ⁢ 
                 
                   
                     max 
                     i 
                   
                   ⁢ 
                   
                     
                       P 
                       ^ 
                     
                     i 
                     t 
                   
                 
               
             
             , 
             
               c 
               = 
               1 
             
             , 
             2 
             , 
             … 
             ⁢ 
             
                 
             
             , 
             
               n 
               . 
             
           
         
       
     
     In conclusion, the present invention reconstructs a face model via a 3D morphable model, and synthesizes variations of a face image as the training database for face recognition classifier. In the process of constructing 3D face model and training face recognition classifier, only a 2D image is needed for the user of each class. This improves from the statistical approach which requires collecting a large amount of face training samples. 
     The embodiments described above are to demonstrate the technical contents and characteristics of the preset invention to enable the persons skilled in the art to understand, make, and use the present invention. However, it is not intended to limit the scope of the present invention. Therefore, any equivalent modification or variation according to the spirit of the present invention is to be also included within the scope of the present invention.