Patent Publication Number: US-8983203-B2

Title: Face-tracking method with high accuracy

Description:
FIELD OF THE INVENTION 
     The present invention relates to computer vision, and more particularly, to a face-tracking method with high accuracy. 
     BACKGROUND OF THE INVENTION 
     Generally speaking, face-tracking refers to a computer vision technology that extracts the shapes of human faces in arbitrary digital images. It detects facial features and ignores anything else in surrounding, such as furniture or dogs. According to the related art, there are many conventional face tracking methods (e.g., snake, AAM, CLM . . . , etc.) based on face detection to detect face region and then set an initial shape (which is composed by feature points) inside the region, and the content of a given part in face region of an image is extracted to get features and then go fine tuning the face shape to fit features in the image face. 
     However, these methods may result in false shape extractions due to over/under face region detection or target-like background noises, and the following processes (e.g., the power saving application or the camera application) based on the face detection results would be affected by the false shape extractions. Therefore, there is a need for an innovative face-tracking scheme which is capable of extracting face shapes accurately. 
     SUMMARY OF THE INVENTION 
     The present invention provides a face-tracking method with high accuracy. The face-tracking method comprises generating an initial face shape according to a detected rectangle face region in an input image and a learned data base, wherein the initial face shape comprises an initial inner shape and an initial outer shape; generating a refined inner shape by refining the initial inner shape according to the features in the input image and the learned data base; and generating a refined outer shape by searching features composed by edges of the refined outer shape from the initial outer shape outward to limits of the defined possible outer shape. 
     These and other objectives of the present invention will no doubt become obvious to those of ordinary skill in the art after reading the following detailed description of the preferred embodiment that is illustrated in the various figures and drawings. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a flowchart illustrating a face-tracking method with high accuracy according to an exemplary embodiment of the present invention. 
         FIG. 2(   a ) is a diagram illustrating the face region of face detection result and initial inner shape of the face-tracking method with high accuracy according to an exemplary embodiment of the present invention. 
         FIG. 2(   b ) is a diagram illustrating the refined inner shape of the face-tracking method with high accuracy according to an exemplary embodiment of the present invention. 
         FIG. 3  is a diagram illustrating searching edges of the refined outer shape from the initial outer shape toward the limit of outer shape of the face-tracking method with high accuracy according to an exemplary embodiment of the present invention. 
         FIG. 4  is a diagram illustrating a direct searching process of the face-tracking method with high accuracy according to an exemplary embodiment of the present invention. 
         FIG. 5  is a diagram illustrating a 2D directional searching process of the face-tracking method with high accuracy according to another exemplary embodiment of the present invention. 
     
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     The main concept of the present invention is to improve the face-tracking accuracy especially when the background of an image is complicated or messy. In such condition, the performance of conventional face-tracking methods will be deteriorated, especially at face border. However, the present invention proposes a face-tracking method for analyzing images (i.e., face shape border detection) from the inside of a face toward outside with a predetermined distance so as to avoid the background noise interference and improve the face-tracking accuracy. 
     Please refer to  FIG. 1  and  FIG. 2 .  FIG. 1  is a flowchart illustrating a face-tracking method  100  with high accuracy according to an exemplary embodiment of the present invention.  FIG. 2(   a ) is a diagram illustrating the face region of face detection result and initial inner shape of the face-tracking method with high accuracy according to an exemplary embodiment of the present invention.  FIG. 2(   b ) is a diagram illustrating the refined inner shape of the face-tracking method with high accuracy according to an exemplary embodiment of the present invention. Provided that substantially the same result is achieved, the Steps in  FIG. 1  need not be in the exact order shown and need not be contiguous, that is, other Steps can be intermediate. Besides, some Steps in  FIG. 1  may be omitted according to various types of embodiments or requirements. In this embodiment, the detailed operation for face-tracking may comprise the following Steps: 
     Step  101 : Receive an input image  202  (Refer to  FIG. 2 ); 
     Step  102 : Generate an initial face shape according to the detected face region (the rectangle with dot line in  FIG. 2(   a )) of the input image  202  and a learned data base, wherein the initial face shape comprises at least an initial inner shape  204  and an initial outer shape (not shown here, defined in  FIG. 3  as ‘initial shape’);
 
Step  103 : Generate a refined inner shape  208  by refining the initial inner shape  204  in accordance with at least the input image  202  and the learned data base;
 
Step  104 : Generate a refined outer shape by searching an edge of the refined outer shape from the initial outer shape toward the limit of outer shape.
 
     In Step  101 , the input image  202  can be a frame of a video, a picture, and so on. After the input image  202  is received in Step  101 , Step  102  is executed to operate a face detection on the input image  202  to get a face region (the rectangle with dot line in  FIG. 2(   a )) for generating an initial inner shape  204  and an initial outer shape according to a learned data base. Please refer to  FIG. 2(   a ), which is a diagram illustrating the face detection of the face-tracking method  100  with high accuracy according to an exemplary embodiment of the present invention. An initial face shape at a detected face region may be set with help of the learned data base of many various “human faces”. More specifically, in Step  102 , a generic face shape model with n feature points is built according to the learned data base, and a generative shape of the initial face shape S i (θ), i=1, . . . , n, can be described as equation (1):
 
 S   i (θ)= sR (    s   i   +Γ i γ)+ t; θ={s,R,γ,t}   (1);
 
wherein the initial face shape comprises at least the initial inner shape  204  and the initial outer shape, and (  s i   , Γ i ) are the learned data base which comprises average shape  s i    and deformation span basics Γ 1 , and θ comprises geometric factors s, R, t, and Γ, wherein s represents a scaling factor, R represents a rotation matrix (composed by roll, yaw and pitch), t represents a translation factor, and γ represents deformation parameters which are adjustable parameters utilized for fitting various face shape in the input image  202 .
 
     It should be noted that in Step  102 , the geometric factors s, R, t, and γ are just rough values and under no posture conditions, e.g. roll, yaw, or pitch. In other words, these geometric factors are not refined yet, and have to be refined through further fitting process. Consequently, the initial inner shape  204  and the initial outer shape are just rough results as well and need to be refined at the following Step  103 . Moreover, the n feature points may have errors due to the difference between the generic face shape model of the learned data base and the real face shape in the input image  202 . 
     In Step  103 , some optimization algorithms are used to iteratively tune the initial shape to match the extracted features in real image face shape, and do not stop until some criteria are met. For better understanding of technical features of the present invention, one optimization algorithm to match an image (x i , y i ) with model as mentioned above to find optimum θ and z i  are described in equation (2): 
                     min         {     z   i     }       i   =   1     n     ,   θ       ⁢       ∑     i   =   1     n     ⁢       ρ   ⁡     (                [       x   i     ,       y   i     ;     z   i         ]     -       S   i     ⁡     (   θ   )              2     ;   σ     )       .               (   2   )               
However, this is for illustrative purposes only, and is not meant to be a limitation of the present invention. The optimization algorithms or schemes may be modified according to different optimization algorithms or schemes. As a person skilled in the art can readily understand details of the optimization methods described in equation (2), and further description is omitted here for brevity.
 
     Please refer to  FIG. 2(   b ), which is a diagram illustrating the refined inner shape of the face-tracking method  100  with high accuracy according to an exemplary embodiment of the present invention. The refined inner shape  208  thereby is generated through the optimization algorithms mentioned above, and the refined inner shape  208  may be supposed to be more accurate than the initial inner shape  204  as shown in  FIG. 2(   a ). For example, the position and the shape of the mouth in the refined inner shape  208  are presented more accurately comparing with the initial inner shape  204 . Furthermore, the geometric factors which describes a posture of the refined inner shape  208  is also refined in Step  103 , wherein the geometric factors comprises at least the scaling factor s, the rotation matrix R, and the translation factor t as mentioned above. In other words, the refined inner shape  208  comprises the posture, e.g. roll, yaw, or pitch information in Step  103 . 
     In general, since there is only skin around the inner face shape, and the background around the outer face shape may appear with some unexpected objects, the background around the outer face shape of the input image  202  is much more complicated than the background around the inner face shape. Considering the fact mentioned above, the present invention processes the refined inner shape  208  first, and then processes the refined outer shape or the whole face shape. In this way, the refined inner shape  208  would be generated stably and precisely. In Step  104 , after inner shape has been extracted, an initial outer shape and many scan line segments may be set for searching correct outer shape (face border) from inside to outside direction from a face center. 
     Please refer to  FIG. 3 , which is a diagram illustrating searching an edge of the refined outer shape from the initial outer shape toward the limit of outer shape according to an exemplary embodiment of the present invention. In  FIG. 3 , the initial outer face shape has 17 feature points and may be separated into several groups. For example, the 17 feature points may be separated into 4 groups, and more specifically, each of the groups has a common base point  302  which are selected from the points of the refined inner shape  208 . Then, 17 scan lines  304  which corresponding to the 17 feature points are generated from the corresponding common base points  302  toward the limit of outer face shape respectively, and 17 line segments  306  are set respectively on the 17 scan lines to indicate a searching range for the following searching process. 
     Any object outside the line segments  306  would be ignored, hence the searching process may avoid identifying most of the undesired objects at background and the searching process would be efficient. For example, line segments can be defined as a n dimensions 2D image point p i (xi, yi) arrays ArrayP k [p i ], i=0, 1, . . . , n−1 and k=0, 1, . . . , 16. Please note that this setting is initially under no posture (no roll, yaw and pitch) condition. The image coordinates of each point in the scan line segment arrays ArrayP k [p i ] should be transformed to correct position before doing searching operation. For example, each point p i  in ArrayP k [p i ] may be transformed to (as equation (3)):
 
 p′   i   =sRp   i   +t   (3);
 
wherein s, R, t are the scaling factor, the Rotation matrix (composed by head roll, yaw and pitch) and the translation factor as defined in equation (1). Therefore, refined arrays of the scan line segment  306  for searching usage is ArrayP k [p′ i ]. According to the embodiment of the present invention, the searching process may be configured to a direct searching in the one dimensional refined arrays ArrayP k [p′ i ] of the scan line segment  306  in an in-to-out direction.
 
     Firstly, the input image  202  is processed by any one well known edge detection method to get an ‘edge map’ for further processing, for example, the Sobel edge detection. However, this is for illustrative purposes only, and is not meant to be a limitation of the present invention. The edge detection method or scheme may be modified, wherein other methods may be employed according to different edge detection. Please refer to  FIG. 4 , which is a diagram illustrating a direct searching process according to an embodiment of the present invention. An edge map  402  is generated from the input image  202  according to the aforementioned edge detection method or scheme, Then, refined arrays ArrayP k [p′ i ] of the scan line segment  306  are searched from index i=0 to i=n−1 (i.e. from inward to outward direction) to find a maximum edge point, which is the border of the outer face shape we find in this Step, assume it is p max(i,k) , where (i, k) represents the maximum edge point at indexed number i in k scan line segment. 
     In another embodiment of the present invention, a searching process is configured to be 2D directional searching along the refined arrays ArrayP k [p′ i ] of the scan line segment  306  with a 2D patch ‘edge detector’. Please refer to  FIG. 5 , which is a diagram illustrating a 2D directional searching process according to another embodiment of the present invention. A 2D patch  502  as shown in  FIG. 5  is capable of performing a directional searching for the maximum edge point. The patch is a window (for example, 15×15 pixels, but not limited) centering at a point in the refined arrays ArrayP k [p′ i ] of the scan line segment  306  and is suitable for implementing a 2D edge detector. For better understanding of technical features of the present invention, one 2D edge detection method to search along the refined arrays ArrayP k [p′ i ] of the scan line segment  306  are described as an example. However, this is for illustrative purposes only, and is not meant to be a limitation of the present invention. Many well know 2D edge detection methods or schemes can be used to achieve the same objective also belong to the scope of the present invention. As result, the maximum edge point p max(i,k)  can be found as the border of the outer face shape. 
     After each maximum edge point p max(i,k)  has been found (if no edge found in some line segment, the maximum edge point in p max(i,k)  of the line segment would be omitted), one optimization algorithm similar to Equation (2) can be used to determine the refined outer shape. Equation (4) shows this optimization equation, where the shape (face border) generated from the searching process with total n maximum edge points (e.g., 17 points in this embodiment) is subtracted by predicted shape S k (θ) from the learned data base and z k  and θ are estimated by optimization and minimization process until some converge condition (stop criteria) met. Note that we only need take deformation parameters γ into account in θ (includes the scaling factor s, the rotation matrix (composed by roll, yaw and pitch) R, the translation factor t, and the deformation parameters γ, but not limited to) because the scaling factor s, the rotation matrix R, and the translation factor t are determined when we obtained the refined inner shape  208 . In addition, some geometry constrains, for example, reflection symmetry with respect to the left face and the right face, can be imposed upon the optimization and minimization process to improve the fitting correctness as Equation (5). 
                     min         {     z   k     }     ⁢   k     ,   θ       ⁢       ∑     k   =   0       n   -   1       ⁢     ρ   ⁡     (                [       p     max   ⁡     (     i   ,   k     )         ;     z   k       ]     -       S   k     ⁡     (   θ   )              2     ;   σ     )                 (   4   )               
.Assume center at (0,0,0), no rotation for (k=0; k&lt;n/2; k++)
 
|[ p   max(i,k)   ;z   k   ]−[p   max(i,n−k)   ;z   n−k ]|&lt;δ  (5)
 
Wherein |.| denotes the distance between two points, δ is a threshold value.
 
It is an advantage of the present invention that the present invention method can provide an improved flow for face-tracking process. In addition, the improved flow for face-tracking process is suitable for various kinds of, where a traditional face-tacking or detection process can be altered with ease based upon the embodiments disclosed above, to prevent the related art problems.
 
     Those skilled in the art will readily observe that numerous modifications and alterations of the device and method may be made while retaining the teachings of the invention. Accordingly, the above disclosure should be construed as limited only by the metes and bounds of the appended claims.