Abstract:
A gesture recognition system includes a candidate node detection unit coupled to receive an input image in order to generate a candidate node; a posture recognition unit configured to recognize a posture according to the candidate node; a multiple hands tracking unit configured to track multiple hands by pairing between successive input images; and a gesture recognition unit configured to obtain motion accumulation amount according to tracking paths from the multiple hands tracking unit, thereby recognizing a gesture.

Description:
BACKGROUND OF THE INVENTION 
       [0001]    1. Field of the Invention 
         [0002]    The present invention generally relates to a gesture recognition system, and more particularly to a gesture recognition system capable of being performed in a complex scene. 
         [0003]    2. Description of Related Art 
         [0004]    Natural user interface, or NUI, is a user interface that is invisible and requires no artificial control devices such as a keyboard and mouse. Instead, the interaction between humans and machines is achieved, for example, through hand postures or gestures. Kinect by Microsoft is one example of a vision-based gesture recognition system that uses postures and/or gestures to facilitate interaction between a user and a computer. 
         [0005]    Conventional vision-based gesture recognition systems are liable to make erroneous judgments on object recognition owing to surrounding lighting and background objects. After extracting features from a recognized object (a hand in this case), classification is performed via a training set, from which a gesture is recognized. Conventional classification methods suffer either large training data or erroneous judgments due to unclear feature. 
         [0006]    For the foregoing reasons, a need has thus arisen to propose a novel gesture recognition system that is capable of more accurately and fast recognizing postures and/or gestures. 
       SUMMARY OF THE INVENTION 
       [0007]    In view of the foregoing, it is an object of the embodiment of the present invention to provide a robust gesture recognition system that may perform properly in a complex scene and reduce complexity of posture classification. 
         [0008]    According to one embodiment, a gesture recognition system includes a candidate node detection unit, a posture recognition unit, a multiple hands tracking unit and a gesture recognition unit. The candidate node detection unit receives an input image in order to generate a candidate node. The posture recognition unit recognizes a posture according to the candidate node. The multiple hands tracking unit tracks multiple hands by pairing between successive input images. The gesture recognition unit obtains motion accumulation amount according to tracking paths from the multiple hands tracking unit, thereby recognizing a gesture. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0009]      FIG. 1  shows a block diagram illustrated of a gesture recognition system according to one embodiment of the present invention; 
           [0010]      FIG. 2  shows a flow diagram illustrating steps performed by the candidate node detection unit of  FIG. 1 ; 
           [0011]      FIG. 3  shows a flow diagram illustrating steps performed by the posture recognition unit of  FIG. 1 ; 
           [0012]      FIG. 4  shows an exemplary distance curve; 
           [0013]      FIG. 5  shows exemplary classification of the postures according to the amount of recognized unfolding fingers; 
           [0014]      FIG. 6  exemplifies multiple hands being tracked by pairing between successive frames; 
           [0015]      FIG. 7A  shows a natural user interface for drawing on a captured image with one hand; and 
           [0016]      FIG. 7B  shows an exemplary gesture using the postures of  FIG. 7A . 
       
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       [0017]      FIG. 1  shows a block diagram illustrated of a gesture recognition system  100  according to one embodiment of the present invention. In the embodiment, the gesture recognition system  100  primarily includes a candidate node detection unit  11 , a posture recognition unit  12 , a multiple hands tracking unit  13  and a gesture recognition unit  14 , details of which will be described in the following. The gesture recognition system  100  may be performed by a processor such as a digital image processor. 
         [0018]      FIG. 2  shows a flow diagram illustrating steps performed by the candidate node detection unit  11  of  FIG. 1 . In step  111  (i.e., interactive feature extraction), features are extracted according to color, depth and motion, thereby generating a color reliability map, a depth reliability map and a motion reliability map. 
         [0019]    Specifically speaking, the color reliability map is generated according to skin color of a captured input image. In the color reliability map, a higher value is assigned to a pixel that is more like the skin color. 
         [0020]    The depth reliability map is generated according to hand depth of the input image. In the depth reliability map, a higher value is assigned to a pixel that is within a hand depth range. In one exemplary embodiment, a face is first recognized by a face recognition technique, and the hand depth range is then determined with respect to depth of the recognized face. 
         [0021]    The motion reliability map is generated according to motion of a sequence of input images. In the motion reliability map, a higher value is assigned to a pixel that has more motion, for example, measured by sum of absolute differences (SAD) between two input images. 
         [0022]    In step  112  (i.e., natural user scenario analysis), weightings of the extracted color, depth and motion are determined with respect to operation status, such as initial statement, motion or whether hand is close to face. Table 1 shows some exemplary weightings: 
         [0000]    
       
         
               
               
             
               
               
               
               
             
               
               
               
               
               
               
             
               
               
               
               
               
               
             
           
               
                 TABLE 1 
               
             
             
               
                   
               
               
                 Operation status 
                   
               
             
          
           
               
                 Initial 
                   
                 Hand close 
                 Weight 
               
             
          
           
               
                 statement 
                 Motion 
                 to face 
                 Color 
                 Depth 
                 Motion 
               
               
                   
               
             
          
           
               
                 No 
                 Strong 
                 No 
                 0.286 
                 0.286 
                 0.429 
               
               
                 No 
                 Strong 
                 Yes 
                 0.25 
                 0.375 
                 0.375 
               
               
                 No 
                 Low 
                 No 
                 0.5 
                 0.5 
                 0 
               
               
                 No 
                 Low 
                 Yes 
                 0.4 
                 0.6 
                 0 
               
               
                 Yes 
                 Strong 
                 Don&#39;t 
                 0 
                 0.4 
                 0.6 
               
               
                   
                   
                 care 
               
               
                 Yes 
                 Low 
                 Don&#39;t 
                 0 
                 1 
                 0 
               
               
                   
                   
                 care 
               
               
                   
               
             
          
         
       
     
         [0023]    Finally, in step  113 , the color reliability map, the depth reliability map and the motion reliability map are combined with the respective weightings given in step  112 , thereby generating a hybrid reliability map, which provides a detected candidate node. 
         [0024]      FIG. 3  shows a flow diagram illustrating steps performed by the posture recognition unit  12  of  FIG. 1 . In step  121  (i.e., dynamic palm segmentation), the detected hand (from the candidate node detection unit  11 ) is segmented into a palm (which is used later) and an arm (which is discarded). 
         [0025]    In step  122  (i.e., high accuracy finger recognition), a distance curve is generated by recording relative distances between the center of the segmented palm and perimeter (or boundary) of the segmented palm.  FIG. 4  shows an exemplary distance curve, which has five peaks, indicating that five unfolding fingers have been recognized. 
         [0026]    In step  123  (i.e., hierarchical posture recognition), a variety of recognized postures are classified for facilitating the following process.  FIG. 5  shows exemplary classification of the postures according to the amount of recognized unfolding fingers. When recognizing a posture in a hierarchical manner, the amount of unfolding fingers is first determined. Jointed fingers may be detected by computing the width of the recognized fingers. Next, hole and its width indicating folded finger(s) between unfolding fingers are then determined. 
         [0027]    In the multiple hands tracking unit  13  of  FIG. 1 , multiple hands are tracked by pairing (or matching) between successive frames as exemplified in  FIG. 6 , in which tracking path exists between a pair of matched track hands. In a case of unmatched track hand due to object leave, the corresponding tracking path may be deleted. In another case of unmatched track hand due to occlusion, an expected track hand may be generated by extrapolation technique. In a further case of unmatched track hand due to object arrival, a new posture need be recognized and then a new path may then be tracked. In case of unmatched track hands, feedback may be fed back to the candidate node detection unit  11  (as shown in  FIG. 1 ) to discard the associated candidate node. 
         [0028]    In the gesture recognition unit  14  of  FIG. 1 , the tracking paths are monitored to obtain their motion accumulation amount along axes in a three-dimensional space, thereby recognizing a gesture. The recognized gesture may then be fed to a natural user interface for performing a pre-defined task. 
         [0029]      FIG. 7A  shows a natural user interface for drawing on a captured image with one hand. As exemplified in  FIG. 7B , after the posture No. 1 (not shown in  FIG. 7B ), a user may draw a line using a series of the posture No. 2, constructing a gesture, during which the user may change color using the posture No. 3 or No. 4. 
         [0030]    Although specific embodiments have been illustrated and described, it will be appreciated by those skilled in the art that various modifications may be made without departing from the scope of the present invention, which is intended to be limited solely by the appended claims.