Patent Publication Number: US-11048926-B2

Title: Adaptive hand tracking and gesture recognition using face-shoulder feature coordinate transforms

Description:
BACKGROUND OF THE INVENTION 
     Field of the Invention 
     This invention relates to human-computer interface, and in particular, it relates to a system and method for adaptive hand tracking and gesture recognition. 
     Description of Related Art 
     Hand tracking and gesture recognition system is used to create interactions between a computer or machine and a human user where the recognized gestures of the human user can be used for controlling the machines or conveying information. It falls in the domain of human-computer interface (HCI) or human-machine interface (HMI). 
     One approach that satisfies the advanced requirements of hand gesture based HCI is glove-based sensing, which employs sensors (mechanical or optical) attached to a glove that converts finger flexions into electrical signals for determining the hand posture. This approach is not natural, as it requires the user to carry the special glove with wires, and to perform calibration and setup. 
     Another, more natural, approach is vision-based sensing, which uses a camera to capture the hand gesture. Hand gestures can be static (a posture) or dynamic (a sequence of postures). Natural hand gesture acquisition vision systems use either a color camera (referred to as RGB camera), a depth camera, or the combination of the two (referred to as RGB-D camera, which detects both color and depth information of a scene). 
     The methods for vision-based hand gesture recognition can be divided into model-based methods and appearance-based methods. Model-based methods generate 3D model hypotheses on hand gestures and evaluate them on the acquired images. This is fundamentally an optimization problem whose cost function measures the distance between the images that are expected due to a model hypothesis and the actual ones. Up to 28 degrees of freedom (DOF) have been used in various 3D hand gesture models, leading to significant computational burden on the model-to-image transformations and the optimization routine, although 3D inputs from a depth camera can significantly simplify the computation for model-based methods. 
     Appearance-based methods directly use the information contained in the images, and assume no explicit 3D hand model. When only the hand appearances are known, differentiating between gestures typically involves certain statistical classifiers using a set of image features of the hand, which can be extracted from either RGB or depth images of the hand, or the combination of the two. It should be noted that in recent years, deep neural networks (DNNs) have been widely used as image classifiers, and they are equivalent to traditional statistical classifiers with image feature detectors combined, but generally need more samples to train and can produce more robust classification results when it is done properly. 
     Due to person-to-person variations in hand appearances, gesture habits, and body orientation or movements, training a hand gesture classifier needs large amount of samples with ground truth, which in turn requires significant effort and cost in sample collections and annotations. 
     SUMMARY 
     For many HCI applications, when a person performs hand gestures, the hand is held in front of the upper body. There are some dominant features associated with the human upper body that are relatively easy and robust to detect, for example, the features on the face (eyes and mouth etc.), the shoulder extremes (approximately the upper and outer end point of each shoulder), etc. Based on such insight, in embodiments of the present invention, in the first step of hand tracking and gesture recognition, these features on the face and shoulder can be detected, and their geometrical relations can be used to judge whether the face/head has turned to the side relative to the shoulder. If not, a coordinate system can be established using these face-shoulder features, and the hand images are transformed into this new coordinate system. This essentially defines hand gestures relative to each user&#39;s face and shoulder, thus significantly reducing the dimensions of the feature space for hand gestures. This face-shoulder coordinate transform can be combined either with traditional appearance-based hand gesture recognition methods, or with a hybrid deep neural network for RGB-D images that carries out separate preprocessing of the depth image instead of the conventional convolution layers for RGB images. 
     An object of the present invention is to provide an improved hand tracking and gesture recognition method. 
     Additional features and advantages of the invention will be set forth in the descriptions that follow and in part will be apparent from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims thereof as well as the appended drawings. 
     To achieve the above objects, the present invention provides a method for hand tracking and gesture recognition which includes: receiving a current image from a camera system, the current image containing a face and shoulders of a human user; from the current image, detecting face and shoulder features of the user; establishing a current face-shoulder coordinate system based on the detected face and shoulder features; transforming a previous hand region of interest, which is a region of interest containing a hand of the user and which has been generated from a previous image received from the camera system, from a previous face-shoulder coordinate system into the current face-shoulder coordinate system to generate a transformed previous hand region of interest; extracting a current hand region of interest from the current image, wherein the current hand region of interest is defined in the current face-shoulder coordinate system; and performing gesture recognition using the transformed previous hand region of interest and the current hand region of interest in the current face-shoulder coordinate system. 
     In some embodiments, wherein the detecting step includes detecting the user&#39;s eyes, mouth and shoulders and computing current left and right eye center positions, a current mouth center position, and current left and right shoulder curvature extreme positions of the user, and wherein the establishing step includes establishing the current face-shoulder coordinate system based on the current left and right eye center positions, the current mouth center position, and the current left and right shoulder curvature extreme positions. 
     In another aspect, the present invention provides a computer program product comprising a computer usable non-transitory medium (e.g. memory or storage device) having a computer readable program code embedded therein for controlling a data processing apparatus (e.g. a computer), the computer readable program code configured to cause the data processing apparatus to execute the above method. 
     It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIGS. 1( a ) and 1( b )  schematically illustrate a vision-based hand gesture acquisition system in which embodiments of the present invention may be implemented.  FIG. 1( a )  shows an image acquisition and processing system;  FIG. 1( b )  shows calibration outputs of the RGB-D camera. 
         FIG. 2  is a flow chart that illustrates a hand tracking and gesture recognition process according to embodiments of the present invention. 
         FIG. 3  is a flow chart that illustrates an adaptive hand tracking and gesture recognition process in the process shown in  FIG. 2 . 
         FIG. 4  schematically illustrates a face-shoulder coordinate, showing key face-shoulder features and the definition of a right-hand face-shoulder coordinate system. 
         FIG. 5  is a flow chart that illustrates a face-shoulder coordinate system initialization process in the process shown in  FIGS. 2 and 3 . 
         FIG. 6  is a flow chart that illustrates a face-shoulder coordinate system update process in the process shown in  FIG. 3 . 
         FIG. 7  schematically illustrates a hybrid hand gesture recognition method according to embodiments of the present invention. 
     
    
    
     DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
     The hand tracking and gesture recognition methods according to embodiments of the present invention are based on the inventors&#39; insights and recognition of the fact and assumption that for many HCI applications, most of the time when a person performs hand gestures, the hand is held in front of the upper body. There are some dominant features associated with the human upper body that are relatively easy and robust to detect, for example, the features on the face (eyes, eyebrows, mouth and nose etc.), the shoulder extremes (approximately the upper and outer end point of each shoulder), etc. As such, in embodiments of the present invention, in the first step of hand tracking and gesture recognition, these features on the face and shoulder (called face-shoulder features in this disclosure) are detected, and their geometrical relations are used to judge whether the face/head has turned to the side relative to the shoulder. If not, a coordinate system (referred to as the face-shoulder coordinate system) is established using these face-shoulder features, and the hand images are transformed into this new coordinate system. This essentially defines hand gestures relative to each user&#39;s face and shoulder, thus significantly reducing the dimensions of the feature space for hand gestures. It is possible that one or two of the face-shoulder features is missing due to hand occlusion as the hand is most likely between the face-shoulder plane and the camera; in this case, the face-shoulder coordinate system can also be updated by matching those still visible face-shoulder features to historic records (previous image frames during the same gesture recognition session). 
     A key advantage of embodiments of this invention is to dramatically reduce the dimensions of the feature space for hand gestures, thus to (1) reduce the cost associated with hand gesture sample collection and annotation; (2) improve the robustness of hand tracking and the accuracy of hand gesture recognition; and (3) enable the adoption of simpler algorithms for hand tracing and gesture recognition. 
     An implementation of the hand tracking and gesture recognition system of a preferred embodiment includes an RGB-D camera connected to a computer or machine (for simplicity, “machine” will be used in the remaining of this disclosure). The machine includes one or more processors and non-volatile memories that store machine executable program instructions, and/or logic circuits configured to process signals and data, to implement the hand tracking and gesture recognition methods described below. The machine may further include a display screen or speaker for displaying or otherwise convey information to the user. The machine also includes a camera controller and necessary processors that control the acquisition of images and carry out the computations as detailed below, both in real time (see  FIG. 1( a ) ). RGB and depth information is complementary and can simplify the hand tracking and gesture recognition procedures and improve the robustness. However, it should be noted that the system can also be implemented with only an RGB camera or only a depth camera, with appropriate modifications to the methods described below. As an example, this disclosure uses RGB-D camera and the resulting RGB and depth images in the following descriptions. 
     The RGB-D camera is calibrated prior to use to establish the intrinsic parameters of the individual RGB and depth cameras within the RGB-D camera and their spatial relationship (extrinsic parameters). The intrinsic parameters of a camera include, but are not limited to, focal length, principle point, radial and tangential distortions etc. The extrinsic parameters between the RGB and depth cameras include a rotation matrix and a translation vector (see  FIG. 1( b ) ). Any suitable method may be used to perform the camera calibration, which are generally known to those skilled in the art. 
     The overall hand tracking and gesture recognition procedure according to embodiments of the present invention is shown in  FIG. 2 . While the subject (user) is positioned in front of the camera (step S 201 ), RGB-D images are captured, corrected and aligned (step S 202 ). Aligning (or fusing) of the RGB and depth images is done by using the rotation matrix and translation vector described above, to generate an RGB-D image pair. Then the face-shoulder coordinate system XYZ is initialized with features detected from the RGB-D image pair (step S 203 , details further described below). Afterwards, the machine instructs the subject to start gesturing to interact with the machine (step S 204 ), and the system continuously captures RGB-D images (step S 205 ) and carries out adaptive hand tracking and gesture recognition (step S 206 , details further illustrated below), until a predefined “ending” gesture is recognized or other stop command is received (“Y” in step S 207 ). When no “ending” gesture is recognized and no other stop command is received (“N” in step S 207 ), the system executes tasks corresponding to the recognized hand gesture according to pre-defined rules (step S 208 ). 
       FIG. 3  illustrates the process (S 206  in  FIG. 2 ) of adaptive hand tracking and gesture recognition using face-shoulder feature coordinate transforms. The steps shown in  FIG. 3  are performed for an N-th image. After the face-shoulder coordinate system initialization (step S 310 , or step S 203  in  FIG. 2 ), whenever a new (N-th, or “current”) corrected and aligned RGB-D image pair is received (data D 301 , and “Y” in step S 301 ), the system tries to update the face-shoulder coordinate system with image features from the current image pair to generate an updated (current) face-shoulder coordinate system XYZ N  (step S 302 ). The region of interest (ROI) that contains the hand in the previous ((N−1)-th) hand image pair (hand ROI (N-1)  in data D 302 ) is transformed from the previous face-shoulder coordinate system XYZ (N-1)  to the updated face-shoulder coordinate system XYZ N  (step S 303 ), before conducting hand tracking and gesture recognition in the updated face-shoulder coordinate system (steps S 304 -S 309 ). The transforming step S 303  is performed by computing a transformation that transforms the previous face-shoulder coordinate system to the current face-shoulder coordinate system, and then applying the same transformation to the previous hand ROI, to generate the transformed previous hand ROI. This transforms the hand image pairs from different images ((N−1)-th and N-th) to the same coordinate system (XYZ N ) before applying hand tracking and gesture recognition, and effectively act as a normalization process to reduce the variations in the hand gesture due to body rotation or other body movements. It should be pointed out that, in preferred embodiments, the coordinate transform is done only when the face-shoulder rotation or other movement is consistent. For example, if only the head is rotated while the shoulder is not, the geometry relationship check among the features will detect it and the coordinate system update won&#39;t be carried out. In other words, in the preferred embodiments, step S 303  includes first determining whether the face-shoulder rotation or other movement is consistent, and updating the face-shoulder coordinate system only when the face-shoulder rotation or other movement is consistent. 
     After the hand images are transformed to the updated face-shoulder coordinate system (step S 303 ), hand tracking and gesture recognition may be implemented using any suitable algorithm, one example of which is shown in  FIG. 3 , steps S 304 -S 309 . In this example, hand tracking is first performed in the current face-shoulder coordinate system XYZ N  (step S 304 ) to attempt to generate a hand movement (translation and rotation) that moves the hand in the previous ((N−1)-th) hand ROI to the hand in the current (N-th) image. This tracking is performed in the current face-shoulder coordinate system XYZ N  because the previous hand ROI has already been transformed into this coordinate system. If the hand tracking is successful (“Y” in step S 305 ), the current hand ROI, i.e. the ROI in the current image that contains the user&#39;s hand, is extracted based on the tracking (step S 308 ). This current hand ROI is defined in the current face-shoulder coordinate system. If the hand tracking is unsuccessful (“N” in step S 305 ), a hand search window is defined in the current image based on user statistics or a predefined window size (step S 306 ), the hand is detected in the hand search window using a suitable image recognition algorithm (step S 307 ), and the current hand ROI is extracted from the current image based the detected hand (step S 308 ). Then, hand gesture is recognized based on the hand ROIs from the current and one or more previous images, or the current hand ROI N  is put in the hand ROI queue to be used for gesture recognition in subsequent iterations (step S 309 ). 
     Note that in a simple hand tracking method, only a single previous ROI and the current ROI are needed to perform hand tracking. In more sophisticated tracking methods, multiple (two or more) previous ROIs are used for hand tracking. In the latter case, the multiple previous ROIs that have been put in the queue are transformed into the updated (current) face-shoulder coordinate system XYZ N  in step S 303  to perform tracking. 
     The face-shoulder coordinate system is defined on some key features in the face-should region of the body, as shown in  FIG. 4 . The key features include the two eyes (E R  and E L ) and the mouth (M C ), and the shoulder extremes (S R  and S L ), which may be defined, for example, as the respective points in the shoulder outer profiles that have highest curvatures. In the case that one or both eyes are occluded by the hand, the corresponding one or both eyebrows can be used to replaces the one or both eyes, and in the case the mouth is occluded by the hand, the nose can be used to replace the mouth. For brevity, for all procedures described below, “eyes” can represent either the two eyes or the two eyebrows or one eye and one eyebrow, and “mouth” can represent either the mouth or the nose, though their detections may involve somewhat different algorithms. For establishing the face-shoulder coordinate system, only the center positions of these features are used. The replacement of eyes by eyebrows and mouth by nose only introduce very small error in the origin of the face-shoulder coordinate system and the impact can be ignored for the purpose of hand tracking and gesture recognition. Alternatively, position corrections may be applied before using the eyebrow or nose as the eye or mouth position, for example, based on certain assumptions about the average distances between the eyebrow and the eye and between the nose and the mouth. The face-shoulder coordinate system essentially assume the key features (or their centers) are in the same plane. 
       FIG. 5  details how the face-shoulder coordinate system is initialized (step S 203  of  FIG. 2  and step S 310  of  FIG. 3 ). From the corrected RGB-D image pair (data D 501 ), the process detects a face-shoulder (FS) feature in the RGB-D images (step S 501 ), and extracts a corresponding face-shoulder regions of interest (FS ROI) from the RGB-D images (step S 502 ). The process then detects the eyes and mouth in the extracted ROI (step S 503 ), and computes the left and right eye centers E L  and E R  and the mouth center M C  (step S 504 ). The process also detects the shoulders in the extracted ROI (step S 505 ), and computes the left and right shoulder curvature extremes S L  and S R  (step S 506 ). The process computes the 3D coordinates of the eye centers E L  and E R , the mouth center M C , and the shoulder curvature extremes S L  and S R  (step S 507 ), based on the depth camera&#39;s intrinsic parameters (data D 502 ). Note that these 3D coordinates are defined in the camera&#39;s coordinate system, not the face-shoulder coordinate system. 
     The process then computes a topology of the eye centers E L  and E R , the mouth center M C , and the shoulder curvature extremes S L  and S R  (step S 508 ), and determines whether the topology is ideal (step S 509 ), in the following manner. 
     During this initialization stage, since the subject is instructed to face the camera in a generally front and centered position, the subject&#39;s face-shoulder plane is typically not rotated too far from a plane perpendicular to the camera&#39;s optical axis. The geometrical relationships of the five key feature centers are examined to confirm this. If the subject&#39;s body is not rotated too far from the perpendicular plane (perfect or near perfect orientation), the following ratios should be close to 1: 
                     r   LR     =         E   R     ⁢     S   R           E   L     ⁢     S   L                 (   1   )                 r   LRT     =         E   R     ⁢     M   C           E   L     ⁢     M   C                 (   2   )                 r   LRB     =         S   R     ⁢     M   C           S   L     ⁢     M   C                 (   3   )               
The ratios within a certain range of 1 may be considered “ideal” (for example, the range can be chosen in such a way to guarantee the subject&#39;s face-shoulder plane is no more than 30 degrees rotated from the plane perpendicular to the camera&#39;s optical axis). This is to prevent the subject from facing the camera in substantially rotated fashion that may lead to hand tracking and gesture recognition problems. These initial topology ratios are recorded for future verification of whether the subject rotated his face relative to his shoulders. The ratios in Eqs. 1-3 are only an example of what may define a topology of the five key features; other parameters may be suitably used to define the topology of the five key features.
 
     If the topology is ideal (“Y” in step S 509 ), the positions of the eye centers E L  and E R , the mouth center M C , and/or the shoulder curvature extremes S L  and S R  are used to establish an orthogonal face-shoulder coordinate system XYZ based on a set of pre-defined formulas (step S 510 ). In one example (see  FIG. 4 ), (1) the origin O of the face-shoulder coordinate system is defined to be the midpoint between the left and right shoulder curvature extremes S L  and S R , (2) the X direction is defined to be the (O, M C ) direction, (3) the Y direction is defined to be the (O, S R ) direction, and (4) the Z direction is defined as orthogonal to the plane defined by (S L , S R , M C ). The length unit may be defined based on the distance E L E R  or S L S R . Other suitable convention and formulas may also be used to define the face-shoulder coordinate system. 
     If the topology is not ideal (“N” in step S 509 ), an instruction is displayed to remind the user to rotate toward the camera (step S 511 ). For example, if E R S R  is longer than E L S L , the instruction will instruct the user to turn toward the right. Then, an RGB-D image is obtained (captured, corrected and fused) again (step S 512 ), and steps S 501  to S 509  are repeated using the newly obtained RGB-D image to establish the face-shoulder coordinate system. This coordinate initialization process generates ROIs for the FS area, eyes, mouth and shoulders, as well as the face-shoulder coordinate system XYZ (data D 503 ). 
       FIG. 6  details how the face-shoulder coordinate system is updated once the hand gesture interactions with the machine started (step S 302  of  FIG. 3 ). There are some important differences between update and initialization of the face-shoulder coordinate system. First, in the update process, the detections of face-shoulder features utilize tracking (step S 601 ) whenever possible to quickly find the face-shoulder ROI to reduce the computation. Tracking is performed on the RGB-D image (data D 601 ) using face-shoulder ROI and related features in the previous image (data D 602 ), to attempt to generate a movement (translation and rotation) that moves the features in the previous image to the features in the current image. Based on successful tracking (“Y” in step S 602 ), the FS ROI is extracted (step S 603 ) and the FS features (eyes, mouth, shoulders) are detected in the FS ROI (step S 604 ). If the tracking fails (“N” in step S 602 ), the process falls back on the features (eyes, mouth, shoulders) alone to detect them in the entire image (step S 605 ). 
     If the FS features (eyes, mouth, shoulders) are successfully detected (“Y” in step S 606 ), the process computes the eye centers E L  and E R , the mouth center M C , and the shoulder curvature extremes S L  and S R  (step S 607 ), and computes the 3D coordinates (in the camera&#39;s coordinate system) of these five key features using the depth camera intrinsic parameters (data D 603 ), as well as computing the topology of the five key features (step S 608 ). In step S 608 , the topology is computed from the 3D coordinates using the same equations 1-3 as in step S 508  of the initialization process. 
     A second difference between updating and initialization is that in updating, the key feature topology is checked whether it is affine transformation of the initial topology that was calculated in step S 508  (step S 609 ). Since it is assumed that the key features are in a plane, this can be done by simply respectively comparing the three ratios in Eq. 1-3 to see whether they are respectively equal to the three ratios of the initial topology within predefined thresholds. If the key feature topology is affine transformation of the initial topology (“Y” in step S 609 ), the updated face-shoulder coordinate system XYZ N  is established in a manner similar to step S 510  (step S 610 ). 
     A third difference between updating and initialization is that in updating, if the gesturing hand occlude one or more key face-shoulder features (note that since the eye and eyebrow on the same side is considered to be the same feature, if and only if both the eye and eyebrow on the same side is occluded, that feature is treated as occluded) (“N” in step S 606 ), other successfully detected face-shoulder features or generic local image features (for example, SURF (speed up robust features), FAST (features from accelerated segment test), BRIEF (binary robust independent elementary features) and ORB (oriented FAST and rotated BRIEF), etc.) can be used to establish a mapping relationship (including a rotation matrix R and a translation vector T) that maps the previous face-shoulder ROI to the current face-shoulder ROI (steps S 611  and S 612 ), then the face-shoulder coordinate system can be updated with this relationship, by applying the same mapping (R and T) to the previous face-shoulder coordinate system XYZ (N-1) , and the mapped face-shoulder coordinate system is used as the updated face-shoulder coordinate system XYZ (N)  (step S 613 ). In the special case where image feature mapping fails, mathematically it is equivalent to the rotation matrix R being an identity matrix and the translation vector being zero, thus the update of the face-shoulder coordinate system remains the same as the previous one. 
     After the updated face-shoulder coordinate system XYZ (N)  is established in step S 610  or S 613 , the FS ROI and FS feature ROIs are updated to the updated face-shoulder coordinate system (step S 614 ), and the updated data is stored (data D 604 ). 
     Once the hand ROI is transformed into the adaptive face-shoulder coordinate, gesture recognition can be done using traditional appearance-based or model-based methods. In addition, hand gesture recognition can also be done using a hybrid deep neural network (DNN) as shown in  FIG. 7 , where the modules in the dashed-line box form the hybrid DNN. The depth component of the RGB-D is processed separately apart from the early convolution layers of the RGB component, as convolution is not an ideal way to extract the low-level features of depth image. The face-shoulder coordinate transform can significantly reduce the complexities of the DNN (with much fewer layers and parameters), and alleviate the burden of sample collections and annotations necessary to train the otherwise larger DNN, and reduce the training and prediction time significantly, making it easier to deploy on low-cost hardware platforms. 
     It will be apparent to those skilled in the art that various modification and variations can be made in the hand tracking and gesture recognition method and related apparatus of the present invention without departing from the spirit or scope of the invention. Thus, it is intended that the present invention cover modifications and variations that come within the scope of the appended claims and their equivalents.