Patent Publication Number: US-2023145728-A1

Title: Method and system for detecting hand gesture, and computer readable storage medium

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
     This application claims the priority benefit of U.S. provisional application Ser. No. 63/275,958, filed on Nov. 5, 2021. The entirety of the above-mentioned patent application is hereby incorporated by reference herein and made a part of this specification. 
    
    
     BACKGROUND 
     1. Field of the Invention 
     The present disclosure generally relates to a tracking mechanism, in particular, to a method and a system for detecting a hand gesture, and a computer readable storage medium. 
     2. Description of Related Art 
     See  FIG.  1   , which shows a schematic diagram of hand gesture tracking. In the left part of  FIG.  1   , the head-mounted display (HMD)  100  may be disposed with a tracking element (e.g., a tracking camera on the HMD  100 ), and the tracking element may have a field of view (FOV)  199 . When a to-be-tracked object (e.g., the hand  102 ) is in the FOV  199 , the HMD  100  would be able to track the pose (e.g., the hand gesture of the hand  102 ) of the to-be-tracked object since the images of the to-be-tracked object are available. 
     However, when the to-be-tracked object leaves the FOV  199  as exemplarily shown on the right part of  FIG.  1   , the HMD  100  would be unable to track to pose of the to-be-tracked object since the images of the to-be-tracked object are unavailable. 
     SUMMARY OF THE INVENTION 
     Accordingly, the disclosure is directed to a method and a system for detecting a hand gesture, and a computer readable storage medium, which may be used to solve the above technical problems. 
     The embodiments of the disclosure provide a method for detecting a hand gesture, adapted to a host, including: determining whether information of a hand is enough for identifying a hand gesture of the hand; in response to determining that the information of the hand is enough for identifying the hand gesture, identifying the hand gesture, receiving first hand gesture information from at least one external gesture information provider, and correcting the hand gesture based on the first hand gesture information; in response to determining that the information of the hand is not enough for identifying the hand gesture, receiving second hand gesture information from the at least one external gesture information provider, obtaining a predicted hand gesture, and obtaining the hand gesture based on the predicted hand gesture and the second hand gesture information. 
     The embodiments of the disclosure provide a system for detecting a hand gesture. The system includes a host. The host is configured to perform: determining whether information of a hand is enough for identifying a hand gesture of the hand; in response to determining that the information of the hand is enough for identifying the hand gesture, identifying the hand gesture, receiving first hand gesture information from at least one external gesture information provider, and correcting the hand gesture based on the first hand gesture information; in response to determining that the information of the hand is not enough for identifying the hand gesture, receiving second hand gesture information from the at least one external gesture information provider, obtaining a predicted hand gesture, and obtaining the hand gesture based on the predicted hand gesture and the second hand gesture information. 
     The embodiments of the disclosure provide a non-transitory computer readable storage medium, the computer readable storage medium recording an executable computer program, the executable computer program being loaded by a host to perform steps of: determining whether information of a hand is enough for identifying a hand gesture of the hand; in response to determining that the information of the hand is enough for identifying the hand gesture, identifying the hand gesture, receiving first hand gesture information from at least one external gesture information provider, and correcting the hand gesture based on the first hand gesture information; in response to determining that the information of the hand is not enough for identifying the hand gesture, receiving second hand gesture information from the at least one external gesture information provider, obtaining a predicted hand gesture, and obtaining the hand gesture based on the predicted hand gesture and the second hand gesture information. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The accompanying drawings are included to provide a further understanding of the invention, and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the invention and, together with the description, serve to explain the principles of the disclosure. 
         FIG.  1    shows a schematic diagram of hand gesture tracking. 
         FIG.  2 A  is a functional diagram of a system for detecting hand gesture according to an embodiment of the disclosure. 
         FIG.  2 B  is a schematic diagram of  FIG.  2 A . 
         FIG.  3    shows a flow chart of the method for detecting the hand gesture according to an embodiment of the disclosure. 
     
    
    
     DESCRIPTION OF THE EMBODIMENTS 
     Reference will now be made in detail to the present preferred embodiments of the invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the description to refer to the same or like parts. 
     See  FIG.  2 A  and  FIG.  2 B , wherein  FIG.  2 A  is a functional diagram of a system for detecting hand gesture according to an embodiment of the disclosure, and  FIG.  2 B  is a schematic diagram of  FIG.  2 A . 
     In  FIG.  2 A , the system  200  includes a host  21  and external gesture information providers  221 - 22 N, wherein the host  21  is connected with each of the external gesture information providers  221 - 22 N via wireless communication protocols such as Bluetooth. 
     In various embodiments, the host  21  can be any device capable of performing tracking functions (e.g., inside-out tracking and/or outside-in tracking) on one or more to-be-tracked objects (e.g., the hand O 1 ) within the FOV  299 . In the embodiments of the disclosure, the FOV  299  can be an image-capturing range of one or more camera (e.g., tracking camera) on the host  21 . When the to-be-tracked objects (e.g., the hand O 1 ) is within the FOV  299 , the cameras on the host  21  may capture images of the to-be-tracked objects, and the host  21  may track the pose of each to-be-tracked object based on the captured images, but the disclosure is not limited thereto. 
     In the embodiments of the disclosure, the host  21  can be an HMD for providing reality services to the user thereof, wherein the reality services include, but not limited to, a virtual reality (VR) service, an augmented reality (AR) service, an extended reality (XR), and/or a mixed reality, etc. In these cases, the host  21  can show the corresponding visual contents for the user to see, such as VR/AR/XR/MR visual contents. 
     In  FIG.  2 A , the host  21  includes a storage circuit  212  and a processor  214 . The storage circuit  212  is one or a combination of a stationary or mobile random access memory (RAM), read-only memory (ROM), flash memory, hard disk, or any other similar device, and which records a plurality of modules that can be executed by the processor  214 . 
     The processor  214  may be coupled with the storage circuit  212 , and the processor  214  may be, for example, a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Array (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like. 
     In some embodiments, each of the external gesture information providers  221 - 22 N can be a (external) camera capable of performing tracking functions to one or more to-be-tracked objects, such as a tracking camera external to the host  21 . The external gesture information providers  221 - 22 N can be disposed at an environment where the host  21  locates and moves. 
     In the embodiments of the disclosure, the hand O 1  would be used as an example of the to-be-tracked object, and each of the external gesture information providers  221 - 22 N can track the pose (i.e., the hand gesture) of the hand O 1 . In one embodiment, the hand O 1  is predetermined to consist of a specific number (e.g., 21) of predetermined joints, and each of the external gesture information providers  221 - 22 N can be used to track the pose (e.g., 3D position) of each predetermined joint. 
     In one embodiment, the host  21  may send a first control signal CS 1  to each of the external gesture information providers  221 - 22 N to ask each of the external gesture information providers  221 - 22 N to provide the pose of some specific joint(s) of the predetermined joints on the hand O 1 . For example, the host  21  may ask each of the external gesture information providers  221 - 22 N to only provide the pose of fingertip joint(s) on the hand O 1  to the host  21 . In the embodiment, the pose information provided by each of the external gesture information providers  221 - 22 N in response to the first control signal CS 1  may be referred to as first hand gesture information. In one embodiment, the first hand gesture information may be partial hand gesture information, which represents that each of the external gesture information providers  221 - 22 N does not provide the tracked poses of all predetermined joints on the hand O 1 . 
     For example, the external gesture information providers  221  may provide first hand gesture information P 1  to the host  21 , wherein the first hand gesture information P 1  may exemplarily include the poses of the fingertip joints on the hand O 1  tracked by the external gesture information providers  221 . For another example, the external gesture information providers  222  may provide first hand gesture information P 2  to the host  21 , wherein the first hand gesture information P 2  may exemplarily include the poses of the fingertip joints on the hand O 1  tracked by the external gesture information providers  222 . Similarly, the external gesture information providers  222  may provide first hand gesture information PN to the host  21 , wherein the first hand gesture information PN may exemplarily include the poses of the fingertip joints on the hand O 1  tracked by the external gesture information providers  22 N. 
     In another embodiment, the host  21  may send a second control signal CS 2  to each of the external gesture information providers  221 - 22 N to ask each of the external gesture information providers  221 - 22 N to provide the tracked poses of all predetermined joints on the hand O 1 . In the embodiment, the pose information provided by each of the external gesture information providers  221 - 22 N in response to the second control signal CS 2  may be referred to as second hand gesture information. In one embodiment, the second hand gesture information may be complete hand gesture information, which represents that each of the external gesture information providers  221 - 22 N provides the tracked poses of all predetermined joints on the hand O 1 . 
     For example, the external gesture information providers  221  may provide second hand gesture information C 1  to the host  21 , wherein the first hand gesture information P 1  may exemplarily include the poses of all predetermined joints on the hand O 1  tracked by the external gesture information providers  221 . For another example, the external gesture information providers  222  may provide first hand gesture information P 2  to the host  21 , wherein the first hand gesture information P 2  may exemplarily include the poses of all predetermined on the hand O 1  tracked by the external gesture information providers  222 . Similarly, the external gesture information providers  222  may provide first hand gesture information PN to the host  21 , wherein the first hand gesture information PN may exemplarily include the poses of all predetermined on the hand O 1  tracked by the external gesture information providers  22 N. 
     In the embodiments of the disclosure, the processor  214  may access the modules stored in the storage circuit  212  to implement the method for detecting the hand gesture provided in the disclosure, which would be further discussed in the following. 
     See  FIG.  3   , which shows a flow chart of the method for detecting the hand gesture according to an embodiment of the disclosure. The method of this embodiment may be executed by the host  21  in  FIG.  2 A  and  FIG.  2 B , and the details of each step in  FIG.  3    will be described below with the components shown in  FIG.  2 A  and  FIG.  2 B . 
     Firstly, in step S 310 , the processor  214  determines whether information of the hand O 1  is enough for identifying the hand gesture of the hand O 1 . 
     In one embodiment, the processor  214  detects joints on the hand gesture and determines whether the amount of the detected joints on the hand gesture is higher than an amount threshold. In different embodiments, the amount threshold can be determined by the designer as an amount enough for the processor  214  to track the pose of the hand O 1 . 
     In one embodiment, in response to determining that the amount of the detected joints on the hand gesture is higher than an amount threshold, the processor  214  determines that the information of the hand O 1  is enough for identifying the hand gesture. In this case, even if some joints are untrackable due to, for example, being outside of the FOV  299  of the host  21  or being obstructed when the hand O 1  is in certain angles, the processor  214  may predict/guess the pose of the untrackable joints based on the trackable joints, but the disclosure is not limited thereto. 
     On the other hand, in response to determining that the amount of the detected joints on the hand gesture is not higher than the amount threshold, the processor  214  determines that the information of the hand O 1  is not enough for identifying the hand gesture. 
     In one embodiment, the processor  214  may perform an inside-out tracking based on the images of the hand O 1  captured by the tracking camera to detect the joints on the hand gesture, but the disclosure is not limited thereto. 
     In another embodiment, the processor  214  may determine whether the hand O 1  is in the FOV  299  of the camera (e.g., the tracking camera) of the host  21 . In one embodiment, in response to determining that the hand O 1  is in the FOV  299  of the camera of the host  21 , the processor  214  determines that the information of the hand O 1  is enough for identifying the hand gesture. On the other hand, in response to determining that the hand O 1  is not in the FOV  299  of the camera of the host  21 , the processor  214  determines that the information of the hand O 1  is not enough for identifying the hand gesture. 
     In a first embodiment where the information of the hand O 1  is determined to be enough for identifying the hand gesture of the hand O 1  (e.g., the hand O 1  is within the FOV  299 ), the processor  214  may subsequently perform steps S 321  to S 323  to obtain the hand gesture of the hand O 1 . 
     In step S 321 , the processor  214  identifies the hand gesture. In one embodiment, the processor  214  may track the hand gesture based on, for example, the conventional inside-out tracking mechanism, which would not be discussed in detail. 
     Therefore, after step S 321 , the hand gesture (e.g., the 6 degree-of-freedom (6DOF)) of the hand O 1  has been obtained. 
     In step S 322 , the processor  214  receives the first hand gesture information P 1  to PN from the external gesture information provider  221  to  22 N. As mentioned in the above, each of the first hand gesture information P 1  to PN may only include the poses of some specific joints on the hand O 1  (e.g., the fingertip joints on the hand O 1 ). For better understanding, the poses of the specific joints carried in the first hand gesture information P 1  to PN would be referred to as first poses. 
     In step S 323 , the processor  214  corrects the hand gesture based on the first hand gesture information P 1  to PN. In one embodiment, the processor  214  retrieves the pose of each specific joint in the hand gesture detected by the host in step S 321 . For example, if the considered specific joints are the fingertip joints on the hand O 1 , the processor  214  may retrieve the poses of the fingertip joints from the hand gesture detected in step S 321 , but the disclosure is not limited thereto. For better understanding, the poses of the specific joints retrieved from the hand gesture detected in step S 321  would be referred to as second poses. 
     Next, the processor  214  may correct the hand gesture obtained in step S 321  via combining the second pose of each specific joint with the corresponding first pose. 
     In brief, since the processor  214  can obtain the first poses of the specific joints based on the first hand gesture information P 1  to PN, the processor  214  may refine the second poses of the specific joints on the hand gesture obtained in step S 321  based on the first poses of the specific joints. 
     For example, assuming that the fingertip joint of the thumb on the hand O 1  is one of the considered specific joint, the processor  214  can retrieve the second pose of the fingertip joint of the thumb from the hand gesture obtained in step S 321 . In addition, the processor  214  can retrieve the first pose of the fingertip joint of the thumb from each of the first hand gesture information P 1  to PN. Next, the processor  214  may combine the second pose of the fingertip joint of the thumb with the first pose of the fingertip joint of the thumb carried in each of the first hand gesture information P 1  to PN as a corrected pose of the fingertip joint of the thumb. In one embodiment, the processor  214  may take a linear or nonlinear combination of the first poses and the second pose when combining them, but the disclosure is not limited thereto. 
     After performing similar operation on other specific joint, the corrected poses of other specific joints can be according obtained. As such, the hand gesture obtained in step S 321  can be more accurate since the poses of the specific joints on the hand O 1  have been corrected in step S 323 . 
     In particular, when the information of the hand O 1  is enough for identifying the hand gesture of the hand O 1  (e.g., the hand O 1  is in the FOV  299 ), the processor  214  may need to render the hand gesture with a higher frame rate for providing better visual experience to the user. In this case, the processor  214  may not have enough resources to receive the second hand gesture information C 1  to CN from the external gesture information provider  221  to  22 N and/or perform a more complete correction to the hand gesture obtained in step S 321 . 
     Therefore, when the information of the hand O 1  is enough for identifying the hand gesture of the hand O 1 , the processor  214  may ask the external gesture information provider  221  to  22 N to simply provide the first hand gesture information P 1  to PN and accordingly correct the poses of some specific joints (e.g., some important joints and/or some joints more possible to be occluded). In this case, the processor  214  may provide a better result of hand gesture tracking without consuming too much additional resources. 
     In a second embodiment where the information of the hand O 1  is determined to be not enough for identifying the hand gesture of the hand O 1  (e.g., the hand O 1  is outside of the FOV  299 ), the processor  214  may subsequently perform steps S 331  to S 333  to obtain the hand gesture of the hand O 1 . 
     In the second embodiment, assuming that the hand O 1  was within the FOV  299  and leaves the FOV  299  at an i-th timing point as shown in  FIG.  2 B , wherein i is an index. In this case, the processor  214  may determine the hand gesture based on steps S 321  to S 323  before the i-th timing point. 
     At the i-th timing point, since the hand O 1  has left the FOV  299 , the processor  214  may determine that the information of the hand O 1  is not enough for identifying the hand gesture of the hand O 1  and subsequently perform steps S 331  to S 333 . 
     In step S 331 , the processor  214  receive the second hand gesture information C 1  to CN from the external gesture information provider  221 - 22 N. 
     In one embodiment, in response to determining that the information of the hand O 1  is changed from being enough for identifying the hand gesture to be not enough for identifying the hand gesture, the processor  214  may send the second control signal CS 2  to each external gesture information provider  221  to  22 N for requesting the external gesture information provider  221  to  22 N to provide the corresponding second hand gesture information C 1  to CN, but the disclosure is not limited thereto. 
     As mentioned in the above, each of the second hand gesture information C 1  to CN includes the poses of all predetermined joints on the hand O 1 . For better understanding, the poses of all predetermined joints carried in the second hand gesture information C 1  to CN would be referred to as third poses. 
     In step S 332 , the processor  214  obtain a predicted hand gesture G 1 . In the second embodiment, in response to determining that the information of the hand O 1  has changed from being enough for identifying the hand gesture to be not enough for identifying the hand gesture at the i-th timing point, the processor  214  may retrieve a previous hand gesture G 2  identified at an (i−k)-th timing point, wherein k is a positive integer. Next, the processor  214  may predict the hand gesture at the i-th timing point based on the previous hand gesture as the predicted hand gesture G 1 . 
     In different embodiments, k can be any desired value of the designer. For better predicting the predicted hand gesture G 1 , k may be determined to be 1, such that the processor  214  would retrieve the hand gesture identified right before the hand O 1  leaves the FOV  299  as the previous hand gesture G 2 , but the disclosure is not limited thereto. 
     In the second embodiment, the mechanism for predicting the predicted hand gesture G 1  based on the previous hand gesture G 2  can be referred to related hand gesture prediction arts, which would not be further discussed herein. 
     In step S 333 , the processor  214  obtains the hand gesture based on the predicted hand gesture G 1  and the second hand gesture information C 1  to CN. 
     In one embodiment, the processor  214  obtains a reference hand gesture based on the second hand gesture information C 1  to CN of each of the external gesture information provider  221  to  22 N. In one embodiment, the processor  214  obtains a first hand gesture detected by the respective external gesture information provider  221  to  22 N based on the corresponding second hand gesture information C 1  to CN. For example, the processor  214  may retrieve the third pose of each predetermined joint from the second hand gesture information C 1  and accordingly construct the first hand gesture corresponding to the external gesture information provider  221  by connecting the predetermined joints. For another example, the processor  214  may retrieve the third pose of each predetermined joint from the second hand gesture information C 2  and accordingly construct the first hand gesture corresponding to the external gesture information provider  222  by connecting the predetermined joints. Similarly, the processor  214  may retrieve the third pose of each predetermined joint from the second hand gesture information CN and accordingly construct the first hand gesture corresponding to the external gesture information provider  22 N by connecting the predetermined joints, but the disclosure is not limited thereto. 
     Next, the processor  214  combines the first hand gesture detected by the respective external gesture information provider  221  to  22 N as the reference hand gesture. In one embodiment, the processor  214  may take a linear/nonlinear combination of the first hand gesture corresponding to each external gesture information provider  221  to  22 N as the reference hand gesture. In one embodiment, the processor  214  may take a first average of the first hand gesture detected by the respective external gesture information provider  221  to  22 N as the reference hand gesture, but the disclosure is not limited thereto. 
     Afterwards, the processor  214  may combine the predicted hand gesture G 1  with the reference hand gesture as the hand gesture of the hand O 1  at the i-th timing point. 
     In one embodiment, the processor  214  may take a linear/nonlinear combination of the predicted hand gesture G 1  and the reference hand gesture as the hand gesture of the hand O 1 . In one embodiment, the processor  214  may take a second average of the predicted hand gesture with the reference hand gesture as the hand gesture of the hand O 1  at the i-th timing point. 
     In particular, when the information of the hand O 1  is not enough for identifying the hand gesture of the hand O 1  (e.g., the hand O 1  is not within the FOV  299 ), the processor  214  may need to render the hand gesture with a lower frame rate for preparing more resources for processing the second hand gesture information C 1  to CN, such that the hand gesture of the hand O 1  can be refined accordingly. Therefore, the embodiments of the disclosure provide a solution for determining the hand gesture of the hand O 1  when the information of the hand O 1  is not enough for identifying the hand gesture of the hand O 1  (e.g., the hand O 1  is not within the FOV  299 ). 
     In the second embodiment, the processor  214  may further correct the first hand gesture detected by the respective external gesture information provider  221  to  22 N at least based on an correcting factor of the respective external gesture information provider  221  to  22 N before combining them as the reference hand gesture. 
     In the embodiment where each of the external gesture information provider  221  to  22 N is an external camera, the considered correcting factor of the respective external gesture information provider  221  to  22 N may include the extrinsic parameter(s) of the respective external gesture information provider  221  to  22 N. The detail of correcting the first hand gesture detected by the respective external gesture information provider  221  to  22 N based on the corresponding extrinsic parameter(s) may be referred to the related art, which would not be further discussed herein. 
     In one embodiment, in response to determining that the information of the hand O 1  is changed from being not enough for identifying the hand gesture to be enough for identifying the hand gesture, the processer  214  may send the first control signal CS 1  to each external gesture information provider  221  to  22 N for requesting the external gesture information provider  221  to  22 N to provide the corresponding first hand gesture information P 1  to PN, but the disclosure is not limited thereto. In this case, the processor  214  may proceed to perform steps S 321  to S 323 , and the details may be referred to the descriptions in the above. 
     In one embodiment, after the hand gesture of the hand O 1  has been determined in steps S 321  or S 333 , the processor  214  may accordingly adjust the visual content of the reality service based on the hand gesture of the hand O 1 . For example, the processor  214  may render the hand gesture of the hand O 1  in the visual content for the user to see and interact with the reality service, but the disclosure is not limited thereto. 
     In addition, although only one hand O 1  is used for explaining the concept of the disclosure, the embodiments of the disclosure can be used to simultaneously track the hand gestures of multiple hands. For example, for some hands within the FOV  299 , the processor  214  may perform steps S 321  to S 323  to obtain the hand gestures thereof. For some hands outside of the FOV  299 , the processor  214  may perform steps S 331  to S 333  to obtain the hand gestures thereof, but the disclosure is not limited thereto. 
     The disclosure further provides a computer readable storage medium for executing the method for detecting the hand gesture. The computer readable storage medium is composed of a plurality of program instructions (for example, a setting program instruction and a deployment program instruction) embodied therein. These program instructions can be loaded into the host  21  and executed by the same to execute the method for detecting the hand gesture and the functions of the host  21  described above. 
     In summary, the embodiments of the disclosure provide a mechanism for the host to determine hand gesture while cooperating with one or more external gesture information provider when the to-be-tracked hand is invisible to the host. In addition, when the to-be-tracked hand is visible to the host, the embodiments of the disclosure provides a mechanism to achieve higher accuracy for hand gesture tracking by considering the first hand gesture information provided by the external gesture information provider(s). 
     Moreover, the embodiments of the disclosure can be used to implement the hand gesture tracking for multiple hands. Accordingly, the accuracy of hand gesture tracking can be improved, such that the host can provide a better reality service for the user. 
     It will be apparent to those skilled in the art that various modifications and variations can be made to the structure of the present invention without departing from the scope or spirit of the disclosure. In view of the foregoing, it is intended that the present disclosure cover modifications and variations of this invention provided they fall within the scope of the following claims and their equivalents.