Patent Description:
<NPL>, describe an algorithm for tracking an articulated pose, based upon looking up observations, such as body silhouettes, within a collection of known poses.

<CIT> relates to systems, methods and circuits for human to machine interfacing using one or more 3D hand skeleton models.

<CIT> describes a system of inverse reconstruction of a skeleton model of a hand, comprising: an imager adapted to capture at least one image of a hand; a memory storing a plurality of hand pose features records, each defined by a unique set of discrete pose values; a code store storing a code; at least one processor coupled to the imager, memory and program store for executing the stored code, the code comprising: code instructions to identify a group of discrete pose values from an analysis of the at least one image; code instructions to select a hand pose features record from the hand pose features records according to the group of discrete pose values; and code instructions to reconstruct a skeleton model of the hand in the hand pose from the hand pose features record based on a hand model which maps kinematic characteristics of a plurality of hand organs.

The invention is set forth in the independent claims. Specific embodiments are presented in the dependent claims.

A 3D representation of a hand (which is referred to herein as a skeleton model) is generated in real-time from a 2D image of the hand by identifying keypoints on the hand in the 2D image and determining a 3D pose and location of the hand using locations of the keypoints to access lookup tables that represent potential 3D poses of the hand as a function of locations of the keypoints. The keypoints include locations of tips of the fingers and the thumb, joints that connect phalanxes of the fingers and the thumb, palm knuckles that represent a point of attachment of each finger and the thumb to the palm of the hand, and a wrist location that indicates a point of attachment of the hand to the user's forearm. The lookup tables include finger pose lookup tables that represent 2D coordinates of each finger (or thumb) in a corresponding finger pose plane as a function of a location of the tip of the finger (or thumb) relative to the corresponding palm knuckle of the finger or thumb. Lengths of the phalanxes of the fingers and thumb are determined from a set of training images of the hand. The finger pose lookup tables are generated based on the lengths and anatomical constraints on ranges of motion of the joints that connect the phalanxes. The palm of the hand is represented as a palm triangle and a thumb triangle, which are defined by corresponding sets of vertices. The palm triangle has a vertex at the wrist location, which is opposite a triangle side that includes the palm knuckles of the fingers. The thumb triangle has a vertex at the wrist location, a vertex at the palm knuckle of the thumb, and a vertex at the palm knuckle of the index finger. Parameters that define the palm triangle and the thumb triangle are also determined from the set of training images.

In operation, a skeleton model of a hand is determined from a 2D image of the hand using 2D coordinates determined from the finger pose lookup tables and orientations of the palm triangle and the thumb triangle. The fingers and the thumbs are anatomically constrained to move within the corresponding pose planes, which have fixed orientations relative to the palm triangle and the thumb triangle, respectively. For example, the index finger in the 2D image is constrained to lie in a finger pose plane that connects the palm knuckle of the index finger to the fingertip of the index finger. The 2D coordinates of a finger in its finger pose plane are determined by accessing the corresponding finger pose lookup table using a relative location of the fingertip and the palm knuckle. A 3D pose of the fingers is then determined by rotating the 2D coordinates based on the orientation of the palm triangle. The 3D pose of the thumb is determined by rotating the 2D pose of the thumb (determined from the finger pose lookup table) based on the orientation of the thumb triangle. In some embodiments, the 3D pose of the hand is determined using noisy values of keypoints in the 2D image by determining a 3D skeleton model of the hand from the 2D image based on initial estimates of the keypoints, as discussed above. The 3D locations of the keypoints indicated by the skeleton model are modified based on projections of the 3D locations of the keypoints into an image plane along a line connecting the original 2D keypoints to a vanishing point associated with the 2D image. The modified 3D locations of the keypoints are then used to modify the skeleton model, as discussed above. The process is iterated to convergence.

Some embodiments of the techniques disclosed herein have been validated on different data sets and achieve greater than <NUM>% correctly identified keypoints (and in some cases as high as <NUM>%) when results are not aligned to ground truth data before making a comparison. Aligning the results to the ground truth data before the comparison improves the percentage of correct keypoints.

<FIG> is a two-dimensional (2D) image <NUM> of a hand <NUM> according to some embodiments. The hand <NUM> is represented by a skeleton model <NUM> that models the fingers, thumb, and palm of the hand <NUM> as a set of interconnected keypoints. In the illustrated embodiment, the keypoints include tips <NUM> (only one indicated by a reference numeral in the interest of clarity) of the fingers and the thumb, joints <NUM> (only one indicated by a reference numeral in the interest of clarity) that connect phalanxes <NUM> (only one indicated by a reference numeral in the interest of clarity) of the fingers and the thumb, palm knuckles <NUM> (only one indicated by a reference numeral in the interest of clarity) that represent a point of attachment of each finger and the thumb to the palm of the hand, and a wrist location <NUM> that indicates a point of attachment of the hand to the user's forearm.

<FIG> is a block diagram of a processing system <NUM> that is configured to acquire a 2D image of a hand <NUM> and generate a 3D pose of the hand based on the 2D image according to some embodiments. Generating a 3D pose of the hand <NUM> from the 2D image is referred to as "lifting" the 3D pose of the hand <NUM> from the 2D image. In the illustrated embodiment, the 3D pose of the hand <NUM> is represented by a skeleton model <NUM> such as the skeleton model <NUM> shown in <FIG>. In the interest of clarity, the following discussion uses the hand <NUM> as an example of a body part. However, some embodiments of the techniques discussed herein apply equally to lifting 3D poses of other body parts from corresponding 2D images. For example, the processing system <NUM> is able to lift 3D poses of feet, arms, legs, heads, other body parts, or combinations thereof from 2D images of the corresponding body parts.

The processing system <NUM> includes an image acquisition device such as a camera <NUM>. Examples of image acquisition devices that are used to implement the camera <NUM> include red-green-blue (RGB) cameras such as cameras implemented in mobile phones or tablet computers to perform virtual reality or augmented reality applications, RGB cameras with depth estimation using one or more depth sensors, grayscale cameras such as all-in-one virtual reality devices that use stereo fisheye cameras to provide <NUM>° of freedom, infrared cameras such as nighttime imagers or imagers on depth sensors, and the like. In some embodiments, the camera <NUM> is a lightweight RGB camera that is implemented in a small form factor and consumes a small amount of power. The lightweight RGB camera is therefore useful for implementation in augmented reality glasses. Some embodiments of the camera <NUM> are implemented as video cameras that capture sequences of images to represent movement within a scene.

The camera <NUM> acquires a 2D image of the hand <NUM> and stores information representing the 2D image in a memory <NUM>. A processor <NUM> is able to access the information representing the 2D image from the memory <NUM> and perform operations including learning, lifting, and denoising the 2D image. The learning phase includes generating one or more lookup tables (LUTs) <NUM> using training images of the hand <NUM>. For example, lengths of the phalanxes of the fingers and thumb are determined from the set of training images of the hand <NUM>. The LUTs <NUM> are generated based on the lengths and anatomical constraints on ranges of motion of the joints that connect the phalanxes and then stored in the memory <NUM>. Parameters such as the vertices that define a palm triangle and a thumb triangle are also determined from the set of training images and stored in the memory <NUM>.

In the lifting phase, the processor <NUM> generates the skeleton model <NUM> in real-time from the 2D image of the hand by identifying keypoints on the hand <NUM> in the 2D image. The processor determines a 3D pose and location of the hand <NUM> using locations of the keypoints to access 2D coordinates of the fingers and thumb from the LUTs <NUM>, which store the 2D coordinates of each finger and thumb as a function of a relative location of the fingertip and the palm knuckle. The processor <NUM> determines the 3D pose of the fingers by rotating the 2D coordinates based on the orientation of the palm triangle. The processor <NUM> determines the 3D pose of the thumb rotating the 2D pose of the thumb (determined from the finger pose lookup table) based on the orientation of the thumb triangle.

Some embodiments of the processor <NUM> are configured to perform denoising of noisy values of keypoints extracted from the 2D image of the hand <NUM>. The denoising phase is an iterative process. Initially, the processor <NUM> determines a 3D pose of the hand in the 2D image by determining a 3D skeleton model of the hand from the 2D image based on initial estimates of the noisy keypoints. The processor <NUM> then modifies the 3D locations of the keypoints indicated by the skeleton model based on projections of the 3D locations of the keypoints into an image plane along a line connecting the original noisy keypoints to a vanishing point associated with the 2D image. The vanishing point is determined based on parameters that characterize the camera <NUM>. The processor <NUM> updates the values of the noisy keypoints based on the modified 3D locations of the keypoints indicated by the skeleton model and the process is iterated until the noisy keypoints satisfy corresponding convergence criteria.

<FIG> illustrates a palm triangle <NUM> and a thumb triangle <NUM> that represent a portion of a skeleton model of a hand according to some embodiments. The palm triangle <NUM> and the thumb triangle <NUM> represent portions of the skeleton model <NUM> shown in <FIG> and the skeleton model <NUM> shown in <FIG>.

The palm triangle <NUM> is defined by vertices at the wrist location <NUM> and palm knuckles <NUM>, <NUM>, <NUM>, <NUM> (collectively referred to herein as "the palm knuckles <NUM>-<NUM>") of the hand. The plane that includes the palm triangle <NUM> is defined by unit vectors <NUM>, <NUM>, which are represented by the parameters uI, uL, respectively. A distance <NUM> from the wrist location <NUM> to the palm knuckle <NUM> of the index finger is represented by the parameter <NUM> and a distance <NUM> from the wrist location <NUM> to the palm knuckle <NUM> of the little finger (or pinky finger) is represented by the parameter L. Thus, the location of the palm knuckle <NUM> relative to the wrist location <NUM> is given by a vector having a direction uI and a magnitude of I. The location of the palm knuckle <NUM> relative to the wrist location <NUM> is given by a vector having a direction uL and a magnitude of L. The location of the palm knuckle <NUM> of the middle finger is defined as: <MAT> where λm is a parameter associated with the middle finger. The location of the palm knuckle <NUM> of the ring finger is defined as: <MAT> where λr is a parameter associated with the ring finger. Values of the parameters that define the palm triangle <NUM> are learned using 2D images of the hand while the hand is held in a set of training poses.

The thumb triangle <NUM> is defined by vertices at the wrist location <NUM>, the palm knuckle <NUM> of the index finger, and a palm knuckle <NUM> of the thumb. The plane that includes the thumb triangle <NUM> is defined by unit vectors <NUM>, <NUM>, which are represented by the parameters uI, uT, respectively. As discussed herein, the distance <NUM> from the wrist location <NUM> to the palm knuckle <NUM> of the index finger is represented by the parameter I. A distance <NUM> from the wrist location <NUM> to the palm knuckle <NUM> of the thumb is represented by the parameter T. Thus, the location of the palm knuckle <NUM> relative to the wrist location <NUM> is given by a vector having a direction uT and a magnitude of T. The thumb triangle <NUM> differs from the palm triangle <NUM> in that the thumb triangle <NUM> is compressible and can have zero area. Values of the parameters that define the thumb triangle <NUM> are learned using 2D images of the hand while the hand is held in the set of training poses.

<FIG> illustrates a finger pose <NUM> in a corresponding finger pose plane <NUM> according to some embodiments. The finger pose plane <NUM> is anatomically constrained to maintain an approximately fixed orientation with regard to the plane <NUM>. Movement of a finger is therefore constrained to lie approximately within a corresponding finger pose plane <NUM>. The finger pose plane <NUM> shown in <FIG> represents some embodiments of a plane of motion of an index finger, a middle finger, a ring finger, a little finger, or a thumb. If the finger pose plane <NUM> represents a plane of motion of an index finger, a middle finger, a ring finger, or a little finger, the plane <NUM> is a plane including a palm triangle of the hand, such as the palm triangle <NUM> shown in <FIG>. If the finger pose plane <NUM> represents a plane of motion of a thumb, the plane <NUM> is a plane including a thumb triangle of the hand, such as the thumb triangle <NUM> shown in <FIG>.

A finger of a hand is represented by a skeleton model <NUM> of the finger. The skeleton model <NUM> is characterized by a location of a fingertip <NUM> relative to a palm knuckle <NUM> of the finger. As discussed below, the relative location of the fingertip <NUM> relative to the palm knuckle <NUM> determines the 2D coordinates that define the position of the skeleton model <NUM> of the finger in the finger pose plane <NUM>.

An orientation of the plane <NUM> is determined by a vector <NUM>, which is defined as a vector normal to the plane <NUM>. The direction defined by the vector <NUM> is determined by comparing dimensions of the palm triangle (or thumb triangle) in the 2D image of the hand to the dimensions of a trained representation of the palm triangle (or thumb triangle) such as the dimensions discussed above with reference to <FIG>. An orientation of the finger pose plane <NUM> is determined by a vector <NUM>, which is defined as a vector that is normal to the vector <NUM> and in the finger pose plane <NUM>. A 3D pose of the finger in a 2D image is generated by rotating the skeleton model <NUM> of the finger based on the orientations determined by the vectors <NUM>, <NUM>.

<FIG> illustrates a skeleton model <NUM> of a finger in a finger pose plane according to some embodiments. The skeleton model <NUM> represent some embodiments of the skeleton model <NUM> shown in <FIG>. The skeleton model <NUM> also represents portions of some embodiments of the skeleton model <NUM> shown in <FIG> and the skeleton model <NUM> shown in <FIG>. The skeleton model <NUM> includes a palm knuckle <NUM>, a first joint knuckle <NUM>, a second joint knuckle <NUM>, and a fingertip <NUM>. The skeleton model <NUM> is characterized by a length of a phalanx <NUM> between the palm knuckle <NUM> and the first joint knuckle <NUM>, a length of a phalanx <NUM> between the first joint knuckle <NUM> and the second joint knuckle <NUM>, and a link of a phalanx <NUM> between the second joint knuckle <NUM> and the fingertip <NUM>.

Values of the lengths of the phalanxes <NUM>, <NUM>, <NUM> are learned from a set of training images of a hand that is held in a set of training poses. In some embodiments, the values of the lengths of the phalanxes <NUM>, <NUM>, <NUM> are learned by extracting keypoints corresponding to palm knuckles, joint knuckles, and fingertips from the set of training images. The keypoints are filtered for outliers using techniques such as median or median absolute deviation to find and reject the outlier keypoints. Techniques including quadratic programming are then used to fit values of the lengths to the displacements of the keypoints in the set of training images.

A location of the fingertip <NUM> relative to the palm knuckle <NUM> is determined by a set of angles at the palm knuckle <NUM>, the first joint knuckle <NUM>, and the second joint knuckle <NUM>. A first angle <NUM> represents an angle between the phalanx <NUM> and a plane of a palm triangle (or a thumb triangle), as indicated by the dashed line <NUM>. A second angle <NUM> represents an angle between the phalanx <NUM> and the phalanx <NUM>. A third angle <NUM> represents an angle between the phalanx <NUM> and the phalanx <NUM>. Ranges of the angles <NUM>, <NUM>, <NUM> are anatomically constrained to a limited set of values, which is substantially the same with minor variations for different hands. For example, the ranges of the angles <NUM>, <NUM>, <NUM> are constrained to lie between <NUM>° and <NUM>°.

<FIG> is a representation of an LUT <NUM> that is used to look up 2D coordinates of a finger in a finger pose plane based on a relative location of a palm knuckle and a tip of the finger according to some embodiments. The vertical axis of the LUT <NUM> represents a displacement of the tip of the finger relative to the palm knuckle in the vertical direction. The horizontal axis of the LUT represents a displacement of the tip of the finger relative to the palm knuckle in the horizontal direction. The closed curve <NUM> represents an outer boundary of the possible locations of the tip of the finger relative to the palm knuckle. The closed curve <NUM> is therefore determined based on lengths of the phalanxes of the finger and anatomical constraints on relative angles between the phalanxes due to limits on the range of motion of the corresponding joints. Locations within the closed curve <NUM> represent possible relative positions of the tip of the finger and the palm knuckle.

The LUT <NUM> for a particular hand is determined using a set of training images of the hand in a predetermined set of poses. In order to account for the differing lengths of the phalanxes in different hands, the set of training images is defined to include locations near the boundary defined by the closed curve <NUM>. A large portion of the locations within the closed curve <NUM> uniquely determine 2D coordinates of the finger. However, some embodiments of the closed curve <NUM> include a small set of degenerate cases that map a single point within the closed curve <NUM> to more than one set of 2D coordinates of the finger. The degeneracy between the different sets of 2D coordinates can be broken using other information, such as a previous position of the finger, depth information, shadow or lighting information, and the like.

In some embodiments, information in the LUT <NUM> is used to determine when two or more dissimilar poses result in the same or a similar set of projected 2D coordinates of the finger, e.g. one or more of the keypoints that are derived from the LUT <NUM> for one 3D pose are the same or similar to one or more of the keypoints that are derived from the LUT <NUM> for another 3D pose. A signal can then be generated to identify the dissimilar poses that have the same or similar projected 2D coordinates. The LUT <NUM> can also be used to convert 2D labels to 3D poses, e.g., of a hand, without gathering new data. In some embodiments, a confidence score is derived for the dissimilar poses that can result from the same or similar set of projected 2D coordinates. For example, a distance from a current pose to a most distant pose that has the same or similar 2D coordinates is used to generate a confidence score, such as a high confidence score if the distance is zero (or less than a threshold distance) and a low confidence score if the distance is greater than the threshold distance. In some embodiments, the dissimilar poses are disambiguated on the basis of the confidence scores for the keypoints or 2D coordinates that generate the dissimilar poses. For example, an image of a human being is used in some cases to check or confirm that the 3D lift of the 3D pose from the 2D labels is correct. The image can also be used to generate more accurate data by picking among different possible solutions.

<FIG> illustrates the 2D coordinates of fingers having a relative position of the fingertip and palm knuckle indicated by the circles <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> in <FIG> according to some embodiments. The circle <NUM> indicates a relative position between a tip of the finger and the palm knuckle that corresponds to an extended finger, as shown in the skeleton model <NUM>. The circle <NUM> indicates a relative position between the tip and the palm knuckle that corresponds to the tip of the finger being bent at <NUM>° with respect to a second joint of the finger, as shown in the skeleton model <NUM>. The circle <NUM> indicates a relative position between the tip and the palm knuckle that corresponds to the tip being curled up under a horizontally extended phalanx that connects the palm knuckle and the first joint, as indicated by the skeleton model <NUM>. The circle <NUM> indicates a relative position between the tip and the palm knuckle that corresponds to the tip being curled up adjacent a vertically extended phalanx that connects the palm knuckle and the first joint, as indicated by the skeleton model <NUM>. The circle <NUM> indicates a relative position between the tip and the palm knuckle that corresponds to the finger being extended vertically downward, as indicated by the skeleton model <NUM>.

<FIG> is a flow diagram of a method <NUM> of configuring an LUT that maps relative positions of the tip of a finger and a palm knuckle to 2D coordinates of the finger according to some embodiments. The method <NUM> is used to train some embodiments of the LUTs <NUM> shown in <FIG> and the LUT <NUM> shown in <FIG>. The method <NUM> is therefore performed by some embodiments of the processor <NUM> shown in <FIG>.

At block <NUM>, 2D images of a hand positioned in a training set of poses are captured. For example, the 2D images can be captured by the camera <NUM> shown in <FIG>. The 2D images are stored in a memory such as the memory <NUM> shown in <FIG>.

At block <NUM>, the processor identifies keypoints in the 2D images of the hand. As discussed herein, the keypoints include locations of tips of the fingers and the thumb, joints that connect phalanxes of the fingers and the thumb, palm knuckles that represent a point of attachment of each finger and the thumb to the palm of the hand, and a wrist location that indicates a point of attachment of the hand to the user's forearm. Techniques for identifying keypoints in 2D images are known in the art and in the interest of clarity are not discussed further herein.

At block <NUM>, the processor determines lengths of phalanxes in the fingers and thumb of the hand based on the keypoints, e.g., using quadratic programming as discussed herein.

At block <NUM>, the processor configures the LUT based on the length of the phalanxes and other anatomical constraints on the relative positions of the fingertip and the palm knuckle. The processor stores the LUT in a memory such as the memory <NUM> shown in <FIG>.

<FIG> is a flow diagram of a method <NUM> of lifting a 3D pose of a hand from a 2D image of the hand according to some embodiments. The method <NUM> is implemented in some embodiments of the processing system <NUM> shown in <FIG>.

In the illustrated embodiment, LUTs that map relative locations of tips of the fingers and thumb to corresponding palm knuckles has been generated for the hand, e.g., according to some embodiments of the method <NUM> shown in <FIG>. Thus, other parameters that represent a skeleton model of the hand have also been determined, such as lengths of the phalanxes, parameters that define a palm triangle for the hand, and parameters that define a thumb triangle for the hand.

At block <NUM>, a processor identifies keypoints in the 2D image of the hand. The processor then estimates a translation of the hand in 3D space based on the keypoints. Some embodiments of the processor estimate the translation by comparing the parameters that define the skeleton model of the hand to the relative values of corresponding parameters in the 2D image. For example, the processor can compare lengths of the phalanxes of the fingers and thumb in the skeleton model to lengths of the corresponding phalanxes in the 2D image to account for perspective projection and de-project the 2D image of the hand.

At block <NUM>, the processor learns orientations of the palm triangle and the thumb triangle. Some embodiments of the processor learned the orientations of the palm triangle and the thumb triangle by comparing the parameters that define the palm and thumb triangles to portions of the 2D image. The orientations of the palm triangle and the thumb triangle are characterized by corresponding vectors, which are defined to lie in a direction normal to the planes of the palm triangle and the thumb triangle.

At block <NUM>, the processor learns the orientations of the finger pose planes for the fingers and the thumb. The orientations of the finger pose planes are characterized by corresponding vectors, which are normal to the vectors that define the corresponding palm triangle or thumb triangle and which lie in the corresponding finger pose plane.

At block <NUM>, the processor determines the 2D finger coordinates of the fingers and thumb based on the LUTs and relative locations of the tips of the fingers and the corresponding palm knuckles.

At block <NUM>, the processor generates a 3D skeleton model that represents the 3D pose of the hand. To generate the 3D skeleton model, the processor rotates the 2D coordinates of the fingers and thumb based on the orientations of the palm triangle and the thumb triangle, respectively. The 3D skeleton model is determined by combining the palm triangle, the orientation of the palm triangle, the thumb triangle, the orientation of the thumb triangle, and the rotated 2D finger coordinates of the fingers and thumb.

<FIG> is an illustration <NUM> of iteratively de-denoising 3D keypoints that are listed from a 2D image of a hand according to some embodiments. The iterative process depicted in the illustration <NUM> is implemented in some embodiments of the processing system <NUM> shown in <FIG>. The illustration <NUM> shows an image plane <NUM> of a camera such as the camera <NUM> shown in <FIG>. Images captured by the camera are projected onto the image plane <NUM>. Characteristics of the camera also determine a vanishing point <NUM> that is an abstract point on the image plane <NUM> where 2D projections of parallel lines in 3D space appear to converge.

Initially, a keypoint <NUM> is extracted from the 2D image. In the illustrated embodiment, the 2D image is a noisy image and the initial estimate of the keypoint <NUM> is not necessarily at the correct position in the image of the hand. A 3D skeleton model of the hand is lifted from the 2D image on the basis of the noisy keypoint <NUM>, as well as other potentially noisy keypoints (not shown in <FIG>) that are extracted from the 2D image. For example, the 3D skeleton model of the hand is lifted according to some embodiments of the method <NUM> shown in <FIG> and the method <NUM> shown in <FIG>. The 3D skeleton model of the hand is used to determine a 3D keypoint <NUM> that corresponds to the same location in the hand as the keypoint <NUM>.

The 3D keypoint <NUM>, which is referred to herein as a skeleton-compliant keypoint, is not necessarily consistent with the perspective projection of the initial keypoint <NUM> because the skeleton-compliant keypoint <NUM> is not necessarily on a line <NUM> between the initial keypoint <NUM> and the vanishing point <NUM>. A modified 3D keypoint <NUM> is therefore determined by projecting the skeleton-compliant keypoint <NUM> onto the line <NUM>. The process is then iterated by updating the value of the initial keypoint <NUM> by setting it equal to the modified 3D keypoint <NUM>, which is referred to herein as a camera-compliant keypoint. The process is iterated until a convergence criterion for the keypoint (and any other noisy keypoints in the 2D image) is satisfied.

<FIG> is a flow diagram of a method <NUM> of denoising keypoints extracted from a 2D image of a hand according to some embodiments. The method <NUM> is performed in some embodiments of the processing system <NUM> shown in <FIG>.

At block <NUM>, a processor generates a 3D skeleton model of a hand based on noisy keypoints extracted from a 2D image. In some embodiments, the 3D skeleton model is generated according to embodiments of the method <NUM> shown in <FIG> and the method <NUM> shown in <FIG>.

At block <NUM>, the processor identifies a first set of 3D keypoints that are compliant with the 3D skeleton model of the hand. For example, the first set of 3D keypoints represents keypoints corresponding to tips of the fingers and thumb, joints of the fingers and thumb, palm knuckles of the fingers and thumb, and a wrist location defined by the 3D skeleton model of the hand. In some embodiments, the first set of 3D keypoints includes the skeleton-compliant keypoint <NUM> shown in <FIG>.

At block <NUM>, the processor identifies second 3D keypoints based on the first 3D keypoints and a vanishing point associated with the image. As discussed herein, the vanishing point is determined based on characteristics of a camera that acquired the 2D image. In some embodiments, the second set of 3D keypoints includes the camera-compliant keypoint <NUM> shown in <FIG>.

At block <NUM>, the processor modifies the noisy keypoints extracted from the 2D image based on the second set of 3D keypoints. For example, values of the noisy keypoints are updated to be equal to corresponding values of the second set of 3D keypoints.

At decision block <NUM>, the processor determines whether the values of the noisy keypoints have converged, e.g., based on convergence criteria for the noisy keypoints. If not, the method <NUM> flows to block <NUM> and an updated 3D skeleton model is generated on the basis of the modified values of the noisy keypoints. If the processor determines that the values have converged, the method flows to termination block <NUM> and ends.

In some embodiments, certain aspects of the techniques described above are implemented by one or more processors of a processing system executing software.

Such storage media can include, but is not limited to, optical media (e.g., compact disc (CD), digital versatile disc (DVD), Blu-Ray disc), magnetic media (e.g., floppy disc , magnetic tape, or magnetic hard drive), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or Flash memory), or microelectromechanical systems (MEMS)-based storage media.

Claim 1:
A method (<NUM>) comprising:
capturing, by a camera (<NUM>), a two-dimensional, 2D, image of a hand (<NUM>);
identifying, at a processor (<NUM>), keypoints (<NUM>, <NUM>, <NUM>, <NUM>) on the hand (<NUM>) in the 2D image captured by the camera (<NUM>), wherein the keypoints (<NUM>, <NUM>, <NUM>, <NUM>) comprise locations of tips of fingers and a thumb of the hand (<NUM>), joints that connect phalanxes of the fingers and the thumb, palm knuckles that represent a point of attachment of the fingers and the thumb to a palm of the hand (<NUM>), and a wrist location that indicates a point of attachment of the hand (<NUM>) to a forearm, the palm of the hand (<NUM>) being represented as a palm triangle and a thumb triangle; and
determining, at the processor (<NUM>), a three-dimensional (3D) pose of the hand (<NUM>) using locations of the keypoints (<NUM>, <NUM>, <NUM>, <NUM>) to access lookup tables, LUTs, (<NUM>) that represent potential 3D poses of the hand (<NUM>) as a function of the locations of the keypoints (<NUM>, <NUM>, <NUM>, <NUM>), wherein the LUTs (<NUM>) comprise finger pose LUTs (<NUM>) that represent 2D coordinates of each of the fingers and the thumb in corresponding finger pose planes as a function of the locations of the tips of the fingers or thumb relative to corresponding palm knuckles of the fingers or thumb, the finger pose planes having fixed orientations relative to the palm triangle and the thumb triangle, respectively, wherein determining the 3D pose of the hand comprises:
determining 2D coordinates of the fingers and the thumb from the finger pose LUTs (<NUM>) based on relative locations of corresponding fingertips and palm knuckles of the fingers and the thumb,
determining orientations of the palm triangle and the thumb triangle from the 2D image,
rotating the 2D coordinates of the fingers and the thumb based on the orientations of the palm triangle and the thumb triangle, respectively, and
determining a 3D skeleton model that represents the 3D pose of the hand by combining the palm triangle, the orientation of the palm triangle, the thumb triangle, the orientation of the thumb triangle, and the rotated 2D finger coordinates of the fingers and thumb.