Dynamics Based 3D Skeletal Hand Tracking
Stan Melax, Leonid Keselman, Sterling Orsten
Intel Corporation
Take off theoven mitts fornatural userinteraction

Abstract

Hand Model

Voxel Subsampling

Results & Usages

This research explores a new approach to tracking
hands, or any articulated model, by using rigid bodysimulation. Based on a depth sensor’s samples, thesystem generates constraints that limit motion
orthogonal to the rigid body model’s surface. Theseconstraints, along with prior motion, collision/contact
constraints, and joint mechanics, are resolved with aprojected Gauss-Seidel solver.

We construct a model of the human hand using seventeen
convex polyhedral parts (bones). These bones overlap
slightly at joints, and are constrained to limit motion tophysically plausible states. We use a reference hand model
20 cm long, authored in 3D Studio Max®, and canparametrically scale different bones to track users withdifferent sized hands. As our model is specified entirely in
data, our generic tracking system is able to track otherarticulated objects without retraining.

We utilize volumetric subsampling to generate improved
depth samples from the camera. It allows us to both discard
sparse camera noise (by rejecting insufficiently densevoxels) and produce a point cloud with uniform spatialsampling across the surface of the tracked object.

Our approach is able to track the fully articulated pose of a user’s hand in a wide varietyof configurations and motions. We are even able to handle the difficult case of a largely
occluded hand model, where the depth information is ambiguous. For example, we areable to correctly determine which finger is bent from a side profile configuration.

CameraSamples

3D HandModel

Rigid Body Dynamics

Joint Limits

Our approach to hand tracking draws its technical core from theworld of ragdoll physics. We phrase pose tracking as a linear
complementarity problem, and use a projected Gauss-Seidel solver
to find a solution.
Each frame, we generate surface constraints based on our depth
camera’s samples, in addition to mechanical constraints from thehand model. We then run a small, fixed number of solver iterations.
This flexible approach enables us to easily incorporate additional
information and heuristics, as elaborated below.

AngularConstraints

To test a variety of competing
hypotheses, we run multiple
biased simulations, eachincorporating the corresponding
constraints. The outcomes ofthe simulations are evaluatedand the best fit one is returned.

Try hypotheses and heuristics

Contact
Constraints

Surface
Constraints

Dynamics
Solver

Grasping
Bias

ParallelFingerConstraints

Dynamics
Solver

PoseClassifier

Fingertip
Position
Constraints

Dynamics
SolverGross
Motion Bias

LockedJoints

Dynamics
Solver

Explorestate space©2013 Intel Corporation

PointCloudCommon
Information

Multiple Simulations
Our efficient simulationframework is extendable andflexible. Conjectures, heuristics
and externally-derived
information can all be
formulated as additional
constraints in the solver.

FingerCollisions

Flip Fingers
with Joint
Constraints

Dynamics
SolverPrior StateSurface Constraints
By using our tracked pose information, it is possible to use a powered ragdoll approachto create an interactive physics playground. In these examples, we use the Bullet
physics engine to enable interaction with blocks and swing sets without any
specialized code. The level design becomes the interaction design.

Magnet-like weak (impulse capped) surfaceconstraints that attract and limit motion to beparallel to surface

Samples mapped tonearest rigid body of hand
tracking model.

Temporal Coherence
Due to camera quality, occlusions, and pose ambiguity
from a single depth image, using temporal coherence isnecessary for any hand tracking solution.

Select Least
Error Pose

3D Error Measure
We evaluate the fitting error with a lower boundapproximation of the distance between the model and what
we can determine about the location of the user’s hand.
In particular, for each bone in the tracking model we sum:• The distance that the bone is known to be in front of (notoccluded by) the point cloud (if applicable)• The distance from the bone surface to the furthest depth
sample point mapped to that boneHand Pose:
6 DOF for eachof the17 bones

References
1.
2.
3.

CATTO, E. 2005. Iterative dynamics with temporal coherence.
In Game Developer Conference (GDC) 2005.
GANAPATHI, V., PLAGEMANN, C., KOLLER, D., AND THRUN, S. 2012.
Real-time human pose tracking from range data. In ECCV 2012OIKONOMIDIS, I., KYRIAZIS, N., AND ARGYROS, A. 2011.
Efficient model-based 3D tracking of hand articulations using Kinect. In BMVC 2011.

stan.k.melax@intel.com
leonid.m.keselman@intel.com
sterling.g.orsten@intel.com