Rigid-body Dynamics for
Articulated Mesh Tracking
Leonid Keselman (Intel)
Sterling Orsten (Intel)
Stan Melax (formerly @ Intel)

Intel Legal Disclaimer• No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.

• Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness
for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, orusage in trade.
• This document contains information on products, services and/or processes in development. All information provided here issubject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications androadmaps.

• The products and services described may contain defects or errors known as errata which may cause deviations from publishedspecifications. Current characterized errata are available on request.
• Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service
activation. Performance varies depending on system configuration. No computer system can be absolutely secure. Check with yoursystem manufacturer or retailer or learn more at [intel.com].
• Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors.
Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software,operations and functions. Any change to any of those factors may cause the results to vary. You should consult other informationand performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that productwhen combined with other products.
• Intel, the Intel logo, Intel® RealSense™ are trademarks of Intel Corporation in the U.S. and/or other countries.
• Other names and brands may be claimed as the property of others.

• © 2015 Intel Corporation

<Demo>• Run on my IT-issued laptop• i5-4300U @ 1.90 GHz• This is the machine used for any demos/timings in talk• Algorithm code is all single-core C++ CPU code
• By choiceTechnical Reference Materials• Physics Engine• https://github.com/melax/sandbox• Stan Melax’s sandbox for physics + graphics code, BSD license• Sequential Iterative Impulse solver, © 1998-2008• Our tracking built on top of improved/expanded version• Intel’s 2013 release of this work, free download• https://software.intel.com/en-us/articles/the-intel-skeletal-hand-tracking-library-experimental-release• Camera input layer is sample code, you could re-purpose on top of whatever data you’d like• Demo Videos & Concepts• https://www.youtube.com/user/smelax

Academic Reference Material• Melax, Keselman, Orsten.
• Dynamics based 3D skeletal hand tracking.
• i3D 2013. Poster• GI 2013, Full-length paperTalk Overview
1. Motivation
2. Dynamics-Based Trackinga)b)c)

Background
Method overview
Hand Model

3. Fast iterative Trackinga)b)c)d)

Our tracking architecture
Benefits of being 3D
Multi-hypothesis architecture
Value of working in a constraintbased solver

4. Cameras and Usagesa)
b)
c)
d)
e)

Basic Filter Architecture
Structured Light: PrimeSense &Kinect v1
Time-of-Flight: SoftKinetic
Projected Texture Stereo: Intel
R200
Structured Light: Intel F200

5. Annotation, Learning andClassification
6. Q&A

BackgroundBackground• Intel interested in depth cameras• Started in ~2011• Most of our work was during 2012

• January 2013: “Senz 3D”• QVGA, TOF depth camera• CES 2013 Launch• Present Day: Intel RealSense• F200 & R200

Background• Intel has 2 depth sensors
available as developer kits
• http://click.intel.com/realsense.html• F200• Structured Light• R200• Projected Texture Stereo

2011: Real-time “Hand Tracking” from 3D cameras

3D Hand Tracking Goal

• Full 6 DOF pose for all finger bodies••Along with sufficient information to provide collisions and interactions
On consumer hardware• Existing work on providing such 3D pose•••

Wang, Popovic. Real-time hand-tracking with a color glove. ‘09. + 6D Hands Pose Template
Hilliges et al. Digits: freehand 3D interactions anywhere using a wrist-worn gloveless sensor UIST ‘12.

Oikonomidis, Kyriazis, Argyros. Efficient model-based 3D tracking of hand articulations usingKinect. BMVC ‘11.

Motivation: Emergent Interaction

Dynamics-based Tracking

Rigid Body Dynamics• Ability to physicallysimulate articulated
models with collisions andjoints.
• Achieved by satisfying
linear and angularconstraints.

Rigid Body Dynamics• Ability to physicallysimulate articulated
models with collisions andjoints.
• Achieved by satisfying
linear and angularconstraints.

Kinematics vs Dynamics• From Adam Finkelstien’s COS426 lecture notes:

GJK 1988

Surface constraints• Like little magnets that attract the surface

Surface constraints created fromsynthetically generated depth data

2nd instance (left), generates depth data (middle), tracking/fitting (right)

Model to track authored in 3DS max• Created a generic hand and scale it as necessary:Custom models could be made.
20

Combining constraints + hand meshRigid Body Dynamics• Picture from E. Coumans’ talk
GDC14 on MLCP solvers• http://goo.gl/84N71q• Many methods• Stable, Approximate• Minimal tuning, temporally consistent• Very, very fast
• (30-1000Hz for modern games)• We use a sequential impulse solver.
• Fast, stable, converges to global solution• See reference slides for more details

Rigid Body Dynamics: Easy to reason with• The use of a single unified solver• Collisions• Angular limits• Data to model minimization• Approximation to real-world• Solves an MLCP: an arbitrary set ofangular and linear constraints• Easy to express new information
into the system
UnconstrainedWith bend angle constraint• Force fingertip to bend at expected
relative angle?
• Just add a conical constraint!

Fast Iterative Tracking

CameraSamples

3D HandModel

Joint Limits

FingerCollisions

Point
Cloud

AngularConstraints

Contact
Constraints

Surface
Constraints

CommonInformation

Try hypotheses and heuristics

Prior State

Dynamics
SolverGrasping
BiasParallel
FingerConstraints

Dynamics
SolverPose
ClassifierFingertip
Position
Constraints

Dynamics
SolverGross Motion
Bias

LockedJoints

Dynamics
Solver

Explorestate space

Flip Fingers
with Joint
Constraints

Dynamics
Solver

Select Least
Error Pose

Hand Pose:
6 DOF for each of
the17 bones

Simplified Architecture view
High Speed Iterative Tracking

CommonInformation

Pose re-initialization

Pose
Classifier

Fingertip
Position
Constraints

Prior State

Dynamics
SolverDynamics
Solver

Select Least
Error Pose

This type of track + reinitialize architecture seems to be catching on• CVPR 14: Qian et al, “Realtime and Robust Hand Tracking from Depth”• CHI 15: Sharp et al, "Accurate, Robust, and Flexible Real-time Hand Tracking."

Hand Pose:
6 DOF for each of
the17 bones

CameraSamples

3D HandModel

Joint Limits

FingerCollisions

Point
Cloud

AngularConstraints

Contact
Constraints

Surface
Constraints

CommonInformation

Try hypotheses and heuristics

Prior State

Dynamics
Solver

Grasping
Bias

ParallelFingerConstraints

Dynamics
Solver

Pose
Classifier

Fingertip
Position
Constraints

Dynamics
SolverGross Motion
Bias

LockedJoints

Dynamics
Solver

Explorestate space

Flip Fingers
with Joint
Constraints

Dynamics
Solver

Select Least
Error Pose

Hand Pose:
6 DOF for each of
the17 bonesMultiple Simulations• System can get stuck in local
minimum• Run multiple simulations andpick the best fit.
• Increase likelihood of regaining
lost tracking

28

Gross Motion Simulation
0.6

0.5

Error

0.4

0.3

Normal Simulation
0.2

Gross Motion
Simulation
0.1

01

6

1116

2126

31

36

Frame Number

41

46

51

56

61

Grasp Biased Simulation

0.4

0.35

0.3

Error

0.25

0.2

0.15

Grasping
Simulation

0.1

0.05

01

6

1116

2126

31

36

41

46

51

56

61

66

7176

Frame Number

81

8691

96

101 106 111 116 121 126 131 136

NormalSimulation

ErrorState Exploration Simulation

StateExplorationSimulation
BestSimulation
1611162126

313641
Frame Number

4651566166

7176

Error Metric• 𝐸𝑚𝑜𝑑𝑒𝑙 = α( 𝐸𝑑𝑎𝑡𝑎 ∞ + 𝐸𝑜𝑐𝑐𝑙𝑢𝑑𝑖𝑛𝑔 )• L-inf norm for point cloud to rigid body surface• We want the pose that best explains all points• Additional penalty for bone centroid existing in front of background• Single raycast per bone (17 total per frame)• Can strengthen penalty to also penalize missing data for ToF cameras• Non-standard simulations have a penalty multiplier• Other metrics available, but generative + reprojection based metrics
much more computationally expensive

Two Handed Interaction• Simple Method• K-Means Merging for Segmentation• K=2• Explicitly seed leftmost & rightmost points• Merge clusters if centroids are close• Run two simulations for 2 hands• Easy Extension• Solve in single simulation• Might require more carefulcorrespondenceHigh speed motion tracking

Results• Tracked hand modelcompared to input

Creative Gesture Camera

Asus xtion
35Voxel Subsampling

Single CPU core performance
600060• Would be too expensive to use every depthsample.

5000

50

4000

40Vertex
3000
Number30

2000

2010001000• High performance• 45-80 FPS on single core
• Flexible subsampling options• Approximate hashing scheme• Added benefit of removing outliers or“flying pixels”.
• Configurable density check• Improves fitting of tracking model.
• For noisy cameras, we also have acustom 16bit spatial median filter anda bilateral filter for photometricaligned data streams

00.5
Voxel Size (cm)1

Frames PerSecond

Solver Performance• We’re processing roughly ~300-400volumetric contacts• Total system clock on my i5-4300U is
about 6-7 milliseconds• Overall, roughly ~5,000 to 6,500
constraints solved per frame inmultiple hypothesis solver architecture.

• In single hypothesis version, full pass ofdense data fitting is ~800uS• This is roughly 50,000 constraintiterations (~10 iterations per step)• Includes everything after voxel
subsample through solver completion,
counting• ~ 1uS/constraint• ~ 0.15uS/constraint-iteration.

• Multiple solvers, multiple passes• Closest surface finding• Solving data constraints• Solving self-collision constraints• Evaluating all error metrics

Benefits of fast iterative tracking• Fewer points run faster can be a lot more robust.
• Also computationally more efficient: Relative to velocity, 2D image search
space is quadratic with resolution increases, but linear with time decreases.

• If you run fast enough, all changes are small• KinectFusion, Newcombe et al, 2011; Lucas & Kanade, 1981
• “Real-Time Camera Tracking: When is High Frame-Rate Best?”, Ankur Handa,
Richard A. Newcombe, Adrien Angeli, and Andrew J. Davison, ECCV 2012
• Develops Pareto wavefront for tracking cameras given compute budget. Lower resolution
with higher framerate performs better than higher resolution at lower frame-rateBenefits of fast iterative tracking• Feel free to throw out whatever data might be noisy• Make system robust by being selective• Can track in extremely sparse data environments• passive stereo or in depth camera saturation conditions (only edge data)Benefits of fast iterative tracking• Robust tracking withminimal data
• Tracking under camera
saturation conditions• Temporal Coherence• Top Right = Input Depth• White = no data
• Gray = depth data

• Bottom Left = EstimatedHand PoseCameras and Usages

Pose Locking• Trivial to force the solver to use reduced state spaces.
• Can be far more robust for tracking in constrained situations
1. Unibody: Internally used – solve system as a single rigid body
2. Duobody: experimental – solve the system as 2 solid parts: arm and hand
3. Arbitrary joint locking

Pose Locking: Best-fit pose given only pointerfinger and wrist as open rotational DOF

Combining tracking + simulation

Application – physical 3D interaction
TrackedHand
PoseForward Dynamics“Powered-Rag-Doll”

Pose from tracking
system drives
a virtual handin the application.

Drives

VirtualHand

Other
VirtualObjects

Dynamics
SolverApplication Physics Scene45

Jenga Case Study• Easy to knock things over, but Hard to grasp/stack blocks• 3D displays helpful for judging distance. (eg zspace)• Intent-assuming artificial systems can enhance interaction. (extramagnetic pull/push.)• SoftBody seems to work better than RigidBody (wet bar of soap vssponge)• But really too hard to play without force-feedback• Interesting area of further work: how to combine tracking
systems with force understanding and communication•Tu-Hoa Pham, Abderrahmane Kheddar, Ammar Qammaz, Antonis A. Argyros; TheIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 28102819
46

Applications to HMD applications
GPU vs CPU• Please build CPU-side tracking

GPU has a ton of high-throughput numerical computebut it has two problems
1. Run to completion on tasks
2. VR+AR applications have high compute needs andyou’re getting in the way• VR is only viable at ~90Hz [Abrash 2014], which is~ 11ms/frame• Users expect visual fidelity, applications will use
8-10ms/frame• Budget of 1-3ms/frame if you’re using the GPU (e.g. 330to 1000Hz tracking)• If you miss it, you’re toast: > 11ms often means 22ms,
which is 45Hz visual updates and your users all get sickCascaded Hand Pose Regression, CVPR 15, Sun et al. 300Hz on CPU!

Annotation, Learning &Classification

Getting ground truth• If you want validation data, or large data-sets for machine learning: use an
iterative geometric tracker• Failures occur but there’s three huge benefits:
1. Fast and easy
2. If you annotate a few corrections, you can propagate the corrections
3. Geometric trackers can handle multiview pose tracking• Using multiple cameras, simply register them and feed the algorithm at once
• We simply minimize cloud -> pose error• SIGGRAPH 14: “Real-Time Continuous Pose Recovery of Human Hands
Using Convolutional Networks” Jonathan Tompson, Murphy Stein, Yann
Lecun, and Ken Perlin.

Training Label Generation
Left=Annotation, Right=Trained ClassifierExplicit a-prior polyhedral modelPros• Not solving unnecessarily high dimensional problem• Easy to render, collide against
Cons• Not as high-fidelity as a generic skinned mesh
• Doesn’t handle variation across users
52

Hand Variation Across Users• We’ve found that most adult humans have very similar sized hands• We’ve been using a simple to use 2-parameter resizing model• Length• Width/Thickness• User-controlled• CVPR 2015: Sameh Khamis, Jonathan Taylor, Jamie Shotton, Cem Keskin,Shahram Izadi, Andrew Fitzgibbon; “Learning an Efficient Model of Hand
Shape Variation From Depth Images”• captures high level-of-detail variation, and also justifies using just 2 or 3 degree offreedom variation model of hand variation

Hand Variation Across Users
Internal work on naïve hand measurement work done in June 2012

Automatic Hand Measurement: Overview

55

1.
2.
3.

Detect blobs, in pre-determined orientation
Find points of interest on the contour
Feeding a 6 parameter model: finger lengths (5), palm WidthAutomatic Hand Measurement: Accuracy

Hand size measurement
Fingerσ (mm)PinkyRingMiddlePointer
Thumb
Palm

0.7
1.0
0.6
0.6
0.8
0.8

9080
70

mm

60

Mean |E|

Mean Median
2.3%
2.3%

Mode
2.8%

50

Computed

40

Measured30
20100Pinky

56

RingMiddle

Pointer

Thumb

Palm

Automatic Hand Measurement: Multiple Subjects

Measured results (10 users)
1
Pinky
RingMiddle
Pointer
Thumb
Palm

1
Mean ( |E| ) *Correlation

57

2

4

56

78

968
64
54
62
68
61
64
54
84
78
70
76
78
74
83
64
92
80
78
84
87
84
86
70
8276
70
76
7978
78
69
• 68 Results 70
across a varietyofusers
arewithin%of
rulermeasuredvalues
65
65
65
62
68
64
90
85
75
83
85
94
98
78

24.17%
0.971

331.19%0.993

44.67%
0.949

52.84%
0.987

65.65%
0.955

72.43%
0.998

8
4.11%
0.965

95.40%
0.951

10
68
81
82
75
66
87

10
4.25%
0.952

62
71
82
72
68
88

AVERAGE
2.12%
0.973

3.69%
0.969

Automatic Hand Measurement: Multiple Subjects
100

90

Computed Values

80

Pinky
Ring
Middle70

Pointer
Thumb
Palm

60

Wrist

50

40
40

50

60

70
Measured Values

58

80

90100

Automatic Hand Measurement: Applied tohand tracking
Tracked*, open hand
Mean0.2

BeforeAfter0.15

Error

0.1
BeforeAfter0.05

0-0.05

0.066
0.048

σ0.0590.021

Q&A: Additional Interactions

Physics Reference WorksClassical Works

Modern/Education Works• GJK, 1988
• Fast Contact Force Computation,Baraff, 1994
• Anitescu & Porta, 1996
• Impulse-based Dynamics
Simulation, Mirtich & Canny,
1994-1996• Open Dynamics Engine, Russell Smith,2004• Iterative Dynamics with Temporal
Coherence, Erin Catto, 2005
• Modeling & Solving Constraints, ErinCatto, GDC 09• Physics for game programmers, GDC2012• Understanding Constraints, Erin Catto,
GDC 2014• Exploring MLCP solvers, Erwin Coumans,
GDC 2014