Real-time hand modeling and tracking using convolution models

Technologies are provided herein for modeling and tracking physical objects, such as human hands, within a field of view of a depth sensor. A sphere-mesh model of the physical object can be created and used to track the physical object in real-time. The sphere-mesh model comprises an explicit skeletal mesh and an implicit convolution surface generated based on the skeletal mesh. The skeletal mesh parameterizes the convolution surface and distances between points in data frames received from the depth sensor and the sphere-mesh model can be efficiently determined using the skeletal mesh. The sphere-mesh model can be automatically calibrated by dynamically adjusting positions and associated radii of vertices in the skeletal mesh to fit the convolution surface to a particular physical object.

FIELD

The technologies described herein relate to the fields of signal processing, computer vision, and object tracking.

BACKGROUND

With the imminent advent of consumer-level virtual and augmented reality technology, the ability to interact with the digital world in more natural ways, such as by using hands instead of keyboards and/or mice, is becoming becomes more important. A number of techniques have been explored to address this problem, from expensive and unwieldy marker-based motion-capture to instrumented gloves as well as imaging systems. Some multi-camera imaging systems can recover the hand pose and hand-objects interactions with high accuracy, but such systems are only capable of operating a low frame rates, such as 10 Hz. Thus, conventional approaches do not provide satisfactory motion tracking with a single RGBD sensor, and are typically limited by low speed, low resolution, or a combination thereof.

SUMMARY

This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. The systems, methods, and apparatus disclosed herein generally permit rapid (real-time) estimation of static and dynamic geometry for a human body, part of a human body (such as a hand or other limb), or other object with accuracies suitable for tracking of complex motions.

In one embodiment, a method for modeling and tracking an object comprises generating a sphere-mesh model of a physical object, wherein the sphere-mesh model comprises an explicit skeletal mesh comprising multiple connected vertices and an implicit surface generated based on the skeletal mesh; receiving a data frame from a sensor camera, wherein the data frame comprises depth information for points within a field of view of the sensor camera, wherein the physical object is within the field of view of the sensor camera; and determining a distance between one of the points within the field of view and the implicit surface of the sphere-mesh model by determining a distance between the one of the points and the explicit skeletal mesh of the sphere-mesh model.

In another embodiment, a system comprises a depth sensor configured to generate a depth map of points within a field of view of the depth sensor; and a computing device connected to the depth sensor and configured to: generate a sphere-mesh model of a physical object, wherein the sphere-mesh model comprises a control skeleton comprising multiple connected vertices, and a convolution surface based on the control skeleton; receive the depth map from the depth sensor; and determine a distance between one of the points in the depth map and the convolution surface of the sphere-mesh model by determining a distance between the one of the points and one or more vertices of the control skeleton of the sphere-mesh model.

In another embodiment, one or more computer-readable media store instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the operations comprising: generating a sphere-mesh model of a human hand, wherein the sphere-mesh model comprises a skeletal control mesh comprising multiple connected vertices and a convolution surface generated based on the skeletal control mesh; receiving a data frame from a sensor camera, wherein the data frame comprises depth values for points on the human hand within the field of view of the sensor camera; and determining distances between the points in the data frame and the convolution surface of the sphere-mesh model, wherein determining the distances comprises using the depth values for the points in the data frame to determine distances between the points in the data frame and the skeletal control mesh of the sphere-mesh model.

In response to the determination of a location or one or more distances associated with a human hand, fingers, other body part, or any other object using the disclosed approaches, a processor generated display is updated or presented, or such location or distance information (typically available as digital data) is provided to one or more processor-based applications to serve as a user input. In this way, a tracked object can replicate the functions of a conventional user input device such as a mouse or other conventional pointing device. In other examples, positions of movable objects can be assessed to permit processor-based control of such objects.

As described herein, a variety of other features and advantages can be incorporated into the technologies as desired.

DETAILED DESCRIPTION

Previous systems for real-time tracking from data points comprising depth information rely on a combination of discriminative approaches and generative approaches. The per-frame re-initialization of discriminative methods can prevent error propagation by offering a continuous recovery from tracking failure. However, as these discriminative models are learned from data, they typically only estimate a coarse pose. Generative models can be used to refine the estimate by aligning a geometric template of a physical object (such as a human hand) to a measured point cloud and to regularize its motion through time.

FIG.1is a flowchart of an example method100for mapping data points received from a depth center to a sphere-mesh model of a physical object. At110, optionally a sphere-mesh model of a physical object (such as a human hand) is generated. The sphere-mesh model can comprise an explicit skeletal mesh and an implicit surface that is generated based on the skeletal mesh. The explicit skeletal mesh can comprising multiple connected vertices. In an embodiment where the physical object is a human hand, at least some of the multiple vertices of the explicit skeletal mesh can represent joints of the human hand. In at least some embodiments, the example method100can be performed using a predefined sphere-mesh model instead of generating the model at110.

At120, a data frame comprising depth information for points within a field of view of a sensor camera is received. If the physical object is within a field of view of the sensor camera, the points in the data frame can represent locations on and around the physical object. At130, a distance between one of the points within the field of view and the implicit surface of the sphere-mesh model is determined. The distance can be determined, at least in part, by determining a distance between the one of the points and the explicit skeletal mesh of the sphere-mesh model. As described in more detail below, using vertex information of the skeletal mesh, a closest correspondence point on the implicit surface to the one of the points can be determined. This process can be repeated for multiple points in the data frame.

The sensor camera can be any camera or sensor configured to generate depth information for points within a field of view of the camera or sensor. The depth information can be, for example, values indicating distances at which various points in the field of view are located from the camera or sensor. Example sensor cameras include red, green, blue, and depth (RGBD) sensors (e.g., INTEL REALSENSE cameras and MICROSOFT KINECT cameras). A sensor camera can comprise one or more lenses.

Generating the implicit surface of the sphere-mesh model from the explicit skeletal mesh can comprise creating spheres centered on the multiple vertices of the skeletal mesh with radii associated with the multiple vertices. The radii can be stored in association with the vertices in the skeletal mesh and/or stored separately. Determining the distance between the one of the points within the field of view and the implicit surface can comprise using the vertices and the associated radii to determine a point of the implicit surface closest to the one of the points in the data frame.

In at least some embodiments, multiple convolution primitives can be generated, wherein a convolution primitive is based on two or more of the spheres and a surface connecting the two or more spheres (such as a hull surface, e.g. a convex or concave hull). In such an embodiment, for at least one of the convolution primitives, a closest point correspondence on a surface of the convolution primitive to the one of the points can be determined that is within the field of view of the sensor camera.

In a further embodiment, at least one of the convolution primitives is a pill defined by two of the spheres and the surface of the at least one convolution primitive is a plane tangential to the two spheres. In a different or further embodiment, at least one of the convolution primitives is a wedge defined by three of the spheres and the surface of the at least one convolution primitive is a plane tangential to the three spheres.

In at least one embodiment, the example method100can further comprise generating a calibrated sphere-mesh model. In such an embodiment, generating the calibrated sphere-mesh model can comprise adjusting positions of the multiple vertices and a shape of the generated implicit surface of a default sphere-mesh model to match joint positions and a shape of the physical object (such as a human hand) in the field of view of the sensor camera. In a different or further embodiment, one or more additional or different sensor cameras can be used for the calibration.

FIG.2is a diagram depicting an example system200for modeling and tracking a physical object using a depth sensor and one or more sphere-mesh models. The example system200can be used to perform any of the example methods described herein.

The example system200comprises a depth sensor220with a field of view222. A physical object210is depicted within the field of view222of the depth sensor220. The depth sensor can be, for example, a sensor camera as described herein. The depth sensor220is configured to generate a depth map240of points within the field of view222of the depth sensor220(including one or more points on and/or near the physical object210).

A computing device230is connected to the depth sensor220and configured to receive the depth map240from the depth sensor. The computing device230comprises an object modeler and tracker232. The object modeler and tracker232can comprise one or more hardware components and/or software executed by one or more processors of the computing device230.

The object modeler and tracker232can be configured to generate a sphere-mesh model250of the physical object210. The sphere-mesh model250can comprise a control skeleton comprising multiple connected vertices and a convolution surface generated based on the control skeleton. In at least some embodiments, the vertices of the control skeleton comprise predefined degrees of freedom. The object modeler and tracker232can be further configured to determine a distance between one of the points in the depth map240and a convolution surface of the sphere-mesh model250by determining a distance between the one of the points and one or more of the vertices of the control skeleton of the sphere-mesh model250.

In addition to controlling user interfaces of traditional applications, virtual reality (VR) and/or augmented reality (AR) applications are also possible. For example, the depth sensor220and the computing device230can be incorporated into a head-mounted computing device (such as a VR and/or AR headset) worn by a user and used to track interactions of one or more physical objects (such as one or more of the user's hands) with virtual or holographic elements.

In many cases, the quality of a model template used directly affects the quality of pose refinement.FIG.3is a diagram depicting three side-by-side comparisons of example models of hands generated using techniques described herein (302,312, and322) compared with models generated using previous techniques (304,314, and324). The higher quality of the pose refinement of the models302,312, and322is readily apparent.

At least some of the tracking models described herein improve a balance between accuracy and performance, as compared to models generated using previous techniques (e.g.,304,314, and324). At least some of the models described herein are geometric models that more accurately captures a user's hand geometry, while retaining the ability to answer registration queries in closed form with very high efficiency.

FIG.4is a diagram depicting tracking using two models of a hand generated using previous techniques (402and404) and tracking using an example model of the hand generated using technologies described herein (406). The402tracking illustrates tracking performed with a model without proper coarse scale calibration. The404tracking illustrates tracking with a model that has been roughly manually calibrated. Although the manual calibration can help increase fitting fidelity, tuning becomes increasingly difficult as the number of degrees of freedom increase. The406tracking illustrates tracking using a model that has been automatically calibrated using calibration techniques described herein.

Implicit and Explicit Templates.

In modern digital production, the de-facto standard is to represent objects by a surface mesh of their boundary (e.g. triangle or quad meshes). Fast rendering and easy direct manipulation make explicit surface representation attractive for many applications. However, unlike implicit models, explicit representations cannot efficiently answer queries such as the distance from a point to the object's boundary, or whether a point lies inside/outside the model. In tracking applications, these queries play a fundamental role, as the optimization attempts to find configurations where the average distance from model to data is minimized. Similarly, a tracker should prevent the model from assuming implausible configurations, for example by preventing self-intersections as measured by inside/outside predicates. For all these reasons, implicit models appear highly suitable for registration applications; indeed, compelling results in joint rigid registration and reconstruction as well as its recent non-rigid variant leverage implicit models. However, in many cases, such techniques assume the frame-rate is high compared to motion velocity, a condition that is in general not satisfied in our setting. To address this challenge, a hybrid model for tracking can be employed that combines the advantages of explicit and implicit representations.

Hybrid Tracking Model.

A hybrid tracking model, can be a variant of a convolution surface, wherein one or more convolution surfaces are generated from an explicit skeletal mesh.FIG.5is a diagram depicting an example skeletal meshand example generated convolution surfaces502-508. A convolution surface can be defined as the zero iso-level of the scalar function:

whereis a skeletal control mesh (a segment or a triangle in the simple examples ofFIG.5), andis the implicit function of a sphere parameterized by its center c and radius r:
c,r(c)(x)=∥x−c∥2−r(c)2(Equation 2).

The sphere centers c span the skeleton, while the radii are a function of the position c within an element, linearly interpolated from values r*=r(c*)specified on the skeletal mesh vertices c*. This is indeed a hybrid model, as Eq. 1 defines an implicit surface={x∈n|ϕ(x)=0}, while the underlying skeleton S is an explicit representation (i.e. a simplicial complex).

This construct can be generalized to devise a model suitable to represent a human hand.FIG.6is a diagram depicting an example explicit skeletal mesh602and parameterized convolution surfaces604for modelling a human hand. The skeletal mesh602parameterizes the convolution surface604, providing radii value properties in vertices of the skeletal mesh. In example skeletal mesh602, articulated components are shown in dark green while flexible components are shown in purple. Calibration can be used to adjust the vertex positions and radii to alter the convolution surface. Distances tocan conveniently be computed by querying distances to the piecewise linear elements of; seeFIG.5.

Tracking and Calibration with Sphere-Mesh Models.

This hybrid tracking model has at least two significant advantages. (1) Distance queries to the surfacecan be executed by measuring the distance to the skeletal structure. The number of elements inis often significantly smaller (30in example model602) than the number of polygons in a typical triangular mesh surface representation. Therefore, distance queries can be performed efficiently using a brute force approach, which leads to a simple algorithm that is trivially parallelizable and executes at a fixed frame-rate. (2) The parameterization of the hand model is compact, as a family of models can be generated by simply adjusting positions and radii of the control skeleton vertices c, ∈. This allows adapting the model to the hand geometry of a specific user.

Such convolutional models can provide superior hand tracking performance for single-view depth sensors. The tracking model can be optimized to adapt to different human hands with a high level of accuracy. The improved geometric fidelity compared to previous representations leads to quantifiable reductions in registration error and allows accurate tracking even for intricate hand poses and complex motion sequences that previous methods have difficulties with. At the same time, due to a very compact model representation and closed-form correspondences queries, at least some of the generative models described herein retain high computational performance, in some cases leading to sustained tracking at rates of 60 Hz or more.

Hybrid Discriminative and Generative Template Creation.

Pose estimation techniques can be grouped into discriminative and generative techniques, also known respectively as appearance-based and model-based approaches. Generative approaches fit a template through a temporal sequence of images. Given an accurate template of an object being tracked, these methods can resolve highly accurate motion. As the optimization is initialized from the previous frame, tracking loss can occur, although simple geometric reinitialization heuristics can be employed to overcome this issue.

Discriminative methods estimate a pose by extracting features from each image independently by learning from a large dataset of annotated exemplars. While discriminative methods can avoid drift, they lack the accuracy of generative methods, and joint estimates often violate kinematic constraints, like consistent finger lengths and joint limits.

Example Object Tracking

Given a calibrated hand model, a real-time tracking algorithm can be used that optimizes degrees of freedom θ (e.g., degrees of freedom for global rotation, translation, and articulation) so that a hand model matches sensor input data. (The generation of a calibrated modelfor a user is detailed below). The example tracking optimization can be written in Gauss-Newton/Levenberg-Marquardt form

where fitting energies are combined with a number of priors to regularize the solution and ensure the estimation of plausible poses. Example energy termsTrackin the optimization include:

d2meach data point is explained by the modelm2dthe model lies in the sensor visual-hullposehand poses sample a low-dimensional manifoldlimitsjoint limits must be respectedcollisionfingers cannot interpenetratetemporalthe hand is moving smoothly in time

Additional details regarding computational elements that can be adapted to support modeling and tracking using sphere-mesh models described herein are discussed in Tagliasacchi, et al.,Robust Articulated-ICP for Real-Time Hand Tracking(2015), a copy of which is attached.

The similarity of two geometric models can be measured by the symmetric Hausdorff distance dX↔Y

The terms Ed2mand Em2dcan be interpreted as approximations to the asymmetric Hausdorff distances dX→Yand dY→X, where the difficult to differentiate max operators are replaced by arithmetic means, and a robust l1distance is used for d(x, y).

The first asymmetric distance minimizes the average closest point projection of each point p in the current data frame:

Adapting this energy, as well as its derivatives, to the sphere-mesh models requires the specification of a projection operatorthat is described in more detail below.

The second asymmetric distance considers how a monocular acquisition system may not have a complete view of the model. While a 3D location is unknown, the model can be penalized from lying outside the sensor's image-space visual hull:

In the equation above, the set of pixels x∈(Θ) can produced by a rasterization process (described in more detail below) that renders the model with the same implicit and explicit parameters of the sensor camera.

An example correspondence search can leverage the structure of Eq. 1, by decomposing the surface into several elementary convolution elements (or convolution primitives) εe, where e indexes elements of the model template.

FIG.7is a diagram depicting closest point correspondences on example convolution primitives. The computation of closest point correspondences on an example pill convolution primitive702and an example wedge convolution primitive704can be performed by tracing a ray along a normal of the circles' (702) or spheres' (704) tangent line (702) or tangent surface or plane (704).

Convolution elements can be classified into pill and wedge implicit primitives, with an associated implicit functions ϕe. Given a point p in space, the implicit function of the whole surface can be written by evaluating the expression:

Given a query point p, the closest-points qe=┌εe(p) to each element can be independently. Within this set, the closest-point projection to the full model q=(p) is the one with the smallest associated implicit function value ϕe(p).

In any of the examples described herein, a correspondence of a pill convolution element can be represented by q=┌pill(p). A pill can be defined by two spheres1(c1, r1) and2(c2, r2). By construction, the closest point correspondence lies on a plane passing through the triplet {c1, c2, p}, thus allowing a solution in 2D; see702. An intersection point s of the ray r(t)=p+tn with the segmentc1c2is computed and its location is parametrizes in barycentric coordinates as s=αc1+(1−α)c2. If α∈[0,1], a closest point correspondence is given by q=(p), that is, the intersection ofc1c2and r(t). If α<0 or α>1, then the closest point will be q=(p) or q=(p), respectively.

In any of the examples described herein, a correspondence of a wedge convolution element can be represented q=┌wedge(p). A wedge is defined by three spheresi={ci, ri}.704illustrates how a wedge element can be decomposed in three parts: spherical, conical, and planar elements, associated with vertices, edges, and faces of the triangular convolution skeleton. For the planar element(t0, n) with normal n and tangent t0to0, the skewed projection s can be computed by finding the intersection of the r(t)=p+tn with the triangleformed by c1, c2, c3. According to the position of s, there are two possible solutions: If s lies inside the triangle, then the footpoint is q=(p). Otherwise, the barycentric coordinates of s inare used to identify a closest pill convolution element and compute q=Πpill(p)

In monocular (i.e., single sensor) acquisition, an oracle registration algorithm aligns the portion of the model that is visible from the sensor viewpoint to the available data. Hence, when computing closest-point correspondences, only a portion of the model currently visible by the camera should be considered. Given a camera direction v, it can be determined whether a retrieved footpoint q is back-facing (i.e., not facing the camera) by testing the sign of v·(q), where the second term is the object's normal at q.

FIG.8is a diagram depicting closest point correspondences on example convolution surfaces802and804, disregarding correspondences whose normals point away from a direction of a camera. Example back-facing footpoints812,814,822,824, and826are depicted. Whenever a footpoint is determined to be back-facing, additional candidates for closest point can be checked. Such additional candidates can include: (1) the closest-point on the silhouette of the model (e.g., p2,3,6,7), and (2) the front facing planar portions of convolution elements (e.g., p5). These additional correspondences for the query point are computed, and the one closest to p is selected as a front-facing footpoint q (e.g., (q2,3,5,6,7).

The object-space silhouette ∂is a (3D) curve separating front-facing from back-facing portions of a shape. To simplify the silhouette computation, the perspective camera of the sensor can be approximated with an orthographic one.

FIG.9is a diagram depicting three stages in a generation of an example object-space silhouette of an example sphere-mesh model of a human hand using such an orthographic approximation. All convolution elements can then be offset on the 2D camera plane, and a cross-section with this plane can be performed. Spheres are replaced with circles and planes/cylinders with segments; see902. An arrangement can then be computed, splitting curves whenever intersection or tangency occurs to generate a graph; see904. This graph can be traversed, starting from a point that is guaranteed to be on the outline (e.g. a point on the bounding box). The traversal selects the next element as the one whose tangent forms the smallest counter-clockwise angle thus identifying the silhouette. Once the 2D silhouette has been computed, it can be re-projected to 3D; see906.

Note the process described above would compute the image-space silhouette of the sphere-mesh model. In the case of a model of a human hand, the process can be applied to palm and fingers separately, and merged in a subsequent second phase. Such a merge process can check whether vertices v∈∂are contained within the model, and discards those vertices where(v)<0.

Rendering the sphere-mesh model in real time can be employed for visual verification of tracking performance. The real-time tracking techniques described above can be used to perform a 2D registration in the image plane that requires the computation of an (image-space) silhouette. However, alternatives for rendering a sphere-mesh model also exist. One alternative is to explicitly extract the surface of individual convolution elements by computing the convex hull of pairs or triplets of spheres. While this process may be suitable in applications where the model is fixed, it may not be suitable appropriate in a scenario where the model is calibrated to a specific physical object (such as a particular user's hand).

Another alternative is to ray-trace the sphere-mesh model. Such a ray-trace can be performed, in at least some cases, on a Graphical Processing Unit (GPU) of a computing device. For example, a unit fullscreen quad can be rendered and, in a fragment shader, camera implicits can be used to compute a camera ray r(x) associated with each pixel x. Each ray is intersected with each convolution element of the model in closed form, and only the closest intersection point is retained. Intersection tests can be performed with the planar, conical, and/or spherical primitives that compose convolution elements.

Example Model Calibration

Automatic calibration can be performed to adapt a default model (a.k.a., a template model) to a specific physical object (such as a particular user's hand) from a set of N 3D measurements {1. . .N} of the physical object in different poses. Multiple measurements are necessary in many cases, as it is not possible to understand the kinematic behavior of many complex objects by analyzing static geometry. In such cases, a redundancy of information improves fitting precision. In monocular acquisition this redundancy is of greater importance, as single-view data is normally incomplete to a large degree. The datasets {1. . .N} can be acquired via multi-view stereo sensors and/or a single sensor. The calibration techniques described herein can be employed for both acquisition modalities, as well as others.

A rest-pose geometry of a sphere-mesh model can be fully specified by two matrices specifying the set of sphere positions C and radii r. The geometry is then posed through the application of kinematic chain transformations.FIG.10depicts a posed kinematic framesT*1002, a kinematic chain and degrees of freedom for posing an example sphere-mesh model of a human hand1004, and optimal (1006) and non-optimal (1008) kinematic transformation frames. Given a pointpon the modelat rest pose, its 3D position after posing can be computed by evaluating the expression:
p=[Πk∈K(i)TkTkTk−1]p(Equation 7),

where T are the pose transformations parameterized by θ and Π left multiplies matrices by recursively traversing the kinematic chain K(i) of element i towards the root. Each node k of the kinematic chain is associated with an orthogonal frameTkaccording to which local transformations are specified. In some embodiments, the framesT*1002can be manually set by a 3D modeling artist and kept fixed across multiple instances. However, in some cases incorrectly specified kinematic frames can be highly detrimental to tracking quality. For example, note the differences in tracking quality between the optimal kinematic transformation frame1006and the non-optimal kinematic transformation frame1008. Thus in at least some cases, tracking quality can be improved by directly optimizing the kinematic structure from acquired data.

A set of energiescalibcan be employed to account for different requirements. Example requirements can include a good fit of the model to the data and a non-degenerate convolution template that has been piecewise-rigidly posed. The following example calibration energiescalibencode these example requirements:

d2m data to model Hausdorff distance (approx.)

m2d model to data Hausdorff distance (approx.)

rigid convolution elements are posed rigidly

valid convolution elements should not degenerate

To make this calibration more approachable numerically, Eq. 8 can be rewritten as an alternating optimization problem:

The first step adjusts radii and sphere centers of the sphere-mesh model, by allowing the sphere-mesh model to fit to the data without any kinematic constraint beyond rigidity, and returning as a side product a set of per-frame posed centers {Cn}. The second step takes the set {Cn} and projects it onto the manifold of kinematically plausible template deformations. This results in the optimization of the rotational components of rest-pose transformationsT, as their translational components are simply derived from C.

The example energies described above are non-linear and non-convex, but can be optimized offline. The pre-calibrated model can then be used for real-time tracking. In a particular embodiment the lsqnonlin Matlab routine can be used, which requires the gradients of the energies as well as an initialization point. The initialization of C is performed automatically by anisotropically scaling the vertices of a generic template to roughly fit the rest pose. The initial transformation frame rotations δ are retrieved from the default model, while {θn} are obtained by either aligning the scaled template to depth images, or by executing inverse kinematics on a few manually selected keypoints (multi-view stereo).

The example fitting energies are analogous to the ones described above for use in tracking. They approximate the symmetric Hausdorff distance, but they are evaluated on a collection rather than a single data frame:

Note that the projection operatorchanges according to the type of input data. If a multi-view acquisition system is used to acquire a complete point cloud, then the projection operator fetches the closest point to p in the point cloud of frame. Ifis acquired through monocular acquisition, thencomputes the 2D projection to the image-space silhouette of the model.

It can be important in some embodiments to estimate a template that jointly fits the set of data frames {n}. In such embodiments, each posed model can be a piecewise-rigid articulation of a rest pose. This can be achieved by constraining each segment e in the posed centers Cnto have the same length as the corresponding segment e in its rest pose configuration:

Note that only a subset of the edges of a control skeleton are required to satisfy the rigidity condition.

The calibration optimization should avoid producing degenerate configurations. For example, a pill degenerates into a sphere when one of its spheres is fully contained within the volume of the other. Analogously, a wedge can degenerate into a pill, or even a sphere in a similar manner. To avoid such degeneration, validity can be monitored by an indicator function χ(i) that evaluates to one, ifiis degenerate, and zero otherwise. This leads to the following penalty function:

χ(ci) can be used, which verifies whether ciis insidei, the model obtained by removing a vertex, as well as all its adjacent edges, from.

FIG.11is a diagram depicting four example data frames1102-1106undergoing three iterations of calibration using calibration techniques described herein. InFIG.11, the same model is rigidly articulated to fit to a different pose in each of the data frames1102-1106. The use of multiple frames in different poses can be useful for automatically adjusting the centers locations {Cn} to create an articulated model that consensually fits the whole dataset.

The importance in some scenarios of adjusting kinematic chain transformations is shown in1006and1008ofFIG.10, as well as302and304inFIG.3. With incorrect transformations, joint limits and the articulation restrictions of the kinematic chain can prevent the model from being posed correctly.

FIG.12depicts a generalized example of a suitable computing system1200in which the described innovations may be implemented. The computing system1200is not intended to suggest any limitation as to scope of use or functionality, as the innovations may be implemented in diverse general-purpose or special-purpose computing systems.

With reference toFIG.12, the computing system1200includes one or more processing units1210,1215and memory1220,1225. InFIG.12, this basic configuration1230is included within a dashed line. The processing units1210,1215execute computer-executable instructions. A processing unit can be a general-purpose central processing unit (CPU), processor in an application-specific integrated circuit (ASIC) or any other type of processor. In a multi-processing system, multiple processing units execute computer-executable instructions to increase processing power. For example,FIG.12shows a central processing unit1210as well as a graphics processing unit or co-processing unit1215. The tangible memory1220,1225may be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of the two, accessible by the processing unit(s). The memory1220,1225stores software1280implementing one or more innovations described herein, in the form of computer-executable instructions suitable for execution by the processing unit(s).

A computing system may have additional features. For example, the computing system1200includes storage1240, one or more input devices1250, one or more output devices1260, and one or more communication connections1270. An interconnection mechanism (not shown) such as a bus, controller, or network interconnects the components of the computing system1200. Typically, operating system software (not shown) provides an operating environment for other software executing in the computing system1200, and coordinates activities of the components of the computing system1200.

The tangible storage1240may be removable or non-removable, and includes magnetic disks, magnetic tapes or cassettes, CD-ROMs, DVDs, or any other medium which can be used to store information in a non-transitory way and which can be accessed within the computing system1200. The storage1240stores instructions for the software1280implementing one or more innovations described herein.

The input device(s)1250may be a touch input device such as a keyboard, mouse, pen, or trackball, a voice input device, a scanning device, or another device that provides input to the computing system1200. For video encoding, the input device(s)1250may be a camera, video card, TV tuner card, or similar device that accepts video input in analog or digital form, or a CD-ROM or CD-RW that reads video samples into the computing system1200. The output device(s)1260may be a display, printer, speaker, CD-writer, or another device that provides output from the computing system1200.

FIG.13depicts an example cloud computing environment1300in which the described technologies can be implemented. The cloud computing environment1300comprises cloud computing services1310. The cloud computing services1310can comprise various types of cloud computing resources, such as computer servers, data storage repositories, networking resources, etc. The cloud computing services1310can be centrally located (e.g., provided by a data center of a business or organization) or distributed (e.g., provided by various computing resources located at different locations, such as different data centers and/or located in different cities or countries).

The cloud computing services1310are utilized by various types of computing devices (e.g., client computing devices), such as computing devices1320,1322, and1324. For example, the computing devices (e.g.,1320,1322, and1324) can be computers (e.g., desktop or laptop computers), mobile devices (e.g., tablet computers or smart phones), or other types of computing devices. For example, the computing devices (e.g.,1320,1322, and1324) can utilize the cloud computing services1310to perform computing operators (e.g., data processing, data storage, and the like).