Patent ID: 12249030

Reference numerals that are repeated across plural figures are intended to identify the same features in various implementations.

DETAILED DESCRIPTION

1. Introduction

Generally, the present disclosure is directed to a statistical, articulated 3D human shape modeling pipeline within a fully trainable, modular, deep learning framework. In particular, aspects of the present disclosure are directed to a machine-learned 3D human shape model with at least facial and body shape components that are jointly trained end-to-end on a set of training data. Joint training of the model components (e.g., including both facial and body components) enables improved consistency of synthesis between the generated face and body shapes.

More particularly, in some implementations, a set of training data can include high-resolution complete 3D body scans of humans captured in various poses, optionally together with additional closeups of their head and facial expressions and/or hand articulation. One example training dataset can include over 34,000 diverse human configurations. In some implementations, each of these scans can be registered to one or more initial, artist designed, gender neutral rigged quad-meshes to obtain registered shape scans of a human body.

Some example machine-learned 3D human shape models described herein can include a number of sub-models or other modular components. As examples, a human shape model can include a machine-learned facial expression decoder model trained to process a facial expression embedding associated with a human body to generate facial expression data for the human body; a machine-learned pose space deformation model trained to process a set of pose parameters to generate pose-dependent shape adjustments for the human body; and/or a machine-learned shape decoder model trained to process a rest shape embedding associated with the human body to generate identity-based rest shape data for the human body. For example, in some implementations, the decoder models can be trained as a portion of a respective autoencoder (e.g., variational autoencoder) trained to receive an input shape mesh (e.g., facial mesh or body rest shape mesh) and generate the respective embeddings (e.g., facial expression embedding or rest shape embedding).

Additional example models that can be include in the machine-learned 3D human shape model can include a machine-learned joint centers prediction model trained to process the identity-based rest shape data to generate a plurality of predicted joint centers for a plurality of joints of a skeleton representation of the human body and/or a machine-learned blend skinning model trained to process the facial expression data, the pose-dependent shape adjustments, the identity-based rest shape data, and the one or more predicted joint centers to generate the posed mesh for the human body.

According to an aspect of the present disclosure, some or all of the models described above or otherwise included in the machine-learned 3D human shape model can be trained jointly end-to-end on a shared loss function. Thus, in some implementations, all model parameters including non-linear shape spaces based on variational auto-encoders, pose-space deformation correctives, skeleton joint center predictors, and/or blend skinning functions can be trained in a single consistent learning loop.

Simultaneously training all of the models on the 3D dynamic scan data (e.g., over 34,000 diverse human configurations) can improve the overall model's ability to capture correlations and ensure consistency of various components (e.g., the modeled face, body, and/or hands). Stated differently, through the use of joint training of facial, body, and/or hand components on training data that can include facial, body, and/or hand scans, the resulting models can more naturally and consistently support facial expression analysis, as well as body (with detailed hand) shape and pose estimation.

The present disclosure provides two example fully trainable gender-neutral generic human models structured and trained as described herein and having two different resolutions—the moderate-resolution GHUM consisting of 10,168 vertices and the low-resolution GHUML(ite) of 2,852 vertices. Example experimental data is also provided for these two example human models, which demonstrates improved quality and consistency with reduced error. As examples,FIG.1provides example evaluations of GHUM and GHUML on data from GHS3D, with heatmaps of both models to the left. The renderings show registrations of different body poses of a subject (back row), as well as GHUML and GHUM fits in the middle and front rows, respectively. Both models demonstrate good quality estimates, with lower error for GHUM.

Thus, aspects of the present disclosure are directed to an end-to-end learning pipeline for constructing full body, statistical human shape and pose models capable of actuating body shape, as well as facial expressions and/or hand motion. End-to-end pipelines and unified loss functions are provided which enable computing system to perform deep learning, allowing for the simultaneous training of all model components, including non-linear shape spaces, pose-space deformation correctives, skeleton joint center estimators, and/or blend skinning functions in the context of minimal human skeleton parameterizations with anatomical joint angle constraints. The models can be trained with high-resolution full body scans, as well as closeups of moving faces and/or hands, in order to ensure maximum detail and design consistency between body part components.

In addition, a newly collected 3D dataset of generic human shapes, GHS3D, is described and consists of over 30,000 photo-realistic dynamic human body scans. Example embodiments also use over 4,000 full body scans from Caesar. Both a moderate-resolution model, GHUM, and a specially designed (not down-sampled) low-resolution model GHUML, are provided and their relative performance is assessed for registration and constrained 3D surface fitting, under different linear and non-linear models (PCA or variational auto-encoders for body shape and facial expressions). Recovery of shape and pose from images is also illustrated.

The systems and methods described herein provide a number of technical effects and benefits. As one example technical effect, the systems and methods of the present disclosure can provide for more realistic two- or three-dimensional renderings or models of human shapes, including human shapes which have improved consistency between body, facial, and/or hand features. Specifically, model components can learn to be consistent with each other and also learn cross-domain patterns or relationships. Thus, the systems and methods of the present disclosure can enable a computing system to perform improved human modeling functionality.

As another example technical effect, by jointly training multiple model components, the total amount of training time needed to produce a human shape model can be reduced. More particularly, previous approaches separately trained facial and body components and then sought to combine them after the fact, resulting in two different training processes and then additional work to facilitate the combination, that may still be inconsistent. The proposed approach jointly trains all model components in one end-to-end process, thereby making training and the resulting models more consistent.

With reference now to the Figures, example embodiments of the present disclosure will be discussed in further detail.

2. Overview of Example Implementations

Given a training set of human body scans, represented as unstructured point clouds {Y∈3P}, where the number of points P varies, techniques are provided which enable a statistical human model X(α)∈3Vto be learned which represents the variability of body shapes and deforming naturally as a result of articulation. The body model X can have consistent topology with V vertices, as specified by, for example, an artist-provided (rigged) template mesh, and α can be a set of variables that control the body deformation as a result of both shape and articulation. As illustrated inFIG.2, to learn a data-driven human model from scan data Y, the body template can first be registered to the point clouds in order to obtain new, registered ground truth meshes of the same topology, marked as {X*∈3V}.

The registered ground truth meshes X* can then be fed into an end-to-end training network where model parameters α are adjusted to produce outputs that closely match the input as a result of both articulation and shape adjustment. Various techniques can be used, including direct model parameter adjustment to the point cloud via iterative closest point (ICP) losses (identical to the ones used for registration) or with alignment to the proxy meshes X*. Having as targets input meshes X* of the same model topology, makes the process considerably faster and the training losses better behaved.

Thus,FIG.2illustrates an overview of an example end-to-end statistical 3D articulated human shape model construction. The training input can include a set of high-resolution 3D body scans including both resting (or ‘A’) pose and arbitrary poses exposing a variety of articulation and soft tissue deformations. Additionally, head closeup scans of detailed facial expressions and/or hand closeup scans to capture different gestures and object grabs can be collected. These scans are shown generally at Y.

Body landmarks can be automatically identified by rendering the photorealistic 3D reconstructions of the data (e.g., obtained using multi-view stereo triangulation techniques) from multiple virtual viewpoints (e.g., which may be different from the original set of cameras used for collecting the data), detecting those landmarks in the generated images and triangulating the landmark detections in images in order to obtain their corresponding 3d reconstruction. An artist designed full body articulated mesh can be progressively registered to point clouds using losses that combine sparse landmark correspondences and dense iterative closest point (ICP) residuals (e.g., implemented as point scan to mesh facet distances), under as conformal as possible surface priors. The registered ground truth shape scans are shown generally at X.

The example human shape model illustrated inFIG.2can have non-linear shape spaces implemented as deep variational auto-encoders (VAEs) for the body ϕb, and offset VAEs for the facial expressions ϕf. The example human shape model also includes trainable pose-space deformation functions D, modulated by a skeleton K with J joints, centers predictor C, and blend skinning functions M.

In some implementations, during training, all high-resolution scans of the same subjects (both full-body and closeups for face and hands) can be used (c.f.FIG.3), with residuals appropriately masked by a filter F. For model construction, N captured subjects can be used, with B full body scans, F closeup hand scans, and H closeup head scans. In some implementations, during learning, the training algorithm can alternate between minimizing the loss function w.r.t. pose estimates in each scan θ, and optimizing it with respect to the other model parameters (ϕ, γ, ψ, ω).

In operation, for pose and shape estimation, the model can be controlled by parameters α=(θ, β), including kinematic pose θ and VAE latent spaces for body shape and facial expressions β=(βf, βb), with encoder-decoders given by ϕ=(ϕf, ϕb).

2.1 Example Human Model Representation

Example implementations of the present disclosure can represent the human model as an articulated mesh, specified by a skeleton K with J joints and the skin deformed with Linear Blending Skinning (LBS) to explicitly encode the motion of joints. In addition to the skeletal articulation motion, nonlinear models can be used to drive facial expressions. A model X with J joints can be formulated as M(α=(θ, β), ϕ, γ, ω), or in detail, as
X(α)=M(θ,{tilde over (X)}(β),Δ{tilde over (X)}(θ),Δ{tilde over (X)}f(βf),C(X),ω)  (1)
where {tilde over (X)}(β)∈3Vis the identity-based rest shape in ‘A’ pose (seeFIG.2), with β a low-dimensional rest shape embedding vector encoding body shape variability (different low-dimensional representations including PCA or VAEs can be used); similarly, Δ{tilde over (X)}f(βf), is the facial expression at neutral head pose controlled with low-dimensional facial expression embedding βf; c=C(ψ)∈3Jare the skeletal joint centers dependent on the body shape; θ∈3×(J+1)is a vector of skeleton pose parameters consisting of (up to) 3 rotational DOFs in Euler angles for each joint and 3 translational variables at the root; ω∈V×1are per-vertex skinning weights (e.g., which may in some implementations be influenced by at most some number of joints (e.g., I=4)); and lastly pose-dependent corrective blend shapes Δ{tilde over (X)}(θ) are added to the rest shape to fix skinning artifacts.

Two example implementations of the proposed human models which are referred to herein as GHUM and GHUML are generated using artist-defined rigged template meshes (Vghum=10,168, Vghuml=2852, J=63), respectively. For both GHUM and GHUML, the pipeline illustrated inFIG.2estimated all the parameters (θ, ϕ, γ, ψ, ω) while the mesh topology and the joint hierarchy K are considered fixed. The hierarchy is anatomically (hence minimally) parameterized so that bio-mechanical joint angle limits can also be leveraged during optimization. Vertices xi∈X can be written

xi=∑j=1Iωi,j⁢Tj(θ,c)⁢Tj(θ¯,c)-1[x˜i+Δ⁢x˜i+Δ⁢x˜if1](2)Tj(θ,c)=∏a∈K⁡(j)[Ra(θa)ca01]∈S⁢E⁡(3),(3)
where Tj(θ, c) is the world transformation matrix for joint j, integrated by traversing the kinematic chain from the root to j. The transformation from the rest to the posed mesh can be constructed by multiplying by the inverse of world transformation matrix at rest poseθ.

3. Example End-to-End Statistical Model Learning

This section provides description of an example end-to-end neural network-based pipeline that optimizes the skinning weights ω, and learns a rest shape embedding {tilde over (X)}(βb), a facial expression embedding {tilde over (X)}f(βf), identity shape-dependent joint centers c(β), and pose-dependent blend shapesXp(θ) from multi-subject, multi-pose registered surface meshes X* to full body and close-up face and hand scans (seeFIG.2). As a result of ICP registration, some example reconstruction losses can be formulated using per-vertex Euclidean distance error under one-to-one correspondences as

Lr(X*,X⁡(α))=1V⁢∑i=1VFi(xi-xi*),(4)
where F is a filter that accounts for different types of data (e.g., full body scans as opposed to closeups). In some implementations, in order to construct X(α), the pose θ can be co-estimated jointly with the statistical shape parameters. As one example, block coordinate descent can be performed, including alternating between estimation of pose parameters θ under the current shape parameters, based on a BFGS layer, and updating the other model parameters with θ fixed. As one example, skinning can be initialized from artist-provided defaults, all other parameters to 0. Each sub-module can update the parameters α based on the global loss (4).

3.1 Example Variational Body Shape Autoencoder

In some example implementations, the multi-subject shape scans can be obtained by registering the models to the Caesar dataset (4,329 subjects) as well as captured scans in GHS3D, in resting or neutral ‘A’-pose. As one example,FIG.3illustrates close ups of head and face scans. Some example implementations estimate the full body shape at a neural A pose by fusing the body scan and the closeup hand and head scans. Compared with body shape estimation from a single body scan, these example implementations can take advantage of additional head and hand shape detail.

Given rest shapesXestimated for multiple subjects, a compact latent space can be built for the body shape variation. As one example, instead of simply building a PCA subspace, the body shapes can be represented using a deep nonlinear variational autoencoder with a lower-dimensional latent subspace. Because in some implementations mesh articulation is estimated, the input scansXto the autoencoder(s) are all well aligned at ‘A’ pose without significant perturbation from rigid transformations and pose articulations.

In some example implementations, the encoder and decoder can use parametric ReLU activation functions, as they can model either an identity transformation or a standard ReLU, for certain parameters. As standard practice, the variational encoder can output a mean and a variance (μ, Σ), which can be transformed to the latent space through the re-parametrization trick, in order to obtain the sampled code βb. In some implementations, a simple distribution,(0, I) can be used and the Kullback-Leibler divergence can be integrated in the loss function to regularize the latent space. Thus, one example formulation is as follows:

X~(βb)=1N⁢B⁢∑1N⁢BX¯+SD(βb)(5)βb=SE(X¯-1N⁢B⁢∑1N⁢BX¯)(6)
where the encoder SEcaptures the variance from the mean body shape into the latent vector βband the decoder SDbuilds up the rest shape from βbto match the input target rest shape. In particular, some example implementations initialize the first and last layer of the encoder and decoder, respectively, to the PCA subspace U∈3V×Lwhere L is the dimensionality of the latent space. All other fully-connected layers can be initialized to identity, including the PReLU units. In some example implementations, the sub-matrix of log-variance entries can be initialized to 0, and the bias can be set to a sufficiently large negative value. In this way, the network will effectively initialize from the linear model, while keeping additional parameters to a minimum, e.g., as compared to PCA.

3.2 Example Variational Facial Expression Autoencoder

The variational body shape autoencoder can represent various body shapes, including the variances of face shapes. To additionally support complex facial expressions (as opposed to just anthropometric head and face rest variations) additional facial modeling can optionally be introduced. For example, the model can be built from thousands of facial expression motion sequence scans in GHS3D. In addition to a 3-DOF articulated jaw, two 2-DOFs eyelids and two 2-DOFs eyeballs, the parameters of the articulated joints on the head, including skinning weights and pose space deformation, can be updated together with the rest of the pipeline.

For facial motion that is due to expression not articulation, a nonlinear embedding βfcan be built within the same network structure as the variational body shape autoencoder. The input to the VAE can be a facial expression Δ{tilde over (X)}f∈3Vf(Vf=1,932 for GHUM and 585 for GHUML) at neutral head pose by removing all articulated joint motion (including neck, head, eyes and jaw). In some implementations, to un-pose the registered head mesh to neutral, the articulated joint motion θ for the neutral head shape (without expression) that matches the registration can be fit as much as possible (4). The displacement field between the posed head and the registration is accounted to facial expressions and before the comparison the effect of articulated joints θ can be undone (unposed).

3.3 Example Skinning Model

After applying nonlinear shape and facial expression models, an optimal skinning function can be estimated statistically from multi-subject and multi-pose mesh data. Specifically, the same date term as in (4) can be used but now the optimization variables can be the parameters of the joint center estimator C(ψ), C:{tilde over (X)}→K, pose-dependent corrections to body shape D (θ, γ), and/or skinning weights ω.

One possible choice for skeletonal joint centers is to place them at average positions on the ring of boundary vertices connecting two mesh components (segmentations) maximally influenced by a joint. The average of boundary vertices,C{tilde over (X)}∈3J, imposes that the skeleton lies in the convex hull of the mesh surface, to adapt the centers to different body proportions. However, for better skinning, the estimateCcan be kept but a linear regressor ΔC:3V→3Jcan be built on top to learn joint center corrections from the body shape
c({tilde over (X)})=C{tilde over (X)}+ΔC{tilde over (X)}(7)

In some implementations, instead of learning joint centers globally by pooling over all mesh vertices, estimation can be performed only locally from those vertices skinned by the joint. This leads to considerably fewer trainable parameters going down from 3N×3J to 3N×3I, with e.g., I=4 in practice. Sparsity can also be encouraged through L1regularization, and also alignment of the bone directions to the template. To avoid singularities and prevent joint centers from moving outside the surface, the magnitude of center corrections ∥ΔC{tilde over (X)}∥2can be regularized.

In some implementations, to fix skinning artifacts as a result of complex soft tissue deformation, a data-driven pose-dependent corrector (PSD) Δ{tilde over (X)}(θ) can be learned and applied to the rest shape. A nonlinear mapping D:Ri(θi)−Ri(θi)∈9J→Δ{tilde over (X)}(θ)∈3ncan be estimated. However, pose space corrections on a mesh vertex should intuitively be sourced from neighboring joints. Therefore, some example implementations can use a fully-connected ReLU activated layer to extract a much more compact feature vector (e.g., 32 units) than the input, from which the pose space deformation can be linearly regressed.

As one example,FIG.5shows an example Pose Space Deformation architecture sketch and illustration showing the benefit of PSD, here around non-passive articulation points, e.g. right hip and thigh, as well as chest and armpits. For simplicity of illustration, here θ is used as the input feature instead of Ri(θi)−Ri(θi).

Moreover, in some instances {tilde over (X)}(θ) is sparse, and a joint can only generate local deformation correctives to its skinned mesh patch. Compared to the dense linear regressor in SMPL, the proposed network produces similar quality deformations with considerably fewer trainable parameters. The system can regularize the magnitude of pose space deformation to be small, preventing matching the targets by over-fitting through PSD corrections. This can be implemented by a simple L2penalty as
Lp(Δ{tilde over (X)})=∥Δ{tilde over (X)}(θ)∥2.  (8)

High-frequency local PSD is often undesirable and most likely due to overfitting. Therefore smooth pose space deformations can be encouraged with

Ls(Δ⁢X~)=∑i=1V∑j∈N⁡(i)li,j(Δ⁢x˜i-Δ⁢x˜j)2,(9)
where N(i) are the neighboring vertices to vertex i and li,jare cotangent-based Laplacian weights.

Even with PSD regularizers and a reduced number of trainable weights, overfitting could still occur. Differently from SMPL or MANO, where pose space deformation were built specifically for only certain regions (body or hand), a PSD model is in some implementations of the present disclosure constructed for the entire human model, trained jointly based on high-resolution body, hand and head data closeups. Consequently the body data has limited variation on hand and head motions, whereas head and hand data has no motion for the rest of the body. Hence, there is a large articulation space where all joints can move without an effect on the loss. This is undesirable. To prevent overfitting, the input pose feature vector can be filtered or masked into 4 feature vectors, taking head, body, left hand and right hand joints. Each feature vector can be taken into the same ReLU layer and the outputs can be summed before the next regressor. Thus, one example loss is as follows:
LF(Δ{tilde over (X)})=∥FΔ{tilde over (X)}−Δ{tilde over (X)}∥2,  (10)
which enforces PSDs outside masked regions to be small, thus biasing the correctives produced by the network towards limited global impact. However, deformations of shared surface regions corresponding to areas between the head, hand, and the rest of the body, are learnt from all relevant data.

In some implementations, to estimate skinning weights, at the end of the pipeline, a linear blending skinning layer can be used which, given poses θ and pose-corrected rest shape with facial expression {tilde over (X)}+Δ{tilde over (X)}+Δ{tilde over (X)}f, outputs a posed mesh (2) controlled by trainable skinning weight parameters ω. Each skinned vertex can optionally be maximally influenced by some number (e.g., I=4) joints in the template. The system can then regularize ω to be close to the initial artist painted valuesω, to be spatially smooth, and/or per-vertex skinning weights to be non-negative and normalized

Lωs(ω)=∑i=1V∑j∈N⁡(i)∑k=1Ili,j(ωi,k-ωj,k)2⁢Lωi(ω)=∑i=1V∑k=1Ili,j(ωi,k-ω¯i,k)2⁢s.t.⁢∑k=1Iωi,k=1,ωi,k≥0.(11)

The final skinned mesh X can also be weakly regularized to be smooth by adding

Lm(X)=∑i=1V∑j∈N⁡(i)li,j(xi-xj)2.(12)

Pose Estimator. Given body shape estimates and current skinning parameters, the poses θ can be reoptimized over the training set. To limit the search space, enforce consistency, and avoid unnatural local minimums, the anatomical joint angle limits available with the anthropometric skeleton can be leveraged. The problem can be efficiently solved using an L-BFGS solver with box constraints, and gradients evaluated by TensorFlow's automatic differentiation.

4. Example Experiments

This section describes example experiments conducted on example implementations of the systems and methods described herein.

Datasets. In addition to Caesar, which contains diverse body and face shapes (4,329 subjects), the example experiments described herein also used multiple 3dmd systems operating at 60 Hz to capture 32 subjects (16 females and 16 males) with 55 body poses, 60 hand poses and 40 motion sequences of facial expressions. The subjects have a BMI range from 17.5 to 39, height from 152 cm to 192 cm and are aged from 22 to 47. For all multi-pose data, we use 3 subjects for evaluation, and 4 subjects for testing, based on a freestyle motion sequence containing poses generally not in the training set. Each face capture sequence starts from a neutral face to a designated facial expression and each sequence lasts about 2s.

Registration samples from the data are shown inFIG.6. Specifically,FIG.6shows sample registrations for data from Caesar (top left) as well as GHS3D. Notice the quality of registration that captures facial detail and soft tissue deformation of the other body parts as a result of articulation.

Registration. Table 1 reports registration to the point clouds using ICP and the (extended) Chamfer distance. ICP error is measured as point-to-plane distance to the nearest registered mesh facet, whereas Chamfer is estimated point to point, bidirectionally.

TABLE 1Registration error on Caesar and GHS3D (with detail for faces,hands, and the rest of the body) for GHUM and GHUML.ICP error (mm)Chamfer distance (mm)DatasetGHUMGHUMLGHUMGHUMLCaesar0.2650.46519.1331.84body0.3710.72520.7633.64head0.4420.51910.1212.38hand0.1640.42314.8822.01

The proposed registration technique has low error and preserves local point cloud detail (FIG.6).

Model Evaluation. Both a full resolution and a low-resolution human model (GHUM and GHUML) were built using our end-to-end pipeline. Both models share the same set of skeleton joints but have 10,168 vs. 2,852 mesh vertices (with 1,932 vs. 585 vertices for facial expressions). For both models, the example experiments evaluated the mean vertex-based Euclidean distances of meshes X to registrations X* on testing data. Numbers are reported in Table 2 and visualizations are shown inFIGS.4,1, and9(please find hand evaluations in Sup. Mat). We compare the outputs of both models to registered meshes under their corresponding topology. Both models can closely represent a diversity of body shapes (e.g., modeled as VAEs), produce natural facial expressions (e.g., represented as facial VAEs) and pose smoothly and naturally without noticeable skinning artifacts for a variety of body shapes and poses (e.g., resulting from optimized skinning parameters).

TABLE 2Mean vertex-based Euclidean error for registration (mm).DatasetCaesarGHS3D → bodyfacehandGHUM1.965.261.642.96GHUML2.045.152.294.76

GHUM vs GHUML. The low resolution model preserves the global features of the body shape and correctly skins the body and facial motion. Compared with GHUM, it can be observed that GHUML loses some detail for lip deformations, muscle bulges at the arms and fingers, and wrinkles due to fat tissue. Performance-wise, GHUML is 2.5× faster, in feed-forward evaluation mode, than GHUM.

FIG.4shows evaluation on Caesar. The left side ofFIG.4shows per-vertex Euclidean distance error to the registration for GHUM and GHUML. The right side ofFIG.4shows, from top to bottom, registrations, GHUM, and GHUML. VAE-based models can represent body shape very well. Compared to GHUML, additional muscle or waist soft tissue detail is preserved by GHUM.

FIG.9shows evaluation and rendering as inFIG.1with emphasis on the hand reconstruction of GHUM and GHUML. Similar conclusions as inFIG.1hold. Notice additional deformation details around the flexion region of the palm preserved by GHUM over GHUML.

VAE Evaluation. For body shape, the proposed VAE supports both a 16-dim and a 64-dim latent representation where the former has 1.72× higher reconstruction error (report in Table 2 and figures is based on a 64-dim representation). In some examples, a 20-dim embedding can be used for the facial expression VAE.

FIG.7shows the reconstruction error of facial expressions as a function of the latent dimension, for both VAE and PCA. The 20-dimensional VAE has a reconstruction error similar to the one that uses 96 linear PCA basis, at the cost of 2.9× slower performance. Specifically,FIG.7shows an analysis of VAE and PCA models which illustrates the advantages of non-linearity in the low-dimensional regime.

GHUM vs SMPL. InFIG.8, GHUM and SMPL are compared for visual quality. In particular,FIG.8shows, from left to right, registration, GHUM, and SMPL for each of two poses. GHUM produces posing of comparable visual quality, albeit notice fewer pelvis artefacts for this motion sequence.

GHUM has different mesh and skeleton typologies from SMPL and SMPL does not have hand and facial joints. To compare, a captured motion sequence (all the poses, not in our training dataset) from GHS3D is taken, and the captured sequence is registered with SMPL and GHUM mesh respectively. When the error is evaluated, one-to-one point-to-plane Euclidean distance is used (e.g., to avoid sensitivity to surface sliding during registration), and the error is only evaluated on the body region for fair comparison with SMPL. The mean reconstruction error from GHUM is 4.4 mm whereas SMPL has 5.37 mm error, and visual skinning quality for GHUM is observed to be on par with SMPL.

3D Pose and Shape Reconstruction from Monocular Images. This section illustrates image inference with GHUM. In this case, the kinematic prior of the model (for hands and the rest of the body, excluding the face) has been trained with data from Human3.6M, CMU, and GHS3D. An image predictor was not used for pose and shape. Instead, initialization was performed at6different kinematic configurations and α parameters were optimized under anatomical joint angle limits. As loss, the skeleton joints reprojection error and a semantic body-part alignment were used. The results are shown inFIG.10. Specifically,FIG.10shows monocular 3D human pose and shape reconstruction with GHUM by relying on non-linear pose and shape optimization under a semantic body part alignment loss.

5. Example Devices and Systems

FIG.11Adepicts a block diagram of an example computing system100according to example embodiments of the present disclosure. The system100includes a user computing device102, a server computing system130, and a training computing system150that are communicatively coupled over a network180.

The user computing device102can be any type of computing device, such as, for example, a personal computing device (e.g., laptop or desktop), a mobile computing device (e.g., smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

The user computing device102includes one or more processors112and a memory114. The one or more processors112can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory114can include one or more non-transitory computer-readable storage mediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory114can store data116and instructions118which are executed by the processor112to cause the user computing device102to perform operations.

In some implementations, the user computing device102can store or include one or more machine-learned models120. For example, the machine-learned models120can be or can otherwise include various machine-learned models such as neural networks (e.g., deep neural networks) or other types of machine-learned models, including non-linear models and/or linear models. Neural networks can include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks or other forms of neural networks. Example machine-learned models120are discussed with reference toFIG.2.

In some implementations, the one or more machine-learned models120can be received from the server computing system130over network180, stored in the user computing device memory114, and then used or otherwise implemented by the one or more processors112. In some implementations, the user computing device102can implement multiple parallel instances of a single machine-learned model120.

Additionally or alternatively, one or more machine-learned models140can be included in or otherwise stored and implemented by the server computing system130that communicates with the user computing device102according to a client-server relationship. For example, the machine-learned models140can be implemented by the server computing system140as a portion of a web service (e.g., a body shape modeling and/or rendering service). Thus, one or more models120can be stored and implemented at the user computing device102and/or one or more models140can be stored and implemented at the server computing system130.

The user computing device102can also include one or more user input component122that receives user input. For example, the user input component122can be a touch-sensitive component (e.g., a touch-sensitive display screen or a touch pad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component can serve to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

The server computing system130includes one or more processors132and a memory134. The one or more processors132can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory134can include one or more non-transitory computer-readable storage mediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory134can store data136and instructions138which are executed by the processor132to cause the server computing system130to perform operations.

In some implementations, the server computing system130includes or is otherwise implemented by one or more server computing devices. In instances in which the server computing system130includes plural server computing devices, such server computing devices can operate according to sequential computing architectures, parallel computing architectures, or some combination thereof.

As described above, the server computing system130can store or otherwise include one or more machine-learned models140. For example, the models140can be or can otherwise include various machine-learned models. Example machine-learned models include neural networks or other multi-layer non-linear models. Example neural networks include feed forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Example models140are discussed with reference toFIG.2.

The user computing device102and/or the server computing system130can train the models120and/or140via interaction with the training computing system150that is communicatively coupled over the network180. The training computing system150can be separate from the server computing system130or can be a portion of the server computing system130.

The training computing system150includes one or more processors152and a memory154. The one or more processors152can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory154can include one or more non-transitory computer-readable storage mediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory154can store data156and instructions158which are executed by the processor152to cause the training computing system150to perform operations. In some implementations, the training computing system150includes or is otherwise implemented by one or more server computing devices.

The training computing system150can include a model trainer160that trains the machine-learned models120and/or140stored at the user computing device102and/or the server computing system130using various training or learning techniques, such as, for example, backwards propagation of errors. For example, a loss function can be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on a gradient of the loss function). Various loss functions can be used such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and/or various other loss functions. Gradient descent techniques can be used to iteratively update the parameters over a number of training iterations.

In some implementations, performing backwards propagation of errors can include performing truncated backpropagation through time. The model trainer160can perform a number of generalization techniques (e.g., weight decays, dropouts, etc.) to improve the generalization capability of the models being trained.

In particular, the model trainer160can train the machine-learned models120and/or140based on a set of training data162. The training data162can include, for example, full body, hand, and/or facial scans and/or ground truth registrations of such scans.

In some implementations, if the user has provided consent, the training examples can be provided by the user computing device102. Thus, in such implementations, the model120provided to the user computing device102can be trained by the training computing system150on user-specific data received from the user computing device102. In some instances, this process can be referred to as personalizing the model.

The model trainer160includes computer logic utilized to provide desired functionality. The model trainer160can be implemented in hardware, firmware, and/or software controlling a general purpose processor. For example, in some implementations, the model trainer160includes program files stored on a storage device, loaded into a memory and executed by one or more processors. In other implementations, the model trainer160includes one or more sets of computer-executable instructions that are stored in a tangible computer-readable storage medium such as RAM hard disk or optical or magnetic media.

The network180can be any type of communications network, such as a local area network (e.g., intranet), wide area network (e.g., Internet), or some combination thereof and can include any number of wired or wireless links. In general, communication over the network180can be carried via any type of wired and/or wireless connection, using a wide variety of communication protocols (e.g., TCP/IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and/or protection schemes (e.g., VPN, secure HTTP, SSL).

FIG.11Aillustrates one example computing system that can be used to implement the present disclosure. Other computing systems can be used as well. For example, in some implementations, the user computing device102can include the model trainer160and the training dataset162. In such implementations, the models120can be both trained and used locally at the user computing device102. In some of such implementations, the user computing device102can implement the model trainer160to personalize the models120based on user-specific data.

FIG.11Bdepicts a block diagram of an example computing device10that performs according to example embodiments of the present disclosure. The computing device10can be a user computing device or a server computing device.

The computing device10includes a number of applications (e.g., applications1through N). Each application contains its own machine learning library and machine-learned model(s). For example, each application can include a machine-learned model. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.

As illustrated inFIG.11B, each application can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and/or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

FIG.11Cdepicts a block diagram of an example computing device50that performs according to example embodiments of the present disclosure. The computing device50can be a user computing device or a server computing device.

The computing device50includes a number of applications (e.g., applications1through N). Each application is in communication with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and model(s) stored therein) using an API (e.g., a common API across all applications).

The central intelligence layer includes a number of machine-learned models. For example, as illustrated inFIG.11C, a respective machine-learned model (e.g., a model) can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine-learned model. For example, in some implementations, the central intelligence layer can provide a single model (e.g., a single model) for all of the applications. In some implementations, the central intelligence layer is included within or otherwise implemented by an operating system of the computing device50.

The central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized repository of data for the computing device50. As illustrated inFIG.11C, the central device data layer can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and/or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

6. Additional Disclosure

The technology discussed herein makes reference to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and functionality between and among components. For instance, processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

While the present subject matter has been described in detail with respect to various specific example embodiments thereof, each example is provided by way of explanation, not limitation of the disclosure. Those skilled in the art, upon attaining an understanding of the foregoing, can readily produce alterations to, variations of, and equivalents to such embodiments. Accordingly, the subject disclosure does not preclude inclusion of such modifications, variations and/or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art. For instance, features illustrated or described as part of one embodiment can be used with another embodiment to yield a still further embodiment. Thus, it is intended that the present disclosure cover such alterations, variations, and equivalents.