Patent Publication Number: US-2023141392-A1

Title: Systems and methods for human pose and shape recovery

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application is a continuation application of U.S. patent application Ser. No. 16/995,446, filed Aug. 17, 2020, which claims the benefit of Provisional U.S. Patent Application No. 62/941,203, filed Nov. 27, 2019, the disclosure of which is incorporated herein by reference in its entirety. 
    
    
     BACKGROUND 
     Computer-generated human body models that realistically represent a patient&#39;s pose and shape can be used in a wide range of medical applications including, e.g., patient positioning, surgical navigation, unified medical record analysis, etc. For instance, in radiation therapy and medical imaging, success often hinges on the ability to place and maintain a patient in a desirable position so that the treatment or scan can be performed in a precise and accurate manner. Having real time knowledge about the patient&#39;s s physical characteristics such as the patient&#39;s body shape and pose in these situations may bring many benefits such as faster and more accurate positioning of the patient in accordance with a scan or treatment protocol, more consistent results across multiple scans or treatment sessions, etc. In other example situations such as during surgical procedure, information about a patient&#39;s physiques may offer insight and guidance for both surgery planning and execution. For instance, the information may be utilized to locate and navigate around a surgical area of the patient. And when visually presented in real time, the information may also provide means for monitoring the state of the patient during the procedure. 
     Conventional techniques for recovering a human model for a patient rely on having comprehensive knowledge about the joint locations of the patient and may only be able to recover the pose of the patient based on the joint location. In many medical settings, however, a patient&#39;s body is often blocked, at least partially, by medical equipment and/or clothing items, and it may also be beneficial in those settings to obtain information about the body shape of the patient. Therefore, it is highly desirable to have the ability to recover both the pose and shape of a patient despite only having limited information about the physical characteristics of the patient. 
     SUMMARY 
     Described herein are systems, methods and instrumentalities for recovering the pose and shape of a person based on one or more joint locations of the person. The joint locations may be determined based on an image of the person such as an image that includes color and/or depth information representing the physical characteristics of the person. The joint locations may be a subset of all joint locations of the person (e.g., not including joints that are obstructed or otherwise unknown). The systems, methods and instrumentalities may include one or more processors and one or more storage devices storing instructions that, when executed by the one or more processors, cause the one or more processors to implement an artificial neural network and provide information regarding the one or more joint locations of the person to the artificial neural network. The artificial neural network may determine a first plurality of parameters associated with a pose of the person and a second plurality of parameters associated with a shape of the person based on the information regarding the one or more joint locations of the person. Based on the first and second pluralities of parameters, one or more human models may be generated that represent the pose and shape of the person. 
     The artificial neural network may be trained to perform one or more of the tasks described above using training data that comprises joint locations of a human body. During the training, the artificial neural network may predict pose and shape parameters associated with the human body based on the joint locations comprised in the training data. The artificial neural network may then infer joint locations of the human body from the predicted pose and shape parameters, and adjust (e.g., optimize) the operating parameters (e.g., weights) of the artificial neural network based on a difference between the inferred joint locations and the joint locations comprised in the training data. In examples, the training data may also comprise pose and shape parameters associated with the joint locations of the human body and the artificial neural network may adjust (e.g., optimize) its operating parameters further based on a difference between the predicted pose and shape parameters and the pose and shape parameters comprised in the training data. 
     To acquire the ability to predict the pose and shape parameters based on partial knowledge about the joint locations of a person (e.g., certain joint locations of the person may be obstructed, unobserved or otherwise unknown to the neural network), the training of the artificial neural network may involve providing a subset of joint locations to the artificial neural network and forcing the artificial neural network to predict the pose and shape parameters using the subset of joint locations. For instance, the training may utilize an existing parametric human model associated with a human body to determine a plurality of joint locations of the human body and then randomly exclude a subset of the plurality of joint locations from the input of the artificial neural network (e.g., by artificially treating the subset of joint locations as unobserved and unavailable). 
     The joint locations described herein may include two-dimensional (2D) and/or three-dimensional (3D) joint locations of a person. When the training is conducted using at least 2D joint locations, the artificial neural network may, during the training, predict pose and shape parameters based on the 2D joint locations, infer 3D joint locations of the human body using the predicted pose and shape parameters, and project the 3D joint locations into an image space to obtain corresponding 2D joint locations. The artificial neural network may then adjust its operating parameters based on a difference between the projected 2D joint locations and the 2D joint locations comprised in the training data. 
     The pose and shape parameters described herein may be recovered separately (e.g., independent of each other). For example, the pose parameters recovered may be body shape independent (e.g., independent of the height and weight of the person). Hence, in addition to medical applications, the techniques described herein may be used with various human motion capture systems (e.g., which may output 3D joint locations), for example, in game development, animation development, special effects for motion pictures, etc. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       A more detailed understanding of the examples disclosed herein may be had from the following description, given by way of example in conjunction with the accompanying drawing. 
         FIG.  1    is a block diagram illustrating an example system for recovering the pose and shape of a person based on joint location information associated with the person. 
         FIG.  2    is a block diagram illustrating an example of training an artificial neural network to learn a model for predicting the pose and shape of a person. 
         FIG.  3    is a flow diagram illustrating an example neural network training process. 
         FIG.  4    is a block diagram illustrating an example neural network system as described herein. 
     
    
    
     DETAILED DESCRIPTION 
     The present disclosure is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings. 
       FIG.  1    is a diagram illustrating an example system  100  configured to recover the pose and shape of a person and generate a human model representing such recovered pose and shape of the person based on joint location information associated with the person. As shown, the system  100  may be configured to receive information regarding the joint locations  102  of a person. The joint locations  102  may include 2D joint locations (e.g., 2D coordinates, 2D keypoints, 2D feature maps (binary heatmaps), etc., that represent the 2D joint locations). The joint locations  102  may include 3D joint locations (e.g., 3D coordinates, 3D keypoints, 3D feature maps, etc., that represent the 3D joint locations). The joint locations  102  may also include a combination of 2D and 3D joint locations. The joint locations  102  may be derived from an image of the person such as a color image (e.g., a red-green-blue or RGB image) of the person, a depth image of the person, or an image of the person that includes both color and depth information (e.g. a color plus depth or RGB-D image). Such images may be captured by a sensor (e.g. an RGB sensor, a depth sensor, an infrared sensor, etc.), a camera, or another suitable device (e.g. such as a medical imaging device) that is capable of producing a visual representation of the person having a certain pose and body shape. The 2D and/or 3D joint locations may be derived based on 2D and/or 3D features extracted from the image, for example, using one or more convolutional encoders. Examples of convolutional encoders may be found in commonly assigned U.S. patent application Ser. No. 16/863,382, filed Apr. 30, 2020, entitled “Systems and Methods for Human Mesh Recovery,” the disclosure of which is hereby incorporated by reference in its entirety. 
     The joint locations  102  may also be derived from a system or application configured to output the joint location information of a person. Such a system or application may be, for example, another medical imaging system configured to determine and output the joint locations of a person, a human motion capture system that is used in gaming development, animation development, special effects generation (e.g., for motion pictures), etc. Accordingly, the system  100  described herein may be used as an add-on to another system. As such, when referred to herein, providing joint location information to a neural network may include receiving the joint location information from another medical imaging system or human motion capture system and providing the received information to the neural network. 
     The joint locations  102  may not include all of the joint locations of the person as depicted in an image or output by a human motion capture system. For example, one or more joint locations of the person may be obstructed or unobserved in the image, mis-predicted by an upstream device or program, or otherwise unknown to the system  100 . As a result, the joint locations  102  used by the system  100  to predict the pose and shape of a person may only include a subset of the joint locations of the person. 
     The system  100  may include a pose and shape regressor  104  configured to receive the joint locations  102  (e.g., as an input) and determine a plurality of parameters θ associated with a pose of the person and a plurality of parameters β associated with a shape of the person based on the joint locations  102 . The pose and shape regressor  104  may be realized using an artificial neural network (ANN) that comprises a plurality of layers such as one or more input layers, one or more hidden layers, and/or one or more output layers. The input layers of the ANN may be configured to receive the joint locations  102  and pass them to subsequent layers for processing. Each of the input layers may include one or more channels and each channel may be configured to receive data from a respective data source. The hidden layers of the ANN may include one or more convolutional layers, one or more pooling layers, and/or one or more fully-connected (FC) layers (e.g., regression layers). For example, the hidden layers may include a plurality of (e.g., a stack of ten) FC layers with rectified linear unit (ReLU) activation functions and each of the FC layers may include a plurality of units (e.g., neurons) having respective weights that, when applied to features associated with the joint locations  102 , regress the pose parameters θ and the shape parameters β based on features associated with the joint locations  102 . 
     In examples, N joints may be represented with a vector: [x 1 , y 1 , x 2 , y 2 , x 3 , y 3  . . . ], where x and y represent the location of each joint. This may result in a  2 N-dimensional vector that corresponds to the x and y of each of the N joints. Responsive to receiving such a vector input, the ANN (e.g., the FC layers of the ANN) may progressively transform the input vector into one of different dimensionality. Each FC unit may be followed by a non-linear activation unit (e.g., an ReLU) and/or the next FC unit. An output vector with K dimensions may be derived based this operation, where K may represent the number of parameters estimated. For example, if the pose parameters θ have 75 dimensions and the shape parameters θ have 10 dimensions, then K may be equal to 85 (e.g., K=75+10). 
     In examples, a joint location may be represented by a feature map such as a binary heatmap, and as such the joint may have a matrix representation. Multiple such matrices (e.g., corresponding to N joints) may be combined to produce an N-channel (e.g., one channel per joint) input to the ANN. The input may be processed through one or more convolutional layers of the ANN before being culminated by the FC units of the ANN to produce a K-dimensional output vector representing the pose parameters θ and the shape parameters β. 
     In examples, the regressed pose parameters θ may include 72 parameters (e.g., 3 parameters for each of  23  joints and 3 parameters for a root joint) and the regressed shape parameters β may include a plurality of coefficients (e.g., the first  10  coefficients) of a principle component analysis (PCA) space. Once the pose and shape parameters are determined, a plurality of vertices (e.g.,  6890  vertices based on  72  pose parameters and  10  shape parameters) may be obtained for constructing a human model representing the pose and shape of the person. For instance, the human model may be a statistical parametric differentiable model such as a skinned multi-person linear (SMPL) model that defines a function M(β, θ; Φ):R 82 →R 3xN (where Φ may represent learned SMPL model parameters) for producing N mesh vertices associated with a 3D mesh of the person. The mesh vertices may be used by a mesh generator  108  of the system  100  to generate a visual representation  110  of the human model, for example, by shaping template body vertices conditioned on θ and β, articulating bones according to the joint rotations indicated by θ (e.g., via forward kinematics), and deforming 3D surfaces with linear blend skinning. 
     The pose and shape regressor  104  may be configured to recover the pose parameters θ separately or independently from the shape parameters β. Such separation or independence may be accomplished, for example, by applying pose normalization during the recovery operation so that the pose parameters may be estimated without knowledge of the subject&#39;s specific body shape and vice versa. The pose parameters θ thus recovered may be body shape independent (e.g., independent of the height and/or weight of the person) and likewise the recovered shape parameters β may be also pose independent (e.g. independent of the joint angles of the person). This way, either of the recovered pose parameters or shape parameters may be used in a system or application (e.g., a human motion capture system) without the other one of the recovered pose parameters or shape parameters. 
     The pose and shape regressor  104  may be trained to predict the pose parameters θ and the shape parameters β using training data that include joint location information of a human body. The joint location information may be paired with human model parameters (e.g., SMPL parameters) to introduce a cycle consistency objective to the training.  FIG.  2    is a diagram illustrating an example of training an ANN  204  (e.g., the pose and shape regressor  104 ) to learn a model for predicting the pose and shape parameters associated with a human model based on joint location information. The training may be conducted using data that include joint location information  202  (e.g., 2D or 3D keypoints or features representing the joint locations) and/or corresponding parameters  206  of a human model from which the joint locations  202  may be derived. Such paired training data may be obtained, for example, from publicly available human motion capture (MoCap) databases and used as ground truth for optimizing the parameters of the ANN  204 . 
     During the training, the ANN  204  may receive the joint location information  202  and predict pose and shape parameters  208  based on the joint location information using initial operating parameters of the ANN  204  (e.g., initial weights associated with one or more fully-connected layers of the ANN  204 ). The initial operating parameters may be derived, for example, by sampling them from one or more probability distributions and/or based on parameter values of another neural network with a similar architecture. Upon making the prediction, the ANN  204  may compare the predicted pose and shape parameters  208  to the ground truth parameters  206  and calculate a parameter loss  210  in accordance with a loss function. The loss function may be, for example, based on an Euclidean distance between the predicted pose and shape parameters  208  and the ground truth parameters  206 , e.g., L′=∥[β,θ]−[{circumflex over (β)}, {circumflex over (θ)}]∥ 2   2 , where [{circumflex over (β)},{circumflex over (θ)}] represents the predicted pose and shape parameters  208  and [β, θ] represents the ground truth pose and shape parameters  206 . The loss function may also be based on L 1  distance, Haussdorf distance, and/or the like. 
     The ANN  204  may additionally consider a joint location loss when adjusting its operating parameters. For example, the training of the ANN  204  may further utilize a joint location regressor  212  to infer joint locations  214  (e.g., 3D joint locations) based on the predicted pose and shape parameters  208  and adjusting the operating parameters of the ANN  204  by minimizing the difference between the inferred joint locations  214  and the input joint locations  202 . The joint location regressor  212  (e.g., which may comprise a joint regression layer) may be configured to output joint locations based on pose and shape parameters received at an input. The joint locations  214  may be inferred, for example, by applying linear regression to one or more mesh vertices determined from the predicted pose and shape parameters  208  and/or an SMPL model. A joint location loss  216  may be calculated to represent the difference between the inferred joint locations  214  and the input joint locations  202 . For example, denoting the input joint locations  202  as J and the pose and shape parameters  208  predicted by the ANN  204  as G(J), the inferred joint locations  214  may be represented by X(G(J)), where X may represent the joint location regressor  212 . Consequently, the joint location loss  216  may be calculated in accordance with L″=∥ J−X(G(J))∥ 2   2 , and an overall loss function L for training the ANN  204  may be derived as: 
         L =∥[β,θ]−[{circumflex over (β)},{circumflex over (θ)}]∥ 2   2   +∥J−X ( g ( J ))∥ 2   2  
 
     Accordingly, the ANN  204  may adjust its operating parameters (e.g., weights) with an objective of minimizing the overall loss function L. For example, upon obtain the initial pose and shape estimate [{circumflex over (β)}, {circumflex over (θ)}], the ANN  204  may update its operating parameters via a backpropagation process (e.g., based on a gradient descent of the loss function with respect to the current set of parameters). The ANN may then repeat the prediction and adjustment process described above until one or more training termination criteria are satisfied (e.g., after completing a pre-determined number of training iterations, until the change in the value of the loss function L between consecutive training iterations falls below a predetermined threshold, etc.). 
     In example implementations, the training of the ANN  204  may be conducted using annotated 2D joint locations (e.g., using only annotated 2D joint locations or using 2D and 3D join locations) as an input. During the training, the ANN  204  may predict the pose and shape parameters  208  based on the input 2D joint locations and infer, via the joint location regressor  212 , 3D joint locations based on the predicted pose and shape parameters. To verify the accuracy of the predicted parameters  208 , the ANN  204  may project the inferred 3D joint locations onto a 2D image plane to obtain corresponding 2D joint locations (e.g., 2D coordinates, keypoints and/or features indicating the 2D joint locations). The ANN  204  may then compare the projected 2D joint locations with the annotated 2D joint location received at the input and adjust the operating parameters of the ANN  204  based on a loss function associated with the annotated 2D joint locations and the projected 2D joint locations. For example, the projection may be performed based on a weak-perspective camera model and the 2D joint locations may be derived as x=sπ(RX (β,θ))+t, where R∈ R 3  may represent a global rotation in an axis-angle representation, tε R 2 : and s may correspond to translation and scale, respectively, and π may denote an orthographic projection. Consequently, the training of the ANN  204  may be conducted with an objective to minimize the loss between the annotated input 2D joint locations and the projected 2D joint locations, which may be represented as L=∥J−x∥ 2   2  with J denoting the annotated input 2D joint locations. The training may also be supplemented by further considering one or more of a parameter loss (e.g., between input pose and/or shape parameters comprised in the training data and predicted pose and/or shape parameters) or a 3D joint location loss (e.g., between the inferred 3D joint locations and the 3D joint locations comprised in the training data), as described herein. 
     The ANN  204  may also be trained to predict the pose and shape of a person based on incomplete (e.g., partial) knowledge of the joint locations of the person. As described above, the ANN  204  may only possess such incomplete or partial knowledge of the joint locations of the person because some joint locations of the person may be obstructed, unobserved, or otherwise unknown to the ANN  204 . The training of the ANN  204  may account for such situations. For example, during the training of the ANN  204 , one or more randomly selected subsets of the joint locations comprised in the training data may be excluded (e.g., marked as unavailable or unobserved) from the input to the ANN  204  and the ANN  204  may be forced to adjust its operating parameters to accommodate the incomplete input (e.g., the ANN may be forced to make prediction that approximates the given ground truth despite having only partial information about the joint locations). 
       FIG.  3    is a flow diagram of an example process  300  for training a neural network system described herein (e.g., the pose and shape regressor  104  in  FIG.  1    or the ANN  204  in  FIG.  2   ). The process  300  may start at  302  and at  304  the neural network system may initialize its operating parameters such as the weights associated with one or more hidden layers (e.g., fully-connected layers) of the neural network system. The parameters may be initialized, for example, by sampling one or more probability distributions or based on parameter values of another neural network with a similar architecture. At  306 , the neural network system may receive joint location information associated with a human body at an input and process the joint location information using the initial operating parameters. The neural network system may predict a pose and a shape of the human body based on the input joint location information. At  308 , the neural network system may determine the adjustments that need to be made to its operating parameters based on a loss function and a gradient descent (e.g., a stochastic gradient decent) associated with the loss function (e.g. for minimizing the loss function). The loss function may be implemented based on a mean squared error (MSE) or an Euclidean distance between the predicted pose and shape parameters and ground truth parameters (e.g., which may be paired with the joint location information in the training dataset) and/or a MSE or Euclidean distance between the input joint locations and/or joint locations inferred from the predicted pose and shape parameters (e.g., as described herein). The loss function may also take into consideration L 1  norm, L 2  norm, or L 2  norm with L 1  norm. 
     At  310 , the neural network system may perform the adjustments to its current operating parameters, for example, via a backpropagation process. At  312 , the neural network system may determine whether one or more training termination criteria are satisfied. For example, the system may determine that the training termination criteria are satisfied if the system has completed a pre-determined number of training iterations, if the difference between the predicted parameters and the ground truth parameters is below a predetermined threshold, or if the change in the value of the loss function between two training iterations falls below a predetermined threshold. If the determination at  312  is that the training termination criteria are not satisfied, the system may return to  306 . If the determination at  312  is that the training termination criteria are satisfied, the system may end the training process  300  at  314 . 
     The neural network system described herein (e.g., the pose and shape regressor  104  in  FIG.  1    or the ANN  204  in  FIG.  2   ) may be implemented using one or more processors, one or more storage devices, and/or other suitable accessory devices such as display devices, communication devices, input/output devices, etc.  FIG.  4    is a block diagram illustrating an example neural network system  400  as described herein. As shown, the neural network system  400  may include a processor  402 , which may be a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, a reduced instruction set computer (RISC) processor, application specific integrated circuits (ASICs), an application-specific instruction-set processor (ASIP), a physics processing unit (PPU), a digital signal processor (DSP), a field programmable gate array (FPGA), or any other circuit or processor capable of executing the functions described herein. The neural network system  400  may further include a communication circuit  404 , a memory  406 , a mass storage device  408 , an input device  410 , and/or a communication link  412  (e.g., a communication bus) over which the one or more components shown in  FIG.  4    may exchange information. The communication circuit  404  may be configured to transmit and receive information utilizing one or more communication protocols (e.g., TCP/IP) and one or more communication networks including a local area network (LAN), a wide area network (WAN), the Internet, a wireless data network (e.g., a Wi-Fi, 3G, 4G/LTE, or 5G network). The memory  406  may include a storage medium configured to store machine-readable instructions that, when executed, cause the processor  402  to perform one or more of the functions described herein. Examples of the machine-readable medium may include volatile or non-volatile memory including but not limited to semiconductor memory (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)), flash memory, and/or the like. The mass storage device  408  may include one or more magnetic disks such as one or more internal hard disks, one or more removable disks, one or more magneto-optical disks, one or more CD-ROM or DVD-ROM disks, etc., on which instructions and/or data may be stored to facilitate the operation of the processor  402 . The input device  410  may include a keyboard, a mouse, a voice-controlled input device, a touch sensitive input device (e.g., a touch screen), and/or the like for receiving user inputs to the neural network system  400 . 
     It should be noted that the neural network system  400  may operate as a standalone device or may be connected (e.g., networked or clustered) with other computation devices to perform the functions described herein. And even though only one instance of each component is shown in  FIG.  4   , a skilled person in the art will understand that the neural network system  400  may include multiple instances of one or more of the components shown in the figure. Furthermore, although the examples are described herein with reference to various types of neural networks, various types of layers, and/or various tasks being performed by certain types of neural networks or layers, those references are made merely for illustration purposes and not meant to limit the scope of the disclosure. In addition, the operation of the example neural network system is depicted and described herein with a specific order. It should be appreciated, however, that these operations may occur in various orders, concurrently, and/or with other operations not presented or described herein. And not all operations that the neural network system is capable of performing are depicted and described herein, and not all illustrated operations are required to be performed by the system. 
     For simplicity of explanation, the operations of the neural network system may have been depicted and described with a specific order. It should be appreciated, however, that these operations may occur in various orders, concurrently, and/or with other operations not presented or described herein. Furthermore, it should be noted that not all operations that the neural network system is capable of performing are depicted and described herein. It should also be noted that not all illustrated operations may be required to be performed by the neural network system. 
     While this disclosure has been described in terms of certain embodiments and generally associated methods, alterations and permutations of the embodiments and methods will be apparent to those skilled in the art. Accordingly, the above description of example embodiments does not constrain this disclosure. Other changes, substitutions, and alterations are also possible without departing from the spirit and scope of this disclosure. In addition, unless specifically stated otherwise, discussions utilizing terms such as “analyzing,” “determining,” “enabling,” “identifying,” “modifying” or the like, refer to the actions and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (e.g., electronic) quantities within the computer system&#39;s registers and memories into other data represented as physical quantities within the computer system memories or other such information storage, transmission or display devices. 
     It is to be understood that the above description is intended to be illustrative, and not restrictive. Many other implementations will be apparent to those of skill in the art upon reading and understanding the above description. The scope of the disclosure should, therefore, be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.