MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images

1 Introduction . Representing clothed humans as neural implicit functions is a rising research topic in the computer vision community . Earlier works in this direction address geometric reconstruction of clothed humans from static monocular images [ 35 , 36 , 63 , 64 ] , RGBD videos [ 37 , 38 , 71 , 78 , 80 ] or sparse point clouds [ 12 ] as direct extensions of neural implicit functions for rigid objects [ 11 , 45 , 46 , 52 ] . More recent works advocate to learn shapes in a canonical pose [ 7 , 27 , 75 ] in order to not only handle 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . reconstruction , but also build controllable neural avatars from sensor inputs . However , these works do not model pose-dependent cloth deformation , limiting their realism . On the other hand , traditional parametric human body models [ 41 , 50 , 54 , 77 ] can represent pose-dependent soft tissue deformations of minimally-clothed human bodies . Several recent methods [ 13 , 48 ] proposed to learn neural implicit functions to approximate such parametric models from watertight meshes . However , they can not be straightforwardly extended to model clothed humans . SCANimate [ 65 ] proposed to learn canonicalized dynamic neural Signed Distance Fields ( SDFs ) controlled by human pose inputs and trained with Implicit Geometric Regularization ( IGR [ 21 ] ) , thus circumventing the requirement of watertight meshes . However , SCANimate works only on dense full-body scans with accurate surface normals and further requires expensive per-subject/cloth-type training . These factors limit the applicability of SCANimate for building personalized human avatars from commodity RGBD sensors . Contrary to all the aforementioned works , we propose to use meta-learning to effectively incorporate priors of dynamic neural SDFs of clothed humans , thus enabling fast fine-tuning ( few minutes ) for generating new avatars given only a few monocular depth images of unseen clothed humans as inputs . More specifically , we build upon recently proposed ideas of meta-learned initialization for implicit representations [ 67 , 72 ] to enable fast fine-tuning . Similar to [ 67 ] , we represent a specific category of objects ( in our case , clothed human bodies in the canonical pose ) with a neural implicit function and use meta-learning algorithms such as [ 16 , 49 ] to learn a meta-model . However , unlike [ 67 , 72 ] , where the implicit functions are designed for static reconstruction , we target the generation of dynamic neural SDFs that are controllable by user-specified body poses . We observe that directly conditioning neural implicit functions ( represented as a multi-layer perceptron ) on body poses lacks the expressiveness to capture high-frequency details of diverse cloth types , and hence propose to meta-learn a hypernetwork [ 25 ] that predicts the parameters of the neural implicit function . Overall , the proposed approach , which we name MetaAvatar , yields controllable neural SDFs with dynamic surfaces in minutes via fast fine-tuning , given only a few depth observations of an unseen clothed human and the underlying SMPL [ 41 ] fittings ( Fig . 1 ) as inputs . Code and data are public at https : //neuralbodies.github.io/metavatar/ . 2 Related Work . Our approach lies at the intersection of clothed human body modeling , neural implicit representations , and meta-learning . We review related works in the following . Clothed Human Body Modeling : Earlier works for clothed human body modeling utilize parametric human body models [ 5 , 26 , 28 , 41 , 50 , 54 , 77 ] combined with deformation layers [ 2 , 3 , 7 , 8 ] to model cloth deformations . However , these approaches can not model fine clothing details due to their fixed topology , and they can not handle pose-dependent cloth deformations . Mesh-based approaches that handle articulated deformations of clothes either require accurate surface registration [ 33 , 43 , 79 , 83 ] or synthetic data [ 22 , 24 , 53 ] for training . Such requirement for data can be freed by using neural implicit surfaces [ 10 , 51 , 65 , 73 ] . For example , SCANimate [ 65 ] proposed a weakly supervised approach to learn dynamic clothed human body models from 3D full-body scans which only requires minimally-clothed body registration . However , its training process usually takes one day for each subject/cloth-type combination and requires accurate surface normal information extracted from dense scans . Recent explicit clothed human models [ 9 , 42 , 44 , 81 ] can also be learned from unregistered data . Like our method , concurrent work [ 44 ] also models pose-dependent shapes across different subjects/cloth-types , but it requires full-body scans for training . In contrast , our approach enables learning of clothed body models in minutes from as few as 8 depth images . Neural Implicit Representations : Neural implicit representations [ 11 , 45 , 46 , 52 , 55 ] have been used to tackle both image-based [ 27 , 35 , 36 , 57 , 59 , 63 , 64 , 87 ] and point cloud-based [ 7 , 12 ] clothed human reconstruction . Among these works , ARCH [ 27 ] was the first one to represent clothed human bodies as a neural implicit function in a canonical pose . However , ARCH does not handle posedependent cloth deformations . Most recently , SCANimate [ 65 ] proposed to condition neural implicit functions on joint-rotation vectors ( in the form of unit quaternions ) , such that the canonicalized shapes of the neural avatars change according to the joint angles of the human body , thus representing pose-dependent cloth deformations . However , diverse and complex cloth deformations make it hard to learn a unified prior from different body shapes and cloth types , thus SCANimate resorts to per-subject/cloth-type training which is computationally expensive . Meta-Learning : Meta-learning is typically used to address few-shot learning , where a few training examples of a new task are given , and the model is required to learn from these examples to achieve good performance on the new task [ 1 , 14 , 15 , 17–19 , 23 , 29 , 32 , 58 , 60 , 62 , 66 , 70 , 74 , 76 , 82 , 86 ] . We focus on optimization-based meta-learning , where Model-Agnostic Meta Learning ( MAML [ 16 ] ) , Reptile [ 49 ] and related alternatives are typically used to learn such models [ 4 , 6 , 20 , 34 , 39 , 61 ] . In general , this line of algorithms tries to learn a `` meta-model '' that can be updated quickly from new observations with only few gradient steps . Recently , meta-learning has been used to learn a universal initialization of implicit representations for static neural SDFs [ 67 ] and radiance fields [ 72 ] . MetaSDF [ 67 ] demonstrates that only a few gradient update steps are needed to achieve comparable or better results than slower auto-decoder-based approaches [ 52 ] . However , [ 67 , 72 ] only meta-learn static representations , whereas we are interested in dynamic representations conditioned on human body poses . To our best knowledge , we are the first to meta-learn the hypernetwork to generate the parameters of neural SDF networks . 3 Fundamentals . We start by briefly reviewing the linear blend skinning ( LBS ) method [ 41 ] and the recent implicit skinning networks [ 48 , 65 ] that learn to predict skinning weights of cloth surfaces in a weakly supervised manner . Using the learned implicit skinning networks allows us to canonicalize meshes or depth observations of clothed humans , given only minimally-clothed human body model registrations to the meshes . Canonicalization of meshes or points is a necessary step as the dynamic neural SDFs introduced in Section 4 are modeled in canonical space . 3.1 Linear Blend Skinning . Linear blend skinning ( LBS ) is a commonly used technique to deform parametric human body models [ 5 , 26 , 41 , 50 , 54 , 77 ] according to user-specified rigid bone transformations . Given a set of N points in a canonical space , X̂ = { x̂ ( i ) } Ni=1 , LBS takes a set of rigid bone transformations ( in our case we use 23 local transformations plus one global transformation , assuming an underlying SMPL model ) { Bb } 24b=1 as inputs , each Bb being a 4× 4 rotation-translation matrix . For a 3D point x̂ ( i ) ∈ X̂ 1 , a skinning weight vector is a probability simplex w ( i ) ∈ [ 0 , 1 ] 24 , s.t . ∑24 b=1 w ( i ) b = 1 , that defines the affinity of the point x̂ ( i ) to each of the bone transformations { Bb } 24b=1 . The set of transformed points X = { x ( i ) } Ni=1 of the clothed human is related to X̂ via : x ( i ) = LBS ( x̂ ( i ) , { Bb } , w ( i ) ) = ( 24∑ b=1 w ( i ) b Bb ) x̂ ( i ) , ∀i = 1 , . . . , N ( 1 ) x̂ ( i ) = LBS−1 ( x ( i ) , { Bb } , w ( i ) ) = ( 24∑ b=1 w ( i ) b Bb ) −1 x ( i ) , ∀i = 1 , . . . , N ( 2 ) 1with slight abuse of notation , we also use x̂ to represent points in homogeneous coordinates when necessary . where Eq . ( 1 ) is referred to as the LBS function and Eq . ( 2 ) is referred to as the inverse-LBS function . The process of applying Eq . ( 1 ) to all points in X̂ is often referred to as forward skinning while the process of applying Eq . ( 2 ) is referred to as inverse skinning . 3.2 Implicit Skinning Networks . Recent articulated implicit representations [ 48 , 65 ] have proposed to learn functions that predict the forward/inverse skinning weights for arbitrary points in R3 . We follow this approach , but take advantage of a convolutional point-cloud encoder [ 56 ] for improved generalization . Formally , we define the implicit forward and inverse skinning networks as hfwd ( · , · ) : ( R3×K , R3 ) 7→ R24 and hinv ( · , · ) : ( R3×K , R3 ) 7→ R24 , respectively . Both networks take as input a point cloud with K points and a query point for which they predict skinning weights . Therefore , we can analogously re-define Eq . ( 1 , 2 ) respectively as : x ( i ) = ( 24∑ b=1 hfwd ( X̂ , x̂ ( i ) ) bBb ) x̂ ( i ) , ∀i = 1 , . . . , N ( 3 ) x̂ ( i ) = ( 24∑ b=1 hinv ( X , x ( i ) ) bBb ) −1 x ( i ) , ∀i = 1 , . . . , N ( 4 ) Training the Skinning Network : We follow the setting of SCANimate [ 65 ] , where a dataset of observed point clouds { X } and their underlying SMPL registration are known . For a sample X in the dataset , we first define the re-projected points X̄ = { x̄ } Ni=1 as X mapped to canonical space via Eq . ( 4 ) and then mapped back to transformed space via Eq . ( 3 ) . We then define the training loss : L ( X ) = λrLr + λsLs + λskinLskin , ( 5 ) where Lr represents a re-projection loss that penalizes the L2 distance between an input point x and the re-projected point x̄ , Ls represents L1 distances between the predicted forward skinning weights and inverse skinning weights , and Lskin represents the L1 distances between the predicted ( forward and inverse ) skinning weights and the barycentrically interpolated skinning weights w ( i ) on the registered SMPL shape that is closest to point x ( i ) ; please refer to the Supp . Mat . for hyperparameters and details . We train two skinning network types , the first one takes a partial point cloud extracted from a depth image as input and performs the inverse skinning , while the second one takes a full point cloud sampled from iso-surface points generated from the dynamic neural SDF in the canonical space and performs forward skinning . Canonicalization : We use the learned inverse skinning network to canonicalize complete or partial point clouds { X̂ } via Eq . ( 4 ) which are further used to learn the canonicalized dynamic neural SDFs .

Summary: This paper looks at learning animatable avatars given a small number of depth images with associated known SMPL pose transformations. The proposed approach (similar to prior work e.g. SCANimate) learns: a) forward/inverse skinning weight predictors,  which given a query point x in canonical/posed space respectively predict the forward/inverse LBS weights, and b) a canonical space implicit SDF function, which conditioned on SMPL pose, predicts a pose-dependent shape s.t. its forward transformation looks correct.  While the paper uses a fixed network for a) above, the key contributions relate to how the pose-dependent canonical shape is modeled. This paper proposes to capture each object via a hyper-network which predicts the weights of the SDF function conditioned on SMPL pose parameters. It shows that such a hypernetwork can be meta-learned using training data with strong supervision, but can then be finetuned using only a small number of depth images in varying poses for a novel instance.  The experiments convincingly show the benefits of this approach across different datasets and in particular show how using only a small number of depth images suffice.