MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images

1 Introduction . Representing clothed humans as neural implicit functions is a rising research topic in the computer vision community . Earlier works in this direction address geometric reconstruction of clothed humans from static monocular images [ 35 , 36 , 63 , 64 ] , RGBD videos [ 37 , 38 , 71 , 78 , 80 ] or sparse point clouds [ 12 ] as direct extensions of neural implicit functions for rigid objects [ 11 , 45 , 46 , 52 ] . More recent works advocate to learn shapes in a canonical pose [ 7 , 27 , 75 ] in order to not only handle 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . reconstruction , but also build controllable neural avatars from sensor inputs . However , these works do not model pose-dependent cloth deformation , limiting their realism . On the other hand , traditional parametric human body models [ 41 , 50 , 54 , 77 ] can represent pose-dependent soft tissue deformations of minimally-clothed human bodies . Several recent methods [ 13 , 48 ] proposed to learn neural implicit functions to approximate such parametric models from watertight meshes . However , they can not be straightforwardly extended to model clothed humans . SCANimate [ 65 ] proposed to learn canonicalized dynamic neural Signed Distance Fields ( SDFs ) controlled by human pose inputs and trained with Implicit Geometric Regularization ( IGR [ 21 ] ) , thus circumventing the requirement of watertight meshes . However , SCANimate works only on dense full-body scans with accurate surface normals and further requires expensive per-subject/cloth-type training . These factors limit the applicability of SCANimate for building personalized human avatars from commodity RGBD sensors . Contrary to all the aforementioned works , we propose to use meta-learning to effectively incorporate priors of dynamic neural SDFs of clothed humans , thus enabling fast fine-tuning ( few minutes ) for generating new avatars given only a few monocular depth images of unseen clothed humans as inputs . More specifically , we build upon recently proposed ideas of meta-learned initialization for implicit representations [ 67 , 72 ] to enable fast fine-tuning . Similar to [ 67 ] , we represent a specific category of objects ( in our case , clothed human bodies in the canonical pose ) with a neural implicit function and use meta-learning algorithms such as [ 16 , 49 ] to learn a meta-model . However , unlike [ 67 , 72 ] , where the implicit functions are designed for static reconstruction , we target the generation of dynamic neural SDFs that are controllable by user-specified body poses . We observe that directly conditioning neural implicit functions ( represented as a multi-layer perceptron ) on body poses lacks the expressiveness to capture high-frequency details of diverse cloth types , and hence propose to meta-learn a hypernetwork [ 25 ] that predicts the parameters of the neural implicit function . Overall , the proposed approach , which we name MetaAvatar , yields controllable neural SDFs with dynamic surfaces in minutes via fast fine-tuning , given only a few depth observations of an unseen clothed human and the underlying SMPL [ 41 ] fittings ( Fig . 1 ) as inputs . Code and data are public at https : //neuralbodies.github.io/metavatar/ . 2 Related Work . Our approach lies at the intersection of clothed human body modeling , neural implicit representations , and meta-learning . We review related works in the following . Clothed Human Body Modeling : Earlier works for clothed human body modeling utilize parametric human body models [ 5 , 26 , 28 , 41 , 50 , 54 , 77 ] combined with deformation layers [ 2 , 3 , 7 , 8 ] to model cloth deformations . However , these approaches can not model fine clothing details due to their fixed topology , and they can not handle pose-dependent cloth deformations . Mesh-based approaches that handle articulated deformations of clothes either require accurate surface registration [ 33 , 43 , 79 , 83 ] or synthetic data [ 22 , 24 , 53 ] for training . Such requirement for data can be freed by using neural implicit surfaces [ 10 , 51 , 65 , 73 ] . For example , SCANimate [ 65 ] proposed a weakly supervised approach to learn dynamic clothed human body models from 3D full-body scans which only requires minimally-clothed body registration . However , its training process usually takes one day for each subject/cloth-type combination and requires accurate surface normal information extracted from dense scans . Recent explicit clothed human models [ 9 , 42 , 44 , 81 ] can also be learned from unregistered data . Like our method , concurrent work [ 44 ] also models pose-dependent shapes across different subjects/cloth-types , but it requires full-body scans for training . In contrast , our approach enables learning of clothed body models in minutes from as few as 8 depth images . Neural Implicit Representations : Neural implicit representations [ 11 , 45 , 46 , 52 , 55 ] have been used to tackle both image-based [ 27 , 35 , 36 , 57 , 59 , 63 , 64 , 87 ] and point cloud-based [ 7 , 12 ] clothed human reconstruction . Among these works , ARCH [ 27 ] was the first one to represent clothed human bodies as a neural implicit function in a canonical pose . However , ARCH does not handle posedependent cloth deformations . Most recently , SCANimate [ 65 ] proposed to condition neural implicit functions on joint-rotation vectors ( in the form of unit quaternions ) , such that the canonicalized shapes of the neural avatars change according to the joint angles of the human body , thus representing pose-dependent cloth deformations . However , diverse and complex cloth deformations make it hard to learn a unified prior from different body shapes and cloth types , thus SCANimate resorts to per-subject/cloth-type training which is computationally expensive . Meta-Learning : Meta-learning is typically used to address few-shot learning , where a few training examples of a new task are given , and the model is required to learn from these examples to achieve good performance on the new task [ 1 , 14 , 15 , 17–19 , 23 , 29 , 32 , 58 , 60 , 62 , 66 , 70 , 74 , 76 , 82 , 86 ] . We focus on optimization-based meta-learning , where Model-Agnostic Meta Learning ( MAML [ 16 ] ) , Reptile [ 49 ] and related alternatives are typically used to learn such models [ 4 , 6 , 20 , 34 , 39 , 61 ] . In general , this line of algorithms tries to learn a `` meta-model '' that can be updated quickly from new observations with only few gradient steps . Recently , meta-learning has been used to learn a universal initialization of implicit representations for static neural SDFs [ 67 ] and radiance fields [ 72 ] . MetaSDF [ 67 ] demonstrates that only a few gradient update steps are needed to achieve comparable or better results than slower auto-decoder-based approaches [ 52 ] . However , [ 67 , 72 ] only meta-learn static representations , whereas we are interested in dynamic representations conditioned on human body poses . To our best knowledge , we are the first to meta-learn the hypernetwork to generate the parameters of neural SDF networks . 3 Fundamentals . We start by briefly reviewing the linear blend skinning ( LBS ) method [ 41 ] and the recent implicit skinning networks [ 48 , 65 ] that learn to predict skinning weights of cloth surfaces in a weakly supervised manner . Using the learned implicit skinning networks allows us to canonicalize meshes or depth observations of clothed humans , given only minimally-clothed human body model registrations to the meshes . Canonicalization of meshes or points is a necessary step as the dynamic neural SDFs introduced in Section 4 are modeled in canonical space . 3.1 Linear Blend Skinning . Linear blend skinning ( LBS ) is a commonly used technique to deform parametric human body models [ 5 , 26 , 41 , 50 , 54 , 77 ] according to user-specified rigid bone transformations . Given a set of N points in a canonical space , X̂ = { x̂ ( i ) } Ni=1 , LBS takes a set of rigid bone transformations ( in our case we use 23 local transformations plus one global transformation , assuming an underlying SMPL model ) { Bb } 24b=1 as inputs , each Bb being a 4× 4 rotation-translation matrix . For a 3D point x̂ ( i ) ∈ X̂ 1 , a skinning weight vector is a probability simplex w ( i ) ∈ [ 0 , 1 ] 24 , s.t . ∑24 b=1 w ( i ) b = 1 , that defines the affinity of the point x̂ ( i ) to each of the bone transformations { Bb } 24b=1 . The set of transformed points X = { x ( i ) } Ni=1 of the clothed human is related to X̂ via : x ( i ) = LBS ( x̂ ( i ) , { Bb } , w ( i ) ) = ( 24∑ b=1 w ( i ) b Bb ) x̂ ( i ) , ∀i = 1 , . . . , N ( 1 ) x̂ ( i ) = LBS−1 ( x ( i ) , { Bb } , w ( i ) ) = ( 24∑ b=1 w ( i ) b Bb ) −1 x ( i ) , ∀i = 1 , . . . , N ( 2 ) 1with slight abuse of notation , we also use x̂ to represent points in homogeneous coordinates when necessary . where Eq . ( 1 ) is referred to as the LBS function and Eq . ( 2 ) is referred to as the inverse-LBS function . The process of applying Eq . ( 1 ) to all points in X̂ is often referred to as forward skinning while the process of applying Eq . ( 2 ) is referred to as inverse skinning . 3.2 Implicit Skinning Networks . Recent articulated implicit representations [ 48 , 65 ] have proposed to learn functions that predict the forward/inverse skinning weights for arbitrary points in R3 . We follow this approach , but take advantage of a convolutional point-cloud encoder [ 56 ] for improved generalization . Formally , we define the implicit forward and inverse skinning networks as hfwd ( · , · ) : ( R3×K , R3 ) 7→ R24 and hinv ( · , · ) : ( R3×K , R3 ) 7→ R24 , respectively . Both networks take as input a point cloud with K points and a query point for which they predict skinning weights . Therefore , we can analogously re-define Eq . ( 1 , 2 ) respectively as : x ( i ) = ( 24∑ b=1 hfwd ( X̂ , x̂ ( i ) ) bBb ) x̂ ( i ) , ∀i = 1 , . . . , N ( 3 ) x̂ ( i ) = ( 24∑ b=1 hinv ( X , x ( i ) ) bBb ) −1 x ( i ) , ∀i = 1 , . . . , N ( 4 ) Training the Skinning Network : We follow the setting of SCANimate [ 65 ] , where a dataset of observed point clouds { X } and their underlying SMPL registration are known . For a sample X in the dataset , we first define the re-projected points X̄ = { x̄ } Ni=1 as X mapped to canonical space via Eq . ( 4 ) and then mapped back to transformed space via Eq . ( 3 ) . We then define the training loss : L ( X ) = λrLr + λsLs + λskinLskin , ( 5 ) where Lr represents a re-projection loss that penalizes the L2 distance between an input point x and the re-projected point x̄ , Ls represents L1 distances between the predicted forward skinning weights and inverse skinning weights , and Lskin represents the L1 distances between the predicted ( forward and inverse ) skinning weights and the barycentrically interpolated skinning weights w ( i ) on the registered SMPL shape that is closest to point x ( i ) ; please refer to the Supp . Mat . for hyperparameters and details . We train two skinning network types , the first one takes a partial point cloud extracted from a depth image as input and performs the inverse skinning , while the second one takes a full point cloud sampled from iso-surface points generated from the dynamic neural SDF in the canonical space and performs forward skinning . Canonicalization : We use the learned inverse skinning network to canonicalize complete or partial point clouds { X̂ } via Eq . ( 4 ) which are further used to learn the canonicalized dynamic neural SDFs .

Summary: This paper presents a framework that can learn an animatable implicit avatar for clothed humans from very few observations. The key idea is using meta-learning to learn a prior (specifically speaking, a good initialization) to enable fast fine-tuning at inference time. To enable animations of various cloth types, this paper proposes to perform meta-learning on a hyper-network which controls the parameters of the neural implicit function conditioned on pose configurations. Results show that the proposed method is able to generate clothed human avatars from few shot observations.