MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images
Authors: Shaofei Wang
Marko Mihajlovic
Qianli Ma
Andreas Geiger
Siyu Tang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Neural Implict Functions
3D Computer Vision
Clothed Human Modeling
Meta-Learning
Cloth Modeling
HyperNetworks

Abstract: In this paper, we aim to create generalizable and controllable neural signed distance fields (SDFs) that represent clothed humans from monocular depth observations. Recent advances in deep learning, especially neural implicit representations, have enabled human shape reconstruction and controllable avatar generation from different sensor inputs. However, to generate realistic cloth deformations from novel input poses, watertight meshes or dense full-body scans are usually needed as inputs. Furthermore, due to the difficulty of effectively modeling pose-dependent cloth deformations for diverse body shapes and cloth types, existing approaches resort to per-subject/cloth-type optimization from scratch, which is computationally expensive. In contrast, we propose an approach that can quickly generate realistic clothed human avatars, represented as controllable neural SDFs, given only monocular depth images. We achieve this by using meta-learning to learn an initialization of a hypernetwork that predicts the parameters of neural SDFs. The hypernetwork is conditioned on human poses and represents a clothed neural avatar that deforms non-rigidly according to the input poses. Meanwhile, it is meta-learned to effectively incorporate priors of diverse body shapes and cloth types and thus can be much faster to fine-tune, compared to models trained from scratch. We qualitatively and quantitatively show that our approach outperforms state-of-the-art approaches that require complete meshes as inputs while our approach requires only depth frames as inputs and runs orders of magnitudes faster. Furthermore, we demonstrate that our meta-learned hypernetwork is very robust, being the first to generate avatars with realistic dynamic cloth deformations given as few as 8 monocular depth frames.

Learning Animatable Clothed Human Models from Few Depth Images
MetaAvatar:
Shaofei Wang
Marko Mihajlovic' marko. marko.miha www.amaolviiiiif...mmmmm wtw..eeeeehh..c.... ch
shaofei.. wang@inf. ethz.
Qianli Ma qianli. Claani.ma@tual ma@tueu mpg......... de
Andreas Geiger
a. gei.gegrrrmiituuiiieeee de
Siyu Tang'
siyu. tange @nf .et...h.cc. ch
'ETH Zürich
Max Planck Institute for Intelligent Systems, Tübingen
University of Tübingen
Abstract
In this paper, we aim to create generalizable and controllable neural signed distance fields (SDFs) that represent clothed humans from monocular depth observations. Recent advances in deep learning, especially neural implicit representations, have enabled human shape reconstruction and controllable avatar generation from dif- ferent sensor inputs. However, to generate realistic cloth deformations from novel input poses, watertight meshes dense full-body scans are usually needed inputs. Furthermore, due to the difficulty of effectively modeling pose-dependent cloth deformations for diverse body shapes and cloth types, existing approaches resort to per wuwjsct/cloott-yyeeee
optimization from scratch, which is computation- ally expensive. In contrast, we propose an approach that can quickly generate realistic clothed human avatars, represented as controllable neural SDFs, given only monocular depth images. We achieve this by using meta-learning to learn initialization of a hypernetwork that predicts the parameters of neural SDFs. The hypernetwork conditioned on human poses and represents a clothed neural avatar that deforms non-rigidly according the input poses. Mcanwhile, it is meta- learned to effectively incorporate priors of diverse body shapes and cloth types and thus can be much faster to fine-tune, compared models trained from scratch. We qualitatively and quantitatively show that our approach outperforms state-of- oftte--f-hh-arr approaches that require complete meshes inputs while approach requires only depth frames as inputs and runs orders of magnitudes faster. Furthermore, demonstrate that our meta-leamee hypernetwork is very robust. being the first to generate avatars with realistic dynamic cloth deformations given as few as 8 monocular depth frames.
Introduction
Representing clothed humans as neural implicit functions is rising research topic in the computer vision community. Earlier works in this direction address geometric reconstruction of clothed humans from static monocular images [35, 36, 63, 64], RGBD videos [37, 38, 71, 78, 80] or sparse point clouds [12] as direct extensions neural implicit functions for rigid objects [1 45, 46, 52]. More recent works advocate to learn shapes in a canonical pose [7, 27, 75] in order to not only handle
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Clufl
Clothed
Given as few as 8 monocular depth images and their SMPL fittings, our meta-learned model yields a controllable neural SDF in 2 minutes which synthesizes realistic cloth deformations for unseen body poses. Here we show results of two different subjects wearing different clothes
Figure 1:
reconstruction, but also build controllable neural avatars from sensor inputs. However, these works do not model pose-dependent cloth deformation, limiting their realism.
On the other hand, traditional parametric human body models [41, 50, 54, 77] can represent pose-dependent soft tissue deformations of minimally-clothed human bodies. Several recent meth- ods [13, 48] proposed to learn neural implicit functions to approximate such parametric models from watertight meshes. However, they cannot be straightforwardly extended model clothed humans. SCANimate [65] proposed to learn canonicalized dynamic neural Signed Distance Fields (SDFs) controlled by human pose inputs and trained with Implicit Geometric Regularization (IGR [21]. thus circumventing the requirement of watertight meshes. However, SCANimate works only on dense full-body scans with accurate surface normals and further requires expensive per-subject/cloth-yye training. These factors limit the applicability of SCANimate for building personalized human avatars from commodity RGBD sensors.
Contrary to all the aforementioned works, we propose to use meta-learning to effectively incorporate priors of dynamic neural SDFs of clothed humans, thus enabling fast fine-tuning (few minutes) for generating new avatars given only a few monocular depth images of unseen clothed humans as inputs. More specifically, we build upon recently proposed ideas of meta-lcarned initialization for implicit representations [67. 72] to enable fast fine-tuning. Similar to [67], we represent a specific category of objects (in our case, clothed human bodies in the canonical pose) with a neural implicit function and use meta-learning algorithms such as [16, 49] to learn a meta-model. However, unlike [67, 72], where the implicit functions are designed for static reconstruction, we target the generation of dynamic neural SDFs that are controllable by user-specified body poses. We observe that directly conditioning neural implicit functions (represented as multi-layer perceptron) on body poses lacks the expressiveness to capture high-frequency details of diverse cloth types, and hence propose to meta-learn a hypernetwork [25] that predicts the parameters of the neural implicit function. Overall, the proposed approach, which we name MetaAvatar, yields controllable neural SDFs with dynamic surfaces in minutes via fast fine-tuning, given only a few depth observations of an unseen clothed human and the underlying SMPL [41] fittings (Fig. 1) as inputs. Code and data are public at http://nneuralbodiss. github. mo
2
Related Work
Our approach lies at the intersection of clothed human body modeling, neural implicit representations, and meta-learning We review related works in the following.
Clothed Human Body Modeling: Earlier works for clothed human body modeling utilize parametric human body models [5, 26, 28, 41, 50, 54, 77] combined with deformation layers [2, 3, 7, 8] to model cloth deformations. However, these approaches cannot model fine clothing details due to their fixed topology, and they cannot handle pose-dependent cloth deformations. Mesh-based approaches that handle articulated deformations of clothes either require accurate surface registration [33, 43, 79, 83] or synthetic data [22, 24, 53] for training. Such requirement for data can be freed by using neural implicit surfaces [10, 51, 65, 73]. For example, SCANimate [65] proposed a weakly supervised approach to learn dynamic clothed human body models from 3D full-body scans which only requires minimally-clothed body registration. However, its training process usually takes one day for each subject/cloth-tye combination and requires accurate surface normal information extracted from dense scans. Recent explicit clothed human models [9, 42, 44, 81] can also be learned from unregistered data. Like our method, concurrent work [44] also models desendent shapes across different

---

subjects/cloth-tyee but it requires full-body scans for training. In contrast, our approach enables learning of clothed body models in minutes from as few depth images.
Neural Implicit Representations: Neural implicit representations [11, 45, 46, 52, 55] have been used to tackle both image-based [27, 35, 36, 57, 59, 63, 64, 87] and point cloud- [7, 12] clothed human reconstruction. Among these works, ARCH [27] was the first one to represent clothed human bodies neural implicit function in canonical pose. However, ARCH does not handle pose- dependent cloth deformations. Most recently, SCANimate [65] proposed to condition neural implicit functions joint-rotation vectors (in the form of unit quaternions), such that the canonicalized shapes of the neural avatars change according to the joint angles of the human body, thus representing pose-dependent cloth deformations. However, diverse and complex cloth deformations make hard to learn a unified prior from different body shapes and cloth types, thus SCANimate resorts to per-ssubject/clot--yye training which is computationally expensive.
Meta-Learning: Meta-learning typically used to address few-shot learning, where few training examples of new task are given, and the model is required to learn from these examples to achieve
good performance on the new task [1. 14, 15, 17-19, 23, 29, 32, 58, 60, 62, 66, 70, 74, 76, 82, 86]. meta-learning, where Model-Agnostic Meta Learning (MAML [16]),
We focus optimization-base Reptile [49] and related alternatives are typically used to learn such models [4, 6, 20, 34, 39, 61]. In general, this line of algorithms tries to learn a "meta-model" that can be updated quickly from new observations with only few gradient steps. Recently, meta-lcarning has been used to learn a universal initialization of implicit representations for static neural SDFs [67] and radiance fields [72]. MetaSDF [67] demonstrates that only a few gradient update steps are needed to achieve comparable or better results than slower auto-decoder-basee approaches [52]. However, [67, 72] only meta-learn static representations, whereas we are interested in dynamic representations conditioned on human body poses. To our best knowledge, we the first to meta-learn the hypernetwork generate the parameters of neural SDF networks.
3 Fundamentals
We start by briefly reviewing the linear blend skinning (LBS) method [41] and the recent implicit skinning networks [48, 65] that learn to predict skinning weights of cloth surfaces in a weakly supervised manner. Using the learned implicit skinning networks allows us to canonicalize meshes or depth observations of clothed humans, given only minimally-clothed human body model registrations to the meshes. Canonicalization of meshes or points is a necessary step as the dynamic neural SDFs introduced in Section 4 are modeled in canonical space.
3.1 Linear Blend Skinning
Linear blend skinning (LBS) is commonly used technique to deform parametric human body models [5, 26, 41, 50, 54, 77] according to user-specified rigid bone transformations. Given a set of N points in a canonical space, x = (x()) I. LBS takes a set of rigid bone transformations (in our case we use 23 local transformations plus one global transformation, assuming an underlying SMPL model) [B, 24 6= as inputs, each Bg being a 4 x 4 rotation-translation matrix. For a 3D point
( X
skinning weight vector is a probability simplex w(() E [0, 1]24, s.t. Σ w/ 124
(i)
that defines the affinity of the point ( to each of the bone transformations [Bo] b=1- The set of
transformed points X = [ of the clothed human is related to X via:
)
x LBS (Bb), w(())
Σ
B
(()
Vi= LI
(1)
1 LBS-1 (x((), (x)S),)((,(BB)))))) (Bl))..())
wo (6)
Bb
x(0
=i= 1I .
(2)
with slight abuse of notation, we also x represent points in homogeneous coordinates when necessary.
3

---

where Eq. (1) is referred to as the LBS function and Eq. (2) is referred to as the inverse- LBS function. The process of applying Eq. (1) to all points in X is often referred to as forward skinning while the process of applying Eq. (2) is referred to as inverse skinning.
3.2 Implicit Skinning Networks
Recent articulated implicit representations [48, 65] have proposed to learn functions that predict the forward/inverse skinning weights for arbitrary points in IR We follow this approach, but take advantage of a convolutional point-cloud encoder [56] for improved generalization, Formally, we define the implicit forward and inverse skinning networks as A(wa(::)) (IR TR,3* IR$) + R24 and
hime( (:n -) (R5x K R + R respectively. Both networks take as input point cloud with K points and query point for which they predict skinning weights. Therefore, we can analogously re-define Eq. (1, 2) respectively as:
x(1
Vi= 1, I,...,N N hrwd X (1))BB) x((),
(3)
x0 hime (X, xx,x
x(() Vi 1I..,...,
(4)
Training the Skinning Network: We follow the setting of SCANimate [65], where a dataset of
observed point clouds [X} and their underlying SMPL registration are known. For sample X in the dataset, we first define the re-projected points X { as X mapped canonical space via Eq. (4) and then mapped back to transformed space via Eq. (3). We then define the training loss:
((X) = XrCr + AgCo + Agkin Cakin
(5)
where C, represents a re-projection loss that penalizes the L2 distance between an input point x and the re-projected point x, L, represents LI distances between the predicted forward skinning weights and inverse skinning weights, and Lskin represents the I.1 distances between the predicted (forward and inverse) skinning weights and the barycentrically interpolated skinning weights w on the registered SMPL shape that is closest to point (+); please refer to the Supp. Mat. for hyperparameters and details.
We train two skinning network types, the first one takes partial point cloud extracted from a depth image as input and performs the inverse skinning, while the second one takes a full point cloud sampled from sur-seee points generated from the dynamic neural SDF in the canonical space and performs forward skinning.
Canonicalization: We use the learned inverse skinning network to canonicalize complete or partial point clouds (X) via Eq. (4) which are further used to learn the canonicalized dynamic neural SDFs.
4 MetaAvatar
Our approach meta-learns unified clothing deformation prior from the training that consists of different subjects wearing different clothes. This meta-learndd model further efficiently fine-tuned to produce dynamic neural SDF from arbitrary amount of fine-tur fine-tuning data of unseen subjects. In extreme cases, AetaAvatar requires as few as 8 depth frames and takes only 2 minutes for fine-tuning to yield aubject/cloth-yypessecffff dynamic neural SDF (Fig. 1).
We assume that each subject/cloth-tyee combination in the training set has a set of registered bone transformations and canonicalized points, denoted as { (2 124 1 X). Points in are normalized to the range [-1, 1 according to their corresponding registered SMPL shape. With slight abuse of notation, we also define X as all possible points in 1|3 Our goal is to meta-learn hyper- network [25, 69] which takes [Ba]21 ((B.) for shorthand) as inputs and predicts parameters of the neural SDFs in the canonical space. Denoting the hypernetwork as gu (B,)) and the predicted

---

end for end for neural SDF as fx(x)) ((B.) we use the following IGR [21] loss to supervise the learning of g:
Σ xEX
Asdt
+ An (1 (n(x), (x
+ A
V (x)
(on-surface loss)
Σ
ty exp (-a
x~XXX
A Vxfo(x))
(off-surface loss)
(6)
where ny the surface normal of point We assume that this information, along with the ground- truth correspondences from transformed space to canonical space is available when learning the meta-model on the training set, but do not require this for fine-tuning g on unseen subjects.
In practice, we found that directly learning the hypernetwork gø via Eq. (6) does not converge, and thus decompose the meta-learning of ge into two steps. First, learn meta-SDF [67] (without conditioning on (Ba). Sec. 4.1), and then we meta-learn a hypernetwork that takes {B,} as input and predicts the residuals to the parameters of the previously learned meta-SDF (Sec. 4.2).
4.1 Meta-learned Initialization of Static Neural SDFs
To effectively learn a statistical prior of clothed human bodies, we ignore the input bone transfor- mations {B,} and meta-learn the static neural SDF fx) 1]3 R. parameterized by 6, from all canonicalized points of subjects with different genders, body shapes, cloth rypes, and poses. Furthermore, for faster and more stable convergence, the neural SDF f& function addi- tionally leverages the periodic activation func- tions [68].
Training Stago
Lear Mets-SDF
Lineer
-MP sino
Linear
-ljy-- sine
Lineer
-+pp sine
Linear
Linear
sino
for (x)
Figure 2: Overview of the meta-SDF network. We use 5-layer SIREN [67] network with 256 neurons for each layer.
The full meta-learning algorithm for the static neural SDFs is described in Alg. 1.
Algorithm Meta-learning SDF with Reptile [49]
Initialize: meta-network parameters á, meta learning rate 8, inner learning rate o, max training iteration N, inner-loop iteration m, batch size M
1: for 1,...,N do
2: Sample a batch of M training samples (X6) K
3: for j 1,..., . do
4:
@ = Φ
5: for h:
do
6:
(1)
k- a & (IIC((o(X()) (=
7:
8:
9:
o+ E σ)
end for
4.2 Meta-learned Initialization of HyperNetwork for Dynamic Neural SDFs
The meta-learned statie neural SDF explained in the previous section can efficiently adapt to new observations, however it is not controllable by user-specified bone transformations (B,). Therefore, enable non-rigid pose-dependent cloth deformations, we further meta-learn a hypernetwork [25] to predict residuals to the learned parameters of the meta-SDF in Alg. 1.

---

The key motivation for meta-learning the hypernetwork is to build an effective unified prior for articulated clothed humans, which enables the recovery of the non-rigid clothing deformations at test time via the efficient fine-tuning process from several depth images of unseen subjects.
Algorithm 2 Meta-learning hypernetwork with Modified Reptile
Initialize: veeta-hypemetworf parameters v. pre-trained meta-SDF parameters d", meta learning rate 3, inner learning rate a, max training iteration N, inner-loop iteration m. for I.....,NN do
2: 4'u =
3: Randomly choose subject/cloth--ye type combination
4: Uniformly sample . {I....,tmy D(")) where D the number of datapoints of
subjectcloth-type combination
5:
Sample M datapoints from subject/cloth-type combination n, denoting these datapoints as
S [[B,) (1) } 1==1
6:
for k 1 ....,, m do
7:
L
1 Σ (BB)) (fo (o=e ((o,))))
M
8:
1/k 1 aV Uix
9:
end for
1 3 + (1 vo)
11: end for
Denoting the meta-SDF learned by Alg. 1 as ø² and our hypernetwork g (( )) imple- ment Alg. 2. This algorithm differs from the original Reptile [49] algorithm in that it tries to optimize the inner-loop on arbitrary amount of data. Note that for brevity the loss in the inner- loop (line 7-line 8) is computed over the whole batch S, whereas practice we used stochastic gradient descent (SGD) with fixed mini batch size over S since S can contain hundreds of sam- ples; SGD is used with the mini-bate size of 12 for the inner-loop.
Training Stage 2: Leam Meta Hypernewoork
Hyperkal goll (B,)
B,,
MLP
MLP
Linear
Linear
Linear
Linear
-Wp-
Linear
[x)
10
9. (B,)
Inference: At test-time, we are given a small fine-tuning set (ine),i), fine.(i)) M Jj=1 and the validation set (Bl,) ) The fine-tuning set is used to optimize the hypernet- work parameters & (m 256 SGD epochs) that
Figure 3: Overview of the meta-hypernetwork. It predicts residuals to o² which is learned in Sec. 4.1
are then used to generate neural SDFs from bone transformations available in the validation set. The overall inference pipeline including the inverse and the forward LBS stages is shown in Fig. 4.
Bone Transformation Encoding: We found that small hierarchical MLP proposed in LEAP [48] for encoding bone transformations works slightly better than the encoding of unit quaternions used in SCANimate [65]. Thus, we employ the hierarchical MLP encoder to encode {B,} for unless specified otherwise; we ablate different encoding types in the experiment section.
5 Experiments
We validate the proposed MetaAvatar model for learning meta-models and controllable dynamic neural SDFs of clothed humans by first comparing our MetaAvatar to the established approaches [13, 48, 65]. Then, we ablate the modeling choices for the proposed controllable neural SDFs. And lastly, we demonstrate MetaAvatar's capability to tackle the challenging task of learning animatable clothed human models from reduced data, the point that only depth images are available as input.
Datasets: We use the CAPE dataset [43] as the major test bed for our experiments. This dataset consists of 148584 pairs of clothed meshes, capturing 15 human subjects wearing different clothes

---

Fine- buning
Forward LBS
LBG
Generation
[Dy]
Notekenndd
Hypolkr
Canonicalized depti points
Subjec/Clofh-yype Spacific Dynamic
Novel dlof-defoomation canonical space
depth frames
Animatedd Meehes
SDF
Figure 4: Overview of our inference pipeline. The inverse LBS net (Sec. 3.2) takes a small set of input depth frames together with their underlying SMPL registrations canonicalize the depth points; then the meta-learned hypernetwork (Sec. 4.2) is fine-tuned to represent the instance specific dynamic SDF; given novel poses, the updated hypernetoor generates pose-dependent cloth-deformations in canonical space, and the animated meshes are obtained via the forward LBS network (Sec. 3.2).
while performing different actions. We use 10 subjects for meta-learning, which we denote as the training set. We use four unseen subjects (00122, 00134, 00215, 03375)² for fine-tuning and validation; for each of these four subjects, the corresponding action sequences split into fine- tuning set and validation set. The fine-tuning set is used for fine-tuning the MetaAvatar models, is also used to evaluate pose interpolation task. The validation set is used for evaluating novel pose extrapolation. Among the four unseen subjects, two them (00122, 00215) perform actions that are present in the training set for the meta-learning: we randomly split actions of these 70% fine-tuning and 30% validation. Subject 001 00134 34 and 03375 perform two trials actions unseen in the training set for meta-learning. We use the first trial the fine-tuning set and the second trial as the validation set. Subject 03375 also has one cloth type (blazer) that is unseen during meta-lcarning.
subjects with
Baselines: We use NASA [13], LEAP [48], and SCANimate [65] as our baselines. NASA and SCANimate cannot handle multi- sitti--uubeee--lothh with single model so we train per-subject/cloth-yy models from scratch for each of them on the fine-tuning set. LEAP is generalizable neural-implicit human body model that has shown to work on minimally-clothed bodies. We extend LEAP by adding one-hot encoding to represent different cloth types (similarly to [43]) and train it jointly on the full training and the fine-tuning set.
As for the input format, we use depth frames rendered from CAPE meshes for our MetaAvatar, To render the depth frames, we fixed the camera and rotate the CAPE meshes around the y-axis (in SMPL space) at different angles with an interval of 45 degrees; note that for each mesh we only render on one angle, simulating a monocular camera taking round-vier of moving person. For the baselines, we use watertight meshes and provide the occupancy [45] loss to supervise the training of NASA and LEAP, while sampling surface points and normals on watertight meshes to provide the IGR loss supervision for SCANimate. Note that our model is at great disadvantage, as for fine-tuning we only use discrete monocular depth observations without accurate surface normal information.
Tasks and Evaluation: Our goal is to generate realistic clothing deformations from arbitrary input human pose parameters. To systematically understand the generalization capability of the MetaAvatar representation, we validate the baselines and MetaAvatar on two tasks, pose interpolation and extrapolation. For interpolation, tuning, and sample every 5th frame (excluding the training frames) also on the fine-tuning set for validation. For extrapolation, sample every 5th frame on the validation set for validation.
sample every 10th frame on the fine-tuning set for training/fine-
sample every 5th frame the fine-tuning set for training, and
Interpolation is evaluated using three metrics: point-based ground toattdd-trutttoooo mo-mesh distance (Dp J. in cm), face-based ground-truth-tor truth-to-mest distance (D, J. in cm), and point-based soound-truth-to-me normal consistency (NC' 1. in range [-1, 1J). For computing these interpolation metrics ignore non-clothed body parts such as hands, feet, head, and neck. For extrapolation, we note that cloth- deformation often stochastic; in such case, predicting overly smooth surfaces result in lower distances and higher normal consistency. Thus, we also conduct large-scale perceptual study using
We ignore subject 00159 because it has significantly less data compared to other subjects.

---

NASA
LEAP
SCANimate
STREE
Ours
10%
5%
depth images
Figure 5: Qualitative comparison extrapo- lation results with blazer outfit. NASA shows consistent blocky artifacts. LEAP predicts overly smooth surfaces missing the tails of the blazer outfit. SCANimate does not generalize as this specific pose has not been seen during training. Directly meta-learning SIREN [68] network that conditions on input poses produces a smooth surface that does not capture the blazer tails well.
Figure 6: Qualitative comparison on extrapola- tion results when reducing fine-tuning data on subject 00215 wearing poloshirt. The caption indicates the amount of fine-tuning data used to fine-tune the meta-hypernetwork on this un- seen subject. Our meta-learned model captures for this unseen subject the sliding effect of the poloshirt at this pose in which the person raising arms, even fine-tuned with just 8 depth images.
Amazon Mechanical Turk, and report the perceptual scores (PS f) which reflects the percentage of users who favor the outputs of baselines over MetaAvatar. Details about user study design can be found in the Supp. Mat.
5.1 Evaluation Against Baselines
In this section, we report results on both interpolation and extrapolation tasks against various baselines described above. For NASA and SCANimate, we train model for each subject/cloth- type combination on the fine-tuning set. For them it usually takes several thousand of epochs to converge for each subject/cloth-tye -type combination, which roughly equals to 10-24 hours of training. For LEAP, we train a single model on both the training and the fine-tuning set using two days. For the MetaAvatar, we meta-learn single model on the training set, and for each subject/clothttype combination we fine-tune the model for 256 epochs to produce subject/cloth-type specific models. The exact fine-tuning time ranges from 40 minutes to hours depending on the amount of available data since we are running a fixed number of cpochs; see the Supp. Mat. for detailed runtime comparison on each subject/cloth-yyp combination. Note that MetaAvatar uses partial depth observations while the other basclines are trained on complete meshes. The results are reported in Table 1.
Importantly, models of NASA and SCANimate are over-fitted to each subject/cloth-tyee type combination as they cannot straightforwardly leverage prior knowledge from multiple training subjects. LEAP is trained on all training and fine-tuning data with input encodings to distinguish different body shapes and cloth types, but it fails to capture high-frequency details of clothes, often predicting smooth surfaces (Fig. 5); this is evidenced by its lower perceptual scores (PS) compared to SCANimate and our Avaaar. In contrast to these baselines, Avtaav successfully captures a unified clothing deformation prior of diverse body shapes and cloth types, which generalizes well to unseen body shapes (00122, 00215), unseen poses (00134, 03375), and unseen cloth types (03375 with blazer outfit): although we did not outperform LEAP on the interpolation task for subject 00134 and 03375, we note that 1) our method uses only 2.5D input for fine-tuning, while LEAP has access to ground- truth canonical meshes during training: 2) subject 00134 and 03375 comprise much more missing frames than subject 00122 and 00215, resulting in higher stochasticity and thus predicting smooth surfaces (such as LEAP) may yield better performance; this also evidenced by LEAP's much lower perceptual scores on subject 00134 and 03375, although obtaining the best performance for pose interpolation. We encourage the readers to watch the side-by-side comparison videos available on our project page: https://neuralbodies.gggg itt.hub.oommttttttar..
8

---

3D Input
2.30 Input
NASA
LEAP
SCANimate
Ours
Subj 00122, 00215
Ex.
PS 1
0.078
0.314
0.333
0.5
D2 +
0.484
0.454
0.586
0.450
MLP
Postinc
SIREN
Hyper
Hyper
Tnt.
0.327
0.293
0.489
0.273
Quat
BoneFinc
NC
f
0.752
0.807
0.793
0.821
Subj 00122, 00215
Subj
00134, 03375
Dp 4
3.278
1.806
0.472
0.460
0.461
Ex.
P5
0.182
0.224
0.481
0.5
Int.
Dj
2.201
0.998
0.301
0.288
0.288
D2
0.595
0.483
0.629
0.518
NC
t
-0.279
-0.045
0.815
0.818
0.820
Dy
0.469
0.340
0.542
0.367
Suhj
00134,
03375
NC1 f
0.693
0,780
0.755
0.773
Dp 3.320
1.498
0.532
0.526
0.523
Averge per-model training/Tine hineinggginneuuuen tuning time (hours)
Tnt.
Dy 1
2.190
0.772
0.385
0.378
0.374
>10
>10
1.60
NC1 1
-0.300 -0.099
0.772
0.772
Table I: Comparison to baselines. Dp. Dy and NC are reported for interpolation (Int.) while PS is reported for extrapolation (Ex.). Note that MetaAvatar is fine-tuned on depth images while all other baselines are trained on complete meshes. The traiiing/fie-tuning times are just rough estimates, as ours does not include the time for leta-learning, while many factors, in- cluding varying training schedules, disk-IOs and hardware setups, can affect the final speed.
Table 2: Ablation for different architectures
the interpolation task. Hyper- Quat is our model that takes the relative joint-rotations (in the form of unit quaternions) as inputs. Hyper- BoneEnc is our full model with hierarchical bone encoding MLP of LEAP [48]. Models in the table are fine-tuned for 128 epochs.
5.2 Ablation Study on Model Architectures
We further ablate model architecture choices for MetaAvatar, We compare against (1) a plain MLP that takes the concatenation of the relative joint-rotations (in the form of unit quaternions) and query points as input (MLP), (2) a MLP that takes the concatenation of the relative joint-rotations and the positional encodings of query point coordinates input (PosEnc), and (3) SIREN network that takes the concatenation of the relative joint-rotations and query points as input (SIREN). The evaluation task is interpolation; results are reported in Table 2. For the baselines (MLP, PosEnc and SIREN), we directly use Alg. 2 to meta-learn the corresponding models with 6² = O. For MLP and PosEne, the corresponding models fail produce reasonable shapes. For SIREN, it produces unnaturally smooth surfaces which cannot capture fine clothing details such as wrinkles (Fig. 5).
5.3 Few-shot learning of MetaAvatar
In this section, we evaluate the few-shot learning capabilities of MetaAvatar. As shown in Table 3, we reduce the amount of data on the fine-tuning set, and report the performance of models fine- tuned on reduced amount of data. Note that with <1% data, we require only one frame from each action sequence available for subject/cloth-type combination, this roughly equals to 8-20 depth frames depending on the amount of data for that subject/cloth-type combination. For interpolation, the performance drops because the stochastic na- ture of cloth deformation becomes dominant when the amount of fine-tuning data decreases. On the other hand, the perceptual scores (PS) are better than NASA and LEAP even with <1 % data in the form of partial depth observations, and better than or comparable to SCANimate in most cases. The qualitative comparison on extrapolation results of reduced fine-tuning data is shown in Fig. 6. Please see the Supp. Mat. for more qualitative results on few-shot learning, including results on depth from raw scans, results on real depth images and comparison with pre-trrined SCANimate model.
Fine-tuee data (86
100
30
20
10
5
<1
Suby 00122,
00215
Ex.
P8
0.471
0.509
0.473
0.373
0,510
D2
0.450
0.480
0.512
0.543
0.592
Int.
Dr
0.273 0,310
0.353
0.391
0.450
NC 1
0.821
0.808
0.795
0.785
0.768
Suby
00134,
05375
Ex.
PS
0.476
0.424
0.463
0.439
0.387
0.518
0.545
0.576
0.603
0.619
Int.
Dr
0.367
0.400
0.438
0.471
0.489
NC
0.773
0.762
0.753
0.745
0.737
Average per-model training/ sunning Lime (bours)
1.60
0.8
0.32
0.16
0.08
0.02
Table 3: Ablation for few-shot learning. We report performance of MetaAvatar on reduced amount of fine-tuning data. Fine-tuning time scales linearly with the amount of data, since we run for fixed number of epochs.

---

6 Conclusion
We introduced MetaAvatar, meta-leamed hypernetwork that represents controllable dynamic neural SDFs applicable for generating clothed human avatars. Compared to existing methods, MetaAvatar learns from less data (temporally discrete monocular depth frames) and requires less time to represent novel unseen clothed humans. We demonstrated that the meta-learned deformation prior is robust and can be used to effectively generate realistic clothed human avatars in 2 minutes from as few depth observations.
MetaAvatar is compatible with automatic registration methods [8, 75], human motion models [84, 85], implicit hand models [30, 31J and rendering primitives [40, 47] that could jointly enable an efficient end-to-end photo-realistic digitization of humans from commodity RGBD sensors, which has broad applicability in movies, games, and telepresence applications. However, this digitization may raise privacy concerns that need to be addressed carefully before deploying the introduced technology.
7 Acknowledgment
Siyu Tang acknowledges funding by the Swiss National Science Foundation under project 200021 204840. Andreas Geiger was supported by the ERC Starting Grant LEGO-3D (850533) and DFG EXC number 2064/1 project number 390727645. We thank Jinlong Yang for providing results of SCANimate on the CAPE dataset. We thank Yebin Liu for sharing POSEFusion [38] data and results during the rebuttal period. We also thank Yan Zhang, Siwei Zhang and Korrawe Karunratanakul for proof reading the paper.
References
[1] Ferran Alet, Tomas Lozano-Perez, and Leslie P. Kaelbling. Modular meta-learning. In Proc, of The 2nd Conference on Robot Learning, 2018.
[2] Thiemo Alldieck, Marcus Magnor, Bharat Bhatnagar, Christian Theobalt, and Gerard Pons-Moll. Learning to reconstruct people in clothing from a single RGB camera. In Proc. of CVPR, 2019. 2
[3] Thiemo Alldieck, Mareus Magnor, Weipeng Xu, Christian Theobalt, and Gerard Pons-Moll. Video based reconstruction of 3d people models. In Proc. CVPR, 2018.
[4] Marcin Andrychowicz, Misha Denil, Sergio Gómez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando de Freitas. Learning to learn by gradient descent by gradient descent. In Proc. of NeurlPS, 2016.
[5] Dragomir Anguelov. Praveen Srinivasan, Daphne Koller, Sebastian Thrun. Jim Rodgers, and James Davis. Scape: shape completion and animation of people. ACM Transasctions Graphics, 24, 2005. 2.3
[6] Antreas Antoniou, Harrison Edwards, and Amos Storkey. How to train your maml. In Proc. of ICLR, 2019.
Bharat Lal Bhatnagar, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll. Combining implicit function learning and parametric models for 3d human reconstruction. In Proc. of ECCV, 2020.
2.
[8] Bharat Lal Bhatnagar, Cristian Sminchisescu, Christian Theobalt, and Gerard Poll.Moll. Loopreg: Self- supervised learning of implicit surface correspondences, pose and shape for 3d human mesh registration.
In Proc. of NeurlPS, 2020. 2, 10
191 Andrei Buroy, Matthias NicBner, and Justus Thics. Dynamic surface function networks for clothed human
bodies. In Proc. of ICCV, 2021.
[10] Xu Chen, Yufeng Zheng, Michael J Black, Otmar Hilliges, and Andreas Geiger. SNARF: Differentiable forward skinning for animating non-rigid neural implicit shapes. In Proc. of ICCV, 2021. 2
[1]] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proc. of CVPR,
2019. 3,
[12] Julian Chibane, Thiemo Alldieck, and Gerard Pons-Moll. Implicit functions in feature space for 3d shape reconstruction and completion. In Proc, of CVPR, 2020. 1,3
[13] Boyang Deng, JP Lewis, Timothy Jeruzalski, Gerard Pons-Moll, Geoffrey Hinton, Mohammad Norouzi, and Andrea Tagliasacchi. Nasa: Neural articulated shape approximation. In Proc. of ECCV, 2020. 2, 6, 7
[14] Nanqing Dong and Eric P. Xing. Few-shot semantic segmentation with prototype learning. In Proc. of
BMVC, 2018.
[15] SM Ali Eslami, Danilo Jimenez Rezende, Frederie Besse, Fabio Viola, Ari S Morcos, Marta Garello, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, (63()))4):: 104--1200 2018.
[16] Chelsea Finn. Pieter Abbeel, and Sergey Levine. Model- Model-agnostic meta- letanig for fast adaptation of deep networks. In Proc. of [CML, 2017. 2.3
10

---

[17] Marta Garnelo, Dan Rosenbaum, Chris J. Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo J. Rezende, and S. M. Ali Eslami. Conditional neural processes. In Proc. of ICML, 2018. 3
[18] Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, S. M. Ali Eslami, and Yee Whye Teh, Neural processes. In ICML Workshop on Theoretical Foundations and Applications of Deep Generative Models, 2018.
[19]
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard E. Turner. Meta- lcarning probabilistic inference for prediction. In Proc. of ICLR, 2019. 3
[20] Edward Grefenstette, Brandon Amos, Denis Yarats, Phu Mon Htut, Artem Molchanov, Franziska Meier, Douwe Kiela, Kyunghyun Cho. and Soumith Chintala. Generalized inner loop meta-learning, 2019. 3
[21] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization
for learning shapes. In Proc. of ICML, 2020. 2,
[22] Peng Guan. Loretta Reiss, David A. Hirshberg. Er Weiss, and Michael J. Black. Drape: Dressing any person. ACM Transaxctions Graphics 31()), 2012. 2
[23] Liang- Lan Gui, Yu-Xiong Wang. Deva Ramanan, and Jose M. F. Moura. Fow-shot human motion prediction
via meta-learning In Proc. of ECCV 2018. 3
[24] Erhan Gundogdu, Victor Constantin, Amrollah Seifoddini, Minh Dang, Mathieu Salzmann, and Pascal
Fua. Garnet: two-stream network for fast and accurate 3d cloth draping. Proc, ICCV, 2019.
[25]
David Ha, Andrew Dai, and Quoe V. I.c. Hypernetworks. In Proc. ICT.R 2017. 2, 2,4,
[26] N. Hasler, C. Stoll, M. Sunkel, B. Rosenhahn, and H.-P.. Seidel. A Statistical Model of Human Pose and
Body Shape. Computer Graphics Forum, 28:337-346, 2009. 2,
[27] Zeng Huang. Yuanlu Xu, Christoph Lassner. Hao Li, and Tony Tung. ARCH: Animatable Reconstruction
of Clothed Humans. Proc. of CVPR, 2020. 3
[28] Hanbyul Joo, Tomas Simon, and Yaser Sheikh, Total capture: A 3d deformation model for tracking faces.
hands, and bodies. In Proc. of CVPR, 2018. 2
[29] Bingyi Kang. Zhuang Liu, Xin Wang. Fisher Yu, Jiashi Feng, and Trevor Darrell. Few-shot object detection
via feature reweighting. In Proc. of ICCV, 2019. 3
[30] Korrawe Karunratanakul, Adrian Spurr, Zicong Fan, Otmar Hilliges, and Siyu Tang. A skeleton-die -riven
neural occupancy representation for articulated hands. In Proc. of 3DV, 2021. 10
[31] Korrawe Karunratanakul, Jinlong Yang, Yan Zhang, Michael Black, Krikamol Muandet, and Siyu Tang. Grasping field: Learning implicit representations for human grasps. In Proc. of 3DV, 2020. 10
[32] Hyunjik Kim, Andriy Mnih. Jonathan Schwarz, Marta Garnelo. Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Tch. Attentive neural processes. In Proc. of ICLR, 2019. 3
[33] Zorah Lähner, Daniel Cremers, and Tony Tung. Deepwrinkles: Accurate and realistic clothing modeling.
In Proc. of ECCV, 2018. 2
[34] Ke Li and Jitendra Malik. Learning to optimize. In Proc. of ICLR, 2017. 3
[35] Ruilong Li, Yuliang Xiu, Shunsuke Saito, Zeng Huang, Kyle Olszewski, and Hao Li. Monocular real-tin -il-time
volumetric performance capture. In Proc. of ECCV, 2020. 1,
[36] Zhongguo Li, Magnus Oskarsson, and Anders Heyden. Geo-pifu: Geometry and pixel aligned implicit
functions for single-view human reconstruction. In Proc. of NeurlPS, 2020. 1, 3
[37] Zhe Li, Tao Yu, Chuanyu Pan, Zerong Zheng, and Yehin Liu. Robust 3d self-portraits in seconds. In Proc.
of CVPR. 2020. 1
[38] Zhe Li, Tao Yu, Zerong Zheng, Kaiwen Guo, and Yebin Liu. Posefusion: Pose-guided selective fusion for
single-view human volumetric capture. In Proc. of CVPR, 2021. 1, 10
[39] Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot
learning, 2017. 3
[40] Stephen Lombardi, Tomas Simon, Gabriel Schwartz, Michael Zollhoefer, Yaser Sheikh, and Jason Saragih.
Mixture of volumetric primitives for efficient neural rendering. ACM Transactions on Graphics (TOG),
2021. 10
[41] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A
skinned multi-person linear model. ACM Transasctions Graphics, 34(6), 2015, 2, 3
[42] Qianli Ma, Shunsuke Saito, Jinlong Yang, Siyu Tang, and Michael J. Black. SCALE: Modeling clothed
humans with a surface codec of articulated local elements. In Proc. of CVPR, 2021.
[43] Qianli Ma. Jinlong Yang. Anurag Ranjan, Sergi Pujades, Gerard Pons-Moll, Siyu Tang, and Michael J.
Black. Learning to dress 3D people in generative clothing. In Proc. of CVPR, 2020. 2, 6.7
[44] Qianli Ma, Jinlong Yang. Siyu Tang, and Michael J. Black. The power of points for modeling humans in clothing. Proc. of ICCV, 2021.
[45] Lars Mescheder. Michael Oechsle, Michael Niemeyer. Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proc. of CVPR, 2019. 3,
[46] Matcusz Michalkiewicz, Jhony K. Pontes, Dominic Jack, Mahsa Baktashmotlagh, and Anders Eriksson.
Implicit surface representations as layers in neural networks. In Proc. of ICCV, 2019. 1,
[47] Marko Mihajlovic, Silvan Weder, Mare Pollefeys, and Martin R Oswald. DeepSurfels: Learning online
appcarance fusion. In Proc. of CVPR, 2021. 10
[48] Marko Mihajlovic, Yan Zhang, Michael J. Black, and Siyu Tang. LEAP: Learning articulated occupancy
of people. In Proc. CVPR, 2021. 2, 3, 4, 6, 7,9
[49] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-Icarning algorithms, 2018 2, 3, 5, 6 [50] Ahmed A. A. Osman, Timo Bolkart, and Michael J. Black. Star: Sparse trained articulated human body regressor. In Proc. of ECCV, 2020. 2,
11

---

[51] Pablo Palafox, Aljaz Bozic. Justus Thies, Matthias Nießner, and Angela Dai. Neural parametric models for
3d deformable shapes. In Proc. of ICCV, 2021. 2
[52]
Jeong Joon Park, Peter Florence, Julian Straub. Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proc. CVPR, 2019. 1,3
[53] Chaitanya Patel. Zhouyingcheng Liao, and Gerard Pons-Moll. Tailomet: Predicting clothing in 3d as
function human pose, shape and garment style. In Proc. of CVPR, 2020.
[54] Georgios Pavlakos, Vasilcios Choutas, Nima Ghorbami, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3d hands, face, and body from single image. In Proc. of CVPR, 2019. 2.3
[55] Songyou Peng. Chiyu "Max" Jiang. Yiyi Liao, Michael Niemeyer, Mare Pollefeys, and Andreas Geiger.
Shape as points: A differentiable poisson solver. In Proc. of NeurlPS, 2021. 3
[56] Songyou Peng. Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Convolutional
occupancy networks. In Proc. of ECCV, 2020. 4
[57] Sida Peng, Yuanqing Zhang. Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowci Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In Proc. of CVPR. 2021.
[58]
Juan-Manuel Perez-Rua, Xiatian Zhu, Timothy M. Hospedales, and Tao Xiang. Incremental few-shot
object detection. In Proc. of CVPR, 2020. 3
[59]
Amit Raj. Julian Tanke, James Hays, Minh Vo, Carsten Stoll, and Christoph Lassner. Anr-articulated neural rendering for virtual avatars. In Proc. of CVPR, 2021.
[60] Kate Rakelly, Evan Shelhamer, Trevor Darrell, Alexei A. Efros, and Sergey Levine. Few-shot segmentation propagation with guided networks. In Proc, of ICML, 2019, 3
[61]
Sachin Ravi and Hugo Larochelle. Optimization as a model for fewshot learning. In Proc. ICLR, 2017.
[62] Scott Reed, Yutian Chen, Thomas Paine, Aäron van den Oord, S. M. Ali Eslami, Danilo Rezende, Oriol Vinyals, and Nando de Freitas. Few-shot autoregressive density estimation: Towards learning to learn distributions. In Proc. of ICLR, 2018. 3
[63]
Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proc. of ICCV, 2019. 3,3 Shunsuke Saito. Tomas Simon, Jason Saragih, and Hanbyul Joo. Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization. In Proc. of CVPR, 2020. 1,3
[64]
[65]
Shunsuke Saito. Jinlong Yang, Qianli Ma, and Michael J. Black. SCANimate: Weakly supervised learning of skinned clothed avatar networks. In Pruc. of CVPR, 2021. 2, 3, 4, 6,7
[66]
Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. In Proc. of BMVC, 2017. 3
[67]
Vincent Sitzmann, Eric R. Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein. Metasdf: Meta- learning signed distance functions. In Proc. of NeurlPS, 2020. 2, 3.
[68]
Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Proc. of NewrlPS, 2020. 5,8 Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous
[69]
3d- structure- aware neural scene representations. In Proc. of NeurlPS, 2019. 4
[70] Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In Proc. of NeurlPS, 2017. 3
[71] Zhuo Su. Lan Xu, Zerong Zheng, Tao Yu, Yebin Liu, and Lu Fang. Robustfusion: Human volumetric
capture with data-driven visual cues using a rgbd camera. In Proc. of ECVV, 2020.
[72] Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P. Srinivasan, Jonathan T. Barron, and Ren Ng. Learned initializations for optimizing coordinate-basee based neural representations. In Proc. of
CVPR, 2021. 2,3
[73] Garvita Tiwari, Nikolaos Sarafianos, Tony Tung, and Gerard Pons-Moll, Neural-GIF: Neural generalized implicit functions for animating people in clothing. In Proc. of ICCV, 2021.
[74] Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng. Panet: Few-shot image
semantic segmentation with prototype alignment. In Proc. of ICCV, 2019. 3
[75] Shaofci Wang. Andreas Geiger, and Siyu Tang. Locally aware piecewise transformation fields for 3d human mesh registration. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 1, 10
[76] Ting- Cang-Cuun/Pe Wang, Ming- Yu Liu, Andrew Tao. Guilin Liu, Jan Kautz, and Bryan Catanzaro. Few-shot
video-to- viddv synthesis. In Prec. of NewrlPS, 2019. 3
[77] Hongyi Xu, Eduard Gabriel Bazavan, Andrei Zanfir, William T. Freeman, Rahul Sukthankar, and Cristian Sminchisescu. Ghum & ghuml: Generative 3d human shape and articulated pose models. In Proc. of CVPR, 2020. 2.3
[78] Lan Xu, Zhuo Su, Lei Han, Tao Yu, Yebin Liu, and Lu Fang. Unstructuredfusson: Real-time 4d geometry and texture reconstruction using commercial rgbi cameras. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42. 2020.
[79] Jinlong Yang. Jean- -ean-Séébstien Franco. Franck Hlétroy-Wheele and Stefanie Wuhrer. Analyzing clothing layer deformation statistics of 3d human motions. In Proc. of ECCV, 2018. 2
[80] Tao Yu. Zerong Zheng, Kaiwen Guo, Jianhui Zhao, Qionghai Dai, Hao Li, Gerard Pons-Moll, and Yebin Liu. Doublefusion: Real-time capture of human performances with inner body shapes from a single depth sensor. In Proc. of CVPR, 2018 I
12

---

[81] Ilya Zakharkin, Kirill Mazur. Artur Grigorev, and Victor Lempitsky. Point-based modeling of human clothing. In Proc. ICCV, 2021.
[82] Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky, Few-shot adversarial learning of realistic neural talking head models. In Proc. ICCV, 2019.
[83] Chao Zhang. Sergi Pujades. Michael J. Black, and Gerard Pons-Moll. Detailed, accurate, human shape
estimation from elothed 3d sean sequences. Prec. CVPR. 2017.
[84] Siwei Zhang. Yan Zhang, Federica Bogo, Pollefeys Mare, and Siyu Tang. Learning motion priors for 4d
human body capture 3d scenes. Proc. of ICCV, 2021.
[85] Yan Zhang. Michael Black, and Siyu Tang. We are more than our joints: Predicting how 3D bodies
move. In Proc. of CVPR, 2021. 10
[86] Dominic Zhao, Johannes von Oswald, Seijin Kohayashi, João Sacramento, and Benjamin F. Grewe. Meta- learning via hypernetworks. In 4th Workshop on Meta-l-earming at NeurIPS 2020, Voncouver, Canada, 2020. 3
[87] Z. Zheng. T. Yu, Y. Liu, and Q Dai. Pamir: Parametric image- based human reconstruction. IEEE Transactions Pattern Analysis and Machine Intelligence, pages 1-1, 2021.
implicit representation for
13


Reviews:
[{"review_id": "wSv6LJpBcyU", "reviewer": "Reviewer_XKUG", "summary": "This paper aims to create generalizable and controllable neural signed distance fields from monocular depth observations. The authors propose a hypernetwork to represent neural SDFs across different human shapes and cloth types. For fast fine-tuning, they additionally leverage meta-learning to learn an initialization of the hypernetwork. This method achieves impressive reconstruction results and demonstrates a cool application that reconstruct high-quality human shapes from 8 monocular depth frames in 2 minutes.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n\n1. This paper propose a novel representation for human shapes. It first learns a meta-SDF to fit canonical 3D subjects with diffrent genders, body shapes, cloth types, and poses. To model the shape deformation caused by the poses, it then meta-learn a hypernetwork that takes the poses as input and predicts the residuals to the parameters of the previously learned meta-SDF.\n2. This representation can be seen a combination of coarse and fine representations, which effectively and efficiently models the shape deformation.\n\nQuality:\n\n1. The submission is technically sound.\n2. The claims are well supported.\n3. The comparisons and ablation studies are sufficient.\n\nClarity:\n\n1. The submission is clearly written and well organized.\n\nSignificance:\n\n1. This paper proposes a method that quickly reconstructs high-quality human shapes from monocular depth frames, which is very practical and largely promotes the field of human reconstruction. It will probably enlighten follow-up works.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not see potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "jgmKAKWmhuQ", "reviewer": "Reviewer_xRBs", "summary": "This paper proposes a novel learning method to generate controllable neural SDF for non-rigid clothed human body from a few depth scans. The entire pipeline consists of three main parts, i.e., canonicalization, neural SDF, and hyper-net. The neural SDF takes canonicalized points as inputs and learns a static representation. The hyper-net is used to fine-tune the static neural SDF and generate dynamic SDF efficiently for forward LBS based animation.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "As far as I am concerned, the method proposed in this work is novel and technically sound. The target application can be useful for animation of clothed 3D human body from a few depth incomplete input. The presented animation results demonstrate some advantages over those obtained via state-of-the-art methods in certain cases from the CAPE dataset.\n\nOn the other hand, I think this paper needs some major revisions on paper writing, especially Section 4. I suggest to add an overview section or paragraph to present the entire pipeline and clearly describe the input and output of each stage/component. The inference part should be put into an individual subsection. In Section 5, the CAPE dataset contains 15 human objects but only 14 are used for the experiments (among which 10 for training, 4 for validation and test). Is there any particular reason for this choice? Also, the depth frames are synthetically rendered from the CAPE dataset. There are no examples using input from real depth sensor or dataset other than CAPE.\n\nOverall, I think this work is technically sound but the presented results are not convincing enough.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please see the main review part.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "F4KnaIec4oB", "reviewer": "Reviewer_x2Rq", "summary": "This paper presents a framework that can learn an animatable implicit avatar for clothed humans from very few observations. The key idea is using meta-learning to learn a prior (specifically speaking, a good initialization) to enable fast fine-tuning at inference time. To enable animations of various cloth types, this paper proposes to perform meta-learning on a hyper-network which controls the parameters of the neural implicit function conditioned on pose configurations. Results show that the proposed method is able to generate clothed human avatars from few shot observations. \n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths:\n* The idea of meta-learning a pose-conditioned hyper-network is interesting and inspiring. To the best of my knowledge, this paper is the first one that applies meta-learning in human modeling. The usage of meta-learning can accelerate network optimization by a large margin, e.g., from 10 hours to 1.6 hours as presented in the paper. The design of meta-learning a hyper-network is also a valuable idea for the community. \n\n* This paper is well motivated and addresses an important problem: current methods for neural SDFs-based clothed avatars take too much optimization time. Compared to SOTA methods like SCANimate, the proposed method can generate comparable and even better results, while being much more efficient. \n\n* This paper is overall well-written and easy to follow. \n\n### Weaknesses:\n* My major concern is about the claim that this method requires \"only\" depth frames as input.  Although the authors keep emphasizing that their method only needs partial depth observations for fine-tuning, in fact the method assumes perfect SMPL registrations and ground-truth bone transformations, which is only possible when complete 3D scans are available. In the experiments, the authors only use depth rendering from *complete* CAPE meshes. In the supplemental material, the authors conduct an additional experiment on raw scans, but that experiment still requires ground-truth SMPL registrations to clean and complete the scans, and also to calculate the bone transformations. In one word, I don't think the experiments can support the claim in L55-56 (\"given only a few depth observations of an unseen clothed human as inputs\") because of its dependence on ground-truth SMPL registrations. It would make this paper stronger if the authors can conduct experiments on *real* partial depth inputs, i.e., raw depth images captured by an RGBD sensor like Microsoft Kinect or Intel RealSense. \n\n* The numerical evaluation in Table 2 is not very convincing. The numbers in the last 3 columns are really closed to each other. Table 2 makes me feel that the performance gains mainly stem from the usage of SIREN activations, and that the slight improvement in the last column is due to the increased number of network parameters. \n\n\n### Minor:\n* L226: Although reader can guess that the meshes are rotated around the yaw axis, it would be better if the authors can explicitly specify it. \n\n* L311: to representation -> to represente\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have discussed the limitations in the supplemental document and the potential societal impact in Section 6. No more improvement is needed. \n\n", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours."}, "extra_text": {}}, {"review_id": "1Zhi4sJ6-L", "reviewer": "Reviewer_ZigT", "summary": "This paper looks at learning animatable avatars given a small number of depth images with associated known SMPL pose transformations. The proposed approach (similar to prior work e.g. SCANimate) learns: a) forward/inverse skinning weight predictors,  which given a query point x in canonical/posed space respectively predict the forward/inverse LBS weights, and b) a canonical space implicit SDF function, which conditioned on SMPL pose, predicts a pose-dependent shape s.t. its forward transformation looks correct.\n\nWhile the paper uses a fixed network for a) above, the key contributions relate to how the pose-dependent canonical shape is modeled. This paper proposes to capture each object via a hyper-network which predicts the weights of the SDF function conditioned on SMPL pose parameters. It shows that such a hypernetwork can be meta-learned using training data with strong supervision, but can then be finetuned using only a small number of depth images in varying poses for a novel instance.\n\nThe experiments convincingly show the benefits of this approach across different datasets and in particular show how using only a small number of depth images suffice.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a very well-motivated and a very-well executed paper which looks at learning an avatar for a novel instance given a small number of depth images with known poses. In particular:\n\na) I really liked the insight of modeling the pose-dependent shape variations of each instance via a  hyper-network, and the paper convincingly demonstrates that this network can be effectively finetuned via a small number of depth images (unlike prior methods which use full scans/meshes).\n\nb) The insight of factoring this hyper-network into a base network, only whose residual weights are predicted is also an elegant one.\n\nc) The approach for predicting generalizable skinning weights via a convolutional network is also well-motivated, although I am not sure why a similar hyper-network based approach would not be suitable (especially for forward skinning weights).\n\nd) The experiments, and the visualizations in the supplementary convincingly show that this approach allows more detailed reconstruction over the baselines - similar to prior methods which overfit to a single instance, it can capture details of the 3D shape, while only requiring a fraction of data or optimization time.\n\ne) The ablations, in particular the MLP, PosEnc, and SIREN baselines clearly justify the benefits of learning a hypernetwork to model each instance instead of a 'normal' conditional network to predict shape by concatenating the SMPL pose.\n\nOverall, this is a good paper that proposes a novel and insightful solution for learning animatable avatars from a small number of depth images, and I would argue for acceptance.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission introduces a method that enables the generation of realistic clothed human avatars from monocular depth images. While the initial reviews are mixed, after rebuttal, all reviewers are positive and recommend acceptance.  The AC agrees.  The authors should try to address the reviewers' concerns in the camera-ready version. This includes adding the results from the rebuttal period, clarifying the concerns on meta-learning as reviewer XKUG suggested, among others.
Decision: Accept (Poster)