Towards Animatable Human Avatars
Andreas Geiger
Autonomous Vision Group
University of Tübingen and MPI for Intelligent Systems

Covered Papers

SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit
ShapesX. Chen, Y. Zheng, M. Black, O. Hilliges and A. Geiger
ICCV 2021
MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images
S. Wang, M. Mihajlovic, Q. Ma, A. Geiger and S. Tang
NeurIPS 20212

Collaborators

Xu ChenShaofei Wang

Yufeng Zheng

Marko Mihajlovic

Qianli MaMichael BlackOtmar Hilliges

Siyu Tang

3SNARF: Differentiable Forward Skinning for
Animating Non-Rigid Neural Implicit Shapes

Neural Implicit Shapes

Text

Neural implicit representations are useful for 3D human modeling:
I Topological ﬂexibility
I Resolution independent
However:
I Animating such representations is not straightforward
Chen, Zheng, Black, Hilliges and Geiger: SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes. ICCV, 2021.

5

Animating Neural Implicit Shapes
Text?
Goal:
I Generate implicit shapes in given poses
I Learn shape representation from deformed observations
Key question:
I How to model and learn skeletal deformation of implicit shapes?
Chen, Zheng, Black, Hilliges and Geiger: SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes. ICCV, 2021.

6

Existing Solutions

Animating Implicit Shapes
Canonical Parts

Transformed Parts
Deformed Shape

Rigid
Transform

Ground Truth

Composite
Rigid
Transform

Piecewise rigid model:[Deng et al., ECCV 2020]

I Model shape as parts, and each part can be rigidly transformed
I Discontinuous artifacts at joints
Chen, Zheng, Black, Hilliges and Geiger: SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes. ICCV, 2021.

8

Animating Implicit Shapes
Canonical Shape

Skinning Weights

Deformed Shape

Ground Truth

Backward
LBS

Backward skinning:[Jeruzalski et al., ArXiv 2020][Mihajlovic et al., CVPR 2021]

I Backward LBS with pose-dependent skinning weights in deformed space

Chen, Zheng, Black, Hilliges and Geiger: SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes. ICCV, 2021.

9

Animating Implicit Shapes
Canonical Shape

Skinning Weights

Deformed Shape

Ground Truth

BackwardLBSBackward skinning:[Jeruzalski et al., ArXiv 2020][Mihajlovic et al., CVPR 2021]

I Backward LBS with pose-dependent skinning weights in deformed space
I Does not generalize to unseen poses
I Cannot handle one-to-many mapping
Chen, Zheng, Black, Hilliges and Geiger: SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes. ICCV, 2021.

9

Animating Implicit Shapes
Canonical Shape

Skinning Weights

Deformed Shape

Ground Truth

ForwardLBS

This work - forward skinning:
I Forward LBS with pose-independent skinning weights in canonical spaceChen, Zheng, Black, Hilliges and Geiger: SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes. ICCV, 2021.

10

Animating Implicit Shapes
Canonical Shape

Skinning Weights

Deformed Shape

Ground Truth

Forward
LBS

This work - forward skinning:
I Forward LBS with pose-independent skinning weights in canonical space
I Generalization to unseen poses
I Can handle one-to-many mapping
Chen, Zheng, Black, Hilliges and Geiger: SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes. ICCV, 2021.

10

Differentiable Forward SkinningCorrespondence search:
I Given query x0 , its canonical correspondences x∗ satisfy dσw (x∗ , B) − x0 = 0I x∗ can be numerically determined via iterative root ﬁnding

Chen, Zheng, Black, Hilliges and Geiger: SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes. ICCV, 2021.

11

Differentiable Forward Skinning

Multiple correspondences:
I Multiple solutions might exist → apply root ﬁnding with multiple initializations
0I Rigidly transform the query point with each bone as initialization x0i = B −1i ·xI Collect valid solutions by convergence X ∗ = x∗i | kdσw (x∗i , B) − x0 k2 < 
Chen, Zheng, Black, Hilliges and Geiger: SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes. ICCV, 2021.

11

Differentiable Forward SkinningOccupancy query:
I Model the canonical shape as a single occupancy network
I Condition the canonical shape on pose to model pose-dependent deformations
I Aggregate multiple correspondences o(x0 , p) = maxx∗ ∈X ∗ {fσf (x∗ , p)}Chen, Zheng, Black, Hilliges and Geiger: SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes. ICCV, 2021.

11

Training Objective

LBCE (o(x0 , p), ogt (x0 ))Lbone = LBCE (fσf (xbone , p), 1)

Ljoint = kwσw (xjoint ) − wtarget k22

L = LBCE + Lbone + Ljoint{z}|only ﬁrst epoch

Chen, Zheng, Black, Hilliges and Geiger: SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes. ICCV, 2021.

12

Gradients

I Analytical gradients via implicit differentiation:∂L∂L ∂o ∂fσf (x∗ ) ∂x∗=···,∂σw∂o ∂fσf∂x∗
∂σw
∂x∗
∂dσw (x∗ , B) −1 ∂dσw (x∗ , B)=−·∂σw∂x∗∂σw
Chen, Zheng, Black, Hilliges and Geiger: SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes. ICCV, 2021.

13

Results

Backward Skinning(DFaust held-out Test Sequence)

NASAOurs

Chen, Zheng, Black, Hilliges and Geiger: SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes. ICCV, 2021.

Ground Truth14Summary
Differentiable forward skinning:
I Learn forward skinning and shape ﬁelds in pose-independent space
I Learn from deformed shapes without direct supervision or prior (e.g., SMPL)I Generalize to challenging unseen poses at test time
However:
I Root ﬁnding is time-consuming (10x slower than occupancy query)
I Each iteration requires a skinning network query

I Requires 3D data ⇒ combine with differentiable renderer to learn from images
I Requires accurate poses for training ⇒ jointly optimize pose, shape and skinning
I So far only a single subject ⇒ generative model of animatable avatars
Chen, Zheng, Black, Hilliges and Geiger: SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes. ICCV, 2021.

15

MetaAvatar: Learning Animatable Clothed
Human Models from Few Depth Images

MetaAvatar

InputsClothed Avatar

InputsClothed Avatar

Goal:
I Controllable avatars learned from few monocular depth observations
I No fully-body scans or per-subject/cloth-type optimization required
I Fast optimization (2 minutes with 8 depth maps as input)
Idea: Meta-learn pose conditioned hypernetwork to predict parameters of neural SDF
Wang, Mihajlovic, Ma, Geiger and Tang: MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images. NeurIPS, 2021.

17

MetaAvatar

Approach:
I Using learned Inverse LBS, transform input depth frames into canonical pose
I Fine-tune a meta-learned HyperNet to predict parameters of neural SDF
I Given novel poses, our approach generates pose-dependent animated meshes
Wang, Mihajlovic, Ma, Geiger and Tang: MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images. NeurIPS, 2021.

18

Meta-learning a SDF

Meta-learning a SDF:
I 5-layer SIREN network with 256 neurons in each layer: fφ∗ (x)I Point-based on-surface and off-surface loss function [Gropp et al. 2020]
I Meta-learn network parameters on all subjects using Reptile [Nichol et al. 2018]
I Allows fast ﬁne-tuning on new subject, but no pose-dependent deformations

Wang, Mihajlovic, Ma, Geiger and Tang: MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images. NeurIPS, 2021.

19

Meta-learning a pose-conditioned SDF

Meta-learning a pose-conditioned SDF:
I 5-layer SIREN network with 256 neurons in each layer: fφ (x, {Bb })I Condition network on bone transformations {Bb }I Does not work very well, leads to overly smooth results

Wang, Mihajlovic, Ma, Geiger and Tang: MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images. NeurIPS, 2021.

20

Meta-learning a pose-conditioned SDF

Wang, Mihajlovic, Ma, Geiger and Tang: MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images. NeurIPS, 2021.

21

MetaAvatar

Meta-learning a pose-conditioned HyperNetwork:I Learn HyperNetwork gψ ({Bb }) on parameters of neural SDF
I HyperNetwork predicts residuals to meta-learned SDF parameters φ∗
I At test time, ﬁne-tune parameters ψ of HyperNetwork
Wang, Mihajlovic, Ma, Geiger and Tang: MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images. NeurIPS, 2021.

22

Learning with Raw Sensor Inputs

Wang, Mihajlovic, Ma, Geiger and Tang: MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images. NeurIPS, 2021.

23Fine-tuning on Kinect Data

Wang, Mihajlovic, Ma, Geiger and Tang: MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images. NeurIPS, 2021.

24Fine-tuning on Reduced Data

Wang, Mihajlovic, Ma, Geiger and Tang: MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images. NeurIPS, 2021.

25

Training/fine-tuning time (minutes)

Fine-tuning on Reduced Data
500478.0

MetaAvatar
NASA400
300
200
100
00.97

14.36

20

11.71

22.56

50100

Data percentage

Wang, Mihajlovic, Ma, Geiger and Tang: MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images. NeurIPS, 2021.

26

Comparison to Baselines

Wang, Mihajlovic, Ma, Geiger and Tang: MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images. NeurIPS, 2021.

27

SummaryMetaAvatar:
I MetaAvatar enables generation of controllable clothed human avatars
I Meta-learning allows for fast subject-speciﬁc ﬁne-tuning from few depth images
I MetaAvatar enables realistic clothed avatars in 2 minutes from 8 depth maps
I HyperNetworks are required to capture detailed pose dependent deformations
I Learned Inverse/Forward LBS models and bone transformations required as inputWang, Mihajlovic, Ma, Geiger and Tang: MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images. NeurIPS, 2021.

28

Thank you!
http://autonomousvision.github.io