SNARF: Differentiable Forward Skinning for
Animating Non-Rigid Neural Implicit Shapes
Xu Chen1,3 Yufeng Zheng1,3 Michael J. Black3 Otmar Hilliges1 Andreas Geiger2,3
1
ETH Zürich, Department of Computer Science 2 University of Tübingen
3
Max Planck Institute for Intelligent Systems, Tübingen

Abstract
Neural implicit surface representations have emerged asa promising paradigm to capture 3D shapes in a continuous and resolution-independent manner. However, adapting them to articulated shapes is non-trivial. Existing approaches learn a backward warp field that maps deformed
to canonical points. However, this is problematic since thebackward warp field is pose dependent and thus requires
large amounts of data to learn. To address this, we introduce SNARF, which combines the advantages of linear
blend skinning (LBS) for polygonal meshes with those ofneural implicit surfaces by learning a forward deformation field without direct supervision. This deformation field
is defined in canonical, pose-independent, space, enabling
generalization to unseen poses. Learning the deformation
field from posed meshes alone is challenging since the correspondences of deformed points are defined implicitly andmay not be unique under changes of topology. We propose aforward skinning model that finds all canonical correspondences of any deformed point using iterative root finding.
We derive analytical gradients via implicit differentiation,
enabling end-to-end training from 3D meshes with bonetransformations. Compared to state-of-the-art neural implicit representations, our approach generalizes better tounseen poses while preserving accuracy. We demonstrate
our method in challenging scenarios on (clothed) 3D humans in diverse and unseen poses.

1. Introduction
Modeling the shape and deformation of articulated 3D
objects has traditionally been achieved by deforming apolygonal mesh via linear blend skinning (LBS) with posecorrectives. However, meshes are inherently limited bytheir resolution-to-memory ratio and their fixed topology.
Therefore, neural implicit surface representations [10, 30,
31, 38] have recently attracted much attention because they
provide a resolution-independent, smooth and continuousalternative to discrete meshes. However, updating an im-

Input 3D Posed Meshes

Learned Canonical 3D Shape and Skinning Weights

Continuous Implicit Surfaces in Unseen Poses

Figure 1: SNARF: From a sequence of posed meshes (top),
we learn a neural implicit 3D shape and a skinning fieldin canonical pose (middle) without supervision of skinningweights or part correspondences. Learned forward skinningenables generalization to unseen poses (bottom) while capturing local details via pose conditioning.
plicit surface representation as a function of the underlying
pose changes is challenging since it requires modifying acontinuous function rather than a discrete set of points.
To address this, we propose SNARF (Skinned Neural Articulated Representations with Forward skinning), a novel
approach to learning articulated 3D shapes represented byneural implicit surfaces directly from 3D watertight meshes
and corresponding bone transformations with no need forsupervision via pre-defined skinning weights. SNARF combines the simplicity of skeletal-driven deformation of LBS
with the fidelity and topological flexibility of implicit sur-

Canonical Space

Deformed Space

Backward

Canonical Space

Deformed Space

ForwardFigure 2: Forward vs. Backward Skinning. Forwardskinning weights are defined in pose-independent canonical space and therefore naturally generalize to unseen poses
as the one in the bottom left panel. In contrast, backward
skinning weights are defined in pose-dependent deformed
space and thus struggle with unseen poses.
faces, enabling animation of complex human bodies asshown in Fig. 1. Moreover, SNARF goes beyond LBS
by conditioning the neural shape on poses to capture posedependent non-linear deformations. The main challenge is
to express the mapping between surface points in canonicalpose and their deformed counterparts. Existing approaches
attempt to learn shape in the canonical pose and a backward deformation field, transforming deformed points tothe canonical pose [12, 34, 39, 43]. However, as illustrated
in Fig. 2, backward skinning is problematic since the deformation field depends on the pose of the deformed object,
limiting generalization to unseen poses.
To tackle this problem, we devise a method that learns
a dense forward skinning weight field without requiring direct supervision. Once learned, this skinning field can beleveraged to generate shape deformations even for poses
outside of the training set. However, to jointly learn the forward skinning field and the object shape from posed meshes
alone, we must establish the correspondence of any 3D
point in deformed space to the undeformed space. Yet, thisrequires the availability of the backward mapping which isonly implicitly defined and has no analytical solution.
To overcome this issue, we propose a forward skinningmodel that exploits an iterative root finding algorithm tofind the corresponding canonical point for any deformed

point. Our approach is able to retrieve multiple correspondences for any deformed point and therefore naturally handles topology changes. We further derive the gradients ofour forward skinning module, hence making it differentiable and enabling end-to-end learning of the canonicalshape and skinning weights jointly from deformed observations. Importantly, and in contrast to prior work, our method
does not require any a priori skinning weights or pose correctives defined on the surface and hence can be applied in
scenarios where pre-rigged mesh models are not available.
We experimentally demonstrate that our method is able
to generate high-quality shapes with arbitrary desired bonetransformations, even those far beyond the training distribution, where other recent methods like NASA [12] fail.
Since our approach operates in continuous space, it enables
reconstruction of fine geometric details. By conditioning
the neural implicit function on poses, our method faithfullymodels local pose-dependent deformations, e.g., the movement of clothing or soft tissue. Our code is available atgithub.com/xuchen-ethz/snarf.

2. Related Work
Skinning Polygonal Meshes: Modeling the deformation of
non-rigid and articulated 3D objects is a fundamental problem in computer vision and graphics with many applications. Traditionally, this problem is formulated for polygonal meshes and is referred to as skinning. Skinning enables
deformation of a high-resolution surface mesh with loworder control primitives such as skeletal bones. The mostcommon approach is linear blend skinning (LBS), whichmodels each mesh vertex’s deformation as a convex combination of input bone transformations as defined by skinning weights. These skinning weights are typically definedby an artist or learned from data. LBS produces wellknown artifacts that many methods attempt to address, e.g.
with dual quaternion blend skinning [20] or multi-weightenveloping [29, 54]. The key concept is to define posedependent “corrective blend shapes” that are added to ashape such that, when it is posed, the LBS errors are minimized [21, 46]. Classically, these “pose correctives” areartist defined, though they can also be learned [24]. Here
we extend the concept of LBS and pose correctives to neural implicit surface representations.
Learning both blend weights and rigs from examples
has a long history, starting with James and Twigg [17].
Specifically for human bodies, numerous learning methods
have been proposed, many of which learn the LBS weights[15, 24, 37, 56]. Recent methods attempt to disentangle
shape and pose in an unsupervised fashion given registeredtraining meshes [19, 60]. RigNet [57] uses a deep network
to learn both articulated rigs and skinning weights jointly.
NeuroSkinning [23] also uses a deep network to learn blend
weights and can cope with complex surface topology. Incontrast to us, these methods require a large dataset ofrigged models with hand-painted skinning weights and do
not consider implicit surface representations.
Neural Implicit Shapes: Neural implicit shape representations can model complex shapes with arbitrary topology in a continuous fashion. Given a 3D location, these
networks regress the distance to the surface [38], occupancy probability [30], color [36] or radiance [33] of a 3D
point. Conditioning on local information such as 2D image features or 3D point cloud features has been shownto yield more detailed reconstructions [11, 16, 41, 47, 48].
While early methods require watertight meshes for training, several recent approaches have demonstrated unsupervised training from raw 3D points clouds [5, 13, 49] or images [33, 35, 51, 58]. A current limitation of most existing implicit models is that they do not support high-quality
skeletal deformation. Our method addresses this key limitation, enabling learning and generation of realistic skeletal
deformations of neural implicit surfaces.
Deformable Neural Shapes: Compared to meshes, deforming neural implicit shapes is more challenging as one
needs to deform continuous space rather than a fixed setof vertex points. Very recently, various approaches havebeen proposed to model backward deformation fields [18,34, 39, 43]. These fields map points in deformed space
to canonical ones, where geometric properties (e.g. occupancy) are queried from a canonical shape network. The deformation field is modeled as a neural network that outputs
velocity [34], translation [43] or rigid transformation [39]and is jointly trained with the canonical occupancy network
using observations in deformed space. NiLBS [18] learns
skinning weights for each point and then derives the deformation via LBS according to the bone transformations.
An inherent limitation of learned backward deformation,
however, is poor generalization to unseen poses. As illustrated in Fig. 2, backward deformation fields are defined in
deformed space and, hence, inherently deform with pose.
Thus, the network must memorize deformation fields fordifferent spatial configurations, making it difficult to generate deformations that have not been seen during training.
Part-based Models: In recent work, NASA [12], proposes
to represent a 3D human body model as a combination of
independent parts, each of which is represented by an occupancy network [30]. Rigidly transforming these parts
according to the input bone transformations produces deformed shapes. While such a formulation preserves theglobal structure after articulation, the continuity of surface
deformations is violated, causing artifacts at intersectionsof body parts. Although each part can learn to deform itself to partially compensate for this undesired effect, noticeable artifacts remain, particularly for poses that are beyond the training distribution. Moreover, NASA requires

ground-truth surface skinning weights to learn correct partassignments. In contrast to NASA, our method learns forward skinning weights without such supervision and captures pose-dependent deformations.
More generally, the previous approaches suffer from artifacts due to overly simple assumptions about deformationor do not generalize well to unseen poses as shown in Fig. 4.
In contrast, SNARF generates continuous shapes in arbitrary poses, even those far beyond the training distribution,by learning pose-independent forward skinning weights andpose-dependent correctives in canonical space.
3D Human Avatars: While more general, we demonstrate
our approach on the problem of learning and animating realistic 3D human avatars. Recent [2, 3, 4, 6, 59] and concurrent works [9, 14, 22, 25, 26, 32, 40, 42, 44, 49, 50, 52,
53, 55] on learning 3D human models typically require a
template mesh model with fixed topology, e.g. SMPL [24],or are limited in resolution due to the underlying 3D representation [2, 3, 55, 59]. In contrast, our method is able torepresent articulated shapes at high fidelity without strong
prior assumptions about the object’s shape. This allows us
to better model deformations of objects with more flexible
topology, e.g., humans in clothing.

3. Method
In this section, we first define our representation for thecanonical shape and forward skinning weights. Next, we
introduce our forward mapping and derive the gradients forlearning the canonical shape representation and skinningweights in an end-to-end manner.

3.1. Representation
We represent an articulated object by its shape and skinning weights in canonical space. Similar to classical approaches like SMPL, we split the problem into LBS withpose-independent skinning weights and pose-dependent
non-linear deformations. LBS captures many important aspects of the shape change, thus the pose-dependent model
only has to learn a corrective. This makes training with limited data feasible and aids generalization to unseen poses.
Shape: We use a neural network to predict the occupancy
probability for any input 3D point x in canonical space. To
model pose-dependent local deformations such as wrinkles
or soft tissue, we inject the object pose p as additional input:
fσf : R3 × Rnp → [0, 1].

(1)Here, σf are the network parameters and np is the dimensionality of the pose condition p ∈ Rnp which we specify
in terms of joint angles. The canonical shape is implicitly
defined as the 0.5 level set of the neural function S:
S = {x | fσf (x, p) = 0.5}.

(2)

...

...

...

...

Figure 3: Generating Deformed Shapes with Forward Skinning. Given a query point in deformed space x0 , our method
first finds its canonical correspondences x∗ which satisfy the forward skinning equation (4) via iterative root finding. Multiple
correspondences may exist due to topological changes, which can be reliably found by initializing the root finding algorithmwith multiple starting points derived from the bone transformations. The canonical occupancy network fσf then predicts theoccupancy probabilities at {x∗ } which are finally aggregated to yield the occupancy probability of the query point x0 .
Neural Blend Skinning: We model the non-rigid deformation induced by skeleton changes using linear blend skinning (LBS). Towards this goal, we represent an LBS weight
field in canonical space using a second neural network:wσw : R3 → Rnb ,(3)

where σw are the network parameters and nb denotes thenumber of bones. Following traditional LBS, we enforce
the weights P
w = {w1 , . . . , wnb } of each point x to satisfy
wi ≥ 0 and i wi = 1 using a softmax activation function.
Note that wσw does not depend on the pose p.
Given the LBS weights w of a 3D point x and the bonetransformations B = {B 1 , . . . , B nb } corresponding to aparticular body pose p, the deformed point x0 is determinedby the following convex combination:x0 = dσw (x, B) =nbXwσw ,i (x) · B i · x.

(4)

procedure that is able to retrieve all potential canonical correspondences {x∗i } of any deformed point x0 from the implicitly defined relationship and then composite these correspondences using standard operations for implicit shapecomposition. An overview is provided in Fig. 3.
Correspondence Search: Unlike backward skinning, forward skinning defines the canonical correspondence x∗ of
x0 implicitly as the root of the following equationdσw (x, B) − x0 = 0,(5)which cannot be solved in closed form. The solution of
Eq. (4) can be attained numerically via standard Newton or
quasi-Netwon methods:
xk+1 = xk − (Jk )−1 · (dσw (xk , B) − x0 ),

(6)
0

k

where J is the Jacobian matrix of dσw (x , B) − x . To
prevent computing the Jacobian at each iteration, we apply
Broyden’s method [8] using a low-rank approximation of J.

i=1Note that the canonical pose is a free hyper-parameter.
Empirically, we found the canonical pose shown in Fig. 1 towork well and used it for all experiments on human shapes.

3.2. Differentiable Forward Skinning
To predict the occupancy probability o0x of an observed
3D point x0 in deformed space, we must first determine thecanonical correspondence x∗ of the deformed query x0 inorder to evaluate the occupancy o(x0 , p) = f (x∗ , p) withthe canonical occupancy network.
At the core of our forward skinning approach lies the
problem of finding canonical correspondence x∗ of any
query point x0 . This is non-trivial because (i) their relationship is defined implicitly via Eq. (4) without an analytical
inverse form, and (ii) multiple canonical points might correspond to the same deformed point as space can overlap after
warping (cf. Fig. 3). To address this problem, we propose a

Handling Multiple Correspondences: We find multiple
roots {x∗i } by initializing the optimization procedure withdifferent starting locations and exploiting the local convergence of iterative root finding. The initial states {x0i } arethereby obtained by transforming the deformed point x0
rigidly to the canonical space for each of the nb bones, andthe initial Jacobian matrices {J0i } are the spatial gradientsof the LBS weight field at the corresponding initial states:
0x0i = B −1
i ·x

Ji0 =

∂dσw (x, B)
∂x
x=x0

(7)

i

Initial states that are far from the optima lead to either convergence to one of the optima and can be safely included
for further computation, or divergence, and can therefore beeasily discarded by thresholding. Consequently, we define
the final set of correspondences as:
X ∗ = {x∗i | kdσw (x∗i , B) − x0 k2 < } ,(8)where  is the convergence threshold which we set to 10−5
in our experiments. This allows us to retrieve all canonical correspondences of any deformed point x0 even under
topological changes which induce one-to-many mappings.
Note that if any of the canonical correspondences is occupied, the deformed point x0 is occupied as well. Thus, themaximum over the occupancy probabilities of all canonicalcorrespondences gives the final occupancy prediction:o(x0 , p) = max{fσf (x∗ , p)}.
∗
∗
x ∈X

(9)

This union operator is commonly used to composite independent shapes [45]. Similar to NASA [12], in practice we
use softmax instead of a hard maximum to allow gradients
to back-propagate to all canonical correspondences.

3.3. Training Losses
Our model is trained via minimizing the binary cross entropy loss LBCE (o(x0 , p), ogt (x0 )) between the predictedoccupancy of the deformed points o(x0 , p) and the corresponding ground-truth ogt (x0 ) for all posed 3D meshes ofa single subject. In addition, we apply two auxiliary losses
during the first epoch to bootstrap training. We randomly
sample points along the bones that connect joints in canonical space and encourage their occupancy probabilities to beone. Moreover, we encourage the skinning weights of all
joints to be equal to 0.5 for their respective two neighboringbones. No ground truth skinning weights or part segmentations are required by our method.

3.4. Gradients
During training, we must determine the gradient of theoverall loss L w.r.t. the network parameters σ = {σf , σw }.
For the occupancy network fσf , the gradient is given by∂L ∂o ∂fσf
∂L
·
=
·
∂σf
∂o ∂fσf ∂σf(10)

which can be easily obtained by backpropagating gradients
through the corresponding computation graph. For the LBS
weight field wσw , the gradient is given by∂L∂L ∂o ∂fσf (x∗ ) ∂x∗
=
·
·
·
∂σw
∂o ∂fσf
∂x∗
∂σw

(11)

∗

(12)
∗

∗

∂dσw (x , B) ∂dσw (x , B) ∂x
+
·
=0
(13)
∂σw
∂x∗
∂σw

−1
∂x∗
∂dσw (x∗ , B)
∂dσw (x∗ , B)⇔=−·
. (14)
∗
∂σw
∂x
∂σw
⇔

We first conduct toy experiments on synthetic 2D data
to analyze different methods and model design choices
in a controlled setting. Next, we apply our approach tomodel minimally clothed human bodies and compare it toNASA [12] and other self-implemented baselines. Finally,
we demonstrate that our method can handle clothed humans, generalizing well to unseen poses.

4.1. Datasets
We use the following datasets in our experiments:
2D Stick: We simulate a 2D stick articulated by two bones.
We set the true skinning weights of each point as the theinverse of its distance to each bone. To simulate topology
changes, we include a further rigid object. While this object is separate in canonical space, the two may intersect inposed space and therefore cause topology changes to simulate human self-contact or object interaction.
Minimally Clothed Humans: Following NASA [12], we
use the DFaust [7] subset of AMASS [28] for training andevaluating our model on SMPL meshes of people in minimal clothing. This dataset covers 10 subjects of varying
body shapes. For each subject, we use 10 sequences, fromwhich we randomly select one sequence for validation, using the rest for training. For each frame in a sequence, 20K
points are sampled, among which, half are sampled uniformly in space and half are sampled in near-surface regions
by first applying Poisson disk sampling on the mesh surface,followed by adding isotropic Gaussian noise with σ = 0.01
to the sampled point locations. Besides the “within distribution” evaluation on DFaust, we also include another subset
named PosePrior [1] from AMASS for an “out of distribution” evaluation. This dataset contains natural, more challenging, poses beyond those in DFaust.
Clothed Humans: We use the registered meshes fromCAPE [27] and corresponding joints and bone transformations derived from the accompanied SMPL model registration to train our model. We use 8 subjects from thedataset with different clothing types including short/longlower body clothing and short/long upper body clothing.
We train a model for each subject and clothing condition.

4.2. Baselines

where x∗ is the root as defined in Eq. (8) and the last term
can be analytically obtained via implicit differentiation:dσw (x∗ , B) − x0 = 0

4. Experiments

We consider the following baselines in our evaluation. For“Back-LBS”,“Back-D” and “Pose-ONet” we use the same
training losses and hyperparameters as in our approach.
Pose-Conditioned Occupancy Networks (Pose-ONet):
This baseline extends Occupancy Networks [30] by directly
concatenating the pose input to the occupancy network.
Backward Skinning (Back-LBS): This baseline imple-

Pose-ONet Piecewise Back-LBS

Ours

Extrapolation Accuracy

GTInterpolation Accuracy
100
95

80
60
40

ments the concept of backward skinning similar to [18]. A
network takes a deformed point and pose condition as input and outputs the skinning weights of the deformed point.
The deformed point is then warped back to canonical space
via LBS and the canonical correspondence is fed into thecanonical shape network to query occupancy.
Backward Displacement (Back-D): This baseline directlypredicts the displacement from deformed space to canonicalspace, similar to D-NeRF [18].
NASA: NASA [12] models articulated human bodies as acomposition of multiple parts, each of which transforms
rigidly and deforms according to the pose. Note that in contrast to us, NASA requires ground-truth skinning weights
for surface points as supervision. We use the official NASA
implementation provided by the authors.
Piecewise: For evaluation on the 2D toy dataset, we createda variant of NASA for 2D which we refer to as “Piecewise”.

4.3. Results on 2D Stick Dataset
For our results on the simple 2D stick dataset, we do not
use local pose-conditioning as the shape does not locally
deform with pose. We consider the following three settings:
Extrapolation: An essential requirement for articulated
models is the ability to deform into arbitrary poses. In thissetting, we generate training data using the articulated 2D
stick with joint angles from the interval [−60, 60]◦ . At test
time, the models are tasked to generate deformed shapeswith larger joint angles in [−120, 60]◦ ∪ [60, 120]◦ . Fig. 4
and Fig. 5 (left) show our results. While our forward skinning model follows the ground truth closely, Pose-ONetfails to generate a meaningful structure as it learns a direct
mapping from poses to shapes and thus cannot produce un-

8580Pose O-Net
Piecewise
Back-LBS
Forward (Ours)

75
70

20
0

Figure 4: Qualitative Results on 2D Toy Experiment.
Row 1 (2 bones): Our deformed shape appears similar to theground-truth. In contrast, Back-LBS and Pose-ONet produce distorted shapes. The piecewise rigid model (Piecewise) leads to artifacts around bone intersections. Row 2(2 bones + 1 rigid object): Our forward skinning algorithm
can handle topology changes while artifacts at the intersection are noticeable in the result of the backward skinningbaseline (Back-LBS).

90

IoU [%]

IoU [%]

2 bones
1 object

2 bones

100

65
60

with
without
Toplogical Change

5

10

20

40

60

Training Angle Step Size [degree]

Figure 5: Quantitative Results on 2D Toy Experiment.
Left: For pose extrapolation, our method outperforms allbaselines on both test cases, with and without topologicalchanges. Right: When interpolating, the performance gap
increases as training angles are sampled more sparsely.
seen shapes. By disentangling deformations from shapes,
Back-LBS preserves the structure better, but the learned
pose-dependent skinning weights do not generalize. The
piecewise model (Piecewise) generates the correct global
pose configuration but exhibits visible artifacts as the rigidity assumption is violated at the joint.
Topological Changes: To simulate topological changes,
we include a rigid object but otherwise keep the setting thesame as in the previous experiment. Changing topology ischallenging for Back-LBS since it is not able to model oneto-many backward correspondences. To compensate for
this, the occupancy field gets distorted as shown in Fig. 4. In
contrast, our model gracefully handles topological changes,
as also shown quantitatively in Fig. 5 (left).
Interpolation: To assess interpolation performance, we
evaluate the accuracy of the generated shapes with angles
sampled continuously from [−60, 60]◦ while increasing thesampling step size of the training poses. As shown in Fig. 5(right), with increasing difficulty, the gap between the baseline methods (Pose O-Net and Back-LBS) and ours becomes larger. An exception is the piecewise model (Piecewise), whose performance is invariant to the training sample
density, but instead exhibits artifacts at part intersections.

4.4. Results on Minimally Clothed Humans
Following NASA [12], we now consider the more challenging case of modeling articulated 3D human bodies. Human bodies are challenging due to their complex skeletal
structure and local deformations that are non-linearly dependent on the bone transformations. While NASA requires ground-truth skinning weights as additional supervision, our method does not require such knowledge.
Within Distribution: Overall, all methods perform wellin this relatively simple setting, as shown in Tab. 1. However, our method still provides an improvement over all
baselines. In particular, compared to NASA [12], we im-Within Distribution

Out of Distribution

Subject

P.-ONetBack-DIoU bbox
Back-LBS

NASA

Ours

P.-ONetBack-DIoU surface
Back-LBS

NASA

Ours

P.-ONetBack-DIoU bbox
Back-LBS

NASA

Ours

P.-ONetBack-DIoU surface
Back-LBS

NASA

Ours

50002
50004
50007
50009
50020
50021
50022
50025
50026
50027
Avg.

84.80%
80.09%
88.31%
71.67%
69.21%
79.30%
86.60%
80.14%
79.39%
73.91%
79.34%

87.89%
84.52%
89.09%
74.75%
73.37%
79.48%
90.59%
79.81%
84.58%
76.71%
82.08%

47.34%
93.53%
50.13%
65.36%
93.04%
96.86%
97.60%
95.28%
97.32%
80.33%
81.68%

96.56%
96.31%
96.72%
94.94%
95.75%
95.92%
97.94%
95.50%
96.65%
95.53%
96.14%

97.50%
97.84%
97.96%
96.68%
96.27%
96.86%
97.96%
97.54%
97.64%
96.80%
97.31%

63.86%
57.79%
67.14%
50.87%
48.73%
57.80%
66.82%
59.47%
60.52%
53.91%
58.61%

66.42%
59.93%
68.02%
53.96%
53.72%
64.02%
74.27%
60.37%
64.07%
57.46%
62.22%

85.41%
88.07%
83.46%
85.38%
86.03%
89.96%
93.51%
87.33%
90.17%
85.04%
87.44%

84.02%
85.45%
86.28%
84.52%
87.57%
87.01%
91.91%
86.19%
87.72%
86.13%
86.98%

89.57%
91.16%
91.02%
89.19%
88.81%
90.16%
92.06%
91.25%
91.09%
89.47%
90.38%

60.61%
55.44%
40.53%
38.17%
42.66%
45.50%
52.17%
52.78%
56.09%
48.22%
49.21%

70.02%
64.63%
59.68%
50.18%
52.43%
58.99%
60.41%
56.93%
64.33%
57.00%
59.46%

73.42%
65.17%
62.66%
63.34%
64.98%
69.89%
67.83%
68.91%
65.20%
67.86%
66.93%

87.71%
86.01%
80.22%
78.15%
83.06%
81.80%
87.54%
83.14%
84.58%
83.97%
83.16%

94.51%
95.61%
93.99%
91.22%
93.57%
93.76%
94.67%
94.48%
94.13%
93.76%
93.97%

31.94%
34.26%
17.80%
23.24%
26.56%
29.07%
33.00%
31.37%
32.07%
27.56%
28.69%

39.84%
38.62%
34.76%
30.85%
33.62%
37.19%
34.71%
34.49%
37.71%
32.56%
35.44%

71.01%
69.43%
59.53%
64.40%
68.24%
61.69%
73.46%
70.60%
71.85%
70.55%
68.93%

60.25%
62.53%
51.82%
55.86%
62.01%
65.49%
70.23%
60.88%
59.78%
61.82%
60.21%

79.75%
83.34%
77.08%
75.84%
81.37%
81.49%
83.37%
82.48%
80.01%
81.81%
80.65%

Ground Truth

Ours

NASA

Back-LBS

Back-D

Pose-ONet

Table 1: Quantitative Results on Minimally Clothed Humans. The mean IoU of uniformly sampled points in space (IoUbbox) and points near the surface (IoU surface) are reported. Our method outperforms all baselines including NASA [12].
Improvements are more pronounced for points near the surface, and for poses outside the training distribution.

Within Distribution
Out of Distribution
Figure 6: Qualitative Results on Minimally Clothed Humans. Our method produces results similar to the ground-truth
with correct body pose and plausible local details, both for mild poses within the training distribution and more extreme
poses. In contrast, the baseline methods suffer from various artifacts including incorrect poses (Pose-ONet), degenerate
shapes (Pose-ONet, Back-D, LBS), and discontinuities near joints (NASA) which become more severe for unseen poses.

Out of Distribution: In this setting, we test the trainedmodels on a different dataset, PosePrior [1], to assess theperformance in more realistic settings, where poses can be
far from those in the training set. Similar to the observations in the 2D toy setting, unseen poses may cause drastic
performance degradation to the baseline methods as shownin Tab. 1. In contrast, our method degrades gracefully
despite test poses being drastically different from training
poses and very challenging. Hence, the performance gap on
IoU surface between our method and NASA increases from
4.6% to 20.4%. As can be seen in Fig. 6, our method generates natural shapes for the given poses while NASA fails togenerate correctives at bone intersections for unseen poses,
leading to noticeable artifacts. Pose-ONet and Back-D failto generate meaningful shapes and Back-LBS produces distorted bodies due to incorrect skinning weights.
Learned Skinning Weights: We demonstrate our learned
skinning weights in Fig. 1. Our model learns plausible skinning weights with smooth transitions for all moving bodyparts, reflecting the correct body part assignment. More results can be found in the supplementary material.

4.5. Results on Clothed Humans
Our method can also be applied to modeling clothedhumans. We train SNARF using meshes from the CAPE
dataset. The results are shown in Fig. 7. Our method isable to model different clothing types with flexible topology and generates realistic results in novel poses with plausible local details, such as wrinkles. The clothing deforms
naturally with the body pose, except for very extreme poses
where prediction quality degrades gracefully. Fig. 8 shows
the canonical shapes corresponding to different body poses.

Posed

prove the IoU of uniformly sampled points by 1.2% and
the IoU of near-surface points by 4.6%. This improvement
can also be observed in the qualitative results Fig. 6. Our
method produces bodies with smooth surfaces and correct
poses. In contrast, NASA suffers from discontinuous artifacts near joints. Back-D, Back-LBS and Pose-ONet sufferfrom missing body parts.

Canonical

Figure 7: Qualitative Results for Clothed Humans. Our method can model 3D clothed humans in various clothing types,
with rich details including wrinkles, and in novel poses. Moreover, our method faithfully learns the non-linear relationshipbetween cloth deformations and body poses. On the right, we show a failure case where the cloth does not fall naturally foran extreme, unseen pose. However, note how our method still degrades gracefully in this situation.

Figure 8: Pose-dependent non-linear deformations (correctives) in canonical space. The heatmaps show the differences (yellow=large, zoom in for more details) between
the canonical shape for the current pose and the one for thecanonical pose, demonstrating the flexibility of the deformations that can be captured by our model.

5. Conclusion
In this paper, we proposed a differentiable forwardskinning model for articulating neural implicit surfaces.
Our method learns continuous pose-conditioned shapes andskinning weights from meshes and is able to generate plausible shapes in nearly arbitrary poses. We obtain state-ofthe-art results on articulated neural implicit representationsfor 3D human bodies and demonstrate significantly better
generalization to unseen poses than the baselines. We show
SotA results on challenging cases of (clothed) 3D humans
with diverse shapes and poses. In future work, we plan toextend our method to learn across subjects and from images
only using differentiable rendering [35].
Acknowledgements: Xu Chen and Yufeng Zheng were
supported by the Max Planck ETH Center for Learning Systems. Andreas Geiger was supported by the DFG EXC
number 2064/1 - project number 390727645.
Disclosure: MJB has received research gift funds from
Adobe, Intel, Nvidia, Facebook, and Amazon. While MJB
is a part-time employee of Amazon, his research was performed solely at, and funded solely by, Max Planck. MJB
has financial interests in Amazon, Datagen Technologies,
and Meshcapade GmbH.

References[1] Ijaz Akhter and Michael J. Black. Pose-conditioned joint angle limits for 3D human pose reconstruction. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
2015. 5, 8
[2] Thiemo Alldieck, Marcus Magnor, Weipeng Xu, Christian
Theobalt, and Gerard Pons-Moll. Detailed human avatars
from monocular video. In Proc. of the International Conf.
on 3D Vision (3DV), 2018. 3[3] Thiemo Alldieck, Gerard Pons-Moll, Christian Theobalt,
and Marcus Magnor. Tex2Shape: Detailed full human bodygeometry from a single image. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019. 3[4] Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. SCAPE:
Shape completion and animation of people. ACM Trans. on
Graphics, 24(3):408–416, 2005. 3[5] Matan Atzmon and Yaron Lipman. SAL: Sign agnostic
learning of shapes from raw data. In Proc. IEEE Conf. on
Computer Vision and Pattern Recognition (CVPR), 2020. 3[6] Bharat Lal Bhatnagar, Cristian Sminchisescu, Christian
Theobalt, and Gerard Pons-Moll. Loopreg: Self-supervised
learning of implicit surface correspondences, pose and shapefor 3D human mesh registration. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 3[7] Federica Bogo, Javier Romero, Gerard Pons-Moll, andMichael J. Black. Dynamic FAUST: Registering human bodies in motion. In Proc. IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), 2017. 5[8] Charles G Broyden. A class of methods for solving nonlinear simultaneous equations. Mathematics of computation,
19(92):577–593, 1965. 4[9] Andrei Burov, Matthias Nießner, and Justus Thies. Dynamic
surface function networks for clothed human bodies. In Proc.
of the IEEE International Conf. on Computer Vision (ICCV),
2021. 3[10] Zhiqin Chen and Hao Zhang. Learning implicit fields forgenerative shape modeling. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019. 1[11] Julian Chibane, Thiemo Alldieck, and Gerard Pons-Moll.
Implicit functions in feature space for 3D shape reconstruction and completion. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020. 3[12] Boyang Deng, JP Lewis, Timothy Jeruzalski, Gerard PonsMoll, Geoffrey Hinton, Mohammad Norouzi, and Andrea
Tagliasacchi. Neural articulated shape approximation. In
Proc. of the European Conf. on Computer Vision (ECCV),
2020. 2, 3, 5, 6, 7
[13] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, andYaron Lipman. Implicit geometric regularization for learning shapes. In Proc. of the International Conf. on Machinelearning (ICML), 2020. 3[14] Marc Habermann, Lingjie Liu, Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll, and Christian Theobalt. Real-time
deep dynamic characters. ACM Trans. on Graphics, 40(4),aug 2021. 3[15] Nils Hasler, Thorsten Thormählen, Bodo Rosenhahn, andHans-Peter Seidel. Learning skeletons for shape and pose.
ACM Trans. on Graphics, page 23–30, 2010. 2[16] Tong He, John Collomosse, Hailin Jin, and Stefano Soatto.
Geo-PIFu: Geometry and pixel aligned implicit functions for
single-view human reconstruction. In Advances in Neural
Information Processing Systems (NeurIPS), 2020. 3[17] Doug L. James and Christopher D. Twigg. Skinning meshanimations. ACM Trans. on Graphics, 24(3):399, 2005. 2[18] Timothy Jeruzalski, David IW Levin, Alec Jacobson, Paul
Lalonde, Mohammad Norouzi, and Andrea Tagliasacchi.
NiLBS: Neural inverse linear blend skinning. arXiv.org,
2004.05980, 2020. 3, 6
[19] B. Jiang, J. Zhang, J. Cai, and J. Zheng. Disentangled humanbody embedding based on deep hierarchical neural network.
IEEE Transactions on Visualization and Computer Graphics,
26(8):2560–2575, 2020. 2[20] Ladislav Kavan, Steven Collins, Jiri Zara, and CarolO’Sullivan. Geometric skinning with approximate dual
quaternion blending. ACM Trans. on Graphics, 27(4):105,
2008. 2[21] J. P. Lewis, Matt Cordner, and Nickson Fong. Pose space
deformation: A unified approach to shape interpolation andskeleton-driven deformation. ACM Trans. on Graphics,
pages 165–172, 2000. 2[22] Lingjie Liu, Marc Habermann, Viktor Rudnev, Kripasindhu
Sarkar, Jiatao Gu, and Christian Theobalt. Neural actor:
Neural free-view synthesis of human actors with pose control. arXiv.org, 2004.05980, 2021. 3[23] Lijuan Liu, Youyi Zheng, Di Tang, Yi Yuan, Changjie Fan,and Kun Zhou. Neuroskinning: Automatic skin binding
for production characters with deep graph networks. ACM
Trans. on Graphics, 38(4), July 2019. 2[24] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard
Pons-Moll, and Michael J. Black. SMPL: A skinned multiperson linear model. ACM Trans. on Graphics, 2015. 2, 3
[25] Qianli Ma, Shunsuke Saito, Jinlong Yang, Siyu Tang, andMichael J. Black. SCALE: Modeling clothed humans witha surface codec of articulated local elements. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
2021. 3[26] Qianli Ma, Jinlong Yang, , Siyu Tang, and Michael J Black.
The power of points for modeling humans in clothing. In
Proc. of the IEEE International Conf. on Computer Vision(ICCV), 2021. 3[27] Qianli Ma, Jinlong Yang, Anurag Ranjan, Sergi Pujades,
Gerard Pons-Moll, Siyu Tang, and Michael J. Black. Learning to dress 3D people in generative clothing. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
2020. 5[28] Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black. AMASS: Archive of
motion capture as surface shapes. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019. 5[29] Bruce Merry, Patrick Marais, and James Gain. Animation
space: A truly linear framework for character animation.
ACM Trans. on Graphics, 25(4):1400–1423, 2006. 2[30] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks:
Learning 3D reconstruction in function space. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
2019. 1, 3, 5

[31] Mateusz Michalkiewicz, Jhony K Pontes, Dominic Jack,
Mahsa Baktashmotlagh, and Anders Eriksson. Deep levelsets: Implicit surface representations for 3D shape inference.
arXiv.org, 1901.06802, 2019. 1[32] Marko Mihajlovic, Yan Zhang, Michael J Black, and Siyu
Tang. LEAP: Learning articulated occupancy of people. In
Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021. 3[33] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik,
Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF:
Representing scenes as neural radiance fields for view synthesis. In Proc. of the European Conf. on Computer Vision(ECCV), 2020. 3[34] Michael Niemeyer, Lars Mescheder, Michael Oechsle, andAndreas Geiger. Occupancy flow: 4D reconstruction bylearning particle dynamics. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019. 2, 3[35] Michael Niemeyer, Lars Mescheder, Michael Oechsle, andAndreas Geiger. Differentiable volumetric rendering: Learning implicit 3D representations without 3D supervision. In
Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020. 3, 8[36] Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo
Strauss, and Andreas Geiger. Texture fields: Learning texture representations in function space. In Proc. of the IEEE
International Conf. on Computer Vision (ICCV), 2019. 3[37] Ahmed A. A. Osman, Timo Bolkart, and Michael J. Black.
Star: Sparse trained articulated human body regressor. In
Proc. of the European Conf. on Computer Vision (ECCV),
2020. 2[38] Jeong Joon Park, Peter Florence, Julian Straub, Richard
Newcombe, and Steven Lovegrove. DeepSDF: Learning
continuous signed distance functions for shape representation. In Proc. IEEE Conf. on Computer Vision and Pattern
Recognition (CVPR), 2019. 1, 3[39] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien
Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla. Deformable neural radiance fields.
arXiv.org, 2011.12948, 2020. 2, 3[40] Sida Peng, Junting Dong, Qianqian Wang, Shangzhan
Zhang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Animatable neural radiance fields for human body modeling. In
Proc. of the IEEE International Conf. on Computer Vision(ICCV), 2021. 3[41] Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc
Pollefeys, and Andreas Geiger. Convolutional occupancy
networks. In Proc. of the European Conf. on Computer Vision (ECCV), 2020. 3[42] Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang,
Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body:
Implicit neural representations with structured latent codes
for novel view synthesis of dynamic humans. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
2021. 3[43] Albert Pumarola, Enric Corona, Gerard Pons-Moll, andFrancesc Moreno-Noguer. D-NeRF: Neural radiance fields
for dynamic scenes. In Proc. IEEE Conf. on Computer Vision
and Pattern Recognition (CVPR), 2021. 2, 3
[44] Amit Raj, Julian Tanke, James Hays, Minh Vo, Carsten Stoll,and Christoph Lassner. ANR-articulated neural rendering for
virtual avatars. In Proc. IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), 2021. 3[45] Antonio Ricci. A constructive geometry for computer graphics. The Computer Journal, 16(2):157–160, 1973. 5[46] C. Rouet and J. Lewis. Method and apparatus for creating
lifelike digital representations of computer animated objects
by providing corrective enveloping, Mar. 16 1999. US Patent
5,883,638. 2[47] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. PIFu: Pixel-aligned
implicit function for high-resolution clothed human digitization. In Proc. of the IEEE International Conf. on Computer
Vision (ICCV), 2019. 3[48] Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul
Joo. PIFuHD: Multi-level pixel-aligned implicit function for
high-resolution 3D human digitization. In Proc. IEEE Conf.
on Computer Vision and Pattern Recognition (CVPR), 2020.
3[49] Shunsuke Saito, Jinlong Yang, Qianli Ma, and Michael J.
Black. SCANimate: Weakly supervised learning of skinned
clothed avatar networks. In Proc. IEEE Conf. on Computer
Vision and Pattern Recognition (CVPR), 2021. 3[50] Aliaksandra Shysheya, Egor Zakharov, Kara-Ali Aliev,
Renat Bashirov, Egor Burkov, Karim Iskakov, Aleksei
Ivakhnenko, Yury Malkov, Igor Pasechnik, Dmitry Ulyanov,
et al. Textured neural avatars. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019. 3[51] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3Dstructure-aware neural scene representations. In Advancesin Neural Information Processing Systems (NeurIPS), 2019.
3[52] Shaofei Wang, Andreas Geiger, and Siyu Tang. Locallyaware piecewise transformation fields for 3D human meshregistration. In Proc. IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), 2021. 3[53] Shaofei Wang, Marko Mihajlovic, Qianli Ma, Andreas
Geiger, and Siyu Tang. MetaAvatar: Learning animatable
clothed human models from few depth images. arXiv.org,
2004.05980, 2021. 3[54] Xiaohuan Corina Wang and Cary Phillips. Multi-weightenveloping: least-squares approximation techniques forskin animation. In Proceedings of the 2002 ACM SIGGRAPH/Eurographics symposium on Computer animation,
pages 129–138, 2002. 2[55] Chung-Yi Weng, Brian Curless, and Ira KemelmacherShlizerman. Vid2actor: Free-viewpoint animatable person
synthesis from video in the wild. arXiv.org, 2012.12884,
2020. 3[56] Hongyi Xu, Eduard Gabriel Bazavan, Andrei Zanfir,
William T Freeman, Rahul Sukthankar, and Cristian Sminchisescu. GHUM & GHUML: Generative 3D human shapeand articulated pose models. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020. 2[57] Zhan Xu, Yang Zhou, Evangelos Kalogerakis, Chris Landreth, and Karan Singh. RigNet: Neural rigging for articulated characters. ACM Trans. on Graphics, 39(4), 2020. 2[58] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, MatanAtzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. In Advances in Neural Information Processing
Systems (NeurIPS), 2020. 3[59] Zerong Zheng, Tao Yu, Yixuan Wei, Qionghai Dai, andYebin Liu. Deephuman: 3D human reconstruction from asingle image. In Proc. of the IEEE International Conf. on
Computer Vision (ICCV), 2019. 3[60] Keyang Zhou, Bharat Lal Bhatnagar, and Gerard PonsMoll. Unsupervised shape and pose disentanglement for 3D
meshes. In Proc. of the European Conf. on Computer Vision(ECCV), 2020. 2