Supplementary for Locally Aware Piecewise Transformation Fields
for 3D Human Mesh Registration

1ETH ZürichShaofei Wang1 , Andreas Geiger2,3 , Siyu Tang1
2Max Planck Institute for Intelligent Systems, Tübingen 3 University of Tübingen
Abstract

In this supplementary material, we first elaborate on the difference between the original NASA [5] and our modified NASA.
Next, we present detailed model architectures of PTF-Piecewise and PTF-FC which are introduced in the main paper. Thenwe present additional quantitative and qualitative results on the CAPE dataset, as well additional qualitative results on theBUFF dataset. Lastly, we discuss limitations of our approach.

1. Difference Between the Original NASA [5] and Our Modified NASAIn the original NASA [5] paper, the bone transformations {B0b } are defined as:
B0b = Gb (β, θ)Y Rb0 (θ) tb0 − t ¯0 b
Gb (β, θ) =010(1)(2)b ∈A(b){tb } = J (β)(3)

The {B0b } are relative to the origin, whereas in our formulation we have:
Bb = Gb (β, θ)Gb (β, 0)−1Y Rb0 (θ) tb0 − t ¯0 bGb (β, θ) =010(4)(5)

b ∈A(b){tb } = J (β)(6)

Eq. (4)-(6) are the same as Eq. (2)-(4) in the main paper. The {Bb } are relative to the rest-pose joints.

2. Model Architectures
We provide illustrations for our model architectures in Fig. 1. Note that throughout our implementations, we use 1Dconvolutions with kernel size 1 to substitute the fully-connected layers used in typical occupancy networks. This enables
us to use the off-the-shelf implementation of grouped convolutions [7] in PyTorch [8] to represent piecewise occupancy
classifiers {Ōωb } and PTF {T̄ψb }.
Fig. 1a shows our fully-piecewise model, PTF-Piecewise. The key observation we make here is that, the pre-activation
occupancy logits {σb } ∈ R3×B can be used to compute the softmax probability over parts. To construct inputs to theoccupancy classifier module, we concatenate the local point cloud feature cx to each x̂b , ∀b ∈ {1, · · · , B}, thus a piecewise
occupancy function Ōωb of bone b takes the concatenation of cx and x̂b as the input and outputs the occupancy logits σ̂b . We
set B = 24 for PTF-Piecewise.
Fig. 1b shows our PTF-FC model. Here we use a similar structure as in IPNet [2] which predicts part probabilities with aseparate part classifier. The difference between IPNet and our PTF-FC is that we apply PTF to query point x before feeding
it to the occupancy classifier. The first layer (in blue) of the PTF module takes the concatenation of cx and x as the input andoutputs a 128B dimensional feature for later stages. We set B = 14 for PTF-FC.

1×BePx33×B
x{x̂b }

P◦3e256 256
BPart Classifierx̂cx
M1cx{σ̂b }

|cx |

3×B

e

M2|cx |p̂
1×BeP

e3
◦

380 × 80 × 80 × 3×B
B
B
B
PTF
ePp̂

Px̂Pô

{x̂b }
3×B
128B 128× 128× 3×B
B
BPTF
◦ô
380 × 80 × 80 × 3×B
B
B
B
Occupancy Classifier3{ôb }
3×B
128× 128× 128× 3×B
B
B
B
Occupancy Classifier(a) PTF-Piecewise(b) PTF-FCFigure 1: Illustration for model architectures. Blue rectangles indicate 1D convolution layers, green rectangles indicate
1D group-convolutions with B groups, Pe e squares indicate the softmax function, ◦ indicates the Hadamard product withP
broadcasting,squares indicate summation over part dimension, M1 and M2 squares indicate max over the occupancy
dimension and part dimension, respectively. Both PTF-Piecewise and PTF-FC take a query point x and the corresponding
local point feature cx as input, and output a multi-class occupancy probability ô and the rest-pose correspondence x̂.
MethodOuter Err. Max Outer Err.
IPNet
28.2 mm562.7 mm
IPNet-128
26.4 mm564.4 mm
Stitched Puppet [11] 36.1 mm
454.9 mm
3D-CODED [6]23.7 mm614.8 mm
PTF-FC23.1 mm76.6 mm
PTF-FC-128
21.2 mm83.6 mm

Inner Err. Max Inner Err.
28.3 mm
544.2 mm
26.9 mm
546.5 mm
NANANANA
23.1 mm88.9 mm
121.4 mm82.3 mm

1 evaluation on the CAPE dataset. IPNet-128 and PTF-FC-128 indicate IPNet model and ourTable 1: Additional registration
model trained with encoder resolution of 128 × 3, where size of the input point cloud remains 5K.

3. Additional Quantitative Results on the CAPE Dataset
In this section we compare our approach to other related baselines, namely 3D-CODED [6] and the Stitched Puppet [11].
Note that to our best knowledge, IPNet is the only published method that addresses automatic model-fitting of both undercloth body and clothed body, given unoriented, sparse point-clouds of clothed humans. Both IPNet and ours can produce
controllable parameters and clothed avatars from input point clouds.
3D-CODED [6] only focuses on correspondence prediction. Their method can not produce controllable parameters forunder-cloth or clothed body models. Furthermore, 3D-CODED uses exhaustive initial rotation search and is allowed todeform the template freely, while we can only optimize for θ, β, t and D, where θ, β are limited to SMPL parameter space,and D is in the A-pose and regularized to be small.
The Stitched Puppet [11] tackles model-fitting as a pure optimization problem via belief propagation. Since [11] is basedon SCAPE [1] topology, we fit it to FAUST [4] scans with the neutral pose, and then obtain correspondence from SCAPE
vertices to SMPL vertices via barycentric interpolation.
We report the numbers in Tab. 1. For clothing surface registration, 3D-CODED produces slightly inferior results than ours

(a) IPNet surface(b) IPNet SMPL(c) IPNet SMPL+D(d) Ours surface(e) Ours SMPL(f) Ours SMPL+DFigure 2: One common failure case of IPNet [2] happens when the global orientation deviates significantly from zero. Thisoften results in catastrophic failures in the subsequent (b) SMPL fit and (c) SMPL+D fit.

(a) IPNet surface(b) IPNet SMPL(c) IPNet SMPL+D(d) Ours surface(e) Ours SMPL(f) Ours SMPL+DFigure 3: Another type of failure cases of IPNet [2] is caused by poor generalizability. As can be seen in (a), although
IPNet has been trained using random augmentation to global orientation, it still struggles to generalize to the case in which
the person’s two arms are roughly aligned with the z-axis. IPNet misclassifies both arms of the person as the left-arm(same color), this results in catastrophic failure in the subsequent (b) SMPL fit and (c) SMPL+D fit. In (d) our model
generalizes better and correctly distinguishes the left-arm and the right-arm under this rare pose, resulting in much moreaccurate registrations (e+f).

although it is less constrained. The results from the Stitched Puppet are inferior to all others probably due to 1) the sensitivityof belief propagation to noise and 2) the fact that clothed surfaces vary more than SCAPE can capture. Most importantly,
our approach completely avoids catastrophic failures (> 100 mm error) that happen for the baselines, demonstrating strong
generalization.

4. Additional Qualitative Results on the CAPE Dataset
We show additional qualitative results on the CAPE dataset in Fig. 2 and Fig. 3. We observe that IPNet [2] often fails
when the global orientation deviates too much from zero. As can be seen in Fig. 2, even if the local poses are relatively
simple, IPNet still fails catastrophically; this is most likely due to the fact that, in order to make the optimization stable, theoptimization objective often has regularization terms that penalize poses which deviate too much from the mean-pose. This
choice is statistically meaningful, but without pose initialization, it will also make it impossible for the optimizer to converge
to rare poses that are realistic but deviate too much from the mean-pose.
Another type of failure cases of IPNet is caused by its network’s poor generalizability. Fig. 3 shows that IPNet fails tocorrectly distinguish between the left-arm and the right-arm under a rare pose. This is because IPNet learns the occupancy
functions of the two arms in posed space, and thus these occupancy functions need to memorize all possible locations ofarms in posed space. On the other hand, our model learns to canonicalize points before the occupancy classification, thus theoccupancy functions of arms only need to memorize a small region in rest-pose space. This results in better generalizability
and our model correctly distinguishes the left-arm and the left-arm in this case.

(a) Raw scan(b) IPNet SMPL(c) IPNet SMPL+D(d) Ours SMPL(e) Ours SMPL+D(f) Ours reposed

Figure 4: More qualitative results on the BUFF dataset. Note that our SMPLD/SMPL+D fits perform consistently better thanIPNet [2], especially around faces and hands

GTOurs surface

Ours SMPL

GT

Ours surface

Ours SMPL

Figure 5: Failure cases: we note that implicit surface reconstruction often fails in self-contact or near-self-contact scenarios.
This leads to failures in registration.

5. Additional Qualitative Results on the BUFF Dataset
We present more qualitative results on the BUFF dataset in Fig. 4. Note that the BUFF dataset [10] does not containground-truth pose parameters, thus it is not possible to evaluate quantitatively on registration errors as we did on the CAPE
dataset. These qualitative results are meant to demonstrate generalization performance of our trained model to real scans,even though the model is only trained on synthetically sampled point clouds from registered dressed people.

6. Limitations
Our approach often fails in self-contact or near-self-contact scenarios. We show typical failure cases in Fig 5. Anotherlimitation of our approach is that it requires fully-supervised training on accurate surface registration. This kind of data isvery hard to acquire in practice, thus limiting the scalability of our approach. A straightforward improvement would be
integrating the self-supervised loop of [3] into our pipeline, or utilizing the weakly supervised approach of [9] to generate
training data using registered under-cloth SMPL body.

References[1] Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. Scape: shape completion
and animation of people. ACM Transactions on Graphics, (Proc. SIGGRAPH), 24, 2005. 2[2] Bharat Lal Bhatnagar, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll. Combining implicit function learning andparametric models for 3d human reconstruction. In Proc. of ECCV, 2020. 1, 3, 4
[3] Bharat Lal Bhatnagar, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll. Loopreg: Self-supervised learning ofimplicit surface correspondences, pose and shape for 3d human mesh registration. In Proc. of NeurIPS, 2020. 5[4] Federica Bogo, Javier Romero, Matthew Loper, and Michael J. Black. FAUST: Dataset and evaluation for 3D mesh registration. InProc. of CVPR, 2014. 2[5] Boyang Deng, JP Lewis, Timothy Jeruzalski, Gerard Pons-Moll, Geoffrey Hinton, Mohammad Norouzi, and Andrea Tagliasacchi.
Nasa: Neural articulated shape approximation. In Proc. of ECCV, 2020. 1[6] Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan Russell, and Mathieu Aubry. 3d-coded : 3d correspondences by deepdeformation. In Proc. of ECCV, 2018. 2[7] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Proc.
of NeurIPS, 2012. 1[8] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia
Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank
Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deeplearning library. In Proc. of NeurIPS, 2019. 1[9] Shunsuke Saito, Jinlong Yang, Qianli Ma, and Michael J. Black. SCANimate: Weakly supervised learning of skinned clothed avatarnetworks. In Proc. of CVPR, 2021. 5[10] Chao Zhang, Sergi Pujades, Michael J. Black, and Gerard Pons-Moll. Detailed, accurate, human shape estimation from clothed 3dscan sequences. In Proc. of CVPR, 2017. 4[11] Silvia Zuffi and Michael J. Black. The stitched puppet: A graphical model of 3D human shape and pose. In Proc. of CVPR, 2015. 2