Neural Localizer Fields for Continuous 3D Human Pose and Shape Estimation
Authors: István Sárándi
Gerard Pons-Moll
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D human pose estimation
human shape estimation
computer vision
human mesh recovery
Primary area: machine_vision
TL;DR: We train a state-of-the-art generalist human pose and shape estimation model that can localize any point of the human body.

Abstract: With the explosive growth of available training data, single-image 3D human modeling is ahead of a transition to a data-centric paradigm.
A key to successfully exploiting data scale is to design flexible models that can be supervised from various heterogeneous data sources produced by different researchers or vendors.
To this end, we propose a simple yet powerful paradigm for seamlessly unifying different human pose and shape-related tasks and datasets.
Our formulation is centered on the ability - both at training and test time - to query any arbitrary point of the human volume, and obtain its estimated location in 3D.
We achieve this by learning a continuous neural field of body point localizer functions, each of which is a differently parameterized 3D heatmap-based convolutional point localizer (detector).
For generating parametric output, we propose an efficient post-processing step for fitting SMPL-family body models to nonparametric joint and vertex predictions.
With this approach, we can naturally exploit differently annotated data sources including mesh, 2D/3D skeleton and dense pose, without having to convert between them, and thereby train large-scale 3D human mesh and skeleton estimation models that outperform the state-of-the-art on several public benchmarks including 3DPW, EMDB, EHF, SSP-3D and AGORA by a considerable margin.
We release our code and models to foster downstream research.

Neural Localizer Fields for Continuous 3D Human Pose and Shape Estimation
István Sárándi! Gerard Pons-Moll
University of Tübingen, Germany, TTbbingen AI Center, Germany. Max Planck Institute for Informatics, Saarland Informatics Campus, Germany https://istvansarandde com/n1f
Abstract
With the explosive growth of available training data, single-image 3D human modeling is ahead of transition data-centric paradigm. A key successfully exploiting data scale is to design flexible models that can be supervised from various heterogeneous data sources produced by different researchers or vendors. To this end, we propose simple yet powerful paradigm for seamlessly unifying different human pose and shape-related tasks and datasets. Our formulation is centered on the ability both at training and test time to query any arbitrary point of the human volume, and obtain estimated location in 3D. We achieve this by learning a continuous neural field of body point localizer functions, each which is a differently parameterized 3D heatmap-based convolutional point localizer (detector). For generating parametric output, we propose efficient post-processing step for fitting SMPL-famill body models to nonparametric joint and vertex predictions. With this approach, annotated data sources including mesh 2D/3D skeleton and dense pose, without having to convert between them, and thereby train large-scale 3D human mesh and skeleton estimation models that considerably outperform the state-of-thed oftte-of-the-art on several public benchmarks including 3DPW, EMDB, EHF, SSP-3D and AGORA.
naturally exploit differently
Introduction
Along with the wider field of computer vision, the human pose and shape estimation community has recently started a transition towards leveraging large-scale training data to produce robust models [93, 30, 12, 83, 60]. When assembling large training sets from different sources, it is inevitable that they will use different types annotations, including different parametric mesh models, 3D skeletal joints and markers from different motion capture vendors and manual 2D keypoint annotations. Re-annotation to a single format for training is a costly and error-prone process, and therefore difficult to scale. Similarly, current models trained to output specific format, such pre-defined joints, keypoints, or parametric body meshes, which limiting for downstream applications which require other formats. We therefore argue that a good model design should be able to ingest and output highly heterogeneous forms of annotations a unified manner.
Despite the different kinds of labels, these tasks all share a common underlying factor: the correspondence problem of mapping from canonical human body space to the observation space of the camera. Hence, if model could localize arbitrary human points (both surface and body-internal ones, e.g., joints), its predictions could be supervised from any human-centric data source with point labels and could be customized to new landmark sets even after deployment. In this work, we propose a method that is able to achieve exactly this.
To localize individual points, we leverage the machinery of volumetric heatmap estimation with soft-argmax decoding, which has worked well in skeletal keypoint estimation methods [86, 104, 92]. However, generalizing this to arbitrary points is nontrivial. In prior work, each body joint heatmap is
38th Conference on Neural Information Processing Systems (NeurIPS 2024).

---

Concat DontePooe Clicks
Heterogenecus annotations at training time
Estimate any point after deployment
SMPL-X Meeh
MoCap Skalsoon
localization functions stored in neural field
Figure 1:
Can one model learn to localize any point of the human body in 3D from a single RGB image? We propose to build a generalist human pose and shape estimator that can readily learn from any annotated points at training time and can estimate any user-chosen points at test time.
produced through separate set of convolution weights, tying the model to a particular set of joints keypoints. Extending this paradigm to localize an infinite number of points would require infinite memory store the weights, and learning would be highly inefficient as weights would need to be trained independently for each point.
Our key idea and contribution is to learn a continuous field of point localizers, which we call Neural Localizer Field. The principle is to modulate a deep network's s prediction layer on-the-fly. -ly, based on what point need to predict: at training time querying points for which have annotations, and at test time sampling the points that an application requires. Here, we are inspired by the recent success of neural field representations, such as neural radiance [75], distance [84, 18] and occupancy fields [74], but instead of predicting a physical property (occupancy, color) we predict a function.
Specifically, this localizer field's s input domain is the full 3D volume of the canonical human body and its output range is the parameter space of modulated convolutional output layer (making the localizer field hypernetwork [33, 46, 13]). By optimizing the neural field's parameters, we obtain smooth field of localizers (functions) over the human volume domain, improving robustness, and knowledge sharing. In the paper, we investigate various design choices related to this architecture, including positional encodings. This architecture results consistent and accurate skeletal joint predictions along with smooth nonparametric mesh output, whose format is chosen by the user at test time, see Fig. 1. Since a parametric representation (for example based on SMPL [65]) is often preferred for further downstream processing, the second contribution of this paper, derive a simple and efficient algorithm for fitting S.MPL-family body models to our non- non-parametric predictions. This involves alternating between estimating global body part orientations through the Kabsch algorithm [44], and solving for the shape coefficients in a regularized linear least squares 2-4 rounds, and is well-suited for GPU acceleration.
formulation. This converges after few
In summary, our main contribution is the Neural Localizer Field method for the 3D localization of an infinite continuun of points within the human body volume, based single RGB image, allowing seamless mixed-dataset training using various skeleton or mesh annotation formats without the need for tedious and error-pron re-annotation. Through this formulation, we are able to train generalist human pose and shape estimator that outperforms prior work on a wide range of important benchmarks. We will make code and trained models publicly available for research.
2 Related Work
Methods for 3D pose/shape estimation use a wide variety of formats: different skeletons, meshes, body models volumetric representations. For a comprehensive review, see [108]. Here focus on methods that closely relate to ours.
Large-scale multi-dataset human-centric training. A few recent works [93, 30, 12] have started to explore scaling up 3D human body reconstruction methods beyond the previously common practice of training on one or few datasets. Models trained at scale from many sources tend to outperform specialized models that were trained for specific dataset. However, current methods have to unify

---

their datasets to common format [12], which is often an ioo-pd problem. This can introduce errors if done through pseudo-amnotation. requiring significant human effort and manual quality checks. Another option is to train model to predict all possible outputs and optimize consistency losses between them [93], but this is not scalable to a large, possibly infinite number of annotation types. By stark contrast, our model can ingest data from many different formats, and can be asked to output any point set at test time.
Nonparametric mesh estimation and sparse keypoints. Some works [53, 68] predict further keypoints on the surface beyond the skeleton, noting that this carries richer shape information. Ma et al. [68] predict sparse set of virtual markers and approximate the rest through convex interpolation. Nonparametric mesh estimation has been a successful line of recent research [19, 122, 62, 63, 22, 20, 52, 76, 67], mainly due to better preservation of alignment in contrast to parametric estimation, which often suffers from mean mean-shape bias [96]. In contrast to our work, the prominent Transformer- based approaches of this category [62. 19, 63] feed a static set of keypoints into their architecture throughout training, which then exchange information through self-attention layers, and require a coarse to fine strategy and complex design for efficient processing. In contrast, our architecture is more lightweight, and can directly produce predictions for all possible points, without having to interpolate them from some other set of points.
Continuous and dense representations. While the dominant paradigm supervise pose estimators has been sparse keypoints and skeletons, a few works have explored dense, continuously varying keypoints for supervision and prediction. DenseReg [1] predicts dense 2D deformation grid to self-supervise a face regressor. In DensePose [31, 79], 2D keypoint estimation is generalized to a continuous representation, where arbitrary surface points can be regression targets, instead of having a fixed set of sparse keypoints throughout the dataset. DensePose3D [100] shows how 2D dense maps can be used to lift estimates to 3D. Similar in spirit, DenseBody [120, 127] predicts 3D mesh points on a 2D UV-map. Chandran et al. [15] estimate 2D facial landmarks in a continuous manner. Common to all these works is that either the prediction or some intermediate representations are given in 2D, e.g., UV-maps, dense semantic maps or 2D output coordinates. By contrast, we are able to perform a direct 3D-to-3D mapping from canonical observation space, to localize an infinite continuum of points in the human volume, to harness heterogeneous annotations.
Distinctions from DensePose. DensePose [31, 79] may seem similar enough to our motivation that we should expand upon why it does not solve the same problem are tackling. In a sense, problem formulation is the reverse of DensePose's ss DensePose starts out in image space and asks, per pixel, what canonical surface point is visible the pixel (facing the camera). This inherently restricts the formulation to the frontal surface of the body. Occluded points, body-internal joints and the opposite-side surface of the human are ignored in DensePose. Instead, we start in canonical space and ask the more natural question of where each body point (identified in canonical space) can be found in 3D observation space. This is more in line with the goals in practical applications. However, because of this reverse relation between the two formulations, the annotations in DensePose datasets [78, 31] can still be directly used to supervise our model well (with the roles reversed), further boosting the data sources available to us.
3D keypoint estimation through heatmaps. A well-established paradigm to predict 3D keypoints is to use 3D heatmaps [86, 104, 92, 68, 73, 71J. The advantage is that convolutional architectures are well suited for localizing joints in the image, and inputs and outputs are well aligned. However, predicting infinite number of heatmaps for continuum of points is not possible, it would require storing an infinite number of last-layer weights. With our Neural Localizer Fields, we can harness the well well-performing heatmap paradigm and adapt it to predicting keypoints chosen at test time.
Parametric fitting to nonparametric predictions. Fitting parametric mesh to keypoint predictions needed for certain applications and for compactness. If correspondences are known, the classical way is to solve nonlinear least squares problem as in the original SMPL [65] paper, but this requires many iterations and is slow. To speed up and to make the process differentiable, some works train an MLP to predict body parameters from sparse markers [52, 1 126]. This requires generalizing to all possible poses which is hard. By contrast, we propose a solver based on limb rotation estimation via the Kabsch algorithm and linear least-squares shape fitting. This direct geometry-basee algorithm orders of magnitude faster than a classical optimization procedure, and does not suffer from generalization issues of the MLP methods since it is not learning-basse based.
3

---

Heatmap logits
Observation point
Vision backbone
Feature maps
*
Soft- argmax
P' E R
Convix1
11
1 xO
E
Weights
((p) E R XCon
Image I E RxWx3
Localizer field
R
Canonical point
Figure 2: Overview of NLF. Given image features F and any arbitrarily chosen 3D query point within the canonical human volume, we aim to estimate the observation-space 3D point p'. To control which point gets estimated, dynamically modulate convolutional layer the output, to produce heatmaps for the requested point. We achieve this modulation by predicting the convolutional weights through a neural field. During training, the points p can be picked per training example based on whichever points are annotated for it, allowing natural dataset mixing. At test time, the model can flexibly estimate any surface point and any skeletal joint inside the body volume, as required.
3 Method
Our method consists of three main parts: (1) point localizer network (PLN) for localizing 3D points from images, (2) a neural localizer field (NLF) to control what point the PLNN should predict, and (3) an efficient body model fitting algorithm to create parametric representation for set of points that were localized nonparametrically. We metonymically refer to our complete approach as NLF. This modular structure establishes clear interfaces and can enable independent improvements on the parts. The architecture overview is depicted Fig.
3.1 Point Localizer Network
We start with describing the basic mechanism of localizing a point our architecture, which we then generalize to arbitrary points afterward.
Given RGB image crop I [0, 1] 11" of person, first extract feature maps F R through an off-the-shelf backbone at stride & = W/w = H/h. We then follow the MeTRAAbs [92, 93] architecture in skeletal 3D pose estimation, and produce metric-scale truncation-robuss volumetric heatmaps HAD E hxx as well as 2D pixel-space heatmaps Hzo E Txx through a 1 x convolutional layer. The heatmaps are decoded to coordinates via soft-argmax [54, 80, 104], and the resulting 2D and root-relative 3D points are combined to an absolute camera-space estimate, including estimates for truncated body parts. Additionally, we predict an uncertainty map U E B which will be averaged to an uncertainty scalar per point using weights from Hap-
pose
A classical, joint-based pose estimator would be trained to produce fixed set of heatmaps, many as there are joints in the skeleton, and it would have to be trained and tested with the corresponding fixed set of joints. Taking a functional view, observe that this type of architecture realizes discrete set of localizer functions, mapping image features to 3D locations as
IR. kXxxC + R F-P' iE [1.J]
(1)
with // being the number of joints the model can predict. In a pose estimator, each f. mapping is represented through a section of the weight matrix W of the convolutional output head. Hence, in this paradigm, implementing localizer functions to detect further points requires extending the weight matrix, giving linear growth in the parameters of the output layer. It clearly infeasible store the weights for every possible point we might want to predict. Furthermore, even though the functions for localizing nearby points the body have more in common than those detecting faraway points, this spatial structure is lost when representing them as discrete outputs. In the next part, we describe our method for retrieving the full continuum of localizer functions, allowing predictions for any point, as well as ensuring spatial information sharing across the functions.

---

3.2 Neural Localizer Field
The goal of our work is go beyond predicting only predetermined finite set of localizable points in network, and to instead learn to localize all points of the body with one universal model, by learning whole localizer field of functions V. The field v associates a function fo to every point p in the canonical human volume QH C TR3:
V : SHH + ( + R*), p+ (=b::Fpp)
(2)
where the posed point p' is determined based on image features F by fp. typically used to predict points vectors, here we use them predict localizer functions fp- This way, we represent information on a continuous domain, while appropriately modeling both smooth structure and fine-grained details of signal. Specifically, fP( is implemented
Although neural fields are
single-layer convolutional head whose weights are predicted as a function (p)) of the canonical point p we want to localize. The function w : Ω2 - ((C+I)((D-2) is realized as an MLP, taking three canonical coordinates of p as input and producing parameters (W. b) for Sp 's convolutional layer to control which point gets predicted (cf. hypernetworks). Here the C'+1 is the backbone feature dimensionality plus one bias, and D + are the total number of output channels which are D depth channels of the volumetric heatmap HAD. one 2D heatmap H2D and one uncertainty map U. In summary, given the image features F and canonical query point p. the following computation takes place:
(W, b) == w(p)
(3)
(Hw, HAD, U) == convixi W PAD rostrel == coft-argmax((He
(4)
Pn == soft ooft-argmax((11))
(5)
Uz,y softmax(Hav) 2.
softplus(u) + c,
(6)
after which the 3D root-relative and the 2D predictions are fused into camera-space output p' following [92]. The localizer field's architecture consists of a positional encoding part and an MLP with GELU activation (see the appendix for the details of the MLP layers).
Positional encoding. To express high-frequency signals, neural fields typically employ positional encodings [90, 107]. Since the field's domain is the canonical human volume flg, we use the eigenbasis d; of the volumetric Laplacian A (the Fourier basis equivalent for bounded volumes), inspired by the use of the Laplace-Bell Beltrami operator in [79]. We then compute the global poins signature (GPS) [91]
<P) = (dd(())// (hol///A,,.
T AM
E CRM.
widely used descriptor in shape matching. So far, this only gives us the eigenfunction values 6 on the discrete tetrahedral mesh nodes. To obtain an approximate interpolant for the entire domain, we distill the eigenfunctions into a small learnable Fourier feature network [58] h R - RM with (p)))= R (p). The network h can be finetuned end-to-end, Details are in the appendix. We will compare the effectiveness of plain coordinates, global point signature, and learnable Fourier features.
Mathematical formulation for training. Our formulation allows us to supervise the point- localization task on continuous domain. Given an image I with features F, and the ground-trute continuous warp function g Q2 + RR3 mapping between the canonical and observation spaces, the loss measures the distance between ground truth g(-) and prediction f(: F)) in function space. through the volume integral
C --
co(p: d'p,
(8)
where C. is the pointwise loss comparing prediction f(p: = p' to ground truth g(p) = p' in observation space. We adopt the Euclidean loss
(. p') = 11'0
p'll -pll
without squaring, for better outiier-rouustnes. (In the appendix, we show how the uncertainty output can be combined into the loss function, as this aspect not the main focus here.) This loss is applied to the predictions in the 3D camera coordinate frame, the 3D root-relative frame and the 2D image
5

---

space as well, with weighting factors. Naturally, only the 2D loss is applied for training examples that only have 2D labels.
In practice, we often do not have the full volume annotation (p) available for all p as in (8), and it would be too expensive to evaluate the integral for every image. Since train on a mixture of datasets, each training example (Iz, k= )) can contain a different annotated point set {p()k= K depending on the dataset the example comes from. Hence, we turn the integral (8) into a discrete sum over the available sptaset-specific locations P& E Qq. For training examples with parametric body annotations, we perform Monte Carlo approximation of the integral by random sampling of points.
Zoo of annotations. Data in HPE contains a wide variety of annotation types including parametric meshes, 3D keypoints, 2D keypoints and DensePose samples. Thanks to our formulation described above, we can directly consume this data, since the zoo of annotations consists of subsets of points in the canonical volume. For examples that are annotated with a parametric mesh model, we sample 640 points uniformly over the surface and 384 points from the interior of the human volume. The sampled points are unique to each example of the batch. We supervise interior points densely to enable the network to predict new joint sets anywhere in the volume at test time. To deform interior points, we use scattered data interpolation [102, 9] to propagate SMPL deformations to the volume. For datasets annotated with 2D or 3D keypoints, the corresponding canonical positions need to be established per joint. We use an approximate initialization per skeleton type (see appendix), and finetune the precise canonical positions during the main training process. In case of DensePose annotations, we map the U,U, V labels to the corresponding SMPL surface points in the canonical pose.
Implementation details. We use EfficientNetV2-S (256 px) and L (384 px) [106] initialized from [93], and train with AdamW [66], linear warmup and exponential learning rate decay for 300k steps. Training the S model takes ~2 days on two 40 GB A100 GPUs, while the L takes 44 days on A100s.
3.3 Efficient Body Model Fitting in Post-Processing
Above, we introduced our method to nonparametrically localize any human point from an image, in 3D. While nonparametric prediction has its advantages (e.g., closer fits to the image data), in some cases parametric representation (joint rotations 0 and shape vector B) is preferable due to its compactness, and disentanglement of pose and shape. We therefore develop a fast algorithm to solve for pose and shape parameters that closely reproduce our predicted nonparametric vertices.
Our algorithm alternates between two steps for a small number of iterations (~3),, followed by an adjustment step. We first independently fit global orientations per body part by applying a weighted Kabsch algorithm. Then, keeping the orientations fixed, we solve for , and the translation vector via linear least squares, since SMPL's vertex positions are linear in the shape parameters. Finally, we refine the part rotations in one pass along the kinematic tree, anchoring the rotations at the parent joint. For improved efficiency, it is possible to consider only a subset of the vertices when fitting. Further, given that our model estimates per-point uncertainties, these can be used in a weighted variant of the above algorithm. Splitting the shape estimation task into nonparametric point localization and a fitting step has the further benefit that easy to share the body shape parameters 3 for multiple observations of the same person (multi-view and temporal use cases) during the least-squares regression. More details and the pseudocode of the algorithm are given in the appendix.
4 Experiments
We extensively evaluate our method on a variety of benchmarks: 3DPW [114] and EMDB [47] for SMPL body, AGORA [85] and EHF [87] for SMPL-X, SSP-3D [96] for SMPL focusing body shape, as well as Human3.66 3.6 [40], MPI-INF- 3DHP [70] and MuPOTS-3D [72] for 3D skeletons.
Our goal is to enable large-scale datti-d training from heterogeneous annotation sources in order to train strong models. To show that our formulation is effective at this, we assemble large meta-dataset for use in training. Due to space constraints we provide the detailed dataset description
in the appendix. For parametric meshes we include the datasets [85, 8, 6, 35, 50, 109, 24, 36, 16, 130, 121, 11, 119, 123, 88, 10, 1 112], with points sampled directly from each dataset's native body model (one of SMPL, SMPL-X or SMPL.-, neutral or gendered, as provided) without a need for any conversions. For 3D skeleton annotations, use [40, 70, 72, 124, 131, 43, 57, 81, 115, 89,

---

Table I: Ablation of positional encodings. Input representation matters in NLF. The fixed Laplacian-
derived global point signature helps most metrics, but is outperformed by learnable Fourier features. Best results are achieved by initializing the learnable Fourier features to the GPS.
SSP3D mloU 3DWW P-MVEI, EMDB P-MVEL 3DWW MPJAE,
Plain XYZ coordinates
0.777
54.0
57.7
16.9
Global point signature (GPS)
0.789
54.2
57.5
15.8
Learnable Fourier features Learnable Fourier (GPS init.)
0.804
52.8
57.0
15.4
0.812
52.8
56.4
15.3
34, 29, 132, 111, 82, 110, 23, 28, 26, 27, 4, 48, 17]. Several of these have custom skeletons that are nontrivial to convert to the SMPL. family, preventing existing 3D HPS methods from exploiting these rich data sources. For 2D keypoint annotations we use [37, 41, 113, 2. 116, 25] and further use DensePose datasets [78, 31] which provide pairs of SMPL surface points and corresponding pixel locations. Integrating all these datasets would be a great challenge if we had to choose one specific representation. Instead, can easily train using the strategy described in Sec. 3.2, we simply map each annotation convention to the canonical human volume and learn a continuous localizer field.
We use standard metrics including per-joint error (MPIPE), per-vertex error (MVE). orientation angle error (MPIAF) and their Procrustes variants ("P-"). Metrics are explained in detail in the appendix.
Pose prediction. NLF outperforms all baselines on EMDB (Tab. 3), which has challenging outdoor sequences. The improvement is drastic, we get an MPJPE of 68 4 mm compared to the second-best (98.0 mm). Remarkably, we outperform the temporal method WHAM (79.7 mm) although we work frame-by-frame 3DPW results confirm the same (Tab. 2). (For reference, the standard error of the mean under normal assumptions for 3DPW MPJPE 0.06 mm.) We attribute this performance to the fact that we can natively train on many sources without error-prone conversions. (For better comparability to temporal methods, we also provide NLF results using a (non-learned) 5-frame temporal smoothing filter.) On AGORA (Tab. 5), we outperform all prior works in SMPL-X prediction. Even though we do not specially target facial and hand keypoints, we obtain the second best scores for these. We also achieve state-of-the-an body results on EHF (Tab. 7). The higher hand error is due to few detailed hands in the training data. We obtain excellent performance on the skeleton estimation benchmarks Human3.6M, MPI-INF-3DHP and MuPoTS-3) as well (Tab. 6). This evaluation is easy with NLF, as we can pick at test time which points of the body to predict.
Shape prediction. Mesh recovery methods are often biased towards an average shape. Keypoint- based methods are pixel-aligned but ignore shape, and SMPL parametric regressors tend to produce subpar image alignment. By contrast, since we model the full continuum of points, we can estimate shape more accurately while being pixel-aligned. This is reflected on SSP-3D (Tab. 4), dataset specifically designed to evaluate shape estimation, where we again improve over all baselines, obtaining low error of 10.0 mm. Notably, we outperform works, such as ShapeBoost [7], which are specialized to shape estimation and have not been tested on general pose estimation benchmarks. Figure 3 shows a qualitative example. Following the protocol in [98], we also evaluate combined shape estimation from multiple (max. 5) images of the same person, by shared estimation the / shape parameters during body model fitting. This reduces the error to 9.6 mm. Our NLF achieves state-of-te the art shape estimation while simultaneously obtaining SOTA results on the most challenging pose benchmarks such EMDB, using the same model weights.
Positional encoding. To assess the impact of positional encoding on shape fidelity, we evaluate the mIoU silhouette overlap measure on the shape-diverse SSP-3D, the Procrustes-aligned vertex error on 3DPW and EMDB, as well as the orientation error on 3DPW after SMPL-fitting using the small backbone. As seen in Sec. 4, the use of the Laplacian- leads to better mloU of 0.789 compared to the 0.777 achieved using plain X, Y, Z coordinates. We attribute this to better representation of fine- fine-grained geometry. Learnable Fourier features obtain even better results than GPS, but best is the combination where we use the GPS as initialization for the learnable Fourier feature network. Do note however that NLF achieves good performance even with plain coordinates.
global point signature (GPS) encoding
Uncertainty estimation. We provide an ablation for uncertainty predictions in the appendix.
7

---

Table 2: Results on 3DPW (14 joints). denotes temporal (multi- frame) method
Trained without 3DPW-train
Trained with 3DPW-train
Method
MPJPE P-MPJPE
MVE P-VVE
MPIPE
P-MPJPE
MVE P-MVE
PYMAF [128]
78.0
47.1
91.3
74.2
45.3
87.0
HaPLer-X-HBB. [12]
75.0
50.6
71.7
48.0
BEDLAM-CLIFE [8]
72.0
46,6
85.0
66.9
43.0
78.5
HybriK [55]
71.6
41.8
82.3
HMR 2.0a [30]
70.0
Multi-HMR [3]
69.5
46.9
88.8
61.4
41.7
75.9
WHAM-B B (VIZ, w/ BEDLAM)" [101]
56.9
35.7
67.4
NLF-S
60.9
38.5
73.3
52.8
55.6
35,9
67.0
48,9
NLF-S +fit
60.8
37.9
72.2
51.4
56.6
35.7
66.7
47.9
NLF-
60.3
37.3
71.4
50.2
54.1
33.7
63.7
45.3
NLF-L +fil
59.0
69.7
48.8
54.9
63.7
NLF-L +fit +smooth*
57.2
35.4
67.8
47.7
53.2
32.6
62.1
43.5
Table 4: Results SSP-3D. PVE-T- SC is the per-vertex error T-pose with scale correction, mloU measures silhouette overlap. means estimating body shape
Table Results on EMDB1 (24j). *temporal method
P-MPIPE
P-MPIAE
MPJPE
MVE
P.MVE
MPIAE
Method
from multiple images of a person (max. 5).
PyMAF [128]
131.1
82.9
160.0
98.1
28.5
25.7
LGD [103]
115.8
81.1
140.6
95.7
25.2
25.6
Method
PVE-T-SCL
mloU?
ROMP [105]
112.7
75.2
134.9
90.6
26.6
24.0
HMR [45]
22.9
0.69
PARE [49]
113.9
72.2
133.2
85.4
24.7
22.4
SPIN [S]]
22.2
0.70
GLAMR' [125]
107.8
71.0
128.2
85.5
25.5
23.5
SHAPY [21]
19.2
0.71
FastMETRO-L [19]
115.0
72.7 133.6
86.0
25.1
22.9
SoY [94]
15.8
0.76
CLIFF [59]
103.1
68.8 122.9
81.3
23.1
21.6
STRAPS [96]
15.9
0.80
HybrlK [55]
103.0
65.6 122.2
80.4
24.5
23.1
Sengupta et al. [98]
HMR 2.0 [30]
98.0
60.6 120.3
Sengupta et al. [98]
13.3
REDLAM-CLIFF [8]
98.0
60.6 111.6
Sengupta et al. [97]
13.6
WHAM-B (VIT)* [101]
79.7
50.4
94.4
LASOR [118]
0.67
HuManiFlow [99]
NLF-S
74.5
44.9
87.8
56.4
KBody [133]
25.6
0.80
NLF-S +fit
72.0
44.6
85.2
55.4
17.1
16.4
ShapeBoost [7]
NLF-
69.6
41.2
82.4
52.0
NLF-L +fit
68.4
40.9
80.6
51.1
16.1
15.4
NLF-S +fit
10.0
0.85
NLF-S +fit (shared 33*
9.6
0.85
NLF-L +fit +3mooth*
78.7
50.0
16.0
15.2
NLF-L +fit
11.1
0.83
Table 5: Results on AGORA-test (SMPL-X). Models are fine-tuned on AGORA.
N
NMJE
MVE
MPJPE
Method
All Body
All Body
All
Body
Face
LHan RHan
All
Body
Face
LHan RHan
Hand4Whole [77]
144.1
96.0 141.1 92.7 135.5
90.2
41.6
46.3
48.1
132.6
87.1
46.1
44.3
46.2
BEDLAM [8]
142.2
102.1 141.0
101.8 103.8
74.5
23.1
31.7
33.2
102.9
74.3
24.7
29.9
31.3
PYMAF-X [129]
141.2
94.4
140.0
93.5
125.7
84.0
35.0
44.6
45.6
124.6
83.2
37.9
42.5
43.7
OSX [61]
130.6
85.3 127.6
122.8
36.2
45.4
119.9
78.3
37.9
43.0
43.9
HybrlK-X [56]
120.5
73.7 115.7
72.3
112.1
68.5
37.0
46.7
47.0
107.6
67.2
38.5
41.2
41.4
SMPLer-X [12]
107.2
68.3
104.1
66.3
99.7
63.5
29.9
39.1
39.5
96.8
61.7
31.4
36.7
37.2
Multi-HMR [3]
102.0
63.4
101.8
64.1
95.9
59.6
27.7
40.2
40.9
95.7
60.3
29.2
38.1
39.0
NLFS
108.6
67.9
106.3
67.4
102.1
63.8
29.5
42.7
42.8
99.9
63.4
31.7
38.7
39.2
NLF-
98.6
62.1
96.6
61.9
92.7
58.4
27.0
37.9
38.1
90.8
58.2
28.5
34.4
34.9
Table 6: Results on skeleton estimation benchmarks.
Method
Human3.66
MPI-INF-3DHP
MuPoTS-3D
MPJPEJ
P. P-PPIEE
PCKt
AUC
MPJPF_
PCK-detected+
McTRAbs-ACAE-S [93]
40.2
31.1
96.3
58.7
57.9
94.7
MeTRAbs-ACAF-L [93]
36.5
27.8
97.1
60.1
55.4
95.4
NLF-S
40.4
30.6
96.6
57.9
59.9
94.7
NLF-
39.7
28.5
97.5
61.0
54.9
95.5
8

---

Table 7: Results on EHF (SMPL.-X).
Method
MVE
P-MVE
P-MPIPE
Full-body
Hands
Face
Full-body
Body
Hands
Face
Body (14j)
Hands
HybrIK-X [56]
121.4
52.1
41.9
59.8
50.0
17.6
8.1
60.8
17.9
OSX [61]
70.8
53.7
26.4
48.7
15.9
6.0
PyMAF-X [129]
64.9
29.7
19.7
50.2
44.8
10.2
5.5
52.8
10.3
SMPLer-X [12]
62.4
47.1
17.0
37.1
14.1
5.0
Multi-HMR [3]
42.0
28.9
18.0
28.2
10.8
5.3
NLF-S
39.7
49.7
15.2
30.9
30.4
24.5
7.4
32.8
28.2
NLF-S +fit
40.0
49.7
15.4
30.4
29.7
24.0
6.5
32.0
25.4
NLF-L
36.3
43.2
13.5
25.8
24.8
21.3
6.7
26.3
25.3
NLF-L +fit
36.4
43.0
13.9
26.0
24.4
20.7
6.3
26.1
21.8
Qualitative result on SSP-3D. left: NLF's nonparametric output (front and side view), right: result of our proposed fast SMPL fitting algorithm (front and side). Our nonparametric prediction already has high quality, allowing us to use a simple and efficient fitting algorithm obtain body model parameters that faithfully represent the nonparametric output.
Figure 3:
SMPL
SMPL-X
COCO
Humaa3.6M
Random points
Figure 4: Customizable point localization. By selecting points p the continous canonical space, we can predict any landmark set both at training and test time. The first column depicts the query points we estimate: SMPL(-X) joints and vertices, COCO joints, Human3.6M joints, and arbitrary points sampled within the human volume. The fourth and seventh column show rotated views.

---

Table 8: Training data ablation. Using all datasets yields the best results compared only the real subset or only the synthetic subset, showing that NLF benefits from both kinds of data and that more data improves results. (Here we trained for 100k steps as opposed to 300k for the main evaluations.)
SSP-3D
H36M
3DHP
3DPW
3DPW
EMDB
EMDB
Method / data
PVE-T-SCL
MPIPEL
PCK1 P-MPJPF4 P-MVEL
P-MPJPEJ
P-MVEL
NLF-S, real datascts only
11.9
43.0
96.0
40.4
55.5
47.8
59.3
NLF-S, synthetic datasets only
10.0
48.0
95.4
39.2
53.5
45.7
57.9
NLF-S, all datascts
9.9
41.6
96.3
38.4
53.1
45.8
57.2
Effect of datasets. Assessing the contribution of each individual dataset is computationally infeasible, nonetheless, to validate that NLF benefits from diverse supervision, we partition the data into real and synthetic subsets. As Tab. 8 shows, best results are achieved when all datasets are used in training.
Real-time inference. NLF-S has batched throughput of 410 fps and unbatched throughput of 79 fps on an Nvidia RTX 3090 GPU. For NLF-L these are 109 fps and 41 fps respectively.
Qualitative results. To demonstrate the versatility of our method in localizing any point of the human body, we show qualitative results in Fig. 4 according to different representations. Qualitative results with the final fitted SMPL are given in the appendix.
Efficient body model fitting. On both 3DPW and EMDB, fitting SMPL to the nonparametric estimation (Sec. 3.3) generally reduces the error, but the difference is small, indicating that the original predictions already have high quality. To measure the fitting speed, we consider the task of fitting SMPL-X to vertices derived from SMPL (model transfer) using sample data from the SMPL-X website (33 meshes). The official code for the transfer takes 33 minutes with mean vertex error 5.0 mm. Our algorithm takes 28.4 ms with error 7.8 mm, i.e., orders of magnitude faster, and error increase is negligible relative to the typical prediction error of centimeters.
Limitations. NLF works frame-by-frame without temporal cues, and estimates each person independently, highly-overlapping people remain challenging. Further, our predictions may have self-intersections and the absolute distance may be inaccurate due to depth/scale ambiguity.
Broader social impact. Human pose and shape estimation can help advance assistive technologies, human- robot interaction or interactive entertainment. However, like other vision methods, a potential for misuse exists in e.g., illegitimate surveillance. We will release models for research only.
Conclusion
We proposed Neural Localizer Fields to upgrade 3D human pose estimators from fixed joint sets to predicting any point of the body surface and volume. We store point-localizer functions in a neural field, as opposed to storing explicit convolutional weights for every joint the network can predict. This enables us to train a large-scale generalist pose and shape estimation model, without any effort on relabeling all data to one format. Instead, we can supervise our model with any points that happen to be annotated for particular training examples. Besides this training advantage, we can choose at runtime which points in the volume to predict, allowing us to output any desired format. To make our output even more useful in downstream applications, we proposed an efficient hiints-to-parameet fitting algorithm to obtain SMPL parameters. Our model trained on several datasets with different formats achieves sate-of-the-ate performance with significant improvement over baselines on several benchmarks. Our model exhibits good performance on outdoor, indoor and synthetic data, regardless of the skeleton/mesh format used for different benchmarks. With the recent trend of more datasets from different sources being available, NLF paves the way to leverage these elegantly and efficiently.
Acknowledgments and Disclosure of Funding
This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 011S18039) This work is funded by the Deutsche Forschungsgemeeinschatt (DFG, German Research Foundation) - 4097921 80 (Emmy Noether Programme, project: Real Virtual Humans). GPM is member of the Machine Learning Cluster of Excellence, EXC number 2064/1 Project number 390727645. The project was made possible by funding from the Carl Zeiss Foundation.
10

---

References
[1] R. Alp Guler, Trigeorgis, E. Antonakos, P. Snape, S. Zafeiriou, and I. Kokkinos. DenseReg: Fully convolutional dense shape regression in-the-wild. In CVPR, 2017.
[2] M. Andriluka, U. Iqbal, E. Insafutdinov, L. Pishchulin, A. Milan, J. Gall, and B. Schiele. PoseTrrak: A benchmark for human pose estimation and tracking. In CVPR, 2018. 7, 18
[3] F. Baradel, M. Armando, S. Galaaoui, R. Brégier, P. Weinzaepfel, G. Rogez, and T. Lucas. Multi-HMR: Multi-person whole-body human mesh recovery in single shot. In ECCV, 2024. 8,
[4] E. G. Bazavan, A. Zanfir, M. Zanfir. W. T. Freeman, R. Sukthankar, and C. Sminchisescu. HSPACE:
Synthetic parametric humans animated in complex environments. arXiv:2112 12867, 2021. 7,
[5] Y. Ben-Shabat, X. Yu, F. Salch, D. Campbell, C. Rodriguez- H. Li, and S. Gould. The IKEA ASM
datasct: Understanding people assembling furniture through actions, objects and pose. WACV, 2021.
18
[6] L. Bhatnagar, Xie, I. Petrov, C. Sminchisescu, Theobalt, and Pons-Moll. BEHAVE: Dataset and method for tracking human object interactions. In CVPR, 2022. 6, 17
(7) Bian, J. Li, J. Tang. and C. Lu. ShapeBoost: Boosting human shape estimation with part-based parameterization and slothing-preserving augmentation. In AAAI, 2024. 7, 8
[8] M. J. Black, P. Patel, J. Tesch, and J. Yang. BEDLAM: A synthetic dataset of badies exhibiting detailed lifelike animated motion. In CVPR, 2023. 6, 8, 17
[9] T. A. Bobach. Natural neighbor interpolation-
assessment and new contributions. PhD thesis.
Technische Universität Kaiserslautern, 2009. 6
[10] F. Bogo, J. Romero, G. Pons-Moll, and M. J. Black. Dynamic FAUST: Registering human bodies in motion. CVPR, 2017. 6, 17
[11] Z. Cai, D. Ren, Zeng. Z. Lin. T. Yu. W. Wang. X. Fan, Y. Gao, Y. Yu, L. Pan, F. Hong, M. Zhang, C. C. Loy, Yang, and Z. Liu. HuMMan: Multi-modal 4D human dataset for versatile sensing and modeling. In ECCV, 2022. 6, 17
[12] Cai, W. Yin, A. Zeng, C. Wei, Q. Sun, Y. Wang, H. E. Pang, H. Mei, M. Zhang, L. Zhang, C. C. Loy, L. Yang, and Z. Liu. SMPLer-X: Scaling up expressive human pose and shape estimation. In NeurIPS: Datasets and Benchmarks, 2023. 2, 3,8, 8,9
[13] C. Cao, T. Simon, J. K. Kim, G. Schwartz, M. Zollhoefer, S.-S. Saito, S. Lombardi, S.-E. Wei, D. Belko, S.-L. Yu, et al. Authentic volumetric avatars from phone scan. In TOG (Proc. SIGGRAPH), 2022. 2
[14] D. Casas and M. Comino-Trinidad. SMPLitex: A generative model and datasct for 3D human texture estimation from single image. In BMVC, 2023.
[15] Chandran, G. Zoss, Gotardo, and D. Bradley. Continuous landmark detection with 3D queries. In CVPR, 2023. 3
[16] W. Cheng. S. Xu, Piao, C. Qian, W. Wu, K.-Y. Lin, Li. Generalizable neural performer: Learning robust radiance fields for human novel view synthesis. arXiv:2044117999 2022. 6, 17
[17] W. Cheng, R. Chen, W. Yin, S. Fan, Chen, H. Hc, H. Luo, Cai, J. Wang, Y. Gao, 7. Yu, 7. Lin, D. Ren, I.. Yang, 7. Liu, C. C. Loy, C. Qian, W. Wu, D. Lin, B. Dai, and K.-Y. Lin. DNA-Rendering: A diverse neural actor repository for high-fidelity human-centric rendering. In ICCV, 2023. 7. 18
[18] J. Chibane, A. Mir, and G. Pons-Moll. Neural unsigned distance fields for implicit function learning. In NeurlPS, 2020. 2
[19] J. Cho, K. Youwang, and T.-H. Oh. Cross- aonssatttntionnoooo of disentangled modalities for 3D human mesh recovery with transformers. ECCV, 2022. 3,8
[20] H. Choi, G. Moon, and K. M. Lee. Pose2Mesh: Graph convolutional network for 3D human pose and mesh recovery from a 2D human pose. ECCV, 2020.
[21] Choutas, L. Müller, C.-H. Huang, S. Tang, Tzionas, and M. J. Black. Accurate 3D body shape regression using metric and semantic attributes. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (C (CVPR), 2022. 8
[22] E. Corona, G. Pons-Moll, G. Alenyà, and F. Moreno-Noguer, Learned vertex descent: A new direction for 3D human model fitting. In ECCV, 2022. 3
[23] M. Fabbri, F. Lanzi, S. Calderara, A. Palazzi, R. Vezzani, and R. Cucchiara. Learning to detect and track visible and occluded body joints in virtual world. In ECCV, 2018. 7, 18
11

---

[24] Z. Fan, O. Taheri. D. Tzionas. M. Kocabas, M. Kaufmann. M. J. Black, and O. Hilliges. ARCTIC: A dataset for dexterous bimanual hand-object manipulation. In CVPR, 2023. 6, 17
[25] H.-S. Fang, J. Li, H. Tang, C. Xu, H. Zhu, Y. Xiu, Y.-L. Li, and C. Lu, AlphaPose: Whole-body regional multi-person pose estimation and tracking in real-time. TPAMI, 2022. 7,
[26] M. Fieraru, M. Zanfir, E. Oneata, A.-I. Popa, V. Olaru, and Sminchisescu. Three-dimensional reconstruction of human interactions. In CVPR, 2020. 7,
[27] M. Fieraru, Zanfir, E. Oneata, A. Popa. Olaru, and Sminchisescu. Learning complex 3D human self-contact. AAAI, 2021. 7, 18
[28] M. Fieraru, M. Zanfir, S.-C. Pirlea, V. Olaru, and Sminchisescu. AlFit: Automatic 3D human- interpretable feedback models for fitmess training. In CVPR, 2021. 7, 18
[29] Ghorbani, Mahdaviani, A. Thaler, K. Kording, D. J. Cook, G. Blohm, and Troje. MoVi: large multi-purpose human motion and video dataset. PLOS One, 16())c0253157, 2021. 7, 18
[30] S. Goel, G. Pavlakos, J. Rajasegaran, A. Kanazawa*, and J. Malik Humans in 4D: Reconstructing and tracking humans with transformers. In ICCV, 2023. 1, 8
[31] R. A. Güler, N. Neverova, and I. Kokkinos. DensePose: Dense human pose estimation in the wild. In
CVPR, 2018. 3, 7, 18
[32] V. Guzov, 1. A. Petrov, and G. Pons-Moll. Blendify - Python rendering framework for Blender. arXiv:2410. / 17858, 2024. 25
[33] D. Ha, A. M. Dai, and Q. V. Le. Hypernetworks. In ICLR, 2017. 2
[34] Y.T. Hu, H.-S. Chen, K. Hui, J.-B. Huang, and A. G. Schwing. SAIL-VOS: Semantic amodal instance level video object segmentation synthetic dataset and baselines. In CVPR, 2019. 18
[35] C.-H. P. Huang, H. Yi, M. Höschle, M. Safroshkin, T. Alexiadis, S. Polikovsky, D. Scharstein, and M. J. Black. Capturing and inferring dense full-hody human-scene contact. In CVPR, 2022. 17
[36] Y. Huang, O. Taheri, M. Black, and D. Tzionas. InterCap: Joint markerless 3D tracking of humans and objects interaction. GCPR, 2022. 6.
[37] Insafutdinov, Pishchulin, B. Andres, M. Andriluka, and Schiele. DeeperCut: A deeper, stronger, and faster mulli-persoo pose estimation model. ECCV, 2016. 7,
[38] loffe. Batch renormalization: Towards reducing minibatch dependence natch-normalized models. In
NIPS, 2017. 19
[39] Ionescu, Li, and Sminchisescu. Latent structured models for human pose estimation. ICCV, 2011. 18
[40] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu. Human3 6M: Large scale datasets and predictive methods for 3D human sensing in natural environments. TPAMI, 36(7):1325-1333 2014, 6, 18
[41] S. Jin, L. Xu, J. Xu, C. Wang, W. Liu, C. Qian, W. Ouyang, and P. Luo. Whole-body human pose estimation in the wild. In ECCV, 2020. 7, 18
[42] G. Jocher, A. Chaurasia, and J. Qiu. Ultralytics YOLO, Jan. 2023. 18
[43] H. Joo, T. Simon, X. Li, H. Liu, L.. Tan, I.. Gui, S. Banerjee, T. Godisart, B. C. Nabbe, I. A. Matthews, T. Kanade, S. Nobuhara, and Y. Sheikh. Panoptic Studio: A massively multiview system for social interaction capture. TPAMI, 41(1):190- 2049 2019. 6, 18
[44] W. Kabsch. A solution for the best rotation to relate two sets of vectors. Acta Crystallographica Section A, 32:922-923, 1976. 2
[45] A. Kanazawa, M. J. Black, D. Jacobs, and J. Malik. End-to-end recovery of human shape and pose. In CVPR, 2018.
[46] T. Karras, Laine, and Aila. A style-based generator architecture for generative adversarial networks. CVPR, 2019.
[47] M. Kaufmann, Song, C. Guo, K. Shen, T. Jiang, C. Tang, J. Zárate, and O. Hilliges. EMDB: The Electromagnetic Database of Global 3D Human Pose and Shape in the Wild. In International Conference
on Computer Vision (ICCV), 2023. 6
[48] R. Khirodkar, A. Bansal, L. Ma, R. Newcombe, M. Vo, and K. Kitani. Egohumans: An egocentric 3D
multi-human benchmark. In ICCV, 2023. 7, 18
[49] M. Kocabas, C.HH. P. Huang. O. Hilliges, and M. J. Black. PARE: Part attention regressor for 3D human body estimation. In ICCV, 2021. 8
12

---

[50] M. Kocabas, C.-II. P. Huang. J. Tesch, L. Müller, O. Hilliges, and M. J. Black. SPEC: Seeing people in the wild with estimated camera. In ICCV, 2021. 17
[51] N. Kolotouros, G. Pavlakos, M. J. Black, and K. Daniilidis. Learning to reconstruct 3D human pose and shape via model-fitting in the loop. In ICCV, 2019. 8, 18
[52] N. Kolotouros, G. Pavlakos, and K. Danilidis. Convolutional mesh regression for single-image human shape reconstruction. In CVPR, 2019. 3
[53] C. Lassner, J. Romero, M. Kiefel, F. Bogo. J. Black, and P. V. Gehler. Unite the people: Closing the loop between 3D and 2D human representations. In CVPR, 2017. 3
[54] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. JMLR, 17 (39):1-40, 2016. 4
[55] Li. C. Xu, Z. Chen, S. Bian, L. Yang, and C. Lu. HybrIK: A hybrid analytical-neural inverse kinematics solution for 3D human pose and shape estimation. In CVPR, 2021. 8
[56] J. Li. S. Bian, C. Xu, Z. Chen, L. Yang, and C. Lu. Hybrik-x: Hybrid analytical maalyicaa-nnuual inverse kinematics for whole-body mesh recovery. arXi1:2304.05690 2023. 8,
[57] R. Li. S. Yang. D. A. Ross, and A. Kanazawa. Al Choreographer: Music conditioned 3D dance generation
with AIST++ In ICCV, 2021. 6, 18
[58] Y. Li. S. Si, G. Li. C.-J. Hsieh, and S. Bengio. Learnable Fourier features for nulti-diiensional spatial positional encoding. NeurlPS, 2021.
[59] Z. Li, J. Liu, Z. Zhang, S. Xu, and Y. Yan. CLIFF: Carrying location information in full frames into human pose and shape estimation. In ECCV, 2022. 8
[60] J. Lin, A. Zeng, S. Lu, Y. Cai, R. Zhang, H. Wang, and L. Zhang. Motion-X: A large-scale 3D expressive bhole-body human motion dataset. In NeurlPS, 2023. 1
[61] J. Lin, A. Zeng, H. Wang. L. Zhang, and Y. Li. One-stage 3D whole-body mesh recovery with component aware transformer. In CVPR, 2023. 8,9
[62] K. Lin, L. Wang, and Z.. Liu. End-to-end human pose and mesh reconstruction with transformers. In CVPR, 2021. 3
[63] K. Lin, L.. Wang, and Z. Liu. Mesh graphormer. In ICCV, 2021. 3
[64] T.-Y. Lin, M. Maire, S. Belongic, Hays, P. Perona, D. Ramanan, Dollár, and C. L. Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014, 18
[65] M. Loper, N. Mahmood, Romero, G. Pons-Moll and M. J. Black. SMPL: A skinned multi-persad linear model. TOG (Proc. SIGGRAPH Asia), 34(6):248: 1-::::1228::: 16, 2015. 2,
[66] I. Loshchilov and Hutter. Decoupled weight decay regularization. ICLR, 2019.
[67] T. Luan, Y. Wang, J. Zhang, Z. Wang, 7. Zhou, and Y. Qiao. PC-HMR: Pose calibration for 3D human mesh recovery from 2D images/videos. In AAAI, 2021. 3
[68] X. Ma, J. Su, C. Wang, W. Zhu, and Y. Wang. 3D human mesh estimation from virtual markers. In CVPR, 2023.
[69] Martin-Martin M. Patel, Rezatofighi, Shenoi, Gwak, E. Frankel, Sadeghian, and Savarese. JRDB: A dataset and benchmark of egocentric robot visual perception of humans built environments. TPAMI, 2021. Early access. 18
[70] D. Mehta, H. Rhodin, Casas, P. Fua, O. Sotnychenko, Xu, and C. Theobalt. Monocular 3D human pose estimation in the wild using improved CNN supervision. In 3DV, 2017. 6, 18
[71] D. Mehta, S. Sridhar, O. Sotnychenko, H. Rhodin, M. Shafiei, H.-P. Seidel, W. Xu, D. Casas, and
C. Theobalt. VNect: Real-time 3D human pose estimation with a single RGB camera. TOG (Proc.
SIGGRAPH), 36(4))4 2017. 3
[72] D. Mehta, O. Sotnychenko, F. Mueller, W. Xu, S. Sridhar, G. Mons-Moll, and C. Theobalt. Single-shot multi-person 3D pose estimation from monocular RGB. In 3DV, 2018. 6
[73] D. Mehta. O. Sotnychenko, F. Mueller, W. Xu, M. Elgharib, P. Fua, H.-P. Seidel, H. Rhodin, G. Pons-Moll, and C. Thcobalt. XNect: Real-time multi 3D human pose estimation with a single RGB camera. TOG (Proc. SIGGRAPH). 39(4), 2020. 3
[74] L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3D reconstruction in function space. In CVPR, 2019. 2
[75] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. NeRF: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020. 2
13

---

[76] G. Moon and K. M. Lee. I2L-MeshNet: Image-to-lixel prediction network for accurate 3D human pose and mesh estimation from single RGB image. In ECCV, 2020.
[77] G. Moon, H. Choi, and K. M. Lee. Accurate 3D hand pose estimation for whole-body 3D human mesh estimation. In CVPR Workshops, 2022. 8
[78] N. Neverova, R. A. Güler, and I. Kokkinos. Dense pose transfer. In ECCV, 2018. 3, 18
[79] N. Neverova, D. Novotny, M. Szafraniec, V. Khalidov, P. Labatut, and A. Vedaldi. Continuous surface
embeddings. NeurlPS, 2020. 3,5
[80] A. Nibali, Z. He, S. Morgan, and L. Prendergast. Numerical coordinate regression with convolutional neural networks, 1r01 07372,, 2018. 4
[81] A. Nibali, J. Millward, Z. He, and S. Morgan. ASPset: An outdoor sports pose video dataset with 3D keypoint annotations. Image and Vision Computing, 111:104196, 2021. 6, 18
[82] F. Ofli, R. Chaudhry, G. Kurillo, R. Vidal, and R. Bajesy. Berkeley MHAD: A comprehensive multimodal
human action database, In WACV, 2013. 7, 18
[83] H. E. Pang. Z. Cai, L. Yang. T. Zhang. and Z. Liu. Benchmarking and analyzing 3D human pose and shape estimation beyond algorithms. In NewrIPS: Datasets and Benchmarks, 2022. 1
[84] J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. DeepSDF: Learning continuous signed distance functions for shape representation. In CVPR, 2019.
[85] P. Patel, C.-H. P. Huang, J. Tesch, D. T. Hoffmaann, S. Tripathi, and M. J. Black, AGORA: Avatars in goography optimized for regression analysis. In CVPR, 2021. 6, 17
[86] G. Pavlakos, X. Zhou, K. G. Derpanis, and K. Daniilidis. Coarse-to-fine volumetric prediction for single-image 3D human pose. In CVPR, 2017. 1,3
[87] G. Pavlakos, V. Choutas, N. Ghorbani, T. Bolkart, A. A. A. Osman, D. Tzionas, and M. J. Black. Expressive body capture: 3D hands, face, and body from single image. In CVPR, 2019, 6
[88] S. Peng. Y. Zhang. Y. Xu, Q. Wang. Q. Shuai, H. Bao, and X. Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In CVPR, 2021.
6, 17
[89] A. Pumarola, A. Agudo, A. Sanfeliu, and F. Moreno-Noguer. Unsupervised person image synthesis in arbitrary poses. In CVPR, 2018. 6, 18
[90] N. Rahaman, A. Baratin, D. Arpit, F. Draxler, M. Lin, F. Hamprecht, Y. Bengio, and A. Courville. On the spectral bias of neural networks. In ICML, 2019. 5
[91] R. M. Rustamov. Laplace-beltrami eigenfunctions for deformation invariant shape representation. In Symposium Geometry Processing, 2007. 5
[92] I. Sárándi, T. Linder, K. O. Arras, and B. Leibe. MeTRAbs: Metric-scale truncation-robust heatmaps for absolute 3D human pose estimation. IEEF Transactions on Biometrics, Behavior, and Identity Science (T-BIOM), 3(1):16-30. 30. 2021. 1, 3,
[93] 1. Sárándi, A. Hermans, and B. Leibe. Learning 3D human pose estimation from dozens of datasets using a geometry-aware autoencoder to bridge between skeleton formats. In WACV, 2023. 2. 3, 6, 19, 25
[94] R. Sarkar, A. Dave, G. Medioni, and B. Biggs. Shape of you: Precise 3D shape estimations for diverse body types. In CVPR, 2023. 8
[95] M. Seitzer, A. Tavakoli, D. Antic, and G. Martius. On the pitfalls of heteroscedastic uncertainty estimation with probabilistic neural networks. In ICLR, 2022. 20
[96] A. Sengupta. 1. Budvytis. and R. Cipolla. Synthetic training for accurate 3D human pose and shape estimation in the wild. In BMVC, 2020. 3, 6.8
[97] A. Sengupta, 1. Budvytis. and R. Cipolla. Hierarchical kinematic probability distributions for 3D human shape and pose estimation from images in the wild. In ICCV, 2021. 8
[98] A. Sengupta. 1. Budvytis, and R. Cipolla. Probabilistic 3D human shape and pose estimation from multiple unconstrained images in the wild. In CVPR, 2021. 7,8
[99] A. Sengupta, I. Budvytis, and R. Cipolla. HuManiFlow: Ancestor-conditionaa normalising flows on SO(3) manifolds for human pose and shape distribution estimation. In CYPR, 2023, 8
[100] R. Shapovalov, D. Novotny, B. Graham, P. Labatut, and A. Vedaldi. DensePose 3D: Lifting canonical surface maps of articulated objects the third dimension. In ICCV, 2021. 3
[101] S. Shin. J. Kim. E. Halilaj, and M. J. Black. WHAM: Reconstructing world-grounded humans with accurate 3D motion. In CVPR, 2024. 8, 18
14

---

[102] R. Sibson. A brief description of natural neighbour interpolation. Interpreting multivariate data, pages 21-36, 1981. 6, 19
[103] J. Song, X. Chen, and O. Hilliges. Human body model fitting by learned gradient descent. In ECCV, 2020.
[104] X. Sun, B. Xiao, S. Liang, and Y. Wei. Integral human pose regression. In ECCV, 2018. 1,3,4
[105] Y. Sun, Q. Bao, W. Liu, Y. Fu, J. Black, and T. Mei. Monocular, one-stage, regression of multiple 3D people. In ICCV, 2021. 8
[106] M. Tan and Q. Le. ElficientNetV2: Smaller models and faster training. In JCML, 2021.
[107] M. Tancik, Srinivasan, B. Mildenhall, S. Fridovich- Keil, N. Raghavan, U. Singhal, R. Ramamoorthi.
J. Barron, and R. Ng. Fourier features let networks learn high frequency functions in low dimensional domains. NeurlPS, 33:7537-7547, 2020. 5
[108] Y. Tian, H. Zhang, Y. Liu, and L. Wang. Recovering 3D human mesh from monocular images: A survey. TPAMI, 45(12):15406- 15425, 2023. 2
[109] S. Tripathi, L. Müller, C.-H. P. Huang. T. Omid, M. J. Black, and D. Tzionas. 3D human pose estimation via intuitive physics. In CVPR, 2023, 6, 17
[110] M. Trumble, A. Gilbert, C. Malleson, A. Hilton, and J. Collomosse. Total Capture: 3D human pose estimation fusing video and inertial sensors. In BMVC, 2017. 7, 18
[111] N. P. van der Aa, X. Luo, G.-J. Giezeman, R. T. Tan, and R. C. Veltkamp. Utrecht Multi-Person Motion (UMPM) benchmark: a multi-person datasct with synchronized vidco and motion capture data for evaluation of articulated human motion and interaction. In ICCV Workshops, 2011. 7, 18
[112] G. Varol, J. Romero, X. Martin, N. Mahmood, M. J. Black, I. Laptev, and C. Schmid. Learning from synthetic humans. In CVPR, 2017. 6, 17
[113] E. Vendrow, D. T. Lc, and H. Rezatofighi. JRDB- Pose: A large-scale dataset for multi pose estimation and tracking. In CVPR, 2023. 7, 18
[114] T. von Marcard, R. Henschel, M. J. Black, B. Rosenhahn, and G. Pons-Moll. Recovering accurate 3D human pose in the wild using IMUs and a moving camera. In ECCV, 2018. 6
[115] Z. Wang, L. Chen, S. Rathore, D. Shin, and C. Fowlkes. Geometric Pose Affordance: 3D human pose with scene constraints. arXiv: /905 2019, 6, 18
[116] J. Wu, H. Zheng, B. Zhao, Y. Li, B. Yan, R. Liang, W. Wang, S. Zhou, G. Lin, Y. Fu, et al. Al Challenger:
A large-scale dataset for going deeper in image understanding. (xXiv:/711.06477 2017. 7, 18
[117] L. Xu, S. Jin, W. Liu, C. Qian, W. Ouyang, P. Luo, and X. Wang. ZoomNAS: Searching for whole-body human pose estimation in the wild. TPAMI, 2022. 18
[118] K. Yang, R. Gu, M. Wang, M. Toyoura, and G. Xu. LASOR: Learning accurate 3D human pose and shape via synthetic occlusion-aware data and neural mesh rendering. IEEE Transactions on Image Processing, 31:1938- 1111888-948,8 2022. 8
[119] Z. Yang. Z. Cai, H. Mei, S. Liu. Z. Chen, W. Xiao, Y. Wei, Z. Qing, C. Wei, B. Dai. W. Wu, C. Qian, D. Lin. Z. Liu, and L. Yang. SynBody: Synthetic dataset with layered human models for 3D human perception and modeling. In /CCV, 2023. 6, 17
[120] P. Yao. Z. Fang. F. Wu, Y. Feng, and Li. DenseBody: Directly regressing dense 3D human pose and shape from single color image. arXiv preprint arXiv:1903. 10153, 2019. 3
[121] Y. Yin, C. Guo, M. Kaufmann. J. Zarate, J. Song, and O. Hilliges. Hi4D: 4D instance segmentation of close human interaction. In CVPR, 2023. 6,
[122] Y. Yoshiyasu. Deformable mesh transformer for 3D human mesh recovery. In CVPR, 2023. 3
[123] T. Yu, Z. Zheng, K. Guo, P. Liu, Q Dai, and Y. Liu. Funetion4D: Real-time human volumetric capture from very sparse consumer RGBD sensors. In CVPR, 2021. 6, 17
[124] Z. Yu. J. S. Yoon, I. K. Lee, P. Venkatesh. J. Park, J. Yu, and H. S. Park. HUMBI: A large multiview dataset of human body expressions. In CVPR, 2020. 6, 18
[125] Y. Yuan, U. lqbal, P. Molchanov, K. Kitani, and J. Kautz. GLAMR: Global occlusion-awar human mesh recovery with dynamic cameras. In CVPR, 2022. 8
[126] M. Zanfir, A. Zanfir, E. G. Bazavan, W. T. Freeman, R. Sukthankar, and C. Sminchisescu. THUNDR: Transformer- based 3D human reconstruction with markers. In ICCV, 2021. 3
[127] W. Zeng. W. Ouyang, P. Luo, W. Liu, and X. Wang. 3D human mesh regression with dense correspondence. In CVPR, 2020. 3
15

---

[128] H. Zhang, Y. Tian. X. Zhou, W. Ouyang, Y. Liu, L. Wang, and Z. Sun. PyMAF: 3D human pose and shape regression with pyramidal mesh alignment feedback loop. In ICCV, 2021. 8
[129] H. Zhang, Y. Tian, Y. Zhang, M. Li, L. An, Z. Sun, and Y. Liu. PyMAF-X: X: Towards well-aligned full-body model regression from monocular images. TPAMI, 2023. 8,
[130] S. Zhang, Q. Ma, Y. Zhang, Z. Qian, T. Kwon, M. Pollefeys, F. Boga, and S. Tang. EgoBody: Human body shape and motion of interacting people from head mounted devices. In ECCV, 2022. 6, 17
[131] Zhang, B. Huang, and Y. Wang. Object-nceluded human shape and pose estimation from a single color image. In CVPR, 2020. 6, 18
[132] W. Zhang, Liu, L. Zhou, Leung, and A. B. Chan. Martial Arts, Dancing and Sports Dataset: challenging stereo and multi-view dataset for 3D human pose estimation. Image and Vision Computing,
61:22-39 39, 2017. 7, 18
[133] Zioulis
O'Brien. KBody: Towards general, robust, and aligned monocular whole-body
estimation, In CVPR, 2023. 8
16

---

Appendix: Neural Localizer Fields for Continuous 3D Human Pose and Shape Estimation
In this appendix, we provide further technical details on
the experimental setup (datasets, training configuration),
uncertainty estimation evaluation,
our proposed efficient parametric body model fitting algorithm (pseudocode, convergence
analysis, fitting to subsets of vertices, uncertainty- weighted fitting),
our use of positional encodings,
the neural field MLP architecture,
qualitative results.
For video results, including visual comparison to prior work, refer to supplementary video.
This video includes a demonstration of continuous querying in the canonical space.
A Mixture Dataset Description
The design goal of our work is to naturally mix data sources using different annotation formats. To show this capability, we perform mixed- batdhh-bhhhhhh training with batch size 160, where every batch contains examples with parametric annotations, examples with 3D keypoint annotations, examples with 2D keypoint annotations and examples with DensePose annotations. In the following, we describe the specific datasets we use for each of these types of annotations.
A.1 Parametric Annotations
We use the following datasets with parametric annotations, grouped by the body model:
SMPL-X: AGORA [85]. BEDLAM [8], RICH [35], MOYO [109], ARCTIC [24], Inter-
Cap [36], GeneBody [16], EgoBody [130]
SMPL: SPEC [50], Hi4D [121], HuMMan [11], SynBody-Neke NeRF [119], THuman2.0 [123].
ZJU-Mocap [88], DFAUST [10], and SURREAL [112].
SMML+H: BEHAVE [6]
Rendering. DFAUST does not provide RGB images but contains diverse shapes that are useful for training, so we render 26K synthetic images with Blender, using randomized SMPLitex [14] textures, cameras and lights. Similarly, since Hi4D contains high-quality fits with good image-alignment, as well as high-quality textured meshes, we also render 100K synthetic images of these meshes with random camera angles and lights.
A.1.1 Shape Accuracy Considerations
Datasets have varying levels of shape accuracy, depending the annotation pipeline used create them, which means that we need to be careful about what we can treat as "ground truth" for training purposes.
Several datasets include annotations in SMPL(-X) format, but these were obtained by fitting only to sparse skeletal body joint locations, which were in turn triangulated from 2D skeletons (obtained e.g. from OpenPose). Therefore, the surface shape information in these fits does not line up well with the image, and the body shapes are biased towards the mean shape. To avoid similarly biasing model to the mean shape, we only treat the joint annotations as valid in these datasets but not use the surface. This applies to GeneBody, HuMMan and EgoBody.
Some datasets provide both the SMPL(-X) fit and raw body joints, either from triangulation or motion capture systems. We observed that the raw body joints tend to line up better with the image, so we ignore the SMPL(-X) fits in these cases. These datasets are: AIST-Dance++, ONA-Rendering HUMBI and FugoH (these provide COCO-style keypoints besides the body fits) and BML-MoVi (which provides more than 80 markers on the body surface, as well as body joints these exhibit better mmaage-alignmen than the provided SMPL fits).
17

---

A.2 3D Skeleton Annotations
We use skeleton annotations (non-SMPL--asee)) from the following dataset sources: Human3.6M [40,
391. MPI-INF-3DHP [70]. MuCo-3DHP [70]. HUMBI [124]. 3DOH [131]. CMU-Panoptic [43].
AIST-Dance++ [57], ASPset510 [81], GPA [115]. 3DPeople [89], SAIL-VOS [34], BML-MoVi [29].
MADS [132], UMPM[1 [1]. Berkeley MHAD [82], TotalCapture [1 [1]0], JTA [23], IKEA-ASM [5].
Fit3D [28], CHI3D [26], HumanSC3D [27], HSPACE [4], EgoHumans [48], and DNA-Rendering [17].
Some of these datasets generated their annotations via multi-view triangulation of 2D pose estimation results, others used marker- based motion capture, and yet others are rendered through graphics and
annotated according the skeleton of the underlying 3D assets, which may deviate from the SMPL skeleton.
A.3 2D Annotations
We are able to use training examples that only provide 2D keypoint information in this case, the loss is computed solely by comparing the projection of our model's output to the ground-truth pixel positions. We use the following 2D keypoint datasets: MPII [37]. COCO-WholeBode [64, 41, 117], JRDB-Pos [69, 113], Pose'Track [2], Al Challenger [116] and Halpe [25].
We also use the DensePose annotations from DensePose-COCO [31] and PensePose-PoseTraas [78].
B Evaluation Protocol
On 3DPW, evaluate 14 joints, obtained through the same Human3. Human3.6.Msstye 6M-style joint regressor that all prior works use. We only report evaluation results on the test set section 3DPW.
Bounding boxes. For testing, some datasets provide bounding boxes (e.g.. SSP-3D), while others do not (e.g., AGORA). In the latter case, we use the YOLOv8 [42] detector (trained on COCO). In the case of AGORA, we evaluate in the fine-tuned setting (NLF fine-tuned just on AGORA), and here we also fine-tune the YOLOv8 detector on AGORA-tran.
Evaluation metrics. In our evaluations, we apply the conventionally used metrics for each individual benchmark. MPJPE is the average Euclidean joint error after alignment at the pelvis. (All distance- based errors are given as millimeters in this work.) Following prior works, e.g., [101], on 3DPW and EMDB the reference point the midpoint between the two hip joints. P-MPJPE the Euclidean after Procrustes alignment, i.e., least-squares optimal rigid and uniform scaling alignment. When 14 joints indicated for evaluation, these obtained with Human3.6M-like regressor from [51], which commonly used in the literature.
MVE is analogous MPJPE but for mesh vertices instead of joints, similarly P-MVE is the Procrustes-aligned version of it.
MPIAE and P-MPJAE are the mean per joint angle errors without and with Procrustes alignment, in degrees. It is the geodesic distance on the SO((3) rotation manifold, comparing the global body part orientations between ground truth and prediction.
On SSP-3D, PVE-T-SC evaluates the body shape, by computing the average Euclidean the predicted and the ground truth vertices in the default T-pose, after the prediction is optimally scale-aligned the ground truth. The mloU is the mean intersection over union metric comparing the binary masks obtained through projecting the ground truth mesh and the predicted mesh on the image plane.
for
On AGORA, NMVE is a normalized version of MVE that takes into account the detection performance well. It is the product of MVE and the Fl measure (harmonic mean of precision and recall). Generally, MVE can be improved by not making predictions for hard cases, and the F1 penalty is aimed compensate for this.
MPI-INF-3DHP and MuP6TS-3D, PCK stands for the percentage of correct keypoints, namely those predicted joints that are within 150 mm of the ground truth.
18

---

C
Training Details
Definition of the canonical volume. The canonical space is defined in reference to a specific T-like pose of the default SMPL mesh. All other keypointss (as e.g. shown in the referenced Fig. 4 column 1) are represented in this same coordinate system, as points within this canonical human volume. No explicit conversion between formats necessary.
Canonical positions for 3D skeletal joints.
The skeleton-based datasets do not provide their correspondence to our SMPl--compatile canonical space. Hence, we perform an approximate initialization and then treat the canonical position of each of several hundred joints (across the skeleton types) as trainable variables during the main training.
One could obtain this initialization in various ways. For this work, we first trained model for predicting the separate skeleton formats (similar to the separate-heads baseline in [93].. We then ran inference with this predictor on the SURREAL dataset and trained linear regressors to interpolate from SURREAL GT vertices the predicted keypoints and applied this regressor to the canonical template to obtain the approximate initialization.
The corresponding left and right joints forced be symmetrically placed in canonical space. To avoid divergence, we impose a constraint that the canonical positions cannot move away more than 70 mm from their initial positions.
Continuous deformation of body models. In SMPL(-X), skinning weights only provided for the discrete set of mesh vertices, so by default forward kinematics can only be applied to these points. To be able to transform any continuous point (e.g.. inside the body), perform natural neighbor interpolation [102]. To avoid having to compute this interpolation on the fly for hundreds of points per training example,
precompute the interpolation weights for ~ million quasi- ransiirandomly sampled points (according to the Sobol sequence). Then during training, we apply forward kinematics to all SMPL vertices and produce the transformed internal point locations by computing weighted average of the mesh vertices (according to the pre-computed natural-neighbor interpolation weights).
Point sampling- Using the above-mentioned interpolation method, we can choose to supervise any set of points in the canonical volume for training examples annotated with high-qualite parametric models. We use a total of 1024 points for each such training cxample, of which 384 are inside the body (sampled from large, precomputed set of more than a million points) and 640 are the surface. The surface points are sampled uniformly by picking a random mesh triangle (with probability proportional to the triangle areas) and sampling random point within the triangle (through barycentric coordinates).
Batch normalization. We found that in our mixture-datasee setting, the batch normalization layers in EfficientNerV2 can accumulate statistics during training that do not work well at inference time. This applies especially to longer trainings of the large backbone. We found effective to use batch renormalization [38] instead, with default settings. Batch renorm mitigates the discrepancy between the training and inference behavior of batch norm. For numerical stability (i.e., avoiding crashes due to NaN values), we also found that constraining the convolutional kernel norms (to maximum value of 20) helped.
Training schedule. We initialize with the 3D pose estimator-based weights from [93], and initially, for 3000 steps, we freeze the image backbone weights and only train the neural field. Subsequently. we decay the backbone learning rate from 10 to 10 with exponential decay with larger drop after 90% of the training is done.
We use random rotation, scaling, translation, truncation, color distortion, synthetic occlusion, random erasing and JPEG compression for data augmentation during training.
Augmentations.
D Inference Speed Optimization
NLF's inference-time overhead (for predicting weights through the field MLP) can be eliminated by precomputing the weights once for chosen set of canonical points. (Typically one wants to localize the same points, i.. same skeleton formats, for many images.) For reference, in case of NLF-S with
19

---

Table Ablation of uncertainty estimation. We measure the average error of joints and vertices of SMPL on the 3DW benchmark, along with Pearson's s correlation (PCC) between the predicted pointwise uncertainty and the true error. Naively optimizing the log-likelihood harms the mean predictions, but the INL.. loss [95] reduces the degradation and improves uncertainty quality. (NLF-S
nonparam., on full 3DPW, 24 joints)
MPJPE4
MVEJ
PCC-Jointst
PCC-Verticest
Euclidean loss (no uncertainty) Negative log-liiklihood (NLL) loss 3-NLL loss [95] (3 1)
59.2
71.3
61.2
73.6
0.50
0.34
59.4
71.5
0.54
0.40
no image batching, and about 8000 points to be predicted (mesh vertices and skeletons), forwarding the field MLP to obtain convolution weights takes 7.7 ms, while the rest of the network including the backbone takes 12.7 ms. For NLF-L with batch size 64 the latter takes 587 ms, making the MLP cost negligible in comparison even if we do not precompute it.
E
Uncertainty Estimation
To also consider our uncertainty prediction in the loss computation, we use the negative log-likelihood loss as
P.N.L. - + +og a.
(10)
However, we noticed that this significantly diminishes the prediction quality for localization com- pared to the variant without predicting uncertainties. We attribute this to the effect described by Seitzer al. [95], wherehy the NLL loss downweights challenging training examples and impedes the learning progress. We therefore also experiment with applying their proposed (-NLL loss, which looks follows when adapted to our context:
|p'' (IIB' P'l +
L
+logoo log -g(o))
(1I)
where sg(-) the "stop gradient" operation that blocks the gradient flow during backpropagation. Intuitively, for / = 1, multiplication by sg(a) "cancels out" the division of || p'll by a, resulting in the same gradients for P as in the case of vanilla Euclidean loss (removing the above-mentioned downweighting of challenging examples), without impacting the gradients for a (due to sg(-)).
To evaluate these different options, we measure Pearson's correlation coefficient (PCC) between the predicted pointwise uncertainty and the true error for all joints and vertices the SMPL body model on the 3DPW benchmark test set, using all 24 joints and 6890 vertices. As seen in Tab. 9, the standard NLL loss significantly harms the mean prediction quality, but SNLL largely fixes this problem and also yields uncertainties that are better correlated with the true error.
To illustrate that the obtained uncertainties are visually plausible, we show an example in Fig. 5, where the occluded hand has high predicted uncertainty, in line with intuition.
F Baseline Architecture Comparison without Localizer Field
To emphasize the key importance of using a localizer field in tying together the heatmap-producing convolutional weights for nearby points of the body, perform the following ablation. We create baseline architecture which uses no localizer field, and instead defines separate Iearnable parameters for each SMPL vertex and each joint of each skeleton format.
As seen in Fig. when using the baseline model for inference, the different skeleton format predictions are visibly inconsistent with each other and with the SMPL mesh (see e.g. the H36M arm outside the SMPL body in the lower example), since the weights to localize each point have no enforced relation to each other. This also results in more scattered vertex predictions (see e.g. the hand region). NLF, by contrast, ensures that the different skeletons are localized consistently with each other, and the spatial smoothness of the prediction is improved.
20

---

100
150
250
350
50
100
150
200
250
300
350
Figure 5: Uncertainty estimation results. High uncertainty is indicated in yellow, while low is shown in blue. Occluded body parts tend have higher uncertainty prediction.
NLF
450
100
300
100
1000
Baseine
NLF
K30m
zahoo
Figure 6: Side-views of the predictions by our NLF vs. baseline architecture that trains separate, explicit convolutional weights for each skeletal joint and SMPL vertex.
21

---

11.2
11.0
10.8
PCE-1-5(mmm)
10.6
PVE-TSC
mlou
10.4
0.841
10.2
10.0
Witerations
Figure 7: Convergence of SMPL fitting. We analyze the convergence properties of our iterative SMPL fitting algorithm the SSP-3D benchmark. We the nonparametric predictions from NLF-S model as the fitting target. We can observe that approximately three iterations are sufficient for convergence to the SOTA result. (For PVE-T-SC lower is better, and for mIoU higher is better.)
18000
0.844
16000
0.842
14000
whroughhsst (1:50
12000
miuu
10000
0.838
MioU
thraughput
8000
0.836
6600
0.834
4000
#verrices used in fitting
Figure 8: Vertex-subset-basee fitting of SMPL. For additional efficiency, we evaluate fitting SMPL to only a subset of the 6890 vertices. The joints are still all used. To decide which vertices become part of each subset, we simplify the template mesh by quadric decimation and choose the vertices distance-based Hungarian matching. We use 3 iterations in all cases.
G Efficient Body Model Fitting
In Algorithm we provide the simplified pseudocode for our body model fitting algorithm used in the main paper. The following is an intuitive description of the steps. After initialization at the default T-pose, we iterate over the body parts. For each body part, we select the vertices and joints that form part of this body part. (Note that non-endeffector joints belong to two body parts as well.) We then use the (weighted) Kabsch algorithm to find the rotation matrix between the body part points in the current fit and in the input (fitting target).
body parts, it is important that fit matches the orientation of the bone vectors as closely as possible, so that there no error accumulation along the kinematic chain. We therefore weigh the joints much higher than the surface vertices in the rotation estimation. We specifically set weight of 10 for vertices. Even though this weight is tiny, the vertices are meaningfully used well, since many body parts consist of single bone with only two joints, and fitting rotation to two joints would leave one degree of freedom ambiguous (the rotation around the bone's s main axis). By also including the vertices albeit with small weight this additional degree of freedom is determined by the vertex positions. For body parts that contain at least three joints, the rotation based practically solely on the position of these joints (e.g., the pelvis joint is the parent of three joints: two hips and one spine, making this body part contain four joints). Note that these estimated rotations are not parent-relative. Fach body part's orientation fitted independently of each other, to avoid any error accumulation.
Since most joints connect
22

---

Iterations
25
37
50
100
Time min 18 s
6 min 54 s
16 min 35 s
33 min
Error
14.0 mm
8.0 mm
6.2 mm
5,0 mm
Table 10: Evaluation of the official SMPL-to- SMPLX converter according to total used time and achieved average error for 33 sample meshes under default settings and various maximum iteration counts. The time required is on the order of minutes, while our efficient body model fitter completes the same task with 7.8 mm error in about 28 milliseconds.
Table 11: Ablation of uncertainty- bassed weights body-model fitting. Weighting points according to their estimated uncertainty brings modest benefits on 3DPW and EMDB (NLF-L model).
3DPW-test (14 joints)
EMDB1 (24 joints)
MPJPEJ
P-MPJPEL MVE.
P-MEE,
MPJPEL
P-MPJPEL MVEI P.MVEL
w/o weighting 59.3
36.6
69.6
48.8
68.9
41.1
81.2
51.4
w/ weighting 59.0
36.5
69.7
48.8
68.4
40.9
80.6
51.1
The body model is then forwarded with the new orientation parameters, yielding new estimated fit vertices and joints, be used in the second step.
In the second step, the 3 shape parameters and the translation vector are estimated. Since SMPL uses linear blendshapes and linear blend skinning, the vertex and joint positions are linear function of the shape parameters (while treating the pose rotation parameters as fixed). We simply need to compute the Jacobian matrix expressing this linear relationship (which involves traversing the kinematic tree). Given the Jacobian, we can solve for 3 (and t) via L2-regularized linear least squares (solved via Cholesky decomposition). L2 regularization is used to avoid obtaining unrealistic shape vectors that have too large coefficients. We found that it is best not to penalize the first two components of 3, which correspond the person's general size (height and weight), in order to avoid biasing the process to an average-sizz sized shape too strongly.
After obtaining the new shape estimate, can re-estimate the per-body-part orientations, and this time the body parts have better shape correspondence, hopefully resulting in a more accurate orientation estimation. With the more accurate orientations, we re-estimate the shape parameters, and so on.
As final step, re-estimate the orientations, but now not independently per body part (anchoring at the mean of each body part), but sequentially, traversing the kinematic chain, anchoring the rotation estimation of each body part at the position of its main joint (as determined by the previous rotation estimations). Here there is no reason to downweight the contribution of vertices, as the sequential estimation along the kinematic chain ensures that the body parts will link up correctly. This step reduces misalignment and error accumulation due to mismatched bone lengths between the shape fit and the nonparametrically estimated skeleton.
This process converges very quickly, as shown in Fig. 7, with the main improvement happening between the first and second iterations. From 3 iterations onward, convergence is reached.
To increase efficiency further, we may use only subset of the vertices of the body model during fitting. In Fig. 8, we can see that, e.g., by fitting to only 1024 vertices, we increase the throughput by a factor of 3, while mIoU only decreases from 0.845 to 0.843. The timing was measured on an NVIDIA RTX 3090 GPU with batch size of 2048.
In the model transfer experiment described in the main paper, where fit SMPL-X parameters to clean SMPL mesh vertices with our fitting algorithm, we do not regularize the betas (as the input is a clean mesh without noise), one iteration and 4096 vertices for fitting (out of the 10475). We also provide a more detailed comparison to the official model transfer code in Tab. 10, showing that our algorithm is much faster even the same accuracy levels.
As described in the previous section, our algorithm outputs uncertainty estimates per point. These can be used to obtain a weighted version of the fitting algorithm. Specifically, given estimated uncertainty for point i, we use the weighting factors w; -115 when solving for the rotations and the shape coefficients. As shown in Tab. 11, this gives a slight improvement on both 3DPW and EMDB.
23

---

Algorithm 1 Efficient fitting of body model M to given target vertices V and joints J. Iss(3)) denotes projection to SO(3) via SVD, the subscript {k} selects joints or vertices that belong to body part k. J is the joint regressor. We use
10 in practice.
Require: VER N JER xx3, I,N, a E [0, 1]
V - Vn, JV, t+ 0, Rs <- Tax3
D. Initialize the fit to mean-shaped T-pose D 1-4 iterations
for i [1../] do
> (1) Keeping 9 and t fixed, solve for global rotations R via weighted Kabsch
for k E [1..K] do
P(k} - [[K)), J4n]
P Concatenate vertices and joints together to a list of points
P(a) IVin], [un]
W [a, 1- ai
Weights for the vertices and joints, respectively
Eg WeightedCov
((K) P(a): W Weighted cov. mat. with vertices downweighted
R3 + -I[so(3) (E
> Project to the L2-nearest rotation matrix via SVD
Pose the vertices and joints with newly fitted R D (2) Keeping the rotations R. fixed, solve for shape 3 and translation t
V, + MRR,, 3) +
P- [.
> Concatenate vertices and joints together for uniform treatment
P+ [V
Compute the Jacobian V3, P via forward-mod autodiff. along the kinematic tree 3, (V (P-P)
> Linear least squares (with regularization)
> Update vertices and joints with new B, t D (3) Keeping 3 and fixed, refine the rotations R by traversing the kinematic tree
v.J + M(R, 3) +
Jnew + J1
> Root position is not changed
for k E [L..K] do
In topological order of the kinematic tree
p + parent-index(())
For non-root, determine Jaew from jnew
Rp and the T-pose k-p bone implied by B
Paa < [V(x] Jnew J(m)= jrem b Get the estimated body part relative to new pivot
Piuy + [V[N] - Ju. J(K] Ju] "Covariance" anchored at the respective pivots instead of the means
P Get the fitted body part relative to its (old) pivot
Ea +- P(K) P P(())
Rs < (E,) RE
return B,
We will make all code publicly available, including the fitting algorithm, ensure its precise reproducibility.
H Neural Field Architecture and Details
Figure shows the architecture of the neural field that parameterizes our point localizer network. The neural field has an MLP-like structure, starting with learnable Fourier features (fully connected layer followed by sine and cosine activations). After two further fully connected layers with GELU activation inbetween, we arrive at the layer whose output is initially trained to approximate the global point signature (GPS) derived from the volumetric Laplacian. Further two FC layers with GELU inbetween yield the parameters to modulate the convolutional layer of the point localizer network as explained in the main paper.
We perform the following steps to obtain the global point signature and bake it into our neural field as initialization. We take the SMPL mesh in our canonical pose and finely subdivide it with Delaunay tetrahedralization. We process the resulting tetrahedral mesh with finite element methods to compute the eigenbasis of the Laplacian evaluated at each node the tet mesh. These samples become the training set for training an MLP on top of learnable Fourier feature positional encoding. After this initial setup, the main training commences, where the weights of the neural field are tuned further, together with the backbone net.
24

---

W, b A
FC: 1024 384
GELU
FC: 1024 384
Pretrained to output g global Focin deccgp Comeeriii
FC: 2048 1024
GELU
FC: 2048
Learnable rourier
sin
cO5
FC: +26
p E Qg
Figure 9: Architecture of the neural localizer field itself.
Table 12: Results 3DPW (24 joints). In contrast to the main paper, here the whole 3DPW dataset used for evaluation, not only the part designated as test set, and all 24 joints are evaluated directly. Method
MPJPE
P-MPJPE
MVE
P-MVE
AbSCAASS [93]
60.6
41.7
NLF-S
59.2
41.0
71.7
52.0
NLF-S +fit
59.4
41.5
71.1
50.8
McTRAbs-ACAE-L [93]
58.9
39.5
NLF-L
59.2
39.5
71.0
49.2
NLF-L +fit
59.0
40.0
70.1
48.3
I
Additional 3DPW Protocol
For fair comparison to [93], in Tab. 12, we provide 3DPW evaluations matching their evaluation, i.e., using the entire dataset the test (and none of for training), and evaluating all 24 joints directly (without applying a Human3.6M-like 14-joint regressor).
J Further Qualitative Results
Figs. and 11 we show further results of NLF- with SMPL-fiting post-processing (rendered using the Blendify framework [32]).
25

---

Figure 10: Qualitative results in the wild.
26

---

Figure 11: Qualitative results in the wild.
27

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Ys]]
Justification:
We substantiate each claim through experimental evaluation.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed the end of the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or a speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
28

---

This paper is focused on experimental work and does not prove theorems or other theoretical results.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We describe experimental details in the appendix. Code release will ensure the details of reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but should possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
29

---

Question: Does the paper provide open access to the data and code. with sufficient instructions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We will make our code and models available for research use before the conference takes place. We will also provide data processing scripts. However, the code is not part of the submission package at this time.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips.ce
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips
http:/oooduuiissssooooiiiiioiiie ssionPolicy) for more details. http///lllgggudds//eeeee
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Ye]]
Justification: We provide necessary details for understanding the method and its impact, and will provide the full code to exactly specify the minor details.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We specify the standard error of the mean for MPJPE on 3DPW, It is also worth noting that carefully seed the pseudorandom number generators the same way between trainings, and we verify that we obtain the same weight initialization, the same training example order and the same augmentation sequence in all runs. This reduces the experimental noise, leading to more reliable comparisons.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes"if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
30

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: (Ys]]
Justification: We specify the hardware (GPU type, memory and count) as well as the training time length in the main paper.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (c.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethies httt:s///enrrpsscc/pplliiiiiiiii hctpppcccc/tthiiccsiiee.........
Answer: [Yes]
Justification:
We have thoroughly reviewed and conform to the code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss positive and negative potential uses.
Guidelines:
The answer NA means that there is socictal impact of the work performed.
31

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We do not release any dataset ourselves, and our model will only be made available for research use.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All used datasets are identified unambiguously through citations. We do not release any dataset ourselves.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
32

---

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/coo////oooo con/datasess has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We do not release new assets as part of the paper submission. Models will be made available exclusively for academic research purposes later time.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We perform crowdsourcing nor research with human subjects. We use standard academic datasets, and did not collect our own data for this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
33

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval. you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
34


Reviews:
[{"review_id": "9cJdQfDCDj", "reviewer": "Reviewer_Dcsh", "summary": "This paper deals with the task of 3d human pose estimation. It contains three main contributions:\n\n1. A hypernetwork that takes as input a point in a 3d body volume (in a canonical pose) and outputs the weights of a network (a single layer, really) that, when applied to the features of a vision backbone, is able to localize said 3d point in R^3 given an image (plus a 2d point and 2d uncertainty).\n2. An application of this approach to train on multiple datasets with SMPL and SMPL-like annotations, 3d, and 2d annotations.\n3. An algorithm to fit SMPL parameters given joints and vertices.\n\nThese approaches, combined, (plus a series of engineering tricks, such as creating a synthetic dataset and treating some annotations as themselves learnable) result in network that yields state-of-the-art results on several 3d pose estimation benchmarks.", "questions": "0. Could the authors elaborate on how datasets with 2d and 3d annotations are used for training? How does the \"approximate initialization\" work? Is this some approximate initialization to 3d registration (via SMPL fitting)?\n\n1. The supplementary material discusses the creation of a large synthetic dataset using SMPL fittings of the DFAUST dataset, which is not mentioned in the abstract or the paper -- how important is this for the overall results?\n\n2. What is the dimensionality of the volumetric heatmap? Is this depth defined over the entire scene or only over the depth of the human body? If so, is the range of the function over the entire $\\mathbb{R}^3$ in the human body, or a discretized subset?\n\n3. Why does the architecture predict a 2d and a 3d heatmap? Is it possible for the 2d heatmap to disagree with the projection of the 3d prediction?\n\n4. The last two layers of Fig 6 show FC layers going from 1024 to 384 dimensions, and later going from 1024 to 384 again. Is this a typo? If so, what does the actual architecture look like?\n\n5. The paper uses the number 384 several times in seemingly unrelated areas\n* The size of the images used in the larger network variant\n* The number of channels predicted by the localization field (or is it both the size of the input plus output?)\n* The number of points sampled from the interior of the human volume\n\nIs this a coincidence?\n\n6. What is the time it takes the official SMPL fitting code to achieve an error comparable to the one achieved by the proposed method?", "limitations": "Limitations are addressed adequately.", "rating": 7, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "## Originality\nUsing a hypernetwork to predict arbitrary points in a human volume is a novel idea. Putting together a super dataset for this task is very nice and novel as well as far as I am aware.\n\n## Quality\nThe results, whether they come from a novel architecture or a novel super-dataset, are strong across the board.\n\n## Significance\nRegardless of the soundness of the contributions, the fact that the paper promises to make all the contributions easily reproducible is a big plus. The field could really benefit from a way of sourcing multiple datasets together, and I can see multiple people building upon the ideas presented here if everything is released in decent shape.", "weaknesses": "## Soundness\nThe main weakness of this paper is the lack of experiments that independently test the importance of each of the contributions. The paper proposes two main ideas: a hypernetwork for 3d human modelling, and a superset of datasets used to train this system; the former being primarily a methodological contribution, and the latter being primarily an engineering contribution. Unfortunately, there is no experiment or ablation distilling the importance of each contribution. Concretely, this could be achieved by, for example\n* Training the novel architecture on a single dataset\n* Training the novel architecture on a subset of the compiled datasets (eg, on the datasets with SMPL annotations), or\n* Training a baseline architecture on the superset of datasets (or a subset thereof, such as the ones with SMPL annotations)\n\nThese results would help the readers understand whether and to what extent the access to more data or the novel architecture make a difference in the SOTA results reported. As is, this crucial question remains unfortunately unanswered, and takes away from what would otherwise be a very, very strong paper.\n\nI think these experiments are extra important because the paper is implicitly making a very bold and counterintuitive claim: that by posing the task of 3d human pose estimation as 3d registration (a more complicated task), it is possible to achieve better 3d poses than SOTA. Furthermore, this is achieved by exploiting data that is not annotated for 3d registration; this is very counterintuitive and, in my opinion, likely to be untrue. Therefore, I am inclined to think that it is the extra data that helps the most towards the strong results.\n\n## Clarity\nIn my opinion, the treatment of the \"localizer field\" is overly convoluted. While yes, it is true that the localizer field technically defines a neural field of functions, the paper makes it sounds like this is a very new idea (L163-164 \"Although neural fields are typically used to predict points or vectors, here we use them to predict localizer functions\"). This is not the case; at the end of the day this is a hypernetwork, which has been a staple of work in human modelling for a long time (eg [a, b]). The authors seem to be aware of this connection, since the paper mentions that the localize field \"modulates\" (L731) the convolutional layer of the point localization network, which is the terminology used in [a] for hypernets. I believe S3.2 could benefit from rewriting to make this part clearer and more in line with previous notation and descriptions.\n\nRe: Efficient body model fitting. The method is described as really fast, compared to the official code which is said to take 33 minutes and achieves a slightly lower error. Most optimization methods have exponential error decreases, so it is not uncommon to see exponentially longer times for slightly lower errors. I think it would be clearer to plot the error as a function of time for both the official and new methods.\n\nRe: Using 2d and 3d annotations. I am unable to understand how datasets annotated with only 3d poses are used to supervise an approach to volumetric registration -- the description in the paper is very terse (1 line). Is this done by fitting SMPL to the 3d points and obtaining an approximate place in the human volume? If so, it seems like training with these fitted SMPL meshes would be another baseline worth trying; ie, bring all the datasets to SMPL, then train on it. This would further disambiguate whether the architecture or the use of extra data is the main contribution.\n\n[a] Karras et al, A Style-Based Generator Architecture for Generative Adversarial Networks, CVPR'19\n[b] Chen et al, Authentic Volumetric Avatars from a Phone Scan, SIGGRAPH'22", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that R1 sees our idea as \"novel\", further acknowledging the novel aspect of putting together such a \"super-dataset\". R1 further praises the model quality as \"strong across the board\" and foresees significant community impact.\n\nImportance of the hypernetwork and datasets: see the global answer.\n\nComplicated explanation of localizer field, simply a hypernetwork: our description emphasizes the connection to neural fields, given the similar 3D-spatial input and the use of positional encodings. However, we agree that the hypernetwork view is also important. While we do mention this connection and cite the HyperNetworks paper (Ha et al., 2017), we will extend the manuscript with further connections to other relevant uses of hypernetworks.\n\n> how datasets with 2d and 3d annotations are used for training? How does the \"approximate initialization\" work?\n> Is this done by fitting SMPL to the 3d points[...]?\n\nWe will make our existing explanation around L197-214 clearer. Importantly, our formulation allows us to sidestep the generally ill-posed problem of converting annotations to a single format, such as SMPL. Each dataset with 3D joint annotations can have different skeleton formats, i.e., the joints may designate different anatomical locations, e.g. Human3.6M's shoulder point is not the same as the shoulder point of the MPI-INF-3DHP dataset. We directly train the network with points that are annotated for a training example, i.e. we query those points for which we have annotations, so we can compute and minimize the average loss for those points. For this, we need to know where to query the field, e.g. where the Human3.6M shoulder point is in the canonical space. (For SMPL this is given, since the canonical space is derived from a SMPL mesh of mean shape and a T-like pose). Training can be started with an approximate placement of e.g. the Human3.6M shoulder point in the canonical human volume in the general shoulder area, and we let the gradient-based optimization update all parameters jointly, including the backbone parameters, the neural field parameters and the 3D query location of each skeletal format in the canonical space.\nDatasets with 2D point annotations are treated similarly, except that the prediction is projected before computing the loss in 2D.\n\n> bold and counterintuitive claim: that by posing the task of 3d human pose estimation as 3d registration (a more complicated task), it is possible to achieve better 3d poses than SOTA. Furthermore, this is achieved by exploiting data that is not annotated for 3d registration\n\nWe do not make such a “bold claim” in the paper, instead our claims are as follows. We are able to jointly train on heterogeneous data sources, some annotated for 3D/2D skeleton pose estimation with different formats as well as some for 3D human mesh recovery (also different ones, SMPL/SMPLX/SMPLH male/female/neutral), obtaining a single model that achieves SOTA on both kinds of tasks. Our work enables this common treatment by designing a generalist model that can estimate any arbitrarily chosen points, and then by casting each task as a point localization task (with different point sets) that are simple to tackle with the proposed generic point localizer.\nThe end goal is to obtain a strong model for body pose and shape estimation, that is test-time configurable for user-chosen skeleton formats and mesh formats. Our model is designed to make spatially smooth predictions w.r.t. the selected points, resulting in consistent predictions for the different output formats.\n\n> The supplementary material discusses the creation of a large synthetic dataset using SMPL fittings of the DFAUST dataset, which is not mentioned in the abstract or the paper -- how important is this for the overall results?\n\nIndividually ablating the effect of each dataset is computationally infeasible. However, we provide experimental results with using only synthetic or only real 3D-annotated training examples, see the global answer.\n\n> dimensionality of the volumetric heatmap? Is this depth defined over the entire scene or only over the depth of the human body?\n> Why does the architecture predict a 2d and a 3d heatmap? Is it possible for the 2d heatmap to disagree with the projection of the 3d prediction?\n\nThe heatmap does not cover the entire scene but a cube of side length 2.2 meters around the human. The 2D and 3D heatmaps are used in order to estimate the human scale and distance from the camera. They could theoretically disagree, but we did not observe such a problem in practice - training them together results in compatible predictions.\n\n> The last two layers of Fig 6 show FC layers going from 1024 to 384 dimensions, and later going from 1024 to 384 again. Is this a typo? If so, what does the actual architecture look like?\n    \nThe architecture was designed like this to express the Global Point Signature-based positional encoding, which is inspired from [72] in 2D surface modeling. The 1024-dimensional vector is hence initially trained to output the Global Point Signature. As explained in the paper from L268, we found it best to finetune the full MLP after this initialization. Without this special pretraining, the two linear layers could indeed be replaced by a single layer without losing expressivity.\n\n> The paper uses the number 384 several times in seemingly unrelated areas\n    \nThere is no special connection there, the reason is simply that 384 is the sum of two high powers of 2 (256+128) and hence a \"round number\" in binary. Tensor sizes divisible by high powers of two are often more convenient and hardware-efficient in practice.\n\n> What is the time it takes the official SMPL fitting code to achieve an error comparable to the one achieved by the proposed method?\n\nPlease refer to Table 2 in the PDF. The official code runs for about 7 minutes (for all 33 samples included with the code) achieving an average error of 8.0 mm, while our method achieves 7.8 mm in just 13 milliseconds.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fyf92z4h3I", "reviewer": "Reviewer_M7cf", "summary": "This paper focuses on 3D human pose and shape estimation from a single RGB image. The main insight is, to avoid the influence of a fact that different human pose dataset defines different skeleton in their annotations, this paper proposes a point-based representation to ensure the model can learn from many datasets without suffering the skeleton mis-alignment problem between existing datasets. The idea follows the mechanism of dynamic convolution, encoding a point (inside the 3D human body) in canonical coordinates as the weight of the dynamic convolution, and converting the image features into a heat map that estimates the 3D position of the point in the target 3D human body mesh. The proposed model is trained with nearly 50 datasets with different annotations, including SMPL parameters, 3D / 2D keypoints, densepose, etc. Then they compare the performance on multiple benchmarks.", "questions": "1. About fair comparisons. \nIf we use the same training dataset but remove the point-based representation, will the results be similar? To what extent does this new point-based representation help? However, the current paper does not answer this fundamental question very well.\n\nIn rebuttal, this question doesn't get well answered. The concern about what is one solid technical contribution of this paper is still there.", "limitations": "No.", "rating": 4, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Extensive ablation studies to exploit the effects of many important settings, e.g. different way of encoding canonical position, uncertainty estimation. These results would be helpful for HMR developer to make better decision on model designs. \n2. Great qualititive results on InterNet videos, especially the 2D alignment seems pretty well. \n3. Video demo of sampling random canonical points proves the effectiveness of position encoding.", "weaknesses": "1. The quantitative comparison is not fair and cannot verify the superiority of the proposed point-based representation over previous ones.The model is trained with nearly 50 datasets, while none of the compared methods are using the same experiment settings. Without fair experiment settings, readers can't tell wether the proposed point-based representation helps or not. See the following questions section for details. \n2. Some typos. For example, a period is missing between \"process\" and \"I\" at L#213. At L#215, \"We use EfficientNetV2-S (256 px) and EfficientNetV2-L (384 px) [98]\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer M7cf (R2) for the suggestions. R2 notes that we performed \"extensive ablation studies\", which are seen as \"helpful\" for future model design decisions, and further praises our strong qualitative results with good pixel alignment.\n\n> If we use the same training dataset but remove the point-based representation, will the results be similar? To what extent does this new point-based representation help?\n\nPlease refer to the global response regarding the role of the neural-field-based point querying.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qst9XSsfPL", "reviewer": "Reviewer_YzeS", "summary": "The authors proposes a Neural Localizer Field (NLF) to learn a continuous representation of the canonical human pose by learning to predict a set of functions that map a query point in the canonical human volume to a point in the human posed space, given a single rgb image. By introducing a meta-learning architecture, they are able to train on diverse datasets with different annotations in both 2D and 3D. The authors claim that this scaling from a large number of datasets allows for a better pose predictor than prior work and show relevant results.", "questions": "* Its not clear what the canonical space representation is. Since the query points share the same domain across multiple datasets, I presume all of them are sampled from a single canonical space. But Fig. 4 column 1 shows query points that are with respect to each dataset. \n\n* Corresponding to the previous comment, how are 3D losses weighted across datasets such that scale is handled appropriately?", "limitations": "Yes, authors have adequately addressed limitations and societal impact of the work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Clear Insight/Idea: The insight is simple, clearly explained and well motivated. Having a single architecture that ingests all sorts of human pose, shape annotation would certainly benefit from the diversity if handled correctly during training time. Although, the current architecture might not be the *best* design choice, the paper does show that a simple architecture + large datasets boosts metrics. \n\n* Impressive results: The quantitative metrics in Table 2-6 are quite impressive and perform better on most comparison axes. The shape estimation results also are convincing and show benefits from better pose prediction. \n\n* Although not trained for temporal stability, the method does show some temporal stability in the supp. video.", "weaknesses": "* Better data inspection: The core contribution is that simple architecture + more data gives better results. Since data is the main focus here, a thorough ablation on the data sources is missing. Its not clear if the performance is derived from just a few data sources or all of them i.e. how each dataset affects results. Without this understanding, its hard to argue that more diverse data improves metrics while a few datasets might have the biggest quality impact.\n\n* Extent of generalizability: Usual suspects for human pose&shape estimation failures/limitations are loose clothing, occluded views, unique poses. It would be nice to see how the method works on such cases and if the method generalizes well to such cases. Additionally, points that are often not annotated in pose estimation datasets might be prone to failures. The lower accuracy for face and hands in Table 5 makes me believe that this could be the case. It would be great if the authors could comment on the performance of the method in such cases. \n\n* 3D Loss weighting: Given the 3D loss for 3D datasets, the network would have to account for the different dataset scales which might be widely off. This can affect training and test time results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer YzeS (R3) for the review and questions. R3 considers that our \"insight is simple, clearly explained and well motivated\", and finds the quantitative metrics \"impressive\" and \"convincing\".\n\n> Its not clear if the performance is derived from just a few data sources or all of them i.e. how each dataset affects results.\n\nWhile an extensive ablation for assessing each individual dataset's contribution is computationally not feasible, we provide ablations for using only synthetic or only real 3D data. Please refer to the global answer regarding this.\n\n> Extent of generalizability: Usual suspects for human pose&shape estimation failures/limitations are loose clothing, occluded views, unique poses. It would be nice to see how the method works on such cases and if the method generalizes well to such cases.\n\nWe include further qualitative examples in the attached PDF that cover such cases. (Such challenging examples are rare in quantitative benchmarks, hence the qualitative examples.)\n\n> lower accuracy for face and hands in Table 5\n\nAlthough our focus throughout the paper is mainly on body pose and body shape, we still achieve second-best scores for hands and faces in Table 5 (AGORA benchmark). It is true that many of the training datasets do not provide detailed annotations for hands and faces, and we indeed attribute our lack of SOTA results on hands and faces to this property of the training data.\n\n> different dataset scales which might be widely off\n\nWe adjust for different dataset scales (i.e. different number of training examples in each) by sampling more training examples from larger datasets. We did not tune these sampling-proportion hyperparameters, as this can lead to combinatorial explosion and would be very resource-intensive.\n\n> Its not clear what the canonical space representation is.\n\nThe canonical space is defined in reference to a T-like pose of the default SMPL mesh. All other keypoints (as e.g. shown in the referenced Fig. 4 column 1) are represented in this same coordinate system, as points within this canonical human volume. Again, no explicit conversion between formats is necessary and the exact locations of points in the canonical volume are tuned automatically during training. We will include this important point in the paper, and we thank R3 for calling attention to this omission.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qx4WIkI0mx", "reviewer": "Reviewer_R9xH", "summary": "This paper proposes Neural Localizer Field, a continuous field of point localizers, for localizing any point of the human body in 3D from a single RGB image. The method enables mixed-dataset training using various skeleton or mesh annotation formats. The method has three main parts: a point localizer network, a neural localizer field, and a body model fitting algorithm. Trained on a mix of datasets with different annotations, the model achieves good performance across various benchmarks.", "questions": "How does the model do on small details such as fingers?", "limitations": "Limitations (lack of temporal cues) and are explained.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "+ Novelty. The idea of utilizing a neural field to unite different data sources is interesting and novel. The paper does a good job in explaining the motivation as well as laying out the technical details.\n\n+ Impressive performance on extensive benchmarks and experiments. The method enables training with multiple annotation sources, and it achieves better performance as compared to SoTA. Notably, it achieves good performance on shape prediction, which is a hard problem in human mesh recovery due to the lack of training data with shape annotations.", "weaknesses": "- Lack of discussion on inference speed. Since this method would require on the fly point inference, I wonder what is the inference speed and whether this method is suitable for real-time inference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer R9xH (R4) for the assessment and questions. R4 considers the idea \"interesting and novel\" and finds that we do \"a good job in explaining\" the motivation and technical details. R4 further sees the performance as \"impressive\" on \"extensive benchmarks and experiments\".\n\n> Lack of discussion on inference speed. [...] whether this method is suitable for real-time inference.\n\nThe method is suitable for real-time inference. NLF-S has a batched throughput of 410 fps and unbatched throughput of 79 fps on an Nvidia RTX 3090 GPU. For NLF-L these are 109 fps and 41 fps respectively. (Bounding box detection needs to be performed on top of this, but fast off-the-shelf detectors are readily available.)\n\nNote that NLF’s inference-time overhead (for predicting weights through the field MLP) can be eliminated by precomputing the weights once for a chosen set of canonical points. (Typically one wants to localize the same points, i.e. same skeleton formats, for many images.) For reference, in case of NLF-S with no image batching, and about 8000 points to be predicted (mesh vertices and skeletons), forwarding the field MLP to obtain convolution weights takes 7.7 ms, while the rest of the network including the backbone takes 12.7 ms. For NLF-L with batch size 64 the latter takes 587 ms, making the MLP cost negligible in comparison even if we do not precompute it.\nWe will add this information to the paper.\n\n> How does the model do on small details such as fingers?\n\nFor this, we refer to our AGORA results on hand keypoints (Table 5, RHan and LHan columns), where we achieve second-best results. However, given our focus on body pose and shape, most of our training datasets do not contain detailed finger annotations, and hence we can consider the second-best results obtained for this subset of keypoints still a strong result (see also our answer to R3.)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their thoughtful suggestions and questions. Their assessments are unanimously on the positive side, recommending acceptance. R1 (Dcsh) sees both our architectural idea and our extensive dataset combination as "novel" and foresees significant community impact. R3 (YzeS) finds that our "insight is simple, clearly explained and well motivated". R4 (R9xH) considers our idea "interesting and novel" and finds that we do "a good job in explaining" the motivation and technical details. R2 (M7cf) highlights our "extensive ablation studies" as "helpful" and R4 commends the "extensive benchmarks and experiments". All four reviewers emphasize our results that are "strong across the board" (R1), have accurate pixel alignment (R2), are "impressive" (R3, R4) and "convincing" (R3).

A common question by reviewers is about separately analyzing the contribution of our novel architecture and the effect of data scale. First, we emphasize that the two are linked: thanks to our architectural contribution, we can train from multiple datasets that are annotated with different pose and skeleton formats. This would otherwise require tedious and difficult conversions – for example it is not well-defined to convert a sparse skeleton to the full body pose of SMPL as some degrees of freedom such as shape, and axial arm rotation is missing, and differences in the skeleton definitions and joint placements can introduce further problems. It is even less well-defined if there can be any number of missing joints in each training example, which is typically the case for datasets that are triangulated from multi-view 2D predictions.

Nevertheless, we include additional experimental results obtained with newly trained ablation models to demonstrate that both our architectural contribution and the data scale are important.

The architectural contribution (i.e., localizer functions encoded as a neural field / hypernetwork) is ablated by training a baseline model where separate, explicit convolutional weights are learned for localizing every skeletal point and every SMPL vertex, instead of predicting these weights via an MLP. As shown in Fig. 1 (note that the 3D views show a rotated side view), the different skeleton formats in the baseline prediction are visibly inconsistent with each other and with the SMPL mesh; see e.g. in the bottom example how the green H36M arm is outside the SMPL body for the baseline. This is because the weights for localizing each point have no enforced relation to each other in the baseline. This also results in scattered and disorganized vertex predictions (see e.g. the hand region). NLF, by contrast, ensures that the different skeletons are localized consistently with each other and with the mesh, and the predicted mesh is spatially smooth and less scattered. (Note that these aspects are not straightforward to measure quantitatively on individual benchmarks.)
Note also that the baseline architecture requires one to pre-determine and fix the number and definition of the points before training; they cannot be changed at runtime by the user. Furthermore, increasing the number of points that the baseline network can predict requires linearly scaling the number of network parameters in the prediction head. In contrast, NLF allows choosing arbitrary points at test time, and its number of parameters is independent of how many different points we want to be able to localize.

The data contribution is ablated by training our novel architecture also on subsets of the 3D datasets - first only on the synthetic datasets (computer graphics renderings) then only on the real ones (photos). (The 2D-annotated real datasets are always used). As Tab. 2 shows, the best results are achieved when combining these data sources. (All results in Tab. 2 are obtained with the small model and a shorter training of 100k steps compared to the 300k used in the main paper, due to rebuttal time constraints.)

We answer the further, individual questions as a reply to each review.

Meta-review: The paper was championed for acceptance. One condition that was explicitly mentioned is the release of the codebase, especially if it is easy to use and enables the use of multiple datasets for 3d human pose estimation as a single unit. This was recognized as a limitation that is holding back progress in the area, and this paper has the potential to change that. 

The main concerns remaining were soundness and presentation. In particular, it seems that all pre-processing steps require some sort of explicit mapping to SMPL. The reviewer argued that this step may be inevitable today, and this paper does the next best thing by using synthetic data to bridge that gap.
Decision: Accept (poster)