A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose
Authors: Shih-Yang Su
Frank Yu
Michael Zollhoefer
Helge Rhodin
Venue: NeurIPS 2021 Poster (2021)
Keywords: neural scene presentation
3D human modeling
pose estimation
character animation

Abstract: While deep learning reshaped the classical motion capture pipeline with feed-forward networks, generative models are required to recover fine alignment via iterative refinement. Unfortunately, the existing models are usually hand-crafted or learned in controlled conditions, only applicable to limited domains. We propose a method to learn a generative neural body model from unlabelled monocular videos by extending Neural Radiance Fields (NeRFs). We equip them with a skeleton to apply to time-varying and articulated motion. A key insight is that implicit models require the inverse of the forward kinematics used in explicit surface models. Our reparameterization defines spatial latent variables relative to the pose of body parts and thereby overcomes ill-posed inverse operations with an overparameterization. This enables learning volumetric body shape and appearance from scratch while jointly refining the articulated pose; all without ground truth labels for appearance, pose, or 3D shape on the input videos. When used for novel-view-synthesis and motion capture, our neural model improves accuracy on diverse datasets.

A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose
Shih- Shih-Yang Sul
Frank Yu!
Michael Zollhöfer
Helge Rhodin
Uniierssity of British Columbia
2 Facebook Reality Labs
Abstract
While deep learning reshaped the classical motion capture pipeline with feed- forward networks, generative models are required to recover fine alignment via iterative refinement. Unfortunately, the existing models are usually hand-crafted learned in controlled conditions, only applicable to limited domains. We propose a method learn generative neural body model from unlabelled monocular videos by extending Neural Radiance Fields (NeRFs). We equip them with skeleton to apply to time-varying and articulated motion. key insight that implicit models require the inverse of the forward kinematics used in explicit surface models. Our reparameterization defines spatial latent variables relative to the pose body parts and thereby overcomes ill-posed inverse operations with an overparameterization. This enables learning volumetric body shape and appearance from scratch while jointly refining the articulated pose; all without ground truth labels for appearance, pose, or 3D shape on the input videos. When used for novel-view-symtheiis and motion capture, neural model improves accuracy diverse datasets. Project website: htttps://lemonl
github. 1o/ anerf/
1 Introduction
Generative models have evolved from Generative Adversarial Networks (GANs) recreating images
providing control and image understanding for downstream tasks via structured latent variables. However, most 3D models require 3D labels natural images and require dedicated depth sensors. It is hence important research problem to learn 3D representations from 2D observations, which is particularly challenging for humans with diverse body shapes and appearances and their non-rigid motion.
[4.21] to neural scene representations (39 56 57
that cannot be crowd-soureed
Modern human motion capture techniques typically combine the advantages of discriminative and generative approaches. A feed-forward 3D human pose estimation approach provides rough initial estimate of human pose. Afterward, a generative approach based on either high-quality 3D scan of the person [1],, or parametric human body model leameed from laser scans 44 refines the estimate iteratively based on the image evidence. Although achieving unprecedented accuracy, existing models require a low-dimensional, restrictive, shape body model or personalized 3D scan of the user.
We introduce Articulated Neural Radiance Fields (A-NeRF) for learning a user-specific neural 3D body model and underlying skeleton pose from unlabelled videos (see Figure Db. When applied to motion capture, it alleviates the need for template models while maintaining the advantages and accuracy of current generative approaches. A-NeRF extends Neural Radiance Fields (NeRF) [39 to work with single videos and articulated motion. NeRF parameterizes the scene implicitly as
(z), ((d))) + (a,c), c), with E R, E R23 q E
3 and d E IR33
(1)
by chaining F,, Multi- layer Perceptron (MLP), with r, the Positional Encoding (PE) [66]. First, the PE maps the input scene point q and view direction d to a higher dimensional space that enables the
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Initial pose
Input view
(a) A-NeRF body model for animation and vovel-view-synthesis
Refined pose
3D rec. (c) Body shape
(b) 3D pose refinement for real images
Figure 1: Our A-NeRF jointly learns neural body model of the and works with diverse body poses (left) while also refining the initial 3D articulated skeleton pose estimate from a single or, available, multiple views without tedious camera calibration (center). Underlying is a template- free neural representation (right) and skeleton-based embedding coupled with volume volumetric rendering. Real faces and their reconstructions are blurred in all figures for anonymity.
MLP to learn meaningful scene representation function Fs that subsequently outputs the radiance and opacity a at every point in space. Second, the implicitly described scene (via conditioning on query locations) rendered via classical ray-marching from computer graphics. The advantage of
the MLP representation is that avoids the complexity volumetric grids [29]. circumvents the
artifacts induced by the implicit bias of screen-space convolution 40 [44] and, unlike surface meshes, can have flexible topology. However, the original NeRF only works for static scenes captured from dozens of calibrated cameras such that each 3D point is seen from multiple views.
Our conceptual contribution lies learning a neural latent representation relative to an articulated skeleton. While explicit models such the popular SMPL body model (30) deform a surface via forwards kinematics, the implicit form of A-NeRF makes us re-think how skeletons can be integrated- implicit networks require the inverse transformation from 3D world coordinates to the reference skeleton, a significantly harder task that has not been fully explored. Our core technical novelty is to local coordinates relative to the articulated skeleton. Since a point 3D world coordinates cannot uniquely be associated with body part, resolve the mentioned ill-posed inverse problem by overparameterizing with one embedding per bone. This embeds domain knowledge of how humans move and provides common frame for the neural network to combine body shape and appearance constraints across the entire captured sequence (see Figure 2).
up with and evaluate different parameterizations of [(q), [(d) in Eq. to create
We demonstrate that all our contributions together enable learning of neural body model from monocular video, requiring only rough 3D pose estimates for initialization, that reaches a level of detail previously only attained with parametric surface models multi-view approaches [51]
Scope. We apply the model to motion capture, character animation, and appearance and motion transfer and demonstrate that the pose refinement improves existing monocular skeleton recon- struction. We learn in the transductive setting, for a specific target video that is known at training time but has ground truth. A-NeRF enables novel view synthesis of dynamic motions, with plausible however non-physical illumination. Additional steps are needed to enable relighting applications.
General impact. Building self-supervised approach for personalized human body modelling promises to be more inclusive to people and activities that are not well represented in supervised datasets. However, it bears the risk that 3D models of people are created without consent. We urge users to only use datasets collected for developing and validating motion capture algorithms.
2 Related Work
Our approach builds upon and is related to the following work on human pose and shape estimation, human modeling, and neural scene representations (62].

---

Discriminative Human Pose Estimation. [26] 32 34
While feed-forward estimation of the 3D joint positions
69 72 or joint angles and bone lengths [53] 73 of the skeleton
is highly accurate, such discriminative estimates are prone to misalignment when overlayed onto the input image due to the generalization gap. The skeleton pose can be refined better match the 2D pose estimates, but this usually leads to larger errors in 3D [36 37]. skeleton pose and combine it with a neural body model.
We use 23] for initializing
Surface-based Generative Body Models. These are obtained by either constraining template meshes via deformation energies [7] or learning parametric human body models from a large
collection of laser scans 66 30). Their low-dimensional parameters constrain the space of plausible
human shapes and motions. This enables real-time reconstructions from single images 7 [5].
detailed texturing and displacement mapping
3 [], and alleviates manual rigging 22] It also
enables optimization within the bounds of the learned prior [6 25] and weak supersuupevisiio when
integrated in a differentiable form [27] into neural training processes 20 23
42
45
65 Closest
to our approach in this category are the model fitting methods by [74] that textures and geometrically
refines an untextured parametric quadruped model to zebra images and to [88 that uses optical flow to refine human pose. Although in similar setting, our surface-free neural body model and volumetric rendering is fundamentally different to their textured triangle mesh that is rigged with forward kinematics and needs to be obtained a-priori.
Implicit Body Models. A-NeRF bears close similarities with body models defined implicitly in terms of level-sets (58) and density of sum of Gaussians [8] 48 49) that used for refining human pose, shape, and appearance via differentiable ray-tracing. However, sum of Gaussians and other primitives only provide rough approximations.
Neural Scene Representations.
Recent neural scene representations learn low-dimensional non- point clouds 1
linear representations of meshes 28
38 67], sphere sets (24). and dense volumetric grids 22 55 Their respective geometric output representations enable rendering with classical rendering techniques but have limited expressiveness, e.g., due to the fixed connectivity of surface mesh and large memory footprint of discretized volumes. This limitation is overcome by using unconstrained MLPs (39 paired with positional encoding (60] to characterize an arbitrary point in 3D space. Common surface definitions via level-sets of the MLP that are rendered with sphere tracing [56] and density representations rendered with ray-marching [39]. The rendering step required for maximum likelihood estimation, to define likelihood over observable variables- rcal images-while learning a 3D model. The rendering can be learned too (40 50 51 but usually leads to inconsistencies, particularly when training data is scarce. Some concurrent works also neural scene representations for refining camera motion [71], video reenactment [47), and facial models 1 [3]. Orthogonal these works, our A-NeRF learns an articulated body model from estimated poses and uncalibrated cameras.
633,
Closely related to ours is the NASA surface body model [10] that also defines implicit function as the minimum of individual implicit functions that are rigidly attached to the bones of a skeleton, each conditioned on the entire human pose to model dependencies and learned from 3D scans. By contrast, we Iearn a volumetric model instead of a surface model and include appearance and rendering. Even more similar is the recent NeuralBody [46] representation, which combines a NeRF with surface body model and underlying skeleton. By contrast to both approaches, we do not require surface supervision or initialization, condition pose differently, and refine pose, which enables us to learn from single videos in unconstrained environments.
3 Formulation
N Objective. Given a sequence [&] 1=1 of N images I E #xxx3 stemming from one or several videos of the same person, our goal is to simultaneously estimate the time-varying skeleton poses [0 and learn a detailed body model. Our A-NeRF body model C. is parametrized by neural network parameters ø that define volumetric shape and color while the skeleton captures motion over time. Figure gives overview of this generative model. It enables a rendering of the virtual body
3

---

Ray sampling
Oef albow) (0,,1)
Puse
Estimatio matistiation only)
onadds Irale knes?
cSy (0,
Comm....)
A
Cg (P
Gaye
T (0,,
(o,c)
Volumetric Rendering
NeRT (MLP)
Skeletoe-reaative Encodiing
Food Servard
AΦ
cuttallationss
Sumgle
Figure Overview, A-NeRF is generative model that can be rendered and optimized on photometric loss _SV (white). First, the skeleton pose is initialized with an off-the-shelf estimator (orange). Second, this pose is refined via a skeleton-relative embedding (blue) that, when fed to NeRF (green), drives the implicit body model that is rendered by ray- ray-marching (red). A key property of the skeleton-relatiio embedding is that single 3D query location maps to an overcomplete reparametrization, with the same point represented relative to each skeleton bone (right).
(c) w/o v,
(f) w/o v
(g) Ours
(a) NeRF
(b) Rotate 24
(c) Ours
(d) Rotate 24
Figure 3: Importance of our skeleton-relative encodings. The original NeRF breaks (a) when training on diverse set of poses and (b) further degrades when the poses are rotated. Even if (e) conditioned directly on 0g. the NeRF still suffers from artifacts due to the complexity and ambiguity of human articulation. With our skeleton-relative encoding (f, g), the geometry for the subject consistent under rotation, and the quality is greatly improved, with the full model working best.
model in unseen poses and optimizes its parameters 0 and á on the image reconstruction objective
8 012
SV (0, Σ (OC)(O)) Ie 1 + Nod(On dn) +
data term
pose regularizer
smoothness prior
with the influence of all three terms balanced by hyperparameters X; and Ag. The data term measures the distance between the images generated by C and the input image with the LI distance. The pose regularizer encourages the solution stay close to an initial pose estimate 9 obtained from an off- the-shelf predictor 233,, tolerating small shifts up to e = 0.01 with d(x) = min([1æ113 - c, c).). Lastly, the smoothness prior penalizes acceleration d' between poses of consecutive frames. Minimizing E can be seen as maximizing a corresponding probabilistic model, with the quadratic energy terms being the log-likelln likelihoodds of Gaussian distributions. Our focus is on formalizing the neural body model. For simplicity, we continue to write equations in terms of the objective functions used during inference with stochastic gradient descent.
3.1 NeRF and A-NeRF Image Formation Model
Instead of modeling the scene as collection of triangles or other primitives, we define the human implicitly by neural network as function (Eq. [ defined over all possible 3D points and view

---

directions in space 39 Similar to NeRF. we render the image of the human subject via ray marching
Calu, 03 020a),
Σ - exp(==;p:););. Ti = exp Σ و5
(3)
with (u, v) the 2D pixel location on the image. i the index to 3D query positions qi sampled along d and òg the distance to neighboring samples-a constant if samples would be taken at regular intervals. The T: is the accumulated transmittance for the ray traveling from the near plane q qi-the fraction of light reaching the sensor from sample point i. The C₁ is the light color emitted or reflected at i. The final pixel color is the sum over all Q samples, with the last sample taking the special role of the background. The background color is easily inferred as the median pixel color over the entire video for static camera setups. The ray direction d = K (u, v) is computed using the estimated camera intrinsics [23]. In the following, we introduce our skeleton parametrization 85, and how to use it to effectively model dynamic articulated human motion.
3.2 Articulated Skeleton Pose Model
Our skeleton representation encodes the connectivity and static bone lengths via a rest pose of 3D joint locations. Dynamics are modeled with per-frame skeleton poses 02, which define an affine transformation T(O,. m) for each bone m. Specifically, T(0x, m) maps 3D position Pk.m R in the mn-h local bone coordinates world coordinates q IR3 using homogeneous coordinates,
qq I
Pk.m
T(0s, m)
where subscript k.m. indicates that a variable is related to the m-th joint of image I,- Conversely, T(Ox, m) maps world local bone coordinates. Note that our skeleton is equivalent to SMPL [30 and others, but without their parametric surface model, and can therefore be initialized with any skeleton pose estimator. We include more details of our skeleton representation in the supplementary.
3.3 A-NeRF Eicteton-Relltivv Encoding
Our core contribution is to transform the query locations q and view direction relative to the skeleton before determining the color and opacity at that transformed point via NeRF. It is form of reparameterizatioon that explicitly incorporates domain knowledge of how the human body parts are linked and transformed relative to each other. Intuitively, implicit formulation turns explicit models, such as SMPL [30], on its head. Instead of deforming the output surface via skinning, the query location is mapped in the inverse direction to local bone-relative coordinate systems before processing through the NeRF network. Our final model uses combined encoding (dxx) as input to the NeRF Fs. Note that the skeleton embedding
eş [h(vv.)'' 11 (vv)), Fx, introduces the desired time dependency, denoted by subscript k. The inlet shows our most crucial contribution, the relative distance encoding Vi followed by PE with Cutoff to reduce the influence of irrelevant bones. We derive the components of our encoding O₁ and the other alternatives below.
Reference Pose Encoding One could compensate motion by attaching the query q world coordinates at frame l: to the closest bone and transforming it with
a = T(f,, m))T=((As m)q.
((k,,
(5)
This puts the query relative to the bone m as in frame & but with the skeleton in rest pose 6n- NeRF could then learn without change in the 3D space of the rest pose as done before for surfaces [61]. However, this cannot capture non-rigid pose-dependent effects, such as muscle bulging, and has ambiguities when q is at equal distance to two bones.
Bone-relative Position (Rel. Pos.) To remove these ambiguities and the ill ill-posed association to a single part, we map relative to each bone m with.
q [9 24.2 and q =T 1 m m)q.
(6)
5

---

The resulting individual bone coordinates are well suited to model the overwhelmingly rigid motion of the corresponding body part. Moreover. the overparameterization of position by concatenating all local encodings enables learning when complex interactions are necessary. However, such an embedding for all bones increases the dimensionality by an order of magnitude.
Relative Distance (Rel. Dist.) Much simpler compute distances from q all bones m,
V& [VK,1, ... Vx,24], with Vk,m = | 1l IR.
(7)
This radial encoding is used in our final model in favor of q because it naturally captures spherically shaped limbs, is lower-dimensional, and thereby improves reconstruction accuracy.
Relative Direction (Rel. Dir.) Since the distance encoding is invariant to direction, we addi- tionally obtain the direction vector to capture the orientation information of q.
F2 [ P6,224, fr.m
9k.m
E R
(8)
Note that by contrast to all other embeddings, this direction encoding did not profit from subsequent PE. We therefore pass it directly into ek.
Relative Ray Direction (Rel. Ray.) NeRF models the illumination effects in static 3D scene using the position and view direction. By contrast, our goal is to learn a body model that produces plausible colors with dynamic skeleton poses. Therefore, we transform d to obtain d, the outgoing ray direction relative each bone, similar to query position,
d₃ d dr
demm -T'' (0x, m)]3x3d R
with (0k, m) m)]axs the rotational part of the Mone-to-world transformation T- m). also optimize an appearance code for each image to handle dynamic light effects. The combination of d and the per- ver-image code enables A-NeRF to approximate the light effects in I plausibly. See the supplemental material for detailed discussions on modeling view-dependent effects in our setting.
Following concurrent works [3 46
Cutoff. We desire a local embedding where points should not be influenced by all but only nearby bones. To this end, we introduce a windowed version of positional encoding by multiplying the encoding with respect to bone m by (.(v,.m) = 11 ((T(Vk.m 1)). with S the sigmoid step function, the cutoff point, the sharpness. This leaves queries unaffected by distant bones.
Our embedding choice of e [h(vv))((vi)) Tk, h(v (d+ has the advantage of being invariant to the global shift and rotation of the person and preserves the piece- wiieewwieeae rigidity of articulated motion while still allowing for pose-dependent deformation (see Figure 3). In addition to Ck. also consider other embedding variants. See the supplementary and Section for a detailed discussion.
Evaluation
We performed experiments to validate that A-NeRF learns accurate body models and poses, with fewer assumptions (single view, uncalibrated, and w/o a parametric surface model) than the related works. This makes it applicable to fine- ane-ggained pose refinement that improves the estimates of state- of-the-art methods. The supplements provide the implementation details, additional comparisons and ablation studies.
Inference and Implementation Details Our A-NeRF model is learned without supervision on a single multiple videos of the same person. Camera intrinsies, bone lengths for setting am, and pose 02 are initialized with 23 for every frame k. These poses are then optimized on objective Eq alongside the generative A-NeRF model. See supplementary for more details.
Datasets. We evaluate on the following benchmarks, and additionally on synthetic data created from SURREAL and Mixamo characters, which are listed in the supplement.
Human 3.6M [9 The dataset consists of 5 training and 2 testing subjects ($9/S11) with ground truth 3D joint locations. We follow two widely adopted test protocols denoted as
Protocol 20 22 23 and Protocol II 41 59], which we evaluate 14/17-joint estimation error every tth t6 frame of the test videos, respectively. See the supplement for details.

---

This dataset is a standard benchmark for human pose estimation. It consists of 4 indoor and 2 outdoor subjects with challenging human poses. The number of frames per subject range from 276 to 603.
MPI-INF-3DHP (35
MonoPerfCap [70] The dataset consists of human performance video captured with a monocular camera in both indoor and outdoor settings. We two subjects, Weipeng outdoor and Nadia outdoor, for our qualitative experiments. The two subjects have 1151 and 1635 frames, respectively, for training.
Pose Metries. We report the PA-MPJPE metric, the Euclidean distance between Procrustes-aligned (PA) predictions and ground truth 3D joint position averaged over all frames and joints of the test set. The PA alignment scale and orientation is essential for comparing approaches that not assume knowledge of the ground truth calibration and are, hence, ill-posed to the factors that the alignment removes. Following prior work [23] 22 35), we also report percentange of correct keypoints (PCK) for MPI-INF-3DHP, the percentage of joints that lie within distance of 150mm the ground truth.
Visual Metrics. We quantify the visual quality on MonoPerfCap and Human 3.6M datasets by training on subset and testing held-out test set of the same character. Image quality is quantified via the PSNR and SSIM of the rendering compared with the reference image within the character bounding boxes. Because no ground truth pose is available in the required skeleton format these datasets, we train our model once on the entire dataset to get reliable skeleton pose estimates as pseudo ground truth, and second time with part withheld to learn the body model for visual quality evaluation. Since MonoPeríCap has one sequence per actor, we exclude the last 20% of each video. For Human 3.6M exclude entire actions, namely Geeting- 1,2 Posing- 1,2 and Walking 1,2. The body model then transferred to the held-out portion by using the pseudo ground truth poses as the driving motion. Thereby, viewpoints, irrespective of the underlying skeleton model provided the dataset.
still test the generalization of different models to new poses and
Reference
Reference
Novel view (-+ Mixamo
MonoPeríCap
"James"
NeuralBody
Ours w/o rf.
Ours
NeuralBody
Ours w/o rf.
Ours
Novel view synthesis from all models. All models are trained with SPIN 23 estimated human pose and camera parameters. A-NeRF renderings (ours) align better with the reference images (top row). and the rendered novel views (2nd and 3rd rows) show better details (limbs, facial features).
Figure 4:
Sevel-View-Syaee View-Synthhesi and Character Animation. Our body model is generative, which allows us to train on single or multiple uncalibrated videas and alter viewpoint and human pose. Figure 4 shows renderings of the same persons from a new camera angle; novel-view-synthesi Likewise. Figure [5] demonstrates character animation, where the view is fixed, and the underlying skeleton re-posed by manually changing joint angles or by transferring
motion between characters. A-NeRF is the first model that learns detailed human body model with such capabilities without needing a 3D surface or multi-view supervision. While NeuralBody [46] can also learn photo-realistic human models from monocular images, their model anchors its representation on the SMPL 3D

---

Source
NeuralBody
Ours
Source
NeuralBody
Ours
Source
NeuralBody
Ours
Figure Motion retargeting and animation. Top rows: Pose transfer, with the source pose reconstructed by A-NeRF and rendered with different target body models. Last row: Animating the A-NeRF model while keeping the lower or upper part of the body fixed.
Table 1: Quantitative evaluation on Human3.6M [19] and MPI-INF-3DHP B5 Our test-time pose refinement improves consistently upon the SPIN baseline, with largest improvements for extremities (PA-Wrist).
Human 3.6M
MPI-INF-3DHP
Protocol I
Protocol II Prot. II Wrist Prot. Il Multi-viww
Method
PA-MPJPEJ PA-MPIPEJ PA-Wrist
PA-MPIPEJ
PA-MPJPEJ
PCK
MotioNct 53
54.6
HoloPose
46.5
VIBE 22
41.4
n/a
n/a
n/a
64.6
89.3
SPIN 23
41.1
67.5
76.4
Baseline (SPIN github 23)
42.7"
41.9*
66.5*
34.0"
68.2"
79.3*
SPIN-SMPLify 23
57.7
59.2
100.9
Ours smoothness prior)
39.4
39.6
28.0
66.9
Ours
39.3
n/a
n/a
n/a
66.8
80.4
Reevaluation of publicly-available model because missing evaluation protocols reproducible. We refine the SPIN estimated pose using SMPLify. We adopt the implementation from the SPIN repository.
single-view setup with the noisy estimated poses Ô as input, NeuralBody suffers from artifacts and less detail when training, as it assumes 3D ground truth, which only available in controlled conditions. The importance of our joint body and pose optimization is further validated by the ablation with refinement disabled (Ours w/o rf.), which similarly produces ghosting artifacts around extremities.
surface. When trained
Human Pose Estimation. Training A-NeRF includes a form of test-time optimization (see Figure
right), only the initialization from 223 trained supervised on the Human3.66 training set. Table I shows that on Human 3.6M, A-NeRF reaches comparable results with other single-view approaches, and achieves 8.0% improvement PA-MPJPE upon the baseline used for pose initialization on Protocol (42.7 - 39.3) and 5.5% on Protocol II (41.9 - 39.6). Note that these are average numbers across all joints, including easy-to-predict hip, shoulder, and head joints. Our largest gains are on the extremities, e.g., with an improvement of 14% (9.2 mm) for the wrist joint on Protocol (PA-Writ (Pr-Wtist). We also compare to applying SMPLify 7 a method that refines 3D poses using 2D joint locations (estimated with [8] as constraints, at test time. It tends to explain the 2D joint estimates perfectly but degrades the pose. The final pose estimations become less accurate than the initial ones. In contrast, A-NeRF optimizes the poses by implicitly minimizing the disagreement

---

Table 2:
Visual quality evaluation on the Human3.6M 19 and MonoPerfCap [70] held-out
sets. Our full A-NeRF model significantly improves the visual quality. The body model itself attains a higher quality than NeuralBody (2nd vs. 3rd row), and additional detail gained with the proposed pose refinement (last row).
MonoPeriCap
Human 3.6M
PSNR
SSIM f PSNR
SSIM
NeuralBody, driving motion from 23
21.80
0.8476
22.08
0.8766
NeuralBody, driving motion from A-NeRF refinement
21.75
0.8468
22.55
0.8782
A-NeRF w/o pose refinement
21.99
0.8405
23.33
0.8776
A-NeRF (Our full model)
24.39
0.8851
27.45
0.9277
Ref. Image
Geometry
Novel views
Figure 6: Geometry. The isosurface of our density model rendered from multiple unseen views.
A-NeRF learns plausible geometry without using explicit surface templates and accurate initial poses.
among the 3D body representation in different images, and thus achieves better performance. On MPI-INF-3DHP, T the results averaged over the test subjects from MPI-INF-DDHP. Despite having a low number of frames and human poses available for learning the skeleton-relative encoding, A-NeRF still provides moderate improvements over the baseline estimations.
Video-based volumetric reconstruction. Figure 6 visualizes the learned density using Marching Cubes [31], with voxel grid resolution of 256 and density threshold 10. Despite only learned from monocular videos (no stereo, depth camera, or multi-view constraints) and without using a pre- defined template model, A-NeRF reconstructs detailed volumetric body with details that could be captured by offsets to parametric surface model, such as the head phones and basebal cap in the last row of Figure Note that no geometric smoothness term enforeed.
Multi-view extension. A-NeRF can also leverage multi-view refinement (Table I Protocol II Multi-view), even without access to ground truth camera calibration (see supplementary).
Visual Quality Comparison. We report the results in Table 2 We compare to NeuralBody with both the initial estimates from 23 and our refined pose since NeuralBody has refinement step. Compared to both variants, our A-NeRF shows significantly better reconstruction performance on held-out poses. We observe that NeuralBody models can retain the facial features and hands as they anchor their representation on a 3D surface model. However, the rendered limbs and faces are blurry and distorted. Results are similar to training A-NeRF without refinement (see Figure 7). As estimation for these joints is often inaccurate and noisy, the models without pose refinement simply learn to predict mean pixel values. To conclude, it is important train with pose refinement, with which A- A-NeRF suffers less from artifacts with overall better visual quality.

---

Ground Truth Photometric emrrr (-)
Ground Truth
Photometric error (+)
NeuralBody w/ pose from [23
NeuralBody w/ pose from A-NeRF
A-NeRF w/o rf.
A-NeRF
Figure 7: A-NeRF with pose refinement generalizes better to test sequences. We visualize both the rendered images, well the photometric error (squared distance, normalized [0, 1J) between the rendered and the ground truth images (warmer color indicates higher error) from our MonoPerfCap (1-2th rows) and Human 3.6M (3-4th rows) held-out sets. NeuralBody models produce artifacts around body contours. A-NeRF w/o refinement cannot reproduce facial features and limbs. With pose refinement, A-NeRF can produce both appearances and shapes more plausibly.
Ablation study. Our detailed ablation studies are reported in the supplemental document. In sum- mary, they reveal: i) Embedding relative 3D position, qR, instead of our proposed radial embedding yields only half as good pose refinements. ii) Our embedding choices keep the dimensionality moderate while improving on or matching the PSNR and SSIM of higher-dimensional variants. iii) For fixed number of images with accurate poses, learning from a long video with diverse poses has visual quality comparable to learning from multiple shorter clips.
Limitations and Failure Cases, Our computation time is the biggest bottle- neck in extending A-NeRF long sequences and multiple actors. Although single static camera suffices, A-NeRF requires to see the person from all sides in varying poses to learn pose dependencies from data. The inlet shows rendering of an extreme breakdance pose retargeted to model trained on normal walking motions. Hence, the source pose is unseen during training and far from the data distribution, which leads to artifacts.
Reference
Failure
Conclusion
We propose a new way for integrating articulated skeleton models and implicit functions via an overcomplete ce-parametrization It includes learning an interpretable 3D representation from 2D images; personalized volumetric density field with texture detail and time-varying poses the actor depicted in the input. The underlying ill pol-p problem of mapping single query location to multiple parts addressed with an overparametrizztion over nearby parts. To the best of our knowledge, A- NeRF is the first approach to define NeRF models for extreme and articulated motion unconstrained video and this new approach scores high on Human 3.6M benchmark. Importantly, it works from single video and naturally extends to multi-view and does not require camera calibration in either scenario. This an important step towards making motion capture more accurate and practical. In future work, we will learn a general human model from database of subjects instead of individuals.
10

---

Funding in direct support of this work: NSERC Discovery grant, UBC Advanced Research Computing (ARC) GPU cluster, Compute Canada GPU servers, and gift by Facebook Reality Labs.
References
[1] K.-A. Aliev, A. Sevastopolsky, M. Kolos, D. Ulyanov, and Lempitsky. Neural point- based graphics. arXiv preprint arXiv:1906.08240, 2019.
[2] Alldieck, M. Magnor, W. Xu, C. Theobalt, and G. Pons-moll. Video Based Reconstruction of 3D People Models. In CVPR, 2018.
[3] Alldicck, M. Magnor, W. Xu, C. Theobalt, and G. Pons-Moll. Detailed human avatars from monocular video. In 3DV, pages 98-109. IEEE, 2018.
[4] Alldieck, M. Magnor, L. Bhatnagar, C. Theobalt, and G. Pons-Moll. Learning to reconstruct people in clothing from single rgb camera. In CVPR, pages 1175-1186, 2019.
[5] Alldieck, G. Pons-Moll, C. Theobalt, and M. Magnor. Tex2shape: Detailed full human body geometry from single image. In ICCV, 2019.
[6] A. O. Balan, L. Sigal, M. J. Black, J. E. Davis, and H. W. Haussecker. Detailed human shape and pose from images. In CVPR, pages 1-8, 2007.
[7] Bogo, Kanazawa, C. Lassner, Gehler, J. Romero, M. J. Black, Keep smpl: Automatic estimation of 3d human pose and shape from single image. In ECCV, pages 561-578. Springer, 2016.
[8] Z. Cao, G. H. Martinez, T. Simon, S.-E. Wei, and Y. A. Sheikh. Openpose: Realtime multi- person 2d pose estimation using part affinity fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.
[9] V. Choutas, G. Pavlakos, T. Bolkart, D. Tzionas, and M. Black. Monocular expressive body regression through body-driven attention. In ECCV, 2020. URL https://eeppsse is tue mpg.de
[10] B. Deng, J. Lewis, T. Jeruzalski, G. Pons-Moll, G. Hinton, M. Norouzi, and A. Tagliasacchi. Nasa: neural articulated shape approximation. arXiv preprint arXiv:1912.032000 2019.
[11] J. Dong, Q. Shuai, Y. Zhang, X. Liu, X. Zhou, and H. Bao. Motion capture from internet videos.
In ECCV, pages 210-227. Springer, 2020.
[12] G. Gafni, J. Thies, M. Zollhöfer, and M. Nielner. Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. CVPR, 2021.
[13] C. Gao, A. Saraf, J. Kopf, and J.-B. Huang. Dynamic view synthesis from dynamic monocular vidco. In ICCV, 2021.
[14] I. Goodfellow, J. Pouget-abadie, M. Mirza, B. Xu, D. Warde-farley, S. Ozair, A. Courville, and Bengio. Generative Adversarial Nets. In NeurIPS, pages 2672-2680, 2014.
[15] P. Guan, A. Weiss, A. O. Bälan, and M. J. Black. Estimating human shape and pose from single image. In CVPR, pages 1381-1388, Sep. 2009. doi: 10.1109/CCV.2009544555
[16] R. A. Guler and I. Kokkinos. Holopose: Holistic 3d human reconstruction in-the-wild. In CVPR, pages 10884-10894, 2019.
[17] M. Habermann, W. Xu, M. Zollhoefer, G. Pons-Moll, and C. Theobalt. Livecap: Real-time human performance capture from monocular video. ACM TOG (Proc. SIGGRAPH), 2019.
[18] Z. Huang, Y. Xu, C. Lassner, H. Li, and T. Tung. ARCH: animatable reconstruction of clothed humans. In CVPR, 2020. doi: 10.1 0WW09//CVPP42220000000000000 URI. hhtpps //doi org 100. 1109/ 1 Cwtp42600 2020.

---

[19] C. lonescu, D. Papava, V. Olaru, and C. Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. PAMI, 36(7):1325-133) 13399 2014.
[20] A. Kanazawa, M. J. Black, D. W. Jacobs, and J. Malik. End-to-end recovery of human shape and pose. In CVPR, pages 7122-7131, 2018.
[21] Karras, S. Laine, and Aila. style-based generator architecture for generative adversarial networks. In CVPR, 2019.
[22] M. Kocabas, N. Athanasiou, and M. J. Black. Vibe: Video inference for human body pose and shape estimation. In CVPR, 2020.
[23] N. Kolotouros, G. Pavlakos, M. Black, and K. Daniilidis. Learning reconstruct 3d human pose and shape via model-fitting in the loop. In ICCV, 2019.
[24] C. Lassner and M. Zollhöfer. Pulsar: Efficient sphere-based neural rendering. In CVPR, 2021.
[25] C. Lassner, J. Romero, M. Kiefel, F. Bogo, M. J. Black, and P. V. Gehler. Unite the people: Closing the loop between 3d and 2d human representations. In CVPR, July 2017.
[26] Li, Ke, K. Pratama, Y.-W. Tai, C.-K. Tang, and K.-T. Cheng. Cascaded deep monocular 3d human pose estimation with evolutionary training data. In CVPR, June 2020.
[27] S. Liu, T. Li, W. Chen, and H. Li. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. In ICCV, pages 7708-7717, 2019.
[28] LLobardi, J. Saragih, Simon, and Sheikh. Deep appearance models for face rendering. ACM TOG (Proc. SIGGRAPH), 37(4). July 2018. ISSN 0730-0301. doi: 10.1145/3197517. 3201401. URL https: org/ 10. 1145/3197511 3201401
[29] Lombardi, T. Simon, J. Saragih, G. Schwartz, A. Lehrmann, and Y. Sheikh. Neural volumes: Learning dynamic renderable volumes from images. ACM TOG (Proc. SIGGRAPH), 38(4), July 2019. ISSN 0730-0301. doi: 10. 111114343366.333300000 1145/3306344 3323020
bttps///:///oo org::////1..r///0
[30] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black. Smpl: A skinned multi- person linear model. ACM TOG (Proc. SIGGRAPH), 34(6):1-16, 2015.
[31] W. E. Lorensen and H. Cline. Marching cubes: A high resolution surface construction algorithm. ACM SIGGRAPH, 1987.
[32] C. Luo, X. Chu, and A. L. Yuille. Orinet: A fully convolutional network for 3d human pose estimation. In BMVC, page 92, 2018. URL http:://mmnc220e..rrg//oeeom wwwp:///00....orr//oeeeeeeeeeeee pttpoottttttttteeeeeeeeee 0289. pdf
[33] R. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In CVPR. 2021.
N. Radwan, M. S. M. Sajjadi, J. T. Barron, A. Dosovitskiy, and D. Duckworth.
[34] J. Martinez, R. Hossain, J. Romero, and J. Little. A simple yet effective baseline for 3d human pose estimation. In ICCV, 2017.
[35] D. Mehta, H. Rhodin, D. Casas, Fua, Sotnychenko, W. Xu, and Theobalt. Monocular 3d human pose estimation in the wild using improved cnn supervision. In 3DV. IEEE, 2017. doi: 10.1109/3dv.207..00 7.00644. URL http:///t http:::///wmmmpiintttt.. mpg. deedddeeddddaaeeeeeee dataset.
[36] D. Mehta, Sridhar, O. Sotnychenko, H. Rhodin, M. Shafiei, H.-P. Seidel, W. Xu, D. Casas, and C. Theobalt. Vnect: Real-time 3d human pose estimation with a single rgb camera. In TOG, volume 36, July 2017. doi: 10. 0111445377995930733333 mpg pog..de/projjcts//Nect///
URIhtttp httttp://gv..pp. wwwpwwwwp.i..............w... inf
[37] D. Mehta, O. Sotnychenko, F. Mueller, W. Xu, M. Elgharib, P. Fua, H.-P. Seidel, H. Rhodin, G. Pons-Moll, and C. Theobalt. XNect: Real-time multi-person 3D motion capture with single RGB camera. In ACM TOG (Proc. SIGGRAPH), 2020.
12

---

[38] M. Meshry, D. B. Goldman, S. Khamis, H. Hoppe, R. Pandey, N. Snavely, and R. Martin-Brualte Neural rerendering in the wild. In CVPR, pages 6778-6887,27 2019.
[39] B. Mildenhall, P. Srinivasan, M. Tancik, T. Barron, Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[40] T. Sguyen-Phuoc, C. Li, L. Theis, C. Richardt, and Y.-L. Yang. Hologan: Unsupervised learning of 3d representations from natural images. ICCV, pages 7588-7597, 2019.
[41] A. Nibali, Z. He. Morgan, and L. Prendergast. 3d human pose estimation with 2d marginal heatmaps. In WACV, 2019.
[42] M. Omran, C. Lassner, G. Pons-Moll, P. Gehler, and B. Schiele. Neural body fitting: Unifying deep learning and model based human pose and shape estimation. In 3DV, 2018
[43] S. Park, Hwang, and N. Kwak. 3d human pose estimation using convolutional neural networks with 2d pose information. In European Conference on Computer Vision (ECCV). pages 156- 169, 2016. doi: 10.1007/978-3-319-4490 _1. URL [tL/sttps:///oo 10 3-
[44] G. Pavlakos, X. Zhou, K. G. Derpanis, and K. Daniilidis. Coarse-to-fine volumetric prediction for single-image 3d human pose. Conference on Computer Vision and Pattern Recognition (CVPR), pages 1263-127777 1272, 2017.
[45] G. Pavlakos, L. Zhu, X. Zhou, and K. Daniilidis. Learning to estimate 3D human pose and shape from single color image. In CVPR, 2018.
[46] S. Peng, Y. Zhang, Xu, Q. Wang. Q. Shuai, H. Bao, and X. Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In CVPR, 2021.
[47] A. Pumarola, E. Corona, Pons-Moll, and Moreno-Noguer. D-NeRF: Neural Radiance Fields for Dynamic Scenes. In CVPR, 2020.
[48] H. Rhodin, N. Robertini, C. Richardt, H.-P. Seidel, and C. Theobalt. A versatile scene model with differentiable visibility applied generative pose estimation. In ICCV, pages 765-773, 2015.
[49] H. Rhodin, N. Robertini, D. Casas, C. Richardt, H.-P. Seidel, and C. Theobalt. General automatic human shape and motion capture using volumetric contour cues. In ECCV, pages 509-526, 2016.
[50] H. Rhodin, M. Salzmann, and P. Fua. Unsupervised Geometry-Aware Representation for 3D Human Pose Estimation. In ECCV, 2018
[51] H. Rhodin, Constantin, Katircioglu, M. Salzmann, and P. Fua. Neural scene decomposition for multi-person motion capture. In CVPR, 2019.
[52] G. Rogez, P. Weinzaepfel, and C. Schmid. I.RRNet: Cecaliaatioo-Clsssffccttooonnnp for Human Pose. In CVPR, pages 1216-1224, Honolulu, United States, July 2017. IEEE. doi: 10. 10110//RP.2000.11444
htRphhtps:://aa //haal:://ba.... frr.ria.ffraaa-1555500000
[53] M. Shi, K. Aberman, A. Aristidou, T. Komura, D. Lischinski, D. Cohen-Or, and B. Chen. Motionet: 3d human motion reconstruction from monocular video with skeleton consistency. ACM TOG (Proc. SIGGRAPH), 40(1):1-15, 2020.
[54] A. Shysheya, E. Zakharov, K.-A. Aliev, R. Bashirov, E. Burkov, K. Iskakov, A. Ivakhnenko, Malkov, Pasechnik, D. Ulyanov, et al. Textured neural avatars. In CVPR, pages 2387-2397, 2019.
[55] V. Sitzmann, J. Thies, F. Heide, M. Nießner, G. Wetzstein, and M. Zollhöfer. Deepvoxels: Learning persistent 3d feature embeddings. In CVPR, 2019.
[56] V. Sitzmann, M. Zollhöfer, and G. Wetzstein. Scene representation networks: Continuous ad-structure-wware neural scene representations, In NeurIPS, pages 1121-1132, 2019.
13

---

[57] V. Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein. Implicit neural representations with periodic activation functions. NeurIPS, 33, 2020.
[58] C. Stoll, N. Hasler, J.
Gall, H.-P. Seidel, and C. Theobalt, Fast articulated motion tracking using a sums of Gaussians body model. In ICCV, pages 951-958, 2011.
[59] X. Sun, J. Shang, S. Liang, and Y. Wei. Compositional human pose regression. In ICCV, pages 2602-200 2611, 1, 2017.
[60] M. Tancik, P. P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ra- mamoorthi, Barron, and Ng. Fourier features let networks learn high frequency functions in low dimensional domains, NeurIPS, 2020.
[61] J. Taylor, J. Shotton, T. Sharp, and A. Fitzgibbon. The vitruvian manifold: Inferring dense correspondences for one-shot human pose estimation. In CVPR, pages 103-110. IEEF, 2012.
[62] A. Tewari, O. Fried, Thies, V. Sitzmann, S. Lombardi, K. Sunkavalli, R. Martin-Brualla, T. Simon, J. Saragih, M. Nießner, R. Pandey, S. Fanello, G. Wetzstein, J.-Y. Zhu, C. Theobalt, M. Agrawala, E. Shechtman, D. B. Goldman, and M. Zollhöfer. State of the Art on Neural Rendering. Computer Graphics Forum (EG STAR 2020), 2020.
[63] Thies, M. Zollhöfer, and M. Nießner. Deferred neural rendering: Image synthesis using neural textures. ACM TOG (Proc. SIGGRAPHI) 2019.
[64] D. Tome, C. Russell, and L. Agapito. Lifting from the deep: Convolutional 3d pose estimation from a single image. In CVPR, July 2017.
[65] H.-Y. Tung, H.-W. Tung, Yumer, and K. Fragkiadaki. Self-supervised learning of motion capture. In NeurlPS, pages 5242-5252, 2017.
[66] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention all you need. NeurIPS, 2017.
[67] O. Wiles, G. Gkioxari, R. Szeliski, and J. Johnson. Synsin: End-to-end view synthesis from single image. In CVPR, pages 7467-7477, 2020.
[68] D. Xiang, H. Joo, and Sheikh. Monocular total capture: Posing face, body, and hands in the wild. In CVPR, pages 10965-10974. 2019.
[69] J. Xu, Z. Yu, B. Ni, Yang, X. Yang, and W. Zhang. Deep kinematics analysis for monocular 3d human pose estimation. CVPR, June 2020.
[70] W. Xu, A. Chatterjee, M. Zollhöfer, H. Rhodin, D. Mchta, H.-P. Scidel, and C. Theobalt. Monoperfcap: Human performance capture from monocular video. TOG, 37,2):27,201
[71] L. Yen-Chen, P. Florence, J. T. Barron, A. Rodriguez, P. Isola, and T.-Y. Lin. inerf: Inverting neural radiance fields for pose estimation. In IROS, 2020.
[72] K. Zhou, X. Han, N. Jiang, K. Jia, and J. Lu. Hemlets pose: Learning part-centric heatmap triplets for accurate 3d human pose estimation. In ICCV, pages 3344-2353, 2019.
[73] X. Zhou, X. Sun, W. Zhang, S. Liang, and Y. Wei. Deep kinematic pose regression. In ECCV Workshop, 2016.
[74] S. Zuffi, A. Kanazawa, T. Berger- Wolf, and M. J. Black. Three-d safari: Learning estimate zebra pose, shape, and texture from images "in the wild". In ICCV, pages 5358-5367. IEEE, Oct. 2019.
14


Reviews:
[{"review_id": "ncjMeM8vNMd", "reviewer": "Reviewer_Zhik", "summary": "This submission presents a way for adding articulation to neural radiance fields. The main contribution is an encoding of the query points and view directions relative to a skeleton. The proposed formulation is trained per subject on a video sequence and optimizes for the 3D skeleton configuration in each frame and a global MLP which represents the NeRF. This defines the motion tracking throughout the input sequence and the recovered model can be reposed by rendering frames with new joint locations.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Adding an articulation model to a NeRF seems a novel and significant contribution\n- Novel poses can be rendered photorealistically.\n\nWeaknesses:\n- The formulation seems very expensive to get the motion tracking of a single subject 60 hours of training on 2 high end GPUs is needed.\n- It is unclear if the model can work with very short sequences. How long are the videos and how much pose variation in the videos is needed to avoid overfitting of the NeRF? My concern is that this might behave similar to when a vanilla NeRF is trained on sparse views and just overfit to the small number of poses.\n- To me it is not clear how with the skeleton relative encoding the NeRF is still able to reason about view dependent lighting effects given the NeRF does not have access to the skeleton configuration. How is the network still able to recover the relative configuration of the lighting environment, view direction, surface position and orientation in the original world coordinate system. It seems without this it would be hard for the network to recover view dependent lighting effects.\n- Similar to the comment above it is not clear to me how the network could understand how the lighting behaves under camera motion, given everything is expressed relative to the camera. From the pictures in the submission it is hard to tell if view dependent lighting effects are recovered. Example renderings that show how the appearance changes if the view direction input to the network is changed but the actual rendered view direction stays constant would be helpful to better understand the capabilities of the proposed network.\n\n\nSummary:\nMay main concern with this paper is that the effect of the skeleton relative encoding on the view dependent effects is not discussed. Given that the view dependent effects is one of the key properties of NeRF I find it important that this would be discussed and evaluated better. At this point it is not clear to me if the proposed formulation is even still able to generate view dependent effects or if this part of the formulation could even be removed. \n\n\nAfter Author Reply:\nIn the author's replies it turns out that the problem of modeling illumination more accurately is non-straight forward and the model proposed in this paper even though in my opinion slightly unintuitive works better than obvious alternatives. I don't think these issues need to be resolved for publishing this work. In my opinion the original submission is lacking a discussion of this limitation but I feel confident that this can be incorporated in the final version and left for future work. I adjusted my rating and time spent on the review accordingly. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think potential risks of creating models without consent of people is addressed. One point that could be emphasized more is that this method allows to render people in unseen poses which adds the risk of generating fakes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3h"}, "extra_text": {}}, {"review_id": "ItSU90msyCY", "reviewer": "Reviewer_EsYF", "summary": "This paper aims to refine the 3D human poses. To this end, the authors equip neural radiance fields with human skeletons, where they design a skeleton-relative embedding based on the human skeleton. By training the A-NeRF on the video, the human poses are optimized to fit the observations.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n\n1. This paper proposes a novel skeleton-relative embedding, which enables the neural radiance field to represent dynamic humans.\n2. A-NeRF allows the human pose refinement during training, which improves the quality of rendering and estimated poses.\n\nQuality:\n\n1. Although the results in the submission look good, the proposed skeleton-relative embedding does not make sense to me. Why does relative distance work better than the bone-relative position. I think that reducing a position to a distance may cause ambiguity, since points on a spherical surface have the same distance.\n2. For experiments, the results of NeuralBody always have black blobs, which seems strange. Is the NeuralBody trained with black background and visualized with the white background?\n3. The training and test data is unclear. What are the training views and test views?\n4. The motivation of this submission is unclear. At the first, I thought that it aims to refine the human poses with differentiable rendering. However, the experimental results are mostly about the view synthesis and animation. So I thought that it aims for view synthesis. However, the quantitative comparisons are only presented in the supplementary material, and the authors only compare their method with NeuralBody.\n\nClarity:\n\n1. This submission is poorly written. Although I can understand the overall idea, it is still difficult for me to learn about the technical details. The authors could leave some unrelated contents in the supplementary material and describe their core method firstly.\n2. For example, the readers probably cares about how to combine NeRF with human skeletons, so after describing the basic NeRF, the authors could introduce how to improve the input of NeRF based on 3D skeletons. The position encoding (line 163), 4D space-time encoding (line 169), and conditioning on pose (line 172) are unrelated with the skeleton-relative embedding and should be put in other places.\n3. Similarly, the NeRF and articulated skeleton pose model are well-known techniques, which can be described more simply and can be put in the supplementary material.\n4. The quantitative comparisons on view synthesis and the ablation studies are important, which should be put in the experiment section, while not in the supplementary material.\n\nSignificance:\n\n1. This paper develops a human representation for view synthesis and animation from sparse camera views. It probably enlighten follow-up works in the field of human view synthesis.\n\nTypos:\n\n1. Line 201: \"Much simpler to compute are distances from q to all bones m\".", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not see potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "ETsXX8uQiv", "reviewer": "Reviewer_gmac", "summary": "This paper presents modelling articulated neural radiance fields for human dynamics from monocular videos. To model NeRF for dynamic human, it encodes 3d spatial points and ray directions w.r.t the articulated skeleton poses, which are fine-tuned together when training the NeRF field. Various skeleton-based feature encoding are experimented in consideration of both performance and memory. ", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "There are 2 major contributions from the work. It first extends the static NeRF modeling to dynamic human motion by encoding the spatial points w.r.t. the bone transformations. Secondly, it utilizes the differential NeRF rendering to fine tune the estimated human pose that better aligns with image pixels. i am not fully convinced with either of them and have quite a few concerns (see detailed comments below). \n\n- I do not find any sentences in the paper describing how the background is removed from the NeRF rendering. A foreground mask is needed for training NeuralBody and how do you obtain good masks for the evaluation datasets? Could the dark noise around the person come from imperfect segmentation? Figure A1 shows both your results and NeuralBody with a background and how do you obtain them? Related to this background question, is your image metrics (SSIM and PSNR) evaluated over the whole image or only for the region of interest (around the target human)? All of these important details are missing in the paper.\n\n- The paper provides comparisons to NeuralBody and with ablation studies, demonstrates it is helpful to fine tune the estimated skeleton poses. I agree for in-the-wild videos, fine-tuning the poses seem a good idea. However,  it does not demonstrate the superiority of the proposed skeleton-based feature encoding over other approaches, such as anchoring over statistical human \nsurface as in NeuralBody. It seems to me that NeuralBody can also be easily extended to fine tune the skeleton but i realize it might be an unfair comparison to this submission. However, I would like to see both methods compared on synthetic dataset or dataset with ground-truth pose estimations for a better understanding on different feature encoding approaches.  \n\n- Pose estimation:  how important the regularization term is? The pose estimation error could theoretically be compensated by the NeRF field (overfitting). How do you ensure they are decoupled well? Besides, such a pose estimation procedure with NeRF training is very expensive and it does not seem a very fair comparison by comparing to feed-forward approaches (such as SPIN which the method is initialized from). I would suggest to also compare with some nonlinear optimization based methods, such as Simplify or Zanfir et al ECCV 2020, followed with SPIN initialization. \n\n- Qualitative results:  it will be good to visualize the 3d geometry reconstructions from the trained NeRF field. The appearance looks very noisy for the subject in fig 1 and 3 (with clothing color diluted into the skin).  Any reasons?  \n\n- I think there could be a fundamental problem by transforming spatial points w.r.t. the skeleton as NeRF feature encodings. Suppose there is a constant lighting source in the scene and now how the subject is located w.r.t. this lighting source should largely affects its appearance. However, the proposed encoding scheme will result in global transformation invariant features and will the ANeRF produce right lighting effects, say the subject is rigidly translating in the scene?\n\n- Table A7: for this monocular and multiview comparison, it is not clear from which camera views you evaluated the approach?  \n\n- Some missing related works:\n   * D-NeRF: Neural Radiance Fields for Dynamic Scenes\n   * Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Seems ok. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "DGs4xtsgtFp", "reviewer": "Reviewer_2h5q", "summary": "The paper presents A-Nerf which is a generative neural body model capable of rendering a human under novel viewpoints and poses. A-Nerf is an extension of Nerf to capture dynamic human bodies. It's achieved by conditioning Nerf on the skeletal body pose information. The paper demonstrates that naively conditioning Nerf on the body pose information does not lead to optimal performance and therefore proposes a set of relative encodings of the pose information that results in significantly better renderings without the need of a full-body mesh model. The experiments are performed for motion capture as well as novel view synthesis where the proposed method demonstrates superior performance as compared to existing methods. ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros:**\n\n+ The paper is well written and easy to read. \n+ The proposed pose relative embeddings are novels and intuitive. I like the idea of windowed positional encodings to reduce the influence of parts that are far away from the sampled point. \n+ The supplementary material provides an extensive ablation study of the proposed approach. \n+ The proposed method provides better renderings as compared to the state-of-the-art method NeuralBody even though it doesn't require surface mesh. \n+ I like the narrative of A-Nerf as a \"Neural Body Model\". Unlike existing body models, e.g., SMPL, A-Nerf can also capture additional finer details such as Caps, glasses, etc. that are otherwise not easily possible with existing human body models. \n+ The paper demonstrates the ability to refine initial pose estimates by rendering pose-conditioned images using A-Nerf. This is a solid contribution of the paper. Unlike, other differentiable rendering-based methods, A-Nerf does not require a mesh model hence it can inspire future research into test-time optimization with model-free differentiable rendering. \n+ The proposed approach doesn't require MV data to work and relies only on monocular video data. \n+ The experiments are performed for motion capture as well as novel-view synthesis. The evaluation protocols are well-defined and reproducible. \n\n\n**Cons:**\n\nI cannot think of any significant limitation of the paper except its slightly incremental nature i.e., it mainly augments Nerf with pose conditioned embeddings. \n\n**Questions to authors:**\n\n- Have the authors tried learning the cutoff parameters?  \n- Have the authors visualized the underlying geometry of the person? Does it show anything reasonable? \n-  Since most available methods for skeletal human pose estimation only provide joint positions, it will be great if the authors can comment on what will happen if the body pose information only consists of joints positions, rather than joint rotation angles and a T-pose? What has to be changed? \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. ", "review_text": "", "extra_scores": {"ethical_concerns": "Ethical concerns have been addressed appropriately in the paper. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes A-Nerf, a generative neural body model capable of rendering a human under novel viewpoints and poses. A-Nerf is an extension of Nerf to capture dynamic human bodies. This is achieved by conditioning Nerf on the skeletal body pose information. The paper demonstrates that naively conditioning Nerf on the body pose information does not lead to optimal performance and therefore proposes a set of relative encodings of the pose information that results in significantly better renderings without the need of a full-body mesh model.

Reviewers raised concerns regarding the similarity or advantages to the proposed formulation with NeuralBody, which the rebuttal addressed to some extent. Reviewers agree on the novelty of the formulation, specifically the spatial encodings for ray and query point with respect to the skeleton bones. The paper is recommended for publication.  

Decision: Accept (Poster)