A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose
Year: 2021

Abstract:
While deep learning reshaped the classical motion capture pipeline with feedforward networks, generative models are required to recover fine alignment via iterative refinement. Unfortunately, the existing models are usually hand-crafted or learned in controlled conditions, only applicable to limited domains. We propose a method to learn a generative neural body model from unlabelled monocular videos by extending Neural Radiance Fields (NeRFs). We equip them with a skeleton to apply to time-varying and articulated motion. A key insight is that implicit models require the inverse of the forward kinematics used in explicit surface models. Our reparameterization defines spatial latent variables relative to the pose of body parts and thereby overcomes ill-posed inverse operations with an overparameterization. This enables learning volumetric body shape and appearance from scratch while jointly refining the articulated pose; all without ground truth labels for appearance, pose, or 3D shape on the input videos. When used for novel-view-synthesis and motion capture, our neural model improves accuracy on diverse datasets. Project website: https://lemonatsu.github.io/anerf/.

heading: 1 Introduction
text: Generative models have evolved from Generative Adversarial Networks (GANs) recreating images [14, 21] to neural scene representations [39, 56, 57] providing control and image understanding for downstream tasks via structured latent variables. However, most 3D models require 3D labels that cannot be crowd-sourced on natural images and require dedicated depth sensors. It is hence an important research problem to learn 3D representations from 2D observations, which is particularly challenging for humans with diverse body shapes and appearances and their non-rigid motion.
Modern human motion capture techniques typically combine the advantages of discriminative and generative approaches. A feed-forward 3D human pose estimation approach provides a rough initial estimate of human pose. Afterward, a generative approach based on either a high-quality 3D scan of the person [17], or a parametric human body model learned from laser scans [4] refines the estimate iteratively based on the image evidence. Although achieving unprecedented accuracy, existing models require a low-dimensional, restrictive, shape body model or a personalized 3D scan of the user.
We introduce Articulated Neural Radiance Fields (A-NeRF) for learning a user-specific neural 3D body model and underlying skeleton pose from unlabelled videos (see Figure 1). When applied to motion capture, it alleviates the need for template models while maintaining the advantages and accuracy of current generative approaches. A-NeRF extends Neural Radiance Fields (NeRF) [39] to work with single videos and articulated motion. NeRF parameterizes the scene implicitly as
Fφ(Γ(q),Γ(d)) 7→ (σ, c), with σ ∈ R, c ∈ R3,q ∈ R3, and d ∈ R3, (1) by chaining Fφ, a Multi-layer Perceptron (MLP), with Γ, the Positional Encoding (PE) [66]. First, the PE maps the input scene point q and view direction d to a higher dimensional space that enables the
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
MLP to learn a meaningful scene representation function Fφ that subsequently outputs the radiance c and opacity σ at every point in space. Second, the implicitly described scene (via conditioning on query locations) is rendered via classical ray-marching from computer graphics. The advantage of the MLP representation is that it avoids the complexity of volumetric grids [29], circumvents the artifacts induced by the implicit bias of screen-space convolution [40, 54], and, unlike surface meshes, can have flexible topology. However, the original NeRF only works for static scenes captured from dozens of calibrated cameras such that each 3D point is seen from multiple views.
Our conceptual contribution lies in learning a neural latent representation relative to an articulated skeleton. While explicit models such as the popular SMPL body model [30] deform a surface via forwards kinematics, the implicit form of A-NeRF makes us re-think how skeletons can be integrated—implicit networks require the inverse transformation from 3D world coordinates to the reference skeleton, a significantly harder task that has not been fully explored. Our core technical novelty is to come up with and evaluate different parameterizations of Γ(q),Γ(d) in Eq. 1 to create local coordinates relative to the articulated skeleton. Since a point in 3D world coordinates cannot uniquely be associated with a body part, we resolve the mentioned ill-posed inverse problem by overparameterizing with one embedding per bone. This embeds domain knowledge of how humans move and provides a common frame for the neural network to combine body shape and appearance constraints across the entire captured sequence (see Figure 2).
We demonstrate that all our contributions together enable learning of a neural body model from monocular video, requiring only rough 3D pose estimates for initialization, that reaches a level of detail previously only attained with parametric surface models or multi-view approaches [51].
Scope. We apply the model to motion capture, character animation, and appearance and motion transfer and demonstrate that the pose refinement improves on existing monocular skeleton reconstruction. We learn in the transductive setting, for a specific target video that is known at training time but has no ground truth. A-NeRF enables novel view synthesis of dynamic motions, with plausible however non-physical illumination. Additional steps are needed to enable relighting applications.
General impact. Building a self-supervised approach for personalized human body modelling promises to be more inclusive to people and activities that are not well represented in supervised datasets. However, it bears the risk that 3D models of people are created without consent. We urge users to only use datasets collected for developing and validating motion capture algorithms.

heading: 2 Related Work
text: Our approach builds upon and is related to the following work on human pose and shape estimation, human modeling, and neural scene representations [62].
Discriminative Human Pose Estimation. While feed-forward estimation of the 3D joint positions [26, 32, 34, 35, 43, 44, 52, 59, 64, 69, 72] or joint angles and bone lengths [53, 73] of the skeleton is highly accurate, such discriminative estimates are prone to misalignment when overlayed onto the input image due to the generalization gap. The skeleton pose can be refined to better match the 2D pose estimates, but this usually leads to larger errors in 3D [36, 37]. We use [23] for initializing skeleton pose and combine it with a neural body model.
Surface-based Generative Body Models. These are obtained by either constraining template meshes via deformation energies [17, 70] or learning parametric human body models from a large collection of laser scans [6, 9, 30]. Their low-dimensional parameters constrain the space of plausible human shapes and motions. This enables real-time reconstructions from single images [7, 15], detailed texturing and displacement mapping [3, 5], and alleviates manual rigging [2]. It also enables optimization within the bounds of the learned prior [11, 16, 25] and weak-supervision when integrated in a differentiable form [27] into neural training processes [4, 20, 23, 42, 45, 65]. Closest to our approach in this category are the model fitting methods by [74] that textures and geometrically refines an untextured parametric quadruped model to zebra images and to [68] that uses optical flow to refine human pose. Although in a similar setting, our surface-free neural body model and volumetric rendering is fundamentally different to their textured triangle mesh that is rigged with forward kinematics and needs to be obtained a-priori.
Implicit Body Models. A-NeRF bears close similarities with body models defined implicitly in terms of level-sets [58] and density of a sum of Gaussians [18, 48, 49] that are used for refining human pose, shape, and appearance via differentiable ray-tracing. However, sum of Gaussians and other primitives only provide rough approximations.
Neural Scene Representations. Recent neural scene representations learn low-dimensional nonlinear representations of meshes [28, 63], point clouds [1, 38, 67], sphere sets [24], and dense volumetric grids [29, 55]. Their respective geometric output representations enable rendering with classical rendering techniques but have limited expressiveness, e.g., due to the fixed connectivity of a surface mesh and large memory footprint of discretized volumes. This limitation is overcome by using unconstrained MLPs [39] paired with positional encoding [60] to characterize an arbitrary point in 3D space. Common are surface definitions via level-sets of the MLP that are rendered with sphere tracing [56] and density representations rendered with ray-marching [39]. The rendering step is required for maximum likelihood estimation, to define a likelihood over observable variables—real images—while learning a 3D model. The rendering can be learned too [40, 50, 51] but usually leads to inconsistencies, particularly when training data is scarce. Some concurrent works also use neural scene representations for refining camera motion [71], video reenactment [47], and facial models [12, 13]. Orthogonal to these works, our A-NeRF learns an articulated body model from estimated poses and uncalibrated cameras.
Closely related to ours is the NASA surface body model [10] that also defines an implicit function as the minimum of individual implicit functions that are rigidly attached to the bones of a skeleton, each conditioned on the entire human pose to model dependencies and learned from 3D scans. By contrast, we learn a volumetric model instead of a surface model and include appearance and rendering. Even more similar is the recent NeuralBody [46] representation, which combines a NeRF with a surface body model and underlying skeleton. By contrast to both approaches, we do not require surface supervision or initialization, condition pose differently, and refine pose, which enables us to learn from single videos in unconstrained environments.

heading: 3 Formulation
text: Objective. Given a sequence [Ik]Nk=1 of N images Ik ∈ RH×W×3 stemming from one or several videos of the same person, our goal is to simultaneously estimate the time-varying skeleton poses [θk] N k=1 and learn a detailed body model. Our A-NeRF body model Cφ is parametrized by neural network parameters φ that define volumetric shape and color while the skeleton captures motion over time. Figure 2 gives an overview of this generative model. It enables a rendering of the virtual body
model in unseen poses and optimizes its parameters θ and φ on the image reconstruction objective
LSV(θ, φ) = ∑ k
‖Cφ(θk)− Ik‖1︸ ︷︷ ︸ data term +λθd(θk − θ̂k)︸ ︷︷ ︸ pose regularizer +λt
∥∥∥∥∂2θk∂t2 ∥∥∥∥2 2︸ ︷︷ ︸ smoothness prior . (2)
with the influence of all three terms balanced by hyperparameters λt and λθ. The data term measures the distance between the images generated by Cφ and the input image with the L1 distance. The pose regularizer encourages the solution to stay close to an initial pose estimate θ̂ obtained from an offthe-shelf predictor [23], tolerating small shifts up to = 0.01 with d(x) = min(‖x‖22 − , 0). Lastly, the smoothness prior penalizes acceleration ∂
2θk ∂t2 between poses of consecutive frames. Minimizing
Eq. 2 can be seen as maximizing a corresponding probabilistic model, with the quadratic energy terms being the log-likelihoods of Gaussian distributions. Our focus is on formalizing the neural body model. For simplicity, we continue to write equations in terms of the objective functions used during inference with stochastic gradient descent.

heading: 3.1 NeRF and A-NeRF Image Formation Model
text: Instead of modeling the scene as a collection of triangles or other primitives, we define the human implicitly by a neural network as a function (Eq. 1) defined over all possible 3D points and view
directions in space [39]. Similar to NeRF, we render the image of the human subject via ray marching
Cφ(u, v; θk) = Q∑ i=1 Ti(1− exp(−σiδi))ci, Ti = exp − i−1∑ j=1 σjδj  , (3) with (u, v) the 2D pixel location on the image, i the index to 3D query positions qi sampled along d and δi the distance to neighboring samples—a constant if samples would be taken at regular intervals. The Ti is the accumulated transmittance for the ray traveling from the near plane to qi—the fraction of light reaching the sensor from sample point i. The ci is the light color emitted or reflected at i. The final pixel color is the sum over all Q samples, with the last sample taking the special role of the background. The background color is easily inferred as the median pixel color over the entire video for static camera setups. The ray direction d = K−1k (u, v) is computed using the estimated camera intrinsics [23]. In the following, we introduce our skeleton parametrization θk and how to use it to effectively model dynamic articulated human motion.

heading: 3.2 Articulated Skeleton Pose Model
text: Our skeleton representation encodes the connectivity and static bone lengths via a rest pose of 3D joint locations. Dynamics are modeled with per-frame skeleton poses θk, which define an affine transformation T (θk,m) for each bone m. Specifically, T (θk,m) maps a 3D position pk,m ∈ R3 in the m-th local bone coordinates to world coordinates q ∈ R3 using homogeneous coordinates,[
q 1
] = T (θk,m) [ pk,m
1
] , (4)
where subscript k,m indicates that a variable is related to the m-th joint of image Ik. Conversely, T (θk,m)
−1 maps world to local bone coordinates. Note that our skeleton is equivalent to SMPL [30] and others, but without their parametric surface model, and can therefore be initialized with any skeleton pose estimator. We include more details of our skeleton representation in the supplementary.

heading: 3.3 A-NeRF Skeleton-Relative Encoding
text: Our core contribution is to transform the query locations q and view direction d relative to the skeleton before determining the color and opacity at that transformed point via NeRF. It is a form of reparameterization that explicitly incorporates domain knowledge of how the human body parts are linked and transformed relative to each other. Intuitively, our implicit formulation turns explicit models, such as SMPL [30], on its head. Instead of deforming the output surface via skinning, the query location is mapped in the inverse direction to local bone-relative coordinate systems before processing through the NeRF network. Our final model uses a combined encoding ek = [h(ṽk)Γ(ṽk), r̃k, h(ṽk)Γ(dk)] as input to the NeRF Fφ. Note that the skeleton embedding introduces the desired time dependency, denoted by subscript k. The inlet shows our most crucial contribution, the relative distance encoding ṽk followed by PE with Cutoff to reduce the influence of irrelevant bones. We derive the components of our encoding ek and the other alternatives below.
• Reference Pose Encoding One could compensate motion by attaching the query q in world coordinates at frame k to the closest bone m and transforming it with
ak = T (θ0,m)T −1(θk,m)q. (5)
This puts the query relative to the bone m as in frame k but with the skeleton in rest pose θ0. NeRF could then learn without change in the 3D space of the rest pose as done before for surfaces [61]. However, this cannot capture non-rigid pose-dependent effects, such as muscle bulging, and has ambiguities when q is at equal distance to two bones.
• Bone-relative Position (Rel. Pos.) To remove these ambiguities and the ill-posed association to a single part, we map q relative to each bone m with,
q̃k = [q̃k,1, · · · , q̃k,24] and q̃k,m = T−1(θk,m)q. (6)
The resulting individual bone coordinates are well suited to model the overwhelmingly rigid motion of the corresponding body part. Moreover, the overparameterization of position by concatenating all local encodings enables learning when complex interactions are necessary. However, such an embedding for all bones increases the dimensionality by an order of magnitude.
• Relative Distance (Rel. Dist.) Much simpler to compute are distances from q to all bones m,
ṽk = [ṽk,1, · · · , ṽk,24], with ṽk,m = ‖q̃k,m‖2 ∈ R. (7) This radial encoding is used in our final model in favor of q̃ because it naturally captures spherically shaped limbs, is lower-dimensional, and thereby improves reconstruction accuracy.
• Relative Direction (Rel. Dir.) Since the distance encoding is invariant to direction, we additionally obtain the direction vector to capture the orientation information of q,
r̃k = [r̃k,1, · · · , r̃k,24], r̃k,m = q̃k,m ‖q̃k,m‖2 ∈ R3. (8)
Note that by contrast to all other embeddings, this direction encoding did not profit from subsequent PE. We therefore pass it directly into ek.
• Relative Ray Direction (Rel. Ray.) NeRF models the illumination effects in a static 3D scene using the position and view direction. By contrast, our goal is to learn a body model that produces plausible colors with dynamic skeleton poses. Therefore, we transform d to obtain d̃, the outgoing ray direction relative to each bone, similar to query position,
d̃k = [d̃k,1, · · · , d̃k,24], d̃k,m = [T−1(θk,m)]3×3d ∈ R3, (9)
with [T−1(θk,m)]3×3 the rotational part of the bone-to-world transformation T−1(θk,m). Following concurrent works [33, 46], we also optimize an appearance code for each image to handle dynamic light effects. The combination of d̃ and the per-image code enables A-NeRF to approximate the light effects in Ik plausibly. See the supplemental material for detailed discussions on modeling view-dependent effects in our setting.
• Cutoff. We desire a local embedding where points should not be influenced by all but only nearby bones. To this end, we introduce a windowed version of positional encoding by multiplying the encoding with respect to bone m by h(ṽk,m) = 1− S(τ(ṽk,m − t)), with S the sigmoid step function, t the cutoff point, τ the sharpness. This leaves queries unaffected by distant bones.
Our embedding choice of ek = [h(ṽk)Γ(ṽk), r̃k, h(ṽk)Γ(dk)] has the advantage of being invariant to the global shift and rotation of the person and preserves the piece-wise rigidity of articulated motion while still allowing for pose-dependent deformation (see Figure 3). In addition to ek, we also consider other embedding variants. See the supplementary and Section 4 for a detailed discussion.

heading: 4 Evaluation
text: We performed experiments to validate that A-NeRF learns accurate body models and poses, with fewer assumptions (single view, uncalibrated, and w/o a parametric surface model) than the related works. This makes it applicable to fine-grained pose refinement that improves the estimates of stateof-the-art methods. The supplements provide the implementation details, additional comparisons and ablation studies.
Inference and Implementation Details Our A-NeRF model is learned without supervision on a single or multiple videos of the same person. Camera intrinsics, bone lengths for setting am, and pose θk are initialized with [23] for every frame k. These poses are then optimized on objective Eq.2, alongside the generative A-NeRF model. See supplementary for more details.
Datasets. We evaluate on the following benchmarks, and additionally on synthetic data created from SURREAL and Mixamo characters, which are listed in the supplement.
• Human 3.6M [19] The dataset consists of 5 training and 2 testing subjects (S9/S11) with ground truth 3D joint locations. We follow two widely adopted test protocols denoted as Protocol I [20, 22, 23] and Protocol II [41, 59], in which we evaluate 14/17-joint estimation error on every 5th/64th frame of the test videos, respectively. See the supplement for details.
• MPI-INF-3DHP [35] This dataset is a standard benchmark for human pose estimation. It consists of 4 indoor and 2 outdoor subjects with challenging human poses. The number of frames per subject range from 276 to 603.
• MonoPerfCap [70] The dataset consists of human performance video captured with a monocular camera in both indoor and outdoor settings. We use two subjects, Weipeng_outdoor and Nadia_outdoor, for our qualitative experiments. The two subjects have 1151 and 1635 frames, respectively, for training.
Pose Metrics. We report the PA-MPJPE metric, the Euclidean distance between Procrustes-aligned (PA) predictions and ground truth 3D joint position averaged over all frames and joints of the test set. The PA alignment in scale and orientation is essential for comparing approaches that do not assume knowledge of the ground truth calibration and are, hence, ill-posed to the factors that the alignment removes. Following prior work [23, 22, 35], we also report percentange of correct keypoints (PCK) for MPI-INF-3DHP; the percentage of joints that lie within a distance of 150mm to the ground truth.
Visual Metrics. We quantify the visual quality on MonoPerfCap and Human 3.6M datasets by training on a subset and testing on a held-out test set of the same character. Image quality is quantified via the PSNR and SSIM of the rendering compared with the reference image within the character bounding boxes. Because no ground truth pose is available in the required skeleton format on these datasets, we train our model once on the entire dataset to get reliable skeleton pose estimates as pseudo ground truth, and a second time with a part withheld to learn the body model for visual quality evaluation. Since MonoPerfCap has one sequence per actor, we exclude the last 20% of each video. For Human 3.6M we exclude entire actions, namely Geeting-1,2, Posing-1,2 and Walking-1,2. The body model is then transferred to the held-out portion by using the pseudo ground truth poses as the driving motion. Thereby, we can still test the generalization of different models to new poses and viewpoints, irrespective of the underlying skeleton model provided in the dataset.
Novel-View-Synthesis and Character Animation. Our body model is generative, which allows us to train on a single or multiple uncalibrated videos and alter viewpoint and human pose. Figure 4 shows renderings of the same persons from a new camera angle; novel-view-synthesis. Likewise, Figure 5 demonstrates character animation, where the view is fixed, and the underlying skeleton is re-posed by manually changing joint angles or by transferring the motion between characters. A-NeRF is the first model that learns a detailed human body model with such capabilities without needing a 3D surface or multi-view supervision. While NeuralBody [46] can also learn photo-realistic human models from monocular images, their model anchors its representation on the SMPL 3D
surface. When trained on our single-view setup with the same noisy estimated poses θ̂ as input, NeuralBody suffers from artifacts and less detail when training, as it assumes 3D ground truth, which is only available in controlled conditions. The importance of our joint body and pose optimization is further validated by the ablation with refinement disabled (Ours w/o rf.), which similarly produces ghosting artifacts around extremities.
Human Pose Estimation. Training A-NeRF includes a form of test-time optimization (see Figure 1, right), only the initialization from [23] is trained supervised on the Human3.6M training set. Table 1 shows that on Human 3.6M, A-NeRF reaches comparable results with other single-view approaches, and achieves a 8.0% improvement in PA-MPJPE upon the baseline used for pose initialization on Protocol I (42.7 → 39.3) and 5.5% on Protocol II (41.9 → 39.6). Note that these are average numbers across all joints, including easy-to-predict hip, shoulder, and head joints. Our largest gains are on the extremities, e.g., with an improvement of 14% (9.2 mm) for the wrist joint on Protocol II (PA-Wrist). We also compare to applying SMPLify [7], a method that refines 3D poses using 2D joint locations (estimated with [8]) as constraints, at test time. It tends to explain the 2D joint estimates perfectly but degrades the 3D pose. The final pose estimations become less accurate than the initial ones. In contrast, A-NeRF optimizes the poses by implicitly minimizing the disagreement
among the 3D body representation in different images, and thus achieves better performance. On MPI-INF-3DHP, Table 1 shows the results averaged over the 6 test subjects from MPI-INF-3DHP. Despite having a low number of frames and human poses available for learning the skeleton-relative encoding, A-NeRF still provides moderate improvements over the baseline estimations.
Video-based volumetric reconstruction. Figure 6 visualizes the learned density using Marching Cubes [31], with voxel grid resolution of 256 and density threshold 10. Despite only learned from monocular videos (no stereo, depth camera, or multi-view constraints) and without using a predefined template model, A-NeRF reconstructs a detailed volumetric body with details that could not be captured by offsets to a parametric surface model, such as the head phones and basebal cap in the last row of Figure 6. Note that no geometric smoothness term is enforced.
Multi-view extension. A-NeRF can also leverage multi-view refinement (Table 1, Protocol II Multi-view), even without access to ground truth camera calibration (see supplementary).
Visual Quality Comparison. We report the results in Table 2. We compare to NeuralBody with both the initial estimates from [23] and our refined pose since NeuralBody has no refinement step. Compared to both variants, our A-NeRF shows significantly better reconstruction performance on held-out poses. We observe that NeuralBody models can retain the facial features and hands as they anchor their representation on a 3D surface model. However, the rendered limbs and faces are blurry and distorted. Results are similar to training A-NeRF without refinement (see Figure 7). As the estimation for these joints is often inaccurate and noisy, the models without pose refinement simply learn to predict mean pixel values. To conclude, it is important to train with pose refinement, with which A-NeRF suffers less from artifacts with overall better visual quality.
Ablation study. Our detailed ablation studies are reported in the supplemental document. In summary, they reveal: i) Embedding relative 3D position, q̃k, instead of our proposed radial embedding ṽ yields only half as good pose refinements. ii) Our embedding choices keep the dimensionality moderate while improving on or matching the PSNR and SSIM of higher-dimensional variants. iii) For a fixed number of images with accurate poses, learning from a long video with diverse poses has visual quality comparable to learning from multiple shorter clips.
Reference Failure Limitations and Failure Cases. Our computation time is the biggest bottleneck in extending A-NeRF to long sequences and multiple actors. Although a single static camera suffices, A-NeRF requires to see the person from all sides in varying poses to learn pose dependencies from data. The inlet shows a rendering of an extreme breakdance pose retargeted to a model trained on normal walking motions. Hence, the source pose is unseen during training and far from the data distribution, which leads to artifacts.

heading: 5 Conclusion
text: We propose a new way for integrating articulated skeleton models and implicit functions via an overcomplete re-parametrization. It includes learning an interpretable 3D representation from 2D images; a personalized volumetric density field with texture detail and time-varying poses of the actor depicted in the input. The underlying ill-posed problem of mapping a single query location to multiple parts is addressed with an overparametrization over nearby parts. To the best of our knowledge, ANeRF is the first approach to define NeRF models for extreme and articulated motion on unconstrained video and this new approach scores high on the Human 3.6M benchmark. Importantly, it works from a single video and naturally extends to multi-view and does not require camera calibration in either scenario. This is an important step towards making motion capture more accurate and practical. In future work, we will learn a general human model from a database of subjects instead of individuals.
Funding in direct support of this work: NSERC Discovery grant, UBC Advanced Research Computing (ARC) GPU cluster, Compute Canada GPU servers, and a gift by Facebook Reality Labs.

Review summaries:
This paper aims to refine the 3D human poses. To this end, the authors equip neural radiance fields with human skeletons, where they design a skeleton-relative embedding based on the human skeleton. By training the A-NeRF on the video, the human poses are optimized to fit the observations.

This submission presents a way for adding articulation to neural radiance fields. The main contribution is an encoding of the query points and view directions relative to a skeleton. The proposed formulation is trained per subject on a video sequence and optimizes for the 3D skeleton configuration in each frame and a global MLP which represents the NeRF. This defines the motion tracking throughout the input sequence and the recovered model can be reposed by rendering frames with new joint locations.

The paper presents A-Nerf which is a generative neural body model capable of rendering a human under novel viewpoints and poses. A-Nerf is an extension of Nerf to capture dynamic human bodies. It's achieved by conditioning Nerf on the skeletal body pose information. The paper demonstrates that naively conditioning Nerf on the body pose information does not lead to optimal performance and therefore proposes a set of relative encodings of the pose information that results in significantly better renderings without the need of a full-body mesh model. The experiments are performed for motion capture as well as novel view synthesis where the proposed method demonstrates superior performance as compared to existing methods. 

This paper presents modelling articulated neural radiance fields for human dynamics from monocular videos. To model NeRF for dynamic human, it encodes 3d spatial points and ray directions w.r.t the articulated skeleton poses, which are fine-tuned together when training the NeRF field. Various skeleton-based feature encoding are experimented in consideration of both performance and memory.