A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose

1 Introduction . Generative models have evolved from Generative Adversarial Networks ( GANs ) recreating images [ 14 , 21 ] to neural scene representations [ 39 , 56 , 57 ] providing control and image understanding for downstream tasks via structured latent variables . However , most 3D models require 3D labels that can not be crowd-sourced on natural images and require dedicated depth sensors . It is hence an important research problem to learn 3D representations from 2D observations , which is particularly challenging for humans with diverse body shapes and appearances and their non-rigid motion . Modern human motion capture techniques typically combine the advantages of discriminative and generative approaches . A feed-forward 3D human pose estimation approach provides a rough initial estimate of human pose . Afterward , a generative approach based on either a high-quality 3D scan of the person [ 17 ] , or a parametric human body model learned from laser scans [ 4 ] refines the estimate iteratively based on the image evidence . Although achieving unprecedented accuracy , existing models require a low-dimensional , restrictive , shape body model or a personalized 3D scan of the user . We introduce Articulated Neural Radiance Fields ( A-NeRF ) for learning a user-specific neural 3D body model and underlying skeleton pose from unlabelled videos ( see Figure 1 ) . When applied to motion capture , it alleviates the need for template models while maintaining the advantages and accuracy of current generative approaches . A-NeRF extends Neural Radiance Fields ( NeRF ) [ 39 ] to work with single videos and articulated motion . NeRF parameterizes the scene implicitly as Fφ ( Γ ( q ) , Γ ( d ) ) 7→ ( σ , c ) , with σ ∈ R , c ∈ R3 , q ∈ R3 , and d ∈ R3 , ( 1 ) by chaining Fφ , a Multi-layer Perceptron ( MLP ) , with Γ , the Positional Encoding ( PE ) [ 66 ] . First , the PE maps the input scene point q and view direction d to a higher dimensional space that enables the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . MLP to learn a meaningful scene representation function Fφ that subsequently outputs the radiance c and opacity σ at every point in space . Second , the implicitly described scene ( via conditioning on query locations ) is rendered via classical ray-marching from computer graphics . The advantage of the MLP representation is that it avoids the complexity of volumetric grids [ 29 ] , circumvents the artifacts induced by the implicit bias of screen-space convolution [ 40 , 54 ] , and , unlike surface meshes , can have flexible topology . However , the original NeRF only works for static scenes captured from dozens of calibrated cameras such that each 3D point is seen from multiple views . Our conceptual contribution lies in learning a neural latent representation relative to an articulated skeleton . While explicit models such as the popular SMPL body model [ 30 ] deform a surface via forwards kinematics , the implicit form of A-NeRF makes us re-think how skeletons can be integrated—implicit networks require the inverse transformation from 3D world coordinates to the reference skeleton , a significantly harder task that has not been fully explored . Our core technical novelty is to come up with and evaluate different parameterizations of Γ ( q ) , Γ ( d ) in Eq . 1 to create local coordinates relative to the articulated skeleton . Since a point in 3D world coordinates can not uniquely be associated with a body part , we resolve the mentioned ill-posed inverse problem by overparameterizing with one embedding per bone . This embeds domain knowledge of how humans move and provides a common frame for the neural network to combine body shape and appearance constraints across the entire captured sequence ( see Figure 2 ) . We demonstrate that all our contributions together enable learning of a neural body model from monocular video , requiring only rough 3D pose estimates for initialization , that reaches a level of detail previously only attained with parametric surface models or multi-view approaches [ 51 ] . Scope . We apply the model to motion capture , character animation , and appearance and motion transfer and demonstrate that the pose refinement improves on existing monocular skeleton reconstruction . We learn in the transductive setting , for a specific target video that is known at training time but has no ground truth . A-NeRF enables novel view synthesis of dynamic motions , with plausible however non-physical illumination . Additional steps are needed to enable relighting applications . General impact . Building a self-supervised approach for personalized human body modelling promises to be more inclusive to people and activities that are not well represented in supervised datasets . However , it bears the risk that 3D models of people are created without consent . We urge users to only use datasets collected for developing and validating motion capture algorithms . 2 Related Work . Our approach builds upon and is related to the following work on human pose and shape estimation , human modeling , and neural scene representations [ 62 ] . Discriminative Human Pose Estimation . While feed-forward estimation of the 3D joint positions [ 26 , 32 , 34 , 35 , 43 , 44 , 52 , 59 , 64 , 69 , 72 ] or joint angles and bone lengths [ 53 , 73 ] of the skeleton is highly accurate , such discriminative estimates are prone to misalignment when overlayed onto the input image due to the generalization gap . The skeleton pose can be refined to better match the 2D pose estimates , but this usually leads to larger errors in 3D [ 36 , 37 ] . We use [ 23 ] for initializing skeleton pose and combine it with a neural body model . Surface-based Generative Body Models . These are obtained by either constraining template meshes via deformation energies [ 17 , 70 ] or learning parametric human body models from a large collection of laser scans [ 6 , 9 , 30 ] . Their low-dimensional parameters constrain the space of plausible human shapes and motions . This enables real-time reconstructions from single images [ 7 , 15 ] , detailed texturing and displacement mapping [ 3 , 5 ] , and alleviates manual rigging [ 2 ] . It also enables optimization within the bounds of the learned prior [ 11 , 16 , 25 ] and weak-supervision when integrated in a differentiable form [ 27 ] into neural training processes [ 4 , 20 , 23 , 42 , 45 , 65 ] . Closest to our approach in this category are the model fitting methods by [ 74 ] that textures and geometrically refines an untextured parametric quadruped model to zebra images and to [ 68 ] that uses optical flow to refine human pose . Although in a similar setting , our surface-free neural body model and volumetric rendering is fundamentally different to their textured triangle mesh that is rigged with forward kinematics and needs to be obtained a-priori . Implicit Body Models . A-NeRF bears close similarities with body models defined implicitly in terms of level-sets [ 58 ] and density of a sum of Gaussians [ 18 , 48 , 49 ] that are used for refining human pose , shape , and appearance via differentiable ray-tracing . However , sum of Gaussians and other primitives only provide rough approximations . Neural Scene Representations . Recent neural scene representations learn low-dimensional nonlinear representations of meshes [ 28 , 63 ] , point clouds [ 1 , 38 , 67 ] , sphere sets [ 24 ] , and dense volumetric grids [ 29 , 55 ] . Their respective geometric output representations enable rendering with classical rendering techniques but have limited expressiveness , e.g. , due to the fixed connectivity of a surface mesh and large memory footprint of discretized volumes . This limitation is overcome by using unconstrained MLPs [ 39 ] paired with positional encoding [ 60 ] to characterize an arbitrary point in 3D space . Common are surface definitions via level-sets of the MLP that are rendered with sphere tracing [ 56 ] and density representations rendered with ray-marching [ 39 ] . The rendering step is required for maximum likelihood estimation , to define a likelihood over observable variables—real images—while learning a 3D model . The rendering can be learned too [ 40 , 50 , 51 ] but usually leads to inconsistencies , particularly when training data is scarce . Some concurrent works also use neural scene representations for refining camera motion [ 71 ] , video reenactment [ 47 ] , and facial models [ 12 , 13 ] . Orthogonal to these works , our A-NeRF learns an articulated body model from estimated poses and uncalibrated cameras . Closely related to ours is the NASA surface body model [ 10 ] that also defines an implicit function as the minimum of individual implicit functions that are rigidly attached to the bones of a skeleton , each conditioned on the entire human pose to model dependencies and learned from 3D scans . By contrast , we learn a volumetric model instead of a surface model and include appearance and rendering . Even more similar is the recent NeuralBody [ 46 ] representation , which combines a NeRF with a surface body model and underlying skeleton . By contrast to both approaches , we do not require surface supervision or initialization , condition pose differently , and refine pose , which enables us to learn from single videos in unconstrained environments . 3 Formulation . Objective . Given a sequence [ Ik ] Nk=1 of N images Ik ∈ RH×W×3 stemming from one or several videos of the same person , our goal is to simultaneously estimate the time-varying skeleton poses [ θk ] N k=1 and learn a detailed body model . Our A-NeRF body model Cφ is parametrized by neural network parameters φ that define volumetric shape and color while the skeleton captures motion over time . Figure 2 gives an overview of this generative model . It enables a rendering of the virtual body model in unseen poses and optimizes its parameters θ and φ on the image reconstruction objective LSV ( θ , φ ) = ∑ k ‖Cφ ( θk ) − Ik‖1︸ ︷︷ ︸ data term +λθd ( θk − θ̂k ) ︸ ︷︷ ︸ pose regularizer +λt ∥∥∥∥∂2θk∂t2 ∥∥∥∥2 2︸ ︷︷ ︸ smoothness prior . ( 2 ) with the influence of all three terms balanced by hyperparameters λt and λθ . The data term measures the distance between the images generated by Cφ and the input image with the L1 distance . The pose regularizer encourages the solution to stay close to an initial pose estimate θ̂ obtained from an offthe-shelf predictor [ 23 ] , tolerating small shifts up to = 0.01 with d ( x ) = min ( ‖x‖22 − , 0 ) . Lastly , the smoothness prior penalizes acceleration ∂ 2θk ∂t2 between poses of consecutive frames . Minimizing Eq . 2 can be seen as maximizing a corresponding probabilistic model , with the quadratic energy terms being the log-likelihoods of Gaussian distributions . Our focus is on formalizing the neural body model . For simplicity , we continue to write equations in terms of the objective functions used during inference with stochastic gradient descent . 3.1 NeRF and A-NeRF Image Formation Model . Instead of modeling the scene as a collection of triangles or other primitives , we define the human implicitly by a neural network as a function ( Eq . 1 ) defined over all possible 3D points and view directions in space [ 39 ] . Similar to NeRF , we render the image of the human subject via ray marching Cφ ( u , v ; θk ) = Q∑ i=1 Ti ( 1− exp ( −σiδi ) ) ci , Ti = exp − i−1∑ j=1 σjδj , ( 3 ) with ( u , v ) the 2D pixel location on the image , i the index to 3D query positions qi sampled along d and δi the distance to neighboring samples—a constant if samples would be taken at regular intervals . The Ti is the accumulated transmittance for the ray traveling from the near plane to qi—the fraction of light reaching the sensor from sample point i . The ci is the light color emitted or reflected at i . The final pixel color is the sum over all Q samples , with the last sample taking the special role of the background . The background color is easily inferred as the median pixel color over the entire video for static camera setups . The ray direction d = K−1k ( u , v ) is computed using the estimated camera intrinsics [ 23 ] . In the following , we introduce our skeleton parametrization θk and how to use it to effectively model dynamic articulated human motion .

Summary: This submission presents a way for adding articulation to neural radiance fields. The main contribution is an encoding of the query points and view directions relative to a skeleton. The proposed formulation is trained per subject on a video sequence and optimizes for the 3D skeleton configuration in each frame and a global MLP which represents the NeRF. This defines the motion tracking throughout the input sequence and the recovered model can be reposed by rendering frames with new joint locations.