A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose

1 Introduction . Generative models have evolved from Generative Adversarial Networks ( GANs ) recreating images [ 14 , 21 ] to neural scene representations [ 39 , 56 , 57 ] providing control and image understanding for downstream tasks via structured latent variables . However , most 3D models require 3D labels that can not be crowd-sourced on natural images and require dedicated depth sensors . It is hence an important research problem to learn 3D representations from 2D observations , which is particularly challenging for humans with diverse body shapes and appearances and their non-rigid motion . Modern human motion capture techniques typically combine the advantages of discriminative and generative approaches . A feed-forward 3D human pose estimation approach provides a rough initial estimate of human pose . Afterward , a generative approach based on either a high-quality 3D scan of the person [ 17 ] , or a parametric human body model learned from laser scans [ 4 ] refines the estimate iteratively based on the image evidence . Although achieving unprecedented accuracy , existing models require a low-dimensional , restrictive , shape body model or a personalized 3D scan of the user . We introduce Articulated Neural Radiance Fields ( A-NeRF ) for learning a user-specific neural 3D body model and underlying skeleton pose from unlabelled videos ( see Figure 1 ) . When applied to motion capture , it alleviates the need for template models while maintaining the advantages and accuracy of current generative approaches . A-NeRF extends Neural Radiance Fields ( NeRF ) [ 39 ] to work with single videos and articulated motion . NeRF parameterizes the scene implicitly as Fφ ( Γ ( q ) , Γ ( d ) ) 7→ ( σ , c ) , with σ ∈ R , c ∈ R3 , q ∈ R3 , and d ∈ R3 , ( 1 ) by chaining Fφ , a Multi-layer Perceptron ( MLP ) , with Γ , the Positional Encoding ( PE ) [ 66 ] . First , the PE maps the input scene point q and view direction d to a higher dimensional space that enables the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . MLP to learn a meaningful scene representation function Fφ that subsequently outputs the radiance c and opacity σ at every point in space . Second , the implicitly described scene ( via conditioning on query locations ) is rendered via classical ray-marching from computer graphics . The advantage of the MLP representation is that it avoids the complexity of volumetric grids [ 29 ] , circumvents the artifacts induced by the implicit bias of screen-space convolution [ 40 , 54 ] , and , unlike surface meshes , can have flexible topology . However , the original NeRF only works for static scenes captured from dozens of calibrated cameras such that each 3D point is seen from multiple views . Our conceptual contribution lies in learning a neural latent representation relative to an articulated skeleton . While explicit models such as the popular SMPL body model [ 30 ] deform a surface via forwards kinematics , the implicit form of A-NeRF makes us re-think how skeletons can be integrated—implicit networks require the inverse transformation from 3D world coordinates to the reference skeleton , a significantly harder task that has not been fully explored . Our core technical novelty is to come up with and evaluate different parameterizations of Γ ( q ) , Γ ( d ) in Eq . 1 to create local coordinates relative to the articulated skeleton . Since a point in 3D world coordinates can not uniquely be associated with a body part , we resolve the mentioned ill-posed inverse problem by overparameterizing with one embedding per bone . This embeds domain knowledge of how humans move and provides a common frame for the neural network to combine body shape and appearance constraints across the entire captured sequence ( see Figure 2 ) . We demonstrate that all our contributions together enable learning of a neural body model from monocular video , requiring only rough 3D pose estimates for initialization , that reaches a level of detail previously only attained with parametric surface models or multi-view approaches [ 51 ] . Scope . We apply the model to motion capture , character animation , and appearance and motion transfer and demonstrate that the pose refinement improves on existing monocular skeleton reconstruction . We learn in the transductive setting , for a specific target video that is known at training time but has no ground truth . A-NeRF enables novel view synthesis of dynamic motions , with plausible however non-physical illumination . Additional steps are needed to enable relighting applications . General impact . Building a self-supervised approach for personalized human body modelling promises to be more inclusive to people and activities that are not well represented in supervised datasets . However , it bears the risk that 3D models of people are created without consent . We urge users to only use datasets collected for developing and validating motion capture algorithms . 2 Related Work . Our approach builds upon and is related to the following work on human pose and shape estimation , human modeling , and neural scene representations [ 62 ] . Discriminative Human Pose Estimation . While feed-forward estimation of the 3D joint positions [ 26 , 32 , 34 , 35 , 43 , 44 , 52 , 59 , 64 , 69 , 72 ] or joint angles and bone lengths [ 53 , 73 ] of the skeleton is highly accurate , such discriminative estimates are prone to misalignment when overlayed onto the input image due to the generalization gap . The skeleton pose can be refined to better match the 2D pose estimates , but this usually leads to larger errors in 3D [ 36 , 37 ] . We use [ 23 ] for initializing skeleton pose and combine it with a neural body model . Surface-based Generative Body Models . These are obtained by either constraining template meshes via deformation energies [ 17 , 70 ] or learning parametric human body models from a large collection of laser scans [ 6 , 9 , 30 ] . Their low-dimensional parameters constrain the space of plausible human shapes and motions . This enables real-time reconstructions from single images [ 7 , 15 ] , detailed texturing and displacement mapping [ 3 , 5 ] , and alleviates manual rigging [ 2 ] . It also enables optimization within the bounds of the learned prior [ 11 , 16 , 25 ] and weak-supervision when integrated in a differentiable form [ 27 ] into neural training processes [ 4 , 20 , 23 , 42 , 45 , 65 ] . Closest to our approach in this category are the model fitting methods by [ 74 ] that textures and geometrically refines an untextured parametric quadruped model to zebra images and to [ 68 ] that uses optical flow to refine human pose . Although in a similar setting , our surface-free neural body model and volumetric rendering is fundamentally different to their textured triangle mesh that is rigged with forward kinematics and needs to be obtained a-priori . Implicit Body Models . A-NeRF bears close similarities with body models defined implicitly in terms of level-sets [ 58 ] and density of a sum of Gaussians [ 18 , 48 , 49 ] that are used for refining human pose , shape , and appearance via differentiable ray-tracing . However , sum of Gaussians and other primitives only provide rough approximations . Neural Scene Representations . Recent neural scene representations learn low-dimensional nonlinear representations of meshes [ 28 , 63 ] , point clouds [ 1 , 38 , 67 ] , sphere sets [ 24 ] , and dense volumetric grids [ 29 , 55 ] . Their respective geometric output representations enable rendering with classical rendering techniques but have limited expressiveness , e.g. , due to the fixed connectivity of a surface mesh and large memory footprint of discretized volumes . This limitation is overcome by using unconstrained MLPs [ 39 ] paired with positional encoding [ 60 ] to characterize an arbitrary point in 3D space . Common are surface definitions via level-sets of the MLP that are rendered with sphere tracing [ 56 ] and density representations rendered with ray-marching [ 39 ] . The rendering step is required for maximum likelihood estimation , to define a likelihood over observable variables—real images—while learning a 3D model . The rendering can be learned too [ 40 , 50 , 51 ] but usually leads to inconsistencies , particularly when training data is scarce . Some concurrent works also use neural scene representations for refining camera motion [ 71 ] , video reenactment [ 47 ] , and facial models [ 12 , 13 ] . Orthogonal to these works , our A-NeRF learns an articulated body model from estimated poses and uncalibrated cameras . Closely related to ours is the NASA surface body model [ 10 ] that also defines an implicit function as the minimum of individual implicit functions that are rigidly attached to the bones of a skeleton , each conditioned on the entire human pose to model dependencies and learned from 3D scans . By contrast , we learn a volumetric model instead of a surface model and include appearance and rendering . Even more similar is the recent NeuralBody [ 46 ] representation , which combines a NeRF with a surface body model and underlying skeleton . By contrast to both approaches , we do not require surface supervision or initialization , condition pose differently , and refine pose , which enables us to learn from single videos in unconstrained environments . 3 Formulation . Objective . Given a sequence [ Ik ] Nk=1 of N images Ik ∈ RH×W×3 stemming from one or several videos of the same person , our goal is to simultaneously estimate the time-varying skeleton poses [ θk ] N k=1 and learn a detailed body model . Our A-NeRF body model Cφ is parametrized by neural network parameters φ that define volumetric shape and color while the skeleton captures motion over time . Figure 2 gives an overview of this generative model . It enables a rendering of the virtual body model in unseen poses and optimizes its parameters θ and φ on the image reconstruction objective LSV ( θ , φ ) = ∑ k ‖Cφ ( θk ) − Ik‖1︸ ︷︷ ︸ data term +λθd ( θk − θ̂k ) ︸ ︷︷ ︸ pose regularizer +λt ∥∥∥∥∂2θk∂t2 ∥∥∥∥2 2︸ ︷︷ ︸ smoothness prior . ( 2 ) with the influence of all three terms balanced by hyperparameters λt and λθ . The data term measures the distance between the images generated by Cφ and the input image with the L1 distance . The pose regularizer encourages the solution to stay close to an initial pose estimate θ̂ obtained from an offthe-shelf predictor [ 23 ] , tolerating small shifts up to = 0.01 with d ( x ) = min ( ‖x‖22 − , 0 ) . Lastly , the smoothness prior penalizes acceleration ∂ 2θk ∂t2 between poses of consecutive frames . Minimizing Eq . 2 can be seen as maximizing a corresponding probabilistic model , with the quadratic energy terms being the log-likelihoods of Gaussian distributions . Our focus is on formalizing the neural body model . For simplicity , we continue to write equations in terms of the objective functions used during inference with stochastic gradient descent . 3.1 NeRF and A-NeRF Image Formation Model . Instead of modeling the scene as a collection of triangles or other primitives , we define the human implicitly by a neural network as a function ( Eq . 1 ) defined over all possible 3D points and view directions in space [ 39 ] . Similar to NeRF , we render the image of the human subject via ray marching Cφ ( u , v ; θk ) = Q∑ i=1 Ti ( 1− exp ( −σiδi ) ) ci , Ti = exp − i−1∑ j=1 σjδj , ( 3 ) with ( u , v ) the 2D pixel location on the image , i the index to 3D query positions qi sampled along d and δi the distance to neighboring samples—a constant if samples would be taken at regular intervals . The Ti is the accumulated transmittance for the ray traveling from the near plane to qi—the fraction of light reaching the sensor from sample point i . The ci is the light color emitted or reflected at i . The final pixel color is the sum over all Q samples , with the last sample taking the special role of the background . The background color is easily inferred as the median pixel color over the entire video for static camera setups . The ray direction d = K−1k ( u , v ) is computed using the estimated camera intrinsics [ 23 ] . In the following , we introduce our skeleton parametrization θk and how to use it to effectively model dynamic articulated human motion .

Summary: This paper presents modelling articulated neural radiance fields for human dynamics from monocular videos. To model NeRF for dynamic human, it encodes 3d spatial points and ray directions w.r.t the articulated skeleton poses, which are fine-tuned together when training the NeRF field. Various skeleton-based feature encoding are experimented in consideration of both performance and memory.