H-NeRF: Neural Radiance Fields for Rendering and Temporal Reconstruction of Humans in Motion

1 Introduction . Enabling free-viewpoint video of a human in motion , based on a sparse set of views , is extremely challenging , but has many applications . Our work is motivated by a breadth of transformative 3D use cases , including immersive visualization of photographs , virtual clothing and try-on , fitness , as well as AR and VR for improved communication or collaboration . However , so far static scenes and rigid objects have been the primary subject of research . In pursuing realistic novel-view synthesis two schools of thought have been established : 1 ) 3D reconstruction methods aim to recover the geometry of the observed scene as accurately as possible , with novel views generated using classical rendering pipelines [ 30 , 42 ] . 2 ) Image-based rendering techniques [ 7 , 10 , 40 ] and very recently neural radiance fields [ 29 ] primarily aim at image production quality without explicitly constructing an accurate 3D geometric model . While these techniques explicitly or implicitly reconstruct the scene geometry sometimes , this is however not guaranteed to accurately resemble the true scene geometry . We argue that novel-view rendering and reconstruction are two sides of the same coin , and that reliable viewpoint generalization , especially given relatively few input views , would require good quality for both . To this end , we propose a unified model in order to support both robust reconstruction and photo-realistic rendering . Dynamic scenes , especially those capturing a human in motion , add considerable complexity to the problem : while static scenes can be observed from many views by a camera moving through the scene , any configuration of a dynamic scene is typically observed only from sparse views . Moreover , the scene geometry and its appearance may change considerably over time . To cope with few views , some methods integrate scene knowledge over time by warping observations into a common reference frame [ 35 , 37 ] . At test time , the information is warped back to the desired state and rendered from a novel view . Extrapolating to unseen motion , however , remains challenging . For scenes capturing people , this means that only poses seen during training can be rendered at test time . For some applications , however , rendering the subject over a broad range of motions , or in novel poses , is desirable . To make generalisation over poses and views possible , we rely on additional problem domain knowledge in the form of a human body model , imGHUM [ 4 ] . imGHUM is an implicit signed distance function ( SDF ) conditioned on generative shape and pose 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . codes learned from a large corpus of dynamic human scans . In this work , imGHUM is used as the common reference frame for a neural radiance field and as a structured prior for robust reconstruction . Additionally , since imGHUM can represent a broad distribution of statistically valid human poses and shapes , we can render the reconstructed subject in novel poses and even with modified body shapes . In summary , our system supports photo-realistic free-view point temporal rendering of a human subject given only sparse camera observations . By conditioning on an implicit human body model , we can render considerably different viewpoints , body poses , and body shapes compared to those observed in training . Our carefully designed losses ensure not only good image quality but also plausible temporal reconstructions that can be used for the free-viewpoint visualisation of human performance capture . 2 Related Work . Human performance capture is the process of reconstructing the dynamic ( 4D ) geometry of a human subject as observed in one or multiple synchronized views . Pioneering methods deform a rigged template mesh against silhouettes observed in multiple views [ 6 , 50 ] , estimated skeleton key-points [ 12 ] , or image correspondences [ 9 ] . Later systems relying on RGB-D video streams [ 5 ] or monocular capture [ 3 , 2 , 17 , 52 , 16 , 15 ] have also been presented . All these methods rely either on a pre-scanned template mesh or on a body model that is deformed to explain the image evidence . Neural representation for view synthesis . With the advent of neural networks , researchers have begun to explore alternative solutions to represent a scene in order to support novel view synthesis and photo-realistic rendering ; we refer the reader to [ 48 ] for a survey . Different scene representation have been explored . Some methods use voxel grids to embed the scene [ 23 , 45 ] . For rendering , the voxel grid is probed by shooting a ray and by linearly interpolating voxel values . These samples are then transformed into color values using a neural network . Others have proposed neural textures [ 49 , 43 ] that can be rendered based on view-dependent effects or texture synthesis networks [ 22 ] to realistically render meshes . In a similar spirit , other methods render point clouds , where each point carries local appearance information [ 26 , 1 ] . Riegler and Koltun [ 40 ] reproject features from nearby views and rely on a SfM reconstruction scaffold for novel view synthesis . With the recent advent of implicit function networks [ 25 , 34 , 8 , 27 ] , such 3D representations have been explored for rendering and view synthesis with great success . In contrast to discrete voxel representations , textures , or point clouds , these methods represent the scene as a continuous function and thus are not bound to a specific image or volume resolution . In the pioneering work of Sitzmann et al . an implicit function produces features displayed using a neural renderer [ 46 ] . Follow up methods focus on 3D geometric reconstruction [ 44 ] , and use 2D supervision [ 31 , 53 ] . Neural Radiance Fields ( NeRFs ) are a recent approach to represent scenes for novel view synthesis . Mildenhall et al . [ 29 ] introduce Neural Radiance Fields resented as fully connected neural networks , where the input is a spatial query point and a viewing direction . The output is a volume density and the emitted radiance at the query location in the direction of the viewer . By ray-tracing using this simple representation , one can generate photo-realistic images from novel views . Despite the excellent quality of results , one drawback is the slow rendering time . To this end , researchers have presented faster versions that e.g . transform the radiance field into more efficient sparse grids [ 18 ] or remove the dependence on viewing-direction during rendering by estimating a spherical harmonic representation of the radiance function [ 54 ] . Others improve fidelity or rendering time by tackling ambiguities in the original formulation [ 56 ] , spatial decomposition into multiple NeRFs [ 39 ] , or combining NeRF with sparse voxel fields [ 21 ] . Initial work on adapting NeRF to dynamic scenes has been presented as well . Park et al . [ 35 ] produce “ Nerfies ” ( NeRF-Selfies ) from videos where subjects carefully move a camera around their head . The scene information is fused by warping query points into a canonical reference frame . Similarly , Pumarola et al . [ 37 ] produce dynamic NeRFs from synthetic animation data . Related to our approach , some methods integrate human body models to fuse information over time . A-NeRF [ 47 ] uses a skeleton to rigidly transform NeRF features to refine estimated 3D poses . A similar approach is followed in NARF [ 32 ] for view synthesis . Most related , Neural Body [ 36 ] attaches learnable features to the vertices of a SMPL body model [ 24 ] . These features are processed using a sparse 3D convolutional network , where the output forms a neural radiance field . In contrast to our approach , the resolution is bounded by the spatial resolution of the 3D convolutional network and no geometric supervision is used . We highlight differences in §5 . 3 Background . Given a collection of images capturing a dynamic scene of a human in motion , observed from a sparse set of calibrated camera views ( in the limit a monocular camera , as we will show ) , we aim to learn both the detailed temporal geometry of the human in motion , and to render the sequence from novel camera views and for different human poses . To this end , our work unifies two main methodologies : 1 ) implicit 3D human representations , and 2 ) volumetric radiance fields . In this section , we provide the relevant background on both representations , which we co-learn in a joint framework . Neural Radiance Fields . A neural radiance field ( NeRF ) [ 29 ] represents a 3D scene as a continuous function of color volume densities . The model consists of a neural network function Fω that maps a 3D spatial point x ∈ R3 and a viewing direction v ∈ R3 to a volume density σ ∈ R+ and a radiance c ( x , v ) ∈ R3 emitted towards the viewer . In practice , NeRF encodes the inputs x and v using a sinusoidal positional encoding γ : R3 → R3+6m that projects a coordinate vector into a high-dimensional space using a set of sine and cosine functions of m increasing frequencies . Given a ray r = o + sv with N samples { x } originating from a camera location o , NeRF integrates radiance values along the ray by means of alpha blending . The pixel/ray color is approximated with numerical quadrature [ 33 ] , C ( r ) = N∑ i=1 α ( xi ) ∏ j < i ( 1− α ( xj ) ) c ( xi , v ) , ( 1 ) α ( xi ) = 1− exp ( −σ ( xi ) δi ) , ( 2 ) where α ( xi ) is the transparency by accumulating transmittance along the ray , and δi = |xi+1 − xi| is the distance between adjacent samples . The NeRF function Fw is fully differentiable and its network parameters w can be optimized using an image reconstruction loss [ 29 ] . An approximate 3D scene geometry SF = { x|σ ( x ) = σh } can be extracted from the trained opacity field via Marching Cubes [ 20 ] at a density threshold σh . Implicit Generative Human Models . Implicit human body surfaces are typically represented as the decision boundary of either binary occupancy classifiers [ 11 , 41 , 28 ] or signed distance functions [ 14 , 4 ] . Specifically , our work builds upon the SOTA statistical implicit human model imGHUM [ 4 ] Hω : ( T−1x , β , θ ) → ( d , s ) that maps a 3D spatial point x , unposed with root joint transformation T ∈ R4×3 , to its signed distance value d ∈ R with respect to a body surface parameterized with body shape β ∈ R16 and articulated pose θ ∈ R118 . In addition to d , imGHUM returns implicit continuous semantics s ∈ R3 of the query point which corresponds to the 3D coordinate of the nearest surface point defined on a canonical surface . We refer to the original paper [ 4 ] for details . Essentially , imGHUM builds a human-centric 4D semantic descriptor ( d , s ) for all spatial points in the neighborhood of the surface . imGHUM is trained with a large collection of human scans of diverse body shapes and poses , sharing the same generative shape and pose latent code with the mesh-based statistical human model GHUM [ 51 ] . The implicit articulated 3D human body SH ( β , θ , T ) = { x|d ( x ) = 0 } is defined by the the zero-isosurface of the signed distance field .

Summary: The authors propose a model that combines a parameteric human shape model with NeRF and a residual signed distance function to enable novel-view synthesis of humans in motion, assuming full observation of the subject (i.e., no prior-based completion).  The core contributions is the combination of NeRF with a parametric human body model, which enables the learning of the NeRF in a canonical coordinate frame, as well as regularization of the learned geometry. Several additional tweaks improve the model further, such as leveraging auxiliarly losses via a instance segmentation mask, or feeding the estimated body pose to the NeRF as additional input, enabling dynamic (due to being pose-specific) fine detail.