H-NeRF: Neural Radiance Fields for Rendering and Temporal Reconstruction of Humans in Motion

1 Introduction . Enabling free-viewpoint video of a human in motion , based on a sparse set of views , is extremely challenging , but has many applications . Our work is motivated by a breadth of transformative 3D use cases , including immersive visualization of photographs , virtual clothing and try-on , fitness , as well as AR and VR for improved communication or collaboration . However , so far static scenes and rigid objects have been the primary subject of research . In pursuing realistic novel-view synthesis two schools of thought have been established : 1 ) 3D reconstruction methods aim to recover the geometry of the observed scene as accurately as possible , with novel views generated using classical rendering pipelines [ 30 , 42 ] . 2 ) Image-based rendering techniques [ 7 , 10 , 40 ] and very recently neural radiance fields [ 29 ] primarily aim at image production quality without explicitly constructing an accurate 3D geometric model . While these techniques explicitly or implicitly reconstruct the scene geometry sometimes , this is however not guaranteed to accurately resemble the true scene geometry . We argue that novel-view rendering and reconstruction are two sides of the same coin , and that reliable viewpoint generalization , especially given relatively few input views , would require good quality for both . To this end , we propose a unified model in order to support both robust reconstruction and photo-realistic rendering . Dynamic scenes , especially those capturing a human in motion , add considerable complexity to the problem : while static scenes can be observed from many views by a camera moving through the scene , any configuration of a dynamic scene is typically observed only from sparse views . Moreover , the scene geometry and its appearance may change considerably over time . To cope with few views , some methods integrate scene knowledge over time by warping observations into a common reference frame [ 35 , 37 ] . At test time , the information is warped back to the desired state and rendered from a novel view . Extrapolating to unseen motion , however , remains challenging . For scenes capturing people , this means that only poses seen during training can be rendered at test time . For some applications , however , rendering the subject over a broad range of motions , or in novel poses , is desirable . To make generalisation over poses and views possible , we rely on additional problem domain knowledge in the form of a human body model , imGHUM [ 4 ] . imGHUM is an implicit signed distance function ( SDF ) conditioned on generative shape and pose 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . codes learned from a large corpus of dynamic human scans . In this work , imGHUM is used as the common reference frame for a neural radiance field and as a structured prior for robust reconstruction . Additionally , since imGHUM can represent a broad distribution of statistically valid human poses and shapes , we can render the reconstructed subject in novel poses and even with modified body shapes . In summary , our system supports photo-realistic free-view point temporal rendering of a human subject given only sparse camera observations . By conditioning on an implicit human body model , we can render considerably different viewpoints , body poses , and body shapes compared to those observed in training . Our carefully designed losses ensure not only good image quality but also plausible temporal reconstructions that can be used for the free-viewpoint visualisation of human performance capture . 2 Related Work . Human performance capture is the process of reconstructing the dynamic ( 4D ) geometry of a human subject as observed in one or multiple synchronized views . Pioneering methods deform a rigged template mesh against silhouettes observed in multiple views [ 6 , 50 ] , estimated skeleton key-points [ 12 ] , or image correspondences [ 9 ] . Later systems relying on RGB-D video streams [ 5 ] or monocular capture [ 3 , 2 , 17 , 52 , 16 , 15 ] have also been presented . All these methods rely either on a pre-scanned template mesh or on a body model that is deformed to explain the image evidence . Neural representation for view synthesis . With the advent of neural networks , researchers have begun to explore alternative solutions to represent a scene in order to support novel view synthesis and photo-realistic rendering ; we refer the reader to [ 48 ] for a survey . Different scene representation have been explored . Some methods use voxel grids to embed the scene [ 23 , 45 ] . For rendering , the voxel grid is probed by shooting a ray and by linearly interpolating voxel values . These samples are then transformed into color values using a neural network . Others have proposed neural textures [ 49 , 43 ] that can be rendered based on view-dependent effects or texture synthesis networks [ 22 ] to realistically render meshes . In a similar spirit , other methods render point clouds , where each point carries local appearance information [ 26 , 1 ] . Riegler and Koltun [ 40 ] reproject features from nearby views and rely on a SfM reconstruction scaffold for novel view synthesis . With the recent advent of implicit function networks [ 25 , 34 , 8 , 27 ] , such 3D representations have been explored for rendering and view synthesis with great success . In contrast to discrete voxel representations , textures , or point clouds , these methods represent the scene as a continuous function and thus are not bound to a specific image or volume resolution . In the pioneering work of Sitzmann et al . an implicit function produces features displayed using a neural renderer [ 46 ] . Follow up methods focus on 3D geometric reconstruction [ 44 ] , and use 2D supervision [ 31 , 53 ] . Neural Radiance Fields ( NeRFs ) are a recent approach to represent scenes for novel view synthesis . Mildenhall et al . [ 29 ] introduce Neural Radiance Fields resented as fully connected neural networks , where the input is a spatial query point and a viewing direction . The output is a volume density and the emitted radiance at the query location in the direction of the viewer . By ray-tracing using this simple representation , one can generate photo-realistic images from novel views . Despite the excellent quality of results , one drawback is the slow rendering time . To this end , researchers have presented faster versions that e.g . transform the radiance field into more efficient sparse grids [ 18 ] or remove the dependence on viewing-direction during rendering by estimating a spherical harmonic representation of the radiance function [ 54 ] . Others improve fidelity or rendering time by tackling ambiguities in the original formulation [ 56 ] , spatial decomposition into multiple NeRFs [ 39 ] , or combining NeRF with sparse voxel fields [ 21 ] . Initial work on adapting NeRF to dynamic scenes has been presented as well . Park et al . [ 35 ] produce “ Nerfies ” ( NeRF-Selfies ) from videos where subjects carefully move a camera around their head . The scene information is fused by warping query points into a canonical reference frame . Similarly , Pumarola et al . [ 37 ] produce dynamic NeRFs from synthetic animation data . Related to our approach , some methods integrate human body models to fuse information over time . A-NeRF [ 47 ] uses a skeleton to rigidly transform NeRF features to refine estimated 3D poses . A similar approach is followed in NARF [ 32 ] for view synthesis . Most related , Neural Body [ 36 ] attaches learnable features to the vertices of a SMPL body model [ 24 ] . These features are processed using a sparse 3D convolutional network , where the output forms a neural radiance field . In contrast to our approach , the resolution is bounded by the spatial resolution of the 3D convolutional network and no geometric supervision is used . We highlight differences in §5 . 3 Background . Given a collection of images capturing a dynamic scene of a human in motion , observed from a sparse set of calibrated camera views ( in the limit a monocular camera , as we will show ) , we aim to learn both the detailed temporal geometry of the human in motion , and to render the sequence from novel camera views and for different human poses . To this end , our work unifies two main methodologies : 1 ) implicit 3D human representations , and 2 ) volumetric radiance fields . In this section , we provide the relevant background on both representations , which we co-learn in a joint framework . Neural Radiance Fields . A neural radiance field ( NeRF ) [ 29 ] represents a 3D scene as a continuous function of color volume densities . The model consists of a neural network function Fω that maps a 3D spatial point x ∈ R3 and a viewing direction v ∈ R3 to a volume density σ ∈ R+ and a radiance c ( x , v ) ∈ R3 emitted towards the viewer . In practice , NeRF encodes the inputs x and v using a sinusoidal positional encoding γ : R3 → R3+6m that projects a coordinate vector into a high-dimensional space using a set of sine and cosine functions of m increasing frequencies . Given a ray r = o + sv with N samples { x } originating from a camera location o , NeRF integrates radiance values along the ray by means of alpha blending . The pixel/ray color is approximated with numerical quadrature [ 33 ] , C ( r ) = N∑ i=1 α ( xi ) ∏ j < i ( 1− α ( xj ) ) c ( xi , v ) , ( 1 ) α ( xi ) = 1− exp ( −σ ( xi ) δi ) , ( 2 ) where α ( xi ) is the transparency by accumulating transmittance along the ray , and δi = |xi+1 − xi| is the distance between adjacent samples . The NeRF function Fw is fully differentiable and its network parameters w can be optimized using an image reconstruction loss [ 29 ] . An approximate 3D scene geometry SF = { x|σ ( x ) = σh } can be extracted from the trained opacity field via Marching Cubes [ 20 ] at a density threshold σh . Implicit Generative Human Models . Implicit human body surfaces are typically represented as the decision boundary of either binary occupancy classifiers [ 11 , 41 , 28 ] or signed distance functions [ 14 , 4 ] . Specifically , our work builds upon the SOTA statistical implicit human model imGHUM [ 4 ] Hω : ( T−1x , β , θ ) → ( d , s ) that maps a 3D spatial point x , unposed with root joint transformation T ∈ R4×3 , to its signed distance value d ∈ R with respect to a body surface parameterized with body shape β ∈ R16 and articulated pose θ ∈ R118 . In addition to d , imGHUM returns implicit continuous semantics s ∈ R3 of the query point which corresponds to the 3D coordinate of the nearest surface point defined on a canonical surface . We refer to the original paper [ 4 ] for details . Essentially , imGHUM builds a human-centric 4D semantic descriptor ( d , s ) for all spatial points in the neighborhood of the surface . imGHUM is trained with a large collection of human scans of diverse body shapes and poses , sharing the same generative shape and pose latent code with the mesh-based statistical human model GHUM [ 51 ] . The implicit articulated 3D human body SH ( β , θ , T ) = { x|d ( x ) = 0 } is defined by the the zero-isosurface of the signed distance field .

Summary: This paper proposes a method for dynamic human reconstruction and rendering using sparse view videos.  The high-level strategy is to learn both a neural radiance field and an SDF field with mutual constraints in a canonical frame from temporal observations. Moreover, for generating live rendering and surface results, pose and position conditioned SDF offsets are also learned for better live frame surface reconstruction results.  To guarantee robust surface reconstruction, an implicit generative human model was used as the inner layer which not only incorporates strong human shape constraint, but also provides semantic correspondences for accurate warping between live frames and the canonical frame. Another benefit of using an underlying implicit human model is that shape and pose editing can be achieved through interpolating in the parametric body space, which was not demonstrated in previous works.