H-NeRF: Neural Radiance Fields for Rendering and Temporal Reconstruction of Humans in Motion
Year: 2022

Abstract:
We present neural radiance fields for rendering and temporal (4D) reconstruction of humans in motion (H-NeRF), as captured by a sparse set of cameras or even from a monocular video. Our approach combines ideas from neural scene representation, novel-view synthesis, and implicit statistical geometric human representations, coupled using novel loss functions. Instead of learning a radiance field with a uniform occupancy prior, we constrain it by a structured implicit human body model, represented using signed distance functions. This allows us to robustly fuse information from sparse views and generalize well beyond the poses or views observed in training. Moreover, we apply geometric constraints to co-learn the structure of the observed subject – including both body and clothing – and to regularize the radiance field to geometrically plausible solutions. Extensive experiments on multiple datasets demonstrate the robustness and the accuracy of our approach, its generalization capabilities significantly outside a small training set of poses and views, and statistical extrapolation beyond the observed shape.

heading: 1 Introduction
text: Enabling free-viewpoint video of a human in motion, based on a sparse set of views, is extremely challenging, but has many applications. Our work is motivated by a breadth of transformative 3D use cases, including immersive visualization of photographs, virtual clothing and try-on, fitness, as well as AR and VR for improved communication or collaboration. However, so far static scenes and rigid objects have been the primary subject of research. In pursuing realistic novel-view synthesis two schools of thought have been established: 1) 3D reconstruction methods aim to recover the geometry of the observed scene as accurately as possible, with novel views generated using classical rendering pipelines [30, 42]. 2) Image-based rendering techniques [7, 10, 40] and very recently neural radiance fields [29] primarily aim at image production quality without explicitly constructing an accurate 3D geometric model. While these techniques explicitly or implicitly reconstruct the scene geometry sometimes, this is however not guaranteed to accurately resemble the true scene geometry. We argue that novel-view rendering and reconstruction are two sides of the same coin, and that reliable viewpoint generalization, especially given relatively few input views, would require good quality for both. To this end, we propose a unified model in order to support both robust reconstruction and photo-realistic rendering. Dynamic scenes, especially those capturing a human in motion, add considerable complexity to the problem: while static scenes can be observed from many views by a camera moving through the scene, any configuration of a dynamic scene is typically observed only from sparse views. Moreover, the scene geometry and its appearance may change considerably over time. To cope with few views, some methods integrate scene knowledge over time by warping observations into a common reference frame [35, 37]. At test time, the information is warped back to the desired state and rendered from a novel view. Extrapolating to unseen motion, however, remains challenging. For scenes capturing people, this means that only poses seen during training can be rendered at test time. For some applications, however, rendering the subject over a broad range of motions, or in novel poses, is desirable. To make generalisation over poses and views possible, we rely on additional problem domain knowledge in the form of a human body model, imGHUM [4]. imGHUM is an implicit signed distance function (SDF) conditioned on generative shape and pose
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
codes learned from a large corpus of dynamic human scans. In this work, imGHUM is used as the common reference frame for a neural radiance field and as a structured prior for robust reconstruction. Additionally, since imGHUM can represent a broad distribution of statistically valid human poses and shapes, we can render the reconstructed subject in novel poses and even with modified body shapes. In summary, our system supports photo-realistic free-view point temporal rendering of a human subject given only sparse camera observations. By conditioning on an implicit human body model, we can render considerably different viewpoints, body poses, and body shapes compared to those observed in training. Our carefully designed losses ensure not only good image quality but also plausible temporal reconstructions that can be used for the free-viewpoint visualisation of human performance capture.

heading: 2 Related Work
text: Human performance capture is the process of reconstructing the dynamic (4D) geometry of a human subject as observed in one or multiple synchronized views. Pioneering methods deform a rigged template mesh against silhouettes observed in multiple views [6, 50], estimated skeleton key-points [12], or image correspondences [9]. Later systems relying on RGB-D video streams [5] or monocular capture [3, 2, 17, 52, 16, 15] have also been presented. All these methods rely either on a pre-scanned template mesh or on a body model that is deformed to explain the image evidence.
Neural representation for view synthesis. With the advent of neural networks, researchers have begun to explore alternative solutions to represent a scene in order to support novel view synthesis and photo-realistic rendering; we refer the reader to [48] for a survey. Different scene representation have been explored. Some methods use voxel grids to embed the scene [23, 45]. For rendering, the voxel grid is probed by shooting a ray and by linearly interpolating voxel values. These samples are then transformed into color values using a neural network. Others have proposed neural textures [49, 43] that can be rendered based on view-dependent effects or texture synthesis networks [22] to realistically render meshes. In a similar spirit, other methods render point clouds, where each point carries local appearance information [26, 1]. Riegler and Koltun [40] reproject features from nearby views and rely on a SfM reconstruction scaffold for novel view synthesis. With the recent advent of implicit function networks [25, 34, 8, 27], such 3D representations have been explored for rendering and view synthesis with great success. In contrast to discrete voxel representations, textures, or point clouds, these methods represent the scene as a continuous function and thus are not bound to a specific image or volume resolution. In the pioneering work of Sitzmann et al. an implicit function produces features displayed using a neural renderer [46]. Follow up methods focus on 3D geometric reconstruction [44], and use 2D supervision [31, 53].
Neural Radiance Fields (NeRFs) are a recent approach to represent scenes for novel view synthesis. Mildenhall et al. [29] introduce Neural Radiance Fields resented as fully connected neural networks,
where the input is a spatial query point and a viewing direction. The output is a volume density and the emitted radiance at the query location in the direction of the viewer. By ray-tracing using this simple representation, one can generate photo-realistic images from novel views. Despite the excellent quality of results, one drawback is the slow rendering time. To this end, researchers have presented faster versions that e.g. transform the radiance field into more efficient sparse grids [18] or remove the dependence on viewing-direction during rendering by estimating a spherical harmonic representation of the radiance function [54]. Others improve fidelity or rendering time by tackling ambiguities in the original formulation [56], spatial decomposition into multiple NeRFs [39], or combining NeRF with sparse voxel fields [21]. Initial work on adapting NeRF to dynamic scenes has been presented as well. Park et al. [35] produce “Nerfies” (NeRF-Selfies) from videos where subjects carefully move a camera around their head. The scene information is fused by warping query points into a canonical reference frame. Similarly, Pumarola et al. [37] produce dynamic NeRFs from synthetic animation data. Related to our approach, some methods integrate human body models to fuse information over time. A-NeRF [47] uses a skeleton to rigidly transform NeRF features to refine estimated 3D poses. A similar approach is followed in NARF [32] for view synthesis. Most related, Neural Body [36] attaches learnable features to the vertices of a SMPL body model [24]. These features are processed using a sparse 3D convolutional network, where the output forms a neural radiance field. In contrast to our approach, the resolution is bounded by the spatial resolution of the 3D convolutional network and no geometric supervision is used. We highlight differences in §5.

heading: 3 Background
text: Given a collection of images capturing a dynamic scene of a human in motion, observed from a sparse set of calibrated camera views (in the limit a monocular camera, as we will show), we aim to learn both the detailed temporal geometry of the human in motion, and to render the sequence from novel camera views and for different human poses. To this end, our work unifies two main methodologies: 1) implicit 3D human representations, and 2) volumetric radiance fields. In this section, we provide the relevant background on both representations, which we co-learn in a joint framework.
Neural Radiance Fields. A neural radiance field (NeRF) [29] represents a 3D scene as a continuous function of color volume densities. The model consists of a neural network function Fω that maps a 3D spatial point x ∈ R3 and a viewing direction v ∈ R3 to a volume density σ ∈ R+ and a radiance c(x,v) ∈ R3 emitted towards the viewer. In practice, NeRF encodes the inputs x and v using a sinusoidal positional encoding γ : R3 → R3+6m that projects a coordinate vector into a high-dimensional space using a set of sine and cosine functions of m increasing frequencies. Given a ray r = o + sv with N samples {x} originating from a camera location o, NeRF integrates radiance values along the ray by means of alpha blending. The pixel/ray color is approximated with numerical quadrature [33],
C(r) = N∑ i=1 α(xi) ∏ j<i (1− α(xj))c(xi,v), (1)
α(xi) = 1− exp ( −σ(xi)δi ) , (2)
where α(xi) is the transparency by accumulating transmittance along the ray, and δi = |xi+1 − xi| is the distance between adjacent samples. The NeRF function Fw is fully differentiable and its network parameters w can be optimized using an image reconstruction loss [29]. An approximate 3D scene geometry SF = {x|σ(x) = σh} can be extracted from the trained opacity field via Marching Cubes [20] at a density threshold σh.
Implicit Generative Human Models. Implicit human body surfaces are typically represented as the decision boundary of either binary occupancy classifiers [11, 41, 28] or signed distance functions [14, 4]. Specifically, our work builds upon the SOTA statistical implicit human model imGHUM [4] Hω : (T−1x,β,θ) → (d, s) that maps a 3D spatial point x, unposed with root joint transformation T ∈ R4×3, to its signed distance value d ∈ R with respect to a body surface parameterized with body shape β ∈ R16 and articulated pose θ ∈ R118. In addition to d, imGHUM returns implicit continuous semantics s ∈ R3 of the query point which corresponds to the 3D coordinate of the nearest surface point defined on a canonical surface. We refer to the original paper [4] for details. Essentially, imGHUM builds a human-centric 4D semantic descriptor (d, s) for all spatial points in the neighborhood of the surface. imGHUM is trained with a large collection of human scans of diverse body shapes and poses, sharing the same generative shape and pose latent
code with the mesh-based statistical human model GHUM [51]. The implicit articulated 3D human body SH(β,θ,T) = {x|d(x) = 0} is defined by the the zero-isosurface of the signed distance field.

heading: 4 Method
text: We present the details of our main contribution, H-NeRF, a novel neural network (fig. 1) that exploits the power of volumetric radiance fields to learn complex human structure and appearance, by relying on statistical implicit human pose and shape signed distance functions for accurate geometric reconstruction. Further, H-NeRF relies on imGHUM, an implicit model of articulated human pose and shape, as a rich geometric prior, to integrate scene information over time, and to represent human articulation. Co-learning both a radiance field and a signed distance function of scene geometry consistently in a unified framework, enables accurate 3D geometric reconstruction and volume rendering of a dynamic human in motion, through 1) consistent integration of image observations over time, and 2) good generalization capability for novel viewpoints, human poses, and for statistically extrapolated shapes, given only very few training poses and camera views.
Given a video of a human in motion, observed by a sparse set of calibrated cameras, our objective is to generate free-viewpoint video of the observed person and to reconstruct the underlying 4D geometry. The set of input images, with a resolution of w × h pixels, is denoted as {Ict ∈ Rw×h×3|c = 1, . . . , Nc, t = 1, . . . , Nt}, where c is the camera index, Nc is the number of cameras, t is the frame index, and Nt is the number of frames. For each image, we apply [13] to obtain the binary foreground human mask Mct ∈ Rw×h. In addition, we obtain a temporally consistent imGHUM latent shape and pose codes, β and (θt,Tt), respectively, at each frame index t, by optimizing an imGHUMequivalent parametric replica under multi-view keypoint and body segmentation losses [55]. We refer to our Sup. Mat. for details on the imGHUM fitting process.
In the sequel we first explain our adaptations to NeRF for the static case. imGHUM is used as a prior in order to bias the reconstruction of the observed scene towards a more accurate human geometry. We continue by explaining the additional methodological innovation needed in the dynamic case. Hereby, imGHUM provides spatial-temporal correspondences and is used as the common reference frame to fuse information across different views and time instances.

heading: 4.1 Static Semantic Human NeRF
text: We first formulate H-NeRF for a human capture at a single moment in time (Nt = 1). From multi-view image observations, we co-learn a radiance field Fω : x,v→ (c, σ) for free-viewpoint rendering, and a signed distance function Ĥω : x→ d̂ for 3D geometric reconstruction. We use Ĥω for the dressed subject in order to distinguish it from the body’s imGHUM SDF Hω .
Like NeRF, we formulate an L1 image reconstruction loss to optimize Fw as Lrec = ∑ r∈R ‖C̄(r)−C(r)‖1, (3)
whereR denotes the batched set of all pixels/rays, and C̄(r),C(r) are observed and rendered pixel colors, cf. (1), respectively. However, under sparse training camera views, the volumetric radiance field is not well regularized, leading to poor generalization to novel viewpoints, cf. fig. 2. Specifically, we observe that the model encounters difficulties in correctly representing the scene and in separating the person from the background. The model fails to learn a semantically meaningful opacity field, i.e. α = 0 in free space, and 1 if occupied.
Coarse Scene Structuring. Using imGHUM fits for all training images, we can spatially locate the person in 3D. We define a 3D bounding box B ∈ [SH − ,SH + ] around the detected person, where SH ,SH are the minimal and maximal coordinates of the human body surface S and is a spatial margin reserved for geometry not modeled by imGHUM. All radiance points associated to the rendering of the person should reside inside the bounding box, leading to a 3D segmentation loss
C̃(r) = N∑ i=1 b(xi)α(xi) ∏ j<i (1− b(xj)α(xj))c(xi,v), (4)
Lmask = ∑ r∈R ∥∥M(r)(C̄(r)− C̃(r))∥∥ 1 , (5)
where b(xi) is 0 if outside of B and 1 otherwise, and M(r) the image mask. We rely on the mask, so the loss is only applied to image observations from the subject, and not the remaining scene geometry.
Unifying SDF with NeRF. After structuring the scene coarsely, we now couple the estimated radiance field with an implicit signed distance-based 3D reconstruction, in order to further regularize the opacity distribution. A signed distance function associated to the detected person in the image naturally comes with a 3D classifier for all spatial points, where d̂(xi) > 0 means xi is in free space, whereas xi lies within the subject when d̂(xi) <= 0. We rely on this insight in order to co-learn a SDF of the performer and constrain the radiance field. To this end, we introduce a pseudo alpha value α̇(xi) = φ(γd̂(xi)) where φ is a Sigmoid activation function and γ controls the sharpness of the boundary. To refine the NeRF opacity semantics, especially for the volume in the neighborhood of the human surface B, we formulate two losses
Ĉ(r) = N∑ i=1 α̂(xi) ∏ j<i (1− α̂(xj))c(xi,v), α̂(xi) = b(xi)α̇(xi) + (1− b(xi))α(xi), (6)
Lblend = ∑ r∈R ( ‖M(r) ( C̄(r)− Ĉ(r) ) ‖1 + η‖(1−M(r)) ( C̄(r)− Ĉ(r) ) ‖1 ) , (7)
Lgeom(r) = N∑ i=1 b(xi)BCE ( φ ( λ(σh − σ(xi)) ) , α̇(xi) ) , (8)
where Ĉ(r) is the rendered pixel color with blended alpha values α̂ replacing NeRF alpha values α with SDF-based pseudo alpha α̇ for all ray points inside the bounding box B. The first term in Lblend requires the rendered pixel color for all the intersecting rays within the human mask to come from the surface, whereas the second term assumes that all background color is formed from ray samples outside of B. We set η = 1 when no other geometry than the person is inside B and tune η down if the assumption is violated (e.g. person standing on a floor). The term Lgeom uses the binary cross entropy loss to couple the NeRF surface boundary with the zero-isosurface of the signed distance function describing the subject. While the coupling terms given by (7) and (8) act as strong priors for the opacity distribution, during test time, we still rely on volumetric radiance rendering with learned NeRF alpha values α to support transparency effects and complex geometry for structures like hair.
Image-based SDF learning. Learning the signed distance field Ĥω from scratch using a sparse set of training images is still challenging. The model often fails to reconstruct reasonable human geometry, even when using coupling losses (7) and (8). We therefore leverage imGHUM as an inner layer for the target human reconstruction and combine it with a light-weight residual SDF network ∆Hω : x→ ∆d. The residual SDF models surface details, including hair and clothing, that are not represented by imGHUM. The final signed distance for xi becomes d̂(xi) = d(xi|β,θ,T)+∆d(xi). Given the training images, we learn the personalized residual SDF using our coupling losses, given by (7) and (8), and additionally apply geometric regularization
Lseg = ∑ r∈R BCE(M(r), d̂min(r)), (9)
Leik(r) = N∑ i=1 b(xi)(‖∇xi d̂(xi)‖2 − 1)2, Lreg = ‖ψ(∆d)‖1, (10)
where d̂min(r) denotes the minimal signed distance for all sampled ray points and ψ is the ReLU activation function. The term Lseg ensures that if a pixel is inside the human segmentation mask, there should be at least one intersection between the ray and the 3D human surface and therefore d̂min(r) should be non-positive. Otherwise, all ray samples should have positive signed distances. Using Leik, we enforce the composite SDF d̂ to be approximately a signed distance function, i.e. incorporating Eikonal regularization [14]. The term Lreg regularizes the residual distance ∆d to be non-positive. This is because imGHUM should reside within the geometry and personalized geometric details given by ∆d should be modeled on top of the skin surface.

heading: 4.2 Dynamic Semantic Human NeRF
text: We now extend the framework to model dynamic human motion. To implicitly model scenes where the human moves, we learn a consistent, continuous function Gω : (c, α, d̂|z) → (c′, α′, d̂′|z′) for both the geometry and the appearance variations across frames. For generalization to novel poses or shapes, Gω should be conditioned on a semantically meaningful latent code z that can be interpolated and should ideally have good extrapolation properties. To integrate scene information over time in a single radiance field, we follow the approaches in [35, 37] and warp observations into a canonical reference frame. Given our constraints to Gω, imGHUM, conditioned on its semantic shape and pose latent code (β,θ,T), naturally provides spatial correspondences across frames. Given two spatial points x and x′ in the space of two human instances (β,θ,T) and (β′,θ′,T′) respectively, we decide these are in correspondence if they have the same semantics and signed distances (s, d|β,θ,T) = (s′, d′|β′,θ′,T′). Essentially, imGHUM assigns a 4D point descriptor (s, d) ∈ R4 to any spatial point and deforms the volume continuously with respect to the parameterized articulated human body surface. Specifically, we apply Hω : (T−1x,β,θ)→ (d, s) to map a spatial point x to a canonical point descriptor (s, d). We then modify the input to the NeRF function with the point descriptor as Fw : (s, d)→ (c, σ), and similarly for the residual SDF function as ∆Hω : (s, d)→ ∆d. imGHUM is pre-trained using a large corpus of 3D human scans, thus we keep it fixed in our network and directly use this prior to integrate structured geometric and appearance information across training frames. Given that the background scene is invariant to the human motion, we only apply the imGHUM warping function to spatial points inside the bounding box B, thus largely improving computation time and lowering memory usage.
Pose-Dependent Geometry and Appearance. Training a single canonical NeRF and a residual SDF from a sequence integrates image observations into a consistent implicit geometric and volumetric representation of the radiance. However, we observe that fine-grained geometric and appearance variations caused by human motion are missing in the canonical NeRF and the residual SDF. To model such variations in geometry (clothing deformation) and appearance (lighting, self-shadows), we condition the NeRF function Fω – the volume density σ (geometry) and color value c (appearance) – as well as the residual SDF ∆Hω (geometry), on the body pose code θ. In addition, we also notice that the relative position and rotation of the person with respect to the scene largely affects appearance (due to illumination effects), but should not affect the geometry. Based on this observation, we additionally condition the NeRF color on the person’s root transformation as c(x|θ,T). H-NeRF Articulation. Differently from prior work [35, 37], our H-NeRF modules are conditioned on semantic human latent codes (β,θ,T), leading to an SDF and radiance field that can be interpolated. Moreover, due to imGHUM’s capability to accurately model geometric volume deformation for diverse human poses and shapes, H-NeRF enjoys strong generalisation to novel poses and even body shapes. Concretely, during inference, one can simply assign a different set of (β,θ) to transfer the learned geometry and appearance to an unseen pose and shape configuration. For better generalization, we add Gaussian noise to the input code (θ,T), preventing network overfitting to pose-dependent deformation and appearance effects experienced during training. When only a sparse set of camera views is available, We apply the same technique to NeRF, conditioned on viewing direction v.
Fine-tuning imGHUM Pose and Shape. We observe that imGHUM fitting can be a source of error when learning Fω and ∆Hω from dynamic image sequences. Instead of keeping parameters fixed, we further improve the fit during training by fine tuning a time-consistent shape correction ∆β and per-frame pose correction ∆θ(t) with
Lfit = ∑ r∈R BCE(M(r), dmin(r|β + ∆β,θ + ∆θ(t))), Linc = ‖∆β‖2 + 1 Nt Nt∑ t=1 ‖∆θ‖2, (11)
where Lfit aligns the imGHUM fit with the human foreground segmentation and Linc regularizes the incremental corrections.

heading: 5 Experiments
text: We quantitatively and qualitatively evaluate H-NeRF through ablations and by comparing with other methods. Please refer to our Sup. Mat. for additional results and ablation experiments. We first detail our model architecture, as well as the datasets and evaluation metrics used.
Architecture and Training. We use the same network architecture for all of our experiments, where the trainable modules Fω and ∆Hω consist of eight 256-dimensional and six 128-dimensional MLPs
respectively, with a skip connection to the middle layer and with Swish activation [38]. As in the original NeRF [29], we use 256 coarse and fine-level ray samples, for each of which we use 8- and 1-dimensional positional encoding for Fω and ∆Hω, respectively. We train the network using the Adam optimizer with a learning rate of 0.001 exponentially decayed by a factor 0.1 until the maximum number of iterations is reached (10k iteration of 4k ray batch size). We apply Gaussian noise with σ = 0.1 to the NeRF conditioning code (θ,T,v).
Dataset and Metrics. We provide qualitative and quantitative evaluation on four different datasets: RenderPeople (8 sequences), GHS3D [51] (14 sequences), PeopleSnapshot [3] (7 sequences) and Human3.6M [19] (5 sequences). The first two are synthetic, rendering animated (RenderPeople) or 4D human scans (GHS3D) from four orthogonal cameras, where ground-truth geometry paired with images is available. We evaluate both the image and the 3D geometry reconstruction quality from two novel cameras, and qualitatively demonstrate generalization to shapes and poses not in the training set. The remaining datasets are real captured videos (PeopleSnapshot: monocular, Human3.6M: four cameras) without paired 3D geometry, where we only evaluate the rendered images. For image metrics, we adopt peak signal-to-noise ratio (PSNR ↑), structural similarity index (SSIM ↑) and learned perceptual image patch similarity (LPIPS ↓) [57]. For evaluation, we render the NeRF field inside the human bounding box B and compare to the ground-truth segmented image within a region of interest around the person. To evaluate geometric reconstruction quality, we report bi-directional Chamfer (Ch) L2 distance, Normal Consistency (NC) and Volumetric Intersection over Union (IoU), evaluated on the mesh produced by applying Marching Cubes on the SDF, with a resolution of 2563.
Static Human Reconstruction under Sparse Viewpoints. H-NeRF learns a geometrically regularized volumetric radiance field constrained by an imGHUM-based SDF, which significantly improves robustness under sparse training camera views. In fig. 2, we show reconstruction of a static RenderPeople and GHS3D scan, respectively, using the original NeRF [29], the state-of-the-art multi-view reconstruction approach IDR [53], and H-NeRF, for increasing number of cameras. Both novel view image quality and geometric accuracy improve for all methods as the number of training views increases. However, in contrast to competitors, H-NeRF produces good quality output even under sparse training views (2-8), demonstrating the generalization capability of a co-training approach.
Dynamic Human Reconstruction and Rendering. In the following, we evaluate H-NeRF’s capabilities to reconstruct and render dynamic scenes (fig. 5). We compare H-NeRF against NeuralBody [36]
(tab. 1), the current state-of-the-art for novel view synthesis of humans in motion. For fair comparisons, we have trained NeuralBody with the same data as H-NeRF and use GHUM meshes (instead of SMPL). GHUM meshes with the same pose and shape representation as in our model are used for NeuralBody’s structured latent codes. We further compare with Nerfies [35]. However, Nerfies has difficulties with the high range of motion in our test sequences and fails for most of them. We consider such results less meaningful and we report them only in the Sup. Mat. for completeness.
H-NeRF overperforms NeuralBody across datasets and metrics, especially for geometric reconstruction. One possible explanation for NeuralBody’s lower performance on view synthesis, is that NeuralBody is not conditioned on the root transformation. Thus, global lighting effects are handled less well. Furthermore, pose depended effects that go beyond body articulation are only indirectly modeled through relative distances of input mesh vertices. Finally, NeuralBody conditions on the frame index, which is set to zero for novel poses. On the PeopleSnapshot dataset where no view-dependent and only subtle pose-dependent effects are present, these limitations are not entirely apparent. In contrast, H-NeRF pays special attention to modeling both detailed geometry and appearance, and its explicit conditioning on pose and the root transformation captures pose-dependent geometric and appearance effects. This strategy pays off, especially in more complex scenarios.
Frame Number Ablation. We have illustrated H-NeRF’s rendering and reconstruction capabilities for static and dynamic sequences. Next, we evaluate the necessary number of different poses or time-frames H-NeRF needs during training for good pose generalisation. We again compare against NeuralBody. To this end, we have trained both methods with an increasing number of images, uniformly distributed over the full sequence. Results are shown in fig. 3. As expected, both methods perform better when trained with more data. However, H-NeRF performs overall better and more importantly, the quality degrades less in the sparse training regime, especially for geometric reconstruction. Our results support H-NeRF’s robustness to sparse training data and its capability to reconstruct accurate geometry. In practical terms, H-NeRF can be robustly trained with as little as 10 temporal frames per camera (in a four camera set-up), resulting in a performance capture effort of only a few minutes.
Qualitative Results for Pose Generalisation and Shape Extrapolation. Finally, we illustrate HNeRF’s rendering and reconstruction accuracy qualitatively (fig. 4). Consistently with the reported metrics, our synthesized novel poses appear sharper, more detailed, and contain less noise than the current state-of-the-art. The estimated geometry is complete, smooth, and contains much of the detail present in the original scan, e.g. the clothing folds on the back of the person on the left, the first row.
In contrast, geometries produced by NeuralBody are noisier and sometimes incomplete. To further demonstrate the versatility of our approach, in fig. 5 we show examples of synthesized images and reconstructed geometry from novel viewpoints, in novel poses, and for modified body shape.
Limitations and Broader Impact. In the current setup, we assume full-body views of a single person in every-day clothing. Inconsistency with these assumptions, e.g. by placing another person in the scene and occluding the performer, would break the method although models of partial views, or representation estimates of multiple people could be used for generality. Furthermore, our method exhibits some sensitivity to the estimated body shapes and poses, as well as the quality of image segmentation. Although the process of learning to render could absorb certain inaccuracies, as pose, shape or segmentation are increasingly degraded, the method would fail eventually.
Our method paves the way towards immersive AR and VR applications. In contrast, our approach is not targeted, or particularly useful for applications like visual surveillance or person identification as a particular set-up and a cooperating subject is needed. We do not build an audio-visual model that would be necessary for deep fakes.

heading: 6 Conclusions
text: We have presented novel neural radiance field models (H-NeRF) for the photo-realistic rendering and the temporal reconstruction of humans in motion. Our objective is to extend the scope of the previous state of the art, focused on static scenes observed by a large number of cameras, by building dynamic models, which based on a sparse set of views, can generalize well to novel camera views, human body poses, and extrapolate over human body shapes. Our key contribution is in developing a new model with associated multiple losses, in order to specialize and constrain a generic NeRF formulation by using a compatible implicit statistical 3D human pose and shape model, represented using signed distance functions. Our implicit geometric formulation captures not just the statistical regularities of the human body, but also hair and clothing represented as an implicit residual network. Our model is trained end-to-end based on several novel losses, and achieves good results for both 3D reconstruction and photorealistic rendering. Training the body model in an end-to-end rendering framework carries the promise to learn complex implicit skinning functions based on images only, a performance previously possible only in the realm of human capture using complex and expensive 3D body scanners, in the laboratory. We illustrate the favorable capabilities of H-NeRF through extensive experimentation using several datasets and against other state of the art techniques.

Review summaries:
This paper presents a new NeRF based method for rendering and reconstruction of humans observed from sparse cameras. The main contribution is combining volumetric radiance fields and an implicit SDF for the tasks of novel view synthesis and geometric reconstruction for humans. To achieve this goal, the method utilizes a human prior, imGHUM, which was pretrained on thousands of 3D scans. 

The authors introduce H-NeRF, a geometry-aware neural radiance field for human performance rendering.  The core ideas are to (1) conditioned the neural radiance field (NeRF) on a pre-trained implicit geometric human representation that captures a wide variety of body poses and shapes, and (2) co-learn a signed distance function (SDF) that describes the surface geometry of the NeRF for rendering. Experimental results show that, compared to other baselines, H-NeRF achieves better surface reconstruction and novel view and pose synthesis quality, and are able to extrapolate to new body shape volumes.  

The authors propose a model that combines a parameteric human shape model with NeRF and a residual signed distance function to enable novel-view synthesis of humans in motion, assuming full observation of the subject (i.e., no prior-based completion).  The core contributions is the combination of NeRF with a parametric human body model, which enables the learning of the NeRF in a canonical coordinate frame, as well as regularization of the learned geometry. Several additional tweaks improve the model further, such as leveraging auxiliarly losses via a instance segmentation mask, or feeding the estimated body pose to the NeRF as additional input, enabling dynamic (due to being pose-specific) fine detail.

This paper proposes a method for dynamic human reconstruction and rendering using sparse view videos.  The high-level strategy is to learn both a neural radiance field and an SDF field with mutual constraints in a canonical frame from temporal observations. Moreover, for generating live rendering and surface results, pose and position conditioned SDF offsets are also learned for better live frame surface reconstruction results.  To guarantee robust surface reconstruction, an implicit generative human model was used as the inner layer which not only incorporates strong human shape constraint, but also provides semantic correspondences for accurate warping between live frames and the canonical frame. Another benefit of using an underlying implicit human model is that shape and pose editing can be achieved through interpolating in the parametric body space, which was not demonstrated in previous works.