H-NeRF: Neural Radiance Fields for Rendering and Temporal Reconstruction of Humans in Motion
Authors: Hongyi Xu
Thiemo Alldieck
Cristian Sminchisescu
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Neural Radiance Field
SDF
Human Reconstruction
Dynamics

Abstract: We present neural radiance fields for rendering and temporal (4D) reconstruction of humans in motion (H-NeRF), as captured by a sparse set of cameras or even from a monocular video. Our approach combines ideas from neural scene representation, novel-view synthesis, and implicit statistical geometric human representations, coupled using novel loss functions. Instead of learning a radiance field with a uniform occupancy prior, we constrain it by a structured implicit human body model, represented using signed distance functions. This allows us to robustly fuse information from sparse views and generalize well beyond the poses or views observed in training. Moreover, we apply geometric constraints to co-learn the structure of the observed subject -- including both body and clothing -- and to regularize the radiance field to geometrically plausible solutions. Extensive experiments on multiple datasets demonstrate the robustness and the accuracy of our approach, its generalization capabilities significantly outside a small training set of poses and views, and statistical extrapolation beyond the observed shape.

H-NeRF: Neural Radiance Fields for Rendering and Temporal Reconstruction of Humans in Motion
Hongyi Xu Google Research hongyixu.@google com
Thiemo Alldieck Google Research al slldieck@googge.coom
Cristian Sminchisescu Google Research smi wesinccissscuugooglloo
Abstract
We present neural radiance fields for rendering and temporal (4D) reconstruction of humans in motion (H-NeRF), captured by sparse of cameras even from monocular video. Our approach combines ideas from neural scene representation, novel-view synthesis, and implicit statistical geometric human representations, coupled using novel loss functions. Instead of learning a radiance field with a uniform occupancy prior, we constrain it by a structured implicit human body model, represented using signed distance functions. This allows us to robustly fuse information from sparse views and generalize well beyond the poses or views observed in training. Moreover, we apply geometric constraints to co-learn the structure of the observed subject - including both body and clothing and to regularize the radiance field to geometrically plausible solutions. Extensive experiments on multiple datasets demonstrate the robustness and the accuracy of our approach, its generalization capabilities significantly outside small training set of poses and views, and statistical extrapolation beyond the observed shape.
Introduction
Enabling free-viewpoint video of a human in motion, based on sparse set of views, is extremely challenging, but has many applications. Our work is motivated by breadth of transformative 3DD use cases, including immersive visualization of photographs, virtual clothing and try-on, fitmess, as well as AR and VR for improved communication or collaboration. However, so far static scenes and rigid objects have been the primary subject of research. In pursuing realistic novel-view synthesis two schools of thought have been established: 1) 3D reconstruction methods aim recover the geometry of the observed scene as accurately as possible, with novel views generated using classical rendering pipelines 30 42 2) Image- based rendering techniques 7 10 40 and very recently neural radiance fields (29 primarily aim at image production quality without explicitly constructing an accurate 3D geometric model. While these techniques explicitly or implicitly reconstruct the scene geometry sometimes, this is however not guaranteed to accurately resemble the true scene geometry. We argue that novel-view rendering and reconstruction are two sides of the same coin, and that reliable viewpoint generalization, especially given relatively few input views, would require good quality for both. To this end, we propose a unified model in order to support both robust reconstruction and photo-realistic rendering. Dynamic scenes, especially those capturing human in motion, add considerable complexity to the problem: while static scenes can be observed from many views by camera moving through the scene, any configuration of dynamic scene is typically observed only from sparse views. Moreover, the scene geometry and its appearance may change considerably over time. To cope with few views, some methods integrate scene knowledge
time by warping observations into common reference frame [35] 37]. At test time, the information is warped back to the desired state and rendered from novel view. Extrapolating unseen motion, however, remains challenging. For scenes capturing people, this means that only poses seen during training can be rendered at test time. For some applications, however, rendering the subject over broad range of motions, or in novel poses, is desirable. To make generalisation over poses and views possible, we rely on additional problem domain knowledge in the form of human body model, imGHUM (4) imGHUM is an implicit signed distance function (SDF) conditioned on generative shape and pose
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

M
MLP
A
Residual SDF
Ne
-(s.
InGHUM
(3, O., Tr)
Figure 1: Overview of H-NeRF. Given a set of images of human performer collected from a sparse set of calibrated cameras, we train geometry-awee neural radiance field by co-learning deformable signed distance field. First, we estimate the body shape 3 and track the articulated pose 0, T using the implicit statistical human body model imGHUM (orange). imGHUM encodes all 3D spatial points x across frames with 4D point descriptor (s, d) referencing a canonical frame. Using the foreground mask M. we co-train residual SDF and a NeRF network in that canonical space, in order to integrate all image observations into consistent implicit 3D geometry Ad and its view-depended (v) radiance representation (c, o). The trainable residual SDF and NeRF networks (in blue) conditioned on the body pose and the root transformation T to model pose-dependent geometric and appearance variations. Our framework supports both accurate 3D geometric reconstruction and free-viewpoint rendering, and generalizes well to novel views, shapes, and poses.
codes learned from a large corpus of dynamic human scans. In this work, imGHUM is used as the common reference frame for a neural radiance field and as structured prior for robust reconstruction. Additionally, since imGHUM and shapes, we can render the reconstructed subject in novel poses and even with modified body shapes. In summary, our system supports photo-realistic free-view point temporal rendering of human subject given only sparse camera observations. By conditioning on an implicit human body model, we can render considerably different viewpoints, body poses, and body shapes compared to those observed in training. Our carefully designed losses ensure not only good image quality but also plausible temporal reconstructions that be used for the free- viewvvit visualisation of human performance capture.
represent broad distribution of statistically valid human poses
2 Related Work
Human performance capture is the process of reconstructing the dynamic (4D) geometry of a human subject as observed in one or multiple synchronized views. Pioneering methods deform a rigged template mesh against silhouettes observed in multiple views 50), estimated skeleton key-points [12], or image correspondences [9. Later systems relying on RGB-D video streams (5) or monocular capture 3 2 17
52
16, 15] have also been presented. All these methods rely either on a body model that is deformed to cxplain the image evidence.
pre-scanned template mesh
Neural representation for view synthesis. With the advent of neural networks, rescarchers have begun to explore alternative solutions to represent a scene in order to support novel view synthesis and photo-realistic rendering; we refer the reader to 48 for survey. Different scene representation have been explored. Some methods use voxel grids to embed the scene (33 45). For rendering, the voxel grid is probed by shooting ray and by linearly interpolating voxel values. These samples are then transformed into color values using neural network. Others have proposed neural textures that can rendered based on view-dependent effects or texture synthesis networks [22] to realistically render meshes. In a similar spirit, other methods render point clouds, where each point carries local appearance information [26. M. Riegler and Koltun [40] reproject features from nearby views and rely a SIM reconstruction scaffold for novel view synthesis. With the recent advent implicit function networks [25] 34 27]. such 3D representations have been explored for rendering and view synthesis with great success. In contrast to discrete voxel representations, textures, or point clouds, these methods represent the scene as a continuous function and thus are not bound to specific image or volume resolution. In the pioneering work of Sitzmann et al. an implicit function produces features displayed using a neural renderer [46]. Follow up methods focus on 3D geometric reconstruction 44] and use 2D supervision 33 53
49 43
Neural Radiance Fields (NeRFs) are recent approach to represent scenes for novel view synthesis. Mildenhall et al. [29] introduce Neural Radiance Fields resented as fully connected neural networks,

---

where the input is a spatial query point and a viewing direction. The output is volume density and the emitted radiance at the query location in the direction of the viewer. By ray- ray-tracing using this simple representation, one can generate photo-realistic images from novel views. Despite the excellent quality of results, one drawback the slow rendering time. To this end, researchers have presented faster versions that e.g. transform the radiance field into more efficient sparse grids [8] or remove the dependence on viewing-direction during rendering by estimating a spherical harmonic representation of the radiance function (54). Others improve fidelity or rendering time by tackling ambiguities in the original formulation [56]. spatial decomposition into multiple NeRFs [39], or
combining NeRF with sparse voxel fields [21]. Initial work on adapting NeRF to dynamic scenes has been presented as well. Park et al. [35 produce "Nerfies" (NeRF-Selfies) from videos where subjects carefully move camera around their head. The scene information is fused by warping query points into a canonical reference frame. Similarly, Pumarola et al. [37 produce dynamic NeRFs from synthetic animation data. Related to our approach, some methods integrate human body models to fuse information over time. A-NeRF [47 uses a skeleton to rigidly transform NeRF features to refine estimated 3D poses. A similar approach is followed in NARF [32] for view synthesis. Most related, Neural Body [36] attaches learnable features to the vertices of a SMPL body model [24]. These features are processed using sparse 3D convolutional network, where the output forms neural radiance field. In contrast to our approach, the resolution is bounded by the spatial resolution of the 3D convolutional network and no geometric supervision is used. We highlight differences in $
3 Background
Given collection of images capturing dynamic scene of a human in motion, observed from a sparse set of calibrated camera views (in the limit monocular camera, as we will show), we aim to learn both the detailed temporal geometry of the human in motion, and to render the sequence from novel camera views and for different human poses. To this end, our work unifies two main methodologies: 1) implicit 3D human representations, and 2) volumetric radiance fields. In this section, we provide the relevant background on both representations, which we co-learn in a joint framework.
Neural Radiance Fields. A neural radiance field (NeRF) (29 represents a 3D scene as a continuous function of color volume densities. The model consists of neural network function F. that maps a 3D spatial point x E R and a viewing direction v E R to a volume density a E R and a radiance c(x, v) E R emitted towards the viewer. In practice, NeRF encodes the inputs x and using a sinusoidal positional encoding 7 R + R R3+6m that projects a coordinate vector into a high-dimensional space using set of sine and cosine functions of m increasing frrqueeniess. Given a ray r = o + sv with N samples {x} originating from camera location o, NeRF integrates radiance values along the ray by means of alpha blending. The pixel/ray color is approximated with numerical quadrature 33
N
C(r)
Σ 1=1
(x))
II (1-a(xx))e(xx;, jzt
(1)
((x)) = 1 e
(2)
where (x;)) the transparency by accumulating transmittance along the ray, and 6, = |x+1 x,
is the distance between adjacent samples. The NeRF function F. is fully differentiable and its network parameters w can be optimized using an image reconstruction loss (29). An approximate 3D scene geometry S = [xlo (x) = ) can be extracted from the trained opacity field via Marching Cubes (00 at a density threshold on-
Implicit Generative Human Models. Implicit human body surfaces are typically represented as the decision boundary of either binary occupancy classifiers [I] 41 28 or signed distance functions 11. 4) Specifically, our work builds upon the SOTA statistical implicit human model imGHUM H1w (T- xx B, 0) + (d) that maps 3D spatial point x, unposed with root joint transformation T E R 4x3 to its signed distance value d E R with respect to a body surface parameterized with body shape B E R6 and articulated pose O R 118 In addition to d, imGHUM returns implicit continuous semantics E R of the query point which corresponds to the 3D coordinate of the nearest surface point defined on a canonical surface. We refer to the original paper 4 for details. Essentially, imGHUM builds a human-centric 4D semantic descriptor (d. s) for all spatial points in the neighborhood of the surface. imGHUM is trained with large collection of human scans of diverse body shapes and poses, sharing the same generative shape and pose latent
3

---

code with the mesh-based statistical human model GHUM [S] The implicit articulated 3D human body (3, 0, T) fx]d(x) = 03 is defined by the the zero-isosurface the signed distance field.
4 Method
We present the details of our main contribution, H-NeRF, a novel neural network (fig. D that exploits the power of volumetric radiance fields to learn complex human structure and appearance, by relying on statistical implicit human pose and shape signed distance functions for accurate geometric reconstruction. Further. H-NeRF relies on imGHUM, an implicit model of articulated human pose and shape, as rich geometric prior, to integrate scene information over time, and to represent human articulation. Co-learning both a radiance field and a signed distance function of scene geometry consistently in a unified framework, enables accurate 3D geometric reconstruction and volume rendering of dynamic human in motion, through 1) consistent integration of image observations over time, and 2) good generalization capability for novel viewpoints, human poses, and for statistically extrapolated shapes, given only very few training poses and camera views.
Given video of human in motion, observed by sparse set of calibrated cameras, our objective is to generate free-view viewpint video of the observed person and to reconstruct the underlying 4D geometry. The set of input images, with a resolution of X h pixels. is denoted as {Iī E Ruxhx3 I...., No. t = I..... Nr where e is the camera index, No is the number of cameras, t is the frame index, and N₂ is the number of frames. For each image, we apply [33 obtain the binary foreground human mask M; E R_xx,_ In addition, we obtain a temporally consistent imGHUM latent shape and pose codes, /3 and (0,, Tr), respectively, at each frame index t, by optimizing an imGHUM- equivalent parametric replica under multi-view keypoint and body segmentation losses [5] to our Sup. Mat. for details on the imGHUM fitting process.
We refer
In the sequel we first explain our adaptations to NeRF for the static case. imGHUM is used as a prior in order to bias the reconstruction of the observed scene towards more accurate human geometry. We continue by explaining the additional methodological innovation needed in the dynamic case. Hereby, imGHUM provides spatial-tempont correspondences and used as the common reference frame to fuse information across different views and time instances.
4.1 Static Semantic Human NeRF
We first formulate H-NeRF for a human capture at single moment in time (N, = 1). From multi-view image observations, we co-learn a radiance field F. x, v + (c,o) for free-viewpoint rendering, and signed distance function Hm, x - d for 3D geometric reconstruction. We use H. for the dressed subject in order to distinguish it from the body's imGHUM SDF H
Like NeRF, we formulate an L1 image reconstruction loss to optimize Fw as
Lree Σ [[C(r) C(r)lla,
where denotes the batched set of all pixels/rays, and C(r), C(r) are observed and rendered pixel colors, cf. ( respectively. However, under sparse training camera views, the volumetric radiance field is not well regularized, leading to poor generalization to novel viewpoints, cf. fig. Specifically, we observe that the model encounters difficulties in correctly representing the scene and in separating the person from the background. The model fails to learn a semantically meaningful opacity field, i.c. O in free space, and if occupied.
Coarse Scene Structuring. Using imGHUM fits for all training images, we can spatially locate the person in 3D. We define a 3D bounding box B [Sw - e, Sh + around the detected person, where SH, SH are the minimal and maximal coordinates of the human body surface S and e is a spatial margin reserved for geometry not modeled by imGHUM. All radiance points associated to the rendering of the person should reside inside the bounding box, leading to a 3D segmentation loss
C(r) = - (xx)((x;)
II -i,-, b(x))a JRL
(4)
- ((M
(5)
rER

---

where b(x,) is O if outside of B and otherwise, and M(r) the image mask. We rely on the mask, so the loss is only applied to image observations from the subject, and not the remaining scene geometry.
Unifying SDF with NeRF. After structuring the scene coarsely, we now couple the estimated radiance field with implicit signed distance-basel 3ID reconstruction, in order to further regularize the opacity distribution. A signed distance function associated to the detected person in the image naturally comes with 3D classifier for all spatial points, where d((x,) > 0 means is in free space, whereas x, lies within the subject when d(xx)) <= O. We rely this insight in order to co-learn a SDF of the performer and constrain the radiance field. To this end, we introduce a pseudo alpha value (x,) = 6(>d(( x))) where & is Sigmoid activation function and controls the sharpness of the boundary. To refine the NeRF opacity semanties, especially for the volume in the neighborhood of the human surface B, we formulate two losses
C(r) Σ (xx, II(1 ((xx)) v), a(x,) = b(x,)(xx) + (1 b(x)))axx))
(6)
Lolend
Σ (() - (CT) ((r))
(7)
rER.
N Lgcom (r) = ((x a(x)))
(8)
where C(r) is the rendered pixel color with blended alpha values & replacing NeRF alpha values a with SDF-based pseudo alpha & for all ray points inside the bounding box B. The first term in Calend requires the rendered pixel color for all the intersecting rays within the human mask to come from the surface, whereas the second term assumes that all background color is formed from ray samples outside of B. We set 1) = when no other geometry than the person is inside B and tune 1) down the assumption is violated (e.g. person standing on a floor). The term Leoom uses the binary cross entropy loss to couple the NeRF surface boundary with the zero-isosurface of the signed distance function describing the subject. While the coupling terms given by 77) and (8) act as strong priors for the opacity distribution, during test time, we still rely on volumetric radiance rendering with learned NeRF alpha values a to support transparency effects and complex geometry for structures like hair.
Image-based SDF learning. Learning the signed distance field H.,, from scratch using sparse set of training images is still challenging. The model often fails to reconstruct reasonable human geometry, even when using coupling losses (7) and (8). We therefore leverage imGHUM as inner layer for the target human reconstruction and combine it with a light-weight residual SDF network AH x - Ad. The residual SDF models surface details, including hair and clothing, that not represented by imGHUM. The final signed distance for x; becomes d(x;,) d(x; 0, T) Ad(x)). Given the training images, we learn the personalized residual SDF using our coupling losses, given by and (8). and additionally apply geometric regularization
Lxeg Σ BCE(M(r), dmin(())
(9)
FER
Laik (r)
Σ (IV,)) 1)?, =
(10)
where dmin (r) denotes the minimal signed distance for all sampled ray points and va is the ReLU activation function. The term Lagg ensures that if a pixel is inside the human segmentation mask, there should be at least one intersection between the ray and the 3D human surface and therefore dmin (r) should be non-positive. Otherwise, all ray samples should have positive signed distances. Using Laik. we enforce the composite SDF d to be approximately a signed distance function, i.c. incorporating Eikonal regularization [14]. The term freg regularizes the residual distance Ad to be non-positive. This is because imGHUM should reside within the geometry and personalized geometric details given by Ad should be modeled on top of the skin surface.
5

---

4.2 Dynamic Semantic Human NeRF
We now extend the framework to model dynamic human motion. To implicitly model scenes where the human moves, we learn a consistent, continuous function G (c, a, d|zz [ + (c', a', d' zz) for both the geometry and the appearance variations across frames. For generalization to novel poses or shapes, G. should be conditioned on a semantically meaningful latent code z that can be interpolated and should idcally have good extrapolation properties. To integrate scene information over time in a single radiance field, we follow the approaches in [35] 37 and warp observations into canonical reference frame. Given our constraints to Gu., imGHUM, conditioned its semantic shape and pose latent code (3, 9, '),, naturally provides spatial correspondences across frames. Given two spatial points and x' in the space of two human instances (B, 0, T) and (B', 0', (') respectively, we decide these are in correspondence if they have the same semantics and signed distances (s, 3 0, T) = (s', d' d'',3', 0', T'). Essentially, imGHUM assigns a 4D point descriptor (s, d) E R to any spatial point and deforms the volume continuously with respect to the parameterized articulated human body surface. Specifically, we apply H. : : (T- B, 0) + (d, s) to map spatial point x to canonical point descriptor (s, d). We then modify the imput to the NeRF function with the point descriptor as F.. (s. - (c.o), and similarly for the residual SDF function as AH. (s, d) + Ad. imGHUM is pre-trained using a large corpus of 3D human scans, thus we keep it fixed in our network and directly use this prior to integrate structured geometric and appearance information across training frames. Given that the background scene is invariant to the human motion, we only apply the imGHUM warping function to spatial points inside the bounding box B, thus largely improving computation time and lowering memory usage.
Pose- Dese-Dependent Geometry and Appearance. Training a single canonical NeRF and residual SDF from a sequence integrates image observations into a consistent implicit geometric and volumetric representation of the radiance. However, we observe that fine-grained geometric and appearance variations caused by human motion are missing in the canonical NeRF and the residual SDF. To model such variations in geometry (clothing deformation) and appearance (lighting, self- soffsshadows). we condition the NeRF function F.. - the volume density (geometry) and color value e (appearance) as well as the residual SDF A (geometry), on the body pose code In addition, we also notice that the relative position and rotation of the person with respect to the scene largely affects appearance (due to illumination effects), but should not affect the geometry. Based on this observation, we additionally condition the NeRF color on the person's 's root transformation c(x T).
Differently from prior work [35 37). our H-NeRF modules conditioned on semantic human latent codes (3, 0, '). leading to an SDF and radiance field that can be interpo- lated. Moreover, due imGHUM's 's capability to accurately model geometric volume deformation for diverse human poses and shapes, H-NeRF enjoys strong generalisation to novel poses and even body shapes. Concretely, during inference, one can simply assign a different set of (3, 0) to transfer the learned geometry and appearance to an unseen pose and shape configuration. For better generalization, we add Gaussian noise to the input code (0, T), preventing network overfitting to pose-dependent deformation and appearance effects experienced during training. When only a sparse set of camera views is available, We apply the same technique to NeRF, conditioned on viewing direction v.
H-NeRF Articulation.
Fine-tuning imGHUM Pose and Shape. We observe that imGHUM fitting can be a source of error when learning Fw and AH from dynamic image sequences. Instead of keeping parameters fixed, we further improve the fit during training by fine tuning a time-consistent shape correction A33 and per-frame pose correction (t with
N
Lit BCE(M(()) dmin (r|8 A8, 0 + A0())), Lime FER
||
+
|
(11)
N
where Cat aligns the imGHUM fit with the human foreground segmentation and Line regularizes the incremental corrections.
5 Experiments
We quantitatively and qualitatively evaluate H-NeRF through ablations and by comparing with other methods. Please refer to our Sup. Mat. for additional results and ablation experiments. We first detail our model architecture, well the datasets and evaluation metrics used.
Architecture and Training. We use the same network architecture for all of our experiments, where the trainable modules F., and AH, consist of eight 256 dineennnniioall and six 128-dimensional MLPs

---

PSWRT
SSM
LPPSS
23
0.2
24
16
0.5
2.4
16
04
244
140
Charrier 110"
NO
Figure 2: Left: H-NeRF (red, x-axis: #cameras) outperforms the original NeRF (blue) and IDR
(green) in both novel view synthesis and 3D reconstruction of static scene (16 test cameras). NeRF and IDR fail under sparse camera views (solid line: RenderPeople scan; dashed line: GHS3D scan). Right: qualitatively show novel views and reconstructed geometry for NeRF, IDR, and H-NeRF (from left to right) trained using four cameras.
Model
Dataset
PSNR 1
SSIM T
LPIPS
Ch x10
NC f
IoU f
RenderPoople
27.33/23.52
0.8880.827
0.11770.247
0.336/0.63
0.908A1892
0.864008224
NeuralBody
36
PcopleSnapshet
GHS3D
24.62
24.7
0.849
0.829
0.160
0,236
0.79
0.887
0.81
Human3.66 6M
24.86
0.82
0.189
RenderPoople
28.78/24.31
0,913/0.86
125/0,246
0.217/0,274
0.950N0.939
0.917
H-Nekl' (ours)
GHS3D
24.92
0.852
0.232
0.218
0.932
0.89
PeopleSnupshot
26.33
0.868
0.159
Human3.6M
25.01
0.83
0.17
Table I: Quantitative evaluation of dynamic sequences for various datasets. When two metrics are reported they correspond to evaluation of training poses and novel poses for test cameras, respectively, PeopleSnapshot and Human3.6N 6. are evaluated on novel poses under training views (where ground- truth images exist). Geometric metrics are only reported when ground-trute is available.
respectively, with a skip connection to the middle layer and with Swish activation 38 As in the original NeRF [29], we use 256 coarse and fine-level ray samples, for each of which we use 8- and 1-dimensional positional encoding for F. and AHw. respectively. We train the network using the Adam optimizer with learning rate of 0.001 exponentially decayed by a factor 0.1 until the maximum number of iterations is reached (10k iteration of 4k ray batch size). We apply Gaussian noise with 0.1 to the NeRF conditioning code (0, T, v).
Dataset and Metrics. We provide qualitative and quantitative evaluation on four different datasets: RenderPeople (8 sequences), GHS3D [51] (14 sequences), PeopleSnapshot [3 (7 sequences) and Human3.66 .6M [99 (5 sequences). The first two are synthetic, rendering animated (RenderPeople) or 4D human scans (GHS3D) from four orthogonal cameras, where ground-truth geometry paired with images is available. We evaluate both the image and the 3D geometry reconstruction quality from two novel cameras, and qualitatively demonstrate generalization to shapes and poses not in the training set. The remaining datasets are real captured vidcos (PcopleSnapshot: monocular, Human3, 66: four cameras) without paired 3D geometry, where only evaluate the rendered images. For image metrics, we adopt peak signal-to-noise no-se ratio (PSNR 1), structural similarity index (SSIM t) and learned perceptual image patch similarity (LPIPS J) [57]. For evaluation, render the NeRF field inside the human bounding box and compare to the ground-truth segmented image within a region of interest around the person. To evaluate geometric reconstruction quality, we report bi-directional Chamfer (Ch) L2 distance, Normal Consistency (NC) and Volumetric Intersection over Union (IoU), evaluated on the mesh produced by applying Marching Cubes the SDF, with resolution of 256".
Static Human Reconstruction under Sparse Viewpoints. H-NeRF learns geometrically regular- ized volumetric radiance field constrained by an imGHUM-based SDF, which significantly improves robustness under sparse training camera views. fig. 2 we show reconstruction of static Render- People and GHS3D scan, respectively, using the original NeRF 22). the state-of-the-an multi-view reconstruction approach IDR (53]. and H-NeRF, for increasing number of cameras. Both novel view image quality and geometric accuracy improve for all methods the number of training views increases. However, in contrast to competitors, H-NeRF produces good quality output even under sparse training views (2-8), demonstrating the generalization capability of a co-training approach.
Dynamic Human Reconstruction and Rendering. the following, we evaluate H-NeRF's capabil- ities to reconstruct and render dynamic scenes (fig. 5 We compare H-NeRE against NeuralBody (36)
7

---

30
PSNR
0.25
SSIM
0.35
LPIPS
2.0
Chamfer 1100
0.9
NC 1
1.0
IoU
0.30
25
0.75
1.0
0.6
0.8
0.70
0.5
0.4
15
5
10
20
30
45
6
20
30
45
20
30
45
5
10
20
30
45
20
30
45
5:
20
30
45
Figure 3: Frame Number Ablation. H-NeRF (red, x-axis: #training video frames) outperforms Neural Body (blue), evaluated on RenderPeople (dashed) and GHS3D (solid) sequence.
Figure 4: Qualitative comparisons with NeuralBody four different datasets. Left: GHS3D (top) and RenderPeope (bottom) ground-truth image, our result, NeuralBody, ground-truth geometry (front and back), geometry, NeuralBody's geometry. Right: PeopleSnapshot (top) and Human3.6M (bottom) ground-truth image, our result, NeuralBody, our geometry. NeuralBody's geometry. Pay attention to the sharp renderings and the complete and detailed geometry produced by method. Digital zoom-in recommended.
(tab. [.. the current state-of off-the-arttt for novel view synthesis of humans in motion. For fair compar- isons, we have trained NeuralBody with the same data H-NeRF and use GHUM meshes (instead of SMPL). GHUM meshes with same pose and shape representation as in our model are used for NeuralBody's 's structured latent codes. We further compare with Nerfies [35]. However, Nerfies has difficulties with the high range of motion our test sequences and fails for most them. We
consider such results less meaningful and report them only in the Sup. Mat. for completeness.
H-NeRF overperforms NeuralBody across datasets and metrics, especially for geometric recon- struction. One possible explanation for NeuralBody's lower performance on view synthesis, is that NeuralBody not conditioned on the root transformation. Thus, global lighting effects are handled less well. Furthermore, pose depended effects that go beyond body articulation are only indirectly modeled through relative distances of input mesh vertices. Finally, NeuralBody conditions on the frame index, which is set to zero for novel poses. On the PeopleSnapshot dataset where no view-dependent and only subtle pose-dependent effects are present, these limitations are not entirely apparent. In contrast, H-NeRF pays special attention to modeling both detailed geometry and appearance, and its explicit conditioning on pose and the root transformation captures pose-dependent geometric and appearance effects. This strategy pays off, especially in more complex scenarios.
Frame Number Ablation. We have illustrated H-NeRF's rendering and reconstruction capabilities for static and dynamic sequences. Next, we evaluate the necessary number of different poses or time-frames H-NeRF needs during training for good pose generalisation. We again compare against NeuralBody. To this end, we have trained both methods with an increasing number of images, uniformly distributed over the full sequence. Results are shown fig. 3 As expected, both methods perform better when trained with more data. However, H-NeRF performs overall better and more importantly, the quality degrades less in the sparse training regime, especially for geometric reconstruction. Our results support H-NeRF's robustness to sparse training data and its capability to reconstruct accurate geometry. practical terms, H-NeRF can be robustly trained with as little as 10 temporal frames per camera (in a four camera set-up), resulting performance capture efffrt of only a few minutes.
Qualitative Results for Pose Generalisation and Shape Extrapolation. Finally, we illustrate H- NeRF's rendering and reconstruction accuracy qualitatively (fig. ( Consistently with the reported metrics, our synthesized novel poses appear sharper, more detailed, and contain less noise than the current otate-of-the-art. The estimated geometry is complete, smooth, and contains much of the detail present in the original scan, e.g. the clothing folds the back of the person on the left, the first row.

---

Figure 5: Qualitative results of H-NeRF for novel image synthesis. We show ground truth images and scans (left, available) side-by-side with our results (right). Red icons correspond to test, green icons training configurations. Symbols correspond camera, pose, and shape, respectively. From top to bottom: RenderPeople, GHS3D, PeopleSnapshot and Human3.6M.
In contrast, geometries produced by NeuralBody noisier and sometimes incomplete. To further demonstrate the versatility of our approach, in fig. 5 we show examples of synthesized images and reconstructed geometry from novel viewpoints, in novel poses, and for modified body shape.
Limitations and Broader Impact. In the current setup, we assume full-body views of a single person in every-day clothing. Inconsistency with these assumptions, c.g. by placing another person in the scene and occluding the performer, would break the method although models of partial views, or representation estimates of multiple people could be used for generality. Furthermore, exhibits some sensitivity to the estimated body shapes and poses, as well as the quality of image segmentation. Although the process of learning render could absorb certain inaccuracies, as pose, shape or segmentation are increasingly degraded, the method would fail eventually.
method
Our method paves the way towards immersive AR and VR applications. In contrast, our approach is not targeted, or particularly useful for applications like visual surveillance or person identification as particular set-up and a cooperating subject is needed. We do not build audio-visual model that would be necessary for deep fakes.
Conclusions
We have presented novel neural radiance field models (H-NeRF) for the photo-realistic rendering and the temporal reconstruction of humans in motion. Our objective extend the scope of the previous state of the art, focused on static scenes observed by large number of cameras, by building dynamic models, which based sparse set of views, can generalize well to novel camera views, human body poses, and extrapolate over human body shapes. Our key contribution is in developing new model with associated multiple losses, in order specialize and constrain generic NeRF formulation by using a compatible implicit statistical 3D human pose and shape model, represented using signed distance functions. Our implicit geometric formulation captures not just the statistical regularities of the human body, but also hair and clothing represented as an implicit residual network. Our model trained end-to-end based on several novel losses, and achieves good results for both 3D reconstruction and photorealistic rendering. Training the body model in an end-to-end rendering framework carries the promise to learn complex implicit skinning functions based on images only, performance previously possible only in the realm of human capture using complex and expensive 3D body scanners, in the laboratory. We illustrate the favorable capabilities of H-NeRF through extensive experimentation using several datasets and against other state of the art techniques.

---

References
[1] Kara-Ali Aliev, Dmitry Ulyanov, and Victor Lempitsky. Neural point-based graphics. arXiv preprint
arXiv: (906i 08240.
[2] Thiemo Alldieck, Marcus Magnor, Bharat Lal Bhatnagar, Christian Theobalt, and Gerard Pons-Moll. Learning to reconstruct people in clothing from single RGB camera. In IEEE Conf. Comput. Vis. Pattern Recog.. pages 1175-1186. IEEE, 2019.
[3] Thiemo Alldiock, Marcus Magnor, Weipeng Xu, Christian Theobalt, and Gerard Pons-Moll, Video based reconstruction of 3D people models. In /EEE Conf. Comput. Vis. Pottern Recog,, pages 8387-8397. IEEE,
2018.
[4] Thiemo Alldieck, Hongyi Xu, and Cristian Sminchisescu. imGHUM: Implicit generative models of 3D human shape and articulated pose. In Int. Conf. Comput. Vis., 2021.
[5] Federica Bogo, Michael J. Black, Matthew Loper, and Javier Romero. Detailed full-hody reconstructions
of moving people from monocular RGB-D sequences. In JEEF International Conference on Computer Vision, pages 2300-2308.. IEFF, 2015.
[6] Joel Carranza, Christian Theobalt, Marcus A Magnor, and Hans-Peter Scidel. Free-viewpoint video of human actors. ACM Trans. Graph., 22(3):569-5)) 577, 2003.
[7] Shenchang Eric Chen and Lance Williams. View interpolation for image synthesis. In Proceedings of the
20th annual conference on Computer graphics and interactive techniques, pages 279-288, 1993.
[8] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In JEEE Conf. Comput. Vis. Pattern Recog., Recog.pages 5939-5948, 2019..
[9] Edilson De Aguiar, Carsten Stoll, Christian Theobalt, Naveed Ahmed, Hans-Peter Seidel, and Sebastian Thrun Performance capture from sparse multi-view video. In ACM Trans. Gruph., pages 1-10. 2008.
[10] Paul E Debevec, Camillo J Taylor, and Jitendra Malik. Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 1 1-20, 1996.
[11] Boyang Deng. JP Lewis. Timothy Jeruzalski. Gerard Pons-Moll. Geoffrey Hinton, Mohammad Norouzi. and Andrea Tagliasacchi. Neural articulated shape approximation. In Eur. Conf. Comput. Vis. Springer. August 2020.
[12] Juergen Gall, Carsten Stoll, Edilson De Aguiar, Christian Theobalt, Bodo Rosenhahn, and Hans-Peter Seidel. Motion capture using joint skeleton tracking and surface estimation. In IEEE Conf. Comput. Vis. Pattern Recog. pages 1746-1753. IEEF 2009.
[13] Ke Gong, Xiaodan Liang. Yicheng Li, Yimin Chen, Ming Yang, and Liang Lin. Instance-leved human parsing via part grouping network. In Proceedings of the European Conference on Computer Vision (ECCV), pages 770-785, 2018.
[14] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization shapess [15] Marc Habermann, Lingjie Liu, Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll, and Christian Theobalt. Real-time deep dynamic characters. ACM Trans. Graph., 40(4), aug 2021.
for learning shapes. In Jnt. Conf. on Mach. Learn., pages 3569-3579. 2020.
[16] Marc Habermann. Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll, and Christian Theobalt. Deepcap: Monocular human performance capture using weak supervision. In IEEE Conf. Comput. Vis. Pattern Recog. IEEE, jun 2020.
[17] Mare Habermann, Weipeng Xu, Michael Zollhöfer, Gerard Pons-Moll, and Christian Theobalt. Livecap: Real-time human performance capture from monocular video. ACM Trans. Gruph., 380))14:1-14:177, 2019.
[18] Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan T Barron, and Paul Debevec. Baking neural radiance fields for real-time view synthesis. In Int. Conf. Comput. Vis., 2021.
[19] Catalin lonescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6M, Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Trans. Pattern Anal. Mach. Intell., 2014.
[20] Thomas Lewiner, Hélio Lopes, António Wilson Vieira, and Geovan Tavares. Efficient implementation of marching cubes' cases with topological guarantees. Journal of Graphics Tools, 3(2):1115, 2003.
[21] Lingjie Liu. Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields.
NewrIPS, 2020.
[22] Stephen Lombardi, Jason Saragih, Tomas Simon, and Yaser Sheikh. Deep appearance models for face rendering. ACM Trany. Graph., 37(4): 2018
[23] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh.
Neural volumes: Learning dynamic renderable volumes from images. ACM Trans. Graph., 38(4):65:1-
65:14, July 2019.
[24] Matthew Loper. Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. SMPL: A
skinned multi-person linear model. ACM Trans. Graph., 34(6):248:1 1-28::1124::66666 2015.
[25] Lars M. Mescheder, Michael Occhsle, Michael Niemeyer, Scbastian Nowozin, and Andreas Geiger.
Occupancy networks: Learning 3d reconstruction in function space. In IEEE Conf. Comput. Vis, Pattern
Recog., pages 4460-4470, 2019.
[26] Moustafa Meshry, Dan B Goldman, Sameh Khamis, Hugues Hoppe, Rohit Pandey, Noah Snavely, and Ricardo Brruala. Neural rerendering in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6878 6887, 2019.
10

---

[27] Mateusz Michalkiewicz, Jhony K Pontes. Dominie Jack, Mahsa Baktashmotlagh, and Anders Eriksson. Deep level sets: Implicit surface representations for 3D shape inference. arXiv preprint 1Xii::901.00802
2019.
[28] Marko Mihajlovic, Yan Zhang, Michael J. Black, and Siyu Tang. LEAP:
Learning articulated occupancy of people. In Proceedings IEEE/CVF Conf. on Computer Vixion and Pattern Recognition (CVPR), June 2021. Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren
[29]
Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In Eur. Conf. Comput. Vis., 2020.
[30] Roger Mohr, Long Quan, and Françoise Veillon. Relative 3d reconstruction using multiple uncalibrated
images. The International Journal of Robotics Research, 14(6):61--632, 1995.
[31] Michael Niemeyer, Lars Mescheder, Michael Oechsle. and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In IEEE Conf. Comput. Vis. Pattern
Recog., June 2020.
[32] Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada. Neural articulated radiance field. In Int.
Conf. Comput. Vis., 2021.
[33] Michacl Occhsle, Songyou Peng, and Andreas Geiger. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. Int. Conf. Comput. Vis., 2021.
[34] Jeong Joon Park, Peter Florence, Julian Straub, Richard A. Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functtions for shape representation. In JEEE Conf. Comput. Vis. Pattern Recog, pages 165-174, 2019.
[35] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M Scitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. In Int. Conf. Comput. Vis., 2021. [36] Sida Peng. Yuanqing Zhang. Yinghao Xu, Qianqian Wang. Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
[37] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moogue- D-nerf: Neural radiance
fields for dynamic scenes. In IEEE Conf. Comput. Vis. Puttern Recog. IEEE, jun 2021.
[38] Prajit Ramachandran, Barret Zoph, and Quoc Lc. Searching for activation functions. arXiv preprint
arXiv:1700.00 05941, 2017.
[39] Daniel Rehain, Wei Jiang, Soroosh Yazdani, Ke Li, Kwang Moo Yi, and Andrea Tagliasacchi. Derf:
Decomposed radiance fields. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
[40] Gernot Riegler and Vladlen Koltun. Free view synthesis. In European Conference on Computer Vision,
pages 623-640. Springer, 2020.
[41] Shunsuke Saito, Zeng Huang. Ryota Natsume, Shigeo Morishima, Angjao Kanazawa, and Hao Li. Pifu: Pixel-alic exxll-llgnned implicit function for high nigh-resolution clothed human digitization. In Int. Conf. Comput. Vis., pages 2304 2314, 2019.
[42] Johannes I. Schonberger and Jan-Michael Frahm. Structure-frome frome-momoooooooooooooooo revisited. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 4 4104-4113, 2016.
[43]
Aliaksandra Shysheya, Egor Zakharov, Kara-Ali Aliev. Renat Bashirov, Egor Burkov, Karim Iskakov, Aleksei Ivakhnenko, Yury Malkov, Igor Pasechnik, Dmitry Ulyanov, ct al. Textured neural avatars. In IEEE Conf. Comput. Vis. Pattern Recog., pages 2387-2397, 2019.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Weizstein. Implicit neural representations with periodic activation functions. Adv. Neural Inform. Process. Syst. 33, 2020. [45] Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhöfer.
[44]
Deepvoxels: Learning persistent 3d fcature embeddings. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEF, 2019.
[46] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d structurc-awar aware neural seene representations. In Adv. Neural Inform. Process. Syst., 2019.
[47] Shih-Yang Su, Frank Yu, Michael Zollhoefer, and Helge Rhodin. A-nerf: Surface-free human 3d pose
refinement via neural rendering. In Adv. Neural Inform. Process. Syst., 2021.
[48] Ayush Tewari, Ohad Fried. Justus Thies. Vincent Sitzmann, Stephen Lombardi. Kalyan Sunkavalli, Ricardo Martin-Brualla Tomas Simon, Jason Saragih, Matthias Nießner, al. State of the on neural rendering. In Compur. Graph. Forum, volume 39, pages 701-727. Wiley Online Library, 2020.
[49] Justus Thies, Michael Zollhöfer, and Matthis Nießner. Deferred neural rendering: Image synthesis using
neural textures. ACM Transactions on Graphics (TOG), 1-12) 2019.
[50] Daniel Vlasic. 1lya Baran, Wojeiech Matusik, and Jovan Popović. Articulated mesh animation from multi-view silhouettes. In ACM Trans. Graph., pages 1 1-9. 2008.
[51] Hongyi Xu, Eduard Gabriel Bazavan, Andrei Zanfir, William T Freeman, Rahul Sukthankar, and Cristian Sminchisescu. GHUM & GHUML: Generative 3d human shape and articulated pose models. In IEEE
Conf. Comput. Vis. Pattern Recog,, pages 61 6184-6193, 2021.
[52] Weipeng Xu, Avishek Chatterjee, Michael Zollhoefer. Helge Rhodin, Dushyant Mehta, Hans-Peter Seidel, Human performance capture from monocular video. ACM Trans.
and Christian Theobalt. Monoperfcap:
Graph.. 2018
[53] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Adv. Neural Inform. Process. Syst. 33, 2020.
11

---

[54] Alex Yu, Ruilong Li, Matthew Tancik. Hao Li. Ren Ng. and Angjoo Kanazawa. Plenoctrees for real-time
rendering of neural radiance fields. In Int. Conf. Comput. Vis., 2021.
[55]
Mihai Zanfir, Elisabeta Oneata, Alin-lonut Popa, Andrei Zanfir, and Cristian Sminchisescu. Human
synthesis and
compositing. In AAAI, pages 12749-12756, 2020.
[56] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural
radiance fields. arXiv preprini arXiv:2010.07492 2020.
[57] Richard Zhang, Phillip Isola, Alexci Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
12


Reviews:
[{"review_id": "lPBRVjWYMCb", "reviewer": "Reviewer_VaSk", "summary": "This paper presents a new NeRF based method for rendering and reconstruction of humans observed from sparse cameras. The main contribution is combining volumetric radiance fields and an implicit SDF for the tasks of novel view synthesis and geometric reconstruction for humans. To achieve this goal, the method utilizes a human prior, imGHUM, which was pretrained on thousands of 3D scans. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I like the idea of unifying explicit and implicit representations for rendering as well as reconstruction. \nI also appreciate the important problems that the proposed method attempts to address, i.e. photo-realistic free-viewpoint rendering, accurate geometric reconstruction, as well as synthesis for new poses and new shapes. \nHowever, I am not quite satisfied with the proposed solutions and the quality of the results. \nFor example, the main contribution is to combine NeRF and SDF. The solution is just replacing the density value in volume rendering with a function of the SDF value. However, they did not realize this naive replacement would introduce some geometric errors in the reconstruction results, which was discussed in UNISURF. That is, the formulation of the color accumulation equation in volume rendering would make the point that contributes most to the ray color not lie on the surface. How did the authors solve this issue?\nFurthermore, the authors claimed that they can synthesize photo-realistic rendering, however, the novel view synthesis results are blurry and do not meet the standard of photo-realism. \nThe authors also claimed they can synthesize for new poses, however, the new poses are very simple and thus it is hard to see the generalizability of this method on novel poses. To evaluate the generalizability of this method, the proposed method should be tested on the poses that are very different from training poses, e.g. poses from the AMASS dataset and the AIST++ dataset. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. The authors claimed that the key contribution was a new model with associated multiple losses to constrain a NeRF formulation by using the imGHUM statistic model represented using SDF. To me, the losses are not novel, either the regular losses for training NeRF or training neural SDF (i.e. imGHUM in this paper). Moreover, the way to unifying neural radiance fields and SDF is just replacing volume densities with SDF values, more importantly, the issue of having the geometric errors caused by this replacement has not been discussed. Therefore, the technical contributions sound limited. \n2. The quality of the novel view synthesis results is limited.\n3. The comparisons with the following methods on surface reconstruction are expected: \nPIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization\nARCH: Animatable Reconstruction of Clothed Humans\nVideo-Based Reconstruction of 3D People Models\nLearning to Reconstruct People in Clothing from a Single RGB Camera\n4. The comparisons with the following methods on novel view synthesis are expected:\nANR: Articulated Neural Rendering for Virtual Avatars\n4. For the novel pose synthesis experiment, more challenging poses are expected to test. For example, poses from the AMASS dataset and the AIST++ dataset. \n5. The comparison of IDR in the experiment of static human reconstruction is not fair. For a fair comparison, the prior encoded in imGHUM should also be used in IDR since imGHUM should not be regarded as a contribution of this work. \n6. I understand the paper aims at sparse input settings. However, it's good to see the results (qualitative and quantitative results) of IDR and the proposed method when the camera views are sufficient. ", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "iTZoc-ujs9V", "reviewer": "Reviewer_MJGq", "summary": "The authors introduce H-NeRF, a geometry-aware neural radiance field for human performance rendering. \nThe core ideas are to (1) conditioned the neural radiance field (NeRF) on a pre-trained implicit geometric human representation that captures a wide variety of body poses and shapes, and (2) co-learn a signed distance function (SDF) that describes the surface geometry of the NeRF for rendering. Experimental results show that, compared to other baselines, H-NeRF achieves better surface reconstruction and novel view and pose synthesis quality, and are able to extrapolate to new body shape volumes. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I feel positive about this work. Learning high-fidelity and virtual avatars with animation capability is an important problem. The authors approach this problem with a novel formulation that combines NeRF with human geometric priors, and the empirical evaluations show promising results. \n\nStrength:\nGood generalization to unseen poses, comparing to other NeRF-based approaches (e.g., D-NeRF, NeuralBody).\nBetter surface reconstruction with fewer extraneous points/densities.\nCan handle moderate cloth/body shape deformation.\nMostly consistent geometry across different poses.\nCan learn from a sparse set of cameras.\n\nWeakness:\nComputation/memory consumption/training time is not discussed, please list and add.\nPotentially slow inference speed: NeRFs/Implicit functions are often slow, and the presented work has 3 of them in H-NeRF, so I expect the computation time will be higher than other approaches.\nSensitivity to hyperparameters is not discussed: how sensitive H-NeRF is to the weights of the nine different losses? How are the weights chosen? While the weight configuration is in the supplementary (line 89), it would be better if the authors could provide more insights into these losses. Similarly, how are \\sigma_h and \\gamma determined? \nSimilar to other human NeRF, H-NeRF needs to fit one model for each human subject.\n\nBelow are some further questions and suggestions of how to improve the evaluation:\nHow does H-NeRF without the residual SDF work (e.g., simply train a NeRF conditioned on imGHUM)?\nHow does H-NeRF work in extreme poses (e.g., break dancing)?\n(line 316, Figure 3) it would be nice if the authors can provide some renderings for comparison. Sometimes small PSNR improvements could make a huge difference perceptually.\n\nSome minor issues:\nline 91: citation for NARF [1] is incorrect.\nsupp. line 96: subjectquetly -> subsequently.\n\n[1] Atsuhiro Noguchi, Xiao Sun, Stephen Lin, Tatsuya Harada. Neural articulated radiance field. arXiv preprint arXiv:2104.03110, 2021.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have discussed a few limitations in line 326-332. It would be nice if the authors can also clarify the questions/weaknesses raised above.\n", "review_text": "", "extra_scores": {"ethical_concerns": "no", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "5h"}, "extra_text": {}}, {"review_id": "ZP7D1UHe_x", "reviewer": "Reviewer_vuz9", "summary": "The authors propose a model that combines a parameteric human shape model with NeRF and a residual signed distance function to enable novel-view synthesis of humans in motion, assuming full observation of the subject (i.e., no prior-based completion).\n\nThe core contributions is the combination of NeRF with a parametric human body model, which enables the learning of the NeRF in a canonical coordinate frame, as well as regularization of the learned geometry. Several additional tweaks improve the model further, such as leveraging auxiliarly losses via a instance segmentation mask, or feeding the estimated body pose to the NeRF as additional input, enabling dynamic (due to being pose-specific) fine detail.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Related work*\nThe related work is great.\n\n*Exposition*\nThe exposition is generally good, however, I found the methods section hard to parse. In particular, I found it very confusing that the authors first discuss the case 4.1 *without* using the coordinate frame established by the imGHUM model. After reading the introduction and the background, I had expected that the authors would leverage imGHUM to establish a canonical coordinate frame that is used to train the NeRF in. Instead, the authors start out explaining a vanilla NeRF without any notion of dynamics. I assume that this was done to set up the scene structuring and residual SDF sections, but personally, I believe the pitch would actually get easier if the authors directly established that the input to the NeRF and the SDF will be the imGHUM's (s, d) tuple. Maybe the authors want to reconsider the narrative of this section.\n\nInstead of \"learned\", I would recommend the authors call views in the training set \"context\" or \"training\" views - \"learned\" is confusing.\n\nA minor detail: I would recommend the authors remove the audio track of their video, as they did not record any voiceover, but one can still hear static noise and voices in the background.\n\n*Evaluation*\nI was glad to see a comparison to NeRF and IDR in the static case, and then a comparison to both NeRFies and neural bodies in the dynamic case.\n\nHowever, I am not content with the video results. First, in the \"Novel View Synthesis\" section, I would expect to see the nearest neighbor context camera view, not only the ground-truth for novel camera views. Further, I would like to see what the training set for these sequences looks like - i.e., a mosaic of the context videos before each of the novel view synthesis results.\n\nFurther, I would like to see novel camera *trajectories*, not only novel camera *views*. \n\nOnly with the previous two results is it possible to judge the multi-view consistency and extrapolation capabilities of the proposed approach appropriately.\n\nNext, I would like to see a discussion of both the rendering and inference times of the proposed approach, which I expect to be very, very slow. What FPS can be achieved at what resolution? How long does it take to train the network?\n\n*Final review*.\nI recommend this paper for acceptance. Though I would like to see more comprehensive results in the form of continuous camera trajectories and the context videos, the method is sound and I am convinced that this method performs the way the authors claim it does. The only \"shortcoming\" of the proposed method is its specificity to the problem at hand, i.e., it is not a fundamental contribution that would enable new approaches across applications in vision, for instance. In particular, this method is quite specific to dynamic novel view synthesis where a parameter-based model is available, which is the case for faces and bodies, but not for general objects. Nevertheless, the contributions are clear, the results are cool and the paper is well written, and I believe this to be a valuable contribution to the community.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I would have appreciated an analysis of runtimes, see \"Evaluation\" above.\n\nThe authors adequately discuss the societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "UlmRn5fW6j", "reviewer": "Reviewer_TFXE", "summary": "This paper proposes a method for dynamic human reconstruction and rendering using sparse view videos. \nThe high-level strategy is to learn both a neural radiance field and an SDF field with mutual constraints in a canonical frame from temporal observations. Moreover, for generating live rendering and surface results, pose and position conditioned SDF offsets are also learned for better live frame surface reconstruction results. \nTo guarantee robust surface reconstruction, an implicit generative human model was used as the inner layer which not only incorporates strong human shape constraint, but also provides semantic correspondences for accurate warping between live frames and the canonical frame. Another benefit of using an underlying implicit human model is that shape and pose editing can be achieved through interpolating in the parametric body space, which was not demonstrated in previous works.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Reconstructing/Encoding dynamic humans from light-weight inputs is a very challenging but important topic that has strong potential usage in AR/VR, telepresence, etc.\n \nThe paper writing is clear and easy to follow.\n \nThe overall idea of the paper is novel, the results in the paper look nice, the quantitative comparisons are impressive. \n \nOne of my concerns is that the video results are not well demonstrated, almost all the video results are not very continuous. For example, in video 04:17, the side view rendering results are simply skipped, and this happens to all the results of the PeopoeSnapShot dataset. Moreover, for RenderPeople dataset results in video 01:27, it's easy to get continuous frames for training and rendering, but the videos results are not very continuous, please demonstrate full video results without skipping any frames. \n \nMoreover, I think video comparisons with NeuralBody should be provided to enhance the effectiveness of the proposed method on free-viewpoint rendering and dynamic geometry reconstruction results.\n\nMinor issue:\nThe caption of Fig.4: botton -> bottom", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": " ", "review_text": "", "extra_scores": {"ethical_concerns": " ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission has received 4 positive final ratings: 6, 7, 7, 8.
The reviewers overall were excited about the method and the idea of combining implicit representations with explicit priors, and also acknowledged strong empirical results and solid presentation. The remaining questions and concerns were mostly addressed in the rebuttal: the reviewers were mostly satisfied, but left some recommendations for further improvements (the authors are recommended to follow them while preparing the camera ready version). 
The final recommendation is to accept as a spotlight.
Decision: Accept (Spotlight)