Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a new NeRF based method for rendering and reconstruction of humans observed from sparse cameras. The main contribution is combining volumetric radiance fields and an implicit SDF for the tasks of novel view synthesis and geometric reconstruction for humans. To achieve this goal, the method utilizes a human prior, imGHUM, which was pretrained on thousands of 3D scans. 
main_review: I like the idea of unifying explicit and implicit representations for rendering as well as reconstruction. 
I also appreciate the important problems that the proposed method attempts to address, i.e. photo-realistic free-viewpoint rendering, accurate geometric reconstruction, as well as synthesis for new poses and new shapes. 
However, I am not quite satisfied with the proposed solutions and the quality of the results. 
For example, the main contribution is to combine NeRF and SDF. The solution is just replacing the density value in volume rendering with a function of the SDF value. However, they did not realize this naive replacement would introduce some geometric errors in the reconstruction results, which was discussed in UNISURF. That is, the formulation of the color accumulation equation in volume rendering would make the point that contributes most to the ray color not lie on the surface. How did the authors solve this issue?
Furthermore, the authors claimed that they can synthesize photo-realistic rendering, however, the novel view synthesis results are blurry and do not meet the standard of photo-realism. 
The authors also claimed they can synthesize for new poses, however, the new poses are very simple and thus it is hard to see the generalizability of this method on novel poses. To evaluate the generalizability of this method, the proposed method should be tested on the poses that are very different from training poses, e.g. poses from the AMASS dataset and the AIST++ dataset. 
limitations_and_societal_impact: 1. The authors claimed that the key contribution was a new model with associated multiple losses to constrain a NeRF formulation by using the imGHUM statistic model represented using SDF. To me, the losses are not novel, either the regular losses for training NeRF or training neural SDF (i.e. imGHUM in this paper). Moreover, the way to unifying neural radiance fields and SDF is just replacing volume densities with SDF values, more importantly, the issue of having the geometric errors caused by this replacement has not been discussed. Therefore, the technical contributions sound limited. 
2. The quality of the novel view synthesis results is limited.
3. The comparisons with the following methods on surface reconstruction are expected: 
PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization
ARCH: Animatable Reconstruction of Clothed Humans
Video-Based Reconstruction of 3D People Models
Learning to Reconstruct People in Clothing from a Single RGB Camera
4. The comparisons with the following methods on novel view synthesis are expected:
ANR: Articulated Neural Rendering for Virtual Avatars
4. For the novel pose synthesis experiment, more challenging poses are expected to test. For example, poses from the AMASS dataset and the AIST++ dataset. 
5. The comparison of IDR in the experiment of static human reconstruction is not fair. For a fair comparison, the prior encoded in imGHUM should also be used in IDR since imGHUM should not be regarded as a contribution of this work. 
6. I understand the paper aims at sparse input settings. However, it's good to see the results (qualitative and quantitative results) of IDR and the proposed method when the camera views are sufficient.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors introduce H-NeRF, a geometry-aware neural radiance field for human performance rendering. 
The core ideas are to (1) conditioned the neural radiance field (NeRF) on a pre-trained implicit geometric human representation that captures a wide variety of body poses and shapes, and (2) co-learn a signed distance function (SDF) that describes the surface geometry of the NeRF for rendering. Experimental results show that, compared to other baselines, H-NeRF achieves better surface reconstruction and novel view and pose synthesis quality, and are able to extrapolate to new body shape volumes. 

main_review: I feel positive about this work. Learning high-fidelity and virtual avatars with animation capability is an important problem. The authors approach this problem with a novel formulation that combines NeRF with human geometric priors, and the empirical evaluations show promising results. 

Strength:
Good generalization to unseen poses, comparing to other NeRF-based approaches (e.g., D-NeRF, NeuralBody).
Better surface reconstruction with fewer extraneous points/densities.
Can handle moderate cloth/body shape deformation.
Mostly consistent geometry across different poses.
Can learn from a sparse set of cameras.

Weakness:
Computation/memory consumption/training time is not discussed, please list and add.
Potentially slow inference speed: NeRFs/Implicit functions are often slow, and the presented work has 3 of them in H-NeRF, so I expect the computation time will be higher than other approaches.
Sensitivity to hyperparameters is not discussed: how sensitive H-NeRF is to the weights of the nine different losses? How are the weights chosen? While the weight configuration is in the supplementary (line 89), it would be better if the authors could provide more insights into these losses. Similarly, how are \sigma_h and \gamma determined? 
Similar to other human NeRF, H-NeRF needs to fit one model for each human subject.

Below are some further questions and suggestions of how to improve the evaluation:
How does H-NeRF without the residual SDF work (e.g., simply train a NeRF conditioned on imGHUM)?
How does H-NeRF work in extreme poses (e.g., break dancing)?
(line 316, Figure 3) it would be nice if the authors can provide some renderings for comparison. Sometimes small PSNR improvements could make a huge difference perceptually.

Some minor issues:
line 91: citation for NARF [1] is incorrect.
supp. line 96: subjectquetly -> subsequently.

[1] Atsuhiro Noguchi, Xiao Sun, Stephen Lin, Tatsuya Harada. Neural articulated radiance field. arXiv preprint arXiv:2104.03110, 2021.

limitations_and_societal_impact: The authors have discussed a few limitations in line 326-332. It would be nice if the authors can also clarify the questions/weaknesses raised above.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a model that combines a parameteric human shape model with NeRF and a residual signed distance function to enable novel-view synthesis of humans in motion, assuming full observation of the subject (i.e., no prior-based completion).

The core contributions is the combination of NeRF with a parametric human body model, which enables the learning of the NeRF in a canonical coordinate frame, as well as regularization of the learned geometry. Several additional tweaks improve the model further, such as leveraging auxiliarly losses via a instance segmentation mask, or feeding the estimated body pose to the NeRF as additional input, enabling dynamic (due to being pose-specific) fine detail.
main_review: *Related work*
The related work is great.

*Exposition*
The exposition is generally good, however, I found the methods section hard to parse. In particular, I found it very confusing that the authors first discuss the case 4.1 *without* using the coordinate frame established by the imGHUM model. After reading the introduction and the background, I had expected that the authors would leverage imGHUM to establish a canonical coordinate frame that is used to train the NeRF in. Instead, the authors start out explaining a vanilla NeRF without any notion of dynamics. I assume that this was done to set up the scene structuring and residual SDF sections, but personally, I believe the pitch would actually get easier if the authors directly established that the input to the NeRF and the SDF will be the imGHUM's (s, d) tuple. Maybe the authors want to reconsider the narrative of this section.

Instead of "learned", I would recommend the authors call views in the training set "context" or "training" views - "learned" is confusing.

A minor detail: I would recommend the authors remove the audio track of their video, as they did not record any voiceover, but one can still hear static noise and voices in the background.

*Evaluation*
I was glad to see a comparison to NeRF and IDR in the static case, and then a comparison to both NeRFies and neural bodies in the dynamic case.

However, I am not content with the video results. First, in the "Novel View Synthesis" section, I would expect to see the nearest neighbor context camera view, not only the ground-truth for novel camera views. Further, I would like to see what the training set for these sequences looks like - i.e., a mosaic of the context videos before each of the novel view synthesis results.

Further, I would like to see novel camera *trajectories*, not only novel camera *views*. 

Only with the previous two results is it possible to judge the multi-view consistency and extrapolation capabilities of the proposed approach appropriately.

Next, I would like to see a discussion of both the rendering and inference times of the proposed approach, which I expect to be very, very slow. What FPS can be achieved at what resolution? How long does it take to train the network?

*Final review*.
I recommend this paper for acceptance. Though I would like to see more comprehensive results in the form of continuous camera trajectories and the context videos, the method is sound and I am convinced that this method performs the way the authors claim it does. The only "shortcoming" of the proposed method is its specificity to the problem at hand, i.e., it is not a fundamental contribution that would enable new approaches across applications in vision, for instance. In particular, this method is quite specific to dynamic novel view synthesis where a parameter-based model is available, which is the case for faces and bodies, but not for general objects. Nevertheless, the contributions are clear, the results are cool and the paper is well written, and I believe this to be a valuable contribution to the community.
limitations_and_societal_impact: I would have appreciated an analysis of runtimes, see "Evaluation" above.

The authors adequately discuss the societal impact of their work.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes a method for dynamic human reconstruction and rendering using sparse view videos. 
The high-level strategy is to learn both a neural radiance field and an SDF field with mutual constraints in a canonical frame from temporal observations. Moreover, for generating live rendering and surface results, pose and position conditioned SDF offsets are also learned for better live frame surface reconstruction results. 
To guarantee robust surface reconstruction, an implicit generative human model was used as the inner layer which not only incorporates strong human shape constraint, but also provides semantic correspondences for accurate warping between live frames and the canonical frame. Another benefit of using an underlying implicit human model is that shape and pose editing can be achieved through interpolating in the parametric body space, which was not demonstrated in previous works.
main_review: Reconstructing/Encoding dynamic humans from light-weight inputs is a very challenging but important topic that has strong potential usage in AR/VR, telepresence, etc.
 
The paper writing is clear and easy to follow.
 
The overall idea of the paper is novel, the results in the paper look nice, the quantitative comparisons are impressive. 
 
One of my concerns is that the video results are not well demonstrated, almost all the video results are not very continuous. For example, in video 04:17, the side view rendering results are simply skipped, and this happens to all the results of the PeopoeSnapShot dataset. Moreover, for RenderPeople dataset results in video 01:27, it's easy to get continuous frames for training and rendering, but the videos results are not very continuous, please demonstrate full video results without skipping any frames. 
 
Moreover, I think video comparisons with NeuralBody should be provided to enhance the effectiveness of the proposed method on free-viewpoint rendering and dynamic geometry reconstruction results.

Minor issue:
The caption of Fig.4: botton -> bottom
limitations_and_societal_impact:
review_title: null

Meta-review: The submission has received 4 positive final ratings: 6, 7, 7, 8.
The reviewers overall were excited about the method and the idea of combining implicit representations with explicit priors, and also acknowledged strong empirical results and solid presentation. The remaining questions and concerns were mostly addressed in the rebuttal: the reviewers were mostly satisfied, but left some recommendations for further improvements (the authors are recommended to follow them while preparing the camera ready version). 
The final recommendation is to accept as a spotlight.
Decision: Accept (Spotlight)