Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed several additions onto the PIFU architecture that makes it perform better when trained and evaluated with noisy sparse RGBD views. The paper has three major contributions. 1.  A two scale PIFU model to render detail at different scales and to focus on the face of the human avatar. 2. An extension of PIFU to take in noisy RGBD data from real world sensors. 3. A demonstration of the results on 3 Kinects
strengths_and_weaknesses: The paper is very well written. It provides competitive qualitative and quantitative results with state of the art. The paper is very well written and include useful figures and diagrams. One useful novel contribution their method includes merging the occupancy maps of the face and body. I think this might be the most general contribution in the paper as it stands. The separate modeling of the face and body with their own occupancy fields might also be a good way to optimize the number of parameters needed for high fidelity body construction. The approach is just taking some rather straightforward impovements onto the existing PiFU HD, so the technique itself isn't particularly original. However, it does define an interesting use case for PiFU, sparse RGBD reconstruction with Kinects and demonstrates noticeable improvements on it. I will note that I am not extremely familiar with the latest literature surrounding PiFU so may be missing some related work here, but the the contributions are somewhat novel and useful to future work on digital humans.

One weakness is the lack of qualitative examples describing the weaknesses noted in Section 5. It would be nice as a reader for the paper to demonstrate visually why hands remains challenging. 
questions: * Some of the text on the figure is so small it can be hard to read when written out.
* In Figure 9, what does "Ours" have a different background
* For the limitations section, did the authors try to train a 4 scale occupancy model with seperate occupancy models to model each hand? It sounds like the authors may have experimented with doing so, but failed to get good results. If so, this information would be very helpful to at least include in the paper's supplement.
limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a method for reconstructing 3D human models from sparse RGBD inputs. The proposed method is built upon PIFu. To eliminate the negative impact of depth noise, the authors introduce a multi-task learning framework that incorporates the complementary nature of  depth denoising and 3D reconstruction. The authors also introduce an additional MLP for facial regions to further improve the reconstruction quality of human faces. Experiments show that the proposed method can generate high-quality 3D human models from sparse RGBD sensors. 

strengths_and_weaknesses: Strengths:

* The results look impressive. Given noisy depth measurement from sparse RGBD sensors, the proposed method is able to reconstruct clean 3D models with high-quality geometric details. 

* The main contributions of this paper are convincingly validated. The authors present extensive experiments throughout the paper. 

* This paper is overall well-written and easy to follow. 


Weaknesses:

* The novelty of this paper is a bit of weak. In the introduction, the authors claim the two-scale PIFu representation as one technical contribution. However, a similar design has already been proposed in PIFuHD. I understand that the fine-scale network in PIFuHD is still designed for the whole body while the fine-scale MLP in this paper only focuses on the facial region, but I would regard this difference as engineer choices instead of a "technical contribution". 

* No textured results are demonstrated. The authors are encouraged to present the textured models because it is a more intuitive way to evaluate the geometric reconstruction accuracy. 

* To evaluate the multi-task design, the authors simply remove the depth denoising task. I think a better baseline for this experiment would be a sequential framework where the depth maps are firstly refined by the depth denoising module and then fed into the 3D reconstruction network. 

* Missing citation: Zheng et al. PaMIR: Parametric Model-Conditioned Implicit Representation for Image-based Human Reconstruction. IEEE T-PAMI. 

******************************************************************
Final comments:
Thanks for the authors' response which addresses some of my concerns. After reading other reviews and the rebuttal, I would like to keep my initial rating. The authors should add their additional experimental results into the final paper if the paper gets accepted finally. 
questions: I am skeptical about the experiments in Fig. 3. In the main text of the paper, the authors emphasis that the task of depth denoising preserves local geometric details while 3D reconstruction provides global topology guidance. However, in Fig. 3 (g), which is the results of solely depth denoising, the refined depth looks much smoother than (h) and does NOT preserve local geometric details. This contradicts with the main insight of this paper and makes me feel that the geometric details is acturally produced in the 3D reconstruction task rather than the depth denoising task. I think the authors should discuss more about the experiment in Fig. 3 and provide more results if possible. 


limitations: The authors have discussed the limitations in Sec. 5 and the potential societal impact in the supplement material. No more improvement is needed.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a 3D reconstruction method of clothed humans from sparse RGBD inputs. This work argues that jointly solving depth refinement and PIFu-based 3D reconstruction is mutually beneficial with empirical evidence. For effective information fusion between RGB and depth, the paper also propose cross attention module and geometry aware module. Additionally, the paper further improves the fidelity of face reconstructoin by processing face and body separately and fused them back into a single 3D model. With THuman2.0 dataset, the proposed approach outperforms existing approaches and the ablation study validates their technical contributions.

strengths_and_weaknesses: The strengths of the paper can be summarized as follows:
- The qualitative results are impressive. Given noisy RGB-D inputs, the proposed method successfully generates high-resolution and complete 3D geometry of clothed humans. 
- The paper presents extensive experiments to show the effectiveness of the proposed approach over the existing approaches. 
- The proposed rgb and depth fusion module are interesting and shows improvements in performance. 

However, there are several weaknesses in this work:
- While the paper argues that the multi task learning of depth refinement and 3D reconstruction is the key contribution, the experiments do not necessarily support this claim. More specifically, it is not clear if solving them together as multi-task is critical or simply providing cleaner depth map is beneficial for the following steps. To validate this, I would recommend running experiments by feeding refined depth that is separately predicted as isolated module into the proposed pipeline. Without such an experiment, the key contribution of this work remains unclear.
- This paper misses one highly relevant paper, JIFF, where PIFu-like model predicts body and face regions separately and fuse them to produce high-quality reconstruction (although they are based on RGB inputs). Given the existence of this work, the contribution of two-scale PIFu would diminish. 

JIFF: Jointly-aligned Implicit Face Function for High Quality Single View Clothed Human Reconstruction
Yukang Cao, Guanying Chen, Kai Han, Wenqi Yang, Kwan-Yee K. Wong, CVPR 2022

While the results are impressive, the extent of the paper contributions remain unclear with the current form. Thus, I would stay on the fence, slightly inclined towards rejection.
questions: Please provide evidence that multi-task learning is beneficial, not depth refinement itself.

Other comments:
- I would recommend not using different background color for ours in the qualitative results. 
- L157: I' and D' do not appear anywhere else.
- Please clarify how to get the input mask.
- L30: minimally clothed
- L297: not small?
limitations: The paper does not discuss limitations or its societal impact. Discussion would be highly recommended.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper propose a method for human 3D reconstruction. The main contribution includes: 1. a geometry-aware PIFu method for human reconstruction from RGBD images; 2. a two-scale method dealing with body and face separately and then fusing the results; 3. experiments results showing the effectiveness on Kinect captured data.
strengths_and_weaknesses: Strengths:

+ The experimental results are good. Clothing deformation and facial features are reconstructed better than compared methods.

+ The ablation study is extensive, showing the necessity of each proposed module.

Weaknesses:

- The method seems to be too complex, and more like stacking existing methods and modules, limiting the novelty of the paper.

- The writing needs significant improvement: (1) The pipeline figure 2 needs to be simplified. Showing too many details in one single figure makes readers get lost. (2) The math should be simplified, too. Instead of defining a lot of symbols, it would be good if the author could describe the key idea (or even for a simplified pipeline) using very simple equations. Complex details could go to the supplementary material. (3) I would not recommend using "\copyright" as the concatenation operation, because it already has other meanings. ‖ could be a better choice.

- Since the method deals with face and body separately, it would be good if the comparison with face reconstruction methods is included in the experiments.

- The related work could be expanded. Here I listed some relevant papers, but the author could include more:
[a] Real-time Deep Dynamic Characters, SIGGRAPH'21;
[b] Texmesh: Reconstructing detailed human texture and geometry from rgb-d video, ECCV'20;
[c] MonoClothCap: Towards Temporally Coherent Clothing Capture from Monocular RGB Video, 3DV'20
questions: 1. Many modules are used in the pipeline. Which of them are completely novel? Which are adapted from existing works?
2. How are the parameters selected?
limitations: The authors have discussed the limitations. However, it is unclear whether the method works in some challenging scenarios. E.g., loose dressing, eye glasses, etc.
review_title: null

Meta-review: metareview: All reviewers were in favor of acceptance. The AC examined the paper, reviews, and author response, and is inclined to agree. The AC would encourage the authors to incorporate their responses to the reviewers into the final version of the paper. 
confidence: Certain
Decision: Accept