TotalSelfScan: Learning Full-body Avatars from Self-Portrait Videos of Faces, Hands, and Bodies

Abstract: Recent advances in implicit neural representations make it possible to reconstruct a human-body model from a monocular self-rotation video. While previous works present impressive results of human body reconstruction, the quality of  reconstructed face and hands are relatively low. The main reason is that the image region occupied by these parts is very small compared to the body. To solve this problem, we propose a new approach named TotalSelfScan, which reconstructs the full-body model from several monocular self-rotation videos that focus on the face, hands, and body, respectively. Compared to recording a single video, this setting has almost no additional cost but provides more details of essential parts. To learn the full-body model, instead of encoding the whole body in a single network, we propose a multi-part representation to model separate parts and then fuse the part-specific observations into a single unified human model. Once learned, the full-body model enables rendering photorealistic free-viewpoint videos under novel human poses. Experiments show that TotalSelfScan can significantly improve the reconstruction and rendering quality on the face and hands compared to the existing methods. The code is available at \url{https://zju3dv.github.io/TotalSelfScan}.

Reviews and author responses:

 Thanks for the authors’ effort in the feedback. All my concerns have been answered. I would recommend accepting this work.

 We thank the reviewer for the valuable comments and will add the discussions below to our revised paper.


> More qualitative comparisons and analysis on the non-rigid ray transformation 

We add more qualitative comparisons of the non-rigid ray transformation in Figure 2 in the link https://sites.google.com/view/totalselfscan. The results show that the ray transformation significantly improves the image quality. Without the ray transformation, there will be severe artifacts on the head, hands, and edge of the body. The reason is that the self-rotating video includes very limited human poses, and each point on the body is seen from a very limited range of view directions. Therefore, the color field network has difficulty producing high-quality rendering under novel human poses. Replacing the input view directions with global ones is a reasonable way to improve the generalization.



> The prior of pose-dependent deformations
 
To model pose-dependent deformations, we may leverage the existing multi-view human motion datasets to learn a generalizable pose-dependent deformation regressor, which can be conditioned on the human pose and canonical human geometry feature. Once learned, the generalizable regressor can be applied to new input data or can be further finetuned on the input data to improve the results.

 We thank the reviewer for the valuable comments and will add the discussions below to our revised paper.



> The use of parametric models

The use of parametric models indeed makes the problem easier but will not harm the applicability of the proposed method as we focus on human body reconstruction. 

The SMPL+H model gives reasonable hand geometry but it is quite coarse and lacks details. In Figure 1 in the link https://sites.google.com/view/totalselfscan, we show the comparison of hand geometry from the SMPL+H model and our reconstructed model. The results show that our method produces more detailed hand geometry such as bones and palm print.

 We thank the reviewer for the valuable comments and will add the discussions below to our revised paper.

> Potential of the FLAME model


We thank the reviewer for the suggestion. As shown in RigNeRF [1] and IMavatar [2], the FLAME model can provide fine-grained face animation, given data with sufficient facial expressions. However, the data we currently collect does not contain rich expressions. In future work, we will collect the corresponding data and try to animate the face using the FLAME model.




> More additional details of the method

We will add more details about the method and update the overall diagram in the revised paper. The $L^{body}$ loss equals the $L^p$ loss in Equation (10) when the part $p$ denotes the body.


> Non-rigid ray transformation

To avoid unexpected artifacts under novel human poses, the non-rigid ray transformation makes the color unchanged with view direction, which sacrifices the view-dependent effect and makes the rendering less natural. Thus, there is a trade-off between better generalization to novel poses and more photo-realistic rendering. 


> Comparison with traditional graphics pipeline

The traditional multi-view reconstruction methods cannot deal with the monocular self-portrait video in which the human is moving. A recent work VideoAvatar[3] proposed to deform a template body mesh to fit the image observations (e.g. silhouettes), but the reconstruction is less realistic due to the limited geometry accuracy and blurred texture map. 


### References

[1] Athar, ShahRukh, et al. "RigNeRF: Fully Controllable Neural 3D Portraits." CVPR 2022.

[2] Zheng, Yufeng, et al. "Im avatar: Implicit morphable head avatars from videos." CVPR 2022.

[3] Alldieck, Thiemo, et al. "Video based reconstruction of 3d people models." CVPR 2018.

 The paper presents a new method that creates full-body avatars from self-portrait monocular videos. The proposed method specifically models the body, the head, and the hands, the neural volumetric renderings of which are later fused together. Qualitative and quantitative evaluations demonstrate the superiority of the proposed method. Paper strengths:
- The proposed new method demonstrates clear progress of neural human avatars, achieving sharp and clear renderings under novel views and novel poses.
- The proposed method is compared against strong existing baselines and ablated carefully.

Paper weaknesses:
- Despite the FLAME model being used for the head, its potential does not seem to be fully utilized. The facial expression remains static in the final results which significantly increases the uncanniness.
- The method description could benefit from more additional details. For instance, the latent code is mentioned several times but there lacks a description of it in the overall diagram. The $L^{body}$ loss was not defined in the main text.
- The non-rigid ray transformation seems to make the texture static across view directions. Was this the reason why the results seem quite rigid? Also, how does the proposed method perform when compared with a traditional graphics pipeline such as deformation transfer? It would be great if the authors could address the above mentioned points in the weaknesses section. There are no potential negative societal impact of this paper.

 This paper proposes to reconstruct the human body with detailed hands and face, leveraging separately recorded hand and face videos. Strengths
1. The proposed pipeline is sound. Leveraging separate videos for body, face and hands at different scales makes sense; the fusion of geometry and appearance is well designed; the use of non-rigid ray transformation technique for free-view novel pose synthesis is reasonable.  
2. The experiment results show more visually appealing results.
3. An additional SynTotalHuman dataset is proposed.

Weaknesses
My major concern is related to the use of parametric models as the foundation for reconstruction that undermines the technical significance of the paper. The reconstructed hands are claimed to be more refined than prior works, given that SMPL+H is used. Even without any replacement estimation, the SMPL+H already gives good geometry. This is particularly questionable as the examples shown all have bare hands.  Please see the Weaknesses section. No additional limitations or potential negative societal impact.

 This work aims at reconstructing the full-body human from several monocular self-rotation videos including the face, hands, and body videos. Compared to the priors, this work improves the local details significantly with the acceptable cost of several extra part videos and more training time. The experiment results prove the effectiveness of the proposed framework and each introduced component. Strengths

- The proposed multi-part networks can utilize information from different part videos to learn better local details. The ablation study in  Figure5(a) and Table3 show the effectiveness of multi-part networks. I believe part-based data and modeling is the right way to enhance the details for complex objects like humans.  

- To connect the observation space and the canonical space, this work first uses part-based deformation fields to transform the sample points from each observation space, and then performs the geometry and appearance composition to generate a unified model.
 Sufficient experiments and comparisons have been done to show the superiority of TotalSelfScan.

- Two new datasets are introduced for part-based model evaluation. I believe they can also benefit the computer vision/graphics community.

- Limitations and reasonable potential future solutions have been well discussed. I believe these can also benefit the following works. 

Weaknesses

- The non-rigid ray transformation seems a nice strategy to alleviate the unnatural color artifacts. I would expect more qualitative comparisons and analysis on this part.
 Overall, this work is easy to follow and useful for real human reconstruction applications. I am interested in the prior of pose-dependent deformations discussed in the limitation. I think it is the key limitation of TotalSelfScan when applied to animation. I would expect more discussion on this part. The limitations and potential negative societal impact have been well described.

Meta-review: This paper was reviewed by three experts in the field. Based on the reviewers' feedback, the decision is to recommend the paper for acceptance to NeurIPS 2022. 

The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper. For example, more discussion can be added on the key limitation of TotalSelfScan when applied to animation. The authors are encouraged to make the necessary changes to the best of their ability. We congratulate the authors on the acceptance of their paper!

Decision: Accept