Geometry-aware Two-scale PIFu Representation for Human Reconstruction
Authors: Zheng Dong
Ke Xu
Ziheng Duan
Hujun Bao
Weiwei Xu
Rynson W. H. Lau
Venue: NeurIPS 2022 Accept (2022)
Keywords: 3D human reconstruction
depth denoising
multi-task learning
two-scale PIFu representation
geometry-aware features

Abstract: Although PIFu-based 3D human reconstruction methods are popular, the quality of recovered details is still unsatisfactory. In a sparse (e.g., 3 RGBD sensors) capture setting, the depth noise is typically amplified in the PIFu representation, resulting in flat facial surfaces and geometry-fallible bodies. In this paper, we propose a novel geometry-aware two-scale PIFu for 3D human reconstruction from sparse, noisy inputs. Our key idea is to exploit the complementary properties of depth denoising and 3D reconstruction, for learning a two-scale PIFu representation to reconstruct high-frequency facial details and consistent bodies separately. To this end, we first formulate depth denoising and 3D reconstruction as a multi-task learning problem. The depth denoising process enriches the local geometry information of the reconstruction features, while the reconstruction process enhances depth denoising with global topology information. We then propose to learn the two-scale PIFu representation using two MLPs based on the denoised depth and geometry-aware features. Extensive experiments demonstrate the effectiveness of our approach in reconstructing facial details and bodies of different poses and its superiority over state-of-the-art methods.

Geome Two-scale PIFu Representation for Human Reconstruction
Zheng Dong' Ke Xu² Ziheng Duan'
Hujun Bao'
Weiwei Xu*!
Rynson W.H. Lau?
State Key Lab of CAD&CG, Zhejiang University
2 City University Hong Kong
Abstract
Although PIFu-based 3D human reconstruction methods are popular, the quality of recovered details is still unsatisfactory. In sparse (e.g.. 2. RGBD sensors) capture setting, the depth noise typically amplified in the PIFu representation, resulting flat facial surfaces and geometry-fallibe bodies. In this paper, we propose novel geometry-aware two-scale PIFu for 3D human reconstruction from sparse, noisy inputs. Our key idea is to exploit the complementary properties of depth denoising and 3D reconstruction, for learning a two-scale PIFu representation to reconstruct high-frequency facial details and consistent bodies separately. To this end, we first formulate depth denoising and 3D reconstruction as multi-task learning prob- lem. The depth denoising process enriches the local geometry information of the reconstruction features, while the reconstruction process enhances depth denoising with global topology information. We then propose to learn the two-scale PIFu representation using two MLPs based on the denoised depth and geometry-aware features. Extensive experiments demonstrate the effectiveness of our approach in reconstructing facial details and bodies of different poses and its superiority over state-of-the-art methods.
Introduction
Three-dimensional human reconstruction, which aims to obtain dense surface geometry from single- view multi-view human images, is a fundamental topic in computer vision and computer graphics. While reconstructing high-fidelity 3D human models possible using commercial multi-view/strreo software under the customized studio setting [14, 58, 81, 29, 85], it is highly desirable to lift the studio setting constraint, which may be inaccessible to most users. Low-cost RGBD sensors have recently become popular in 3D human reconstruction, and tracking-basse to fuse the depth data from RGBD sensors for reconstruction [66]. During fusion, the estimation of non-rigid human body deformation is essential to improve the reconstruction quality [65, 101]. However, it technically challenging to ensure the stability of the depth fusion algorithm, due to occlusions and severe noise in the depth data.
methods are developed
Recently, learning-basce 3D human reconstruction methods have significantly simplified the capture setting. The parametric human model [60, 69, 67] is introduced to reduce the modeling difficul- ties from complex poses. After training on images-to-mode pairs, methods [46, 105] may even reconstruct 3D human shapes from single images. However, these methods typically only obtain minimally clothed human bodies. As detail requirements increase, the focus learning- based human reconstruction methods has been shifted to the implicit representation, e.g., pixel-aligned implicit function (PIFu) [73, 74, 40]. PIFu-base based methods [73] can reconstruct human bodies with different types of details (e.g., hair and clothing) without utilizing predefined templates. However, they often produce topology errors in the reconstructed human models, especially the regions
*Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

that are invisible or where the input depth is highly noisy (e.g., hairs). While PIFuHD [74] partially alleviates this problem by synthesizing the body's normal maps at both front and back sides, the details reconstructed from the synthesized normals may not be consistent with the target. Hence, some methods [40, 100] resort to the multi-view feature fusion scheme to reduce these topology errors.
Although implicit methods develop fast, we observe that the quality of their recovered details is still unsatisfactory under the sparse capture settings (e.g.33 3 RGBD sensors). Due to the less overlapping between sparse views, the input depth noise is typically amplified, increasing the difficulty of performing stable reconstruction. In addition. due to the ineffective fusion of RGB and depth features, these methods may not easily reconstruct feegffrey details. As result, we often observe flat or incorrect facial surfaces, body geometries with topology errors, as shown in Fig. I(b,c,)).
Jack
(A) Raw RGBD
(b) PIFu [73]
(c) PIFuHD [74] IPNet [4]
(c) Ours
(f) Our Additional Results
Figure 1: Given sparse and noisy inputs, with one of the three RGBD views shown in (a), existing methods such as Multi-view RGBD-PIFu [73] (b), PIFuHD [74] (c) and IPNet [4] (d) tend to produce over-smoothed facial details (b,d) topology errors (c,d), due to the amplified noise in sparse views. Our method learns the geometry-awan two-scale PIFu representation, which can produce vivid
facial/hair details and accurate bodies under different poses (e,f).
In this paper, we propose a geometry-aware two-scale PIFu representation for reconstructing digital humans from sparse, noisy inputs. Our method is based two observations. First, depth denoising and 3D reconstruction are complementary to each other. The former task preserves local geometric fidelity, while the latter task provides global topology guidance. Second, while function of high complexity may not be easy to express, it is much easier to approximate it piecewise (e.g., in two parts). Inspired by these observations, multi-task learning problem. The two tasks can work together to further improve the reconstruction quality. We encode RGB and depth separately, and use fused features perform depth denoising, enriching deep features with local geometric information. Based on the denoised depth and geometry- aware features, we propose using two ML.Ps to represent the PlFus for the face (particular region) and the body, respectively. This separate modeling increases the network capacity for handling details at different scales. As shown in Fig. (cc f), our method can produce results with high-fidelity body and facial details for different actions. Our main contributions are:
first formulate depth denoising and 3D reconstruction
1) We propose novel scometry-award PIFu method for digital human reconstruction from sparse, noisy RGBD images. Our approach exploits the complementary properties of depth denoising and 3D reconstruction to learn robust geometry information while suppressing the input noise.
2) We propose to learn a two-scale PIFu representation based on geometry-aware features, by using two individual ML.Ps for the face and the body separately. The two-scale formulation enhances the network capacity in producing high-frequent facial details and smooth body surfaces.
3) Extensive experiments demonstrate that the proposed method can produce high-quality digital human reconstruction results, based on noisy depth maps taken by three Azure Kinect sensors.
Related Works
Tracking-basee Human Reconstruction methods [50, 66, 109, 65, 30, 19, 49, 18, 101, 33, 79, 100] track human motions and infer the non-rigid deformations of the references reconstruct 3d human mesh in temporal fusion manner. The references are typically parameterized as the 3D poses/skeletons [24, 98, 99, 92, 101, 33, 34, 55) and/or parametric body models [7, 102, 101, 45] (e.g.,SPLL [60, 69, 45]). Some methods combine tracking with segmentation [59] or optical flow estimation [63] to help compute the references for reconstruction. To tackle the occlusion and large motion problems, some methods develop high-end systems for the dense capture of human

---

performance, consisting of a large number (up to 100) of RGBD sensors [14, 58, 81] or custom color lights [29, 85] (e.g.. 1,200 individually controllable light sources the acquisition setup in [85].
Learning-based Human Reconstruction methods [108, 77, 23, 88, 64, 106, 83, 93, 100] leverage the neural 3D representation for reconstructing the geometry and/or texture details. Some methods obtain 3D human meshes from single RGB [77] or RGBD [88] image by incorporating the SMPL template [80]. which are ineffective for deformed human bodies. Some approaches reconstruct continuous results from RGB [91] or RGBD [107] videos, skeletal motions [32], but are limited by garments types and cannot obtain facial expression details. Other methods adopt the image-to- image translation pipeline to regress the 3D mesh via 2D estimations of intermediate textures [2], silhouettes [64], and depths [23], while some approaches jointly exploit 2D and 3D information, e.g.. body joints and per- per-pixel shading information [108], and 2D/3D poses and segmentation map [83].
These methods typically suffer from over-smoothed surfaces occluded regions.
Recently, the pixel-aligned implicit function (PIFu) [73] has attracted much attention for 3D human reconstruction due to its effective implicit representation, and PIFuHD [74] estimates normal maps to reduce the geometry errors in the occluded regions based on PIFu. Due to its success, many methods promote PIFu with voxel-alignment [104, 40, 37], deformation field [38, 41J, real-time approach [51J, illumination [3], monocular fusion [54]. sparse-view temporal fusion [100]. or apply it for point
clouds based human reconstruction [13, 4, 62]. Typically, PIFu-based methods lack fine facial details in the sparse capture setting due to its all-in-
implicit 3D representation learning. To tackle this problem, JIFF [9] employs the facial 3DMM [5] model as a shape prior, enhancing
features for detailed geometry information. However, their face reconstruction capacity is constrained by the 3DMM parameters, so that JIFF may not handle dense facial geometry well. In this paper, we propose the geometry- two-scale PIFu representation, which utilizes high-resolution RGB images to obtain the face geometric details and explicitly fuses the face and body occupancy fields for high-quality full-body reconstructed results.
Depth Image Denoising is essential for depth images captured from consumer-level depth sensors (e.g., Azure Kinect). Traditional filtering- based methods [47, 10, 17, 71, 36, 76, 35] enhance depth data from various sensors. Color or infrared images are used help depth image denoising [16, 96, 56, 68, 57, 22, 95, 61, 35]. These methods typically assume the intensity image to be edge-aligned with the depth image, and they tend to produce artifacts when this assumption is violated. Some methods [28, 75, 66, 15, 39, 11, 65, 31] fuse multiple frames to refine the depth images, but they tend to produce smoothhd depth images.
Learning-base depth image denoising methods are proposed based on dietionary-learning [48] and deep-learning [27, 53, 52, 44, 93, 78]. Recently, Yan el al. [93] propose the DDRNet for deep denoising using fused geometry and color images. Their method tends to produce inconsistent results. Sterzentsenko et al. [78] propose a sulf-supervised method that leverages photometric supervision from a differentiable rendering model to smooth the depth noise. However, the photometric supervi- sion lacks 3D geometry information and their approach tends to produce homogeneous results. In this paper, we propose to formulate depth denoising and 3D reconstruction as multi-task learning problem. The global topology information facilitates the depth denoising significantly.
Monocular Face Reconstruction methods aim to reconstruct personalized faces from monocular data. The parametric face model, e.g, 3D morphable model (3DMM) [6, 5, 72] multi-linear blend shapes [8, 84, 25], are typically used in conventional methods. However, these methods may not reconstruct accurate or dense facial geometry. Deep-learning-bel based methods [42, 97, 21, 26] that incorporate face landmarks or the 3DMM model for face reconstruction also tend to produce coarse reconstructions results. Other methods [82, 43, 21, 70, 103] estimate the facial dense shape instead of the low-dimensional template parameters. All these approaches only reconstruct faces. In this paper, we apply the PIFu representation and extend it to two scales for modeling human bodies and faces. Our formulation enables producing vivid facial expressions with accurate body reconstructions.
3 Proposed Method
We propose the geometry- aware two-scale PIFu method, to address the problem of PiFu-baser approaches that reconstruct flat facial and gelletry-fallible body surfaces in the sparse capture setting. First, to handle the noise issue of the sparse capture setting, we propose to formulate the depth denoising and the body reconstruction processes in the multi-task learning manner to exploit their complementary properties. Second, although the face only occupies small proportion of the whole
3

---

Selected Front Vew
Geomelry-aware PiFu-Body
M
Depth
Dap
Layer1
Poge
Face. Faca-ta-body Fusion
Layar?
Occapancy pues
Layer3
Fasttrne
Doporr
RGR 11
Baolbone Peatare Estraction Extencement
Dody First
DtrenSaamll
Hiph-rrsolattin PIFU-
PORt X
Solect Bici reglon Projection
UpGamblle
& Sogenet
Model
Map
Dcsupercy Plap
1GB yea Heparott
Enooder
Conpaane,
Body Sibaay
Feat's
Sampled Fastun vitta
Suparsiore
Concileration
Elrrant Addition: Mustplicata
Figure 2: Proposed method overview. Given sparse and noisy RGBDs as inputs, Geametry-avare PIFu-Bodh performs depth denoising and predicts the body occupancy field. High-resolution PIFu- Face predicts the face occupancy field with fine- tneegrained details. The body and face occupancy fields fused produce final results via the Face-to-body Fusion scheme.
human model, it typically contains more nigh-frequency (e.g., vivid expressions) than other parts and plays a vital role in assessing the reconstruction fidelity. To this end, we propose the two-scale PIFu representation to allocate more network capacity for face reconstruction.
As illustrated in Fig. 2, our network contains three parts: (L) the Geometry-aware PIFu-Body, Fo that predicts the body occupancy field O2 and the refined depth maps from the noisy RGBD images of A (i.e., three) perspective views; (2) the High-resolutiin from Fu and the high-resolution RGB image of the front view as inputs; (3) the light-weight Face-to-boot to-body Fusion, W that reconstructs the full human model by fusing the face and body occupancy fields (i.e., O₂ and Of).
PIFu-Face Facce F, which obtains the fine-grained
face occupancy field Oy, using refined depth map
3.1 Geometry-aware PIFu-Body: Fb
PlFu-based methods predict 3D occupancy fields in an implicit manner, enabling image-aligned features be aware of global topological information, which helps suppress depth noise. For example, in Fig. 3(e), the holes can be filled using only geometric supervision. However, this depth map tends be over-smoothed, and the reconstructed surface also lacks details (Fig. 3()). On the other hand, with only depth supervision, image-to to -moieeaaee depth denoising can fill holes and add details. However, the obtained depth contains incorrect details (e.g.. face, hands, and clothing regions shown Fig. 3(g) and Fig. 9b)), when lacking of 3D geometric guidance. Based on these observations, we formulate depth denoising and PlFu-based occupancy estimation in multi-task learning manner. It exploits the global topological information of the 3D occupancy field to guide the denoising process, and the local
details of refined depth improve occupancy estimation (Fig. 3(h,i.) and
Fig. (c.hh))
Inputs
Compartsons
Ablation Study
urs
(8)
(b)
(c)
(d)
(e)
(g)
(h)
(1)
Figure 3: Visualization of depth denoising and the subsequent reconstruction results. Raw RGB and Depth images (a.b). Results of two depth denoising methods, [93] and [78] (c.d). Our results of only using occupancy supervision (e.f). Our refined depth under depth supervision only (g). Our refined depth, its normal map, and full-body mesh (h,i.)).

---

Formulation of Fo. Given the triplet (T) of RGB (I), depth (D) images, body binary masks (M) from /V perspective views, and the query point X E R as inputs, we formulate the F, to predict both the body occupancy value σg E [0, and the refined depth maps Dry. as:
Fo(X, T) = (M (T,), xy). C()),
( (())((T))))) == for, Dyy. (1)
where v5,s) (-) is a mapping function that encodes T into multi-scale Geometry-aware features, X; =
n. (X) E R2, is the 2D projection of point X i-th view, and z; E RR is the depth of X in the local coordinate system of the i-th view. (x(X) = [24, pi (x() where p: px( = T( - B x:)) E
[-6,, is the truncated PSDF value as in [100]. B(,, x.) is the sampling function to obtain pixel-aligned 2D features B ( x;) and depth information Cx)), which are processed by the multi-view feature aggregation module A and further fed into the implicit function M, for occupaney querying. Meanwhile, the decoder D(-) processes features dig (T,) for depth denoising.
Geometry-aware Features: (T,). The geometry aware mapping (-(-) aims to exploit the complementary properties of depth denoising and occupancy field estimation. Hence, is expected to fuse the multi-modal RGB-D inputs effectively. To handle their modal discrepancy, we use two independent HRNets [87] (Fig. 2) to process the RGB (I) and depth (D) inputs respectively, where {I, D) will be pre-processed to filter out the background via the masks, i.e. M(1, D). To fuse the RGB and depth backbone features, we propose a novel Cross Attention Module (CAM) on the highest level of backbone features, and use the CAM output feature to guide the fusion of lower levels in a level-wise manner. We also propose a novel Geometry Aware Module (GAM) to enrich the CAM output features with frguufrryeency information. The enhanced features form the features b ( (T
Cross Attention Module (CAM). The CAM aims to fuse the RGB and depth features by computing their non-local correlations. Since depth typically noisier than RGB information, we first compute the self-attention map based on the RGB feature and then use it to reweight both RGB and depth features before they are fused. The architecture of CAM is shown in Fig. 4. of which the implementation is based on the non-local model [89] but we extend it to handle the RGB-D fusion. Specifically, the fused feature Y can be written as:
MUP
Festure
AGR
MLP
Fused Featurn
MAC
MLP
Query
Depith
RoSCBAM
B Concettraation
X
MelMdll
Figure 4: Cross Attention Module (CAM).
Y R. (F,) & K(F))) II Ra(ga(Fa) @ x(F)))
(2
where R, and R. represent two ResCBAMs [90] used for calculating the local attentions for fusion, || is the channel-wise concatenation and @ denotes the matmul operation. F, and Fa indicate the RGB and depth backbone features output by Layer4 (Fig. 2). x(F,)) computes the non-local feature affinity map as k(F,) = softmar (0( (2)² @ Φ (F))). where 0 and & are learnable linear embedding functions. gr. Id are two functions to compute the value features of F, and Fa.
Geometry Aware Module (GAM). The GAM aims to enrich the fused features of CAM with high-frequency information for reconstructing geometric details. To this end, we propose to calculate the depth contrasted feature between a local region and its surroundings to capture high- frequency depth variations, as shown in Fig. 5. Specifically, we split out the RGB and depth fea- tures: FY, F/. according to the channels number, and calculate the contrasted feature for F' as:
AIN Load Pertore
EDIN SDIN
ROB-
Fuser Festure
Poaluro
Pigct Frot
Daytt
Deneneraton
Bheren Sutnection
Figure S: Geometry Aware Module (GAM).
Ca = (f - (F()) (f((F) - (((E)))((((F))))) ((((()))
(3)
where fi denotes the local convolution with a 1x1 kernel and fy denotes the context convolution with a 3x3 (dilat rate=z). g is a global average pooling operation. || is the concatenation operation. The second item after || represents the local depth feature of the pixels relative to the global feature Fz. As a result, Ca amplifies high-frequency signals at depth transitions, benefiting the prediction of these details (e.g., hairs in Fig. 1). For F,, we maintain its information through local convolutions.
5

---

3.2 High-resolution PIFu-Face: Fy
Face regions typically have more high-frequency components than the body (e.g., mouth v.s.soft clothing) (Fig. 6(a.b.f). Even enhanced with depth denoising, the all-in-one PIFu (FF still struggles to represent high wigh-freeuence facial details (e.g., nose & mouth in Fig. 6(c,f). Hence, we propose to express the implicit function F in a piece-wise manner (ie., Fb and Fy) to reduce the complexity of joint occupancy estimation while producing vivid facial and body details. Specifically, we propose to learn the high-frequency facial PIFu representation (ay [0, 1)) conditioned on the high-resolution
face image ( cropped from the frontal view) and the corresponding denoised facial depth map D
rs
Fauss
Dick
Weigito 30 Po Tage
wit
(a)
(b)
(c)
(d)
(e)
(8)
Figure 6: Facial mesh closure FaceScape [94] dataset for training (a). Raw RGBD input images (b). Our reconstructed face model (w/ and w/o PIFu-Face) (c). The weight map M2 and 3D weights for Face-to-body Fusion (d). Comparison results on Face-to-body Fusion and PIFu-Face (e.f).
Formulation of Fy. Given the above inputs (denoted as T [I,, M2), we formulate the Fy along the query point X, E R3 within the face regions as:
Fy Ty) = (Y (T xx)))))))),
where Hy denotes the feature extractor for facial images Ty. The function (+). up-samples T to the same resolution as T. My is the implicit function for querying Oj. Cx(X,) ) is defined the same as C, (X), but use the up-sampled masked facial depth M, (U (D!,)) to compute pX()..
Facial Points Selection. To determine the facial points for inferring oy. we select the points among all the querying points of which the projection x E IR2 falls inside the bounding-box R, of D and absolute PSDF value is less than a as x, (Fig. 7(b)). We set flag vy(+) to mark the facial points
{
Xf E R2 & abs(z, B(D x/) <a else
8y (X) (X)
3.3 Face-to-body Occupancy Fields Fusion: W
Simply merging the reconstructed face and body (ie.. replacing d with o, for the facial points) would result in the discontinuity artifacts at the stitching (Fig. 6(e), 1st row). To address this issue, we propose to fuse O and O via adaptive weights calculated in 3D space. As shown in Fig. 6(d), compute 2D fusion weight map (M₃) in the x-y plane by eroding edges of the facial mask M,. Along the z axis, we compute the weights through a Gaussian distribution model of the PSDF values. Then, we formulate the joint probability distribution of the final fusion weight as:
Howe porn
Frantt depke Unep
B2lnd Fratuums
W WLO. MAO, Cripiii, MD)
Fbund bacal oecepuncy valte
(a) Points Sampling
(b) Occupuncy Fields Fusion
Figure 7: Equal facial and body points sam-
w = B(M x,) exp(-3 (PPpp--B-PPP/,,,,.. (I(D(),x,)))) X,)))), (6)
pling (a). Face-to-body fusion (b).
where -P.-.)) denotes the function to calculate the PSDF value, i.e., P(D = = B(D/ rf Xy) and the parameter 3 is set to 1 default. In Eq. 6, the first term yields smaller values from the center of the face region to its boundaries, which improves the smoothness around the stitching. The second term emphasizes the occupancy values computed by Fj before and after the face surface. Hence, we can leverage to fuse the σb and o as: W(os, w) w of + (1 w) ob.

---

3.4 Loss Function
We adopt the extended Binary Cross Entropy (BCE) loss [74] to supervise the predicted occupancy values db and o on the sampled body and facial points X [x,, Xy]. which can be written as:
Li Po
Σ Cb( + pr
Σ Lb(a),03),
X.ES.
X,ES,
where og. of the ground-truth occupancy values for X and x sets. Ly represents the BCE loss, Po and μ* are the weights to balance PIFu-Body and PIFu-Face.
f. So and S1 denote the sampled
Regularization Term. We propose a Regularization Loss (Ireg) to reduce the artifacts in depth- jumping regions during multi-views aggregation, as:
breg
L2(n(Xx, TX, X,ES;
Σ
where Sj E Sy the set of points projected on the depth-jumping regions (refer to the supplemental for details of obtaining these regions). The parameter is small random uniform 3D perturbation. n(x,, T) E 3 is the normal vector at Xs, defined as Fo( T)/ Fx, T))l2 Eq. encourages the normals of X, to be consistent with those of the points sampled in its neighborhood, hence enhancing the smoothness along the stitching boundaries.
Depth Denoising Term. We penalize the per-pixel difference between Dry and the rendered ground- truth depth map Dyt. We also penalize the error of caleulated normal maps prevent D, from becoming blurred. The loss for depth denoising is formulated as:
LD PD=p (x) PN (2(n, mg)
(9)
where , denotes the s-scale (S=4, the scale of v (T)) predicted depth value of D,f in pixel p. f E I$ is the normal vector of 1 in pixel p. where N, is the normal map computed from D, (p) and ngt are the corresponding ground-trute touund-truth value and vector. Ln and L2 are the smooth L1 loss and L2 loss. The weights pD, PN and A, are used for balancing different loss terms.
The whole loss function can be defined as L = LL + Areg Lreg + AD LD. where the weights Avegs XD are the balance terms. Refer to the supplemental for detailed implementation information.
Experiments
Datasets and Evaluation Metrics. We use the THuman2.0 [100] dataset which contains 500 high-quality 3D human scans to train and validate our network. We split the dataset into training and test sets with a ratio of 4:1. We rotate each scan along the yaw axis and render RGBD body portraits at every 6-degree rotation with CUDA acceleration. For PIFu-Face, we pretrain Fy using the FaceScape [94] dataset, which contains 3D head models of different people and expressions. We crop the face regions and make the cropped meshes watertight (Fig. 6(a)). For the input raw depth maps, following [20], we synthesize the sensor noise on Dyt to obtain D. For og and oÿ. we follow the sampling strategy of PIFuHD [74] sample body and facial points (Fig. 7(a)), and compute their occupancy values as the ground-truth labels.
We adopt the Point-to-Surface (P2S) distance(cm) and Chamfer distance(cm) for mesh, 1.2 (1e (1e=1) and Cosine distance (1e-3) for normals as the metrics to measure the errors between the reconstructed and the ground-truth surfaces. Lower metric values indicate better performance.
4.1 Comparisons with the State-of-the-arts
We compare method with six state-of-the-and human reconstruction methods, including the Multi-view RGBD-PIFu [73], PIFuHD [74], StereoPIFu [40], IPNet [4], DoubleFusion [101] and
Function4D [100]. Among them, the DoubleFusion [101] and Function4D [100] are tracking-
based, Multi-view RGBD-PIFu [73], PIFuHD (single RGB image) [74], StereoPIFu (stereo RGB
images) [40] are PIFu-based and the IPNet [4] is implicit function based.
7

---

(a)
(b)
(0) (d)
(c)
(h)
Figure 8: Qualitative comparisons on our captured real data, between the proposed method and five latest state-of sate-of-thee-ad art methods. RGBD images of the front view (a). Our refined depths (fused point
clouds) (b). Multi-view RGBD-PIFu [73] (c). PIFuHD [74] (d). IPNet [4] (e). DoubleFusion [101]
(f). Function4D [100] (g). Ours (h). Zoom in to see the details.
Quantitative Comparisons. Tab. ports the quantitative comparisons on our test set between the proposed method and four PIFu (or implicit (unction)-basee ap- proaches. Our method ranks in the first place under all metrics, exceeding the second-best results with a 16.72% redue- tion in Chamfer and P2S distances, 9.50% reduction L2 and Cosine distances. The PIFuHD [74] reconstructs 3D human bod- ies from a single RGB image, which cannot handle the topology errors, resulting in the low recon- struction accuracy. The IPNet [4] receives the fused point clouds as inputs, but it still cannot handle the significant noise issues in depths. also has fit the SMPLL, which may not handle the complex poses. The StereoPIFu [40] can produce reasonable results from the front view but cannot handle the topological errors hidden other perspectives. Obtaining stereo pairs from multiple views may be a solution, but it significantly increases the computational cost introduced by its 3D voxel features. For RGBD-PIFu [73], we use multi-view RGBDs as inputs to handle the topology errors. However, still fails to produce reliable results (e.g., artifacts in hairs, wver-seothhed faces) when depth maps contain larger noise. In contrast, our method achieves statc-of-the-an performance by learning the geometry-aware two-scale PIFu representation.
Nomnal
P25=110
Chamferx10 x10
12xx0011
Cosinex 10-
RGBD-PIFP [38]
0.3188
0:201
0.824
PIEGHD
1.7288
1.7423
0,512
1,576
StereoPIPu [40]
0,5832
1,193
IPNer [4]
0.8%6
0.7347
0,196
0.751
Ours
0.2652
0.3775
0.176
0.685
Table 1: Quantitative comparisons ofate-of-the-a
methods on our test dataset. The best results are marked in bold.
Qualitative Comparisons. Fig. 8 shows the reconstruction results of five existing methods and our method on our captured real data. Although refined depths Diy can be fused (e.g., TSDF-Fusion in [66]) to obtain 3D human models (Fig. 8(b)), the reconstructed results contain large holes and low- quality regions due to sparse inputs and miet inconsistencies. The multi-view RGBD-PIFu and the PIFuHD methods often suffer from floating geometry (Fig. 8(c)) and topology errors (Fig. 8(d)) due to non-negligible depth noise and the lack of other view information. For IPNet, even if take the fused point clouds (from 3 frames) as inputs, the topological errors are still obvious (Fig. 8(e)), and facial details are missing. We also test the DoubleFusion [101] on our captured data. As shown in Fig. 8(f), due to the low quality of the original depths and the sparse embed- -graph of the whole body, this volumetric fusion-based approach tends to smooth out regions where we expect to see the high-frequency information (e.g., faces). In addition, when a pose differs significantly from the initial pose, the reconstructed mesh tends to be over-stretched (blue boxes). The Function4D [100] tracks the former and latter frames for the current frame and fuses the multi-framee point clouds to produce
8

---

0.1261 new depth maps. When the motion changes are not small, their method is not easy to reconstruct reasonable high-frequency details (flat faces, hairs in Fig. 8(g)). Our approach leverages the depth denoising to produce robust depth (e.g., hairs in Fig. 8(h)) for the PlFu-based reconstruction process. Moreover, our two-scale PIFu represents the face and body separately to produce vivid details.
4.2 Ablation Study
Multi-task Formulation. To evaluate the ef- fectiveness of our multi-task design, we com- pare our geometry-aware PIFu with a sequential model that first performs the depth denoising and then the human reconstruction using two in- dividual networks. The two networks are trained individually. For depth denoising, we use an ab- lation version of our full model by removing the PlFu-body MLP and the occupancy supervision. For human reconstruction, we remove the depth supervision and compute the trunc- trune-TSDF from the input raw depth maps. As shown in Tab. 3, the performance of the sequential pipeline is lower than model on all five metries, which illustrates that our geometry-aware features benefit both two tasks. Besides, comparing the refined depths and normals of the sequential model (Fig. 9b)) with our model (Fig. 9c)),, we find solely depth denoising tends to produce incorrect geometric details (e.g., error depths and normals), as shown in Fig. (h,d), which further results in geometric errors (Fig. (g)) in the reconstruction process.
Moakc
Moh
Nomal
P25
Charders:
42x10-1
Coisex
Wio GT Depeh
0,3165
0.K88
0.211
0.001
CAM -s AR
0,2826
0.2539
0.194
0.759
sol LAM
622665
0,2110
0.175
0.004
WAG CAM
0.7001
0:3877
0.301
0.771
Ours
0.3653
0.375
0.176
Table 2: Ablation study. AR indicates the combi- nation of ASPP [12] and ResCBAM [90]. CAM -> AR indicates replacing CAM with AR.
Methods
Mush
Normal From reconstructed mesh)
Retined Depth
P25x 2
Chamferx 10 -2 J. L2xx10 -1
Cosinex 10-3
L1x 1
Sequentia model
0.3125
0.321
0.212
0.800
Ours
0.2652
0.2775
0.176
B.685
0.0911
Table 3: Quantitative comparisons between the sequential model (depth denoising and 3d reconstruc- tion in sequential) and our proposed geometry-aware PIFu.
Imputs
Refined depths and nommass ( esequentaal model v.s ours)
3D mesh
(4)
(c)
(d)
(e)
(
(g)
(h)
(i)
Figure 9: Qualitative comparisons between the sequential model and our proposed geometry-aware
PIFu. RGBD imputs of the front view (a). Refined depths and normals of the sequential model (b),
and our model (c). Normal error maps between the Ground-truth and the sequential model (d), model (c). Ground-truth normals (f). Reconstructed 3D results of the sequential model (g) and our
model (h). Ground-truth meshes (i). Zoom in to see the details.
Depth & Occupancy Supervision. Tab. 2(1st row) shows that by removing the depth denoising task, the performance drops by 14.84% and 18.98% in terms of Mesh and Normal metrics. The decoder D(.) produces ovor-smo depths and the reconstructed results also lack details, as shown in Fig. 3(e.f) and Fig. 10(c). On the other hand, by removing the 3D supervision, the denoised depth often incorporates incorrect details (e.g., face, hands and clothing regions); See Fig. 3(g) and Fig. 9b). This experiment further verifies the effectiveness of our multi-task formulation.
CAM & GAM. The 2nd-to-4th rows of Tab. 2 evaluate our CAM and GAM modules for RGBD fusion and geometry- aware enrichment, respectively. By replacing the CAM with the common combination of ASPP [12] and ResCBAM [90] (i.e., AR in Fig. 2) or removing the GAM, both the performances drop. We also observe that using AR without GAM yields worse performance on average than removing the GAM. Fig. 10(d illustrates the differences, where we can see that these two modules help reconstruct the freuurreyenncy details (e.g., clothes folds of Fig. 10(h.))).
PIFu-Face: Fy. The visual comparisons in Fig. 6(c.f) show that the PIFu-Face, i.e., F, significantly improves the ability of our method to reconstruct high-frequency facial details.
Face-to-body Fusion: W. The visual comparison in Fig. 6(e) shows that our weighted face-to-body fusion can eliminate the artifacts generated by simply merging the reconstructed face and body.

---

Regularization loss: Leeg. Fig. 10(g) shows that without the proposed Lreg, our method tends to produce jagged noise on stitched boundaries where depth may be consistent.
Imputs
win GT Depth
CAM-AR
wel GAM
who Loe
Our
Flct
Froet
[a]
(b)
(c)
(c)
(g)
(h)
Figure 10: Ablation study. RGBD images of two views (a,b). Reconstruction results and refined depth of ablated versions (c-g). Our results (h,i). Zoom to see details.
4.3 More Experimental Results
Geometry and Texture. As shown in Fig. 11, we provide more of our 3d geometric models along with corresponding textured results. We can see that the textured results are of high quality and well aligned with our reconstructed geometries. Moreover, our method is able handle the loose dressing and respond to the boundary of eyeglasses wearing to some extent (Fig. ((c.d,e,)).
(a)
(b)
(d)
Figure 11: Examples of reconstructed and textured results. 3D geometric models (a,c,e). Textured models (b,d,f). The textured results are generated using MVS-Texturin [86] from the original captured RGBs and rendered via Blender [1].
Limitation for Hand Details. As shown in Fig. 12, it remains challenging for our method generate high- quality hand details when the hand is occluded, or the hand pose is complex. In our two-scale PIFu, we model the face surface from only the front view since the front image contains enough details for the face reconstruc- tion. In contrast, hands have more degrees of movement freedom, which makes them harder to locate and re- construct independently via extending additional scales based on PIFu from only one view. Combining hand parametric models with our two-scale PIFu may be feasible solution for hand modeling.
hands
hands
frout
back
front
bock
(b)
Figure 12: Difficulty for modeling hand details. Ours (a). Ground-truth (b).
5 Conclusion
In this paper, we propose the geometry-aware two-scale PIFu representation to reconstruct the digital human body with fine-grained facial details. The first novelty is that we formulate the depth denoising and implicit occupancy estimation in a multi-task learning manner, to exploit their complementary properties. The second novelty lies in that we formulate the two-scale PIFu via two MLPs to represent the face and body separately, to reduce the complexity of modeling oigh-frequency facial details. Finally, lightweight face-to-body fusion scheme fuses the face and body occupancy fields to generate reliable reconstruction results of high fidelity. For the limitation of modeling high-qualin hand details, we are interested in investigating this topic the future and exploring a light-weight approach.
Acknowledgments: We thank all the reviewers for their constructive comments. Weiwei Xu is supported by NSFC (No. 61732016). This work was partially supported by GRF grant from RGC of Hong Kong (Ref.: 11205620). This paper is supported by the Information Technology Center and State Key Lab of CAD&CG, Zhejiang University.
10

---

References
[1] Blender. httpa://www. baaanderoor//. 10
[2] Thiemo Alldieck, Gerard Pons-Moll Christian Theobalt, and Marcus Magnor. Tex2shape: Detailed full
human body geometry from a single image. In /CCV, 2019.
[3] Thiemo Alldieck, Mihai Zanfir, and Cristian Sminchisescu. Photorealistic monocular 3d reconstruction
humans wearing clothing. In CVPR, 2022.
[4] Bharat Lal Bhatnagar, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll. Combining implicit function learning and parametric models for 3d human reconstruction. In ECCV, 2020. 2, 3, 7,8
[5] Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In ACM SIGGRAPH.
1999.
[6] Volker Blanz and Thomas Vetter. Face recognition based on fitting a 3d morphable model. IEEE TPAMI,
2003. 3
[7] Federica Bogo, Michael Black, Matthew Loper, and Javier Romero. Detailed full-body reconstructions of moving people from monocular rgb-d sequences. In ICCV, 2015. 2
[8] Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou. Facewarehouse: A 3d facial expression database for visual computing. JEEE TVCG, 2013. 3
[9] Yukang Cao. Guanying Chen, Kai Han, Wenqi Yang, and Kwan- Yee K Wong. Jiff: Joimtl--aligned implicit face function for high quality single view clothed human reconstruction. In CVPR, 2022. 3 [10] Derek Chan, Hylke Buisman, Christian Theobalt, and Sebastian Thrun. A noise-award aware filter for real- time depth upsampling. In Workshop on Multi-camera and Multi-modal Sensor Fusion Algorithms and Applications, 2008.
[11] Jiawen Chen, Dennis Bautembach, and Shahram Izadi. Scalable real-time volumetric surface reconstruc-
tion. ACM TOG, 2013. 3
[12] Liang-Chieh Chen, George Papandreou, Florian Schroff. and Hartwig Adam. Rethinking atrous convolu-
tion for semantic image segmentation. arXiv:1706.05580 05587, 2017.
[13] Julian Chibane. Thiemo Alldieck, and Gerard Pons-Moll. Implicit functions in feature space for 3d shape
reconstruction and completion. CVPR, 2020.
[14] Alvaro Collet, Ming Chuang, Pat Sweeney, Don Gillett, Dennis Evseev, David Calabrese, Hugues Hoppe, Adam Kirk, and Steve Sullivan. High-quality streamable free-viewpoint video. ACM TOG, 2015. 1,
[15] Yan Cui, Sebastian Schuon, Sebastian Thrun, Didier Stricker, and Christian Theobalt. Algorithms for 3d shape scanning with depth camera. IEEE TPAMI, 2012. 3
[16] James Dichel and Sehastian Thrun. An application of markov random fields to range sensing. In NewrlPS,
2005. 3
[17] Jennifer Dolson, Jongmin Back, Christian Plagemann, and Sebastian Thrun. Upsampling range data in dynamic environments. In CVPR, 2010. 3
[18] Mingsong Dou, Philip I.. Davidson, S. Fanello, S. Khamis, Adarsh Kowdle, Christoph Rhemann, Vladimir Tankovich, and Shahram Izadi. Motion2fusion: real-time volumetric performance capture. ACM TOG,
2017. 2
[19] Mingsong Dou, Sameh Khamis, Yury Degtyarev, Philip Davidson, Sean Ryan Fanello, Adarsh Kowdle, Sergio Orts Escolano, Christoph Rhemann, David Kim, Jonathan Taylor, et al. Fusion4d: Real-time performance capture of challenging scenes. ACM TOG, 2016. 2
[20] Péter Fankhauser, Michael Bloesch, Diego Rodriguez, Ralf Kaestner, Marco Hutter, and Roland Siegwart. Kinect v2 for mobile robot navigation: Evaluation and modeling. In ICAR, 2015. 7
[21] Yao Feng, Fan Wu. Xiaohu Shao, Yanfeng Wang, and Xi Zhou. Joint 3d face reconstruction and dense alignment with position map regression network. In ECCV, 2018. 3
[22] David Ferstl, Christian Reinbacher, Rene Ranfil, Mauhias Rüther, and Horst Bischof. Image guided depth upsampling using anisotropic total generalized variation. In ICCV, 2013.
[23] Valentin Gabeur, Sén-Stéhstiien Franco, Xavier Martin, Cordelia Schmid, and Gregory Rogez. Moulding humans: Non-parametric human shape estimation from single images. In /CCV, 2019. 3
[24] Juergen Gall, Carsten Stoll, Edilson De Aguiar. Christian Theobalt, Bodo Rosenhahn. and Hans-Peter Seidel. Motion capture using joint skeleton tracking and surface estimation. CVPR, 2009.
[25] Pablo Garrido, Michael Zollhöfer, Dan Casas, Levi Valgaerts, Kiran Varanasi, Patrick Pérez, and Christian Thcobalt. Reconstruction of personalized 3d face rigs from monocular video. ACM TOG, 2016. 3
[26] Baris Gecer, Stylianos Ploumpis. Irene Kotsia, and Stefanos Zafeiriou. Fast-ganfit: Generative adversarial network for high fidelity 3d face reconstruction. IEEE TPAMI, 2021. 3
[27] Shuhang Gu, Wangmeng Zuo, Shi Guo, Yunjin Chen, Chongyu Chen, and Lei Zhang. Learning dynamic guidance for depth image enhancement. In CVPR, 2017. 3
[28] Sigurjon Arni Gudmundsson, Henrik Aanaes, and Rasmus Larsen. Fusion of stereo vision and time-of- flight imaging for improved 3d estimation. International Journal of Intelligent Systems Technologies and Applications, 2008.
11

---

[29]
Kaiwen Guo, Peter Lincoln. Philip Davidson, Jay Busch, Xueming Yu, Malt Whalen, Geoff Harvey. Sergio Orts-Esc,,,no, Rohit Pandey, Jason Dourgarian, et al. The relightables: Volumetric performance capture of humans with realistic relighting. ACM TOG, 2019. 1,3
[30] Kaiwen Guo, Feng Xu, Yangang Wang, Ychin Liu, and Qionghai Dai. Robust non-rigid motion tracking
and surface reconstruction using 10 regularization. In /CCV, 2015. 2
[3]] Kaiwen Guo, Feng Xu. Tao Yu, Xiaoyang Liu. Qionghai Dai, and Yebin Liu. Real-time geometry. albedo,
and motion reconstruction using single rgb-d camera. ACM TOG, 2017. 3
[32] Mare Hahermann, I. Lingjie Liu, Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll, and Christian
Theobalt. Real-time deep dynamic characters. ACM TOG, 2021. 3
[33] Marc Habermann, Weipeng Xu, Michael Zollhoefer. Gerard Mons-Moll, and Christian Theobalt. Livecap:
Real-time human performance capture from monocular video. ACM TOG, 2019. 2
[34] Marc Habermann, Weipeng Xu, Michael Zollhofer, Gerard Pons-Moll, and Christian Theobalt. Deepeap:
Monocular human performance capture using weak supervision. In CVPR, 2020. 2
[35] Bumsub Ham. Minsu Cho, and Jean Ponce. Robust image filtering using joint static and dynamic guidance. In CVPR, 2015.
[36] Kaiming He, Jian Sun, and Xiaoou Tang. Guided image filtering. IEEE TPAMI, 2012. 3
[37] Tong He. John Collomosse, Hailin Jin, and Stefano Soatto. Geo-pifu: Geometry and pixel aligned implicit functions for single-view human reconstruction. In NeurlPS, 2020. 3
[38] Tong Hc, Yuanlu Xu, Shunsuke Saito, Stefano Soatto, and Tony Tung. Arch++: Animation-ready clothed
human reconstruction revisited. In ICCV, 2021. 3
[39] Peter Henry. Michael Krainin, Evan Herbst, Xiaofeng Ren, and Dieter Fox. Rgb-d mapping: Using kinect-style depth cameras for dense 3d modeling of indoor environments. International Journal of Robotics Research, 2012.
[40] Yang Hong. Juyong Zhang, Boyi Jiang, Yudong Guo, Ligang I Liu, and Hujun Bao. Stereopifu: Depth
aware clothed human digitization via stereo vision. In CVPR, 2021. 1, 2, 3, 7, 8
[41] Zeng Huang, Yuanlu Xu, Christoph Lassner, Hao Li, and Tony Tung. Arch: Animatable reconstruction of clothed humans. In CVPR, 2020. 3
[42] Patrik Huber, Guosheng Hu, Rafael Tema, Pouria Mortazavian, P Koppen, William Christmas, Matthias Ratsch, and Josef Kittler. A multiresolution 3d morphable face model and fitting framework, In VISAPP,
2016. 3
[43] Loc Huynh, Weikai Chen, Shunsuke Saito, Jun Xing, Koki Nagano, Andrew Jones, Paul Debevec, and Hao Li. Mesascopic facial geometry inference using deep neural networks. In CVPR, 2018. 3
[44] Junho Jeon and Seungyong Lee. Reconstruction-base pairwise depth dataset for depth image enhance- ment using can. In ECCV, 2018. 3
[45] Hanbyul Joo, Tomas Simon, and Yaser Sheikh. Total capture: 3d deformation model for tracking faces, hands, and bodies. In CVPR, 2018. 2
[46] Angjoo Kanazawa, Michael Black, David Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In CVPR, 2018.
[47] Johannes Kopf, Michael Cohen, Dani Lischinski, and Matt Uyttendacle. Joint bilateral upsampling, ACM TOG, 2007.
[48] HyeokHyen Kwon. Yu-Wing Tai, and Stephen Lin. Data-driven depth map refinement via multi-scale sparse representation. In CVPR, 2015.
[49] Vincent Leroy, Jcan-Sébastien Franco, and Edmond Boyer. Multi-view dynamic shape refinement using local temporal integration. In /CCV, 2017. 2
[50] Hao Li, Bart Adams, Leonidas Guibas, and Mark Pauly. Robust single-view geometry and motion reconstruction. ACM TOG, 2009. 2
[51] Ruilong Li, Vuliang Xiu, Shunsuke Saito, Zeng Huang, Kyle Olszewski, and Hao Li. Monocular real-time volumetric performance capture. In ECCV, 2020. 3
[52] Yijun Li, Jia-Bin Huang, Narendra Ahuja, and Ming- Hsuan Yang. Deep joint image filtering. In ECCV, 2016.
[53] Yijun Li. Jia-Bin Huang, Narendra Ahuja, and Ming Masun Yang. Joint image filtering with deep convolutional networks. JEEE TPAMI, 2019. 3
[54] Zhe Li, Tao Yu, Chuanyu Pan, Zerong Zheng, and Yebin Liu. Robust 3d self-portraits in seconds. In CVPR, 2020. 3
[55] Zhe Li, Tao Yu, Zerong Zheng, Kaiwen Guo, and Yebin Liu. Posefusion: Pose-guided seleetive fusion
for single-view human volumetric capture. In CVPR, 2021.
[56] Marvin Lindner, Andreas Kolb, and Klaus Hartmann. Data-fusion of pmd-based distance-information and high-resolution rgb-images. In International Symposium on Signals, Circuits and Systems, 2007. 3
[57] Ming- Yu Liu, Oncel Tuzel, and Yuichi Taguchi. Joint geodesic upsampling of depth images. In CVPR, 2013. 3
12

---

[58]
Yebin Liu, Qionghai Dai, and Wenli Xu, A point-cloud-base multiview stereo algorithm for free- viewpoint video. IEEE TVCG, 2009. 1.3
[59] Yehin Liu, Carsten Stoll, Juergen Gall, Hans-Peter Seidel, and Christian Thcobalt. Markerless motion capture of interacting characters using multi-view image segmentation. In CVPR, 2011. 2
[60] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael Black. Smpl: A skinned multi-person linear model. ACM TOG, 2015. 1,2
[61] Lu, Xiaofeng Ren, and Feng Liu, Depth enhancement via low-rank matrix completion. In CVPR, 2014.
3
[62] Qianli Ma, Jinlong Yang, Siyu Tang, and Michael Black. The power points for modeling humans in clothing. In ICCV, 2021. 3
[63] Armin Mustafa, Hansung Kim, Jean-Y Yves Guillemaut, and Adrian Hilton. General dynamic scene reconstruction from multiple view video. In ICCV, 2015.
[64] Ryota Natsume, Shunsuke Saito, Zeng Huang. Weikai Chen, Chongyang Ma, Hao Li, and Shigeo Morishima. Siclope: Silhouette-basee clothed people. In CVPR, 2019. 3
[65] Richard A Newcombe, Dieter Fox, and Steven M Seitz. Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time. In CVPR, 2015. 1, 2,
[66] Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. Kinectfusion: Real-time dense surface mapping and tracking. In IEEE ISMAR, 2011. 1, 3, 8
[67] Ahmed Osman, Timo Bolkart, and Michael Black. STAR: A sparse trained articulated human body regressor. In ECCV, 2020. 1
[68] Jaesik Park, Hyeongwoo Kim, Yu-W Lo-ging Tai, Michael Brown, and Inso Kweon. High quality depth map upsampling for 3d-tof cameras. In ICCV, 2011. 3
[69] Georgios Pavlakos, Vasilcios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3d hands, face. and body from a single image.
In CVPR, 2019. 1,2
[70] Elad Richardson, Matan Sela, Roy Or-El, and Ron Kimmel. Learning detailed face reconstruction from a single image. In CVPR, 2017. 3
[71] Christian Richardt, Carsten Stoll, Neil Dodgson, Hans-Peter Seidel, and Christian Theobalt. Coherent spatiotemporal filtering, upsampling and rendering of rgbz videos. In Computer Graphics Forum, 2012. 3
[72] Joseph Roth, Yiying Tong. and Xiaoming Liu. Adaptive 3d face reconstruction from unconstrained photo collections. In CVPR, 2016. 3
[73] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In ICCV, 2019. 1, 2,
8
[74) Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo. Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization. In CVPR, 2020. 2, 3, 7,8
[75] Sebastian Schuon, Christian Theobalt, James Davis, and Sebastian Thrun. Lidarboost: Depth superresolu-
tion for tof 3d shape scanning. In CVPR, 2009. 3
[76] Xiaoyong Shen, Chao Zhou, Li Xu, and Jiaya Jia. Mutual-structure for joint filtering. In ICCV, 2015,
[77] D. Smith, M. Loper, X. Hu, P. Mavroidis, and J. Romero. Facsimile: Fast and accurate scans from an image in less than second. In ICCV, 2019.
[78] Vladimiros Sterzentsenko, Lconidas Saroglou, Anargyros Chatzitofis, Spyridon Thermos, Nikolaos Zioulis, Alexandros Doumanoglou, Dimitrios Zarpalas, and Petros Daras. Self-supervised deep depth denoising. In ICCV, 2019. 3,
[79] Zhuo Su, Lan Xu, Zerong Zheng, Tao Yu, Yebin Liu, and Lu Fang. Robustfusion: Human volumetric capture with data-driven visual using rgbd camera. In 202V, 2
[80] Yu Sun, Qian Bao, Wu Liu, Yili Fu, Michael Black, and Tao Mei. Monocular, one-stage, regression of multiple 3d people. In ICCV, 2021. 3
[81] Jing Tong. Jin Zhou, Ligang Liu, Zhigeng Pan, and Hao Yan. Scanning 3d full human bodies using kinects. IEEE TVCG, 2012. 1,3
[82] Anh Tuan Tran, Tal Hassner, Iacopo Masi, Eran Paz, Yuval Nirkin, and Gérard Medioni. Extreme face reconstruction:
Seeing through occlusions. In CVPR, 2018. 3
[83] Gul Varol, Duygu Ceylan, Bryan Russell, Jimei Yang, Ersin Yumer, Ivan Laptev, and Cordelia Schmid. Bodynet: Volumetric inference of 3d human body shapes. In ECCV, 2018.
[84] Daniel Vlasic, Matthew Brand, Hanspeter Pfister, and Jovan Popovic. Face transfer with multilinear models. ACM SIGGRAPH Courses, 2006. 3
[85] Daniel Vlasic, Pieter Peers, Ilya Baran, Paul Debevee, Jovan Popović, Szymon Rusinkiewicz, and Wojeicch Matusik. Dynamic shape capture using multi-view photometric stereo. ACM SIGGRAPH Asia, 2009. 1,3 3
13

---

[86] Michael Waechter, Nils Moehrle, and Michael Goesele. Let there be color! large-scale texturing of 3d reconstructions. In ECCV, 2014. 10
[87] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, and Bin Xiao. Deep high-resolution representation lcarning for visual recognition. IEEE TPAMI, 2021. 5
[88] Lizhen Wang, Xiaochen Zhao, Tao Yu, Songtao Wang, and Yebia Liu. Normalgan: Learning detailed 3d human from a single rgb-d image. In ECCV, 2020. 3
[89] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018. 8. 5
[90] Sanghyun Woo, Jongchan Park, Joon- YounYYung Lee, and In So Kweon. Cbam: Convolutional block attention module. In ECCV, 2018. 5,9
[91] Donglai Xiang. Fabian Prada, Chenglei Wu, and Jessica Hodgins. Monoclotheap: Towards temporally coherent clothing capture from monocular rgb video. In 3DV, 2020. 3
[92] Weipeng Xu, Avishek Chatterjee. Michael Zollhöfer. Helge Rhodin, Dushyant Mehta, Hans-Peter Seidel, and Christian Theobalt. Monoperfeap: Human performance capture from monocular video. ACM TOG,
2018. 2
[93] Shi Yan, Chenglei Wu, Lizhen Wang. Feng Xu, Liang An, Kaiwen Guo, and Yebin Liu. Ddmeet: Depth map denoising and refinement for consumer depth cameras using cascaded cnns. In ECCV, 2018. 3, 4 [94] Haotian Yang, Hao Zhu, Yanru Wang, Mingkai Huang. Qiu Shen, Ruigang Yang, and Xun Cao. Facescape: a large-scale high quality 3d face dataset and detailed riggable 3d face prediction. In CVPR, 2020. 6, 7
[95] Jingyu Yang. Xinchen Ye, Kun Li, Chumping Hou, and Yao Wang. Color-guided depth recovery from rgb-d data using an adaptive autoregressive model. IEEE TIP, 2014. 3
[96] Qingxiong Yang. Ruigang Yang, James Davis, and David Nistér. Spatial-depth super resolution for range images. CVPR, 2007.
[97] Yu Yanga, Xiao-Jun Wu, and Josef Kittler. Landmark weighting for 3dmm shape fitting. 1808. 2018.
[98] Genzhi Ye. Yebin Liu, Nils Hasler. Xiangyang Ji, Qionghai Dai, and Christian Theobalt. Performance capture of interacting characters with handheld kinects. In ECCV, 2012. 2
[99] Mao Ye and Ruigang Yang. Real-time simultaneous pose and shape estimation for articulated ohjects using& a single depth camera. In CVPR, 2014. 2
[100] Tao Yu, Zerong Zheng, Kaiwen Guo, Pengpeng Liu, Qionghai Dai, and Yebin Liu. Function4d: Real-time human volumetric capture from very sparse consumer rghd sensors. In CVPR, 2021. 2, 5, 5, 7,8
[101] Tao Yu, Zerong Zheng, Kaiwen Guo, Jianhui Zhao, Qionghai Dai, Hao Li, Gerard Pons-Moll, and Yebin Liu. Doublefusion: Real-time capture of human performances with inner body shapes from a single depth
sensor. In CVPR, 2018. 1, 2. 7.8 8
[102] Tao Yu, Zerong Zheng, Yuan Zhong. Jianhui Zhao, Qionghai Dai, Gerard Pons Pons-Moll, and Yebin Liu. Simulcap: Single-view human performance capture with cloth simulation. In CVPR, 2019. 2
[103] Xiaoxing Zeng. Xiaojiang Peng, and Yu Qiao. Df2net: dense-fine-fe network for detailed 3d face
reconstruction. In ICCV, 2019. 3
[104] Yebin Liu Zerong Zheng, Tao Yu and Qionghai Dai. Pamir:
Parametric model-conditioned implicit representation for image-based human reconstruction. IEEE TPAMI, 2021. 3
[105] Jason Zhang, Sam Pepose, Hanbyul Joo, Deva Ramanan, Jitendra Malik, and Angjoo Kanazawa. Perceiv- ing 3d hojen- spatial arrangements from single image the wild. In ECCV, 2020. 1
[106] Zerong Zheng, Tao Yu, Yixuan Wei, Qionghai Dai, and Yebin Liu. Deephuman: 3d human reconstruction from single image. ICCV, 2019.
[107] Tiancheng Zhi, Christoph Lassner, Tony Tung, Carsten Stoll, Srinivasa Narasimhan, and Minh Vo. Texmesh: Reconstructing detailed human texture and geometry from rgb-d video. In ECCV, 2020. 3
[108] Hao 7hu, Xinxin Zuo, Sen Wang, Xun Cao, and Ruigang Yang. Detailed human shape estimation from a single image by hicrarchical mesh deformation. In CVPR, 2019.
[109] Michael Zollhöfer. Matthias NieBner. Shahram Izadi, Christoph Rehmann, Christopher Zach, Matthew Fisher. Chenglei Wu, Andrew Fitzgibbon. Charles Loop, Christian Theobalt, et al. Real-time non-rigid reconstruction using an rgb-d camera. ACM TOG, 2014. 2
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Yes, we discuss this in See, 4.3 and Sec. 5.
14

---

(c) Did you discuss any potential negative societal impacts of your work? [Yes] Yes, we discuss this in the supplemental material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)? [Yes] We explain the implementation details in the supplemental material.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yss] All four metrics are computed as the average of 5 runs with different degrees of input depth noise, as stated in the supplemental material.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] This information is provided the supplemental material.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] The THuman2.0 and FaceScape datasets that we used are encrypted prevent unauthorized access. We have submitted a request form and request mail to obtain the datasets.
(e) Did you discuss whether the data you are using/curating contains personally identi- fiable information or offensive content? [Yes] Our captured data contains personal information but with permission, stated in the supplemental material.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "kq7o7Eop1v", "reviewer": "Reviewer_wHR5", "summary": "This paper proposed several additions onto the PIFU architecture that makes it perform better when trained and evaluated with noisy sparse RGBD views. The paper has three major contributions. 1.  A two scale PIFU model to render detail at different scales and to focus on the face of the human avatar. 2. An extension of PIFU to take in noisy RGBD data from real world sensors. 3. A demonstration of the results on 3 Kinects", "questions": "* Some of the text on the figure is so small it can be hard to read when written out.\n* In Figure 9, what does \"Ours\" have a different background\n* For the limitations section, did the authors try to train a 4 scale occupancy model with seperate occupancy models to model each hand? It sounds like the authors may have experimented with doing so, but failed to get good results. If so, this information would be very helpful to at least include in the paper's supplement.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is very well written. It provides competitive qualitative and quantitative results with state of the art. The paper is very well written and include useful figures and diagrams. One useful novel contribution their method includes merging the occupancy maps of the face and body. I think this might be the most general contribution in the paper as it stands. The separate modeling of the face and body with their own occupancy fields might also be a good way to optimize the number of parameters needed for high fidelity body construction. The approach is just taking some rather straightforward impovements onto the existing PiFU HD, so the technique itself isn't particularly original. However, it does define an interesting use case for PiFU, sparse RGBD reconstruction with Kinects and demonstrates noticeable improvements on it. I will note that I am not extremely familiar with the latest literature surrounding PiFU so may be missing some related work here, but the the contributions are somewhat novel and useful to future work on digital humans.\n\nOne weakness is the lack of qualitative examples describing the weaknesses noted in Section 5. It would be nice as a reader for the paper to demonstrate visually why hands remains challenging. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "jvKW01tNPXr", "reviewer": "Reviewer_Aogp", "summary": "This paper presents a 3D reconstruction method of clothed humans from sparse RGBD inputs. This work argues that jointly solving depth refinement and PIFu-based 3D reconstruction is mutually beneficial with empirical evidence. For effective information fusion between RGB and depth, the paper also propose cross attention module and geometry aware module. Additionally, the paper further improves the fidelity of face reconstructoin by processing face and body separately and fused them back into a single 3D model. With THuman2.0 dataset, the proposed approach outperforms existing approaches and the ablation study validates their technical contributions.\n", "questions": "Please provide evidence that multi-task learning is beneficial, not depth refinement itself.\n\nOther comments:\n- I would recommend not using different background color for ours in the qualitative results. \n- L157: I' and D' do not appear anywhere else.\n- Please clarify how to get the input mask.\n- L30: minimally clothed\n- L297: not small?", "limitations": "The paper does not discuss limitations or its societal impact. Discussion would be highly recommended. ", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strengths of the paper can be summarized as follows:\n- The qualitative results are impressive. Given noisy RGB-D inputs, the proposed method successfully generates high-resolution and complete 3D geometry of clothed humans. \n- The paper presents extensive experiments to show the effectiveness of the proposed approach over the existing approaches. \n- The proposed rgb and depth fusion module are interesting and shows improvements in performance. \n\nHowever, there are several weaknesses in this work:\n- While the paper argues that the multi task learning of depth refinement and 3D reconstruction is the key contribution, the experiments do not necessarily support this claim. More specifically, it is not clear if solving them together as multi-task is critical or simply providing cleaner depth map is beneficial for the following steps. To validate this, I would recommend running experiments by feeding refined depth that is separately predicted as isolated module into the proposed pipeline. Without such an experiment, the key contribution of this work remains unclear.\n- This paper misses one highly relevant paper, JIFF, where PIFu-like model predicts body and face regions separately and fuse them to produce high-quality reconstruction (although they are based on RGB inputs). Given the existence of this work, the contribution of two-scale PIFu would diminish. \n\nJIFF: Jointly-aligned Implicit Face Function for High Quality Single View Clothed Human Reconstruction\nYukang Cao, Guanying Chen, Kai Han, Wenqi Yang, Kwan-Yee K. Wong, CVPR 2022\n\nWhile the results are impressive, the extent of the paper contributions remain unclear with the current form. Thus, I would stay on the fence, slightly inclined towards rejection.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GlE7Ts8TK_E", "reviewer": "Reviewer_7cFn", "summary": "This paper presents a method for reconstructing 3D human models from sparse RGBD inputs. The proposed method is built upon PIFu. To eliminate the negative impact of depth noise, the authors introduce a multi-task learning framework that incorporates the complementary nature of  depth denoising and 3D reconstruction. The authors also introduce an additional MLP for facial regions to further improve the reconstruction quality of human faces. Experiments show that the proposed method can generate high-quality 3D human models from sparse RGBD sensors. \n", "questions": "I am skeptical about the experiments in Fig. 3. In the main text of the paper, the authors emphasis that the task of depth denoising preserves local geometric details while 3D reconstruction provides global topology guidance. However, in Fig. 3 (g), which is the results of solely depth denoising, the refined depth looks much smoother than (h) and does NOT preserve local geometric details. This contradicts with the main insight of this paper and makes me feel that the geometric details is acturally produced in the 3D reconstruction task rather than the depth denoising task. I think the authors should discuss more about the experiment in Fig. 3 and provide more results if possible. \n\n", "limitations": "The authors have discussed the limitations in Sec. 5 and the potential societal impact in the supplement material. No more improvement is needed. \n\n", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* The results look impressive. Given noisy depth measurement from sparse RGBD sensors, the proposed method is able to reconstruct clean 3D models with high-quality geometric details. \n\n* The main contributions of this paper are convincingly validated. The authors present extensive experiments throughout the paper. \n\n* This paper is overall well-written and easy to follow. \n\n\nWeaknesses:\n\n* The novelty of this paper is a bit of weak. In the introduction, the authors claim the two-scale PIFu representation as one technical contribution. However, a similar design has already been proposed in PIFuHD. I understand that the fine-scale network in PIFuHD is still designed for the whole body while the fine-scale MLP in this paper only focuses on the facial region, but I would regard this difference as engineer choices instead of a \"technical contribution\". \n\n* No textured results are demonstrated. The authors are encouraged to present the textured models because it is a more intuitive way to evaluate the geometric reconstruction accuracy. \n\n* To evaluate the multi-task design, the authors simply remove the depth denoising task. I think a better baseline for this experiment would be a sequential framework where the depth maps are firstly refined by the depth denoising module and then fed into the 3D reconstruction network. \n\n* Missing citation: Zheng et al. PaMIR: Parametric Model-Conditioned Implicit Representation for Image-based Human Reconstruction. IEEE T-PAMI. \n\n******************************************************************\nFinal comments:\nThanks for the authors' response which addresses some of my concerns. After reading other reviews and the rebuttal, I would like to keep my initial rating. The authors should add their additional experimental results into the final paper if the paper gets accepted finally. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "9GRJBmwdyG", "reviewer": "Reviewer_A5r2", "summary": "The paper propose a method for human 3D reconstruction. The main contribution includes: 1. a geometry-aware PIFu method for human reconstruction from RGBD images; 2. a two-scale method dealing with body and face separately and then fusing the results; 3. experiments results showing the effectiveness on Kinect captured data.", "questions": "1. Many modules are used in the pipeline. Which of them are completely novel? Which are adapted from existing works?\n2. How are the parameters selected?", "limitations": "The authors have discussed the limitations. However, it is unclear whether the method works in some challenging scenarios. E.g., loose dressing, eye glasses, etc.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+ The experimental results are good. Clothing deformation and facial features are reconstructed better than compared methods.\n\n+ The ablation study is extensive, showing the necessity of each proposed module.\n\nWeaknesses:\n\n- The method seems to be too complex, and more like stacking existing methods and modules, limiting the novelty of the paper.\n\n- The writing needs significant improvement: (1) The pipeline figure 2 needs to be simplified. Showing too many details in one single figure makes readers get lost. (2) The math should be simplified, too. Instead of defining a lot of symbols, it would be good if the author could describe the key idea (or even for a simplified pipeline) using very simple equations. Complex details could go to the supplementary material. (3) I would not recommend using \"\\copyright\" as the concatenation operation, because it already has other meanings. ‖ could be a better choice.\n\n- Since the method deals with face and body separately, it would be good if the comparison with face reconstruction methods is included in the experiments.\n\n- The related work could be expanded. Here I listed some relevant papers, but the author could include more:\n[a] Real-time Deep Dynamic Characters, SIGGRAPH'21;\n[b] Texmesh: Reconstructing detailed human texture and geometry from rgb-d video, ECCV'20;\n[c] MonoClothCap: Towards Temporally Coherent Clothing Capture from Monocular RGB Video, 3DV'20", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept