Deep Deformation Based on Feature-Constraint  for 3D Human Mesh Correspondence
Authors: Na Gong
Venue: Submitted to ICLR 2023 (2023)
Keywords: shape correspondence
deep learning
shape deformation

Abstract:  In this study, we address the challenges in mesh correspondence for various types of complete or single-view human body data. The parametric human model has been widely used in various human-related applications and in 3D human mesh correspondence because it provides sufficient scope to modify the resulting model. In contrast to prior methods that optimize both the correspondences and human model parameters (pose and shape), some of the recent methods directly deform each vertex of a parametric template by processing the point clouds that represent the input shapes. This allows the models to have more accurate representations of the details while maintaining the correspondence. However, we identified two limitations in these methods. First, it is difficult for the transformed template to completely restore the input shapes using only a pointwise reconstruction loss. Second, they cannot deform the template to a single-view human body from the depth camera observations or infer the correspondences between various forms of input human bodies. In representation learning, one of the main challenges is to design appropriate loss functions for supervising features with different abilities. To address this, we introduce the feature constraint deformation network (FCD-Net), which is an end-to-end deep learning approach that identifies 3D human mesh correspondences by learning various shape transformations from a predetermined template. The FCD-Net is implemented by an encoder–decoder architecture. A global feature encoded from the input shape and a decoder are used to deform the template based on the encoded global feature. We simultaneously input the complete shape and single-view shape into the encoder and closely constrain the features to enable the encoder to learn more robust features. Meanwhile, the decoder generates a completely transformed template with higher promise by using the complete shape as the ground truth, even if the input is single-view human body data. We conduct extensive experiments to validate the effectiveness of the proposed FCD-Net on four types of single-view human body data, both from qualitative and quantitative aspects. We also demonstrate that our approach improves the state-of-the-art results on the difficult "FAUST-inter" and "SHREC'19" challenges, with average correspondence errors of 2.54 cm  and 6.62 cm, respectively . In addition, the proposed FCD-Net performs well on real and unclean point clouds from a depth camera.

Under review as a conference paper at ICLR 2023
DEEP DEFORMATION BASED ON FEATURE- CONSTRAINT FOR 3D HUMAN MESH CORRESPON- DENCE
Anonymous authors Paper under double-blind review
ABSTRACT
this study, we address the challenges mesh correspondence for various types of complete or single-view human body data. The parametric human model has been widely used in various auman-related applications and in 3D human mesh correspondence because it provides sufficient scope modify the resulting model. In contrast to prior methods that optimize both the correspondences and human model parameters (pose and shape), some of the recent methods directly deform each vertex of a parametric template by processing the point clouds that represent the input shapes. This allows the models have more accurate representations of the details while maintaining the correspondence. However, we identified two limitations in these methods. First, it is difficult for the transformed template completely restore the input shapes using only a pointwise reconstruction loss. Second, they cannot deform the template to a single-view human body from the depth camera observations or infer the correspondences between various forms of input human bodies. In representation learning, one of the main challenges is design appropriate loss functions for supervising features with different abilities. To address this, we introduce the feature constraint deformation network (FCD- Net), which is end-to-end deep learning approach that identifies 3D human mesh correspondences by learning various shape transformations from a prede- termined template. The FCD-Net is implemented by encoder-decoder archi- tecture. A global feature encoded from the input shape and a decoder are used deform the template based the encoded global feature. We simultaneously input the complete shape and single-view shape into the encoder and closely con- strain the features enable the encoder learn more robust features. Mcanwhile, the decoder generates completely transformed template with higher promise by using the complete shape as the ground truth, even the input is single-view hu- man body data. We conduct extensive experiments to validate the effectiveness of the proposed FCD-Net on four types of single-view human body data, both from qualitative and quantitative aspects. We also demonstrate that approach im- proves the state-of-the-an art results on the difficult "FAUST-inter" and "SHREC" 19" challenges, with average correspondence errors of 2.54 em and 6.62 em, respec- tively In addition, the proposed FCD-Net performs well on real and unclean point clouds from depth camera.
INTRODUCTION
The rapid development of 3D sensor devices has led to tremendous growth in the field of 3D vision technologies. essential application of 3D vision technology is the 3D shape correspondence and deformation HHuang & Fangl (2021). which attempts establish reliable correspondences between two 3D shapes (lattoow et al. (2022); (Sahiliooll Yemez (2010); Huang et al. (2017)) and is a hot research topic in 3D vision. In contrast registrations of scenes objects (Gojcic al. (2020); Segal et al. (2009); Ao et al. (022)) that only involves rigid deformations, such as rotations and articulated human bodies requires flexible, com- plex, non-rigid deformations and pose variations(Seratin & Grisetti (2015); Bhatnagar et al. (2020); Groueix et al. (2018)). This makes the correspondence process challenging.
translations, estimating the correspondences

---

geometrical Under review as a conference paper at ICLR 2023
Model-driven shape reconstruction and matching methods for articulated humans utilize paramet- ric body template model (e.g,, a skinned multiperson linear (SMPL) ) (ooper et al. (2015)) or shape completion and animation for people (SCAPE) (Anguelov al. 2005)) model)
prior, and optimize or learn the parameters to deform the template, typically its poses and shapes. The deformed models have the same vertex definition, definite semantic information, and same face connections as the template. This makes the correspondence problem easier than when using methods that require finding associations between variable and large number of points through an optimization strategy to minimize an objective function. However, the low dimensional parameters (shape and pose) limit the description of the details. Researchers have proposed SMPL model displacement approaches to increase the details of the model (Bhatnagar et al. (2020). Other works have combined a parametric model with the deep implicit function to realize free-form human body reconstruction at arbitrary resolutions (Huang et al. (2020); He et al. (202T)). However, SMPI. displacement methods require two stages to output the final results, making them prone to error at two levels. Free-form implicit functions can lose the semantic information and correspondences of the points. Some recent works Wang et al. (2019) have directly deformed each vertex of a para- metric template based on global feature coded from the input shape. This allows the models to provide a more accurate representation of details while maintaining the correspondence of the para- metric model. However, we identified two limitations in these methods. First, it is difficult for the transformed template to completely restore the input shapes using only a pointwise reconstruc- tion loss. Second, they can only deform the template to a complete human body as obtained by scanning (or via reconstruction by other methods et all (221))). In this study we focus on deforming the template to single-view 3D human shapes from depth camera images and inferring the correspondences between various forms of the human body. A single-view shape is the most easily obtained 3D data form, owing to the development of low cost, low power consumption. and low-price RGBD cameras. In Groueix et al. (2018), point-based neural network was used to learn a global feature from an input shape. The global feature stores all of the information of the input shape for directing the template deformation. In our view, the global feature should not only accu- rately represent the complete human shape but should also facilitate recovering the invisible part(s) when the input is single-view shape. In representation learning, one of the main challenges is to design appropriate loss functions for supervising features with different abilities (discriminative, expressive, or restorative). Motivated by (Denng et al./ (2019), we attempted to penalize the loss in the feature space, and specifically, to penalize the angular difference between the deep features obtained from the single-view view shape and complete human shape. This can be achieved by simultaneously inputting the complete and single-view data of one object during the training process.
Therefore, we propose a framework that is suitable for searching for the correspondence relation- ships between various types of complete or single-view human bodies by processing the point clouds that represent the input shapes. We call this framework the feature constraint deformation network (FCD-Net). The FCD-Net is designed with encoder-decoder structure. The encoder comprises a deep neural network for generating global features representing the input shapes. Then, a shape deformation network learns deform template as guided by the encoded global feature to align with the target shape as described in detail in Section [3]
We train our FCD-Net with single-view and complete shapes as input shape pairs, as supervised by the known correspondences between the input shapes and template. These are generally explicit when they are both generated by the same parametric model. During testing, only one type of obser- vation is needed, i.e., single-view or complete. The correspondences between the various types of inputs can be realized under a unified framework. To demonstrate the advantages of the FCD-Net, first, we show that the FCD-Net can achieve single-view 3D human body correspondence. Then, we test FCD-Net on finding the correspondences for scanned 3D humans on several public datasets. The FCD-Net achieves state-of -fth---rt- results in the "INTER" challenge of the FAUST dataset with an average correspondence error of 2.54 cm, and in the SHREC' 19 challenge with an average cor- respondence error of 6.62 cm.
2 RELATED WORKS
The traditional search for correspondence between 3D human body models is often conducted through regression optimization methods or function mapping methods. The normal iterative clos- est point (NICP)(Seratin & Grisetti (aggSiia represents the corresponding relationship be- tween a source model and target model by establishing complex mathematical relationship be-
2

---

Under review as a conference paper at ICLR 2023
tween the source and target model point sets, and then finds the corresponding points through con- tinuous iteration. In addition to constraining the distances between points, the NICP adds con- straints the normal vector and curvature of the surface where the point cloud is located for the model matching, thereby improving the accuracy of the alignment. Yao et al. used Welsch's func- tion to optimize an objective function; this changed the nonconvex discontinuity of the objective function and enabled the use of the majorize-minininn algorithm. This improved the efficiency and accuracy in solving the correspondence Yao et al. (2020). Optimization-ben based methods have strict mathematical logic to ensure their convergence and interpretability, but they often need good initialization. Therefore, features, such as the heat kernel signature (Sun et al. (2009). ware kernel signature (Aubry et al. (20TT), and orientation histogram, are widely used in non-rigid 3D shape processing tasks, and each identified key point is represented by a feature vector (Tombari et al. (20TO)) These features are often used as a priori knowledge as a way to obtain a more reliable correspondence. However, methods that require additional computation to obtain prior knowledge are sensitive to problems, such as priori knowledge, noise, and voids.
algorithm with the limited-memory Broy-
A function mapping method transforms the process of finding the corresponding relationship be- tween the source model and target model into function mapping, and it considers the source model and target model as changes between matrices. By adding normal constraints, approximation func- tion constraints, and other constraints, the quality of the corresponding relationship is improved (Osjannkovv al.) (2016); Vestner et al. (2017)). However, most of these methods require a large amount of computation or have strict requirements for inputs (Nogneng et al. (2018); Rodolà et al. (2015). ) Another common treatment comprises a combination of function mapping and spectral domain; this has achieved good results on synthetic data, but is less effective in practice and requires a longer inference time (Ginzburg & Raviv (2020); Roufosse et al. (2019)).
AlexNet learned 3D features from RGBD datasets (Zeng et al. (2017). and PpfinetDeng et al.
(018)) used a local point pair feature based on a neighborhood point distribution and fed it into a network for deep feature learning. 3DSmoothNet (Gojeic et al. (2099) proposed rotation-invart invariant handcrafted features and fed them into deep neural networks for feature learning. The 3DN (Wang
al. 2019)) network keeps the network topology of the source model unchanged through differ- entiable network sampling operators, and an excellent corresponding relationship can be obtained between source and target models of different densities. Donati al. used hybrid network structure to learn feature descriptors by using feature extractors in the spatial domain (Donati et al.) ). After feature extraction, they were sent to spectral domain function mapping layer to find the correspondences. Wang et al. transferred local features from an image space to a UV space and established new UV mapping to obtain the correspondences between the images and 3D models ZZng et al. (2020)). Correnet3D (Zeng et al. 22221) used corresponding learning module to com- pute the optimal matching matrices for source and target models with different poses and modeled the matching problem minimized cost function solution for further optimization. The above methods have achieved excellent results in 3D human model deformation and the corresponding relationship searches. However, these methods are usually proposed for matching complete 3D hu- man models; there are relatively few methods applicable to single-view human body data collected by a single RGB-D camera. Our proposed method can identify the deformations and corresponding re relationships between a single-view point cloud and complete human body, and obtain state-of-the art results.
METHOD
In this study, we aim to identify the shape correspondence for single-view human body. To this end, we propose the feature constraint deformation network. which takes single-view point clouds of the human body as input. Inspired by Groueix et al. (2018), our network uses the parameterized human body as an intermediate bridge for the correspondence search. Compared with 3D-CODED data, when learning the reconstruction of the human shape model, our method takes the single-view human body as input, and it uses the complete human body model as ground truth to supervise the network training in the pointwise and feature spaces. The obtained network can better realize the deformation and reconstruction of the single-view human body. Moreover, the shape encoder network has seen the complete human body shape of the ground truth, the single-view point cloud correspondence network can realize the reconstruction and deformation of the complete human body
3

---

Under review as a conference paper at ICLR 2023
Templaoe
Decoder
Uefiomnel body
Complete shape
Tepu xingle view
Eacoder
Roonestaction
Mise pood
Clobal &
Training loss
Encoder
L
Max pool
features,
Beyer comp dugge 8,
Figure I: Overview of the learning procedure
model simultaneously. The learning and inference procedures of our method are shown in Fig. and 3 and details are provided below.
3.1 LEARNING SINGLE-VIEN HUMAN BODY RECONSTRUCTION BY TEMPLATE DEFORMATION
It is a difficult and time-consuming task to directly obtain the correspondence relationship between two 3D human body shapes (source model and target model). The proposed method selects pa- rameterized human body model with standard pose as template, and then deforms it into source and target models with arbitrary poses and body shapes, respectively. The relationship between the corresponding points of the generated source model and target model is already established through the template, making the corresponding point detection relatively easy. To achieve this goal, we first design the shape deformation network. This network takes all of the vertices of the template as input and outputs the deformed vertex coordinates. To obtain guidance on the deformation, we must condition our observations (single-view human point clouds in our method) on the input. Therefore, the mapping relationship for shape deformation network is as follows:
Do (p,;z) -
where,p, denotes the jth vertex of the template, and x denotes the observations. The single-view human point cloud data in the original 3ID space has great redundancy; therefore, we first adopt point cloud feature extraction Es to encode the single-view human point cloud, input all of the 3D point cloud coordinates, and output a one-dimensional feature vector with predefined length. This feature is taken as the observed quantity in Equation and in the human template as an input pair to output the deformed human shape vertices.
network shown in the blue part of Fig. D The encoder adopts the simplified PointNet network to extract the features of the input point cloud. After series of one-dimensional convolutions with scales of 64, 1 128, and 1024, features with 1024 dimensions are obtained through the maximum pooling layer. The decoder corresponds to the shape deformation network in Equation(]), ), and is implemented by simple multilayer perceptron. The specific hidden layer sizes are 1024, 512, 254, and 128, i.e., the same as Groueix et al. (018). The 1024-dimensional features extracted by the encoder network template body to complete the shape deformation. In the specific implementation, the human body template adopts the complete human body model, and the complete human body corresponding to the single-view point cloud is taken as the reconstructed ground truth. Thus, the proposed method can achieve deformation of the data of the visible view simultaneously The missing data at the self-occluded regions is completed and the completed part is deformed. The reconstruction loss is defined as follows:
We achieve the above functions by training the
combined with the
LR ΣΣ 11
Do (pji
(s!0)
(9)
4

---

Under review as a conference paper at ICLR 2023
Fearure of single view body 8,
Festure of complete Loanmu budye
penalize
Figure 2: Feature constraints between complete and single view human body
Here Pj E M, Where, represents the parameterized 3D human body model, S!e represents the tt single-view human shape in the input encoder network; qi (8) S wheres.. represents the complete human body model corresponding to the input single-view human body model, P repre- sents the number of vertices, and N represents the number of training samples.
The currently available methods for body shape correspondence (Zhou et al.] (2020); Deprelle et al. (2019); Zeng al. (2020)) can only achieve correspondence between complete bodies. The proposed method enables a determination of the correspondence between two single-view body shapes. Moreover, in addition the need estimate the shape correspondence between single- view body shapes, there are many applications that require estimations of the shape correspondence between single-view body shape and full-body shape. If our network only uses single-view body shapes as observations, it is will be unable to appropriately handle the deformation and reconstruc- tion of a complete human body. Therefore, body from the single-viee view human shape by supervising the network in the reconstruction space point-by-pon by--int basis, we also propose feature-space constraint strategy (shown in yellow in Fig. [ In this strategy, input the intact human shape ground truth into the same shape encoder net- work, and then constrain the single-view human shape and intact human shape in the feature space close possible, so to use the same network framework for both the single-view and full-body shapes. In addition, the proposed feature constraint strategy can also better guide the complementa- tion and deformation of the single-view human shape, shown in Fig. 2 The feature similarity is measured using the cosine distance, and the specific feature constraint loss is calculated as follows:
addition to generating the deformed intact human
(F(#)
EsU
The total loss function used for optimizing the parameters of FCD constructed as follows:
(Øs 0) = (1 L + bLF
(4)
Here, b a scale coefficient for controlling the influence of the eigenvector similarity on the overall reconstruction effect.
3.2 FINDING SHAPE CORRESPONDENCES
Soarce
Taupot
Template
Minimus Singe chamter distance
Ball bic find the Neghbar
Sull tuqel Find New
Comespondence templatee
Gilobal feotrce
Target
(a) Local optimization of shape
(b) Correspondence
Figure 3: Finding shape correspondences in inference procedure
5

---

Under review as a conference paper at ICLR 2023
In the inference procedure, reconstructed parametric human body model is obtained for each input model by the trained encoder and decoder. This procedure comprises a global encoding and decod- ing of the human body and can achieve the overall correspondence of the human body; however, the local and detailed parts of the human body are only approximated, and it can be improved. The symmetric chamfer distance between the reconstructed model and input model was solved for in (Groueix et al. (2018) to represent the similarity between the input model and reconstructed para- metric model; and the global features of the encoder network output were optimized over 3000 iterations to minimize this chamfer distance. Because our model needs to be applicable to both the single-view human model and complete human model, we use the one-sided chamfer distance to achieve the local optimization between the reconstructed parametric human model and input model and further improve the accuracy of the reconstruction (shown Fig. Bta), The same process can be used to obtain the target human model after the local optimization, and the one-sided chamfer distance loss is as follows:
L CD (Eso) = Σ min Do (pt gl
(5)
PE qES...
To obtain the correspondence between the two 3D input shapes (source and target), we first obtain more accurate parametric deformation human models (source M and target M) through the shape deformation network and local shape optimization, and then use the known correspondence between the parametric human models find the correspondence points. The source and Source M are models of the same pose and body type, and we use a ball-tree to find the nearest neighbors as the corresponding points. The source M and target M are different poses and body types, but they are both parametric models derived from the template deformation, and the correspondence is known. Similarly, the target M and target are human bodies with the same pose and body type, and the ball-tree is used to find the nearest neighhor as the corresponding point. Thus, the correspondence between the source and target can be accurately identified.
EXPERIMENT
4.1 EXPERIMENTAL SETUP
Training data Unlike many current human reconstruction methods that rely large number of scanned 3D mannequins as training data, our method trains directly on synthetic SMPL data. The SMPL model is a state-of-the art generative model for synthetic humans. The SMPL data can be adjusted to generate large amount of human data with different poses and sizes. Moreover, the corresponding relationships of parameterized data are known, which reduces the workload for man- ual labeling. First, we manually deleted part of the point cloud behind the SMPL standard model under a T-pose, and recorded the index of the deleted point cloud. Then, we extracted the point cloud data of "230 K SMPI." with a large variety of realistic poses and body shape models generated by 3D-coding as the training ground truth. Finally, parts of the back point cloud of all models was deleted according to the recorded index to simulate the generation of the persspeereee point cloud data as the input to training. The T-Pose SMPL standard model and part of the training input with the back point removed are shown in Fig. 4
pose
ni
Figure 4: Single perspective point cloud and part of training data
Test data We conducted experiments on four human datasets completely different from the train- ing data: FAUST(Bogo al. (2014)), SCAPE(Anguelov al. 2005)), SMPL (Loper et al. (015)).

---

Under review as a conference paper at ICLR 2023
SMPL-expressive (SMPL-X) (Pavlakos al. (2019)), and SHREC' 19 de Benoist
evaluation criterion For the evaluation the corresponding relationships between parameterized human body models of the same type, calculated the Euclidean distance between the corre- sponding point pairs of the network outputs as our evaluation index for SHREC' 19. We calculated the distance between the target model points in the corresponding point pairs of the network output and real points of the dataset as our evaluation metric, as shown in Equation Below, is the serial number of the corresponding point of the target human model, f (æt) the three-dimennionall coordinate of the corresponding point of the target human model, and gr is the three-dimensional coordinate of the ground truth.
erT
Σ (1:) 991ll2 FIET
(6)
T1
4.2
EXPERIMENTAL RESULT
4.2.1 SINGLE- 3D HUMAN BODY CORRESPONDENCE
oaTa
(a) FAUST after alignment.
(b) Scapecomp.
Innn
28-00E)
(c) SMPL.-X
(d) RGB-D camera capture data
Figure 5: Qualitative comparison for the single-view 3D human correspondence task defferent datasets.
To qualitatively and quantitatively evaluate the reconstruction and corresponding effect of our pro- posed feature constraint deformation network method on the single-view point cloud input, we use different parameterized human data sets to generate single-view point cloud data for evaluation. The specific data used included the registered FAUST dataset, registered Scapecomp dataset, and SMPL- X model. The corresponding single-view point cloud for testing generated in a similar way as the training data. To increase the randomness of the data, each dataset was manually generated with its deletion index to simulate the uncertainty of the edges when the RGB-D camera acquires the single-view point cloud. In addition, we also used set of RealSense camera data collected from real scene single-view human point cloud data test the effect. The qualitative and quantitative comparisons are shown in Fig. and Table [ respectively. As there no corresponding parametric model for the Realsense data, we only show the results of the qualitative analysis. For the quanti- tative comparisons, we can compare the corresponding errors of the frontal (visible) part to better reflect the performance of the different methods.
The qualitative comparison is shown in Fig. 5 In Fig. ((a) it is demonstrated that the 3D-CODED approach reconstructs the human body with a depressed back and the normal of the back part of the reconstructed human model is opposite to the normal of the surrounding vertices; whereas, our method is able to reconstruct the complete human body without these problems as shown in Fig. 5(b) The hand and arm parts of the 3D-CODED human model are severely deformed and the prob- lem of the opposite vertex normal remains. Our method reconstructs the complete human body without deformation because there is data for clenched fist in the training data, and our method

---

Under review as a conference paper at ICLR 2023
Inpuc
30-000ED
Durs
Inpur
30-CODED
Ipp
10-00020
Insat
Figure 6: Qualitative comparison on scans of Figure 7: Qualitative comparison on scans of Faust.
SHREC' 19
reconstructs the human hand as open. From Fig. 5(c) it is clear that our network outperforms 3D-CODED for the SMPL-X human model data, especially for the hands, back, arms, and legs. In the real application scene in Fig. ((d) the reconstruction effect of the single-perspective point cloud mannequin as photographed by Realsense more evident. This indicated that our network can reconstruct the single-perspective point cloud mannequin into more detailed and complete mannequin, and there no reverse normal situation, whereas the mannequin reconstructed by 3D- CODED produces serious deformations in the arms, hands, and face, and the normal of the body parts the opposite problem serious. Thus, our method should be more applicable to real scenes. For the quantitative comparison, contained correspondence and computed correspondence as the error. As seen from the experimen- tal results in Table [] for the reconstruction and correspondence of the visible part, method outperforms the 3D-CODED on all four datasets.
calculated the average Euclidean distance between the self-
Table 1: Visible Part Euclidean mean error (EME)
Dataset
Method
EME
FAUST
3D-CODED
0,371003
FAUST
Ours
0.362244
SCAPE
3D-CODED
0.369555
SCAPE
Ours
0.367941
SMPL
3D-CODED
0.037367
SMPL
Ours
0,031093
SMPL-X
3D-CODED
0.414028
SMPL-X
Ours
0.413197
4.2.2 CORRESPONDENCE OF SCANNED 3D HUMAN
Qualitative testing
In this subsection, we describe qualitative tests concerning the reconstruction and correspondence of the proposed method on all types of scanned human data, and the results are shown in Figs 6 In Fig. 6. the input comes from the test set in the FAUST scanning data. As shown in Fig. 6(a), the target human hand is placed behind the head, and 3D-CODED only relies on reconstruction of the loss to train the deformation network. As result,the hand and head modeled as a whole. As shown in Fig. 6(b), there are some missing parts of the target human feet. Both our method and recover the missing parts of the feet; for the hands integrated with the body, method can also achieve relatively accurate modeling and correspondence.
3D-CODED
Fig. 7(a) and Fig. 7(b) are derived from the results of the Princeton dataset (Chen al. (2009) in the SHREC' 19 dataset (de Benoist). As can be seen from the results for Fig. 7(a), which presents the human body with a more ambiguous input, our proposed method can serve complement the de- tailed parts of the human face and can reconstruct more reasonable human body than 3D-CODED. As shown in Fig. 7(b), our method can reconstruct reasonable hand and face information, whereas 3D-CODED reconstructs the human body with three fingers, facial layering, and other errors.
Correspondence error analysis
To further validate the performance of the proposed method in finding correspondences. we per- formed a quantitative comparison with the existing state-of-the-att methods on the FAUST and SHREC' 19 datasets, as these provided challenges and true values of the data.
In the FAUST dataset-INTER challenge, the test data comprised scanned complete human models with different postures, and the comparison method used a complete human dataset as the training
8

---

Under review as a conference paper at ICLR 2023
Table FAUST dataset INTER challenge performance of different methods
Serial number
Method
Error(cm)
Convex-Opp (Chen & Koltun
(0155)
8.304
FMNet Litany et 201
4.826
FARM (Marin et al.
2020a
4.123
LBS-AE (Li et al.
2019)
4.079
Smooth Shells (Eisenberger et (2020)
3,929
BPS (Prokudın et al. (2019)
4.529
SP 3P_inter cww.lengeeeeeeeee (Zuthi & Black
(2055)
3.126
Adversarial GP network (Zhou et al.
(2020)
2.759
3D-CODED (Groueix al. (201
8))
2.878
10
LoopReg (Bhatnagar et al. (2020)
2.660
11
Learning element structure (Deprelle et (2019)
2.578
12
Ours (single-view shape as imput)
2.774
13
Ours (complete shape as input)
2.541
data. To be fair, trained two versions of the model: version the same in all of the previous experiments and its input data was the synthetic point cloud data that simulated the single view and deleted the backside points; and the input data of the other version used the full SMPL parameterized model as the ground truth. The experimental results are shown in Table 2 It can be seen that the training results from our method on the synthetic single-perspective point cloud are better than most comparison methods (including 3D-CODED). When the input data is complete human body during training, our method obtains the lowest error, outperforming other baseline methods.
Similarly, on the SHREC' dataset (de Benoist), we tested both versions of the model and com- pared the results with those given on the challenge website, shown in Table Because this dataset contains 3D human models with varying densities from 5,000 to 50,000 vertices, and with uniform and non-uniform distributions, the models are highly variable; therefore, the overall test results worse than those for the FAUST dataset. However, our proposed method, trained on the input as both synthetic single-view point cloud and complete human point cloud, outperforms the compar- ison method, demonstrating the good generalization capability of the proposed method in terms of finding correspondences between different human bodies.
Table 3: Performance on SHREC' dataset
Serial number
Method
EME(cm)
3D-CODED Groueix et al. (2018)
8.1
Diff-FMaps MMrin et al. 20206)
Elementery Structures (Deprellc et 2019)
7.6
CorrNet3D (Zeng et al. (2021))
6.9
Ours (single-view shape input)
6.83
Ours (complete shape as input)
6.62
CONCLUSION
We propose network structure that can be applied not only to find correspondences between com- plete human models, but also to find correspondences between complete human models and single- view 3D human models. The key point of our method is the use of feature vector similarity and single-side chamfer distance to further constrain the reconstruction; this will improve the recon- struction quality and correspondence search accuracy. The proposed method improves the accuracy in finding the correspondence between the complete human body model and single-view human of 2.74 em and 2.541
body model. It achieves inputs on the FAUST dataset, respectively, and 6.83 em and 6.62 em on the SHREC" 19 dataset, respectively.
the single-vier view and complete shapes as

---

Under review as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was supported by National Natural Science Foundation of China (No. 61901436)
REFERENCES
Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. Scape: shape completion and animation of people. ACM SIGGRAPH 2005 Papers, pp- 408-416. 2005.
Sheng Ao, Qingyong Hu, Bo Yang, Andrew Markham, and Yulan Guo. Spinnet: Learning general surface descriptor for 3d point cloud registration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11753-11762, 2021.
Mathieu Aubry, Ulrich Schlickewei, and Daniel Cremers. The kernel signature: A quantum mechanical approach to shape analysis. In 20// IEEE international conference on computer vision workshops (ICCV workshops). pp. 1626-1633, IEEE, 2011.
Bharat Lal Bhatnagar, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll. Loopreg: Self-supervised learning of implicit surface correspondences, pose and shape for 3d human mesh registration. Advances in Neural Information Processing Systems, 33:12909- 12922, 2020.
Federica Bogo, Javier Romero, Matthew Loper, and Michael Black. Faust: Dataset and evaluation for 3d mesh registration. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3794-3801, 2014.
Qifeng Chen and Vladlen Koltun. Robust nonrigid registration by convex optimization. In Proceed- ings of the IEEE international conference on computer vision, pp. 2039-2047, 2015.
Xiaobai Chen, Aleksey Golovinskiy, and Thomas Funkhouser. A benchmark for 3d mesh segmen- tation. Acm transactions on graphics (tog), 28(3):1-12, 2009.
Jaesung Choe, Byeongin Joung, Francois Rameau, Jaesik Park, and In So Kweon. Deep point cloud reconstruction. arXiv preprint arXiv:2111.11770 2021.
Louis Benoist. learning.
Matching nodes on meshes with different connectivity using geometric deep
Haowen Deng, Tolga Birdal, and Slobodan Ilic. Ppfnet: Global context aware local features for robust 3d point matching. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 195-205, 2018.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos 7afeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 4690-4699, 2019.
Theo Deprelle, Thibault Groueix, Matthew Fisher, Vladimir Kim, Bryan Russell, and Mathieu Aubry. Learning elementary structures for 3d shape generation and matching. Advances in Neural Information Processing Systems, 32, 2019.
Nicolas Donati, Abhishek Sharma, and Maks Ovsjanikov. Deep geometric functional maps: Ro- bust feature learning for shape correspondence. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8592 8601, 2020.
Marvin Eisenberger, Zorah Lahner, and Daniel Cremers. Smooth shells: Multi-scale shape registra- tion with functional maps. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12265-12274, 2020.
Dvir Ginzburg and Dan Raviv. Cyclic functional mapping: Self-supervised correspondence be- tween non-isometric deformable shapes. In European Conference on Computer Vision, pp. 36-52.... Springer, 2020.
10

---

Under review as a conference paper at ICLR 2023
Zan Gojeic, Caifa Zhou, Jan D Wegner, and Andreas Wieser. The perfect match: 3d point cloud matching with smoothed densities. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5545-5554, 2019.
Zan Gojeic, Caifa Zhou, Jan Wegner, Leonidas Guibas, and Tolga Birdal. Learning multiview 3d point cloud registration. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 1759-1769, 2020.
Thibault Groucix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. 3d- coded: 3d correspondences by deep deformation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 230-246 246, 2018.
Tong He, Yuanlu Xu, Shunsuke Saito, Stefano Soatto, and Tony Tung. Arch++: Animation-ready clothed human reconstruction revisited. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 11046-11056, 2021.
Haibin Huang, Evangelos Kalogerakis, Siddhartha Chaudhuri, Duygu Ceylan, Vladimir G Kim, and Ersin Yumer. Learning local shape descriptors from part correspondences with multiview convolutional networks. ACM Transactions Graphics (TOG), 37(1):1-144 2017.
Hao Huang and Yi Fang. Adaptive wavelet transformer network for 3d shape representation learning. In International Conference on Learning Representations, 2021.
Zeng Huang, Yuanlu Xu, Christoph Lassner, Hao Li, and Tony Tung. Arch: Animatable reconstruc- tion of clothed humans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3093-3102, 2020.
James Klatzow, Giovanni Dalmasso, Neus Martfnez-Abadiin James Sharpe, and Virginie Uhlmann. umatch: 3d shape correspondence for biological image data. Frontiers in Computer Science, pp. 7, 2022.
Chun-Liame Liang Li, Tomas Simon, Jason Saragih, Barnabás Póczos, and Yaser Sheikh. Lbs autoencoder: Self sepe-suue fitting of articulated meshes to point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11967-11976, 2019.
Or Litany, Tal Remez, Emanuele Rodola, Alex Bronstein, and Michael Bronstein. Deep functional maps: Structured prediction for dense shape correspondence. In Proceedings of the IEEE inter- national conference computer vision, pp. 5659-5667, 2017.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. ACM transactions graphics (TOG), 34(6):1-16, 2015.
Riccardo Marin, Simone Melzi, Emanuele Rodola, and Umberto Castellani. Farm: Functional automatic registration method for 3d human bodies. In Computer Graphics Forum, volume 39, pp. 160-173. Wiley Online Library, 2020a.
Riccardo Marin, Marie-Julie Rakotosaona, Simone Melzi, and Maks Ovsjanikov. Correspondence learning via linearl-invaarian embedding. Advances in Neural Information Processing Systems, 33:1608-1620, 2020b.
Dorian Nogneng, Simone Melzi, Emanuele Rodola, Umberto Castellani, Michael Bronstein, and Maks Ovsjanikov. Improved functional mappings via product preservation. In Computer Graph- ics Forum, volume 37, pp. 179-190. Wiley Online Library, 2018.
Maks Ovsjanikov, Etienne Corman, Michael Bronstein, Emanuele Rodolå, Mirela Ben-Chen, Leonidas Guibas, Frederic Chazal, and Alex Bronstein. Computing and processing correspon- dences with functional maps. In SICGRAPH ASIA 2016 Courses, pp. 1-60. 2016.
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. Expressive body capture: 3d hands, face, and body from single image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10975-10985, 2019.
11

---

Under review as a conference paper at ICLR 2023
Sergey Prokudin, Christoph Lassner, and Javier Romero. Efficient learning point clouds with basis point sets. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4332 4341, 2019.
Emanuele Rodolà, Michael Moeller, and Daniel Cremers. Point-wise map recovery and refinement from functional correspondence. arXiv preprint TrXiiv:506.0500 2015.
Jean-Michel Roufosse, Abhishek Sharma, and Maks Ovsjanikov. Unsupervised deep learning for structured shape matching. In Proceedings of the IEEE/CVF International Conference on Com- puter Vision, pp. 1617-1627, 2019.
Yusuf Sahillioglu and Yiicel Yemez. 3d shape correspondence by isometry-driven greedy optimiza- tion. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 453 458. IEEE, 2010.
Alcksandr Segal, Dirk Hachnel, and Sebastian Thrun. Generalized-icp. In Robotics: science and systems, volume 2, pp. 435. Seattle, WA, 2009.
Jacopo Serafin and Giorgio Grisetti. Niep: Dense normal based point cloud registration. In 2015 IEEF/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 742-749. IEEE, 2015.
Jian Sun, Maks Ovsjanikov, and Leonidas Guibas. A concise and provably informative multi-scale signature based on heat diffusion. In Computer forph, volume 28, pp. 1383-1392. Wiley Online Library, 2009.
Federico Tombari,
Samuele Salti, and Luigi Di Stefano. Unique signatures of histograms for local surface description. In European conference on computer vision, pp. 356-369. Springer, 2010.
Matthias Vestner, Roee Litman, Emanuele Rodola, Alex Bronstein, and Daniel Cremers. Prod- uet manifold filter: Non-rigid shape correspondence via kernel density estimation the product space. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp- 3327-3336, 2017.
Weiyue Wang, Duygu Ceylan, Radomir Mech, and Ulrich Neumann. 3dn: 3d deformation network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
1046,1046, 2019.
Yuxin Yao, Bailin Deng, Weiwei Xu, and Juyong Zhang. Quasi-newton solver for robust non- rigid registration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7600-7609, 2020.
Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao, and Thomas Funkhouser. 3dmatch: Learning local geometric descriptors from rgb-d reconstructions. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 1802-1811, 2017.
Wang Zeng, Wanli Ouyang, Ping Luo, Wentao Liu, and Xiaogang Wang. 3d human mesh regression with dense correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7054-7063, 2020.
Yiming Zeng, Yue Qian, Zhiyu Zhu, Junhui Hou, Hui Yuan, and Ying He. Cormet3d: Unsupervised end-to-end learning of dense correspondence for 3d point clouds. In Proceedings the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 6052 2021.
Boyao Zhou, Sean-Sébattien Franco, Federica Bogo, Bugra Tekin, and Edmond Boyer. Reconstruct- ing human body mesh from point clouds by adversarial gp network. In Proceedings of the Asian Conference on Computer Vision, 2020.
Silvia Zuffi and Michael J Black. The stitched puppet: A graphical model of 3d human shape and posc. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 3537-3546, 2015.
12


Reviews:
[{"review_id": "P4_pFOkbR8x", "reviewer": "Reviewer_KeGj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper the authors propose a neural model to deform a template based on the single-view 3D observations while correspondences between both states are implicitly computed. To that end, the authors exploit a feature constraint deformation network on the single-view point cloud as input. The method uses well-known ideas and modules in literature, providing competitive results with respect to an ample variety of approaches. ", "summary_of_the_review": "On balance, the method is simple yet effective, the results seem to be promising, but the technical contribution is a bit incremental. As pointed out, Some relevant experiments and details could improve the paper. \n", "strength_and_weaknesses": "In general, the paper is well written and it is clear enough. The motivation is clearly presented, and the discussion of competing approaches seems to be correct. \n \nq_j needs to be introduced in Eq. (1). \n \nFigure 1 should include a self-contained caption. To be honest, every figure in the paper should consider that. \n \nAccording to the learning and inference pipelines, I think the proposed method is a bit incremental. The idea of deforming a template by considering 2D observations is a well-known problem. Once the template is deformed, the initial human shape is fully estimated. To produce this type of results, the authors assume in training a template at rest and a complete shape in a particular state. In my opinion, the amount of 3D priors to learn the model is very high. At the end, this could be reduced to an alignment task with different sets. However, as the visible points in the set can be observed in the input image, this problem is tractable. In particular, the authors should clarify the differences with Groueix et al. (2018). \n\nSimilar results in terms of a chamfer distance could be derived from very different 3D poses, due to the projection ambiguity on deformable bodies. How can that be handled?  \n\nThe method provides competitive solutions with respect to a wide variety of competing approaches. Both quantitative and qualitative evaluation are well executed. I miss some experiments with large occlusions in the input data, the presence of holes, and so on. That means I would like to see more challenging and realistic scenarios. \n\nWhat about the scale? Could very different bodies be matched? For example, using the current template, incorporating a child as an input data. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NdWxJplKru", "reviewer": "Reviewer_tJR5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors address the task of 3D human mesh registration with key focus on registering a complete template to (incomplete) single view point cloud. Authors show better performance than baselines on the registration as well as correspondence prediction task.", "summary_of_the_review": "I have serious concerns with the way authors generate single view data. This is not how single view data is generated and this is the primary contribution of the work (handling single view data). The novelty also appears to be limited (see weakness) as compared to 3dcoded but I'd like to hear from authors on this.", "strength_and_weaknesses": "Strengths:\n+ The task of registering partial/ single view data with a common template is quite challenging and interesting.\n+ The proposed methodology is simple and makes intuitive sense.\n\nWeaknesses:\n- I find the experimental setup (Sec 4.1) problematic. The paper claims to work with single view point clouds, but this is actually not the case. Authors use a very wrong way to generate single view data. They remove the back of the SMPL model in canonical pose and repose the remaining vertices to get the training data. This is not true for actual single view data. *The parts that are missing in actual single view data are the ones that are occluded from the camera view due to occlusion not a fixed set of vertices*. The most common source of occlusion are hands as they have high degree of freedom and can cover various parts of body. In this work the hands and hand-related occlusions are totally ignored.\nModel trained with this data would not have to reason about the occlusion at all and can just memorise how to complete a smooth back, which is fixed as fixed SMPL vertices are always removed.\nThis process is used both for train and testing, making the evaluation highly unreliable.\n\n- Technical  novelty appears to be limited. The encoder-decoder formulation and template deformation look very similar to 3D-CODED. The new component is enforcing that the features from the partial point cloud and the complete point cloud should be same. This is enforced via cosine similarity on the encoded features. Can the authors comment on this?\n\n- Authors primarily compare their method with works that deal with full point cloud registration. Works like IPNet, ECCV'20 (not cited) are more suitable baselines as they have shown that they can complete the missing point cloud using implicit functions and then perform 3D registration.\n\n- Why not report results on FAUST Intra?\n\nClarifications:\n- The feature decoding and per-vertex deformation approach is very similar to 3DCODED, which requires several initialisations to handle global orientation. Is this also the case for the proposed approach?\n\n- Do the authors use the official FAUST test set (GT not available, but results need to be submitted to their portal for evaluation)? I've seen a few works overfitting to FAUST train set and reporting results. This is a bit of cheating :P Just want to make sure.\n\n- More recent works like LoopReg, NeurIPS'20 and IPNet, ECCV'20 show registration on dressed human shapes, which is more challenging than the current setup where synthetic SMPL is used. Can the authors comment why are they restricting themselves to this old setting?\n\nMinor suggestions:\n- Please make figure captions more informative.\n- Fig. 2 just shows cosine similarity loss. Is it really necessary?\n- Typo: \"The source and Source M are\" below eq. 5", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GD23bcw-PEp", "reviewer": "Reviewer_PTZp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper propose a feature constraint deformation network (FCDNet), the networked has tested on different benchmarks.  No clear contribution for this work.", "summary_of_the_review": "Due to the overall major problems, the paper needs to major revision and make the contribution more clear and offer the fair comparison before publication. ", "strength_and_weaknesses": "Some major problems are show below.\n\n1. The main contribution of the paper is unclear. The author did not present it explicitly. Can not find it anywhere.\n\n2. The author did a severe mislead on quantitative analysis on \"MPI Faust dataset\". For most of the experiments, the author did the comparison with 3D-CODED, which is an ECCV 18 paper. I have checked the MPI FAUST dataset for Inter-subject challenge, the SOTA can achieve 1.164cm, at least 5 methods (2 of them are already published, 1 has released the code[1]) has better results than current paper.\n\n3. In the test data, what's the SMPL and SMPL-X mean? From my point of view, they are not dataset, they are just parametric model. Based on SMPL or SMPL-X, the community create several datasets, such as SURREAL or AMASS. \n\n4. The FCDnet do not have clear Novelty. Either end-to-end framework or feature constraints have been widely used.\n\n5. The figures are not clear, please emphasize the difference. \n\n[1] Deep Virtual Markers for Articulated 3D Shapes. ICCV 2021\n[2] Learning from Synthetic Humans (SURREAL). CVPR 2017\n[3] AMASS: Archive of Motion Capture As Surface Shapes. CVPR 2019", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "DKGHiickF7", "reviewer": "Reviewer_waCJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a method to estimate human body shape correspondence from both the full scans and from single-view depth data. The authors build on a well established method 3D-CODED but they add a mechanism to support partial depth observations as well. Specifically, the authors train an encoder extracting a global code both on the full scans and on corresponding partial observations and introduce an additional loss function to force the global code to remain the same.", "summary_of_the_review": "While the authors present a simple and effective method which yields SotA results on the chosen datasets, the paper presentation is of rather low quality. Specifically, the authors often use confusing and unclear formulations, filler statements and unjustified claims. There are quite a few typos (some of them listed above). Some parts are only explained in words in a bit confusing way where a simple mathematical formulation would help (see Weaknesses). The figures presented in the Experiment section use a color coding which does not visually help unless a GT is provided next to them as well. Overall, I believe the quality of the submission is not up to the standard of the ICLR and the paper could use some more polishing.", "strength_and_weaknesses": "# Strengths\nThe method shows simple yet effective approach to extend a template based human body reconstruction/correspondence estimation method to work on partial observations and report quantitative SotA results.\n\n# Weaknesses\n## Related Work\nThe main focus of the method is to support shape correspondence estimation from single-view (RGBD) observations. While the Related Work section discusses various methods for human shape correspondence estimation, it never makes a distinction between the methods which require a full scan on the input and the ones which can handle single-view input only. The authors should position this paper within the SotA so that it is clear what work was already done and who are the competitors. The authors only mention: \"However, these methods are usually proposed for matching complete 3D human models\". What do the authors mean by \"usually\"? Do some of the  methods in fact support a single-view input? If so, which ones are those? The authors further mention: \"there are relatively few methods applicable to single-view human body data collected by a single RGB-D camera.\" However, the authors never discuss any such method.\n\n## Methodology\n- Fig. 2: The Figure seems redundant as the notion of cosine similarity is well known. I would suggest removing the Figure in favor of more substantial content (e.g. experimental results).\n- The last paragraph - the explanation of correspondence retrieval: The text is quite confusing and it is not clear how exactly the authors find the correspondences. The arrows in Fig. 2 indicate that the points are projected from the template to the observations, but more typical approach would be to project a point from observation A to the template and then from the deformed template to the deformed observation B. This would be clarified if the authors perhaps formalized the process with a mathematical formula.\n\n## Experiments\n- \"Unlike many current human reconstruction methods that rely on a large number of scanned 3D mannequins as training data\" -> Please provide citations for this claim.\n- The paragraph \"evaluation criterion\": Please formulate and clarify the paragraph as it is unclear what your metric is and whether it applies in the same way for all the datasets.\n- Fig. 5: The caption talks about a 3D human correspondence task, but the image shows a reconstruction instead.\n- Figures 5, 6, 7 - If a reconstruction of only a single frame is shown, the color coding of the human does not show any useful information unless it is directly compared to the GT color coded human shape.\n- The authors never show qualitative comparison of correspondences between various body poses despite this application being advertised in the Abstract.\n\n## Presentation\nIn general, the paper is quite difficult to read, due to the obscure formulations the authors choose to use and it is thus often unclear what the authors had in mind. Some examples are listed below but the paper contains many more of these:\nStatements where it is hard to understand what the authors mean:\n- \"The parametric human model has been widely used in various human-related applications and in 3D human mesh correspondence because it provides sufficient scope to modify the resulting model.\" - What do the authors mean by \"sufficient scope to modify the resulting model\"?\n- \"In representation learning, one of the main challenges is to design appropriate loss functions for supervising features with different abilities\" - What do the authors mean by \"supervising features with different abilities\"?\n- \"Meanwhile, the decoder generates a completely transformed template with higher promise\" - What do the authors mean by \"higher promise\"?\n- \"The deformed models have the same vertex definition, definite semantic information, and same face connections as the template.\" - What do the authors mean by \"same vertex definition\"?\n- \"Motivated by Deng et al. (2019), we attempted to penalize the loss in the feature space,\" - Perhaps better formulation would be \"penalize the XYZ by adding a loss ABC\".\n- \"this feature is taken as the observed quantity in Equation (1) and in the human template as an input pair to output the deformed human shape vertices.\" - What input pair?\n- \" then constrain the single-view human shape and intact human shape in the feature space as close as possible, so as to use the same network framework for both the single-view and full-body shapes. \" - Please reformulate (the use of \"so as to\").\n- \"Here, b is a scale coefficient for controlling the influence of the eigenvector similarity on the overall reconstruction effect.\" - The use of any eigenvectors was not either mentioned until this point or after. What eigenvectors do the authors have in mind?\n- \"Below, xi is the serial number \" -> Perhaps index number?\n\nFurthermore, there are plenty of unjustified or unexplained statements which come across as filler sentences rather than content of substance, unless reformulated and/or better cited. Some examples are listed below:\nUnjustified, unexplained or incorrect statements such as:\n- \"The obtained network can better realize the deformation and reconstruction of the single-view human body.\" - Better in what sense?\n- \"The proposed method selects a pa- parameterized human body model with a standard pose as a template,\" - What is the \"standard pose\"?\n- \"deforms it into source and target models with arbitrary poses and body shapes, respectively.\" - It was never explained what are the source and target models.\n- \" which reduces the workload for man- ual labeling. \" - It does not reduce the workload but removes it completely.\n\n## Miscellaneous:\n- The authors use the term \"input model\" to refer to the input data/observations, which might be confusing to the reader. Please reformulate.\npg. 6: \"source M and target M\" - Calling the source and target as source M and target M sounds confusing, perhaps use e.g. M_{s}, M_{t}.\n\n## Questions for the authors:\nEq. 4: Why did the authors choose to formulate the final loss as a convex combination of the two members Lr and Lf, rather than a more common way to combine the individual loss terms involving a linear combination, e.g. Loss = LR + b * LS?\n\nTypos:\n- pg1: \"This makes the correspondence process more challenging.\" -> \"This makes the correspondence estimation process more challenging.\"\n- pg2: \"global feature coded from the input shape.\" -> \"global feature encoded from the input shape.\"\n- pg.4: \"where,pj denotes\" -> \"where pj denotes\"\n- pg.4: \" simplified PointNet network\" -> addd citation\n- pg.5: \"where,M\" -> \"where M\"\n- pg.5: \"whereSc  -> \"where Sc\"\n- pg.5: \"it is will be unable \" -> \"it will be unable \"\n- pg.6: \"(Groueix et al. (2018))to\" -> \"(Groueix et al. (2018)) to\"\n- pg.6: \"EXPERIMENT\" -> \"EXPERIMENTS\"\n- pg.7: \"shown in Equation . \" -> \"shown in Equation 6\"\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject