3D Pose Transfer with Correspondence Learning and Mesh Refinement
Year: 2022

Abstract:
3D pose transfer is one of the most challenging 3D generation tasks. It aims to transfer the pose of a source mesh to a target mesh and keep the identity (e.g., body shape) of the target mesh. Some previous works require key point annotations to build reliable correspondence between the source and target meshes, while other methods do not consider any shape correspondence between sources and targets, which leads to limited generation quality. In this work, we propose a correspondence-refinement network to achieve the 3D pose transfer for both human and animal meshes. The correspondence between source and target meshes is first established by solving an optimal transport problem. Then, we warp the source mesh according to the dense correspondence and obtain a coarse warped mesh. The warped mesh will be better refined with our proposed Elastic Instance Normalization, which is a conditional normalization layer and can help to generate highquality meshes. Extensive experimental results show that the proposed architecture can effectively transfer the poses from source to target meshes and produce better results with satisfied visual performance than state-of-the-art methods. Our code and data are available at https://github.com/ChaoyueSong/3d-corenet.

heading: 1 Introduction
text: 3D pose transfer has been drawing a lot of attention from the vision and graphics community. It has potential applications in 3D animated movies and games by generating new poses for existing shapes and animation sequences. 3D pose transfer is a learning-driven generation task which is similar to style transfer on 2D images. As shown in Figure 1, pose transfer takes two inputs, one is identity mesh that provides mesh identity (e.g., body shape), the other is pose mesh that provides the pose of mesh. It aims at transferring the pose of a source pose mesh to a target identity mesh and keeping the identity of the target identity mesh.
A fundamental problem for previous methods is to build reliable correspondence between source and target meshes. It can be very challenging when the source and target meshes have significant differences. Most of the previous methods try to solve it with the help of user effort or other additional inputs, such as key point annotations [3, 34, 41], etc. Unfortunately, it is time-consuming to obtain such additional inputs that will limit the usage in practice. In [37], they proposed to implement pose transfer without correspondence learning. Their method is convenient but the performance will be degraded since they do not consider the correspondence between meshes. In this work, we propose a COrrespondence-REfinement Network (3D-CoreNet) to solve the pose transfer problem for both the human and animal meshes. Like [37], our method does not need key point annotations or other
∗Corresponding author
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
additional inputs. We learn the shape correspondence between identity and pose meshes first, then we warp the pose mesh to a coarse warped output according to the correspondence. Finally, the warped mesh will be refined to have a better visual performance. Our method does not require the two meshes to have the same number or order of vertices.
For the correspondence learning module, we treat the shape correspondence learning as an optimal transport problem to learn the correspondence between meshes. Our network takes vertex coordinates of identity and pose meshes as inputs. We extract deep features at each vertex using point cloud convolutions and compute a matching cost between the vertex sets with the extracted features. Our goal is to minimize the matching cost to get an optimal matching matrix. With the optimal matching matrix, we warp the pose mesh and obtain a coarse warped mesh. We then refine the warped output with a set of elastic instance normalization residual blocks, the modulation parameters in the normalization layers are learned with our proposed Elastic Instance Normalization (ElaIN). In order to generate smoother meshes with more details, we introduce a channel-wise weight in ElaIN to adaptively blend feature statistics of original features and the learned parameters from external data, which help to keep the consistency and continuity of the original features.
Our contributions can be summarized as follows: • We solve the 3D pose transfer problem with our proposed correspondence-refinement network. To the best of our knowledge, our method is the first to learn the correspondence between different meshes and refine the generated meshes jointly in the 3D pose transfer task. •We learn the shape correspondence by solving an optimal transport problem without any key point annotations and generate high-quality final meshes with our proposed elastic instance normalization in the refinement module. • Through extensive experiments, we demonstrate that our method outperforms state-of-the-art methods quantitatively and qualitatively on both human and animal meshes.

heading: 2 Related work

heading: 2.1 Deep learning methods on 3D data
text: The representations of 3D data are various, like point clouds, voxels and meshes. 3DShapeNets [40] and VoxNet [25] propose to learn on volumetric grids. Their methods cannot be applied on complex data due to the sparsity of data and computation cost of 3D convolution. PointNet [29] uses a shared MLP on every point followed by a global max-pooling. Following PointNet, some hierarchical architectures have been proposed to aggregate local neighborhood information with MLPs [20, 30]. [13, 35] proposed mesh variational autoencoder to learn mesh features whose methods consume a large amount of computing resources due to their fully-connected networks. Many works use graph convolutions with mesh down- and up-sampling layers [15], like CoMA [31], CAPE [24] based on ChebyNet [10], and [45] based on SpiralNet [21], SpiralNet++ [16], etc. They all need a template to implement their hierarchical structure which is not applicable to real-world problems. In this work, we use mesh as the representation of 3D shape and shared weights convolution layers in the network.

heading: 2.2 3D pose transfer
text: Deformation transfer in graphics aims to apply the deformation exhibited by a source mesh onto a different target mesh [34]. 3D pose transfer aims to generate a new mesh based on the knowledge of a pair of source and target meshes. In [3, 34, 41, 42], the methods all require to label the corresponding landmarks first to deal with the differences between meshes. Baran et al. [2] proposed a method that infers a semantic correspondence between different poses of two characters with the guidance of example mesh pairs. Chu et al. [8] proposed to use a few examples to generate results which will make it difficult to automatically transfer pose. For this problem, Gao et al. [14] proposed to use cycle consistency to achieve the pose transfer. However, their method cannot deal with new identities due to the limitations of the visual similarity metric. In [37], they solved the pose transfer via the latest technique for image style transfer. Their work does not need other guidance, but the performance is also restrained since they do not learn any correspondence. To solve the problems, our network learns the correspondence and refines the generated meshes jointly.

heading: 2.3 Correspondence learning
text: In CoCosNet [44], they introduced a correspondence network based on the correlation matrix between images without any constraints. To learn a better matching, we proposed to use optimal transport to learn the correspondence between meshes. Recently, optimal transport has received great attention in various computer vision tasks. Courty et al. [9] perform the alignment of the representations in the source and target domains by learning a transportation plan. Su et al. [33] compute the optimal transport map to deal with the surface registration and shape space problem. Other applications include generative model [1, 6, 11, 39], scene flow [28], semantic correspondence [22] and etc.

heading: 2.4 Conditional normalization layers
text: After normalizing the activation value, conditional normalization uses the modulation parameters calculated from the external data to denormalize it. Adaptive Instance Normalization (AdaIN) [18] aligns the mean and variance between content and style image which achieves arbitrary style transfer. Soft-AdaIN [7] introduces a channel-wise weight to blend feature statistics of content and style image to preserve more details for the results. Spatially-Adaptive Normalization (SPADE) [27] can better preserve semantic information by not washing away it when applied to segmentation masks. SPAdaIN [37] changes batch normalization [19] in SPADE to instance normalization [36] for 3D pose transfer. However, it will break the consistency and continuity of the feature map when doing the denormalization, which has a bad influence on the mesh smoothness and detail preservation. To address this problem, our ElaIN introduces an adaptive weight to implement the denormalization.

heading: 3 Method
text: Given a source pose mesh and a target identity mesh, our goal is to transfer the pose of source mesh to the target mesh and keep the identity of the target mesh. In this section, we will introduce our end-to-end Correspondence-Refinement Network (3D-CoreNet) for 3D pose transfer.
A 3D mesh can be represented as M(I, P,O), where I denotes the mesh identity, P represents the pose of the mesh, O is the vertex order of the mesh. Given two meshes Mid =M(I1, P1, O1) and Mpose =M(I2, P2, O2), we aim to transfer the pose of Mpose to Mid and generate the output mesh Moutput = M
′(I1, P2, O1). In our 3D-CoreNet, we take Vid ∈ RNid×3 and Vpose ∈ RNpose×3 as inputs, which are the (x, y, z) coordinates of the mesh vertices. Nid and Npose denote the number of vertices of identity mesh and pose mesh respectively.
As shown in Figure 2, the vertices of the meshes are first fed into the network to extract multi-scale deep features. We calculate the optimal matching matrix with the vertex feature maps by solving an optimal transport problem, then we warp the pose mesh according to the matrix and obtain the warped mesh. Finally, the warped mesh is refined to the final output mesh with our proposed elastic instance normalization (ElaIN). The output mesh combines the pose from the source mesh and the identity from the target. And it inherits the vertex order from the identity mesh.

heading: 3.1 Correspondence learning
text: Given an identity mesh and a pose mesh, our correspondence learning module calculates an optimal matching matrix, each element of the matching matrix represents the similarities between two vertices in the two meshes. The first step in our shape correspondence learning is to compute a correlation matrix with the extracted features, which is based on cosine similarity and denotes the matching similarities between any two positions from different meshes. However, the matching scores in the correlation matrix are calculated without any additional constraints. To learn a better matching, we solve this problem from a global perspective by modeling it as an optimal transport problem.
Correlation matrix We first introduce our feature extractor which aims to extract features for the unordered input vertices. Close to [37], our feature extractor consists of 3 stacked 1× 1 convolution and Instance Normalization layers, the activation functions applied are all LeakyReLU. Given the extracted vertex feature maps fid ∈ RD×Nid , fpose ∈ RD×Npose of identity and pose meshes (D is the channel-wise dimension), a popular method to compute correlation matrix is using the cosine similarity [43, 44]. Concretely, we compute the correlation matrix C ∈ RNid×Npose as:
C(i, j) = fid(i)
>fpose(j)
‖fid(i)‖ ‖fpose(j)‖ (1)
where C(i, j) denotes the individual matching score between fid(i) and fpose(j) ∈ RD, fid(i) and fpose(j) represent the channel-wise feature of fid at position i and fpose at j.
Optimal transport problem To learn a better matching with additional constraints in this work, we model our shape correspondence learning as an optimal transport problem. We first define a matching matrix T ∈ RNid×Npose+ between identity and pose meshes. Then we can get the total correlation as ∑ ij C(i, j)T(i, j). The aim will be maximizing the total correlation score to get an optimal matching matrix Tm.
We treat the correspondence learning between identity and pose meshes as the transport of mass. A mass which is equal to N−1id will be assigned to each vertex in the identity mesh, and each vertex in pose mesh will receive the mass N−1pose from identity mesh through the built correspondence between vertices. Then if we define Z = 1−C as the cost matrix, our goal can be formulated as a standard optimal transport problem by minimizing the total matching cost,
Tm = argmin T∈RNid×Npose+ ∑ ij Z(i, j)T(i, j) s.t. T1Npose = 1NidN −1 id , T >1Nid = 1NposeN −1 pose.
(2)
where 1Nid ∈ RNid and 1Npose ∈ RNpose are vectors whose elements are all 1. The first constraint in Eq. 2 means that the mass of each vertex in Mid will be entirely transported to some of the vertices in Mpose. And each vertex in Mpose will receive a mass N−1pose from some of the vertices in Mid with the second constraint. This problem can be solved by the Sinkhorn-Knopp algorithm [32]. The details of the solved process will be given in the supplementary material.
With the matching matrix, we can warp the pose mesh and obtain the vertex coordinates Vwarp ∈ RNid×3 of the warped mesh,
Vwarp(i) = ∑ j Tm(i, j)Vpose(j) (3)
the warped mesh Mwarp inherits the number and order of vertex from identity mesh and can be reconstructed with the face information of identity mesh as shown in Figure 2.

heading: 3.2 Mesh refinement
text: In this section, we introduce our mesh refinement module which refines the warped mesh to the desired output progressively.
Elastic instance normalization Previous conditional normalization layers [18, 27, 37] used in different tasks always calculated their denormalization parameters only with the external data. We argue that it may break the consistency and continuity of the original features. Inspired by [7], we propose Elastic Instance Normalization (ElaIN) which blends the statistics of original features and the learned parameters from external data adaptively and elastically.
As shown in Figure 2, the warped mesh is flatter than we desired and is kind of out of shape, but it inherits the pose from the source mesh successfully. Therefore, we refine the warped mesh with the identity feature maps to get a better final output. Here, we let hi ∈ RSi×Di×Ni as the activation value before the i-th normalization layer, where Si is the batch size, Di is the dimension of feature channel and N i is the number of vertex. At first, we normalize the feature maps of the warped mesh with instance normalization and the mean and standard deviation are calculated across spatial dimension (n ∈ N i) for each sample s ∈ Si and each channel d ∈ Di,
µi = 1
N i ∑ n hiwarp (4)
σi =
√ 1
N i ∑ n (hiwarp − µi)2 + (5)
then the feature maps of the identity mesh are fed into a 1× 1 convolution layer to get hiid, which shares the same size with hiwarp. We calculate the mean of h i warp, h i id to make them S
i ×Di × 1 tensors. The tensors are then concatenated in channel dimension to get a Si × (2Di) × 1 tensor.
A fully-connected layer is employed to compute an adaptive weight w(hiwarp, h i id) ∈ RS
i×Di×1
with the concatenated tensor. With w(hiwarp, h i id), we can define the modulation parameters of our normalization layer.
γ′(hiwarp, h i id) = w(h i warp, h i id)γ i + (1− w(hiwarp, hiid))σi, β′(hiwarp, h i id) = w(h i warp, h i id)β i + (1− w(hiwarp, hiid))µi. (6)
where γi and βi are learned from the identity feature hiid with two convolution layers. Finally, we can scale the normalized hiwarp with γ ′ and shift it with β′,
ElaIN(hiwarp, h i id) = γ ′(hiwarp, h i id)(
hiwarp − µi
σi ) + β′(hiwarp, h i id) (7)
Mesh refinement module Our mesh refinement module is designed to refine the warped mesh progressively. Following [27, 37, 44], We design the ElaIN residual block with our ElaIN in the form of ResNet blocks [17]. As shown in Figure 2, our architecture contains l ElaIN residual blocks. Each of them consists of our proposed ElaIN followed by a simple convolution layer and LeakyReLU. With the ElaIN residual blocks, the warped mesh is refined to our desired high-quality output. Please refer to the supplementary material for the detailed architecture of ElaIN.

heading: 3.3 Loss function
text: We jointly train the correspondence learning module along with mesh refinement module by minimizing the following loss functions,
Reconstruction loss Following [37], we train our network with the supervision of the ground truth mesh Mgt =M(I1, P2, O1). We first process the ground truth mesh to have the same vertex order as the identity mesh. Then we define the reconstruction loss by calculating the point-wise L2 distance between the vertices of Moutput and Mgt,
Lrec = ‖Voutput − Vgt‖22 (8)
where Voutput and Vgt ∈ RNid×3 are the vertices of Moutput and Mgt respectively. Notice that they all share the same size and order with the vertices of the identity mesh. With the reconstruction loss, the mesh predicted by our model will be closer to the ground truth.
Edge loss In this work, we also introduce edge loss which is often used in 3D mesh generation tasks [26, 37, 38]. Since the reconstruction loss does not consider the connectivity of mesh vertices, the generated mesh may suffer from flying vertices and overlong edges. Edge loss can help penalize flying vertices and generate smoother surfaces. For every v ∈ Voutput, let N (v) be the neighbor of v, the edge loss can be defined as,
Ledg = ∑ v ∑ p∈N (v) ‖v − p‖22 (9)
then we can train our network with the combined loss function L,
L = λrecLrec + Ledg (10)
where λrec denotes the weight of reconstruction loss.

heading: 4 Experiment
text: Datasets. For the human mesh dataset, we use the same dataset generated by SMPL [23] as [37]. This dataset consists of 30 identities with 800 poses. Each mesh has 6890 vertices. For the training data, we randomly choose 4000 pairs (identity and pose meshes) from 16 identities with 400 poses and shuffle them every epoch. The ground truth meshes will be determined according to the identity and pose parameters from the pairs. Before feeding into our network, every mesh will be shuffled randomly to be close to the real-world problem. Notice that the ground truth mesh will share the
same vertex order with identity mesh for the convenience of supervised training and evaluation. For all input meshes, we shift them to the center according to their bounding boxes. When doing the test, we evaluate our model with 14 new identities with 200 unseen poses. We randomly choose 400 pairs for testing. They will be pre-processed in the same manner as the training data. To further evaluate the generalization of our model, we also test our model with FAUST [5] and MG-dataset [4] in the experiment.
For the animal mesh dataset, we generate animal training and test data using SMAL model [46]. This dataset has 41 identities with 600 poses. The 41 identities are 21 felidae animals (1 cat, 5 cheetahs, 8 lions, 7 tigers), 5 canidae animals (2 dogs, 1 fox, 1 wolf, 1 hyena), 8 equidae animals (1 deer, 1 horse, 6 zebras), 4 bovidae animals (4 cows), 3 hippopotamidae animals (3 hippos). Every mesh has 3889 vertices. For the training data, we randomly choose 11600 pairs from 29 identities (16 felidae animals, 3 canidae animals, 6 equidae animals, 2 bovidae animals, 2 hippopotamidae animals) with 400 poses. For the test data, we randomly choose 400 pairs from other 12 identities (5 felidae animals, 2 canidae animals, 2 equidae animals, 2 bovidae animals, 1 hippopotamidae animal) with 200 poses. All the inputs are pre-processed in the same manner as we do in the human mesh.
Evaluation metrics. Following [37], we use Point-wise Mesh Euclidean Distance (PMD) as one of our evaluation metrics. PMD is the L2 distance between the vertices of the output mesh and the ground truth mesh. We also evaluate our model with Chamfer Distance (CD) and Earth Mover’s Distance (EMD) proposed in [12]. For PMD, CD and EMD, the lower is better.
Implementation details. λrec in the loss function is set as 2000. We implement our model with Pytorch and use Adam optimizer. Please refer to the supplementary material for the details of the network. Our model is trained for 200 epochs on one RTX 3090 GPU, the learning rate is fixed at 1e-4 in the first 100 epochs and decays 1e-6 each epoch after 100 epochs. The batch size is 8.

heading: 4.1 Comparison with the state-of-the-arts
text: In this section, we compare our method with Deformation Transfer (DT) [34] and Neural Pose Transfer (NPT) [37]. DT needs to rely on the corresponding points labeled by user and a reference mesh, as the additional inputs. Therefore, we test DT with the reference mesh and 11, 19 labeling points on animal data and human data respectively. For [37], their method does not consider any correspondence. We train their model using the implementations provided by the authors. The qualitative results tested on SMPL [23] and SMAL [46] are shown in Figure 3 and Figure 4. As we can see, when tested on human data, DT and our method can produce better results that are close to the ground truth. However, it is very time-consuming for DT when dealing with a new identity and pose mesh pair. The results generated by NPT always do not learn the right pose and are not very smooth on the arms or legs since they do not consider any correspondence. When tested on animal data, DT fails to transfer the pose even if we add more labeling points. Their method does not work when the identity of the mesh pairs are very different. NPT can produce very flat legs and wrong direction faces. In comparison, our method still produces satisfactory results efficiently.
We adopt Point-wise Mesh Euclidean Distance (PMD), Chamfer Distance (CD) and Earth Mover’s Distance (EMD) to evaluate the generated results of different methods. All metrics are calculated
between the ground truth and the predicted results. The quantitative results are shown in Table 1. Our 3D-CoreNet outperforms other methods in all metrics over two datasets. When doing pose transfer on animal data that contains more different identities, our method has more advantages.

heading: 4.2 Ablation study
text: In this section, we study the effectiveness of several components in our 3D-CoreNet on human data. At first, we test our model without the refinement module. We only use our correspondence module with the correlation matrix C or the optimal matching matrix Tm respectively. Here, the warped mesh will be viewed as the final output and the reconstruction loss will be calculated between the warped mesh and the ground truth. Then we will compare our ElaIN with SPAdaIN in [37] to verify the effectiveness of ElaIN. And we also test the importance of edge loss Ledg . The results are shown in Table 2 and Figure 5. We evaluate the variants with PMD, CD and EMD. As we can see, when we do not add our refinement module, the model does not perform well both qualitatively and quantitatively. And using the optimal matching matrix has a better performance than using correlation matrix. When we replace our ElaIN with SPAdaIN, the surface of the mesh has clear artifacts and is not smooth. The metrics are also worse than the full model. We can know that ElaIN is very helpful in generating high quality results. We also evaluate the importance of Ledg . The connection between vertices will be better and smoother with the edge loss.

heading: 4.3 Generalization capability
text: To evaluate the generalization capability of our method, we evaluate it on FAUST [5] and MG-dataset [4] in this section. Human meshes in FAUST have the same number of vertices as SMPL [23] and have more unseen identities. In MG-dataset, the human meshes are all dressed which have 27554 vertices each and have more realistic details. As shown in Figure 6, our method can also have a good performance on FAUST and MG-dataset. In the first group, we transfer the pose from FAUST to the identity in SMPL. In the second group, we transfer the pose from SMPL to the identity in MG-dataset. Both of them transfer the pose and keep the identity successfully.

heading: 5 Conclusion
text: In this paper, we propose a correspondence-refinement network (3D-CoreNet) to transfer the pose of source mesh to the target mesh while retaining the identity of the target mesh. 3D-CoreNet learns the correspondence between different meshes and refine the generated meshes jointly. Our method generates high-quality meshes with the proposed ElaIN for refinement. Compared to other methods, our model learns the correspondence without key point labeling and achieves better performance when working on both human and animal meshes. In the future, we will try to achieve the 3D pose transfer in an unsupervised manner.

heading: Acknowledgements
text: This study is supported under the RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner. This work is supported by A*STAR through the Industry Alignment Fund - Industry Collaboration Projects Grant. This work is also supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG-RP-2018-003), and the MOE Tier-1 research grants: RG28/18 (S) and RG95/20.

Review summaries:
This paper proposes a 3D-CoreNet  to solve the 3D pose transfer problem. The 3D-CoreNet consists of two steps: a correspondence learning step and refinement step. Firstly, the authors treat  the shape correspondences learning problem as  an optimal transport problem, which can be solved without using any correspondence annotations. A coarse warped mesh can be obtained from the learned correspondence. Then, an Elastic INstance Normalization (ElaIN) is introduced to the refinement module in order to blend statistics of original features and the learned parameters from external data. State-of-the-art performance has been achieved with the proposed 3D-CoreNet.  

This paper tackles the problem of 3D pose transfer: given an identity mesh Mi and a posed mesh Mp, the key problem is to output a mesh M' with the same id with Mi and the same pose with Mp. The proposed method is in a coarse-to-fine manner. It firstly ultizes optimal transport to learn a coarse correspondence, with which a coarse mesh can be generated. After that, a novel "elastic instance normalization" operation is proposed for mesh refinement. 

This paper investigates transferring the pose of a source mesh to a target mesh while keeping its identity. Given a source pose mesh and a target identity mesh, the proposed method first learn the shape correspondence by solving an optimal transport problem and then refine the mesh with the proposed Elastic Instance Normalization. Both quantitative and qualitative results are strong on both human and animal mesh transfer.