3D Pose Transfer with Correspondence Learning and Mesh Refinement

1 Introduction . 3D pose transfer has been drawing a lot of attention from the vision and graphics community . It has potential applications in 3D animated movies and games by generating new poses for existing shapes and animation sequences . 3D pose transfer is a learning-driven generation task which is similar to style transfer on 2D images . As shown in Figure 1 , pose transfer takes two inputs , one is identity mesh that provides mesh identity ( e.g. , body shape ) , the other is pose mesh that provides the pose of mesh . It aims at transferring the pose of a source pose mesh to a target identity mesh and keeping the identity of the target identity mesh . A fundamental problem for previous methods is to build reliable correspondence between source and target meshes . It can be very challenging when the source and target meshes have significant differences . Most of the previous methods try to solve it with the help of user effort or other additional inputs , such as key point annotations [ 3 , 34 , 41 ] , etc . Unfortunately , it is time-consuming to obtain such additional inputs that will limit the usage in practice . In [ 37 ] , they proposed to implement pose transfer without correspondence learning . Their method is convenient but the performance will be degraded since they do not consider the correspondence between meshes . In this work , we propose a COrrespondence-REfinement Network ( 3D-CoreNet ) to solve the pose transfer problem for both the human and animal meshes . Like [ 37 ] , our method does not need key point annotations or other ∗Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . additional inputs . We learn the shape correspondence between identity and pose meshes first , then we warp the pose mesh to a coarse warped output according to the correspondence . Finally , the warped mesh will be refined to have a better visual performance . Our method does not require the two meshes to have the same number or order of vertices . For the correspondence learning module , we treat the shape correspondence learning as an optimal transport problem to learn the correspondence between meshes . Our network takes vertex coordinates of identity and pose meshes as inputs . We extract deep features at each vertex using point cloud convolutions and compute a matching cost between the vertex sets with the extracted features . Our goal is to minimize the matching cost to get an optimal matching matrix . With the optimal matching matrix , we warp the pose mesh and obtain a coarse warped mesh . We then refine the warped output with a set of elastic instance normalization residual blocks , the modulation parameters in the normalization layers are learned with our proposed Elastic Instance Normalization ( ElaIN ) . In order to generate smoother meshes with more details , we introduce a channel-wise weight in ElaIN to adaptively blend feature statistics of original features and the learned parameters from external data , which help to keep the consistency and continuity of the original features . Our contributions can be summarized as follows : • We solve the 3D pose transfer problem with our proposed correspondence-refinement network . To the best of our knowledge , our method is the first to learn the correspondence between different meshes and refine the generated meshes jointly in the 3D pose transfer task . •We learn the shape correspondence by solving an optimal transport problem without any key point annotations and generate high-quality final meshes with our proposed elastic instance normalization in the refinement module . • Through extensive experiments , we demonstrate that our method outperforms state-of-the-art methods quantitatively and qualitatively on both human and animal meshes . 2 Related work . 2.1 Deep learning methods on 3D data . The representations of 3D data are various , like point clouds , voxels and meshes . 3DShapeNets [ 40 ] and VoxNet [ 25 ] propose to learn on volumetric grids . Their methods can not be applied on complex data due to the sparsity of data and computation cost of 3D convolution . PointNet [ 29 ] uses a shared MLP on every point followed by a global max-pooling . Following PointNet , some hierarchical architectures have been proposed to aggregate local neighborhood information with MLPs [ 20 , 30 ] . [ 13 , 35 ] proposed mesh variational autoencoder to learn mesh features whose methods consume a large amount of computing resources due to their fully-connected networks . Many works use graph convolutions with mesh down- and up-sampling layers [ 15 ] , like CoMA [ 31 ] , CAPE [ 24 ] based on ChebyNet [ 10 ] , and [ 45 ] based on SpiralNet [ 21 ] , SpiralNet++ [ 16 ] , etc . They all need a template to implement their hierarchical structure which is not applicable to real-world problems . In this work , we use mesh as the representation of 3D shape and shared weights convolution layers in the network . 2.2 3D pose transfer . Deformation transfer in graphics aims to apply the deformation exhibited by a source mesh onto a different target mesh [ 34 ] . 3D pose transfer aims to generate a new mesh based on the knowledge of a pair of source and target meshes . In [ 3 , 34 , 41 , 42 ] , the methods all require to label the corresponding landmarks first to deal with the differences between meshes . Baran et al . [ 2 ] proposed a method that infers a semantic correspondence between different poses of two characters with the guidance of example mesh pairs . Chu et al . [ 8 ] proposed to use a few examples to generate results which will make it difficult to automatically transfer pose . For this problem , Gao et al . [ 14 ] proposed to use cycle consistency to achieve the pose transfer . However , their method can not deal with new identities due to the limitations of the visual similarity metric . In [ 37 ] , they solved the pose transfer via the latest technique for image style transfer . Their work does not need other guidance , but the performance is also restrained since they do not learn any correspondence . To solve the problems , our network learns the correspondence and refines the generated meshes jointly . 2.3 Correspondence learning . In CoCosNet [ 44 ] , they introduced a correspondence network based on the correlation matrix between images without any constraints . To learn a better matching , we proposed to use optimal transport to learn the correspondence between meshes . Recently , optimal transport has received great attention in various computer vision tasks . Courty et al . [ 9 ] perform the alignment of the representations in the source and target domains by learning a transportation plan . Su et al . [ 33 ] compute the optimal transport map to deal with the surface registration and shape space problem . Other applications include generative model [ 1 , 6 , 11 , 39 ] , scene flow [ 28 ] , semantic correspondence [ 22 ] and etc . 2.4 Conditional normalization layers . After normalizing the activation value , conditional normalization uses the modulation parameters calculated from the external data to denormalize it . Adaptive Instance Normalization ( AdaIN ) [ 18 ] aligns the mean and variance between content and style image which achieves arbitrary style transfer . Soft-AdaIN [ 7 ] introduces a channel-wise weight to blend feature statistics of content and style image to preserve more details for the results . Spatially-Adaptive Normalization ( SPADE ) [ 27 ] can better preserve semantic information by not washing away it when applied to segmentation masks . SPAdaIN [ 37 ] changes batch normalization [ 19 ] in SPADE to instance normalization [ 36 ] for 3D pose transfer . However , it will break the consistency and continuity of the feature map when doing the denormalization , which has a bad influence on the mesh smoothness and detail preservation . To address this problem , our ElaIN introduces an adaptive weight to implement the denormalization . 3 Method . Given a source pose mesh and a target identity mesh , our goal is to transfer the pose of source mesh to the target mesh and keep the identity of the target mesh . In this section , we will introduce our end-to-end Correspondence-Refinement Network ( 3D-CoreNet ) for 3D pose transfer . A 3D mesh can be represented as M ( I , P , O ) , where I denotes the mesh identity , P represents the pose of the mesh , O is the vertex order of the mesh . Given two meshes Mid =M ( I1 , P1 , O1 ) and Mpose =M ( I2 , P2 , O2 ) , we aim to transfer the pose of Mpose to Mid and generate the output mesh Moutput = M ′ ( I1 , P2 , O1 ) . In our 3D-CoreNet , we take Vid ∈ RNid×3 and Vpose ∈ RNpose×3 as inputs , which are the ( x , y , z ) coordinates of the mesh vertices . Nid and Npose denote the number of vertices of identity mesh and pose mesh respectively . As shown in Figure 2 , the vertices of the meshes are first fed into the network to extract multi-scale deep features . We calculate the optimal matching matrix with the vertex feature maps by solving an optimal transport problem , then we warp the pose mesh according to the matrix and obtain the warped mesh . Finally , the warped mesh is refined to the final output mesh with our proposed elastic instance normalization ( ElaIN ) . The output mesh combines the pose from the source mesh and the identity from the target . And it inherits the vertex order from the identity mesh . 3.1 Correspondence learning . Given an identity mesh and a pose mesh , our correspondence learning module calculates an optimal matching matrix , each element of the matching matrix represents the similarities between two vertices in the two meshes . The first step in our shape correspondence learning is to compute a correlation matrix with the extracted features , which is based on cosine similarity and denotes the matching similarities between any two positions from different meshes . However , the matching scores in the correlation matrix are calculated without any additional constraints . To learn a better matching , we solve this problem from a global perspective by modeling it as an optimal transport problem . Correlation matrix We first introduce our feature extractor which aims to extract features for the unordered input vertices . Close to [ 37 ] , our feature extractor consists of 3 stacked 1× 1 convolution and Instance Normalization layers , the activation functions applied are all LeakyReLU . Given the extracted vertex feature maps fid ∈ RD×Nid , fpose ∈ RD×Npose of identity and pose meshes ( D is the channel-wise dimension ) , a popular method to compute correlation matrix is using the cosine similarity [ 43 , 44 ] . Concretely , we compute the correlation matrix C ∈ RNid×Npose as : C ( i , j ) = fid ( i ) > fpose ( j ) ‖fid ( i ) ‖ ‖fpose ( j ) ‖ ( 1 ) where C ( i , j ) denotes the individual matching score between fid ( i ) and fpose ( j ) ∈ RD , fid ( i ) and fpose ( j ) represent the channel-wise feature of fid at position i and fpose at j. Optimal transport problem To learn a better matching with additional constraints in this work , we model our shape correspondence learning as an optimal transport problem . We first define a matching matrix T ∈ RNid×Npose+ between identity and pose meshes . Then we can get the total correlation as ∑ ij C ( i , j ) T ( i , j ) . The aim will be maximizing the total correlation score to get an optimal matching matrix Tm . We treat the correspondence learning between identity and pose meshes as the transport of mass . A mass which is equal to N−1id will be assigned to each vertex in the identity mesh , and each vertex in pose mesh will receive the mass N−1pose from identity mesh through the built correspondence between vertices . Then if we define Z = 1−C as the cost matrix , our goal can be formulated as a standard optimal transport problem by minimizing the total matching cost , Tm = argmin T∈RNid×Npose+ ∑ ij Z ( i , j ) T ( i , j ) s.t . T1Npose = 1NidN −1 id , T > 1Nid = 1NposeN −1 pose . ( 2 ) where 1Nid ∈ RNid and 1Npose ∈ RNpose are vectors whose elements are all 1 . The first constraint in Eq . 2 means that the mass of each vertex in Mid will be entirely transported to some of the vertices in Mpose . And each vertex in Mpose will receive a mass N−1pose from some of the vertices in Mid with the second constraint . This problem can be solved by the Sinkhorn-Knopp algorithm [ 32 ] . The details of the solved process will be given in the supplementary material . With the matching matrix , we can warp the pose mesh and obtain the vertex coordinates Vwarp ∈ RNid×3 of the warped mesh , Vwarp ( i ) = ∑ j Tm ( i , j ) Vpose ( j ) ( 3 ) the warped mesh Mwarp inherits the number and order of vertex from identity mesh and can be reconstructed with the face information of identity mesh as shown in Figure 2 .

Summary: This paper proposes a 3D-CoreNet  to solve the 3D pose transfer problem. The 3D-CoreNet consists of two steps: a correspondence learning step and refinement step. Firstly, the authors treat  the shape correspondences learning problem as  an optimal transport problem, which can be solved without using any correspondence annotations. A coarse warped mesh can be obtained from the learned correspondence. Then, an Elastic INstance Normalization (ElaIN) is introduced to the refinement module in order to blend statistics of original features and the learned parameters from external data. State-of-the-art performance has been achieved with the proposed 3D-CoreNet.