3D Pose Transfer with Correspondence Learning and Mesh Refinement

1 Introduction . 3D pose transfer has been drawing a lot of attention from the vision and graphics community . It has potential applications in 3D animated movies and games by generating new poses for existing shapes and animation sequences . 3D pose transfer is a learning-driven generation task which is similar to style transfer on 2D images . As shown in Figure 1 , pose transfer takes two inputs , one is identity mesh that provides mesh identity ( e.g. , body shape ) , the other is pose mesh that provides the pose of mesh . It aims at transferring the pose of a source pose mesh to a target identity mesh and keeping the identity of the target identity mesh . A fundamental problem for previous methods is to build reliable correspondence between source and target meshes . It can be very challenging when the source and target meshes have significant differences . Most of the previous methods try to solve it with the help of user effort or other additional inputs , such as key point annotations [ 3 , 34 , 41 ] , etc . Unfortunately , it is time-consuming to obtain such additional inputs that will limit the usage in practice . In [ 37 ] , they proposed to implement pose transfer without correspondence learning . Their method is convenient but the performance will be degraded since they do not consider the correspondence between meshes . In this work , we propose a COrrespondence-REfinement Network ( 3D-CoreNet ) to solve the pose transfer problem for both the human and animal meshes . Like [ 37 ] , our method does not need key point annotations or other ∗Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . additional inputs . We learn the shape correspondence between identity and pose meshes first , then we warp the pose mesh to a coarse warped output according to the correspondence . Finally , the warped mesh will be refined to have a better visual performance . Our method does not require the two meshes to have the same number or order of vertices . For the correspondence learning module , we treat the shape correspondence learning as an optimal transport problem to learn the correspondence between meshes . Our network takes vertex coordinates of identity and pose meshes as inputs . We extract deep features at each vertex using point cloud convolutions and compute a matching cost between the vertex sets with the extracted features . Our goal is to minimize the matching cost to get an optimal matching matrix . With the optimal matching matrix , we warp the pose mesh and obtain a coarse warped mesh . We then refine the warped output with a set of elastic instance normalization residual blocks , the modulation parameters in the normalization layers are learned with our proposed Elastic Instance Normalization ( ElaIN ) . In order to generate smoother meshes with more details , we introduce a channel-wise weight in ElaIN to adaptively blend feature statistics of original features and the learned parameters from external data , which help to keep the consistency and continuity of the original features . Our contributions can be summarized as follows : • We solve the 3D pose transfer problem with our proposed correspondence-refinement network . To the best of our knowledge , our method is the first to learn the correspondence between different meshes and refine the generated meshes jointly in the 3D pose transfer task . •We learn the shape correspondence by solving an optimal transport problem without any key point annotations and generate high-quality final meshes with our proposed elastic instance normalization in the refinement module . • Through extensive experiments , we demonstrate that our method outperforms state-of-the-art methods quantitatively and qualitatively on both human and animal meshes . 2 Related work . 2.1 Deep learning methods on 3D data . The representations of 3D data are various , like point clouds , voxels and meshes . 3DShapeNets [ 40 ] and VoxNet [ 25 ] propose to learn on volumetric grids . Their methods can not be applied on complex data due to the sparsity of data and computation cost of 3D convolution . PointNet [ 29 ] uses a shared MLP on every point followed by a global max-pooling . Following PointNet , some hierarchical architectures have been proposed to aggregate local neighborhood information with MLPs [ 20 , 30 ] . [ 13 , 35 ] proposed mesh variational autoencoder to learn mesh features whose methods consume a large amount of computing resources due to their fully-connected networks . Many works use graph convolutions with mesh down- and up-sampling layers [ 15 ] , like CoMA [ 31 ] , CAPE [ 24 ] based on ChebyNet [ 10 ] , and [ 45 ] based on SpiralNet [ 21 ] , SpiralNet++ [ 16 ] , etc . They all need a template to implement their hierarchical structure which is not applicable to real-world problems . In this work , we use mesh as the representation of 3D shape and shared weights convolution layers in the network . 2.2 3D pose transfer . Deformation transfer in graphics aims to apply the deformation exhibited by a source mesh onto a different target mesh [ 34 ] . 3D pose transfer aims to generate a new mesh based on the knowledge of a pair of source and target meshes . In [ 3 , 34 , 41 , 42 ] , the methods all require to label the corresponding landmarks first to deal with the differences between meshes . Baran et al . [ 2 ] proposed a method that infers a semantic correspondence between different poses of two characters with the guidance of example mesh pairs . Chu et al . [ 8 ] proposed to use a few examples to generate results which will make it difficult to automatically transfer pose . For this problem , Gao et al . [ 14 ] proposed to use cycle consistency to achieve the pose transfer . However , their method can not deal with new identities due to the limitations of the visual similarity metric . In [ 37 ] , they solved the pose transfer via the latest technique for image style transfer . Their work does not need other guidance , but the performance is also restrained since they do not learn any correspondence . To solve the problems , our network learns the correspondence and refines the generated meshes jointly . 2.3 Correspondence learning . In CoCosNet [ 44 ] , they introduced a correspondence network based on the correlation matrix between images without any constraints . To learn a better matching , we proposed to use optimal transport to learn the correspondence between meshes . Recently , optimal transport has received great attention in various computer vision tasks . Courty et al . [ 9 ] perform the alignment of the representations in the source and target domains by learning a transportation plan . Su et al . [ 33 ] compute the optimal transport map to deal with the surface registration and shape space problem . Other applications include generative model [ 1 , 6 , 11 , 39 ] , scene flow [ 28 ] , semantic correspondence [ 22 ] and etc . 2.4 Conditional normalization layers . After normalizing the activation value , conditional normalization uses the modulation parameters calculated from the external data to denormalize it . Adaptive Instance Normalization ( AdaIN ) [ 18 ] aligns the mean and variance between content and style image which achieves arbitrary style transfer . Soft-AdaIN [ 7 ] introduces a channel-wise weight to blend feature statistics of content and style image to preserve more details for the results . Spatially-Adaptive Normalization ( SPADE ) [ 27 ] can better preserve semantic information by not washing away it when applied to segmentation masks . SPAdaIN [ 37 ] changes batch normalization [ 19 ] in SPADE to instance normalization [ 36 ] for 3D pose transfer . However , it will break the consistency and continuity of the feature map when doing the denormalization , which has a bad influence on the mesh smoothness and detail preservation . To address this problem , our ElaIN introduces an adaptive weight to implement the denormalization . 3 Method . Given a source pose mesh and a target identity mesh , our goal is to transfer the pose of source mesh to the target mesh and keep the identity of the target mesh . In this section , we will introduce our end-to-end Correspondence-Refinement Network ( 3D-CoreNet ) for 3D pose transfer . A 3D mesh can be represented as M ( I , P , O ) , where I denotes the mesh identity , P represents the pose of the mesh , O is the vertex order of the mesh . Given two meshes Mid =M ( I1 , P1 , O1 ) and Mpose =M ( I2 , P2 , O2 ) , we aim to transfer the pose of Mpose to Mid and generate the output mesh Moutput = M ′ ( I1 , P2 , O1 ) . In our 3D-CoreNet , we take Vid ∈ RNid×3 and Vpose ∈ RNpose×3 as inputs , which are the ( x , y , z ) coordinates of the mesh vertices . Nid and Npose denote the number of vertices of identity mesh and pose mesh respectively . As shown in Figure 2 , the vertices of the meshes are first fed into the network to extract multi-scale deep features . We calculate the optimal matching matrix with the vertex feature maps by solving an optimal transport problem , then we warp the pose mesh according to the matrix and obtain the warped mesh . Finally , the warped mesh is refined to the final output mesh with our proposed elastic instance normalization ( ElaIN ) . The output mesh combines the pose from the source mesh and the identity from the target . And it inherits the vertex order from the identity mesh . 3.1 Correspondence learning . Given an identity mesh and a pose mesh , our correspondence learning module calculates an optimal matching matrix , each element of the matching matrix represents the similarities between two vertices in the two meshes . The first step in our shape correspondence learning is to compute a correlation matrix with the extracted features , which is based on cosine similarity and denotes the matching similarities between any two positions from different meshes . However , the matching scores in the correlation matrix are calculated without any additional constraints . To learn a better matching , we solve this problem from a global perspective by modeling it as an optimal transport problem . Correlation matrix We first introduce our feature extractor which aims to extract features for the unordered input vertices . Close to [ 37 ] , our feature extractor consists of 3 stacked 1× 1 convolution and Instance Normalization layers , the activation functions applied are all LeakyReLU . Given the extracted vertex feature maps fid ∈ RD×Nid , fpose ∈ RD×Npose of identity and pose meshes ( D is the channel-wise dimension ) , a popular method to compute correlation matrix is using the cosine similarity [ 43 , 44 ] . Concretely , we compute the correlation matrix C ∈ RNid×Npose as : C ( i , j ) = fid ( i ) > fpose ( j ) ‖fid ( i ) ‖ ‖fpose ( j ) ‖ ( 1 ) where C ( i , j ) denotes the individual matching score between fid ( i ) and fpose ( j ) ∈ RD , fid ( i ) and fpose ( j ) represent the channel-wise feature of fid at position i and fpose at j. Optimal transport problem To learn a better matching with additional constraints in this work , we model our shape correspondence learning as an optimal transport problem . We first define a matching matrix T ∈ RNid×Npose+ between identity and pose meshes . Then we can get the total correlation as ∑ ij C ( i , j ) T ( i , j ) . The aim will be maximizing the total correlation score to get an optimal matching matrix Tm . We treat the correspondence learning between identity and pose meshes as the transport of mass . A mass which is equal to N−1id will be assigned to each vertex in the identity mesh , and each vertex in pose mesh will receive the mass N−1pose from identity mesh through the built correspondence between vertices . Then if we define Z = 1−C as the cost matrix , our goal can be formulated as a standard optimal transport problem by minimizing the total matching cost , Tm = argmin T∈RNid×Npose+ ∑ ij Z ( i , j ) T ( i , j ) s.t . T1Npose = 1NidN −1 id , T > 1Nid = 1NposeN −1 pose . ( 2 ) where 1Nid ∈ RNid and 1Npose ∈ RNpose are vectors whose elements are all 1 . The first constraint in Eq . 2 means that the mass of each vertex in Mid will be entirely transported to some of the vertices in Mpose . And each vertex in Mpose will receive a mass N−1pose from some of the vertices in Mid with the second constraint . This problem can be solved by the Sinkhorn-Knopp algorithm [ 32 ] . The details of the solved process will be given in the supplementary material . With the matching matrix , we can warp the pose mesh and obtain the vertex coordinates Vwarp ∈ RNid×3 of the warped mesh , Vwarp ( i ) = ∑ j Tm ( i , j ) Vpose ( j ) ( 3 ) the warped mesh Mwarp inherits the number and order of vertex from identity mesh and can be reconstructed with the face information of identity mesh as shown in Figure 2 .

Summary: This paper tackles the problem of 3D pose transfer: given an identity mesh Mi and a posed mesh Mp, the key problem is to output a mesh M' with the same id with Mi and the same pose with Mp. The proposed method is in a coarse-to-fine manner. It firstly ultizes optimal transport to learn a coarse correspondence, with which a coarse mesh can be generated. After that, a novel "elastic instance normalization" operation is proposed for mesh refinement.