SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Motion transfer to seen target
Figure 4: Qualitative results of 3D motion transfer by LART. The left part shows the motion transfer to seen target in training, and the right part shows the motion transfer results to the target mesh, which is unseen in the training set.
Driving motion
Target mesh
Transferred results
Motion transfer to unseen target
Table 1: Comparison with the state-of-the-art methods on the DFAUST datasets. We compute frame-level average point-wise mesh Euclidean distance (PMD) as the metric. Our LART outperforms other compared methods by a large margin with more efficient learning on the training sets.
Method	PMD↓ (×10-4)
Seen Motions	Unseen Motions
NPT-MP [35]	4.51	6.69
NPT [35]	4.37	5.31
3D-CoreNet [31]	1.25	3.05
LART (Ours)	0.41	1.80
Driving Motion Datasets. For training the model, we use the motion sequences from the DFAUST [6] and AMASS [21] datasets. DFAUST is a 3D human motion dataset with 40,000 raw and aligned meshes, captured at 60 fps. There are ten human subjects with 129 different body motions, such as "punching", "chicken wings pose", etc. Each motion consists of hundreds of frames. AMASS is a large and varied database of human motion that contains 15 different sub-datasets (including SMPL-registered DFAUST), spanning over 300 subjects with more than 1,1000 motions. Here we use the DFAUST dataset as a quantitative evaluation and select some representative motions from AMASS for training and presenting the qualitative results. To create training data, we randomly pair one motion (driving sequence) and one shape (target mesh) for training. The corresponding ground truth is obtained by using the SMPL model [4] to synthesize the target animated sequence with the shape and pose parameters for evaluation.
Target Datasets. In testing, we use the learned LART model to transfer motions from given driving sequences to unseen target meshes. To evaluate the generalization ability of our LART, we employ the learned model to drive the target meshes from other unseen datasets, e.g., FAUST [5] and MG-dataset [3], for qualitative evaluation. For quantitative evaluation, we use eight new target meshes generated with the SMPL model by sampling from pose and shape parameter spaces. Then we conduct the 3D motion transfer on these new target meshes. We evaluate the model with two protocols on the DFAUST, i.e., the seen driving sequences that appear in the training set and the unseen driving sequences that are not in the training set.
Other Domains. Lastly, we also use LART to achieve motion transfer on different domains other than human meshes, such as animal meshes from the Animal dataset [32] and hand meshes from the MANO dataset [27] via domain-specific training. The mesh vertices are all shuffled randomly to ensure the network is vertex-order invariant. The training is achieved in a supervised manner, while in the testing, we directly conduct the motion transfer to unseen meshes with the trained model. See more details on how we process the datasets in the Supplementary Materials.
Figure 5: Generalization ability of LART. On the top, we show the domain generalization ability of LART by conducting 3D motion transfer on 3D hands from MANO dataset [27], and 3D animals from Animal dataset [32]. On the bottom, we show the generalizing ability of LART to unseen mesh targets from the FAUST dataset and MG-cloth dataset. Note that meshes in MG-cloth have more than 27,000 vertices with fine-grained geometric details, showing that LART works without the need to comply with a typical SMPL template which has 6,890 vertices.
Evaluation Metrics. We chose the commonly used point-wise mesh Euclidean distance (PMD) as the evaluation metrics [35, 9]. It is the L2 distance between the vertices of the output mesh and the ground truth mesh. In the 3D motion transfer task, we obtain the average PMD over all the frames:
$$
P M D = \frac {1}{T} \frac {1}{N} \sum_ {\mathbf {t}} \sum_ {\mathbf {n}} \left\| M _ {\mathbf {n}} ^ {t} - G _ {\mathbf {n}} ^ {t} \right\| _ {2} ^ {2}, \tag {5}
$$
where  $M_{\mathbf{n}}$  and  $G_{\mathbf{n}}$  are the point pairs from the generated mesh  $M$  and ground truth mesh  $G$ .
Baselines. We compare the proposed approach to the most recent single mesh deformation approaches. Specifically, we compare three state-of-the-art methods, i.e., NPT [35], 3D-CoreNet [31]. For these two methods, we use them to generate long sequences frame by frame as they are originally for 3D pose transfer without considering the temporal domain. Note that we use the same protocols provided to train the models to make sure they are optimized to the greatest extent.
Training and Runtime. Our network receives a sequence of driving meshes of size  $N_{1} \times 3 \times T$ , and a target mesh of size  $N_{2} \times 3$  where  $N$  and  $T$  stand for the vertex number and frame number. Note that  $N_{1}$  might not equal  $N_{2}$  because our LART can handle meshes of different sizes. The network is implemented in PyTorch [22] and optimized using the Adam. The batch size is 2. The total number of training epochs is 200. The learning rate is initialized as 5e-5 and reduced at milestones of 80, 120, and 160 with gamma as 0.1. During the training, we vary the  $\alpha$  to better learn the underlying linear path in the latent space. The length of the driving sequences is fixed as three frames for the computational memory issue. In the first phase (100 epochs), LART takes as input all the continuous frames from a motion clip to stabilize the training. Then, in the later phase, we add latent regularization which allows us to skip frames between the start and end of the motion by encoding them linearly. The total training time on the DFAUST dataset is around 31 hours on an Nvidia GPU V100, shorter than other methods [35, 31, 7]. During inference, we fix  $\alpha$  as 0.5 for both 3D motion transfer and temporal interpolation tasks. We conduct the long sequential motion transfer to the target mesh via a sliding window. Inference runs at  $\sim 120\mathrm{ms}$  per frame on DFAUST (6,890 vertices).
State-of-the-Art Comparisons. We provide experimental results by comparing LART with state-of-the-art methods quantitatively in Table 1. For both settings: "Seen motions" and "Unseen motions"
Figure 6: Qualitatively comparison with the state-of-the-art methods on motions from the AMASS dataset. Given the long driving sequence, compared methods process the deformation frame by frame while LART takes several frames as inputs.
Table 2: Ablation study to latent metric regularization (LMR) by verifying the performances of a single subject from the DFAUST dataset.
Method	NPT	LART single frame	LART w/o LMR	LART (Ours)
Training Time	21 h	25 h	20 h	8 h
PMD↓ (×10-4)	2.82	2.77	1.90	0.32
from the DFAUST dataset, our LART significantly outperforms compared state-of-the-art methods with frame-level average PMD  $(\times 10^{-4})$  of: 0.41 and 1.80 (LART) vs. 1.25 and 3.05 (3D-CoreNet). As shown in the Figure 6, given unseen motion sequences, both NPT and 3D-CoreNet suffer from degeneracy, while LART still can provide relatively robust results.
Ablation Study. We provide an ablation study in Table 2. As shown, we can further verify the effectiveness of each component in the LART, with LMR not only improving the model performance but also shortening the training time.
Non-SMPL Mesh Motion Transfer. LART can be generalized to the animation of non-SMPL meshes. As shown in Figure 5, LART can directly conduct motion transfer to unseen meshes from FAUST [5] and MG-cloth [3] datasets without any further finetuning. Note that those meshes are not in line with the SMPL model and are more challenging (more fine-grained geometry details). Furthermore, we show the results of our model on animating animals and hands in Figure 5, by training on the Animal and MANO datasets, showing the capacity of our LART in different domains.
Potential Applications. Furthermore, we show two potential applications of LART in Figure 7. For denoising, we add Gaussian noise to each vertex of the input mesh. Results show that LART can not only handle random ordered points as input but is also insensitive to noise to some extent (even though some artifacts can be observed on the arm part of the human mesh). For the temporal interpolation, LART takes the start and end frames as input; it can complete the middle frames.
Limitations. One main issue of LART is the computational complexity. To capture the detailed geometries of an unseen target, we use structured latent codes (geometric adaptive embedding) along with the target mesh to have a better performance. However, it is memory-consuming. Besides, since the setting of LART is to directly transfer motion to unseen targets and generate the animation, LART cannot capture and generate the subject/pose-dependent dynamics such as the plausible wrinkles of the cloth and subtle muscle moments. This kind of non-linear local deformation that can efficiently improve the generative visual effects has not been explored in this work yet, which could be the future
Noisy input handling
Temporal interpolation
Figure 7: Potential examples of our LART application. The left side shows a denoising use case of LART. Given a noisy driving sequence, LART can still handle and complete the information and output relatively robust results. The right side shows using LART to interpolate an incomplete sequence.
research direction. Another limitation is that LART (and all other baselines) relies on the latent pose space learned to generate high-quality results. If a motion is far away from the learned distribution, it will lead to degenerated results. Although efforts have been made here to boost latent space learning, the problem of efficiently covering the whole latent space is far from being closed.
In this work, we propose the LART framework, i.e., a novel Transformer network customized for 3D motion transfer with carefully-designed architectures. LART can implicitly learn the correspondence and can also handle large-size full-detailed unseen 3D targets. Besides, we introduce a novel latent metric regularization for better motion generation. Our proposed LART shows a high learning efficiency with promising visual effects. The experimental results verify the potential of our generative model in applications of motion transfer, content generation, temporal interpolation, and motion denoising. The future direction could be compact architecture design and efficient latent space learning as strong priors to unseen domains.
6 Acknowledgement.
This work was supported by the Research Council of Finland (former Academy of Finland) Academy Professor project EmotionAI (grants 336116, 345122), ICT 2023 project TrustFace (grant 345948), the University of Oulu & Research Council of Finland Profi 7 (grant 352788), and Infotech Oulu. As well, the authors wish to acknowledge CSC - IT Center for Science, Finland, for computational resources. Support by the Alexander von Humboldt Foundation and by the Bulgaria government is also gratefully acknowledged.
[1] Noam Aigerman, Kunal Gupta, Vladimir G Kim, Siddhartha Chaudhuri, Jun Saito, and Thibault Groueix. Neural jacobian fields: Learning intrinsic mappings of arbitrary meshes. SIGGRAPH, 2022.
[2] Mirela Ben-Chen, Ofir Weber, and Craig Gotsman. Spatial deformation transfer. In Proceedings of the 2009 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, 2009.
[3] Bharat Lal Bhatnagar, Garvita Tiwari, Christian Theobalt, and Gerard Pons-Moll. Multi-garment net: Learning to dress 3d people from images. In ICCV, 2019.
[4] Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black. Keep it smpl: Automatic estimation of 3d human pose and shape from a single image. In ECCV, 2016.
[5] Federica Bogo, Javier Romero, Matthew Loper, and Michael J Black. Faust: Dataset and evaluation for 3d mesh registration. In CVPR, 2014.
[26] Emanuele Rodolà, Luca Cosmo, Michael M Bronstein, Andrea Torsello, and Daniel Cremers. Partial functional correspondence. In Computer graphics forum, volume 36, pages 222-236. Wiley Online Library, 2017.
[27] Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: Modeling and capturing hands and bodies together. TOG, 36(6), 2017.
[28] Shunsuke Saito, Jinlong Yang, Qianli Ma, and Michael J Black. Scanimate: Weakly supervised learning of skinned clothed avatar networks. In CVPR, 2021.
[29] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. In NeurIPS, 2019.
[30] Richard Sinkhorn. Diagonal equivalence to matrices with prescribed row and column sums. The American Mathematical Monthly, 74(4):402-405, 1967.
[31] Chaoyue Song, Jiacheng Wei, Ruibo Li, Fayao Liu, and Guosheng Lin. 3d pose transfer with correspondence learning and mesh refinement. NeurIPS, 2021.
[32] Robert W Sumner and Jovan Popovic. Deformation transfer for triangle meshes. TOG, 23(3), 2004.
[33] Qingyang Tan, Lin Gao, Yu-Kun Lai, and Shihong Xia. Variational autoencoders for deforming 3d mesh models. In CVPR, 2018.
[34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, undefinedukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
[35] Jiashun Wang, Chao Wen, Yanwei Fu, Haitao Lin, Tianyun Zou, Xiangyang Xue, and Yinda Zhang. Neural pose transfer by spatially adaptive instance normalization. In CVPR, 2020.
[36] Shaofei Wang, Marko Mihajlovic, Qianli Ma, Andreas Geiger, and Siyu Tang. Metaavatar: Learning animatable clothed human models from few depth images. In NeurIPS, 2021.
[37] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-video synthesis. In NeurIPS, 2018.
[38] Guoying Zhao, Yante Li, and Qianru Xu. From emotion ai to cognitive ai. International Journal of Network Dynamics and Intelligence, 2022.
[39] Ce Zheng, Sijie Zhu, Matias Mendieta, Taojiannan Yang, Chen Chen, and Zhengming Ding. 3d human pose estimation with spatial and temporal transformers. ICCV, 2021.
[40] Keyang Zhou, Bharat Lal Bhatnagar, and Gerard Pons-Moll. Unsupervised shape and pose disentanglement for 3d meshes. In ECCV, 2020.