LART: Neural Correspondence Learning with Latent Regularization Transformer for 3D Motion Transfer
Authors: Haoyu Chen
Hao Tang
Radu Timofte
Luc Van Gool
Guoying Zhao
Venue: NeurIPS 2023 poster (2023)
Keywords: 3D motion transfer
3D Transformer
geometric preservation
3D generation
correspondence learning
TL;DR: LART: An efficient framework for 3D motion transfer with rigorous metric presenvation

Abstract: 3D motion transfer aims at transferring the motion from a dynamic input sequence to a static 3D object and outputs an identical motion of the target with high-fidelity and realistic visual effects. In this work, we propose a novel 3D Transformer framework called LART for 3D motion transfer. With carefully-designed architectures, LART is able to implicitly learn the correspondence via a flexible geometry perception. Thus, unlike other existing methods, LART does not require any key point annotations or pre-defined correspondence between the motion source and target meshes and can also handle large-size full-detailed unseen 3D targets. Besides, we introduce a novel latent metric regularization on the Transformer for better motion generation. Our rationale lies in the observation that the decoded motions can be approximately expressed as linearly geometric distortion at the frame level. The metric preservation of motions could be translated to the formation of linear paths in the underlying latent space as a rigorous constraint to control the synthetic motions occurring in the construction of the latent space. The proposed LART shows a high learning efficiency with the need for a few samples from the AMASS dataset to generate motions with plausible visual effects. The experimental results verify the potential of our generative model in applications of motion transfer, content generation, temporal interpolation, and motion denoising. The code is made available: https://github.com/mikecheninoulu/LART.

LART: Neural Correspondence Learning with Latent Regularization Transformer for 3D Motion Transfer
Haoyu Chen'
Hao Tang2 Radu Timofte Luc Van Gool
Guoying Zhao
CMVS University of Oulu *Computer Vision Laboratory, ETH Zurich
Carnegie Mellon University UUniversity of Wurzburg SESAT-PSI, KU Leuven
INSAIT, Sofia Un. St. Kliment Ohridski
chen. haoyu@oulu.fi, hao. tangavision. ee. .e..ee.eeett ethz. ch, radu. siimotteunni-wueee vangool@vision ee. ethz.ch. ch, guoying zhao@oulu.
Abstract
3D motion transfer aims at transferring the motion from dynamic input sequence static 3D object and outputs identical motion of the target with high-fidelity and realistic visual effects. In this work, we propose a novel 3D Transformer frame- work called LART for 3D motion transfer. With carefully-desiineed architectures, LART is able to implicitly learn the correspondence via a flexible geometry per- ception. Thus, unlike other existing methods, LART does not require any key point annotations or pre-defined correspondence between the motion source and target meshes and can also handle large-size full-detailed unseen 3D targets. Besides, we introduce a novel latent metric regularization on the Transformer for better motion generation. Our rationale lies the observation that the decoded motions can be approximately expressed as linearly geometric distortion at the frame level. The metric preservation of motions could be translated to the formation of linear paths in the underlying latent space as rigorous constraint to control the synthetic motions occurring in the construction of the latent space. The proposed LART shows high learning efficiency with the need for few samples from the AMASS dataset to generate motions with plausible visual effects. The experimental results verify the potential of generative model in applications of motion transfer, content generation, temporal interpolation, and motion denoising. The code is made available: https: //gtt/hb//ttbbb com/ http///ikccennnoou//iinne
Driving motion
Unseen target
Generated results
Figure I: Given as driving mesh sequences with random-ordered vertices, our learned LART could transfer the motion from the driving sequences the unseen 3D targets without any further fine- tuning. Here we connect the points based on the vertex order to verify that our LART does not need the dre-defined correspondence, which is vedeex-order invariant. Samples are from the AMASS dataset [21].
"Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
3D motion transfer is one of the most challenging 3D generation tasks. It refers to the task of transferring desired motions from a driving sequence to a static 3D subject and outputting the corresponding high-fidelity and realistic synthetic dynamics. Aside from the research interests, has countless applications across areas of interest, i.e., 3D animated character design in the film industry. VR/AR enhancement, emotion transfer, or real-time remote interaction [38]. It is desirable that one model can automatically generate realistic animated 3D avatars without manually specifying the geometry correspondences and dynamics, which would be cumbersome when using a standard graphics rendering engine.
Although existing works can achieve 3D motion transfer via various fashions with plausible and promising visual results, the question of automatic 3D dynamic generation is far from being closed. There are several major limitations among existing methods. First, the correspondence between source and target meshes needs to be provided, such as point-wise correspondence [32], human key point annotations [2], or template registrations [36] (e.g., SMPL. Unfortunately, these all involve costly manual data annotations and are not always available. Second, most of the existing models strongly rely on the 3D shape prior learned from the training sets to keep the intrinsic attributes [13, 40, 28, 8]. and cannot generalize to unseen targets that are not in the training set. Thus, collecting new samples each time is a must to train a model for each new subject or shape style many [23]. This would make the model not easily scalable. At last, the generation of surfaces needs fine geometric details to be preserved while transferring the extrinsic motions through a highly- nonlinear map.
We cast the task of 3D motion transfer continuous deformation problem, where the goal is to train a model such that the target 3D object has identical dynamics to driving sequences while preserving its appearance information (see Figure 1). In particular, we design new 3D Transformer structure with tross-attention modules to continuously learn the implicit correspondence between target meshes and dynamies from the driving sequences. It also has an adaptive feature encoder with efficient positional embedding scheme so that can handle large-size meshes with fine geometric details fully preserved. Another key contribution is novel latent metric regularization on the Transformer. Inspired by recent geometric deep learning methods [11, 14] that inject geometric regularization in the training. we formulate the decoded motions in the temporal domain an approximation of linearly geometric distortion at the spatial domain on isometric pairs (i.e., just a change in pose). By coupling the Fuclidean distances among latent codes (hence, along linear paths in the latent space) to the metric distortion among decoded motions, we obtain a strong regularizing effect in the construction of the latent space and, in turn, significantly reduce the need for large training datasets. Our method can learn to generate high-fidelity, realistic, and temporally coherent 3D human motions, Moreover, we show it is possible animate completely unseen 3D mesh with randomly ordered-vertex motion sequences, which is extremely challenging. Lastly, we extend our method to other domains, such as animals and different tasks.
In summary, our main contributions are:
We work on the 3D motion transfer problem, especially for unseen target meshes. To the best of our knowledge, our method the first Transformer-basee 3D motion transfer framework that learns the correspondence implicitly between meshes.
We propose the LART, a novel Transformer-baat based architecture for 3D motion transfer, carefully designed to generate fully- pellereeeee geometric details. A novel feature encoder with an adaptive positional coding scheme is introduced to handle large-size meshes and reduce the dimensionality redundancy and computation cost. It can also implicitly learn the correspondence between meshes via an across- across-attention attention layer.
A novel latent geometric regularization on the Transformer is presented for synthesizing realistic dynamic results. It formulates the temporal generation problem as continuous deformations by coupling the Fuclidean distances among latent codes to the metric distortion among decoded motions as a tight constraint, allowing models to gain generability efficiently and process driving sequences with flexible lengths.
Quantitative or qualitative experimental results on four different datasets show that the proposed method achieves satisfying performances with substantially fewer training samples. Moreover, we further represent the capability of generalizing our framework to other domains and various generative applications, showing it strong baseline for data-eriven 3D motion transfer.

---

2 Related Work
3D Mesh Deformation is closely related to our task. It aims to generate a new 3D shape with given pair of source and target shapes, which can be seen as a frame-level task of 3D motion transfer. Towards 3D mesh deformation, previous works demand re-enforcing the correspondence between source and target meshes. For example, some disentanglemment-iade based methods [11, 40] use the shape correspondences between different pose meshes from the same body to decompose shape and pose factors. Furthermore, Gao et al. [13] proposed using cycle consistency to achieve the pose transfer. Aigerman et al. [1] introduced the neural Jacobian field with mesh-specific, basic linear-algebra operations to an intrinsic field of matrices to preserve the detailed geometry, but it relies heavily on discrete differential geometry operators, which need rigorous processed watertight meshes. Generally, all the above methods cannot deal with unseen identities due to the strong dependence on the shape priors. Differently, recent methods [35, 9] utilize the latest technique for image style transfer to achieve the neural pose transfer without the need for other guidance. 3D-CoreNet [31] explicitly learns the correspondence and refines the generated meshes jointly by solving the optimal transporting problem. However, it will be computationally costly when processing multiple continuous meshes for the motion transfer. In contrast, we use a structured adaptive encoder learn the correspondence implicitly. Most importantly, our model involves the temporal consistency from the sequential meshes, which is challenging and not touched by those 3D deformation methods.
Generalized 3D Animation. Generating 3D mesh sequences via motion information has been intensively researched from different aspects. One main direction closely related to our task is to use the linear blend skinning (LBS) method [4] and the recent implicit skinning networks to construct animatable models. For instance, SNARF [10] uses differentiable forward skinning for animating non-rigid neural implicit shapes, which can be extended to unseen and complex shapes. SCANimate [28] learns pose-aware parametric clothed human model by introducing a cycle procedure to achieve the learning of LBS. Similarly, MetaAvatar [36] can learn an animatable model with only few depth images. However, animation of those methods shares strong prerequisite that certain registrations, such as bone transformations between the target meshes, are inevitable. The motion is achieved by manipulating SMPL pose parameters or key joints of the learned parametric human body models. Thus inherently cannot transfer motion from raw driving mesh sequence to unseen targets. Differently, our learned model can directly conduct motion transfer to unseen 3D meshes from different datasets without any further fune-tine, which is much more challenging.
3D Transformer with Temporal Modeling. Transformer architecture was first proposed by [34] and raised emerging research interests in vision tasks with promising performances [16, 12]. To retain positional information of sequential dependencies, trainable linear projection layer to embed each input patch to a high dimensional feature is always applied, such as pose estimation [39], motion synthesis [25]. However, we argue that the 3D animation task differs from the above tasks. 3D surfaces, as opposed to the standard 2D pixel grid, have significant geometric and topological variations. Standard 2D convolutional architectures make assumptions about point ordering and parametrization that do not apply, in general, to collections of 3D surfaces. Thus, it's crucial that the network architecture can adaptively perceive the geometry and topology. Furthermore, the generation of the 3D surfaces needs detailed preservation. Some 3D motion synthesis methods, such as ACTOR [25] and Action2Motion [155, learn generators based on VAE [20] that achieve the action-conditioned generation of realistic and diverse 3D human mesh sequences. Again, SMPL joint regressor is introduced to ACTOR to ensure the realistic generated motions, and Action2Motion post-renders the skeleton joints to SMPL models to obtain vistaleei results. Apparently, those methods fail to capture complex and geometric details, such as the wrinkle of the human cloth models. Thus, we propose novel Transformer architecture, the LART, for the 3D motion transfer task. Compared to the other works, we have two core differences: 1) both the appearance details of target meshes and desired motions can be handled in the generated 3D sequences, and 2) our framework directly processes the raw mesh sequences without any manual intervention (e.g., SMPL parameters or skeleton joints) in an end-to-end manner.
3 Data-Driven 3D Motion Transfer
Inspired by previous 2D video synthesis works [37, 29] for animating still 2D images and 3D pose transfer [35], we are interested in automatically transferring the motion of driving sequence
3

---

Latent repulariaaion
LART framewurk
LART decoder
M
MP
M
Convy
Target mesh
LART decoder
Target mesh
Target
SPAdeIN
Canc 3d
SPAchEN
Comvid
Clny
Target
SPARS
mish
LART decoder
Target
mesh
Carosh iazetiittiionnn
mish
Id
Cony Coneld
Conrld
Geocectry adaptive positional embeeking
Feature Encoder
Z
2 posc
Embredded patches
4por
Zi
Feature Encoder
(1-a)' '2
puss
pase
Festure Encodar
Mat
SPAdaIN
Convla ld
Target mesh
Mf pase
MP puve
Target
usesh
Id
Instince Norm
Driving Motion
Figure 2: Architecture overview of LART. Left part: the whole LART framework. We process the sequential driving meshes into features, add geometric adaptive positional embeddings, and feed the resulting sequence of vectors to our LART decoder. The resulting generated meshes will be regulated via latent metries with ground truths. Right part: the detailed architectures of an LART decoder and a SPAdaIN block.
to a static 3D target. We first present a general introduction to the LART framework for neural correspondence learning. Then, we will demonstrate how to impose the latent regularization on the LART for training.
3.1 LART: Latent Regularization based Transformer with Neural Correspondence Learning
In the task of 3D motion transfer, especially for an unseen target setting where the shape prior is not available to build compact representation of the latent space, we argue that the latent representations of 3D models should be sufficient and strong enough to preserve and manipulate the detailed surface geometry the target mesh. Inspired by NPT and NeuralBody [35, 24), the idea of using conditional normalization directly in the spatial sense and preserving the structured topology latent codes for detailed geometry capturing is very intuitive and proven effective. However, this will make the latent presentation of the meshes considerably big and make it difficult to model the temporal information, as the number of learnable parameters conventional recurrent temporal models like RNNs/LSTM [18] grow quadratically with the memory size, making them extremely large and almost non-trainable.
Recently, the Transformer architectures [12] have become popular as they allow parallel computation inputs, which both speed up the training and save the memory, especially suiting the needs of our task. Thus, we chose Transformer-basee components, a geometry adaptive feature encoder, a LART decoder, and a latent metric regularization. An overview of the LART is depicted in Figure 2.
architecture to build up our model, which consists of three main
Geometry Adaptive 3D Feature Encoder. Given a 3D mesh driving sequence of T frames Mooses" .Mooee where Mpose E I and N stands for the vertex number, we use the same 3D feature extractor as previous works [35, 31, 9] to extract a static 3D feature per frame. The feature encoder consists of three blocks of 1D convolution layers followed by Instance Norm, resulting in a sequence of latent embedding vector Z Zpase:"
zt pose. where Z E RRC and C' is the channel dimension (e.g.. 1,024). The network weights of the feature encoder are shared for all frames. At the same time, we obtain the latent code Z,a of target mesh Mir via the same feature encoder. Next, we will embed the features of the driving motion each frame with our geometry adaptive positional embedding scheme below.

---

Embeddings
Features
Embeddings
Features
Embeddings
Features
T
N1
N1
N1
N1
C
N1
N1
N2
Resulting features
2C Resalting features
Resulting features
Conventional positional
Conventional positional
(c) Geometty-adaativve
embedding (concatenate)
embedding (add)
positional embedding
Figure 3: A comparison of different positional embedding methods. Note that our methods can output flexible size of embedded features that it can be extended to unseen targets with different vertex numbers than the training set.
As shown in Figure 3, to retain the positional information of the sequence, there are commonly two main ways ("concatenating" and "adding"), and "adding" one is more popular as significantly saves memory [17, 39]. However, 3D motion transfer needs to maintain various sizes of the target meshes (vertex number as N2) with motion sequential meshes (vertex number as N,, N1 # N2), leading to the need for flexible coding scheme. These embedding ways with fixed shapes will hinder the model from extending to different vertex sizes and are apparently not feasible our case. Thus, proposed simple but efficient positional embedding scheme for maintaining the features of structured topology. To ensure the network's 's generalizability to different sizes of unseen meshes while preserving the temporal information, we first conduct adaptive max pooling to every pose feature pose (to the mesh at t frame of motion) to obtain a global-wise latent pose code, and then add the same size vector
as positional/temporal embedding and expending the feature along the topology dimension (N₂) as the target mesh has. In this way, we obtain new feature vector of the pose that has both positions embedded and the same size the target mesh 's making the learning of the correspondence between two meshes with different sizes possible (N1 and N2). After the positional embedding, the resulting sequential feature vectors 7pose together with a single feature vector Zia is sent to the LART decoder.
LART Decoder. As discussed in the work of 3D-CoreNet [31], it's beneficial learn the correspon- dence between the given target mesh and driving mesh sequences. However, 3D-CoreNet needs to explicitly learn the correspondence between the meshes by solving the optimal transporting problem. This could be redundant as we need to empirically set hyper-parameters, i.e., iterations in the Sinkhorn algorithm [30]. Instead, we improve the attention mechanism of the Transformer implicitly learn the implicit correspondence, thus, without the need for extra intervention. Specifically, as shown in Figure 2 right part, we feed the feature vectors from the target and pose meshes into two different ID convolution layers to generate the representations q. k, as that in the standard self-attention [34]. The query q is from Zia, and the value and key are from Zpose- Then, the attention weights Ai, based on the geometric pairwise similarity between two elements of q and k is given with the following formula:
exp(qik;)
A Σ exp(qık,)
(1)
After this, a matrix multiplication between 2 and the transpose of A conducted to perceive the geometric inconsistency between meshes. Finally, weigh the result with scale parameter 7 and conduct an element- wise sum operation with the original latent embedding Zpase to obtain the refined latent embedding Z.ooee
posse
(Ai Z poses
where y is initialized as 0 and updated gradually during the training with gradients. The obtained Z/ is followed by typical Transformer structures as introduced above Figure 2 with convolutional layer and SPAdaIN block [19, 35]. Though multiple stacked LART decoders with latent code size gradually decrease, we generate a sequence of final output Mt... outt
In such a crossing way, the geometric- perceived feature code can consistently be rectified by the original identity mesh and its latent embedding representations. Note that input mesh vertices
5

---

all shuffled randomly, while our network can output vertex-aligned meshes, showing that LART is vertex-order invariant and can successfully learn the correspondence between meshes, see Figure 1. Please refer to the Supplementary Materials for more implementation details.
3.2 Latent Metric Regularization
Here, the efforts in this section are motivated by the observation that 3D motion transfer is different from 3D pose transfer or 3D deformation. There is a lot of redundant information in the motion. Thus, directly implementing existing data-driven approaches for 3D motion transfer, completely relying on the training dataset, will impose a heavy burden on the learning process. For instance, the geometric distortion within a motion clip of 30 frames might be considered subtle, making it extremely inefficient to construct the latent pose space by traveling all the training motions. On the other hand, the meshes within the same motion are isometric (i.c., just change in pose) and share the same intrinsic information, which naturally could be utilized as a regularization training.
With the above observations, we adopt the idea of spatial deformation regularization from LIMP [11] into a temporal case. Specifically, we regard the presence of a motion as a continuous interpretation problem of the pose at each frame, which can be regularized by metric interpolation of the pose space.
Treating Motion Continuous Interpretation. In the continuous setting, we give the latent metric regularization as follows:
Lateric De a) 7 + aa pose) - ((1o))D. + oD&(M
(3)
where a (0, 1) is uniformly sampled scalar in (0, 1), standing for the index frame of motion. The dec operator is to decode the latent code Z into mesh Mout via the LART decoders. In the equation above, the matrix D, encodes the pairwise Euclidean distances [26].
At training time, given two poses M1 and MT from a motion. We then assume there exists an abstract metric space where each point is a pose; this "pose space" is the latent space that our generative model seeks to learn. By constructing the latent code with parametric sequence of poses connecting Mout to M
regard each Zpose as continuously deformed version between M to M By using the metric loss Eq. (3), we can enforce the latent code and Euclidean distance of the motion to be linearly consistent. As result, as grows from to we are modeling sequence of approximate isometries as a motion.
4pose = (1 - a) 2pose + aaZpose
Manipulating on the Flattening Latent Space. Like previous works [11, 33), our learning model performs a "flattening" operation, imposing the latent space be as Euclidean as possible. The expression of the motion can be linearly manipulated by varying the a. This enables algebraic manipulation of the latent codes for various applications, such motion interpolation and denoising.
3.3 Full Objective Function
With the latent metric regularization loss above, define the full objective function below:
Crult AvecCree + hedge + Ammetri&mmetrie
where Crees Ledge and Lmetric are the three losses used as our full optimization objective, including reconstruction loss, edge loss, and our newly proposed latent metric regularization loss. is the corresponding weight of each loss. In Eq. (4), reconstruction loss Free is the point-wise L2 distance and the edge loss [14] an edge-wieg wise regularization between the GT meshes and predicted meshes. The mathematical definitions can be found in the Supplementary Materials.
4 Experiments
In this section, we perform an extensive evaluation of the proposed LART. Quantitative and qualitative experimental results four different datasets show that the proposed method achieves satisfying performances with substantially fewer training samples. Moreover, we further represent the capability of generalizing framework to other domains, such animals and various generative applications.

---

Driving motion
Driving motion
Target mesh
Transferred results
Target mesh
Transferred results
Motion transfer to unseen target Figure 4: Qualitative results of 3D motion transfer by LART. The left part shows the motion transfer seen target in training, and the right part shows the motion transfer results to the target mesh, which is unseen in the training set.
Motion transfer to target
Table Comparison with the state-of-the of-the-art methods on the DFAUST datasets. We compute frame- level average point-wise wise mesh Euclidean distance (PMD) as the metric. Our LART outperforms other compared methods by large margin with more efficient learning the training sets.
PMD (x10
Method
Seen Motions
Unseen Motions
NPT-MP [35]
4.51
6.69
NPT [35]
4.37
5.31
3D-CoreNet [31]
1.25
3,05
LART (Ours)
0.41
1.80
4.1 Datasets
Driving Motion Datasets. For training the model, we use the motion sequences from the DFAUST [6] and AMASS [21] datasets. DFAUST is 3D human motion dataset with 40,000 raw and aligned meshes, captured at 60 fps. There are ten human subjects with 129 different body motions, such as "punching", "chicken wings pose", etc. Each motion consists hundreds of frames. AMASS is large and varied database of human motion that contains 15 different sub-datasets (including SMPPL-registered DFAUST), spanning over 300 subjects with more than 1, 101000 motions. Here we use the DFAUST dataset quantitative evaluation and select some representative motions from AMASS for training and presenting the qualitative results. To create training data, we randomly pair one motion (driving sequence) and shape (target mesh) for training. The corresponding ground truth obtained by using the SMPL model [4] to synthesize the target animated sequence with the shape and pose parameters for evaluation.
Target Datasets. In testing, we use the learned LART model transfer motions from given driving sequences to unseen target meshes. To evaluate the generalization ability of our LART, we employ the learned model drive the targer meshes from other unseen datasets, e-g., FAUST [5] and MG-dataset [3], for qualitative evaluation. For quantitative evaluation, we use eight new target meshes generated with the SMPL model by sampling from pose and shape parameter spaces. Then we conduct the 3D motion transfer on these new target meshes. We evaluate the model with two protocols on the DFAUST, ie.. the seen driving sequences that appear in the training set and the unseen driving sequences that are not in the training set.
Other Domains. Lastly, also use LART to achieve motion transfer on different domains other than human meshes, such as animal meshes from the Animal dataset [32] and hand meshes from the MANO dataset [27] via domain -specffcc training. The mesh vertices are shuffled randomly to ensure the network is vertex-order invariant. The training is achieved in a supervised manner, while the testing, we directly conduct the motion transfer to unseen meshes with the trained model. See more details how process the datasets in the Supplementary Materials.
7

---

Target mesh Driving motion
Driving motion
Transferred results
Target mesh
Transferred
MANO dataset
Animal dataset
Driving motion
Driving motion
Target mesh
Transferred results
Target mesh
Transferred results
FAUST datasct
MG-cloth dataset
Figure 5: Generalization ability LART. On the top, we show the domain generalization ability of LART by conducting 3D motion transfer on 3D hands from MANO dataset [27]. and 3D animals from Animal dataset [32]. On the bottom, show the generalizing ability of LART to unseen mesh targets from the FAUST dataset and MG-cloth dataset. Note that meshes in MG-cloth have more than 27,000 vertices with fine ane-grained geometric details, showing that LART works without the need to comply with a typical SMPL template which has 6,890 vertices.
4.2 Quantitative Evaluation
Evaluation Metrics. We chose the commonly used point-wise mesh Euclidean distance (PMD) as the evaluation metrics [35, 9]. It is the L2 distance between the vertices of the output mesh and the ground truth mesh. In the 3D motion transfer task, we obtain the average PMD over all the frames:
1 1 ΣΣ M -
PMD
T N
where Mn, and Gn are the point pairs from the generated mesh M and ground truth mesh G.
Baselines. We compare the proposed approach to the most recent single mesh deformation approaches. Specifically, we compare three state-of-the-art methods, i.e., NPT [35], 3D-CoreNet [3 For these two methods, pose transfer without considering the temporal domain. Note that we use the same protocols provided to train the models make sure they optimized the greatest extent.
them to generate long sequences frame by frame they originally for 3D
Training and Runtime, Our network receives sequence of driving meshes of size N₁ x T, and a target mesh of size N2 x 3 where N and T stand for the vertex number and frame number. Note that N₁ might not equal N2 because our LART can handle meshes of different sizes. The network is implemented in Py PyTorch [22] and optimized using the Adam. The batch size is 2. The total number training epochs 200. The learning rate is initialized as 5e-5 and reduced milestones of 80, 120, and 160 with gamma 0.1. During the training, we vary the to better learn the underlying linear path in the latent space. The length of the driving sequences is fixed as three frames for the computational memory issue. In the first phase (100 epochs), LART takes input all the continuous frames from a motion clip to stabilize the training. Then, the later phase, we add latent regularization which allows to skip frames between the start and end of the motion by encoding them linearly. The total training time on the DFAUST dataset is around 31 hours on an Nvidia GPU V100, shorter than other methods [35, 31, 7]. During inference, transfer and temporal interpolation tasks. We conduct the long sequential motion transfer to the target mesh via sliding window. Inference
fix 0.5 for both 3D motion
at ~ 1120ms per frame DFAUST (6,890 vertices).
State-of-the-Art Comparisons. We provide experimental results by comparing LART with state-of- the-art methods quantitatively in Table 1. For both settings: "Seen motions" and "Unseen motions"

---

Driving motion
Target
NPT
LART (ours)
3D CoreNet
Ground truth
Figure 6: Qualitatively comparison with the state-of-the- methods on motions from the AMASS dataset. Given the long driving sequence, compared methods process the deformation frame by frame while LART takes several frames as inputs.
Table 2:
Ablation study to latent metric regularization (LMR) by verifying the performances of
single subject from the DEAUST dataset.
LART single frame
LART
Method
NPT
LART (Ours)
w/o LMR
Training Time
25
h
PMD| (x10 +)
2.82
2.77
1.90
0.32
from the DFAUST dataset, our LART significantly outperforms compared state-of-the-art methods with frame-level average PMD (x10-1) of: 0.41 and 1.80 (LARTT) vs. 1.25 and 3.05 (3D-CoreNet). As shown in the Figure 6, given unseen motion sequences, both NPT and 3D-CoreNet suffer from degeneracy, while LART still can provide relatively robust results.
Ablation Study. We provide an ablation study in Table 2. As shown, we can further verify the effectiveness of each component in the LART, with LMR not only improving the model performance but also shortening the training time.
4.3 Qualitative Evaluation
Non-SMPL Mesh Motion Transfer. LART can generalized to the animation Non-SMPL meshes. As shown in Figure 5, LART can directly conduct motion transfer to unseen meshes from FAUST [5] and MG-cloth [3] datasets without any further finetuning. Note that those meshes are not in line with the SMPL model and more challenging (more fine-grained geometry details). Furthermore, we show the results our model on animating animals and hands in Figure 5, by training the Animal and MANO datasets, showing the capacity of our LART in different domains.
Potential Applications. Furthermore, we show two potential applications of LART Figure For denoising, we add Gaussian noise to each vertex of the input mesh. Results show that LART can not only handle random ordered points as input but is also insensitive to noise to some extent (even though some artifacts can be observed on the arm part of the human mesh). For the temporal interpolation, LART takes the start and end frames as input; it can complete the middle frames.
Limitations. One main issue of LART is the computational complexity. To capture the detailed geometries of an unseen target, we use structured latent codes (geometric adaptive embedding) along with the target mesh to have better performance. However, it is memory-comee coonuoi.. Besides, since setting LART to directly transfer motion to unseen targets and generate the animation. LART dynamics such the plausible wrinkles of
cannot capture and generate the subject/pose-dependeee the cloth and subtle muscle moments. This kind of non-linear local deformation that can efficiently improve the generative visual effects has not been explored in this work yet, which could be the future

---

Driving motion
Target mesh
Driving motion
Target mesh
Transferred results
Ground traths
Trunsfered results
Ground truths
Noisy input handling
Temporal interpolation
Figure 7: Potential examples of LART application. The left side shows a denoising of LART. Given a noisy driving sequence, LART can still handle and complete the information and output relatively robust results. The right side shows using LART to interpolate incomplete sequence.
research direction. Another limitation is that LART (and all other baselines) relies on the latent pose space learned generate high-quality results. If motion is far away from the learned distribution, will lead to degenerated results. Although efforts have been made here to boost latent space learning, the problem of efficiently covering the whole latent space is far from being closed.
Conclusion
In this work, propose the LART framework, i.e., novel Transformer network customized for 3D motion transfer with carefully-designedd architectures. LART can implicitly learn the correspondence and also handle large-size full-detailed unseen 3D targets. Besides, introduce novel latent metric regularization for better motion generation. Our proposed LART shows a high learning efficiency with promising visual effects. The experimental results verify the potential of our generative model in applications of motion transfer, content generation, temporal interpolation, and motion denoising. The future direction could be compact architecture design and efficient latent space learning as strong priors to unseen domains.
6 Acknowledgement.
This work was supported by the Research Council of Finland (former Academy of Finland) Academy Professor project EmotionAl (grants 3361 16, 345122), ICT 2023 project TrustFace (grant 345948), the University of Oulu & Research Council of Finland Profi 7 (grant 352788), and Infotech Oulu. As well, the authors wish to acknowledge CSC IT Center for Science, Finland, for computational resources. Support by the Alexander von Humboldt Foundation and by the Bulgaria government is also gratefully acknowledged.
References
[1] Noam Aigerman, Kunal Gupta, Vladimir G Kim, Siddhartha Chaudhuri, Jun Saito, and Thibault Groucix. Neural jacobian fields: Learning intrinsic mappings of arbitrary meshes, SIGGRAPH, 2022.
[2] Mirela Ben-Chen, Ofir Weber, and Craig Gotsman. Spatial deformation transfer. Proceedings of the 2009 ACM SIGGRAPti/Eurographhe Symposium on Computer Animation, 2009.
[3] Bharat Lal Bhatnagar, Garvita Tiwari, Christian Theobalt, and Gerard Pons-Moll. Multi-garman net: Learning to dress 3d people from images. In ICCV, 2019.
[4] Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black. Keep it smpl: Automatic estimation of 3d human pose and shape from single image. In ECCV, 2016.
[5] Federica Bogo, Javier Romero, Matthew Loper, and Michael J Black. Faust: Dataset and evaluation for 3d mesh registration. In CVPR, 2014.
10

---

[6] Federica Bogo, Javier Romero, Gerard Pons-Moll, and Michael J Black. Dynamic faust: Registering human bodies in motion. CVPR, 2017.
[7] Haoyu Chen, Hao Tang, Nicu Sebe, and Guoying Zhao. Aniformer: Data-driven 3d animation with transformer. In BMVC, 2021.
[8] Haoyu Chen, Hao Tang, Henglin Shi, Wei Peng, Nicu Sebe, and Guoying Zhao. Intrinsic- extrinsic preserved gans for unsupervised 3d pose transfer. In ICCV, 2021.
[9] Haoyu Chen, Hao Tang, Zitong Yu, Nicu Sebe, and Guoying Zhao. Geometry-contrastive transformer for generalized 3d pose transfer. In AAAI, 2021.
[10] Xu Chen, Yufeng Zheng, Michael Black, Ounar Hilliges, and Andreas Geiger. Snarf: Differ- entiable forward skinning for animating non-rigid neural implicit shapes. In ICCV, 2021.
[11] Luca Cosmo, Antonio Norelli, Oshri Halimi, Ron Kimmel, and Emanuele Rodolà. Limp: Learning latent shape representations with metric preservation priors. In ECCV, 2020.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2021.
[13] Lin Gao, Jie Yang, Yi-Ling Qiao, Yu-Kun Lai, Paul L Rosin, Weiwei Xu, and Shihong Xia. Automatic unpaired shape deformation transfer. TOG, 37(6), 2018.
[14] Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. 3d-coded: 3d correspondences by deep deformation. In ECCV, 2018.
[15] Chuan Guo, Xinxin Zuo. Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Action2motion: Conditioned generation 3d human motions. ACM MM, 2020.
[16] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on visual transformer. arXiv preprint arXiv:2012.12556, 2020.
[17] Pengeheng Hc, Xiaodong Liu, Jianfeng Gao, and Wei Chen. Deberta: Decoding- enhanned bert with disentangled attention. In ICLR, 2021.
[18] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput.. 175: 1997.
[19] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV, 2017.
[20] Diederik Kingma and Max Welling. Auto-encoding variational bayes. CoRR, 2013.
[21] Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black. Amass: Archive of motion capture as surface shapes. In ICCV, 2019.
[22] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019.
[23] Sida Peng, Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, and Hujun Bao. Animatable neural radiance fields for modeling dynamic human bodies. ICCV, 2021.
[24] Sida Peng, Yuanqing Zhang. Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In CVPR, 2021.
[25] Mathis Petrovich, Michael Black, and Gül Varol. Action-conditioned 3d human motion synthesis with transformer vae. In ICCV, 2021.

---

[26] Emanuele Rodolå, Luca Cosmo, Michael M Bronstein, Andrea Torsello, and Daniel Cremers. Partial functional correspondence. In Computer graphics forum, volume 36, pages 222-236. Wiley Online Library, 2017.
[27] Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: Modeling and capturing hands and bodies together. TOG, 36(6), 2017.
[28] Shunsuke Saito, Jinlong Yang, Qianli Ma, and Michael Black. Scanimate: Weakly supervised learning of skinned clothed avatar networks. In CVPR, 2021.
[29] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. In NeurlPS, 2019.
[30] Richard Sinkhorn. Diagonal equivalence to matrices with prescribed row and column sums. The American Mathematical Monthly, (4)4):402405,, 1967.
[31] Chaoyue Song, Jiacheng Wei, Ruibo Li, Fayao Liu, and Guosheng Lin. 3d pose transfer with correspondence learning and mesh refinement. NeurIPS, 2021.
[32] Robert W Sumner and Jovan Popović. Deformation transfer for triangle meshes. TOG, 23(3), 2004.
[33] Qingyang Tan, Lin Gao, Yu-Kun Lai, and Shihong Xia. Variational autoencoders for deforming 3d mesh models. CVPR, 2018.
[34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, undefinedukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[35] Jiashun Wang, Chao Wen, Yanwei Fu, Haitao Lin, Tianyun Zou, Xiangyang Xue, and Yinda Zhang. Neural pose transfer by spatially adaptive instance normalization. CVPR, 2020.
[36] Shaofei Wang, Marko Mihajlovic, Qianli Ma, Andreas Geiger, and Siyu Tang. Metaavatar: Learning animatable clothed human models from few depth images. In NeurlPS, 2021.
[37] Ting-Chun Wang, Ming- Yu Liu, Jun-Yar Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-video synthesis. In NeurIPS, 2018.
[38] Guoying Zhao, Yante Li, and Qianru Xu. From emotion ai to cognitive ai. International Journal of Network Dynamics and Intelligence, 2022.
[39] Ce Zheng, Sijie Zhu, Matias Mendieta, Taojiannan Yang, Chen Chen, and Zhengming Ding. 3d human pose estimation with spatial and temporal transformers. ICCV, 2021.
[40] Keyang Zhou, Bharat Lal Bhatnagar, and Gerard Pons-Moll. Unsupervised shape and pose disentanglement for meshes. In ECCV, 2020.
12


Reviews:
[{"review_id": "RhvB8VojQC", "reviewer": "Reviewer_hZ8Y", "summary": "The paper proposes to improve the SOTA of the learned pose/motion transfer on unrigged 3D meshes. The architecture consists of a geometry adaptive feature encoder, a LART decoder, and a latent metric regularizer.\n\nThe geometry adaptive feature encoder first extracts features similar to NPT [33] by casting each vertex to a higher dimensional feature, e.g. 1,024 dimensions, through a series of 1D convolutions followed by an instance norm. A typical approach then is to either concatenate or add the feature to the embedding, which limits the model to only work on a fixed number of vertices. Instead, the paper proposed to apply max pooling across all vertices of the extracted feature, add to the embedding, then tile this to an arbitrary number of vertices.\n\nThe resulting feature is passed to the LART decoder which is essentially a transformer to attend to the corresponding vertex in the driving mesh and the target mesh. This is unlike 3D-CoreNet [29] where the correspondence is learned explicitly with the optimal transport.\n\nFinally, the model applies the latent metric regularization to encourage the poses in motion to be interpolatable.", "questions": "Can authors provide videos of various results, as the paper's title is \"motion transfer?\" How well does the method handle temporal coherency? Can authors share the video of the ablation on the latent metric regularization?\n\nHave authors tried the method on a driving sequence with varying vertex count, e.g. raw 4D scan output? In theory, the method seems to be able to handle this.\n\nCan authors demonstrate the effect of the flattening by applying algebraic operation? Would it make sense to show the results of motion blending and motion in-betweening?\n\nIs there a way to evaluate correspondence learning compared to others like DiffusionNet [Attaiki et al. 2022]?", "limitations": "As the authors infer, the method will not handle geometric and physical constraints like volume preservation and collisions. I understand that such effects are not in the scope of this work.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "All proposed components are novel and reasonable to achieve the goal. In particular, the use of the transformer to implicitly learn the 3D geometry correspondence is a great idea.", "weaknesses": "No videos are provided, making it hard to qualitatively discuss the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your acknowledgment of the novelty of our work and the constructive feedback! We address your questions and concerns in the following.\n\n**Q1: Can authors provide videos of various results, as the paper's title is \"motion transfer?\" How well does the method handle temporal coherency? Can authors share the video of the ablation on the latent metric regularization?**\n\nA1:  The result videos can be found at this anonymous link:\nhttps://we.tl/t-WDuWeIFY0K\n\nIn the folder ‘SOTA comparison’, we visualize the results of both our model and other STOA methods (3DCoreNet, NPT, LART w/o LMR, LART, and Ground Truth). We summarize the advantages of our LART over others in the following:\n\n(1) Comparison with SOTA methods: with the qualitative results shown in the video, our method performs better than existing SOTA methods in the visual effects, while the quantitative evaluation in the paper also proves it.\n\n(2) Temporal coherency with LMR ablation study: as one can see, although the LART (both with and without LMR) outperforms other SOTA methods in general, the LART without LMR still has observable artifacts such as shrinking arms and body while the LMR can substantially improve the temporal coherence.\n\n**Q2: Have authors tried the method on a driving sequence with varying vertex count, e.g., raw 4D scan output? In theory, the method seems to be able to handle this.**\n\nA2: Thanks for your suggestion. We followed your suggestion and added additional experiments as requested:\n\nSetting: We use the official DFAUST release raw 4D scans, with subject 50027, motion ‘shaking arms’ as input.\nWe randomly sampled 6,890 points from the raw scans (since the vertex point number of the raw scan is too huge with more than 15,000 points) and conducted the motion transfer with a pre-trained LART to verify its performance both with and without any further finetuning settings. The result videos can be found at the above anonymous link, folder ‘raw scan input’.\n\nAs we can see, using a pre-trained model without any not finetuning to transfer motion from raw 4D scans is extremely challenging (existing methods all need domain-specific finetuning to take raw data as inputs, such as MetaAvatar [S Wang 2021, NeurIPS]). Because the domain distribution of watertight meshes in the training and raw scans in the testing are quite different. For instance, the distortion on the hands in the video is caused by the too-sparse point sampling on the hands with the naive random sampling, while the points on watertight meshes are evenly given. Thus, we further finetuning the model on 4D, the raw 4D scan for ten epochs, the visual results get much better, and the domain de. To conclude, the demo shows the potential of our LART to work directly on 4D raw scans with domain-specific finetuning, which could be the future direction.\n\n**Q3: Can authors demonstrate the effect of the flattening by applying algebraic operation? Would it make sense to show the results of motion blending and motion in-between?**\n\nA3: Thanks for your valuable suggestion. Exploring the potential of applying algebraic operations directly on the latent space would be interesting. We demonstrate two examples by 1) taking two frames from the same motion, i.e., interpolating (boxing), and 2) taking two frames from different motions (one_leg_loos and chiken_wings), i.e., blending, as the first and last frame to generate the corresponding mid-frame outputs by varying the alpha in Equ (3). We attach the results in the ‘blending and interpolation’ folder; as we can see, our LART has the potential to generate reasonable ‘in-between’ outcomes: the intermedia pose has both lifting leg and raising arm actions.\n\n**Q4: Is there a way to evaluate correspondence learning compared to others like DiffusionNet [Attaiki et al. 2022]?**\n\nA4: We really appreciate your valuable suggestion to find a way to evaluate correspondence learning and mention the DiffusionNet [Attaiki et al. 2022]. DiffusionNet focuses on the 3D shape-matching problem. Although it’s similar to 3D motion transfer to some extent, it belongs to another research and beyond our research scope. Although aligning the correspondence can be beneficial to the motion/pose transfer, it’s not the ultimate goal in our 3D motion transfer task. One can still achieve robust motion/pose transfer without the need for correspondence alignment. Thus, we think quantitatively evaluating the correspondence learning with rigorous metric evaluation might go too far from the original task and might mislead the research direction. However, we agree with the reviewer that it would be meaningful to quantitatively evaluate and verify the correspondence learning with the LART in different tasks, such as shape matching and deformation, as it shows promising qualitative results, but we will put it as future work.\n\n**Q5: As the authors infer, the method will not handle geometric and physical constraints like volume preservation and collisions. I understand that such effects are not in the scope of this work.**\n\nA5: \nWe agree with your comment that handling geometric and physical constraints can effectively improve the visual effect. How to effectively introduce those physical prior into the learning is a promising direction for the 3D motion transfer.\n\nPlease let us know if you have more questions or concerns; thanks!"}, {"review_id": "FTDALy7kk2", "reviewer": "Reviewer_gKzm", "summary": "This paper describes a method to transfer the dynamic mesh sequences to the unseen 3D mesh target. A transformer-based model is developed to implicitly learn the correspondence. In this model, pose and identity embeddings are separately encoded from the meshes. A decoder is designed to generate mesh sequences while considering the target mesh and encoded embeddings. The proposed method, LART, has been mainly evaluated on DFAUST dataset.", "questions": "I think the most attractive point of the proposed method is its generalization ability. \nIs there any other way to quantitatively / qualitatively demonstrate the generalization ability of the proposed method?", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Transferring 3D mesh motion to the 3D unseen target is very challenging in learning the correspondce between the input and the targetm esh. In this paper, a transformer-based model is trained to implicitly learn such correspondence. The idea is interesting. \n2. The paper is well written to clearly state the differences compared with previous methods and the main contributions.", "weaknesses": "The main concern is about the experiments. \nThe task is not claimed as transferring 3D motions between general subjects, not only for human meshes. While the evaluation, especially quantitative evaluation, is only about human. There are only a few qualitative results shown in the Fig. 5 to demonstrate the generalization ability of the proposed method, which may not be convincing enough. As a temporal method, it would be better if a video could be provided to show its visual performance. Therefore, the contributions may be a little over-claimed. Current experiments may not be enough to support them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your acknowledgment of the challenge of our work and the constructive feedback! We will address your questions and concerns in the following:\n\n**Q: The main concern is about the experiments. I think the most attractive point of the proposed method is its generalization ability. Is there any other way to quantitatively / qualitatively demonstrate the generalization ability of the proposed method?**\n\nA: Please see our response to the first concern of reviewer 1. In short, we include additional qualitative videos to showcase the more generating performance of our LART in folder ‘more results’, including various motions, and providing experimental results with 4D raw scans as input to drive the target meshes in folder ‘4D raw scan input’, as well as noisy driving source in folder ‘noisy driving motion’. Furthermore, we demonstrate that our method can conduct linear operations to the latent space and achieve meaningful manipulation of the motions in the folder ‘blending and interpolation’. All those abilities demonstrate the generalization ability of our LART.\n\nThe result videos can be found at this anonymous link: https://we.tl/t-WDuWeIFY0K\n\nPlease let us know if you have more questions or concerns, thanks!"}, {"review_id": "8GJfYyiWA6", "reviewer": "Reviewer_SFNt", "summary": "The paper presents a method to transfer motion from a dynamic input sequence to a static 3D object. There are several novel components presented in the method: a novel feature encoder with an adaptive positional encoding scheme and a novel latent geometric regularization on the transformer. The paper is evaluated using motions from the AMASS dataset and shapes from the DFAUST dataset. ", "questions": "none", "limitations": "The authors have mentioned several limitations of their method in the main paper. I have mentioned a few of them above as well. But  despite the limitations and weaknesses, I think there is merit in the paper and it deserves acceptance. So I recommend a boderline accept rating. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is the first to address the problem of retargeting motion from a motion sequence to a novel shape. In this regard, it is quite a novel paper. There are some novel components in the method as well as listed above. I like that the method can also work when the driving motion meshes are noisy. ", "weaknesses": "The memory requirements of the method is a weakness; this is listed in the main paper itself. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your acknowledgment of the novelty of our work and the constructive feedback! \n\n**Q1: The memory requirements of the method are a weakness; this is listed in the main paper itself.**\n\nA1: Regarding the concern on the memory requirements, this memory allocation is predominantly attributed to the necessity of incorporating multiple frames as input data. The rationale behind this allocation is to empower the model to capture and comprehend temporal patterns with Latent Metric Regularization effectively. This, in turn, enables the model to capture and synthesize reasonable temporal dynamics (with locally linear in Euclidean space to gain better deformation effects). As discussed in the paper Section 3.1, few efforts have been made for end-to-end 3D motion transfer due to the substantial computational consumption. To our knowledge, it is the first attempt to achieve end-to-end 3D motion transfer with a customized Transformer architecture.\n\n**Q2: The authors have mentioned several limitations of their method in the main paper. I have mentioned a few of them above as well. But despite the limitations and weaknesses, I think there is merit in the paper, and it deserves acceptance. So I recommend a borderline accept rating.**\n\nA2: \nWe sincerely appreciate your acknowledgment of the merit in our work and your recommendation for a borderline accept rating and we are pleased that you recognize the significance of our work despite the identified limitations. \n\nPlease let us know if you have more questions or concerns; thanks!"}, {"review_id": "fPqgjkRq7m", "reviewer": "Reviewer_cJpb", "summary": "1.The paper presents LART, a 3D Transformer framework for 3D motion transfer. One of the distinctions from previous methods is that LART does not require joint annotation or pre-defined correspondence between the source and target mesh. By preserving motion metrics and effectively controlling synthetic motions in the latent space construction, LART achieves accurate motion synthesis.  The experimental results demonstrate the high learning efficiency of LART, requiring only a few samples from the AMASS dataset to generate motions with plausible visual effects. ", "questions": "See weakness", "limitations": "See weakness", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and presents the information in a clear and comprehensible manner.\n\n2. A novel latent geometric regularization is proposed for synthesizing realistic dynamic results.\n\n3. The proposed method demonstrates its versatility by successfully extending its applicability beyond human motion transfer to animal motion transfer, as illustrated in Fig 5.\n\n4. The method achieves good performance in both quantitative and qualitative evaluations, providing solid evidence of its effectiveness.", "weaknesses": "1. The author should provide more explanation and necessary information about specific terms and blocks used in the paper. For instance, a detailed description of the SPAdaIN block is needed, including its purpose and relevance in the decoder. Additionally, the author should further highlight that SPAdaIN is from [33], not just cite it without mention.\n\n2. The design of the cross-attention mechanism depicted in Fig 2 is not clearly described. More detailed explanations regarding its implementation are required.\n\n3. The proposed encoder is named \"geometry adaptive 3D feature encoder,\" but it seems that geometry is only involved in the positional embedding. It would be beneficial to consider incorporating a geometry-aware design in the encoder, in addition to the positional embedding.\n\n4. The paper lacks an ablation study on the three different positional embedding methods presented in Fig 3. It is important to analyze the performance of each positional embedding method, as the supplementary Table 7 only reports the loss without providing any analysis.\n\n5. While the paper includes visualizations in figures, it would be advantageous to provide a demo video to showcase the actual quality of the visualizations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your acknowledgment of the novelty of our work and the constructive feedback! We will address your questions and concerns in the following:\n\n**Q1: The author should provide more explanation and necessary information about specific terms and blocks used in the paper. For instance, a detailed description of the SPAdaIN block is needed, including its purpose and relevance in the decoder. Additionally, the author should further highlight that SPAdaIN is from [33], not just cite it without mention.**\n\nA1: We appreciate your valuable suggestion that can help us to illustrate our work for the following researchers better. SPAdaIN was first proposed by Wang et al. in [33] by injecting the original target mesh into each layer of the network with a residual-like connection. In this way, the detailed spatial geometry can be perceived throughout the whole flow of the network. Thus, the network can be geometry-aware and preserve the geometric information of the target meshes (i.e., the shape). Due to the page limitation, and as SPAdaIN is not the contribution we want to emphasize, we put the details of how we design the network (e.g., the choice of the SPAdaIN and attention structure) and the specific dimension of each layer and hyperparameters of the network are attached in the Appendix. As suggested, we will add some explanation of each introduced component in the paper in the new version.\n\n**Q2: The design of the cross-attention mechanism depicted in Fig 2 is not clearly described. More detailed explanations regarding its implementation are required.**\n\nA2: We appreciate your suggestion; the cross-attention block in Fig. 2 is originally demonstrated in the paper with Eq. (1), where q and k are the latent pose codes from the target mesh and driving pose. We will enhance the illustration of Fig. 2 and indicate that the cross-attention block (green on) is based on Equ. (1) with better indications in the new version.\n\n**Q3: The proposed encoder is named \"geometry adaptive 3D feature encoder,\" but it seems that geometry is only involved in the positional embedding. It would be beneficial to consider incorporating a geometry-aware design in the encoder in addition to the positional embedding.**\n\nA3: Thanks for your suggestion. As discussed in Q1. The geometry-aware function is achieved via SPAdaIN in the decoder part of LART, where it takes the raw target mesh as residual inputs to learn the spatial geometry, which is more efficient for the generation. Meanwhile, our encoder focuses more on the pose presentations and the pose correspondence learning between the target mesh and source pose. If we also introduce spatial and geometric aware design in the encoder, the network might be redundant and intricate as the complicated and too detailed geometric (such as wrinkles and tissues) are less necessary for the pose encoding.\n\n**Q4: The paper lacks an ablation study on the three different positional embedding methods presented in Fig 3. It is important to analyze the performance of each positional embedding method, as the supplementary Table 7 only reports the loss without providing any analysis.**\n\nA4: We appreciate that reviewer notice the ablation study of the embedding methods. Firstly, compared to concatenating embedding, adding embedding takes fewer memory allocations and is proven to perform better than (please see ViT [A Dosovitskiy · 2020], PoseFormer [C Zheng · 2021]), thus in this work, we take adding embedding by default. As the reviewer noticed, Table 7 is the ablation study of different embedding methods. However, we find it tricky to conduct a fair comparison with a good protocol. Precisely, our adaptive embedding is proposed specifically for target mesh with different vertex sizes than the motion sequences, while the fixed scheme embedding cannot; thus, we cannot directly compare those two schemes on different target sizes. As a result, we only report the evaluated reconstruction loss (i.e., the PMD used for evaluation) in training, in which the target meshes are all fixed size. This loss is equivalent to the evaluation of unseen motion settings which we use as the ablation study of different embedding schemes. Note that this loss is not fair enough as it only evaluates the model with target meshes with fixed vertex sizes which cannot fully demonstrate the strength of the adaptive embedding. We will clarify this in the paper.\n\n**Q5: While the paper includes visualizations in figures, it would be advantageous to provide a demo video to showcase the actual quality of the visualizations.**\n\nA5: Please see our response to the first concern of reviewer 1. In short, we include additional qualitative videos to showcase the more generating performance of our LART in folder ‘more results’, and provide experimental results with 4D raw scans as input to drive the target meshes in folder ‘4D raw scan input’ as well as noisy driving source in folder ‘noisy driving motion’. Furthermore, we demonstrate that our method can conduct linear operations to the latent space and achieve meaningful manipulation of the motions in folder ‘interpolation’.\n\nThe result videos can be found at this anonymous link: https://we.tl/t-WDuWeIFY0K\n\nPlease let us know if you have more questions or concerns; thanks!"}]

Author rebuttal:
We would like to thank all the reviewers and AC for dedicating your time and expertise to assess our manuscript thoroughly. We are glad by the positive remarks from the reviewers on various aspects of our work (the novelty of LART, the versatility of the proposed method, the robustness of handling noisy input, the generalization ability, and the paper being well written). 

We have taken each reviewer's suggestions and comments into consideration and have made comprehensive replies in the rebuttal, including a further explanation of specific technical details, addressing some ambiguous issues, and as well as various visualized experimental results requested by the reviewers.

We have provided individualized responses to each reviewer's comments in the box below. The extra experimental results are attached in the anonymous link:
https://we.tl/t-WDuWeIFY0K

Meta-review: The paper introduces a technique for transferring motion from a dynamic mesh sequence to a previously unseen 3D mesh target. Initially, the paper received borderline to positive scores. The rebuttal effectively addressed several concerns raised by the reviewers, resulting in positive evaluations from all reviewers in the end. The AC agreed with the reviewers' assessments and made the decision to accept the paper.

The authors are advised to moderate their claim regarding correspondence learning and adjust the paper title accordingly. Additionally, they are strongly encouraged to include supplementary material containing additional experiments on correspondence learning, as well as results pertaining to animals and hands, as previously promised.
Decision: Accept (poster)