Neural Pose Representation Learning for Generating and Transferring Non-Rigid Object Poses
Authors: Seungwoo Yoo
Juil Koo
Kyeongmin Yeo
Minhyuk Sung
Venue: NeurIPS 2024 poster (2024)
Keywords: Pose Variation Transfer
Mesh Deformation
Diffusion Models
Primary area: diffusion_based_models
TL;DR: We propose a novel method for transferring pose variations from one 3D mesh to another by training a diffusion model on Jacobian fields.

Abstract: We propose a novel method for learning representations of poses for 3D deformable objects, which specializes in 1) disentangling pose information from the object's identity, 2) facilitating the learning of pose variations, and 3) transferring pose information to other object identities. Based on these properties, our method enables the generation of 3D deformable objects with diversity in both identities and poses, using variations of a single object. It does not require explicit shape parameterization such as skeletons or joints, point-level or shape-level correspondence supervision, or variations of the target object for pose transfer.
To achieve pose disentanglement, compactness for generative models, and transferability, we first design the pose extractor to represent the pose as a keypoint-based hybrid representation and the pose applier to learn an implicit deformation field. To better distill pose information from the object's geometry, we propose the implicit pose applier to output an intrinsic mesh property, the face Jacobian. Once the extracted pose information is transferred to the target object, the pose applier is fine-tuned in a self-supervised manner to better describe the target object's shapes with pose variations. The extracted poses are also used to train a cascaded diffusion model to enable the generation of novel poses.
Our experiments with the DeformThings4D and Human datasets demonstrate state-of-the-art performance in pose transfer and the ability to generate diverse deformed shapes with various objects and poses.

Neural Pose Representation Learning for Generating and Transferring Non-Rigid Object Poses
Seungwoo Yoo
Juil Koo
Kyeongmin Yeo KAIST
Minhyuk Sung
(dreamy1534 63days, aaaaa, mhsung) @kaist. kc.kr
Figure 1: Results of motion sequence transfer (left) and shape variation generation (right) using the proposed neural pose representation. On the left, poses from source shapes (first and third rows) are transferred to target shapes (second and fourth rows), preserving intricate details like horns and antlers. the right, new poses sampled from a cascaded diffusion model, trained with shape variations of the bunny (last column), transferred to other animal shapes.
Abstract
We propose novel method for learning representations of poses for 3D deformable objects, which specializes in 1) disentangling pose information from the object's s identity, 2) facilitating the learning of pose variations, and 3) transferring pose information to other object identities. Based on these properties, our method en- ables the generation of 3D deformable objects with diversity in both identities and poses, using variations of a single object. It does not require explicit shape param- eterization such as skeletons or joints, point-level supervision, variations of the target object for pose transfer. We first design the pose extractor to represent the pose keypoint-based hybrid representation and the pose applier to learn an implicit deformation field. To better distill pose information from the object's geometry, we propose the implicit pose applier output an intrinsic mesh property, the face Jacobian. Once the extracted pose information is transferred to the target object, the pose applier is fine-tuned a self-supervised manner to better describe the target object's shapes with pose variations. The extracted poses are also used train a cascaded diffusion model enable the generation of novel poses. Our experiments with the Deform TefformhhinggDD and Human datasets demonstrate state-of oftte-of-hh-arr performance in pose transfer and the ability to generate diverse deformed shapes with various objects and poses.
shape-level correspondence
Introduction
The recent great success generative models [7 39 40 38
has been made possible not only due to advances in techniques but also due the enormous scale of data that has become available, such as LAION 355 for 2D image generation. For 3D data, the scale has been rapidly increasing. exemplified by Xb [10]. However, it is still far from sufficient to cover all possible 3D shapes, particularly deformable, non-rigid 3D shapes such as humans, animals, and characters. The challenge with deformable 3D shapes is especially pronounced due to the diversity in both
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

the identities and poses of the objects. Additionally, for a new 3D character created by designer, information about possible variations of the creature does not even exist.
To remedy the requirement of a large-scale dataset for 3D deformable shape generation, we aim to answer the following question: Given variations of a single deformable object with its different poses, how can we effectively learn the pose variations while factoring out the object's s identity and also make the pose information applicable to other objects? For instance, when we have a variety of poses of a bear (Fig. |left first row), our objectíve is to learn the space of poses without entangling them with the geometric characteristics of the bears. Also, we aim enable a sample from this space to be applied to new object, such as bull, to generate new shape (Fig. left, second row). We believe that such a technique, effectively separating pose from the object's s identity and enabling the transfer of poses to other identities, can significantly reduce the need for collecting large-scate saree-ccale datasets covering the diversity of both object identities and poses. This approach can even enable creating variations of a new creature without having seen any possible poses of that specific object.
Transferring poses from one object to another has been extensively studied in computer graphics and vision, with most methods requiring target shape supervision 422 5, 4 [2] or predefined pose parameterization
Without such additional supervision, our key idea for extracting identity-agnssee pose information and learning their variations is to introduce a novel pose representation along with associated encoding and decoding techniques. For this, we consider the following three desiderata:
50
20 25
1. Pose Disentanglement:
The representation should effectively represent the pose only without resembling the source object's identity when applied to the other object. The representation should be compact enough effectively learn its variation using a generative model, such as a diffusion model.
2. Compactness:
Transferability: The encoded pose information should be applicable new target objects.
As representation that meets the aforementioned criteria, we propose an autoencoding framework and latent diffusion model with three core components. Firstly, we design pose extractor and pose applier to encode an implicit deformation field with a keypoint-based hybrid representation, comprising 100 keypoints in the space, each associated with latent feature. Learning the deformation field enables disentangling the pose information from the object's identity, while the keypoint-basee other objects. However, simply
representation compactly encodes it and makes it transferable learning the deformation as a new position of the vertex is not sufficient to properly adapt the source object's pose information others. Hence, secondly, we propose predicting intrinsic property of the deformed mesh, Jacobian fields [54] 26 41 which can successfully apply the pose while preserving the identity of the target shape. To better preserve the target's identity while applying the pose variation from the source, thirdly, we propose a per-identity refinement step that fine-tunes the decoder in a relf-superviseer way to adapt to the variations of target shapes, with poses transferred from the source object. Thanks to the compact hybrid representation of pose, a pose generative model can also be effectively learned using cascaded diffusion models [8 21. enabling the generation of varying poses of an object with an arbitrary identity different from the source object.
In our experiments, compare our framework against tfate-of-the-art techniques for pose transfer on animals (Sec.(4.2) and humans (Sec. (S3c. Both qualitative and quantitative analyses underscore the key design factors of our framework, demonstrating its efficiency in capturing identity-agnostic poses from exemplars and its superior performance compared to existing methods. Additionally, we extend the proposed representation to the task of unconditional generation of shape variations. Our representation serves as a compact encoding of poses that can be generated using diffusion models (Sec. (466) and subsequently transferred other shapes. This approach facilitates the generation of various shapes, particularly in categories where exemplar collection is challenging.
2 Related Work
Due to space constraints, we focus on reviewing the literature on non-rigid shape pose transfer, includ- ing methods that operate without parameterizations, those that rely predefined parameterizations, and recent learning- based techniques that derive parameterizations from data.
Parameterization-Fre Pose Transfer. Early works [2 49 5 51 focused on leveraging point-
wise correspondences between source and target shapes. A seminal work by Sumner and Popović 42 transfers per-triangle affine transforms applied to the target shape by solving an optimization problem. A follow-up work by Ben-Chen et al. [5) transfers deformation gradients by approximating

---

X7
XT
24 (0.))
Caocaled Deffiman: Kintal
(Sec
H3-N(10.
FV
h
Pose Batrectoc (See 3.5)
Refincuccat Modale (Sec 5.6)
Posc Applics (Bec 3.2
Jacobias Fichla Poisson Solver
(Sec. 3.9
Figure 2: Method overview. Our framework extracts keypoint- based hybrid pose representations from Jacobian fields. These fields are mapped by the pose extractor 9 and mapped back by the pose applier h. The pose applier. conditioned on the extracted pose, acts as an implicit deformation field for various shapes, including those unseen during training. A refinement module a, positioned between g and h, is trained in a self-supervised manner, leveraging the target's 's template shape. The compactness of our latent representations facilitates the training of a diffusion model, enabling diverse pose variations through generative modeling in the latent space.
source deformations using harmonic bases. On the other hand, a technique proposed by Baran et al. 4 instead employs pose-wise correspondences by learning shape spaces from given pairs of poses shared across the source and target identities. The poses are transferred by blending existing exemplars. While these techniques require point-wise or pose-wise correspondence supervision, our method does not require such supervision during training inference.
Skeleton- or Joint-Based Pose Transfer. Another line of work utilizes handcrafted skeletons, which facilitate pose transfer via motion retargeting [1]. This approach has been extended by
incorporating physical constraints [9, 22 43
3] or generalizing the framework to arbitrary objects (50)
33 Several (carning-based methods [1]
25 446 have also been proposed to predict joint transformations involved in forward kinematics from examples. Recently, Chen et al. [8] proposed a framework that does not require skeletons during test time by predicting keypoints at joints. The method is trained to predict both relative transformations between corresponding keypoints in two distinct kinematic trees and skinning weights. However, the tasks of rigging and skinning are labor- intensive, and different characters and creatures often require distinet rigs with varying topologies. Liao et al. (24) notably presented representation that comprises character-agnostic deformation parts and semi-supervised network predicting skinning weights that link each vertex these deformation parts, although its performance hinges on accurate skinning weight prediction. In this work, we design more versatile framework that is applicable various shapes and provides better performance.
45
20,
Pose Transfer via Learned Parameterization. To bypass the need for correspondence pa- rameterization supervision, learning- based approaches [12 53
47 57 24 48, 36 37 explore
alternative parameterizations learned from exemplars. Yifan et al. (53) propose predict source and target cages and their offsets simultaneously. although their method still requires manual landmark annotations. Gao et al. [12] introduce VAE-GAN framework that takes unpaired source and target shape sets, each containing its own set of pose variations. The network is trained without direct pose-wise correspondences between samples from these sets, instead enforcing cycle consistency between latent representations. Although this work relaxes the requirement for correspondence super- vision, it still requires pose variations for both the source and target identities and individual training lift the requirement for gathering
for each new source-target pair. Numerous works 47 7 57 2 variations of target shapes by disentangling identities from poses, enforcing cycle consistency [57], or adapting conditional normalization layers 47 from image style transfer 30 Notably, Aigerman et
al. (2 train a network that regresses Jacobian fields from SMPI. (2 pose parameters. The vertex coordinates are computed by solving Poisson's 's equation [54] effectively preserving the shapes' local details. Wang al. [48] also train a neural implicit function and retrieve a shape latent from a template mesh of an unseen identity via autodecoding 29). This method models local deformations through mooddintte-based network that learns continuous deformation fields. However, such methods struggle to generalize to unseen identities due to their reliance global latent embeddings encoding shapes. We propose a representation that not only disentangles poses from identities but also allows for implicit queries using the surface points, thereby improving generalization new identities.
3

---

Specifically, we design g as a pose extractor that embeds a source object mesh M into a pose latent representation z = (MM)). This representation disentangles the pose information from the object's identity in .M² and facilitates transferring the pose to the target template mesh MM.. Given this pose representation, the pose applier h then applies the pose to M yielding the corresponding variation of the target object " = h( M²) ). Note that our method is not limited to transferring the pose of given variation of the source object the target mesh but can also apply a pose generated 3 Method
3.1 Problem Definition
Consider a source template mesh M² = (V', ,FS), given as a 2-manifold triangular mesh. The mesh comprises vertices V and faces ps. Suppose there exist N variations of the source template mesh, {Ms ..i.....M..... J. where each M = (V; F*) is constructed with a different pose, altering the vertex positions while sharing the same mesh connectivity F
Assume a target template mesh M³ (V", (V",F") is given without any information about its variations or existing pose parameterization (e.g.., skeletons or joints), Our goal is to define functions g and h that can transfer the pose variations from the source meshes to the target template mesh. For each variation of the source shape M its corresponding mesh M. for the target is obtained as:
M. = (h(g(M) M³ FF), for i = 1,2, N.
(1)
by a diffusion model to the target mesh. In S.c.3..3 we explain how a diffusion model can be trained with the latent pose representation extracted from source object variations. In the following, we first describe the key design factors of the functions and / to tackle the problem.
3.2 Keypoint-Basee Based Hybrid Pose Representation
To encode source shape M into latent representation 28, we consider its vertices V as its geometric representation and use them as input to the pose extractor g. which is designed as sequence of Point Transformer [56] 44 layers. These layers integrate vector attention mechanisms (55) with progressive downsampling of input point clouds. The output of the pose extractor g is a set of unordered tuples g = { b))
k=1 where z E RR3 represents a 3D coordinate of keypoint subsampled from VS via farthest point sampling (FPS) and h is a learned feature associated with 2. This set g forms a sparse point cloud of keypoints in 3D space, augmented with latent features. We set K 100 in our experiments.
K
This keypoint-basee based hybrid representation. visualized in Fig. 2 is designed to exclusively transfer pose information from the source to the target while preventing leakage of the source shape's identity characteristics. Since the keypoints {2 are sampled from V using FPS, they effectively capture the overall pose structure of VS while also supporting geometric queries with the vertices of a new mesh. This property is essential during the decoding phase, where the pose applier h predicts the pose-applied mesh from the input template as an implicit deformation field.
The pose applier h is implemented with a neural network that takes the 3D coordinates of a vertex from the input template mesh as a query, along with the hybrid pose latent representation Z, and outputs the new position of the vertex in the pose-applied deformed mesh. Note that h indicates a function that collectively maps all vertices in the input template mesh to their new positions using the network. Like the pose extractor 9. the implicit deformation network is also parameterized as Point Transformer layers [56). It integrates the pose information encoded in z by combining vector attention mechanisms with nearest neighbor queries aggregate features of the keypoints z around each query point. The aggregated features are then decoded by an MLP to predict the vertex coordinates of the deformed shape. (This is base network, and we also introduce better way to design the implicit deformation network in Sec. (.3)
Given only the variations of the source object {MS S.......,., }. we jointly learn the functions g and by reconstructing the variations of the source object as a deformation of its template:
Lv | M 112
While and is are trained using the known variations of the source object MS the latent representation ~ g(MS), when queried and decoded with the target template mesh, effectively transfers the

---

pose extracted by g from MM. However, we also observe that and h, when trained using the loss Cv, often result in geometry with noticeable imperfections and noise on the surfaces. To address this, we explore an alternative representation of a mesh that better captures and preserves geometric details, which will be discussed in the following section.
3.3 Representing Shapes Jacobian Fields
In this work, we advocate employing the differential properties of surfaces as dual representations of a mesh. Of particular interest are Jucobian fields, a gradient-domain representation noted for its efficacy in preserving local geometric details during deformations [2] resulting surfaces maintain smoothness [13].
while ensuring that the
Given a mesh M = (V, F), Jacobian field J represents the spatial derivative of a scalar-vall valued function o M + IR defined over the surface. We discretize e as Φv E RIV,, sampling its value at each vertex v of the vertex set V. The spatial derivative of Φ at each triangle e F is computed as Vrdv using the per- ger-triangle gradient operator Vr. Given that each dimension of vertex coordinates V is such function, we compute its spatial gradient at each triangle as Jr = VrV. Iterating this process for all triangles yields the Jacobian field J = [Jr[f E F}.
To recover V from given Jacobian field J, we solve least-squares problem, referred to Poisson's equation:
V
argmin| LV
V AJ
where L E V is the cotangent Laplacian of M, is the stack of gradient operators defined each f and A E ([(((( is the diagonal mass matrix, respectively. Since the rank of L is at most [V 1, we can obtain the solution by fixing a single point, which is equivalent to eliminating one row of the system in Eqn. Since in Eqn. Deemains constant for given shape M, we can prefactorize the matrix (e.g.,using Cholesky decomposition) and quickly solve the system for different Jacobian fields J's. Furthermore, the upstream gradients can propagated through the solver since it involves only matrix multiplications 2
Employing Jacobian fields as shape representations, we now modify the implicit deformation network described in Sec. 3.2 to take face center coordinates as input instead of vertex coordinates and to output a new face Jacobian for query face instead of a new vertex position. This results in decomposing h into two functions h = (f o h²), where h is function that collectively maps all the faces in the input mesh to the new Jacobian, and & is a differentiable Poisson solver layer. Both g and h' are then trained by optimizing the following loss:
L, IV E(N')(M)) M 1)122
3.4 Per-Identity Refinement using Geometric Losses
While the latent pose representation 2 learned by g and h exhibits promising generalization ca- pabilities in transferring poses, the quality of the transferred shapes can be further improved by incorporating a trainable, identity-specific refinement module into our system. This module is trained in a self-supervised manner with the set of pose-applied target meshes. Similarly to techniques for personalized image generation [19] 52]. we introduce a shallow network a between and h, optimizing its parameters while keeping the rest of the pipeline frozen.
The optimization of a is driven by geometric losses, aiming to minimize the geometric discrepancies in terms of the object's identity between the target template mesh M² and the pose-transferred meshes M In particular, we first extract poses {2 2 } corresponding to the known shapes (MS .M } of the source object. A transformer- based network which maps latent representation 2 to S is plugged in between the pose extractor g and the pose applier h:
V h(a(2Z),
(5)
The parameters of a are updated by optimizing the following loss function:
Liet = Cuap (VT, V")
)+ + Aedge Lodge (VT, VT) + Avog Σ L 2 + | - 11))
(6)
5

---

where (.)p()) is the mesh Laplacian loss [27], edgee (-) is the edge length preservation loss [24], and Map Aedges and Areg are the weights of the loss terms. The definitions of the losses are provided in the appendix. Note that this refinement step leverages only the originally provided template shape M and does not require its given variations or any other additional supervision.
3.5
Learning Latent Diffusion via Cascaded Training
The use of the keypoint- based hybrid representation discussed in Sec. 3.2 offers a compact latent space suitable for generative modeling using diffusion models 39
40]38. Unlike the Jacobian fields with dimensionality [F] x 9, the keypoints and their feature vectors that comprise the pose representation g lie in significantly lower dimensional space, facilitating generative modeling with latent diffusion models [34].
17
We employ a cascaded diffusion framework [T8, 21 to separately capture the layouts of keypoints and the associated feature vectors. Given a set { ...., Z } of N latent embeddings extracted from the
known source shape variations {MS :. M ). we first learn the distribution over 2 {(2]] k=1' To handle unordered sets with small cardinality, we employ a transformer-based network to facilitate interactions between each element within the noise prediction network e, (Z, t) where & is diffusion timestep and Z is noisy 3D point cloud obtained by perturbing clean keypoint set Z (= Z via forward diffusion process [17.. We train the network by optimizing the denoising loss:
Lzs = e N(O.D), ((0.1) [[| e, (Z, ()]1]]
Likewise, the distribution of the set of latent features H { h k=1 is modeled as a conditional diffusion model En. which takes 2 as an additional input to capture the correlation between z and H The network is trained using the same denoising loss:
K
LHS = EHS e No..), ~ u(0.1) [lle 6, (H, .)]]).
(8)
Once trained, the models can sample new pose representations through the reverse diffusion steps [1
3,
-
(Z it)
(9)
v
0
VI
αt
H
2-1
H
E,1 (H, 13
(10)
V
you
VI
ā
where 0tt, 0g and 8g are the diffusion process coefficients.
Experiments
4.1 Experiment Setup
Datasets. In our experiments, we consider animal and human shapes that are widely used in various applications. For the animal shapes, we utilize animation sequences from the Deforming DooommingThinns4ID--- Things4D-AAniall dataset [23). Specifically, we extract 300 meshes from the anima- tion sequences of each of 9 different animal identities, spanning diverse species such as bears, rabbits, dogs, and moose. For humanoids, we use SMPL [28] 31), which facilitates easy generation of syn- thetic data for both training and testing. We sample 300 random pose parameters from VPoser [I to generate variations of an unclothed human figure using the default body shape parameters, which used to train our networks. For testing, we keep the pose parameters constant and sample 40 different body shapes from the parametric space covered by the unit Gaussian. This produces 40 different identities, each in 300 poses. The generated meshes serve as the ground truth for pose transfer. To assess the generalization capability unusual identities that deviate significantly from the default body shape, we increase the standard deviation to 2.5 when sampling SMPL. body parameters for 30 of the 40 identities. Additionally, we collect stylized character meshes from the Mixamo dataset [1] to test the generalizability of different methods. For diffusion model training, the extracted keypoints and their associated features from the given set of source meshes are used as the training data for our cascaded diffusion model, which is trained separately for each identity.
Baselines. To assess the performance of pose transfer, we compare our method against Aigerman et al. 22 (NJF), Liao et al. (24] (SPT), Wang et al. [88 (ZPT), and various modifications of our

---

framework. For NJF 12, we use the official code from the Morphing Humans experiment, em- ploying a PointNet [32 encoder to map imput shapes to global latents, and we train the model on our datasets. For SPT 24 we use the official code and pretrained model on humanoid shapes. Since a pretrained model for animal shapes is not provided, the comparison with SPT on the Saaeniinggiiinnnn--nniiai
dataset is omitted. For ZPT [48], the official implementation is not provided, so we implemented the model based on the description in the paper. In our ablation study, we explore different variations of our method to assess their impact on performance, including: (1) using vertex coordinates as shape representations (as described in Sec. (.2). and (2) omitting the per-identity refinement module (Sec. 3.4). All models (except for SPT [24,, for which employ pretrained model) are trained for each shape identity.
Evaluation Metries, For pose transfer, when the corresponding shapes of the same pose are given for both source and target shapes, in the SMPL case, we measure accuracy using Point-wise Mesh Euclidean Distance (PMD) [57.4],, following our baselines 24 48 Note that this measurement cannot be applied in the DeformingThings4DD. Things4D-Animal case since pose-wise correspondences are not provided. For both pose transfer and shape generation (via pose generation), we measure the visual plausibility of the output meshes using FID [16], KID [6), and ResNet classification accuracy with images rendered from four viewpoints (front, back, left, and right) without texture. For the latter, we train a ResNet-18 [5 network using 10,800 images rendered from the four viewpoints of all ground truth shape variations of each animal.
4.2 Pose Transfer on Deforming TDingg
We begin our experiments by transferring pose variations across different animals, a challenging task that necessitates strong generalization capabilities due to the diverse shapes of the animals involved.
Deforming -nniiss
23
SMPL
28
FID +
KID
ResNet
PMD L
FID 4
KID 1
ResNet Acc. f (%)
(x 10 3)
(x 10
Acc. f (%)
(x10 "
(x10 3)
(x10
NJF
(2)
11.33
5.71
64.43
2.55
1.57
0,82
70.93
SPT 24
0.28
0.83
0.43
75.38
7PT
48
19.88
11.09
48.15
1.28
0.77
0.45
69.88
Ours 1.11
0.42
78.72
0.13
0.30
0.19
79.09
Table 1: Quantitative results on the experiments using the and the human shape dataset populated using SMPL (28).
dataset
24]
We summarize the quantitative metrics in Tab. (left) Our method outperforms NJF 2 and ZPT 48
both of which use global latent codes to encode shapes, while ours uses a keypoint- -based hybrid representation. Note that SPT 24 is not compared in this experiment since the pretrained model not provided for animal shapes. Qualitative results are also shown in Fig. 3 demonstrating the transfer of poses from source mesh MS (second and seventh column, red) to target template mesh
M (first and sixth column, blue). Both NJF (2 and ZPT [88] introduce significant distortions to the results and often fail to properly align the pose extracted from the source the target. Our method, on the other hand, effectively transfers poses to the targets while preserving local geometric details. More results can be found in the appendix.
M
M
NIF 2
7PT
48
Ours
M
M
NJF 2
7PT
48
Ours
Figure 3: Qualitative results of transferring poses of the source meshes S (red) in the Deform- IngThing54D animals (33 to target templates M¹ T 's (blue). Best viewed when zoomed in.
7

---

4.3 Pose Transfer on SMPL and Mixamo
We further test our method and baselines using humanoid shapes ranging from SMPL to stylized characters from the Mixamo [ dataset. While we employ the parametric body shape and pose model of SMPL (28 31], it is important to note that this is only for evaluation purposes; our method does not assume any parametric representations, such skeletons, for either training inference.
Tab. (right) summarizes the evaluation metrics measured across the 40 target shapes. Notably, our method achieves lower PMD than SPT 241. which is trained a large-scale dataset consisting of diverse characters and poses, while ours is trained using only 300 pose variations of the default human body. This is further illustrated in the qualitative results in Fig. 4 where demonstrate pose transfer from source meshes (red) to target template meshes (blue) not seen during training. As shown, the shapes transferred by method accurately match the overall poses. Our method benefits from combining a keypoint-basee hybrid representation with Jacobian fields, outperforning the baselines in preserving local details, especially in areas with intricate geometric features such as the hands. See the zoomed-in views in Fig. More results can be found in the appendix.
M
M
NJF 2
SPT 24
ZPT 48
Ours
M
Figure 4: Qualitative results of transferring poses of the default human meshes M (red) to different target template meshes M (blue). The ground truth targets MEN's 's (grey) are displayed for reference. Best viewed when zoomed in.
Furthermore, we apply our model to more challenging setup involving stylized characters. Fig. we present qualitative results using shapes from the Mixamo [1] dataset. Despite being trained on single, unclothed SMPL body shape, our method generalizes well stylized humanoid characters with detailed geometry (first row) and even to character missing one arm (second row).
M
M
NJF
SPT 24
ZPT 48
Ours
Figure 5: Qualitative results of transferring poses of the default human meshes M (red) to target template meshes M² 's (blue) of Mixamo characters Best viewed when zoomed in.

---

4.4 Ablation Study
Our framework design is further validated by comparisons against different variations of our frame- work, as listed in Sec. 4.1 in the pose transfer experiment animal shapes discussed in Sec. 4.2 Tab. (left) summarizes the image plausibility metrics measured using the results from our internal baselines. Qualitative results presented in Fig. 6 Our method, which extracts pose representations from Jacobian fields and leverages the per-identity refinement module, achieves the best performance among all the variations.
Poses from M
Generated Poses (Sec.
3.5
Jacobian (Sec. 3.3
Refinement (Sec. 3.4
FID (x10 3)
KID
ResNet
FID
KID
ResNet
(x10*
Acc. f (9)
(x10** 3)
(x10 3)
Acc. f (96)
3.52
2.13
55.67
9.52
4.69
44.34
X
1.17
0.47
75.13
4.40
2.39
75.82
1.11
0.42
78.72
4.22
2.24
78.81
Table 2: Ablation study using the poses from the source shapes Deforming dataset (left) and the poses generated from our cascaded diffusion model.
DeformmingThings4D--AAMM Things4D-Animals
M
M
Vertex Only
Jacobian Field Only
Ours
Figure 6: Qualitative results from the ablation study where pose of the source shape M (red) in the Ceeeemiinghingg4--AAne [23 is transferred to the target template shape M (blue).
4.5 Sensitivity to Number of Keypoints
We examine the sensitivity of our method to the number keypoints by testing different variants of our framework while varying the number of keypoints extracted by the pose extractor to 50, 25, and 10, respectively. These variants are trained using the same SMPL (28) human body shapes and animal shapes from the ChfoomminghinggD---Annee
[23] dataset. Our per-identity refinement
stage (Sec. 3.4p is omitted to focus exclusively on the impact of keypoint counts on performance. Tab. summarizes FID and PMD measured using the DifoommnnghhnggsD--Aiiiii
and SMPL As reflected in both quantitative and qualitative results, reducing the number of keypoints does not significantly affect pose transfer accuracy.
dataset, respectively. We showcase qualitative results in Fig. 7 and Fig. g
23
SMPI. 28
Method
Ours-10 Ours-25
Ours-50
Ours-100
Method
Ours-10
Ours-25
Ours-50
Ours-100
FID (x10-3)
PMD
1.25
0.87
0.83
0.72
0.20
0.17
0.17
0.13
(x10-
Table 3: Quantitative results from the variants of our framework trained to extract different number of keypoints. Ours-N denotes variant of our network trained to extract keypoints.
M
M
Ours-10 10
Ours-25
Ours-50
Ours-100
Figure 7: Qualitative results of transferring a pose of the source shape M (red) in Deffomingghinnss4--Aniii
23 to the target template shape M (blue) using variants of framework (Ours-N), trained to extract keypoints.

---

M
M
Ours- 10
Ours-25
Ours-50
Ours-106
Mar
Figure 8: Qualitative results of transferring a pose of the default human mesh M (red) to the target template mesh M² (blue) using variants framework (Ours-N), trained to extract N keypoints.
4.6 Pose Variation Generation Using Diffusion Models
We evaluate the generation capabilities of our diffusion models trained using different pose represen- tations. Since no existing generative model can learn pose representations transferable across various shapes, we focus on analyzing the impact of using Jacobian fields on generation quality. We use shapes obtained by applying 300 generated poses to both MM's (red) and various M 's (blue). The quantitative and qualitative results are summarized Tab, (right) and Fig. 9 respectively. While the poses are generated using the diffusion model, our model still achieves ResNet classification accuracy comparable to pose transfer experiment (Tab. 2 left). This tendency is also reflected in the qualitative results shown in Fig. These results validate that the latent space learned from variations Jacobian fields is more suitable for generating high-quality shape and pose variations compared to the one based on vertices. More results can be found in the appendix.
M
M
MT
MT
M
M
MT
M (Ours)
(Vertex)
(Ours)
(Vertex)
(Ours)
(Vertex)
(Ours)
(Vertex)
Figure 9: Pose variation generation results. Each row illustrates the outcomes of applying generated pose to a source template mesh M (red) and a target template mesh M (blue).
5 Conclusion
We have presented method for learning novel neural representation of the pose of non-rigid 3D shapes, which facilitates: 1) the disentanglement of pose and object identity, 2) the training generative model due its compactness, and 3) the transfer of poses other objects' meshes. In our experiments, we demonstrated the -fate-of-the-arr performance our method in pose transfer, as well as its ability to generate diverse shapes by applying the generated poses to different identities.
Limitations. Our method leverages differential operators to compute the Jacobian field of the given template mesh, requiring additional preprocessing when dealing with meshes that have multiple disconnected components or defects in the triangulation. Our framework also assumes that a template mesh of the shape is known for pose transfer. We plan extend our framework for transferring poses between arbitrary shapes in future work.
Societal Impacts. Our generative model for poses and the pose transfer technique could potentially be misused for deepfakes. Developing robust guidelines and techniques to prevent such misuse is an important for future research.
10

---

Acknowledgments
We appreciate Minh Hieu Nguyen and Jisung Hwang for their support on creating figures. S. Yoo acknowledges the support of the Graduate School National Presidential Science Scholarship provided by Korea Student Aid Foundation. This work was supported by the NRF grant (RS-2023-00209723) IITP grants IRS-2022-1220594, RS-2023-00227592. DS-2024-00399817) and KEIT grant (RS- 2024-00433625) all funded by the Korean government (MSIT and MOTIE), as well as grants from the
Sketch Futuurre Research Center, NAVER-Intel Co-Lab, Hyundai NGV, KT, and Samsung Electronics.
References
[1] Adobe. Mixamo. pettp://www.niiee com/
[2] Noam Aigerman, Kunal Gupta, Vladimir G Kim, Siddhartha Chaudhuri, Jun Saito, and Thibault Groucix. Neural Jacobian Fields: Learning Intrinsic Mappings of Arbitrary Meshes. ACM TOG, 2022.
[3] Mazen Al Borno, Ludovic Righetti, Michael J. Black, Scott L. Delp, Eugene Fiume, and Javier Romero. Robust physics-based motion retargeting with realistic body shapes. Computer Graphics Forum, 2018.
[4] Ilya Baran, Daniel Vlasic, Eitan Grinspun, and Jovan Popović. Semantic deformation transfer. ACM TOG, 2009.
[5] Mirela Ben-Chen, Ofir Weber, and Craig Gotsman. Spatial deformation transfer. In Proceedings of the 2009 ACM SIGGRAPH//Erograppl Symposium on Computer Animation, 2009.
[6] Mikolaj Binkowski, Danica J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. In ICLR, 2018.
[7] Haoyu Chen, Hao Tang, Shi Henglin, Wei Peng, Nicu Sebe, and Guoying Zhao. Intrinsic- extrinsic preserved gans for unsupervised 3d pose transfer. In ICCV, 2021.
[8] Jinnan Chen, Chen Li, and Gim Hee Lee. Weakly-supervised 3d pose transfer with keypoints. In ICCV, 2023.
[9] Kwang-Jin Choi and Hyeong-Seok Ko. On-line motion retargetting. Proceedings. Seventh Pacific Conference Computer Graphics and Applications (Cat. No. PR002933,, 1999.
[10] Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al. Objaverse-xl: A universe of 10m+ 3d objects. In NeurlPS, 2023.
(11) Brian Delhaisse, Domingo Esteban, Leonel Rozo, and Darwin Caldwell. Transfer learning of shared latent spaces between robots with similar kinematic structure. In International Joint Conference Neural Networks (IJCNN, 2017.
[12] Lin Gao, Jie Yang. Yi-Ling Qiao, Yu-Kun Lai, Paul L Rosin, Weiwei Xu, and Shihong Xia. Automatic unpaired shape deformation transfer. ACM TOG, 2018.
[13] William Gao. Noam Aigerman, Groueix Thibault, Vladimir Kim, and Rana Hanocka. TextDe- former: Geometry Manipulation using Text Guidance. ACM TOG, 2023.
[14] Michael Gleicher. Retargetting motion new characters. ACM TOG, 1998.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
[16] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter: Gans trained by a two time-scale update rule converge to local nash equilibrium. In NewrIPS, 2017.
11

---

[17] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2019.
[18] Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. arXiv preprint arXiv:2106. 15282, 2021.
[19] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022.
[20] Hanyoung Jang, Byungjun Kwon, Moonwon Yu, Seong Uk Kim, and Jongmin Kim. variational u-net for motion retargeting. In SIGGRAPH Asia 2018 Posters, 2018.
[21] Juil Koo, Seungwoo Yoo, Minh Hieu Nguyen, and Minhyuk Sung. SALAD: Part-level latent diffusion for 3d shape generation and manipulation. /CCV, 2023.
[22] Jehee Lee and Sung Yong Shin. hierarchical approach to interactive motion editing for human-like figures. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, 1999.
[23] Yang Li, Hikari Takehara, Takafumi Taketomi, Bo Zheng, and Matthias Nießner. 4dcomplete: Non-rigid motion estimation beyond the observable surface. In ICCV, 2021.
[24] Zhouyingeheng Liao, Jimei Yang. Jun Saito, Gerard Pons-Moll, and Yang Zhou. Skeleton-free pose transfer for stylized 3d characters. ECCV, 2022.
[25] Jongin Lim, Hyung Jin Chang, and Jin Young Choi. Pmnet: Learning of disentangled pose and movement for unsupervised motion retargeting. In BMVC, 2019.
[26] Yaron Lipman, Olga Sorkine, Daniel Cohen-Or, David Levin, Christian Rössl, and Hans-Peter Seidel. Differential Coordinates for Interactive Mesh Editing. Proceedings of Shape Modeling International, pages 181-190, 2004.
[27] Minghua Liu, Minhyuk Sung, Radomir Mech, and Hao Su. DeepMetaHandles: Learning Deformation Hand-Ha of 3D Meshes with Biharmonic Coordinates. In CVPR, 2021.
[28] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael Black. SMPL: A skinned multi-person linear model. ACM TOG, 2015.
[29] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. CVPR, 2019.
[30] Taesung Park, Ming- Yu Liu, Ting-Chun Wang, and Jun- Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. CVPR, 2019.
[31] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani. Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3d hands, face, and body from a single image. In CVPR, 2019.
[32] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, 2017.
[33] Helge Rhodin, James Tompkin, Kwang In Kim, Edilson de Aguiar, Hanspeter Pfister, Hans- Peter Seidel, and Christian Theobalt. Generalizing wave gestures from sparse examples for real-time character control. ACM TOG, 2015.
[34] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[35] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. NeurIPS, 2022.
12

---

[36] Chaoyue Song, Jiacheng Wei, Ruibo Li, Fayao Liu, and Guosheng Lin. 3d pose transfer with correspondence learning and mesh refinement. In NeurIPS, 2021.
[37] Chaoyue Song, Jiacheng Wei, Ruibo Li, Fayao Liu, and Guosheng Lin. Unsupervised 3d pose transfer with cross consistency and dual reconstruction. IEEE TPAMI, 2023.
[38] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[39] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NeurlPS, 2019.
[40] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
[41] Olga Sorkine, Daniel Cohen-Or, Yaron Lipman, Marc Alexa, Christian Rössl, and Hans-Pers Peter Seidel. Laplacian Surface Editing. Proceedings of the EUROGRAPHICCS/ACM SIGGRAPH Symposium Geometry Processing, pages 179-188. 2004.
[42] Robert W. Sumner and Jovan Popović. Deformation transfer for triangle meshes. ACM TOG, 2004.
[43] Seyoon Tak and Hyeong- SSok Ko. physically-based motion retargeting filter. ACM TOG, 2005.
[44] Jiapeng Tang, Markhasin Lev, Wang Bi, Thies Justus, and Matthias Nießner. Neural shape deformation priors. In Advances in Neural Information Processing Systems, 2022.
[45] Ruben Villegas, Jimei Yang, Duygu Ceylan, and Honglak Lee. Neural kinematic networks for unsupervised motion retargetting. In CVPR, 2018.
[46] Ruben Villegas, Duygu Ceylan, Aaron Hertzmann, Jimei Yang, and Jun Saito. Contact-aware retargeting skinned motion. In ICCV, 2021.
[47] Jiashun Wang, Chao Wen, Yanwei Fu, Haitao Lin, Tianyun Zou, Xiangyang Xue, and Yinda Zhang. Neural pose transfer by spatially adaptive instance normalization. In CVPR, 2020.
[48] Jiashun Wang, Xueting Li, Sifei Liu, Shalini De Mello, Orazio Gallo, Xiaolong Wang, and Jan Kautz. Zero-shot pose transfer for unrigged stylized 3D characters. CVPR, 2023.
[49] Weiwei Xu, Kun Zhou, Yizhou Yu, Qifeng Tan, Qunsheng Peng, and Baining Guo. Gradient domain editing of deforming mesh sequences. In ACM TOG, 2007.
[50] Katsu Yamane, Yuka Ariki, and Jessica Hodgins. Animating non-humanoid characters with human motion data. In Proceedings of the ACM SIGGRAPH//Emngraphcc Symposium on Computer Animation, 2010.
[51] Jie Yang, Lin Gao, Yu-Kun Lai, Paul L. Rosin, and Shihong Xia. Biharmonic deformation transfer with automatic key point selection. Graphical Models, 2018.
[52] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arxiv:2308.06722 2023.
[53] Wang Yifan, Noam Aigerman, Vladimir G. Kim, Chaudhuri Siddhartha, and Olga Sorkine. Neural Cages for Detail- Preaer-Pres 3D Deformations. In CVPR, 2020.
[54] Yizhou Yu, Kun Zhou, Dong Xu, Xiaohan Shi, Hujun Bao, Baining Guo, and Heung- Yeung Shum. Mesh editing with poisson-bas based gradient field manipulation. ACM Trans. Graph., 23 (3), 2004.
[55] Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun. Exploring self-attention for image recognition. In CVPR, 2020.
13

---

[56] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip Torr, and Vladlen Koltun. Point transformer. In ICCV, 2021.
[57] Keyang Zhou, Bharat Lal Bhatnagar, and Gerard Ons-Moll. Unsupervised shape and pose disentanglement for 3d meshes. In ECCV, 2020.
14

---

Appendix
In the following appendix, we provide implementation details of our method, including dataset processing, network architectures, training details, and hyperparameter selection in Sec. A We also present multi-view renderings pose transfer example showcased in Sec. |4.3 in Sec. B the full list of evaluation metrics reported in Tab. i Sec. transfer and generation experiments Sec.
[ and additional qualitative results from our pose
Implementation Details
Data. We use nine animal shapes from the DeformingTThnng4--AAiiie experiments, specifically: BEAR, BUNNY, CANINE, DEER, DOG, ELK, Fox, MOoSE, and PUMA. We employ the first frame the first animation sequence (alphabetically ordered) as the template for each animal, and the last frame of randomly sampled animation sequences for variations. Additionally, we use T-posed humanoid shapes from both the SMPL 28 and Mixamo meshes.
23 dataset our
datasets as template
Networks. We utilize Point Transformer layers from Zhao et al. (56) and Tang et al. 44 for im-
plementing the pose extractor 9 and the pose applier h. The network architectures for cascaded diffusion models, as detailed in Sec. 4.6 are adapted from Koo et al. (21 These models operate over T = 1000 timesteps with linear noise schedule ranging from {3 1 10-4 to By 5 10-2 For model training, we employ the ADAM optimizer at learning rate of 1 x 10 and standard parameters. Our experiments conducted on RTX 3090 GPUs (24 GB VRAM) and A6000 GPUs (48 GB VRAM).
For per-identity refinement modules, set Alap 1.0, Aedge 1.0, and Xreg 5 x 10 during training.
We provide the definitions of the loss functions used to supervise the training of our per-identity refinement module.
Loss Functions.
The Laplacian loss 27 is defined between two sets of vertices with the vertex-wise correspondences:
Lr (v-v").
Ctap
(11)
where is set of new vertex coordinates, V is set of vertex coordinates of the target template mesh M². and Ly is the cotangent Laplacian of MM.
Likewise, the edge length preservation loss
24 is defined as:
Ledge
Σ IIV V,lla
I
|2|
(12)
(K.)EE
where & is set of edges comprising the target shapes and V. and V are the i-th vertex of set of vertex coordinates and that of M
B Multi-View Renderings of Qualitative Results
While Fig. 5 showcases only single-view images, our method produces high-fidelity 3D shapes after pose transfer as shown in Fig. A10
Ours (View 1)
Ours (View 2)
Ours (View 3)
Ours (View 4)
Figure A10: A pose transfer example showcased in Fig. 5 rendered from 4 different viewpoints.
15

---

C Full List of Quantitative Metrics Reported in Tab.
We report the full lists of evaluation metrics reported in Tab. from the following page.
Table A4: Full list of FIDs reported in Tab, (left)
Bear
Bunny
Canic
Deer
Dog
Moosc
Puma
Vertices (Sec.
3.2
Bear
3.47
11.53
6.76
6.14
8.40
4.92
5.26
9,80
Bunny
4.43
7.71
4.03
6.07
5.94
4.88
8.11
Canie
2.47
3.46
2.95
2.83
5.43
2.62
2.39
4.24
Doer
2.39
2.37
4.80
3.32
3.00
3.12
1.45
6.53
Dog
Filk
2,35
1.74
9.04
3.13
2.79
2.58
2,83
5.02
0.37
0.71
1.40
Moose
3,38
2.12
8.31
4.25
3.76
6.13
3.63
6.74
Pama
1.92
6.33
2.90
2.72
5.58
2.78
2,10
Jacobians (Sec
1.3
Bear
6.77
1.16
0.61
0.46
0.27
0.73
1.20
0.42
Bunny
3,36
6.08
2.16
4.94
3.87
2.12
1.41
4.37
Canie
Deer
0.49
1.94
2.44
1.25
1.68
0.34
0.27
0.98
Dog
Elk
0.44
5.23
0.99
0.63
0.46
0.49
1,08
0,51
Fox
0.99
1.74
0.74
1.47
1.37
0.53
1.27
Mouse
0.39
2.49
1.69
0.25
0.82
1.16
0.48
0.69
Puma
0.39
2,80
0.46
0.46
0.36
0.35
0.24
0.47
Ours (Incobians Refinement (Soc.
3.4)
Bear
0.60
0.56
0.30
0.41
Bunny
3.97
6.22
2.71
4.75
3.39
2,89
2.16
4.72
Canie
Deer
0.68
1.35
2.18
1.29
1.10
0.40
0.28
1.11
Dog
0.45
0.52
0.20
Elk
0.42
3.70
0.89
0.61
0.49
0.46
0.85
0,53
Fox
1.61
0.60
1.73
1.34
1.46
1.04
1.08
1.51
Moose
Puma
0.50
1.87
0.55
0.53
0.46
0.40
0.29
0.47
16

---

Table A5: Full list of KIDs reported in Tab, (left)
Burny
Canie
Dog
Vertices (Sec
3.2
Bear
2.56
8.87
4.95
4.46
6.15
3.65
3.61
6.95
Bunny
2.50
3.79
Canie
1,55
2.38
1.97
1.82
3.79
1.68
1.46
2.82
Doer
2.07
1.53
1.24
3.93
Dog
1.30
1.70
3.99
1.49
2.92
2.08
1.11
2,00
Ellk
1.37
6.57
2.22
1.60
1,07
1.91
3.02
Fox
0.26
0.39
0.16
0.25
0.39
0.64
Mouse
1.63
0.83
5.29
2.91
1.67
3.33
1.39
2,93
Jacobians (Sec
3.3
Bear
2.36
0.29
0.21
0.20
0.01
0.17
0.27
0.03
Bunny
1.97
Canie
0.17
0.67
0.10
0.25
0.25
0.14
0.11
0.19
Deer
0.10
0.11
0.99
0.70
0.93
0.08
0.03
0.41
Dog
0.08
0.11
0.13
0.05
0.18
-0.00
Elk
0.02
1.97
0.09
0.15
0.05
0.06
0.17
0.05
Moose
0.22
1.18
0.72
0.13
0.45
0.46
0,09
0.20
Puma
0.29
0.15
0.22
Ours (Jacobians Refinement (See. 3.4
Bear
1.26
0.29
0.23
0.20
0.01
0.08
0,27
0.07
Bunny
1.42
2.92
1.02
2.10
1.34
0.80
1.97
Canie
0.17
0.44
0.14
0.25
0.25
0.11
0,27
Door
0.19
0.11
0.94
0.66
0.46
0.08
0.03
0.41
Dog
Elk
0.02
1.34
0.09
0.17
0.05
0.04
0.17
0.09
Moose
0.22
0.68
0.72
0.19
0.45
0.46
0.04
0.29
Puraa
0.22
0.77
0.16
0.29
0.21
0.17
0.03
0.25
Table A6: Full list ResNet classification accuracies reported Tab. 2 (left)
Bear
Canie
Deer
Dog
Eilk
Fox
Moose
Puma
Vertices (Scc.
Bear
81.33
37.58
58.00
72.75
53.17
65.83
54.00
41.33
Bunny
57.25
26.83
51.25
41.33
35.58
$2.83
49.75
15.33
Canie
77.67
69.92
74.25
78.75
63.75
77.75
77.
57.58
Doer
70.33
93.50
85.33
97.58
98.08
67.42
Dog
67.50
45.42
17.58
$1.08
38.00
69.17
64.00
18.58
92.08
72.25
86.25
92.50
84.92
90.08
81.17
Fox
78.92
58.08
34.25
81.58
68.75
46.42
84.42
17.17
Moose
44.33
22.33
9.83
74.33
40.17
33.67
25.08
6.42
Puma
82.25
73.92
71.00
69.00
$1.08
71.50
81.00
77.92
Jacobians (Sec.
3.3
Bear
93.50
85.50
92.08
95.58
95.50
96.42
93.83
91.50
Bunny
86.67
60.00
72.83
69.00
66.75
85.33
79.08
63.08
Canic
77.83
80.25
83.92
81.33
76.17
92.83
81.17
80.08
97.25
Dog
85.50
58.58
60.58
82.92
75.92
85,50
76.75
65.33
Elk
86.67
73.17
86.75
81.42
89.00
91.17
85.67
91.00
Fox
86.83
62.33
64.17
82.67
74.08
72.25
84.58
76.50
Moose
94.08
92.92
69.33
92.83
89.42
69.17
80.83
78.75
Puma
90.75
78.08
87.58
87.17
89.75
88.83
88.25
88.25
Ours (Jacobians Refinement (Soe. 3.4)
Bear
91.67
92.50
93.00
96.58
95.75
97.42
94.42
94.67
Bunny
74.83
Canie
80.00
83.92
86,50
84,42
79.92
91.67
81,58
83.33
Door
98.58
94.00
99.08
99.92
99.50
99.75
99.25
99.67
Dog
86.92
66.17
64.25
86.00
80.83
86.58
77.75
74.08
EIk
88.25
75.42
90.17
82.92
91.17
92.08
86.83
92.00
Fox
87.58
81.08
84.25
89.00
91.17
87.42
89.92
91.42
Moose
95.75
94.00
75.50
93.33
91.92
77.25
92.75
86.08

---

Table A7: Full list of FIDs reported in Tab. (right)
Burny
Canie
Dog
Vertices (Sec
3.2
Bear
6.48
30.77
12.79
12.01
20.99
10.43
24.06
12.48
Bunny
36.26
13.34
Canie
4.03
2.58
5.06
4,95
8.12
4.54
9.67
4.71
Doer
23.16
10.69
14,91
15.48
13.75
Dog
2.62
1.69
12.12
4,06
7.53
3.47
9.51
3.57
Ellk
4.58
3.33
16.49
5.82
6.27
12.78
6.82
Fox
20.87
7.74
7.95
6,30
Mouse
7.60
6.18
22.20
9.00
10.64
15.46
10.74
11.63
Jacobians (Sec
3.3
Bear
2.33
9.66
2.12
4.62
14.47
1.82
0.80
2.74
Bunny
7.19
Canie
1.75
0.94
2.22
4.60
10.37
2.21
1,08
3.16
Deer
1.75
0.99
10.24
5.68
17.01
2.35
0.53
3.04
Dog
1.21
5.16
1.69
8.80
0.80
1.51
Elk
0.80
2.39
5.89
2.11
3.61
1.60
0.73
1.78
4.07
Moose
1.97
1.38
10.27
3.59
6.09
16.94
3.70
3.36
Puma
1.85
3.12
0.71
Ours (Jacobians Refinement (See. 3.4
Bear
1.46
9.01
2.20
4.49
13.71
2.14
0.85
2.72
Bunny
8.03
17.47
7.42
11.33
21.57
7.42
3.79
8.62
Canie
2.05
1.04
2.23
4.37
9.95
2.33
1.20
3.21
Door
2.18
1.04
8.96
5.45
14,57
2.53
0.78
3.11
Dog
Elk
0.99
1.84
5.57
2.09
3.58
1.65
0.79
1.79
Moose
2.36
1.25
9.65
3.72
5.98
16.01
3.46
3.50
Puraa
1.66
1.23
5.09
1.94
3.16
8.59
1.60
0.84
Table A8: Full list of KIDs reported in Tab. (right)
Bear
Canie
Deer
Dog
Eil
Fox
Moose
Puma
Vertices (Scc
Bear
3.53
15.17
7.02
5.90
10.50
5.51
10.86
6.91
Bunny
5.15
17.82
6.40
7.48
11.71
6.10
12.39
7.15
Canie
1.74
1.33
2.21
2.05
3.52
1.99
4.47
2.00
Doer
14.00
6.58
8.33
8.77
Dog
1.31
0.83
5.61
1.88
3.36
1.66
4.15
1.65
2.13
2.70
2.85
5,55
3.25
Fox
2.28
1.10
11.40
4.44
4.24
7.07
8.84
3,36
Moose
3.49
2.77
10.45
4.22
5.14
4,92
4,88
Puma
1.48
1.39
5.73
2.24
2.10
3.63
2.09
4.46
Jacobians (Sec
3.3
Bear
0.66
5.20
1.35
2.61
7.83
0.95
0.23
1.37
Bunny
2.85
10.52
3.68
6.35
12.27
2.81
0.97
3.84
Canic
0.65
0.39
1.29
2.25
5.33
0.77
0.24
1.35
Dog
0.52
0.60
2.90
1.13
4.84
0.40
0.19
0.67
Elk
0.20
3.32
1.38
1.97
0.77
0.14
0.85
Fox
1.96
0.27
6.08
2.89
4.52
8.89
0.64
2.48
Moose
0.79
0.54
5.68
2.43
3.41
9.92
1.86
1.51
Puma
0.78
2.99
1.29
1.77
0.38
Ours (Jacoiians Refinement (Sec. 3.4
Bear
0.35
4.87
1.39
2.57
7.38
1.16
0.28
1.38
Bunny
Canie
0.79
0.41
1.26
2.11
5.04
0.83
0.33
1.39
Door
5.93
3,74
10.15
0.28
1.45
Dog
0.71
0.42
2.67
1.20
4.54
0.55
0.28
0.72
EIk
0.29
0.59
3.12
1.39
1.97
0.77
0.18
0.86
Fox
2.23
0.84
4.52
2,46
3.47
1.19
2.11
Moose
0.97
0.44
5.31
2.52
3.35
9.27
1.68
1.61
18

---

Table A9: Full list of ResNet classification accuracies reported in Tab. 2 (right)
Bear
Canie
Deer
Dog
Ell
Fox
Moose
Puma
Bear
59.25
8.17
24.08
48.75
15.50
36.00
20.75
30.17
Bunny
38.50
0.83
15.00
21.25
9.75
24.33
13.83
5.17
Canic
72.58
78.42
56.67
71.92
39.50
61.50
35.67
66.58
Dog
66.75
45,00
6.17
28.08
13.92
51.17
24.25
14.75
OragonOLO
99.67
97.92
82.25
100.00
99.58
99.67
94.83
96.33
Elk
86.83
71.33
32.50
62.92
74.50
57.50
44.50
81.92
Rox
64.42
62.83
10.58
47.17
54,50
22.42
66.67
13:25
Moose
32.58
15.42
1.33
41.50
19.83
11.50
13.00
35.83
Jacobians (Soc.
3.3
Bear
91.58
77:75
90.33
93.92
82.67
94.92
94.83
90.67
Bunny
42.67
70.17
74.33
84.08
Canie
85.17
87.42
77.83
84.67
68.08
96.00
85.75
89.58
Doer
09.67
99.58
99.92
99.58
100.00
99.75
100.00
Dog
90.92
69.75
69.00
83.08
78.92
93.17
87.83
81.33
Elkk
94.50
80.67
94.42
78.58
92.42
95.00
91.92
92.50
Fox
89.17
76.33
59.92
79.25
78.67
39,08
92.17
89.58
Moose
97.00
91.58
57.92
84,50
86.83
45.17
68.67
85,33
Puma
84.75
90.75
Ours (Jacobiins Refinement (Seec: 3.4
Bear
90.83
84.75
90.58
96.08
86.75
96.42
94.75
95.92
Bunny
72.33
77.42
Canie
87.75
89.08
79.33
85.92
69.08
95.25
86.50
90.75
Deer
99.83
98.00
99.92
100.00
100.00
100.00
99.67
99.83
Dog
91.83
73.50
76.92
84,50
85.67
93.58
89.17
87.08
Ellk
95.00
81.92
95.00
81.75
92.50
95.00
93.17
93.75
Fox
88.17
85.58
78.58
82.92
91.25
61.33
90.50
94.75
Monse
97.08
92.00
67.08
84.92
91.08
55.83
86.92
89.75
Puma
93.75
94.50
86.75
92.08
91.17
92.75
19

---

D
Additional Qualitative Results
We showcase additional qualitative results from the pose transfer and pose generation experiments below.
M
M
s
NJF
2
ZPT
48
Ours
M T
M
NJF
2
ZPT
48
Ours
Figure All: Qualitative results of pose transfer across DeforningThing Things4D animals 23
20

---

M
MS
NJF
SPT 24
ZPT
48
Ours
Figure A12: Qualitative results of pose transfer from SMPL (88 mesh to Mixamo characters [ Best viewed when zoomed in.
21

---

M
M
NJF
SPT 24
ZPT 48
Ours
M
Figure A13: Qualitative results of pose transfer across different SMPL 28 human body shapes. Best viewed when zoomed-in.
22

---

M (Vertex)
M (Ours)
MT (Vertex)
M (Ours)
M (Vertex)
M (Ours)
MT (Vertex)
M (Ours)
Figure A14: Unconditional generation results. Each row illustrates the outcome of directly applying
the generated poses to the source shape M and then transferring them to various target shapes M²
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We fully discuss the claims made in the abstract and introduction throughout the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
All results reported in the paper are obtained through experiments.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed descriptions for the implementation of our framework and the data collection procedure.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

reproducible, they Answer: [No]
Justification:
We will publicly release the code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
public/guides www./oooaaiiiiiooooooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/ //nips http:c/cpplllgggiiieeeeeee
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, we provide detailed descriptions of the data collection and split procedure. Our texts also include important hyperparameters, such as the learning rates used for model training.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: bars.
Due to our limited computational resources, we were unable to report error
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
26

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We indicates the GPU resources used for our experiments in our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpp://///uurpps.cccpccccl/////
Answer: [Yes]
Justification: This work conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, we are aware of potential impact on our work and discussed it in the paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
We do not describe safeguards in our main paper.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cite all resources.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
28

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not present any new assets.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We did not conduct any form of crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our research does not include experiments that require IRB approvals and involving human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "T2Zwcjpi1h", "reviewer": "Reviewer_skND", "summary": "This paper introduces a novel 3D generative model for pose-identity disentangled representation of 3D shapes. It proposes to use a set of keypoints with features to represent the pose of a 3D shape and learn a pose-extractor and pose-applier to accomplish pose transfer between instances. Experiments are conducted on widely used benchmarks with two categories: animals and humans, showing superiority of the proposed method compared to the baselines.", "questions": "- How would the number of keypoints affect the result? What would be the fewest number of keypoints that is still feasible for this pipeline?\n- Regarding the experiments:\n  - What is the detailed setting of NJF? If I understand correctly, NJF takes the pose parameter of the target pose as input. What should be the pose parameter for the deformingthings4d dataset?\n  - How is the texture of the generated mesh predicted? In L265, the authors mentioned to use FID to evaluate the visual fidelity, yet how is this implemented, given that the generated mesh seems to not have texture? Also, how is the camera determined to generate the images used to calculate FID?\n- How is the \"template\" mesh defined in the framework, especially for categories without a template, such as animals? For humans, I understand that you may use T-pose human mesh as the template, but what would the case for the animals?", "limitations": "No failure cases are discussed in the paper. Under what condition would the proposed method fail?", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The studies task is novel and important, with broad applications in different communities.\n- The proposed method is sound and effective. Using a dense set of keypoints to represent the pose of 3D shape is natural and well-motivated, which also results good results in the experiments.\n- The evaluation is extensive and informative. Experiments are conducted on widely-used datasets, with comparisons against enough baselines. For baselines without the code, the paper provides results with own implementation. Ablation studies show the effectiveness of the proposed components.\n-  The writing of the paper is mostly clear, illustrating their method and evaluation in a clean manner.", "weaknesses": "- Although the exposition is mostly good to me, there still remains some questions that are not fully addressed. See the questions below.\n- No video results or animated results are provided. Therefore it's hard to evaluate the 3D/4D generated results (e.g., whether the result is 3D consistent from any view or whether the temporal consistency is good).\n- No qualitative results for the ablation study is provided. \n- It would be better if analysis on the keypoints can be provided. This might include analysis on the number of keypoints and the way to sample it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer skND,\n\nThank you for recognizing our work as a study of a \"novel\" and \"important\" task, and for appreciating our \"sound\" and \"well-motivated\" approach. We have carefully reviewed your queries and provide our responses below.\n\n**Temporal Consistency.**\nThank you for providing the constructive comment. In our work, we mainly focus on learning a transferable pose representation from pose variations of a *single* identity, which is more challenging than our prior works. While our framework has not been explicitly designed for motion transfer--such as by using a temporal attention mechanism or training with motion supervision--our framework can transfer motions by transferring each frame individually as demonstrated in Figure 1 in the PDF file in our global response. We applied the One Euro Filter, a simple and easy-to-implement filter, to refine the smoothness of motions. \nWe plan to extend our work for motion transfer in the future. We will include more results in the revised version.\n\n**Spatial Consistency.** While we only showed single-view images in our paper, our method produces meshes consistent in 3D as illustrated in Figure 2 in the accompanied PDF file. In particular, we rendered one of our results shown in Figure 5 of the main paper from 4 different viewpoints. We will add more images rendered from various viewpoints in the revision.\n\n**Qualitative Results for Ablation Study.** The qualitative result can be found in Figure 5 in the PDF file. As shown, directly using vertices results in low-quality meshes with noticeable distortions and artifacts. Our base model employing Jacobian fields produces high-quality shapes, which can be further improved by using the proposed refinement module. Note the preservation of intricate details near limbs. We will add the analysis and more results in the upcoming revision.\n\n**Impact of the Sampling Method and Number of Keypoints.**\nWe appreciate your suggestion for an in-depth analysis. We used FPS for keypoint extraction due to its simplicity and efficacy, which have made it a popular choice in geometry analysis (e.g., PointNet++ [Qi et al., NeurIPS 2017], KeypointDeformer [Jakab et al., CVPR 2021], DeepMetaHandles [Liu et al., CVPR 2021]). Here, we investigate the impact of the number of keypoints and their sparsity. We trained our model using SMPL and DeformingThings-Animals meshes, adjusting the pose extractor to extract 50, 25, and 10 keypoints, respectively. We evaluated the Point-wise Mesh Euclidean Distance (PMD) for SMPL meshes and Fréchet inception distance (FID) for DeformingThings4D-Animals meshes, following our paper. The results are summarized in Tables 1 and 2. Notably, the model trained with just 10 keypoints still outperforms the pretrained SPT model in the pose transfer using SMPL meshes. Please refer to Figures 3 and 4 in the PDF file. We will include this analysis in the future revision.\n\n**Table 1: PMD measured on SMPL pose transfer experiments with varying number of keypoints. \"Ours-$N$\" denotes a variant of our network trained to extract $N$ keypoints.**\n\n| Method      | SPT | Ours-10 | Ours-25 | Ours-50 | **Ours-100** |\n|------------- |-----|---------|---------|---------|--------------|\n| PMD ($\\times 10^{-3}$)| 0.28 | 0.20    | 0.17    | 0.17    | **0.13**     |\n\n**Table 2: FID measured on meshes whose poses are transferred from class **bear3EP** of the **DeformThings** dataset. \"Ours-$N$\" denotes a variant of our network trained to extract $N$ keypoints.**\n\n| Method   | Ours-10 | Ours-25 | Ours-50 | **Ours-100** |\n|----------|---------|---------|---------|--------------|\n| FID ($\\times10^{-2}$) | 1.25    | 0.87    | 0.83    | **0.72**     |\n\n**Clarification on NJF.** In our experiments, we followed the setup of the morphing humans experiment in [Aigerman et al., ACM ToG 2022] where an MLP takes PointNet latents encoded from both source and target shapes and predicts a Jacobian field of the source shape that morphs it to the target shape. We did not use the setup of the re-posing humans experiment as our work does not rely on parameterizations, such as skeletons. We will clarify this in the revision.\n\n**Clarification on FID Computation.** Unlike 2D images and videos with abundant reference data collected over the Internet, assessing the realism of 3D shapes remains challenging as there is no standardized way. We followed previous work including MeshDiffusion [Liu et al., ICLR 2023], 3DShape2VecSet [Zhang et al., ACM ToG 2023], MeshGPT [Siddiqui et al., CVPR 2024], and Make-A-Shape [Hui et al., ICML 2024] that compute FID using images rendered from multiple viewpoints without texture. Specifically, we rendered images from 4 viewpoints (front, back, left, and right) at zero elevation.\nFollowing MeshDiffusion [Liu et al., ICLR 2023], we applied a grayscale diffuse material for reference and output shapes and applied Phong shading to obtain images with depth cues. We will include these details of the evaluation settings in the revised version.\n\n**Template Meshes for Animals.** For quadrupeds, used in our experiments and comprising a large portion of animal species, we consider shapes standing with straight legs as the animal equivalent of the T-pose for humanoids. This convention has been used in several works, including SMAL [Zuffi et al., CVPR 2017], A-CSM [Kulkarni et al., CVPR 2020], BARC [Ruegg et al., CVPR 2022], BITE [Ruegg et al., CVPR 2023], VAREN [Zuffi et al., CVPR 2024], and 3D Fauna [Li et al., CVPR 2024]. Still, there may be cases where obtaining, or even defining a canonical pose of an object is challenging. Therefore, lifting the necessity of template shapes is one of the directions that we are heading to.\n\n**Limitations**\nFollowing NJF [Aigerman et al., ACM ToG 2022], our method uses differential operators to compute Jacobian fields. Therefore, it may not be directly applicable to meshes with defects (e.g., duplicate vertices) or multiple connected components.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5zbwqjyKiF", "reviewer": "Reviewer_xBSu", "summary": "This paper presents a novel representation learning framework for pose estimation which is disentangled from the identity of the object. This implicit representation is used for generating and transferring poses using a cascade diffusion model. A keypoint-based hybrid pose representation with a sparse mesh is used to ensure it is compact enough for the generative model. Moreover, the authors use Jacobian fields as the shape representation, allowing their implicit deformation network to take mesh face coordinates as input as opposed to vertex coordinates. The latent representations learned from this are used in a self-supervised per-identity refinement module for further improvement. This allows the proposed network to transfer motion sequences from one object to another. Moreover, these representations can be used to animate previously unseen characters. The authors provide extensive experiments with both quantitative and qualitative ablations, showing the superiority of their approach over three baseline networks, using both animals and humanoid objects from different datasets.", "questions": "-The results look very promising, and  I am curious how this proposed method can be transferred to images or videos (with textures and backgrounds) insead of only mesh generation. Do you have any insights about how this will work and how your method can be adapted for this scenario?", "limitations": "The limitations of Jacobian fields are discussed in the conclusion", "rating": 9, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper is very well-written with clear descriptions of motivation, research question, methodology, mathematical formulations, and experiments.\n- There are extensive experimental results to establish the superiority of the proposed method. In particular, the qualitative results clearly help visualize the improvements.\n- It is very easy to read", "weaknesses": "- No weaknesses", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer xBSu,\n\nWe highly appreciate your positive comments on our work, particularly regarding the writing and experiment procedures. Our response to your question can be found below.\n\n**Extension to Images and Videos.** We strongly agree that our work can be extended to other modalities, such as images and videos. For instance, the proposed pose representation and our cascaded diffusion model can be utilized as a prior for 3D-aware image editing tasks where estimating motion trajectories and handling occlusions are crucial. Specifically, we believe our method can be adapted to recent drag-based image editing techniques, such as DragGAN [Pan et al., SIGGRAPH 2023] and DragDiffusion [Shi et al., CVPR 2024]. When a user inputs a set of drag-based instructions describing complex articulations of animals that cannot be resolved in 2D, we may consider lifting the object to 3D by using a single-view 3D reconstruction model (e.g., Zero-1-to-3 [Liu et al., ICCV 2023]). The coarse 3D delegate of the image can then be given to our model as an input, which extracts a pose representation from the shape. As the shape is edited following the instructions, our diffusion model, trained on a collection of pose representations, can provide guidance signals via Score Distillation Sampling [Poole et al., ICLR 2023] to ensure the resulting deformation remains realistic while faithfully reflecting the user's input. Nonetheless, this is one possible scenario where our method can be applied, and we hope to extend it to a variety of applications in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mHWElRpAuF", "reviewer": "Reviewer_RKb2", "summary": "The authors propose a skeleton-free pipeline for implicit 3D pose representation and transfer. They introduce the prediction of Jacobian fields to achieve shape-preserving representation, which facilitates the application of transferred poses. To enhance accuracy, a per-identity refinement step is included, utilizing intrinsic-preserving loss. Additionally, a cascaded diffusion model is employed on the compact pose representation to generate novel poses.", "questions": "1.The proposed method may face limitations when animating non-T-pose meshes if it relies heavily on T-pose assumptions for pose estimation and transfer.\n2.It would be beneficial to demonstrate pose transfer results for both source and target meshes from the Mixamo dataset and compare these with results from the ECCV 2022 paper \"Skeleton-free Pose Transfer for Stylized 3D Characters\" to assess performance and efficacy.\n3.Comparing the training and inference times with baseline methods would provide insights into the computational efficiency of the proposed approach.", "limitations": "Yes, they are discussed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.The use of Jacobian fields for shape representation is innovative.\n2.The method of incorporating an intermediate step for intrinsic preservation is justified.\n3.Training a diffusion model on extracted implicit poses further demonstrates the effectiveness of the pose representation.", "weaknesses": "1.FPS is utilized to sample keypoints as a basis for pose representation. However, for complex poses, FPS-sampled keypoints may not be evenly distributed on the surface due to a lack of awareness of mesh connectivity, potentially resulting in poor pose representation.\n2.Given that the pose is represented by keypoints' coordinates and corresponding latent features, the shape information is inherently entangled in the keypoints' coordinates. This could lead to the shape information being encoded in the latent features.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer RKb2,\n\nThank you for your positive comments. We are especially grateful for your recognition of the use of Jacobian fields as \"innovative\" and for acknowledging that our approach is well \"justified\". We provide our responses to your queries below.\n\n**Impact of the Sampling Method and Number of Keypoints.**\nWe appreciate your suggestion for an in-depth analysis. We used FPS for keypoint extraction due to its simplicity and efficacy, which have made it a popular choice in geometry analysis (e.g., PointNet++ [Qi et al., NeurIPS 2017], KeypointDeformer [Jakab et al., CVPR 2021], DeepMetaHandles [Liu et al., CVPR 2021]). Here, we investigate the impact of the number of keypoints and their sparsity. We trained our model using SMPL and DeformingThings-Animals meshes, adjusting the pose extractor to extract 50, 25, and 10 keypoints, respectively. We evaluated the Point-wise Mesh Euclidean Distance (PMD) for SMPL meshes and Fréchet inception distance (FID) for DeformingThings4D-Animals meshes, following our paper. The results are summarized in Tables 1 and 2. Notably, the model trained with just 10 keypoints still outperforms the pretrained SPT model in the pose transfer using SMPL meshes. Please refer to Figures 3 and 4 in the PDF file. We will include this analysis in the future revision.\n\n**Table 1: PMD measured on SMPL pose transfer experiments with varying number of keypoints. \"Ours-$N$\" denotes a variant of our network trained to extract $N$ keypoints.**\n\n| Method      | SPT | Ours-10 | Ours-25 | Ours-50 | **Ours-100** |\n|------------- |-----|---------|---------|---------|--------------|\n| PMD ($\\times 10^{-3}$)| 0.28 | 0.20    | 0.17    | 0.17    | **0.13**     |\n\n**Table 2: FID measured on meshes whose poses are transferred from class **bear3EP** of the **DeformThings** dataset. \"Ours-$N$\" denotes a variant of our network trained to extract $N$ keypoints.**\n\n| Method   | Ours-10 | Ours-25 | Ours-50 | **Ours-100** |\n|----------|---------|---------|---------|--------------|\n| FID ($\\times10^{-2}$) | 1.25    | 0.87    | 0.83    | **0.72**     |\n\n**Potential Drawback of Representing Poses as Keypoints.** To prevent the leakage of source shape details into target shapes, we use Jacobian fields instead of vertex coordinates when extracting our pose representation. This way, our network predicts per-triangle *local transformations* to deform the given template to match the pose example. Furthermore, training the refinement module by optimizing intrinsic-preservation losses helps preserve intricate local details. Please refer to Figure 5 in the attached PDF file for qualitative results of the ablation study. We will include more results in the revised version.\n\n**Assumption on T-Poses.** Thank you for pointing this out. It is true that our method requires a canonical pose for an identity (e.g., T-pose for humanoids). However, we believe our method remains practical for its intended purpose since default or canonical poses of deformable objects have been widely used not only for humanoids but also for animal shapes, including in works like SMAL [Zuffi et al., CVPR 2017], A-CSM [Kulkarni et al., CVPR 2020], BARC [Ruegg et al., CVPR 2022], BITE [Ruegg et al., CVPR 2023], VAREN [Zuffi et al., CVPR 2024], and 3D Fauna [Li et al., CVPR 2024]. These studies define a canonical pose as a shape standing still with its legs straight for quadrupeds. Nonetheless, there may be cases where obtaining, or even defining a canonical pose of an object is challenging. As such, lifting the necessity of template shapes is one of the directions that we are heading to.\n\n**Pose Transfer between Mixamo Meshes.** We appreciate your suggestion. We downloaded 28 motions in the test split of SPT [Liao et al., ECCV 2022] from the Mixamo repository, comprising a total of 3,025 frames. We followed the preprocessing steps of SPT to extract meshes from skeleton configurations. Since the pretrained SPT model was trained on a larger dataset consisting of shapes from AMASS, Mixamo, and RigNet, we re-trained the model to analyze its performance in the same problem setup as ours. For training, we used 3,025 different poses of a single character. After training, we used 300 poses transferred to 8 different characters to compute Point-wise Mesh Euclidean Distance (PMD) by leveraging the ground-truth correspondences. We report the results in Table 3. Please refer to Figure 6 in the attached PDF file for qualitative results. As illustrated, our method better retains the smoothness and details of the surface even when transferring poses involving articulations of limbs. This is also reflected in lower PMD in Table 3. We will add these results in the revision.\n\n**Table 3: PMD measured on Mixamo pose transfer experiments.**\n| Method       | SPT  | **Ours** |\n|--------------|------|----------|\n| PMD ($\\times10^{-3}$) | 3.42 | **2.28** |\n\n**Training \\& Inference Time Comparisons.** We summarize the training and inference time in Table 4. The inference time includes the time to transfer a pose of one identity to another via network forward passes. We used Mixamo meshes with approximately 10K vertices and 20K faces for evaluation. Note that for SPT, we measured the inference time using the official pretrained model. Our method demonstrates better runtime performance than SPT while outperforming NJF and ZPT in terms of pose transfer accuracy. We will include this analysis in the revised version of our paper.\n\n**Table 4: Training and inference time comparison**\n| Method            | NJF  | SPT  | ZPT  | **Ours** |\n|-------------------|------|------|------|----------|\n| Training          | 2h   | 20h  | 9h   | 8h       |\n| Inference (per pair) | 0.005s | 1s   | 0.004s | 0.03s    |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for taking the time to review our submission and for providing constructive and insightful comments and feedback. We have compiled the qualitative results discussed in our rebuttal in the attached PDF file.

Meta-review: This paper is a clear accept. It receives all positive ratings of 1 very strong accept, 1 weak accept and 1 borderline accept. The reviewers agree that the paper studies a novel and important task, the proposed method is sound and effective with clear and well-justified mathematical formulations. The first two reviewers are satisfied with the authors' rebuttal, and the last reviewer asked for some clarifications but did not point out any unsatisfaction. The authors should incorporate all the comments in the review and rebuttal into the final paper.
Decision: Accept (poster)