RigAnyFace: Scaling Neural Facial Mesh Auto-Rigging with Unlabeled Data
Authors: Wenchao Ma
Dario Michael Kneubuehler
Maurice Chu
Ian Sachs
Haomiao Jiang
Sharon X Huang
Venue: NeurIPS 2025 poster ()
Keywords: Creative AI
Auto-rigging
Facial Animation
Primary area: applications
TL;DR: We propose a scalable neural auto-rigging framework for facial meshes of diverse topologies with multiple disconnected components.

Abstract: In this paper, we present RigAnyFace (RAF), a scalable neural auto-rigging framework for facial meshes of diverse topologies, including those with multiple disconnected components. RAF deforms a static neutral facial mesh into industry-standard FACS poses to form an expressive blendshape rig. Deformations are predicted by a triangulation-agnostic surface learning network augmented with our tailored architecture design to condition on FACS parameters and efficiently process disconnected components. For training, we curated a dataset of facial meshes, with a subset meticulously rigged by professional artists to serve as accurate 3D ground truth for deformation supervision. Due to the high cost of manual rigging, this subset is limited in size, constraining the generalization ability of models trained exclusively on it. To address this, we design a 2D supervision strategy for unlabeled neutral meshes without rigs. This strategy increases data diversity and allows for scaled training, thereby enhancing the generalization ability of models trained on this augmented data. Extensive experiments demonstrate that RAF is able to rig meshes of diverse topologies on not only our artist-crafted assets but also in-the-wild samples, outperforming previous works in accuracy and generalizability. Moreover, our method advances beyond prior work by supporting multiple disconnected components, such as eyeballs, for more detailed expression animation.

RagAnyFace: Scaling Neural Facial Mesh Auto-Rigging with Unlabeled Data
Wenchao Ma!
Dario Kneubuehler
Maurice Chu Sharon X. Huang!
Ian Sachs²
Haomiao Jiang" Penn State University Roblox
Mariere
Tedaadd Mean
Jan Drop
Jan Diaa
LoVt Bye Clisa
MOLER Purnet
Comeer Paler
Lyk Chose
Up Prosser
Epe
Bys Look Down
Eaptcity Conistati houmssrrs
Neatrel Flndul Blvskes
Pare Defermation Veete
PACE Shrnhhleges
Figure I: We present RigAnyFace (RAF), an auto-rigging framework that supports facial meshes of diverse topologies with multiple disconnected components such eyeballs. These meshes drawn from diverse sources and cover both humanoid and non-humanoid heads. Given only a neutral facial mesh and explicitly controllable FACS parameters specifying activated action units, RAF accurately deforms the input mesh into corresponding FACS poses, creating an expressive blendshape rig.
Abstract
In this paper, present RigAnyFace (RAF), scalable neural auto-rigging frame- work for facial meshes of diverse topologies, including those with multiple discon- nected components. RAF deforms static neutral facial mesh into andustry-standaan FACS poses to form an expressive blendshape rig. Deformations are predicted by trrangulattonnagnn aagnotiiiinnniiiccicciooo surface learning network augmented with our tailored architecture design to condition on FACS parameters and efficiently process dis- connected components. For training, curated dataset of facial meshes, with subset meticulously rigged by professional artists to serve as accurate 3D ground truth for deformation supervision. Due to the high cost of manual rigging, this subset is limited in size, constraining the generalization ability of models trained exclusively
To address this, design 2D supervision strategy for un- labeled neutral meshes without rigs. This strategy increases data diversity and allows for scaled training, thereby enhancing the generalization ability of models trained on this augmented data. Extensive experiments demonstrate that RAF is able rig meshes of diverse topologies on only our artist-crafted assets but also in-the-wild samples, outperforming previous works in accuracy and generaliz- ability. Moreover, our method advances beyond prior work by supporting multiple disconnected components, such as eyeballs, for more detailed expression animation. Project page: https://wenchao-m. http:://wenchoom.iiiuubb.iii
githubbo
"Equal Contribution.
Work partially completed during internship at Roblox.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

1 Introduction
Facial rigging aims to make a static neutral facial mesh animatable by defining set of controllable deformations, typically represented either as blendshape rigs driven by activated action units in FACS-based systems [49, 31, 32, 54, 8, 7, 50] or as skeletal rigs driven by joint positions [64, 27, 48). This is an essential step for creative AI, bringing digital avatars to life by enabling expressive and realistic facial movements across a wide range of applications. However, creating rig for facial animation is laborious and expensive, often requiring skilled artists tens of hours to complete a single asset. In this paper, we propose a fully automated and generalizable facial rigging framework that alleviates the reliance on manual labor while achieving high-quality facial rigging.
Typically, facial auto-rigging methods transfer a complete set of blendshapes from a predefined template mesh to a neutral target facial mesh, often necessitating dense correspondences [47,58, 33, 11] or fixed mesh topology between the template and the target [34, 6]. Recent approaches [65, 12] utilize per-face VQ-VAEs [65] to build transferable latent spaces between faces or triangulation- agnostic networks [12] to bypass these limitations. However, a template blendshape rig is still required, which can compromise accuracy when the template and target shapes differ substantially. NFR [50] is currently the only approach capable of directly rigging facial meshes from explicitly controllable Facial Action Coding System (FACS) [49] parameters without relying on template, although has so far been demonstrated primarily humanoid heads. Furthermore, existing approaches, including NFR, have yet to accommodate meshes with multiple disconnected components, such as eyeballs or teeth, limiting their ability to animate highly expressive avatars; for example, an "eye lookdown" pose is difficult to reproduce if the mesh lacks eyeballs.
To address the above challenges, we aim to build facial auto-rigging framework with the following advantages: (i) it eliminates the reliance on predefined template blendshapes, removing the constraint that target facial meshes must rigorously resemble predefined template: (ii) it is capable of animating in-the-wild facial meshes with varying topologies and shapes, including humanoid and oon-humadoid samples as shown in Fig I; and (iii) it supports facial meshes with multiple disconnected components to enable realistic and expressive 3D face animations.
We present RigAnyFace (RAF), scalable and generalizable framework for facial auto-rigging, RAF employs a facial mesh deformation network built on DiffusionNet [55], a triangulation-agnostii backbone for meshes of different topologies. Guided by explicitly controllable FACS parameters, this network deforms a neutral facial mesh into a predefined set of FACS poses to form a blendshape rig. Compared to the original DiffusionNet, we introduce two key modifications: (i) a conditional diffusion block that extends the original diffusion block to incorporate FACS parameters as addi- tional conditional inputs, and (ii) global encoder designed to capture holistic mesh characteristics, enabling effective handling of multiple disconnected components. For network training, we curated comprehensive dataset of facial meshes encompassing wide variety of shapes with detailed disconnected components such as eyeballs and teeth. A subset of these meshes was meticulously rigged by professional artists to provide accurate ground-truth for 3D deformations.
Relying solely on rigged heads for training limits the model's 's generalizability in practice, given the scarcity of rigged samples due to the high cost of manual rigging. This motivates us to employ 2D supervision, which offers better accessibility and broader scalability compared to 3D supervision. We developed a 2D supervision strategy for 3D facial mesh deformation models, integrating appearance guidance from RGB images for prominent facial expressions and motion guidance from optical flow-like 2D displacement field for subtle nicro-expresssios. Supported by a generative 2D face animation model that synthesizes posed images from the renderings of neutral mesh, along with an optical flow estimator that predicts the 2D displacement between neutral and posed images as 2D supervisions, we expand the training dataset using unlabeled neutral meshes without rigs. This enables the network to effectively distill rigging knowledge across diverse facial shapes, resulting in more accurate and generalizable 3D facial animations even with limited labeled training data.
Experiments show that our method outperforms prior work across assets from diverse sources, in- cluding our artist-crafted meshes and in-the-wild models from ICT Face [36], Objaverse [15], and CGTrader [9]. In addition, we demonstrate several downstream applications of our auto-rigging sys- tem in user-controlled animation, retargeting human expressions from videos, and rigging generated facial meshes from text-to-3D model.
2

---

2 Related Works
Auto-rigging, Auto-rigging facilitates efficient and realistic animation of 3D models by automatically generating hierarchical control systems. For full-body character auto-rigging [3, 25, 28, 35,40, 42, 44, 68, 23, 57, 39, 70, 18], most approaches follow a two-step pipeline: skeleton construction and skinning to generate the Linear Blend Skinning (LBS) rig. In contrast, facial character rigs are often paatomically-inspire standardized framework that describes facial movements as combinations of muscle activations and is primarily implemented using blendshapes [31, 32, 54, 8].
typically based on the Facial Action Coding System (FACS) [49],
Previous facial auto-rigging works [47, 58, 33, 11, 34, 6, 65, 12] mostly based complete set of blendshapes from a predefined template mesh, transferring the template blendshapes to the target mesh. For example, Li et al.[34] proposed a CNN-based approach that predicts offisets between template and target blendshapes represented by 2D geometry images. Chandran et al. [12] use transformer with positional encodings to map meshes into canonical space from user-marked correspondences, enabling deformation transfer from template to target across different topologies. Several notable works [1, 10. 71, 5, 37, 52, 26] can directly generate animatable 3D faces based on 3D Morphable Models [4]. NFR [50] is able to deform neutral facial mesh into target expressions by decoding FACS-aligned latent codes from mesh auto-encoder, eliminating any template requirement. Its laagulatiionagnosee
backbone, trained on several face-animation datasets, generalizes to in-the- wild meshes with diverse topologies. Compared with NFR and other previous works, our method enables 2D supervision for scaled training and further improves accuracy and generalizability across a wider variety of facial meshes while natively supporting multiple disconnected components to allow finer-grained and more realistic expression animation.
Facial Animation Transfer from 2D. Facial animation transfer aims retarget facial expressions from one character to another. Recent methods (e.g., [51, 2, 13, 30, 29, 45]) show impressive results in transferring expressions to 3D avatars from 2D images or videos. However, these methods focus on transferring expressions to avatars that already have a rig and are hence not directly comparable to our work, which focuses on automatically generating rigs for facial meshes.
Significant progress has also been made in transferring facial animation for both single-view and multi-view images and videos [22, 66, 67, 13, 63, 38, 16, 17, 61]. Given reference identity image, these methods can generate and manipulate facial expressions for the given identity using various control inputs, such as posed images of other identities or landmarks. Recent advancements in generative models [24, 72] and the availability of large-scale face video datasets [46, 73] have enabled those methods to achieve remarkable success in 2D facial expression animation. For instance, MegActor[699 utilizes a diffusion- uuffusion-bbsed generative framework, incorporating a motion disentanglement module separate identity and expression features, and a motion retargeting model to map expressions onto target portraits. In this work, we utilize 2D face animation models to generate 2D supervision for unrigged heads. Our proposed framework is agnostic to the choice of 2D face animation model, provided they deliver satisfactory animation results. In practice, we base our 2D supervision generation on MegActor [69], which is open-source and efficient to fine-tune.
3 Preliminary
3.1 DiffusionNet
DiffusionNet [55], proposed by Sharp et al., is a neural network that learns on 3D surfaces by mimicking the intrinsic heat diffusion process. diffuses per-vertex features across the surface Beltrami operator, which captures the intrinsic geometry of the manifold. The resulting heat operator acts as a geometry-award smoothing filter that blends nearby features over time. In discrete form, DiffusionNet approximates this process using the cotangent Laplacian L and mass matrix M, defined
based on the Laplace-Beltrami
he(wo) = (M + +L) M uo.
where (ua) represents the diffused features after time I, followed by lightweight MLPs for non-linearity. Because diffusion depends only on surface intrinsic geometry, the same learned weights transfer across meshes with different resolutions or triangulation, making the model compact, discretization-agnostt and effective for tasks such as classification and regression on geometric data.
3

---

suage
Eatimaton Nodal
Right Crve
Pight Ors Dowe
Jan Drop Lofk
20 Au mermttio
Oonarmad
0.00
0.25
DOAN
Oree Rigped lead
Trainai
Geansler
(8) Artis--cratted Facial Meeh Datasel
(b) Supervizion Generation
Figure 2: (a) Illustration of our artist-crafted facial mesh dataset. (i) Neutral head meshes from our dataset, each consisting of multiple disconnected components. (ii) A subset of neutral head meshes meticulously annotated with blendshape rigs by professional artists. (iii) To augment the dataset, develop head interpolation strategy based on standardized UV layouts. (b) 2D Supervision Generation Pipeline: Given a posed image rendered from a rigged head and neutral image from an unrigged head, the 2D animation model generates image that replicates the expression in the posed image while preserving the identity of the neutral image. A flow estimation model then applied to the neutral and generated posed images predict the pixel offsets 2D displacement.
3.2 Linear FACS Blendshape Rig
The linear FACS blendshape rig [31] models an animatable 3D face using neutral mesh Mo (Vo, F), where V, represents the vertex positions and F the mesh connectivity. It also defines of N blendshapes (M, (V,, )) each obtained by adding a vertex offset dç to the neutral mesh: V Vo + d, Each blendshape corresponds to an Action Unit (AU) from the Facial Action Coding System (FACS) [20], representing specific muscle movements such as "Right Eye Close." " Complex facial expression animation, involving the activation of multiple action units, achieved by assigning a weight E [0, 1] to each blendshape and computing the final mesh M (V, F), where V Vo E w,d,,
Method
4.1 Data Collection
We collect diverse set of artist-crafted facial meshes for model training and evaluation. As shown in Fig. (a)(i,, dataset includes facial meshes with multiple disconnected components, such as separate eyeballs and features variety of shapes, including both humanoid and non-humanoid heads.
Each dataset sample contains a neutral base mesh Mo- For a select subset, artists manually annotated each mesh with its own complete blendshape rig [M, = (V,, F across FACS training poses, described in Sec. 3.2 and illustrated in Fig. 2 (i). We set N 96, comprising 48 FACS poses and 48 corrective poses; further details are provided in the appendix. We also pair each blendshape with onc-hot-like FACS vector A; as pose representation, where activated action entries are set to 1. Furthermore, those heads were also annotated with facial landmarks specified as vertex indices. For unlabeled heads, only neutral head mesh Mo (Va, F) is included.
Creating head meshes with complex rigs for animation an expensive process. In order to expand our dataset sufficiently for training deep neural network, we developed data augmentation strategy based on standardized UV layout, enabling interpolate between different head geometries through linear blending to increases the size of our dataset, illustrated in Fig. (a)(iii.

---

Glabal Greeder
Tealuse Vap
Diplicement
Gon dilioon
di
Block
Randering
Nx
Displaceeent
ROSA
Fackal Mesh Deformation Model
Conditinaal Dificiion
Suansal
Graddennt
Spetial
Defiuiion Offhnian -LP-eeie
Model
Figure 3: Model Architecture. (a) Given neutral facial mesh, our deformation model predicts the 3D displacement needed deform the mesh into different expressions based on the input FACS vector. During training, 2D supervision is utilized for both rigged and unrigged heads, while 3D supervision exclusively applied to rigged heads. (b) We modify the original diffusion block in DiffusionNet to support the FACS vector additional conditional inputs (left). Additionally, we design global encoder that processes vertex positions and normals the neutral facial mesh to capture holistic information across disconnected components (right).
4.2 Deformation network
4.2.1 Network Architecture
As shown in Fig. 3 (a), our deformation network takes the neutral facial mesh M, = (Vo F) and a FACS pose vector A; as inputs and predicts the displacement d, required to deform the neutral mesh into the corresponding posed mesh M, (V;. F), where Vi = Vo d,. The posed meshes obtained for all FACS poses together form a linear FACS blendshape rig.
We build our deformation network upon DiffusionNet [55] to take advantage of its triangulation- agonistic property. However, DiffusionNet struggles to handle multiple disconnected components as its diffusion mechanism cannot propagate information between them. Furthermore, it is limited processing single mesh without additional input. In our task, we aim to deform facial meshes with multiple disconnected components conditioned on an additional input: the FACS vector. To this end, we introduce two key modifications to the original DiffusionNet: (i) Global Encoder to capture holistic mesh characteristics across multiple disconnected components. As shown in the right of Fig. 3 (b), this branch consists of smaller 2-layer DiffusionNet that processes the input neutral mesh. Global average pooling is applied to the final layer's per-vertex features, producing single vector encoding Go that compresses information about the mesh into a global feature vector. (ii) FACS Conditioning: We modify the original diffusion block in DiffusionNet to integrate a FACS pose vector as a conditional input, guiding the network's generation network to learn the relationship between FACS values and corresponding mesh deformations. As shown on the left of Fig. (b), the FACS pose vector A, is concatenated with the global feature vector Go create latent representation. This latent representation is then injected into each conditional diffusion block of the main network. Within each block, the latent vector is replicated across the vertex dimension and fused with the block's output features. This fused information is then processed by a small MLP to refine the mesh's 's latent features.
facial expressions. This allows the
4.2.2 2D Supervisions for 3D Deformation Model
Relying solely on fully rigged heads limits the training dataset size due to the scarcity of high-quality 3D ground truth, which hampers generalization to unseen facial meshes. In contrast, 2D supervision more readily available thanks to advancements in 2D generation models, enabling the inclusion
5

---

14) Neutral Image
(bi Posed Irage (Jan Lelk) (6) (tee Color Deforence (d) Digiecement Fade
Figure 4: Illustration of our 2D displacement supervision (d), which provides denser feedback for the subtle pose differences between (a) and (b) than the appearance- supervision (c). Subfigure (c) visualizes per-pixel color-difference magnitudes between (a) and (b), whereas subfigure (d) shows the corresponding pixel offsets using the standard optical-flow color map.
of unrigged heads to scale up the training dataset to enhance generalization. Thus, we introduce 2D supervision for the face auto-rigging network in terms of appearance and motion variation. Specifically, for appearance, we use the front-view image and binary segmentation mask of the posed head as supervision. We render the RGB image 1 and binary mask B; of the predicted mesh M; onto the 2D image plane using differentiable rendering [41, 53J. The image loss Limg and mask loss Lmask are defined as the (] distances between i, with the ground truth image 1, and between B; with the ground truth mask B, respectively.
Using appearance- supervisions like image and mask losses, provides straightforward way to optimize the 3D deformation network using 2D supervision. These losses offer strong supervisory signals for poses that result in significant changes in pixel's color value. However, many target FACS poses involve subtle expressions, where changes are less visually apparent. For instance, as shown in Fig. 4, comparing the neutral image in Fig. 4 (a) with the jaw-left pose image in Fig. 4 (b), the differences are barely noticeable to the human eye. Similarly, as illustrated in Fig. (c), the pixel error map RGB value between these two images highlights that only small portion of pixels contribute meaningful supervisory feedback for these subtle deformations. In other words, the magnitude of the loss remains minimal if the deformation model leaves all vertices fixed in the neutral expression.
To address this challenge, we introduce another 2D supervision for the 3D deformation model based on pixel motions. Specifically, we define the 2D displacement d?dd as the offset of each pixel on the image plane between the neutral and posed images, analogous to optical flow. This 2D displacement is computed from the 3D displacement de fully differentiable manner with differentiable rendering [3] (see appendix for implementation). As shown in Fig. 4 (d), the 2D displacement is more distinguishable for subtle facial expressions because explicitly represents the motion of each pixel in 2D, rather than relying on RGB value changes. This is particularly beneficial in areas with uniform texture, such as check, where RGB value changes may be unnoticeable. We define the 2D displacement loss Cdis 2a the (2 distance between the ground truth 2D displacement dad and predicted 2D displacement ddd.
4.2.3 2D Supervision Generation
For all rigged heads, we can obtain the above 2D supervisions by rendering from 3D ground truth. However, for unrigged heads, this is not feasible due to the absence of complete 3D ground truth deformations. To this end, we leverage recent advancements in 2D generation models to generate 2D supervision for unrigged heads. These models effectively distill appearance and motion priors from large-scale 2D image and video datasets, generalizing well across diverse scenarios.
We implement 2D face animation diffusion model based on Megactor [69]. As illustrated in Fig.2 (b), this model takes a neutral reference image rendered from an unrigged head and driving posed image rendered from rigged head, animating the neutral image to replicate the expression in the posed image while preserving its identity. The generated images serve as image-based ground truth for unrigged heads during the training of the 3D deformation model. In practice, we select one rigged head, render all its FACS poses images, and use them as driving images to generate corresponding posed images for all unrigged heads. Ground truth masks are obtained using a traditional image segmentation [59] as all generated images are with a clean white background.

---

For the 2D displacement, we use the optical flow estimation model RAFT [60] to predict pixel offsets between the neutral image and the generated posed image of unrigged heads. These offsets serve as the ground truth 21) displacement for training the 3D deformation model.
To enhance the performance of the 2D face animation and flow estimation models stylized faces in our artist-crafted dataset, we fine-tune their pre-trained weights using the ground truth renderings from small of rigged heads, improving effectiveness.
4.2.4 Network Training and Inference
We train the network in a two-stage, coarse-to to-fiee manner. In the first stage, the 3D deformation network trained on large-scale dataset comprising both rigged and unrigged heads, using only 2D supervision. We use combination of photometric loss and 2D displacement loss, along with l2 regularization loss, Lreg on the predicted 3D displacement. This regularization loss helps to improve model convergence speed and prevent "flying points" for non-line nfn-line-of-sight vertices. The total training loss for the first stage is defined as:
L.1 = a Ling + a2Lmask azLdis 2d, +rg
where a are weighting parameters for different loss terms.
In the second stage, we fine-tune the pretrained model from the first stage using only rigged heads, incorporating both 2D and 3D supervision to achieve high-precision deformation predictions. Since the 3D ground truth deformed mesh M, = (V,, F) for FACS pose is available for rigged heads, we incorporate 3D supervision by applying the MSE loss Lmse-3d d in 3D space between the ground truth and predicted mesh vertices V4 and Vg.
For 2D supervision, in addition to the image loss and mask loss, we added two loss terms, landmark loss Limk and eye close loss Lec, in [21] to provide supervision for specific facial landmarks and poses. We omit the 2D displacement loss this stage since the 3D displacement ground truth is available. The total training loss for the second stage is defined as:
L.2 ai Cimg + a2Cmaak + a3Lmee-3a' + aqLimn agCec.
(3)
The proposed model only consists of 5.4M parameters. Training runs on an instance with 8 NVIDIA A100 GPUs and takes about 2 days. For inference, takes on average 8.72s on an Apple M2 Max CPU and 3. on an Nvidia T4 GPU to generate a FACS blendshape rig on the test set.
5
Experiments
In this section, we evaluate RAF on both the artist-crafted and in- in-the-wild facial meshes and compare it with the prior art NFR [50] and a representative cnoematiion-tansffr method [58].
5.1 Evaluation Artist-craftd Data
Table Quantitative results on our artist-crafted dataset, validating each component of the model.
Table 2: Quantitative comparison with NFR and Deformation Transfer on 12 artist-annotated hu- manoid heads. (* additional inputs needed)
Spree
MAF (mm)
MAF Q95 (mm)
3.0
Deformation Transfer [58]
2.93
8.41
[50] Ours
1.01
2.94
We evaluate our model both quantitatively and qualitatively on our artist-crafted dataset. The evalua- tion includes two test sets: with rigged heads for detailed accuracy analysis, and another with unrigged heads featuring diverse species and shapes to assess generalization on diitt--iiittitttio samples, simulating real-world applications.
For rigged heads with 3D ground-trufe deformations, we compute the Mean Absolute vertex Error (MAE) and the 95th Stth-eercentie vertex error (MAE Q95) to capture challenging cases; both metrics are evaluated over the full set of FACS poses. During evaluation, all facial meshes are normalized to fit within unit sphere with radius of I meter. Quantitative results are presented in Tab. 1, while
7

---

Depfacement
WiSh Stubal Erenderr
wes Lonees
with Unrigge/
2D Skylommend
Refarence
Celomation
Tha
NPR
Care
(a) Ablation Framework Components
(b) Comparison with Baseline Methods
Figure 5: Visual comparisons. Meshes are colored by per-vertex absolute error. (a) Ablation on Framework Components. 1st Col.: without the global encoder, disconnected parts intersect; 2nd Col.: 2D image loss reduces errors; 3rd Col.: additional unrigged heads improve generalization, addressing challenging cases such as animal eye closure; 4th Col.: 2D-displacement loss further refines subtle poses such as "Jaw Left." (b) Comparison with Baseline Methods. Our method achieves more accurate and expressive animation results while handling multiple disconnected components, Reference mesh and corresponding points (marked as blue) are provided for Deformation Transfer.
qualitative results are shown in Fig. 5(a). Together, these results validate the effectiveness of each component in our model.
We also conduct additional ablations on the global encoder to demonstrate how a single feature vector from it enables model to handle multiple disconnected components. We evaluate penetration between inner components (c.g., teeth) and the outer face surface by reporting the percentage of penetrating vertices with and without the global feature (Tab. 3). Furthermore, we perturb the disconnected components by randomly adding offsets or removing varying numbers of components from each sample. As shown Fig. 6, the t-SNE visualization of global features from the perturbed samples forms separate clusters from the original ones. These results demonstrate that the global feature effectively encodes both the position and presence of disconnected components, avoiding penetration and achieving accurate deformation.
Table 3: Ablation on the global encoder.
Global Encoder
All Other Components
MAE
MAE (96
Penetration
3.16
5.66
0-105
2.8
5.54
8.166
1.92
563
0L13
Figure 6: t-SNE visualization of fcatures from the global encoder.
To ensure a fair comparison with NFR [50], each input mesh was preprocessed following the same procedure in their original implementation, retain only the largest connected component of the neutral mesh with auxiliary structures (e.g., eyeballs and the mouth socket) removed and the inner surfaces of the lips and eyelids trimmed. Since NFR is trained and evaluated only on humanoid faces, we limit the test cases to 12 humanoid heads. Another method that we compare to is the Deformation Transfer [58]. which requires an exemplar expression mesh and user-annotated point correspondences as additional input. We choose one rigged head from the training set as the exemplar, deform it into all FACS poses, and provide artist-annotated landmarks as correspondence points. As reported in Tab. and Fig. 5(b), our method outperforms both baselines by wide margin. It also has the additional advantage of not requiring any additional input and being able to handle meshes with multiple disconnected components.

---

Neutral Meeh
Left Eye Close
Mouth Funnel
Left Lip Comner Puller
NFR
Oure
NFRI
Cur
NFR
Objovrsse
Oun
NFR
Objorerse
Que
Figure 7: Auto-rigging results in tn-the-wil wild facial meshes compared with NFR [50].
For the unrigged head test set, we provide qualitative results only, as 3D ground truth data unavailable. Fig. 8 showcases qualitative examples, where model delivers highly accurate and vivid auto-rigging results across facial meshes various shapes and styles. Additional results for all FACS poses and more samples can be found in the supplementary materials.
Noutral
Jow Drop
Chin Up Rabe
Mouth Pannel
Left Eye Close Lefi Chock Rase Eye Look Dowe
Eye Look Left
o
Figure 8: Qualitative results on our crtist-crafted unrigged heads.
5.2 Evaluation on In-the-wild Heads
Our method generalizes effectively to in-the-wild facial meshes with diverse topology and shape variations. To demonstrate this, we present qualitative results on samples from ICT FaceKit [36], Objaverse [15, 14], and CGTrader [9], and compare that with the results by NFR [50]. We do not provide comparison with Deformation Transfer here, as correspondence point annotations are not available for these samples. We similarly preprocess input meshes for the comparison with NFR. As shown in Fig. 7, our model consistently achieves better accuracy and generalizability. In particular, although NFR was trained on the ICTFaceKit dataset and ours was not, our results are comparable to those of NFR. For humanoid assets from Objaverse and CGTrader, neither our method

---

Oigital sample
Left Eye Close
Poor Dinillatton
Felner
Nautral Disoretization Mesh
Drop
Shall-iike Mesh
Figure 9: Failure cases of our RAF model.
NFR was trained on data from these sources, model demonstrates superior performance. For the non-humanoid head (last column), NFR leaves it largely undeformed, whereas our model successfully generalizes this challenging case.
5.3 Applications
We demonstrate three real-world applications of RAF: (i) user-controlled animation, where the predicted FACS rig allows users to pose mesh by editing FACS parameters; (ii) video-to-mes retargeting, which transfers expressions of subject the video via tracked FACS sequences to an unrigged mesh; (iii) animating a facial mesh generated from text-to-3D model, turning it from neutral facial mesh into a fully animatable avatar. Demos can found on our project page.
Conclusion
We propose RAF, framework for auto-rigging facial meshes. Powered by tailored design for multiple disconnected components and FACS conditioning and scaled by training on unrigged heads with 2D supervision, RAF animate meshes of diverse topologies with even multiple disconnected components, across both artist-crafted assets and in-the-wild samples.
Limitations and Future Work. Our model faces challenges in two scenarios: (i) When the input mesh structure deviates significantly from the training data, such as shell-like meshes that lack the fine-grained geometric details necessary for high-quality facial animation, the model's performance may decline (Fig.9 9 (a)). Expanding the dataset to include a broader range of mesh structures could enhance generalization in such cases. (ii) When the mesh has poor discretization that causes the main facial mesh to break into multiple disconnected components (shown in different colors), our model fails maintain spatial coherence among these components after deformation (Fig.9 (b)). Incorporating a diffusion operator defined on a high-quality background triangulation [56] could enhance robustness in such cases.
7
Acknowledgement
We thank Hsueh-Tr Ti Derek Liu, Chrystiano Araújo, and Jinseok Bae for proofreading the draft and providing helpful comments, and Jihyun Yoon for curating the dataset. Wenchao Ma is supported by travel grant from the Penn State College of Information Sciences and Technology for attending the conference.
10

---

References
[1] Victoria Feráandez Abrevaya, Adnane Boukhayma, Stefanic Wuhrer, and Edmond Boyer. A decoupled 3d facial shape model by adversarial training. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 9418-9427. 2019.
[2] Deepali Aneja, Bindita Chaudhuri, Alex Colburn, Gary Faigin, Linda G. Shapiro, and Barbara Mones. l.caming to generate 3d stylized character expressions from humans. In 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018, Lake Tahoe, NV, USA, March 12-15, 2018, pages 160-169. IEEE Computer Society, 2018.
[3] Ilya Baran and Jovan Popovie. Automatic rigging and animation of characters, ACM Trans. Graph.. 26(3):72, 2007.
[4] Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In Warren N. Waggenspack, editor, Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1999, Los Angeles, CA, USA, August 8-13, 1999, pages 187-194. ACM, 1999.
[5] Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou. Facewarehouse: A 3d facial expression database for visual computing. IEEE Trans. Vis. Comput. Graph., 20(3):413 440)),,,,,2,,,,,, 2014.
[6] Emma Carrigan, Eduard Zell, Cédric Guiard, and Rachel McDonnell. Expression packing: As-few-as- possible training expressions for blendshape transfer. Comput. Gruph. Forum, 39(2):219 2020.
[7] Ozan Cetinaslan and Verónica Orvalho. Sketching manipulators for localized blendshape editing. Graph. Model,, 108:10 2020.
[8] Ozan Cctinaslan and Verónica Orvalho. Stabilized blendshape editing using localized jacobian transpose deseent. Graph. Model., 11 12:101191, 2020.
[9] CGTrader. Cgtrader- - 3d models for professionals, 2025.
[10] Prashanth Chandran, Derek Bradley, Markus Gross, and Thabo Beeler. Semantic deep face models. In International Conference on 3D Vision (3DV), pages 345-354. IEFF, 2020.
[11] Prashanth Chandran, Loic Ciccone, Markus Gross, and Derek Bradley, Local aaatomicallyy-constaannd facial performance retargeting. ACM Trans. Graph., 61(4):168:111688) 2022.
[12] Prashanth Chandran, Gaspard Zoss, Markus Gross, Paulo F. U. Gotardo, and Derck Bradley. Shape transformers: Topology-independer 3d shape models using transformers. Comput. Graph. Forum, 21(2):195-207,22 2022.
[13] Byungkuk Choi, Hackwang Fom, Benjamin Mouscadet, Stephen Cullingford, Wan-Duo Kurt Ma, Stefanic Gassel, Suzi Kim, Andrew Moffat, Millicent Maier, Marco Revelant, Joe Letteri, and Karan Singh. Animatomy: an animator-centric, anatomically inspired system for 3d facial modeling, animation and transfer. In SIGGRAPH Asia 2022 Conference Papers, SA, pages 16:1-16:9. ACM, 2022.
[14] Matt Deitke, Ruoshi Liu, Matthew Wallingford. Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, Eli VanderBilt, Aniruddha Kembhavi, Carl Vondrick, Georgia Gkioxari, Kiana Ehsani, Ludwig Schmidt, and Ali Farhadi. Objaverse-xl: A universe of 10m+ 3d objects. In Annual Conference Neural Information Processing Systems (NeurIPS), 2023.
[15] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In IEFF/CVF Conference Computer Vision and Puttern Recognition (CVPR), pages 13142-13153, 2023.
[16] Yu Deng, Duomin Wang, Xiaohang Ren, Xingyu Chen, and Baoyuan Wang. Portrait4d: Learning one-shot 4d head avatar synthesis using synthetic data. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7119 7130, 2024.
[17] Yu Deng, Duomin Wang, and Baoyuan Wang. Portrait4d-v2: Pseudo multi-view data creates better 4d head synthesizer. In European Conference on Computer Vision (ECCV), volume 15075 of Lecture Notes in Computer Science, pages 316-333. Springer, 2024.
[18] Yufan Deng, Yuhao Zhang, Chen Geng, Shangzhe Wu, and Jiajun Wu. Anymate: A datasct and baselines for learning 3d object rigging. arXiv preprint arXiv:2505.06227 2025.
[19] Nikita Drobyshev, Antoni Bigata Casademunt, Konstantinos Vougioukas, Zoe Landgraf, Stavros Petridis, and Maja Pantic. Emoportraits: Emotion enhanced multimodal one-shot head avatars. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8498 8507, 2024.
11

---

[20] Paul Ekman and Wallace V Friesen. Facial action coding system. Envirommental Psychology & Nonverbul Behavior, 1978.
[21] Yao Feng, Haiwen Feng, Michael J. Black, and Timo Bolkart. Learning an animatable detailed 3d face model from in-the-witt wild images. ACM Trans. Graph., 40(4):88:1-88.13 2021.
[22] Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, and Di Zhang. Liveportrait: Efficient portrait animation with stitching and retargeting control. CoRR, abs/2407. 001688 2024.
[23] Zhiyang Guo, Jinxu Xiang, Kai Ma. Wengang Zhou, Houqiang Li, and Ran Zhang. Make-it-animmaable An efficient framework for authoring animation-ready 3d characters. CoRR, abs/2411. 2024.
[24] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Annual Conference on Neural Information Processing Systems (NeurlPS), 2020.
[25] Alec Jacobson, Ilya Baran, Jovan Popovie, and Olga Sorkine. Bounded biharmonic weights for real-time deformation. ACM Truns. Graph., 30(4):78, 2011.
[26] Zi-Hang Jiang. Qianyi Wu, Keyu Chen, and Juyong Zhang. Disentangled representation learning for 3d face shape. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11957-11966, 2019.
[27] Ladislav Kavan, John Doublestein, Martin Prazák, Matthew Cioffi, and Doug Roble. Compressed skinning for facial blendshapes. In ACM SIGGRAPH 2024 Conference Papers, SIGGRAPHI page 47. ACM, 2024.
[28] Ladislav Kavan and Olga Sorkine. -lasticity-inspired deformers for character articulation. ACM Trans. Graph., 3166):19:11196:: 2012.
[29] Seonghyeon Kim, Sunjin Jung, Kwanggyoon Seo, Roger Blanco i Ribera, and Junyong Noh. Deep learning-hased unsupervised human facial retargeting. Comput. Graph. Forum, 40(7):45-55, 2021.
[30] Ariel Larey, Omri Asraf, Adam Kelder, Itzik Wilf, Ofcr Kruzel, and Nati Daniel. Facial expression re-targeting from a single character. CoRR, abs/2306. 12188, 2023.
[31] John P. Lewis, Ken Anjyo, Taehyun Rhee, Mengjie Zhang, Frédéric H. Pighin, and Zhigang Deng. Practice and theory of blendshape facial models. In Sylvain Lefebvre and Michela Spagnuolo, editors, 35th Annual Conference of the European Association for Computer Graphics, Eurographics, pages 199-218. Eurographics Association, 2014.
[32] John P. Lewis and Ken-ichi Anjyo. Direct manipulation blendshapes. IEEE Computer Graphics and Applications, 30(4):42-50, 2010.
[33] Hao Li, Thibaut Weise, and Mark Pauly, Example-based facial rigging. ACM Trans. Graph., 29(4):22:1- 32:6, 2010.
[34] Jiaman Li, Zhengfei Kuang, Yajie Zhao, Mingming He, Karl Bladin. and Hao Li. Dynamic facial asset and rig generation from a single sean. ACM Trans. Graph., 396)):15:1-2215::: 2020.
[35] Peizhuo Li. Kfir Aberman, Rana Hanocka, Libin Liu, Olga Sorkine Hornnnn and Baoquan Chen. Learning skeletal articulations with neural blend shapes. ACM Trans. Graph., 40(4):130: 130:1-13015555 2021.
[36] Ruilong Li, Karl Bladin, Yajie Zhao, Chinmay Chinara, Owen Ingraham. Pengda Xiang, Xinglei Ren, Pratusha Prasad, Bipin Kishore, Jun Xing, and Hao Li. Learning formation of physically-based face attributes. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3407- 3416, 2020.
[37] Tianye Li, Timo Bolkart, Michael Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4d scans. ACM Trans. Graph., 194:):194111944 2017.
[38] Xueting Li, Shalini De Mello, Sifci Liu, Koki Nagano, Umar Iqbal, and Jan Kautz, Generalizable one-shot 3d neural head avatar. In Annual Conference on Neural Information Processing Systems (NeurIPS), 2023.
[39] Isabella Liu, Zhan Xu, Wang Yifan, Hao Tan. Zexiang Xu, Xiaolong Wang, Hao Su, and Zifan Shi. Riganything: Template-free autoregressive rigging for diverse 3d assets. CoRR, abs/2502.09615, 2025.
[40] Lijuan Liu, Youyi Zheng, Di Tang. Yi Yuan, Changjie Fan, and Kun Zhou. Neuroskinning: automatic skin binding for production characters with deep graph networks. ACM Trans. Graph. 38(4): 1:::)::1::11114::: 111:112, 2019.
12

---

[41] Shichen Liu, Weikai Chen, Tianye Li, and Hao Li. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 7707-7716. IEEE, 2019.
[42] Jing Ma and Dongliang Zhang. Tarig: Adaptive template-aware neural rigging for humanoid characters. Comput. Graph., 114:158-167, 2023.
[43] Yue Ma, Hongyu Liu, Hongfa Wang, Heng Pan, Yingqing Hc, Junkun Yuan, Ailing Zeng, Chengfei Cai, Fine-controllable and expressive
Heung-Yeung Shum, Wei Liu, and Qifeng Chen. freestyle portrait animation. In SIGGRAPH Asia 2024 Conference Papers, pages 110:1-110:1. ACM, 2024.
[44] Albert Mosella-Monee and Javier Ruiz Hidalgo. Skinningnet: Two-stream graph convolutional neural network for skinning prediction of synthetic characters. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 18572-18581. IEFE, 2022.
[45] Lucio Moser, Chinyu Chien, Mark Williams, Jose Serra, Darren Hendler, and Doug Roble, Semi-supervised video-driven facial animation transfer for production. ACM Trans. Gruph., 40(6):222:1 2200::22::112::: 1-222: 18, 2021.
[46] Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. Voxceleb: A large-scale speaker identification dataset. In 18th Annual Conference of the International Speech Communication Association, Interspeech 2017, Stockholm, Sweden, August 20-24, 2017, pages 2-16-2620. ISCA, 2017.
[47] Jun-yong Noh and Ulrich Neumann. Expression cloning. In Lynn Pocock, editor, Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), pages 277-288. ACM, 2001.
[48] Verónica Costa Teixeira Orvalho, Ernesto Zacur, and Antonio Susín. Transferring the rig and animations from a character to different face models. Comput. Graph. Forum, 27(8):1997-200 2012, 2008
[49] Ekman Paul and Wallace V Friesen. Facial action coding system: a technique for the measurement of facial movement. Consulting Psychologists, 4, 1978.
[50] Dafei Qin, Jun Saito, Noam Aigerman, Thibault Groueix, and Taku Komura. Neural face rigging for animating and retargeting facial meshes in the wild. ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH, pages 68:1 11-8:11.. 2023.
[51] Feng Qiu, Wei Zhang, Chen I Liu, Rudong An, Lineheng Li, Yu Ding, Changjie Fan, Zhipeng Hu, and Xin Yu. Freeavatar: Robust 3d facial animation transfer by learning an expression foundation model. In SIGGRAPH Asia 2024 Conference Papers, SA '24, page 1-11. ACM, December 2024.
[52] Anurag Ranjan, Timo Bolkart, Soubhik Sanyal, and Michael J. Black. Generating 3d faces using convolu- tional mesh autoencoders. European Conference on Computer Vision (ECCV), volume 1207 of Lecture Notes in Computer Science, pages 725-741. Springer, 2018
[53] Nikhila Ravi, Jeremy Reizenstein, David Novotný, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari. Accelerating 3d deep learning with pytorch3d. CoRR, abss2200.08501.. 2020.
[54] Yeongho Scol, Jacwoo Sco, Paul Hyunjin Kim, John P. Lewis, and Junyong Noh. Artist friendly facial animation retargeting. ACM Trans. Graph., 30(6):162, 2011.
[55] Nicholas Sharp, Souhaib Attaiki, Keenan Crane, and Maks Ovsjanikov. Diffusionnet: Discretization agnostic learning on surfaces. ACM Trans. Graph., 41(3):27:1227111 2022.
[56] Nicholas Sharp, Yousuf Soliman, and Keenan Crane. Navigating intrinsic triangulations. ACM Trans. Graph., 38(4):55:1-55:16, 2019.
[57] Chanyue Song. Jianfeng Zhang, Xiu Li, Fan Yang, Yiwen Chen, Zhongcong Xu, Jun Hao Liew, Xiaoyang Guo, Fayao Liu, Jiashi Feng, and Guosheng Lin. Magicarticulate: Make your 3d models articulation-ready. CoRR, abs/2502. 2025.
[58] Robert W. Sumner and Jovan Popovic. Deformation transfer for triangle meshes. ACM Trans. Graph. 430 2004.
[59] Satoshi Suzuki and Keicchi Abe. Topological structural analysis of digitized binary images by border following. Comput. Vis. Graph. Image Process.. 30(1):32 46, 1985.
13

---

[60] Zachary Teed and Jia Deng. RAFT: recurrent all-pairs field transforms for optical flow. In European Conference on Computer Vision, volume 12347, pages 402-419. Springer, 2020.
[61] Kartik Teotia, Hyeongwoo Kim, Pablo Garrido, Mare Habermann, Mohamed Elgharib, and Christian Theobalt. Gaussianheads: End-to-end learning of drivable gaussian head avatars from coarse-to-fine representations. ACM Trans. Graph., (3(6):264:1126666
2024.
[62] Phong Tran, Egor Zakharov, Long-Nhat Ho. Adilbek Karmanov, Ariana Bermudez Venegas, McLean Goldwhite, Aviral Agarwal, Liwen Hu, Anh Tran, and Hao Li. VOODOO XP: expressive one-shot head reenactment for VR telepresence. ACM Trans. Graph., 43(6):253:1 253:226, 2024.
[63] Phong Tran, Egor Zakharov, Long-Nhat Ho, Anh Tuan Tran, Liwen Hu, and Hao Li. VOODOO 3d: Volumetric portrait disentanglement for one-shot 3d head reenactment. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18366-13388, IEEE, 2024.
[64] Noranart Vesdapunt, Mitch Rundle, Hsiang Tao Wu, and Baoyuan Wang. JNR: joint-based neural rig representation for compact 3d face modeling. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, European Conference on Computer Vision (ECCV), volume 12363, pages 389-405. Springer, 2020.
[65] Jingying Wang. Yilin Qiu, Keyu Chen, Yu Ding, and Ye Pan, Fully automatic blendshape generation for stylized characters. In IEEE Conference Virtual Reality and 3D User Interfaces (VR), pages 347-355, 2023.
[66] Ting-Chun Wang, Arun Mallya, and Ming- Yu Liu. One-shot free-view neural talking- head synthesis for vidco confereneing. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 10039-10049, 2021.
[67] You Xie, Hongyi Xu, Guoxian Song, Chao Wang, Yichun Shi, and Linjic Luo. X-portrait: Expressive portrait animation with hierarchical motion attention. In ACM SIGGRAPH 2024 Conference Papers, page 15, 2024.
[68] Zhan Xu, Yang Zhou, Evangelos Kalogerakis, Chris Landreth, and Karan Singh. Rignet: neural rigging for articulated characters. ACM Trans. Graph., 39(4):58, 2020.
[69] Shurong Yang. Huadong Li, Juhao Wu, Minhao Jing, Linze Li, Renhe Ji, Jiajun Liang, and Haoqiang Fan. Megactor: Harness the power of raw vidco for vivid portrait animation. CoRR, Abs/2405.2085// 2024.
[70] Jia-Peng Zhang, Cheng-Feng Pu, Meng-Hao Guo, Yan-Pei Cao, and Shi-Min Hu. One model to rig them all: Diverse skeleton rigging with unirig. arXiv preprint arXiv:2504. / 12451, 2025.
[71] Longwen Zhang, Qiwei Qiu, Hongyang Lin, Qixuan Zhang, Cheng Shi, Wei Yang, Ye Shi, Sibci Yang. Lan Xu, and Jingyi Yu. Dreamface: Progressive generation of animatable 3d faces under text guidance. ACM Trans. Graph., 42(4):138:1-138:16 2023.
[72] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In /EEE/CVF International Conference on Computer Vision (ICCV), pages 3813-3824, 2023.
[73] Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang, Li Zhang. Ziwei Liu, and Chen Change Loy. Celebv-hq: large-scale video facial attributes dataset. In European Conference on Computer Vision (ECCV), volume 13667, pages 650-667. Springer, 2022.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We outline scope, key contributions, and experimental results of this work abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be pereeived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings. It is fine to include aspirational goals motivation long it is clear that these goals attained by the paper.
2. Limitations
Question: Does paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of this work the paper.
Guidelines:
The answer NA means that paper has no limitation while answer means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust results are violations these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested a few datasets with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
grounds for rejection,
Theory assumptions and proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
Justification: This paper does include thcoretical results.
Guidelincs:
The answer NA means that the paper does not include theoretical results.
All theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
15

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We clearly detail the algorithm and implementation.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on contribution, reproducibility can be accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of a large language model), relcasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproducc that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (c.g., a large language model), then there should either be a way to access this model for reproducing the results a way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome to describe the particular way they provide closed-source source models, it may be that access to the model is limited some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
reproducibility. In the case of
Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No].
Justification: Upon acceptance, we will release our inference code and model weights so that reported evaluation results can be reproduced. We are also working to make the full dataset and training code publicly available, pending internal approval and legal clearance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines
Lideee/CodeSubmiii ssionPoliccy for more details.
While we encourage release of code and data, we understand that this might not possible,
so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g.. for a new open-soure benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: /// //nips...///////////ee/ cc/public/ Seaiddes/CoceeSumisssiooom/
for more details.
16

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) recommended, but including URLs to data and code is permitted.
6. Experimental sefting/details
Question: Does the paper specify all the training and test details (c.g,, data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide enough details for understanding the results in the main paper and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [No]
Justification: Not applicable for this task.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should given (e-g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error than state that they not verified.
CL, if the hypothesis of Normality of errors
asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). crror bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, docs the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Yes, we include this in the paper and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
17

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, every respect, with NeurIPS Code Ethics attps://neurips. cc/ cw/public/Esaeeeeeeec eaGuidelllie ines?
Answer: [Yes]
Justification:
We adhere to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e-g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss socictal impact in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
the authors answer NA or No, they should explain why their work has no socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations. (e.g.. deploy- of technologies that could make decisions that unfairly impact specific groups), privacy considcrations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated relcase of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible relcase of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should deseribe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
18

---

12. Licenses for existing assets
Question: Are creators or original owners assets (e.g., code, data, models), used the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected"'
Answer: [Yes]
Justification: We cite the code and datasets used and comply with their licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged reach out asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training. license, limitations.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, applicable, as as details about compensation (if any)?
Answer: [NA].
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection. curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA].
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
19

---

The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
recognize that procedures for this may vary significantly between institutions and Iocations, and we expect authors adhere to the NeurIPS Code of Ethics the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review,
Declaration LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this rescarch? Note that if the LLM is used only for writing, editing, or formatting purposes and does impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: This research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, or non-standard components.
Please refer to our LLM policy (https: //nner///s/sseeee ipsurips.om SwterrissccCCoooeeeeeeee cc/Confereecess22255 for what should should not be described.
20

---

Appendix
A Training and Inference Details
In the first stage of training, the weights for the image loss, mask loss, 2D displacement loss, and regularization loss set 10.0, 1.0, 1.0, and 0.0001, respectively. In the second stage, the weights for the image loss, mask loss, 3D MSE loss, landmark loss, and 2D eye closure loss are set 10.0, 1.0, 100.0, 0.5, and 0.5, respectively. We train our model on an Nvidia A100 instance with 8 GPUs and a total batch size of 8 (i.c., effectively sample per GPU if using distributed data parallel). The training proceeds in two stages. For the first stage, we train the deformation model on both rigged and unrigged head datasets (8,386 samples in total) using only 2D supervision for 15 epochs. This stage typically takes around 1.5 days complete. For the second stage, we then finetune model
the first stage both 2D and 3D supervision for 20 epochs. This finetuning phase finishes in approximately day. Throughout both stages, we use the Adam optimizer, initializing the learning rate at 0.0001. For learning rate scheduling. employ CosineAnnealing WarmRestarts, allowing to decay from 0.0001 nearly 0 by the end of training. Additionally, we warm-up phase of 20,000 steps stabilize carly training.
rigged head dataset (2,929 samples), incorporating
For inference speed, model runs single forward pass predict blendshapes offline, requiring only one per input mesh. The outputs are converted into classical FACS blendshape rigs, enabling efficient animation by simply linear blending. The proposed model consists of 5.4M parameters and it takes on average 8.72s on an Apple M2 Max CPU and 3.Is on a Nvidia T4 GPU to generate FACS blendshape rig on the test set (1,750 vertices, 3,362 faces average).
B Details for 2D Displacement Calculation
In the following code sample, demonstrate how compute the 2D displacement of each pixel from mesh vertex deformations in fully differentiable manner. This implementation leverages PyTorch3D's differentiable rendering functionality.
def render displ acement (vertices, deformed, deformed_vertiices, faces, renderer, camera, =(512,512)
Parameters
vertices: torch. tensor (V, 3)
veericedveerice:: torch.tensor tensor (V, 3)
faces: torch.tensor tensor (F, 3)
renderer: www.orchddrrendeeeeee MeshRenderer object
camera: pytorch3d. renderer rendererrr.rrrrrrr........ cameras. CamerasBase object res: tuple
Returns
displacement_2D 2D: torch. tensor [0], res [1], 2) un.
verts_ 2d camera. htwp:mmm_ppiitttsscrreeeeetettee
image_size-res)
verts_20 2d_ rd__defoeeed =res)
wwwp:iittteccreeeeeeeeeeeeee (dwformedd image_aize
verts_fles flow (verta_22 Seart__dd_defeee 2d_deforeed verta_2d) [:, :2] # Vx2
verts flow verts_flow verts_ / res 0.5 0.5 0-1
flou_tex torch
pad (verts_flow, pad- 1)) Vx3
texture TexturesVertex (eattee___eett__teereeeeeee wwwtureetteeeeee [flow_tex]) featuree==(llovtte))
meshes pytorch3d. bttoochhd. structuree...u..u.e..... (eshess(
verts= [vertices], faces=[faces]] textures=texture
displacement_2D renderer (meshes, cameras=camera)
return displacement_201 :2].]. squeeze ()
21

---

Brange
maage
irage
Deplacamen
Figure 10: Example results of 2D generation pipeline.
C Effectiveness of 2D Generation Pipeline
To validate the effectiveness of our 2D supervision generation pipeline, we exclude several rigged heads during the fine-tuning of the 2D face animation model and the flow estimation model. In Fig. 10, we present random sample results showcasing different faces and poses. The ground truth images and 2D displacements are rendered using the ground truth deformations the rigged heads. The 2D face animation model generates pose images based on the neutral image input, while the flow estimation model takes the neutral and generated images as input to predict the 2D displacement. The 2D displacement is visualized according to the standard optical flow convention.
D Data Collection Details
FACS Poses For each rigged heads, artist annotated 48 FACS poses and 48 corrective poses blendshape rig. We show those FACS poses Tab. 4, and Fig 11. In addition blendshapes for individual FACS poses, we generate corrective blendshapes by lincarly combining certain poses and manually correcting artifacts. These corrective blendshapes account for the complex deformations resulting from pose interactions.
Semantic Annotation We provide semantic annotation map for rigged heads, labeling different regions on the mesh (e.g., ears, mouth, eyes), along with facial landmark annotations specified as vertex indices. These annotations allow for the application of weighted losses or region-specific training objectives.
Head Interpolation First, standardized the UV layout across all head meshes, ensuring that corresponding facial features like eyes and mouths occupy same region UV space. This consistent mapping enables the identification of 3D correspondences between vertices on different meshes. Using these correspondences, we can smoothly interpolate between different head geometries through linear blending to significantly increases the size of our dataset.
22

---

#
SHORT
FULL
#
SHORT
FULL
neutral
neutral
25
LEC
LeltEyeClosed
2
c_COR
Corrugator
26
LEULR
LeftEyyeUpperidRRaise
3
c_CR
ChinRaiser
27
LIBR
LeftlnnerBrowRRiieer
4
CRUL
ChinRaiserUppertLip
28
LLCD
LeftLippComerDown
5
ELD
EywsLookDown
29
LLCP
LeftLipCComerPlller
6
ELL
EyfsLookLeft
30
LLLD
LeftLLowerLipDepress
c
ELR
RygsLookRight
31 LLS
LeftLipStretcher
ELU
EyesLookUp
32 LNW
LefikoseWrinkler
9
FN
Funneler
33
LOBR
LeftOuterBrowRaiise
10
FP
FlatPucker
34
LULR
LefiUppperLipRaiser
JD
JawDrop
35 BBL
RightBrowLooerer
12
JL
JawLeft
36
CHP
RightCheekPuff
13
JJ
JawRight
37
CHHR
RightCheekRaiser
14
LLS
LowerLipSuck
38
D_M
RightDimpler
LP
LipPresser
39
EC
RightEyeClosed
16
LPT
LipsTogether
40
EULR
RightEyyUUpperLdRRasser
17
ML
MouthLeft
41
IBR
RightlnnerBrowRRisee
18
MR
MouthRight
42
LCD
RightLipCornerDown
PK
Pucker
43
LCP
RightLipCormerPlller
20
UUS
UpperLipSuck
44
LLD
RightLLowerLippepressor
21
1
BL
LefeBrroWowerer
45
LS
RightLipSStretcher
22
CHHP
LeftCheekPuff
46
N_NW
RighnNoseWrinkler
23
LCHR
LeftCheekRaiser
47
T_OBR
RightOuteerrooRRaisrr
24
DDM
LeftDDmpler
48
r ULR
RightUpperLipRRiiser
Table FACS Short and Full Name Mapping.
E
Dataset Split
Our dataset includes rigged heads and unrigged heads. From these, a subset of 24 rigged heads with 3D ground-truth annotations forms the test set to for accurate absolute error evaluation. Additionally, we select 37 diverse unrigged heads as the test set, representing different species and shapes to evaluate the model's generalization out-of-distribution (OOD) faces. For training, we augment the dataset using interpolations, manually filtering poor interpolation results. Specifically, interpolate the remaining 137 unrigged heads with a factor 50, generating 5,457 samples, and interpolate the remaining 137 rigged heads with factor 25, producing 2,929 samples.
F Pre-processing for Baseline Method NFR
All NFR bascline results were obtained after applying the official preprocessing pipeline": we keep only the largest connected component and remove the inner-lip and eyelid surfaces. These steps erucial for NFR generate reasonable deformations. Figure 12 shows that retaining multiple disconnected components causes self-penetration, while Fig 13 shows jarring artifacts when the inner-lip surfaces are not trimmed. In contrast, our method do not need such preprocessing.
'https://github httt::://gttab..co//lfeeeeeo wwt.//oooo/////eeee/e//eeee..tet hit..etptggggn/RR.ttttrthhh
23

---

PARE
COL
S-
LOAK
LUD
UAN
TM
11
URILAT
JJRRIZZ
Figure 11: A sample of all the FACS and corrective poses used this work.
NFR
Input Neutral Mesh
Ours
Figure 12: Compared to NFR during inference meshes with multiple disconnected components from ICT Facekit Dataset. We highlight one of these components: "gums and tongue" in red. While animating a Jaw Drop pose, this component causes penetration issues for NFR.
24

---

wio Preprocassing
Preprocesssog
Inside Mesh
Deformation Result
Figure 13: Illustration of the trimming preprocessing step for NFR. The inner-lip surfaces to be trimmed are highlighted in red in the top-left figure. Omitting this step results implausible deformations produced by NFR.
G Border Impact
Our face-autorigging framework could broaden access to high-quality animation by letting small studios, educators, and assistive-tech developers create expressive avatars quickly, which benefits entertainment, remote communication, and certain medical visualization tasks. However, the same ease of use can lower the barrier for deepfake production, intensifying privacy concerns around emotion tracking and biometric profiling. Careful dataset curation, explicit usage licenses, and watermarking tools are essential to realize the creative upside while limiting misuse and incquitable impacts.
25


Reviews:
[{"review_id": "UfdPtWQGBg", "reviewer": "Reviewer_U3KH", "summary": "The paper introduces RigAnyFace, a framework for facial meshes of diverse topologies. RAF deforms a neutral mesh into FACS poses to create a blendshape rig, using a triangulation-agnostic DiffusionNet backbone augmented with two novel components: (1) a global encoder to handle disconnected parts (2) FACS conditioning blocks to integrate action-unit parameters. To address limited 3D supervision, RAF leverages 2D supervision generated via a pretrained 2D face-animation model for unrigged neutral meshes. This strategy scales training data, enhancing generalization. Experiments show RAF outperforms prior work on artist-crafted and in-the-wild meshes in accuracy and topological flexibility.", "questions": "RAFT optical flow and MegActor may struggle with highly stylized/non-humanoid meshes. How significant is the domain gap for these models? Provide error analysis of generated 2D data vs. artist annotations.\n\nThe global encoder compresses all mesh components into one vector, potentially losing part-specific details. Could a hierarchical encoder improve detail preservation?\n\nHow does RAF handle cases where the input mesh contains severe discretization artifacts, such as shell-like geometry or low-resolution scans? Are there any preprocessing recommendations or failure cases to be aware of? Would data augmentation or mesh-denoising pre-processing mitigate this?", "limitations": "please see the Questions section.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Robust technical design effectively handles disconnected components via the global encoder and achieves precise expression control through FACS conditioning blocks.\n- Superior performance outperforms NFR by >45% in MAE on humanoid heads and generalizes to non-humanoid/out-of-distribution meshes.\n- Significant practical impact eliminates template dependency, supports multi-component meshes, and reduces 3D data reliance via scalable 2D supervision, enabling applications in text-to-3D rigging and video retargeting.\n- Pioneering contributions are the first to support disconnected components in auto-rigging and introduce 2D supervision for 3D deformation networks.\n\nWeaknesses:\n- Global feature bottleneck may compress fine details into a single vector, limiting performance on complex meshes.\n- Degraded performance on shell-like meshes occurs when inputs lack fine geometric details.\n- Unclear handling of stylized meshes in the 2D supervision pipeline; fine-tuning details for non-standard topologies need clarification.\n- Suboptimal inference speed could hinder real-time deployment, Moderate originality as core innovations build incrementally upon DiffusionNet.", "was_revised": true, "final_justification": "The rebuttal has addressed my concerns about the evaluation issues. I am happy to upgrade my score to borderline accept. The additional discussions and experiments should be included in the revised manuscript or supplementary materials.", "rebuttal": "We sincerely appreciate the reviewer’s constructive feedback and recognition of the key strengths of our work, including its support for meshes with multiple disconnected components and the scalable training enabled by unlabeled data. Below, we address the reviewer’s questions and concerns point by point.\n\n### 1. Global features are compressed into a single vector, potentially losing details.\n\nThe primary role of the global feature is not to capture fine-grained geometric details, but rather to encode the relative positions between disconnected components. This information is crucial for preventing penetrations during deformation, as the Diffusion Blocks alone cannot propagate information across disconnected mesh components.\n\nTo quantify this, we evaluate the penetration between inner components (e.g., mouthbag) and the outer face surface by reporting the percentage of penetrating vertices with and without our global feature. Specifically, we compute the signed distance function (SDF) of each vertex on the inner component with respect to the outer face surface in both the neutral and predicted meshes. Vertices are considered to be penetrating when their SDF values change from negative in the neutral mesh to positive in the deformed/predicted mesh. As shown in the table below, involving the global feature significantly reduces the number of such penetrating vertices. Please also refer to Fig. 5A (first column) in the main paper for the qualitative comparison. \n\nRegarding the concern about capturing geometric details, the final deformation predictions are contributed by both the global feature and pre-vertex features from the Diffusion Blocks, enabling accurate deformation even for complex meshes. For instance, while the training set meshes have 1,704 vertices and 3,267 faces on average, our model generalizes well to in-the-wild meshes with significantly higher complexity, averaging 9,497 vertices and 17,958 faces, as shown in Fig. 7 of the main paper. Given the effectiveness of this setup, we found that a more complex global encoder (e.g., a hierarchical encoder) might not be necessary.\n\n| Global Encoder | All Other Components | MAE ↓ | MAE Q95 ↓ | Penetration ↓ |\n|----------------|-----------------------|--------|-----------|--------------------|\n| ✗              | ✗                     | 2.14   | 6.64      | 0.377              |\n| ✗              | ✓                     | 2.16   | 6.08      | 0.405              |\n| ✓              | ✗                     | 2.08   | 5.84      | **0.166**          |\n| ✓              | ✓                     | **1.92** | **5.63** | 0.173         |\n\n### 2. Domain gap of RAFT and MegActor’s pretrained models on stylized/non-humanoid meshes. Provide error analysis. \n\nYes, there is indeed a domain gap when directly applying the pretrained RAFT and MegActor models to stylized head mesh renderings. To address this, we fine-tuned both models using rendered ground-truth optical flow and expression images from our artist-annotated training set, as described in L222–224 of our paper. \n\nFor error analysis, we used similarly rendered ground-truth data from the artist-annotated test set, with all images rendered at 512×512 resolution. The fine-tuned RAFT achieves an end-point error (EPE, lower is better) of 0.99, i.e., the average per-pixel Euclidean distance between predicted and ground-truth optical flow vectors is equal to 0.99. The fine-tuned MegActor yields a mean absolute error (MAE) of 0.00346 between generated and ground-truth images, computed over RGB values normalized to [0, 1]. Please also refer to Figure 1 in our supplementary material for qualitative comparisons of the generated 2D supervisions. \n\nThese results indicate that, after fine-tuning, both RAFT and MegActor handle stylized or non-humanoid meshes with high accuracy and show strong robustness against domain gaps between realistic and stylized inputs.\n\n\n### 3. Degraded performance on shell-like meshes and meshes with poor discretization. How does RAF handle them? \n\nWe will include qualitative examples in the revision to show that our model typically leaves such meshes undeformed. Even though we would also like to clarify that such meshes are not typical inputs for facial rigging pipelines and are generally not considered by prior methods. Shell-like meshes (e.g., deformed spheres) often depict facial features such as eyeballs and mouths using textures instead of explicit geometry, making them ill-suited for animating expressions like eye closure or mouth opening. Meshes with poor discretization, where the main facial surface can break into multiple disconnected pieces, are very challenging to rig and annotate, even for professional artists. We include them in our evaluation to explore the robustness and limitations of our approach beyond its intended scope.\n\n### 4. Are there any preprocessing recommendations for failure cases? \n\nFor poorly discretized meshes, we experimented with preprocessing such as remeshing to improve mesh quality. However, we found that remeshing often mistakenly connects disconnected components (e.g., eyeballs and mouth bags) to the main facial mesh, which deteriorates the pose prediction. For instance, animation of eye gazing requires the eyeballs to deform independently within the eyelid region. If these parts are merged, the model would struggle to predict correct deformations. As a future direction, incorporating a diffusion operator defined on a high-quality background triangulation [1] may help with these failure cases.\n\n[1] Nicholas Sharp, Yousuf Soliman, and Keenan Crane. 2019. Navigating intrinsic triangulations. ACM Trans. Graph. 38, 4 (2019)\n\n### 5. Suboptimal inference speed could hinder real-time deployment.\n\nAs discussed in L242–L244 of our paper, our model is lightweight, with only 5.4M parameters. It performs a single forward pass to generate all the FACS poses to form a blendshape rig for each input mesh, a process that takes on average 8.72s on an Apple M2 Max CPU and 3.1s on an Nvidia T4 GPU (test set average: 1,750 vertices and 3,362 faces). \n\nImportantly, this process is conducted offline. Once the blendshape rig is generated, animations can be executed in real-time at deployment using standard linear blendshape interpolation, achieving over 1000 FPS on an Apple M2 Max CPU in our testing.\n\n### 6. Moderate originality as core innovations build incrementally upon DiffusionNet.\n\nOur architectural design is specifically tailored to address facial rigging challenges that previous neural methods have not effectively handled. The global feature enables information flow across disconnected components, and FACS conditioning allows expression-aware deformation. These designs are essential components of our auto-rigging framework. More importantly, our design of learning 3D deformation with 2D supervision enables scalable training on unlabeled data.\n\nTaken together, the architectural innovations and our scalable training framework form a unified and practical solution for real-world auto-rigging across diverse mesh topologies with multiple disconnected components, which we believe constitutes a meaningful step beyond prior work."}, {"review_id": "sfClPQjdGi", "reviewer": "Reviewer_E5kF", "summary": "This paper introduces RigAnyFace, a scalable and generalizable neural auto-rigging framework for facial meshes with diverse topologies, including those with disconnected components like eyeballs. Its key goal is to automatically create FACS-based blendshape rigs from a neutral 3D facial mesh. \n\nTo achieve this goal, the paper use 2 stage training pipeline. First, train on both rigged and unrigged data using 2D losses. Then, fine-tune on rigged data with 3D supervision for higher precision. The model part is built upon DiffusionNet, use a global encoder to capture holistic information across disconnected components and conditional diffusion blocks to integrate FACS parameters directly into the deformation prediction process.", "questions": "- Could you include visual or quantitative examples of where the model fails or underperforms?\n- How is the model's ability to handle extreme expressions and postures of driving images?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- Strong Methodological Innovation: The paper presents a well-designed auto-rigging framework that combines a modified DiffusionNet backbone with conditional FACS encoding and a global encoder to handle disconnected mesh components. These architectural modifications are technically sound and justified.\n- Well-structured: the paper is logically organized, beginning with motivation, followed by detailed method descriptions, training strategy, and comprehensive evaluations. Figures are clear and informative.\n- Support for Disconnected Components: A significant practical contribution is the ability to rig meshes with disconnected elements like eyeballs—something most prior works fail to address.\n\nWeaknesses\n- Limited Analysis of Failure Cases: The limitations section briefly mentions mesh discretization and poor mesh quality as failure modes, but these are not empirically analyzed or visualized in the main paper.\n- Incremental Architecture Changes: While effective, the architectural improvements to DiffusionNet (e.g., conditional blocks, global encoder) are relatively incremental. Much of the novelty lies in the integration and system design, rather than entirely new algorithms.\n- Two-Stage Training with 2D and 3D Supervision: 3D mesh learning with 2D supervision already have many prior works. The originality and novelty of the framework and loss functions are limited here.\n- Evaluation Setup: This work only performed a comparison with NFR in the evaluation, lacking more comprehensive analysis with other SOTA methods.", "was_revised": true, "final_justification": "The authors have addressed majority of the concerns in the rebuttal.", "rebuttal": "We sincerely thank the reviewer for the valuable feedback and recognition of the key contributions of our work, including the tailored architectural design for handling disconnected mesh components and the scalable training pipeline. We address the reviewer’s concerns as below:\n\n### 1. Limited Analysis of Failure Cases:\n\nThanks for pointing that out. In our revision, we will include qualitative examples of our failure cases particularly where our method struggles to deform shell-like structures or meshes with poor discretization. These challenging cases are mainly in-the-wild samples sourced from Objaverse, which is a large-scale dataset of general-purpose 3D assets but not curated for facial modeling. Therefore, quantitative analysis of failure cases is unfortunately not feasible as the lack of ground-truth rig annotations. These failure cases are primarily caused by the following factors:\n\n* Shell-like meshes (e.g., deformed spheres) often depict facial features such as eyelids and lips using texture rather than geometry, making meaningful deformation ill-posed or ambiguous.\n* Poorly discretized meshes, where the main facial surface breaks into disconnected parts instead of a complete one, hinder feature propagation through the DiffusionNet backbone and make it difficult to establish consistent deformation bases.\n\n### 2. Incremental Architecture Changes:\n\nOur RAF, built on top of DiffusionNet, is specifically aimed at addressing facial rigging challenges that previous neural methods are not capable of handling. The global feature inherits the information of global arrangement between disconnected components, and our FACS conditioning allows for expression-aware deformation. These designs are essential components of our auto-rigging system.\n\n### 3. 3D mesh learning with 2D losses already has many prior works. The originality of the loss functions is limited:\n\nWhile prior works [1, 2, 3] have explored applying 2D losses for 3D face reconstruction or generation using direct 2D supervision from input images or videos, our approach first generates pseudo-2D ground truth tailored to each unrigged mesh via a 2D face animation model and an optical flow estimator. We then train on large-scale unrigged meshes with 2D supervision generated above, eliminating the need for artist-provided annotations while enabling scalable training.\n\nIn addition, we empirically found that relying solely on 2D photometric loss is insufficient for accurate deformation. It often fails to capture subtle expressions as limited changes in pixel RGB values provide only marginal gradients. To address this, we introduce a 2D displacement loss based on pixel motion flow, which provides a stronger signal as an additional supervision for capturing fine-grained facial motions.\n\n[1] Feng, Yao, et al. Learning an Animatable Detailed 3D Face Model from In-The-Wild Images. (TOG2021)\n\n[2] Chan, Eric R., et al. Efficient geometry-aware 3d generative adversarial networks. (CVPR2022)\n\n[3] Trevithick, Alex, et al. Real-time radiance fields for single-image portrait view synthesis. (TOG2023)\n\n### 4. Evaluation Setup: only compared with NFR in the evaluation, lacking analysis with other SOTA methods.\n\nIndeed, Neural Face Rigging (NFR) is our primary baseline as it is the most recent and relevant method for our task, to the best of our knowledge. As also mentioned by Reviewer cQPn, NFR is the latest SOTA method that we can compare with.\n\n### 5. How is the model's ability to handle extreme expressions of driving images? \n\nPlease refer to Section 2 of our supplementary video: Retargeting Human Expression from videos (01:09–03:17), which showcases retargeting of real-world videos with a variety of extreme expressions (including one example from the Facial Extreme Emotions Dataset [4]) to unrigged 3D meshes. These results demonstrate our model’s ability to generalize to intense and diverse facial expressions.\n\n[4] Drobyshev, Nikita, et al. Emoportraits: Emotion-enhanced multimodal one-shot head avatars. (CVPR2024)"}, {"review_id": "TkziL0ZUvc", "reviewer": "Reviewer_cQPn", "summary": "This work tackles the problem of generating blend shapes of an arbitrary 3D human head model, which can also have disconnected geometries such as eyeballs. To this end, the authors propose a learning based approach based on the DiffusioNet architecture and extend it to accept FACS conditioning as well as a global latent code to overcome the issue of disconnected geometry. The authors train on hand-crafted ground truth pairs, i.e. head models and FACS blend shapes. However, this data has limited diversity and thus, the authors also propose a 2D supervision strategy using readily available 2D face animation methods for generating pseudo 2D ground truth. The ablation studies and comparisons partially confirm the authors claims.", "questions": "l.146 How does the interpolation strategy works? I cannot find much details in the manuscript? Does it assume correspondence between two different instances? If so, how can this be guaranteed?\n\nIs NFR retrained on the same data? Otherwise, the comparison is not conclusive as it is not clear whether the improvement comes from data or the algorithmic design. The authors should clarify.", "limitations": "Limitations are discussed in the conclusion section.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:** \n\n- Clear scope and positioning of the work\n- References are more or less complete\n- Challenging and relevant problem setting\n- Interesting idea of using Megactor for generating pseudo ground truth for weakly supervised training\n- Comparisons are slightly limited in terms of prior methods. However, even after careful search, I could not find newer methods tackling the same problem. Thus, I believe the comparisons are complete to the best of my knowledge.\n\n---\n\n**Weaknesses:**\n\n- Incremental design\n    - The global encoder and FACS conditioning can be considered minor (task-specific) adaptations to the DiffusioNet architecture. While interesting, I believe it is rather incremental and does not justify an acceptance at Neurips.\n- Technical correctness\n    - paragraph l175 defines a differentiable rendering loss between the reconstructed FACS and the ground truth rendering. However, this still assumes having paired GT data available. Thus, I am wondering why this should generalize better as it essentially requires the same data as before? Then why not simply supervising in 3D? The authors should clarify.\n    - In paragraph l195, again it remains unclear why supervision is applied in 2D as the ground truth mesh is assumed. Why not simply supervising in 3D directly?\n- Conceptual design choices\n    - The global encoder basically averages all features across all vertices. This completely ignores spatial/local structures. While it is addressing disconnected geometry parts, it completely destroys locality information. Thus, I feel this design is sub-optimal given the task.\n- Writing\n    - FACS not being introduced before using the acronym\n    - l.34 starting the sentence with “But” is grammatically wrong\n    - l.133-134 could be removed as it does not add any valuable information\n    - l158 “. And…” not a proper english sentence\n- Clarity\n    - The DiffusionNet background section is not very self-contained. It would be great to introduce the approach more formally with equations/mathematical notations\n- Ablation studies\n    - The authors provide some ablation studies, which is good. However, I would like to see a study where only one component at the time is changed to a simple baseline. For example, the global encoder is only removed in conjunction with other key designs. Instead, I would like to see an ablation where only the global encoder is removed. This would be a more meaningful experiment. Moreover, the discussion about the ablation study is very shallow and at the level of “every design improves the results”. There is no good answer to why it improves and in which sense it improves results. The authors should extend explanations here.\n- Missing reference for image to 3D avatar\n    - Teotia et al. 2024: GaussianHeads: End-to-End Learning of Drivable Gaussian Head Avatars from Coarse-to-fine Representations\n\nGiven the current weaknesses, I slightly lean towards reject. However, the authors may be able to address some of the concerns during the rebuttal period.", "was_revised": true, "final_justification": "The authors addressed my major concerns about novelty, evaluation, and key design choices during the rebuttal. \n\nThus, I am happy to increase my score.", "rebuttal": "We sincerely appreciate the reviewer’s valuable and constructive feedback and recognition of our work’s scope and complete comparison. For presentation-related issues, we will further polish the writing, correct grammatical errors, clarify technical details, and add the missing references as suggested. We address the reviewer’s technical concerns below.\n\n### 1. Why is supervision applied in 2D, as the ground truth mesh is assumed? Why not simply supervise in 3D directly?\n\nThank you for the question, this gives us a chance to clarify an important part and contribution of our approach. Not all meshes in the training set come with artist-labeled 3D blendshape (deformed mesh) ground truth. Due to the high cost of manual annotation, only a subset of samples have full 3D labels. The remaining unlabeled data consists of only a neutral mesh without any corresponding 3D ground truth deformations.\n\nTherefore, our use of 2D supervision is not intended to replace 3D supervision when it is available, but to enable learning from the unlabeled data. Specifically, we employ a two-stage training pipeline. During the first stage of training, we use 2D losses to supervise both labeled and unlabeled samples. For the unlabeled ones, we generate pseudo-2D ground truth using a 2D face animation model (based on MegActor) and a RAFT-based optical flow estimator.\n\nThen, in the second stage, we fine-tune the model using the subset of data with accurate 3D ground-truth deformations with 3D loss. This two-stage strategy allows us to scale learning while still benefiting from the precision of 3D supervision when available. \n\n### 2. The global encoder and FACS conditioning can be considered minor (task-specific) adaptations. \n\nWhile the global encoder and FACS conditioning are indeed tailored to the facial rigging, we believe this task-specificity reflects a thoughtful and necessary adaptation to challenges that have not been addressed in prior work. The global encoder enables information flow across disconnected components, and FACS conditioning allows expression-aware deformation. These designs are essential components of our auto-rigging system.\n\nImportantly, these architectural components are just one part of our contributions. Another key contribution of our work lies in the 2D supervision design that enables scaling-up training on unlabeled neutral facial meshes without rig annotations. Unlike prior methods that rely exclusively on costly 3D-labeled data, we introduce a tailored 2D animation and optical flow pipeline to generate pseudo-2D ground truth supervision for unlabeled data. To the best of our knowledge, RAF is the first scalable approach to make effective use of unlabeled data for neural auto-rigging.\n\nTaken together, the architectural innovations and our scalable training framework form a unified and practical solution for real-world auto-rigging across diverse mesh topologies with multiple disconnected components, which we believe constitutes a meaningful step beyond prior work.\n\n### 3. Global encoder compressing all vertices' features in one single vector destroys locality information. \n\nThe primary role of the global feature is not to capture fine-grained geometric details, but rather to encode the relative positions between disconnected components. This information is crucial for preventing penetrations during deformation, as the Diffusion Blocks alone cannot propagate information across disconnected mesh components.\n\nTo quantify this, we evaluate the penetration between inner components (e.g., mouthbag) and the outer face surface by reporting the percentage of penetrating vertices with and without our global feature. Specifically, we compute the signed distance function (SDF) of each vertex on the inner component with respect to the outer face surface in both the neutral and predicted meshes. Vertices are considered to be penetrating when their SDF values change from negative in the neutral mesh to positive in the deformed/predicted mesh. As shown in the table below, involving the global feature significantly reduces the number of such penetrating vertices. Please also refer to Fig. 5A (first column) in the main paper for the qualitative comparison. \n\nRegarding the concern about capturing geometric details, the final deformation predictions are contributed by both the global feature and pre-vertex features from the Diffusion Blocks, enabling accurate deformation even for complex meshes. For instance, while the training set meshes have 1,704 vertices and 3,267 faces on average, our model generalizes well to in-the-wild meshes with significantly higher complexity, averaging 9,497 vertices and 17,958 faces, as shown in Fig. 7 of the main paper. \n\n| Global Encoder | All Other Components | MAE ↓ | MAE Q95 ↓ | Penetration ↓ |\n|----------------|-----------------------|--------|-----------|--------------------|\n| ✗              | ✗                     | 2.14   | 6.64      | 0.377              |\n| ✗              | ✓                     | 2.16   | 6.08      | 0.405              |\n| ✓              | ✗                     | 2.08   | 5.84      | **0.166**          |\n| ✓              | ✓                     | **1.92** | **5.63** | 0.173         |\n\n### 4. Ablation study on the global encoder.\n\nWe conducted an additional ablation as the reviewer suggested, removing the global encoder only while keeping all other components unchanged. As shown in the second row of the above table, the model’s performance drops noticeably compared to the full model (the last row) and becomes similar to the variant where the global encoder is removed together with other key components (the first row). This demonstrates that the global encoder plays a critical role, and its removal alone creates a significant bottleneck in the overall performance, even when the rest of the architecture remains intact. \n\n### 5. How does the interpolation strategy work? Does it assume correspondence, and how can it be guaranteed?\n\nYes, our interpolation strategy assumes correspondence between meshes of different topologies. Specifically, we standardize the UV layout across all head meshes so that key facial features (e.g., eyes, mouth) are consistently mapped to the same regions in UV space. This shared UV space enables us to identify correspondences between vertices across meshes, even when their topologies differ. Leveraging these correspondences, we perform linear blending in geometry space to smoothly interpolate between different head shapes.\n\n### 6. Is NFR retrained on the same data?\n\nNFR’s released pretrained model didn’t provide training scripts, and it does not support meshes with multiple connected components by design. Therefore, we could not re-train the NFR model. Given these constraints, the fairest comparison we can make is to evaluate both methods on unseen artist-crafted and in-the-wild head meshes, strictly preprocessed to meet NFR’s input requirements. Specifically, for the data used to compare our model with NFR, we retain only the largest connected component of the neutral mesh, removing auxiliary structures such as eyeballs and mouthbags, and trimming inner surfaces of the lips and eyelids (please refer to Figs 3 and 4 in the supplementary appendix).\n\nBesides, we also evaluated our model on NFR’s training dataset, ICT FaceKit (please refer to the first sample in our Fig.7 ), while our method did not train on this dataset,  we still got comparable performance while handling multiple connected components."}, {"review_id": "vF8qCYzEPd", "reviewer": "Reviewer_r7gx", "summary": "This paper presents RigAnyFace (RAF), a novel auto-rigging framework for facial meshes that supports rigging in-the-wild meshes without manual labeling, addresses limitations of previous methods, particularly Neural Face Rigging (NFR). RAF is designed to handle diverse mesh topologies, including those with multiple disconnected components like eyeballs and mouth sockets. The framework utilizes a triangulation-agnostic surface learning network, augmented with a 2D diffusion block for and a global encoder to handle disconnected components. To overcome the scarcity and high cost of manually rigged 3D ground truth data, RAF introduces a 2D supervision strategy for unlabeled neutral meshes. This strategy leverages a generative 2D face animation model and an optical flow estimator to synthesize posed images and 2D displacement fields for training. Experiments show that RAF outperforms prior work in accuracy and generalizability across artist-crafted assets and in-the-wild samples, and it supports more detailed expression animation by accommodating disconnected components.", "questions": "1. Will the code and the dataset be released?\n2. Why did the authors choose to output vertex offsets directly instead of using Possion systems like in Neural Jacobian Fields or Neural Face Rigging? An ablation on this could better explain the design choices.\n3. As the DiffusionNet layers can't propagate information between disconnected components, how does the Global Encoder, made of several DiffusionNet Layers with average pooling, help to mitigate this issue? Besides, would the model be able to generalize to meshes with unseen components?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "# Strengths\n1. The proposed global encoder helps to deal with multiple disconnected components.\n2. The 2D supervision pipeline enriches the model's exposure to unrigged meshes, which effectively enhances the generalization ability towards in-the-wild meshes.\n3. The general retargetting quality, including the expression faithfulness, the temporal smoothness, and the scope of applicable meshes is better than comparable baselines.\n4. The ablation studies validate the effectiveness of the key design choices.\n\n# Weaknesses\n1. Though the model delivers better temporal consistency, it does not explicitly consider the temporal dimension. \n2. The paper does not analyse the properties of the global encoder, either its feature distribution or why it's effective for different disconnected components.\n3. The discussion of limitations is omitted. It's better to give some specific examples about the limitations.", "was_revised": true, "final_justification": "The authors have provided a thorough rebuttal and engaged in a productive discussion that has successfully addressed my initial questions and concerns. The core contributions of the paper stand strong, and my confidence in its quality and impact has been reinforced. \nThrough rebuttal, the author resolved my issues about the functionality of the Global Encoder and the character-specific expressions.\nThe paper does not explicitly model the temporal dimension for animation consistency. However, the authors acknowledged this and have reasonably positioned it as an area for future work. Given that the paper's main contribution is a scalable auto-rigging framework for static meshes, I assign low weight to this point and do not see it as a flaw in the current work. \nIn summary, I believe this work is a step towards auto-rigging of the face. The proposed method, combining scalable training on 2D unlabeled data and an architecture handling complex topologies and disconnected components, is helpful. I thus recommand accept.", "rebuttal": "We sincerely appreciate the reviewer’s constructive comments and recognition of the key contributions of our paper. We address the reviewer’s concerns and questions point by point.\n### 1. Though the model delivers better temporal consistency, it does not explicitly consider the temporal dimension.\n\nThanks for pointing that out. We will explore the explicit temporary modeling during animation in our future work. \n\n### 2. Properties of the global encoder:\n\nThe primary role of the global feature is to encode the relative positions between disconnected components. This information is crucial for preventing penetrations during deformation, as the Diffusion Blocks alone cannot propagate information across disconnected mesh components.\n\nTo quantify this, we evaluate the penetration between inner components (e.g., mouthbag) and the outer face surface by reporting the percentage of penetrating vertices with and without our global feature. Specifically, we compute the signed distance function (SDF) of each vertex on the inner component with respect to the outer face surface in both the neutral and predicted meshes. Vertices are considered to be penetrating when their SDF values change from negative in the neutral mesh to positive in the deformed/predicted mesh. As shown in the table below, involving the global feature significantly reduces the number of such penetrating vertices. Please also refer to Fig. 5A (first column) in the main paper for the qualitative comparison. \n\n| Global Encoder | All Other Components | MAE ↓ | MAE Q95 ↓ | Penetration ↓ |\n|----------------|-----------------------|--------|-----------|--------------------|\n| ✗              | ✗                     | 2.14   | 6.64      | 0.377              |\n| ✗              | ✓                     | 2.16   | 6.08      | 0.405              |\n| ✓              | ✗                     | 2.08   | 5.84      | **0.166**          |\n| ✓              | ✓                     | **1.92** | **5.63** | 0.173         |\n\nAdditionally, we will include two visualizations in our revision to illustrate the role of the global encoder. Specifically, we will show T-SNE plots of the global features compared against those from unmodified meshes in our dataset: (1) when the positions of connected components are randomly perturbed, and (2) when different numbers or types of connected components are removed. In both cases, the resulting features form separate clusters from the original ones, demonstrating that the global encoder captures structural changes in the mesh.\n\n### 3. Discussion of limitations:\n\nThanks for pointing that out. In our revision, we will include qualitative examples of our failure cases particularly where our method struggles to deform shell-like structures or meshes with poor discretization. These failure cases are primarily caused by the following factors:\n\n* Shell-like meshes (e.g., deformed spheres) often depict facial features such as eyelids and lips using texture rather than geometry, making meaningful deformation ill-posed or ambiguous.\n* Poorly discretized meshes, where the main facial surface breaks into disconnected parts instead of a complete one, hinder feature propagation through the DiffusionNet backbone and make it difficult to establish consistent deformation bases.\n\n### 4. Will the code and the dataset be released?\n\nWe will release the code, trained model weights for reproducibility. We are also working on releasing the full dataset, pending internal approval and legal clearance upon acceptance.\n\n### 5. Ablation on Neural Jacobian Fields\n\nWe initially explored Poisson-based approaches such as Neural Jacobian Fields (NJF), but found several issues. In our early experiments, NJF required double-precision for stable training, which increased memory usage and slowed down the training. Moreover, passing gradients through the Poisson solver introduces additional computational overhead without a prominent benefit in terms of accuracy.\n\nIn contrast, directly predicting vertex displacements proved to be much simpler, more efficient, and yielded strong empirical results. Given these considerations, we decided to focus on the direct offset prediction and discontinued the NJF direction early in development.\n\n### 6. Would the model be able to generalize to meshes with unseen components?\n\nYes. During dataset curation, we ensured coverage of common deformable facial components needed for rigging, such as mouth bags and eyeballs. Unseen components typically refer to non-deforming accessories (e.g., earrings, hats, or hair), which do not require animation.\n\nFor example, in the sample from Objaverse (Please refer to Fig. 1, third column; Fig. 7, last column), earrings are preserved without deformation. Similarly, in our video demo (Please refer to Section 3: Animating Generated Facial Mesh from Text-to-3D Model), components like hair and hats remain unaffected during animation. Even though these parts were unseen during training, the model naturally preserves them on the mesh."}]

Author rebuttal:

Meta-review: Strengths
* Builds on top of DiffusionNet for topology-agnostic facial rigging
* Overcomes the limitation of DiffusionNet with a global encoder to communicate holistic face characteristics across disconnected mesh components, e.g., base face mesh and eyeballs
* 2D supervision pipeline with fine-tuned optical flow model for training with unlabelled data
* SOTA results compared to NFR

Weaknesses
* (debatable) Incremental work on top of DiffusionNet

During the discussion, the authors have clarified the claims and novelty of the paper. The additional experiments, especially the ablations of the global encoder, will strengthen the paper.

Therefore, I recommend accepting the paper.
Decision: Accept (poster)