Title: RigAnyFace: Scaling Neural Facial Mesh Auto-Rigging with Unlabeled Data
Venue: NeurIPS 2025
Accepted: yes
Keywords: Creative AI

Auto-rigging

Facial Animation
Abstract: In this paper, we present RigAnyFace (RAF), a scalable neural auto-rigging framework for facial meshes of diverse topologies, including those with multiple disconnected components. RAF deforms a static neutral facial mesh into industry-standard FACS poses to form an expressive blendshape rig. Deformations are predicted by a triangulation-agnostic surface learning network augmented with our tailored architecture design to condition on FACS parameters and efficiently process disconnected components. For training, we curated a dataset of facial meshes, with a subset meticulously rigged by professional artists to serve as accurate 3D ground truth for deformation supervision. Due to the high cost of manual rigging, this subset is limited in size, constraining the generalization ability of models trained exclusively on it. To address this, we design a 2D supervision strategy for unlabeled neutral meshes without rigs. This strategy increases data diversity and allows for scaled training, thereby enhancing the generalization ability of models trained on this augmented data. Extensive experiments demonstrate that RAF is able to rig meshes of diverse topologies on not only our artist-crafted assets but also in-the-wild samples, outperforming previous works in accuracy and generalizability. Moreover, our method advances beyond prior work by supporting multiple disconnected components, such as eyeballs, for more detailed expression animation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents RigAnyFace (RAF), a novel auto-rigging framework for facial meshes that supports rigging in-the-wild meshes without manual labeling, addresses limitations of previous methods, particularly Neural Face Rigging (NFR). RAF is designed to handle diverse mesh topologies, including those with multiple disconnected components like eyeballs and mouth sockets. The framework utilizes a triangulation-agnostic surface learning network, augmented with a 2D diffusion block for and a global encoder to handle disconnected components. To overcome the scarcity and high cost of manually rigged 3D ground truth data, RAF introduces a 2D supervision strategy for unlabeled neutral meshes. This strategy leverages a generative 2D face animation model and an optical flow estimator to synthesize posed images and 2D displacement fields for training. Experiments show that RAF outperforms prior work in accuracy and generalizability across artist-crafted assets and in-the-wild samples, and it supports more detailed expression animation by accommodating disconnected components.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n1. The proposed global encoder helps to deal with multiple disconnected components.\n2. The 2D supervision pipeline enriches the model's exposure to unrigged meshes, which effectively enhances the generalization ability towards in-the-wild meshes.\n3. The general retargetting quality, including the expression faithfulness, the temporal smoothness, and the scope of applicable meshes is better than comparable baselines.\n4. The ablation studies validate the effectiveness of the key design choices.\n\n# Weaknesses\n1. Though the model delivers better temporal consistency, it does not explicitly consider the temporal dimension. \n2. The paper does not analyse the properties of the global encoder, either its feature distribution or why it's effective for different disconnected components.\n3. The discussion of limitations is omitted. It's better to give some specific examples about the limitations.\n\nQUESTIONS:\n1. Will the code and the dataset be released?\n2. Why did the authors choose to output vertex offsets directly instead of using Possion systems like in Neural Jacobian Fields or Neural Face Rigging? An ablation on this could better explain the design choices.\n3. As the DiffusionNet layers can't propagate information between disconnected components, how does the Global Encoder, made of several DiffusionNet Layers with average pooling, help to mitigate this issue? Besides, would the model be able to generalize to meshes with unseen components?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work tackles the problem of generating blend shapes of an arbitrary 3D human head model, which can also have disconnected geometries such as eyeballs. To this end, the authors propose a learning based approach based on the DiffusioNet architecture and extend it to accept FACS conditioning as well as a global latent code to overcome the issue of disconnected geometry. The authors train on hand-crafted ground truth pairs, i.e. head models and FACS blend shapes. However, this data has limited diversity and thus, the authors also propose a 2D supervision strategy using readily available 2D face animation methods for generating pseudo 2D ground truth. The ablation studies and comparisons partially confirm the authors claims.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:** \n\n- Clear scope and positioning of the work\n- References are more or less complete\n- Challenging and relevant problem setting\n- Interesting idea of using Megactor for generating pseudo ground truth for weakly supervised training\n- Comparisons are slightly limited in terms of prior methods. However, even after careful search, I could not find newer methods tackling the same problem. Thus, I believe the comparisons are complete to the best of my knowledge.\n\n---\n\n**Weaknesses:**\n\n- Incremental design\n    - The global encoder and FACS conditioning can be considered minor (task-specific) adaptations to the DiffusioNet architecture. While interesting, I believe it is rather incremental and does not justify an acceptance at Neurips.\n- Technical correctness\n    - paragraph l175 defines a differentiable rendering loss between the reconstructed FACS and the ground truth rendering. However, this still assumes having paired GT data available. Thus, I am wondering why this should generalize better as it essentially requires the same data as before? Then why not simply supervising in 3D? The authors should clarify.\n    - In paragraph l195, again it remains unclear why supervision is applied in 2D as the ground truth mesh is assumed. Why not simply supervising in 3D directly?\n- Conceptual design choices\n    - The global encoder basically averages all features across all vertices. This completely ignores spatial/local structures. While it is addressing disconnected geometry parts, it completely destroys locality information. Thus, I feel this design is sub-optimal given the task.\n- Writing\n    - FACS not being introduced before using the acronym\n    - l.34 starting the sentence with “But” is grammatically wrong\n    - l.133-134 could be removed as it does not add any valuable information\n    - l158 “. And…” not a proper english sentence\n- Clarity\n    - The DiffusionNet background section is not very self-contained. It would be great to introduce the approach more formally with equations/mathematical notations\n- Ablation studies\n    - The authors provide some ablation studies, which is good. However, I would like to see a study where only one component at the time is changed to a simple baseline. For example, the global encoder is only removed in conjunction with other key designs. Instead, I would like to see an ablation where only the global encoder is removed. This would be a more meaningful experiment. Moreover, the discussion about the ablation study is very shallow and at the level of “every design improves the results”. There is no good answer to why it improves and in which sense it improves results. The authors should extend explanations here.\n- Missing reference for image to 3D avatar\n    - Teotia et al. 2024: GaussianHeads: End-to-End Learning of Drivable Gaussian Head Avatars from Coarse-to-fine Representations\n\nGiven the current weaknesses, I slightly lean towards reject. However, the authors may be able to address some of the concerns during the rebuttal period.\n\nQUESTIONS:\nl.146 How does the interpolation strategy works? I cannot find much details in the manuscript? Does it assume correspondence between two different instances? If so, how can this be guaranteed?\n\nIs NFR retrained on the same data? Otherwise, the comparison is not conclusive as it is not clear whether the improvement comes from data or the algorithmic design. The authors should clarify.\n\nLIMITATIONS:\nLimitations are discussed in the conclusion section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces RigAnyFace, a scalable and generalizable neural auto-rigging framework for facial meshes with diverse topologies, including those with disconnected components like eyeballs. Its key goal is to automatically create FACS-based blendshape rigs from a neutral 3D facial mesh. \n\nTo achieve this goal, the paper use 2 stage training pipeline. First, train on both rigged and unrigged data using 2D losses. Then, fine-tune on rigged data with 3D supervision for higher precision. The model part is built upon DiffusionNet, use a global encoder to capture holistic information across disconnected components and conditional diffusion blocks to integrate FACS parameters directly into the deformation prediction process.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Strong Methodological Innovation: The paper presents a well-designed auto-rigging framework that combines a modified DiffusionNet backbone with conditional FACS encoding and a global encoder to handle disconnected mesh components. These architectural modifications are technically sound and justified.\n- Well-structured: the paper is logically organized, beginning with motivation, followed by detailed method descriptions, training strategy, and comprehensive evaluations. Figures are clear and informative.\n- Support for Disconnected Components: A significant practical contribution is the ability to rig meshes with disconnected elements like eyeballs—something most prior works fail to address.\n\nWeaknesses\n- Limited Analysis of Failure Cases: The limitations section briefly mentions mesh discretization and poor mesh quality as failure modes, but these are not empirically analyzed or visualized in the main paper.\n- Incremental Architecture Changes: While effective, the architectural improvements to DiffusionNet (e.g., conditional blocks, global encoder) are relatively incremental. Much of the novelty lies in the integration and system design, rather than entirely new algorithms.\n- Two-Stage Training with 2D and 3D Supervision: 3D mesh learning with 2D supervision already have many prior works. The originality and novelty of the framework and loss functions are limited here.\n- Evaluation Setup: This work only performed a comparison with NFR in the evaluation, lacking more comprehensive analysis with other SOTA methods.\n\nQUESTIONS:\n- Could you include visual or quantitative examples of where the model fails or underperforms?\n- How is the model's ability to handle extreme expressions and postures of driving images?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces RigAnyFace, a framework for facial meshes of diverse topologies. RAF deforms a neutral mesh into FACS poses to create a blendshape rig, using a triangulation-agnostic DiffusionNet backbone augmented with two novel components: (1) a global encoder to handle disconnected parts (2) FACS conditioning blocks to integrate action-unit parameters. To address limited 3D supervision, RAF leverages 2D supervision generated via a pretrained 2D face-animation model for unrigged neutral meshes. This strategy scales training data, enhancing generalization. Experiments show RAF outperforms prior work on artist-crafted and in-the-wild meshes in accuracy and topological flexibility.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Robust technical design effectively handles disconnected components via the global encoder and achieves precise expression control through FACS conditioning blocks.\n- Superior performance outperforms NFR by >45% in MAE on humanoid heads and generalizes to non-humanoid/out-of-distribution meshes.\n- Significant practical impact eliminates template dependency, supports multi-component meshes, and reduces 3D data reliance via scalable 2D supervision, enabling applications in text-to-3D rigging and video retargeting.\n- Pioneering contributions are the first to support disconnected components in auto-rigging and introduce 2D supervision for 3D deformation networks.\n\nWeaknesses:\n- Global feature bottleneck may compress fine details into a single vector, limiting performance on complex meshes.\n- Degraded performance on shell-like meshes occurs when inputs lack fine geometric details.\n- Unclear handling of stylized meshes in the 2D supervision pipeline; fine-tuning details for non-standard topologies need clarification.\n- Suboptimal inference speed could hinder real-time deployment, Moderate originality as core innovations build incrementally upon DiffusionNet.\n\nQUESTIONS:\nRAFT optical flow and MegActor may struggle with highly stylized/non-humanoid meshes. How significant is the domain gap for these models? Provide error analysis of generated 2D data vs. artist annotations.\n\nThe global encoder compresses all mesh components into one vector, potentially losing part-specific details. Could a hierarchical encoder improve detail preservation?\n\nHow does RAF handle cases where the input mesh contains severe discretization artifacts, such as shell-like geometry or low-resolution scans? Are there any preprocessing recommendations or failure cases to be aware of? Would data augmentation or mesh-denoising pre-processing mitigate this?\n\nLIMITATIONS:\nplease see the Questions section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}