Title: Low-Rank Head Avatar Personalization with Registers
Venue: NeurIPS 2025
Accepted: yes
Keywords: Personalization

Avatar Generation
Abstract: We introduce a novel method for low-rank personalization of a generic model for head avatar generation. Prior work proposes generic models that achieve high-quality face animation by leveraging large-scale datasets of multiple identities. However, such generic models usually fail to synthesize unique identity-specific details, since they learn a general domain prior. To adapt to specific subjects, we find that it is still challenging to capture high-frequency facial details via popular solutions like low-rank adaptation (LoRA). This motivates us to propose a specific architecture, a Register Module, that enhances the performance of LoRA, while requiring only a small number of parameters to adapt to an unseen identity. Our module is applied to intermediate features of a pre-trained model, storing and re-purposing information in a learnable 3D feature space. To demonstrate the efficacy of our personalization method, we collect a dataset of talking videos of individuals with distinctive facial details, such as wrinkles and tattoos. Our approach faithfully captures unseen faces, outperforming existing methods quantitatively and qualitatively.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of personalizing head avatars by adapting a pretrained generalized avatar generation model to a target subject using short videos of 4-15 seconds. The key technical contribution is a \"Register Module\" that enhances low-rank adaptation (LoRA) by combining local per-vertex embeddings with DINO-v2 features. The authors also contribute a video dataset of 50 subjects with distinct facial features for evaluation. They compare their method against 2 other approaches using perceptual quality (LPIPS) and identity similarity (Arcface embeddings) metrics. I would consider the potential impact as limited due to its specificity to LoRA finetuning of pretrained generalized avatar models.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n+ The results with LoRA alone are indeed overly blurry and a better personalization method is needed.\n+ The results after personalization improve details over the baseline (GAGAvatar)\n+ The paper is easy to read.\n+ The paper promises to release the collected dataset.\n\n## Weaknesses\n- Minimal Ablation\n    - The ablation study is minimal and fails to analyze the most important component of the proposed method. The ablation study adds noise (a) and learnable embeddings to the DINO features f_\\text{sense}^\\text{src}, i.e., it studies the effect of the DINO input features (Sec. 4.3, Fig. 3). The correct component to study, however, would be the proposed register module (blue box in Fig. 3). A more meaningful ablation study could replace the learned vertex features with noise or the xyz positions or it could ablate the need for processing the rasterized features with E_\\text{proc-1}. At minimum, the ablation study should provide numbers for the model variant that uses LORA vs. LORA + registers. \n- Limited comparisons\n    - The paper proposes a method for talking head generation from a video. There are dozens of relevant works in the literature, but the paper only compares with a feedforward method (GagAvatar) without personalization and a more generic method, which does not use a 3D Morphable Face Model (LoRA). Given the similar methodology, the comparison with GagAvatar and LoRA is important, but the comparisons should include at least the state-of-the-art for avatar generation from videos like SplattingAvatar, FlashAvatar, and MonoGaussianAvatar. Optional comparisons could be conducted by fine-tuning EG3D using Pivotal Tuning Inversion  \n- Missing citations\n    - The paper does not cite other relevant work in avatar personalization, e.g., by leveraging meta-learning (Tancik et al. 2021), data-driven priors (Buehler et al. 2024), or extensions to LoRA (Liu et al. 2024).\n\n\n## References\n\n      Roich, Daniel, et al. \"Pivotal tuning for latent-based editing of real images.\" __ACM Transactions on graphics (TOG)__ 42.1 (2022): 1-13.\n      Chen, Yufan, et al. \"Monogaussianavatar: Monocular gaussian point-based head avatar.\" __ACM SIGGRAPH 2024 Conference Papers__. 2024.\n      Shao, Zhijing, et al. \"Splattingavatar: Realistic real-time human avatars with mesh-embedded gaussian splatting.\" __Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition__. 2024.\n      Xiang, Jun, et al. \"Flashavatar: High-fidelity head avatar with efficient gaussian embedding.\" __Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition__. 2024.\n      Tancik, Matthew, et al. \"Learned initializations for optimizing coordinate-based neural representations.\" __Proceedings of the IEEE/CVF conference on computer vision and pattern recognition__. 2021.\n      Buehler, Marcel C., et al. \"Cafca: High-quality Novel View Synthesis of Expressive Faces from Casual Few-shot Captures.\" __SIGGRAPH Asia 2024 Conference Papers__. 2024.\n      Liu, Shih-Yang, et al. \"Dora: Weight-decomposed low-rank adaptation.\" __Forty-first International Conference on Machine Learning__. 2024.\n\nQUESTIONS:\n- Ablations\n    - How does the method compare numerically to LoRA and its extensions? Specifically, what are the metrics when comparing LoRA vs. the proposed LoRA+Register Module vs. Dora vs. Dora+Register Module?\n- Comparisons\n    - How does the method compare against the state-of-the-art in avatar generation from videos (SplattingAvatar, FlashAvatar, and MonoGaussianAvatar)?\n\nLIMITATIONS:\nI appreciate mentioning the potential issues about imperfect 3DMM fitting (Sec. F.1). I would also include a note about the expected diversity in the input video sequence, specifically to what extent the method is robust against limited expression and head pose diversity.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new method for personalizing head avatar models through a LoRA approach enhanced by a learnable Register Module. This module introduces a 3D feature space that captures high-frequency identity-specific facial details such as wrinkles and tattoos, which are often missing in generic models trained on large datasets. Additionally, the authors introduce a new dataset, RareFace-50, to demonstrate the effectiveness of their approach. Quantitative and qualitative evaluations on RareFace-50 and VFHQ Test datasets show improvements over baseline and naive LoRA methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is well-written and structured, making it easy to understand.\n- The proposed 3D register extension is effective for the personalization training of the face avatar.\n- Introduction of RareFace-50 fills a valuable gap for testing on underrepresented facial features.\n\nWeaknesses: \n- The technical novelty is limited: the approach primarily reuses LoRA, with the main contribution being the use of registers. However, the register mechanism closely resembles that of the recent SIGGRAPH paper LAM, which also leverages features from 3DMM geometry.\nLAM: https://github.com/aigc3d/LAM\n\n- In the experiments, the authors only apply the proposed method to the GAGAvatar method, while not proving the generalized applicability of the proposed method based on other head avatar methods.\n\n- In the experiments, the authors evaluate their method on only two datasets—VFHQ and RareFace-50—without including comparisons on other relevant datasets such as HDTF (used in GAGAvatar) or Nersemble, which is now widely adopted.\n\nHDTF: https://github.com/MRzzm/HDTF\nNersemble: https://tobias-kirschstein.github.io/nersemble/\n\nQUESTIONS:\nI would like the author to answer the following questions in order to further enhance the score:\n\n-  In the feature learning procedure starting from Line 169, why do you adopt the process of computing visible joints → projection → interpolation, instead of using a rasterization-based approach? Rasterization is a standardized and GPU-efficient method that is typically much faster and more scalable.\n- Why didn’t you evaluate your method on more datasets and with additional baseline methods (other than GAGAvatar)? Is the proposed approach only applicable to GAGAvatar?\n\nLIMITATIONS:\n- The proposed method still requires annotated 3DMM parameters for adaptation, which might limit its accessibility or deployment in the wild. This also makes the method inefficient for personalizing the head avatar. \n\n- Currently, the proposed method is only demonstrated on GAGAvatar. The authors provide no evidence of its applicability to other human head avatar methods, which limits the generalizability of the approach.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of personalizing generic, 3D-aware head avatar models to reconstruct the unique, high-frequency details of an unseen individual (e.g., wrinkles, tattoos, skin texture). The authors posit that standard adaptation techniques like Low-Rank Adaptation (LoRA) are insufficient for this task, as they tend to lose these identity-specific features.\n\nThe core contribution is a novel Register Module, designed to augment LoRA-based personalization. Inspired by registers in Vision Transformers that store global information, this module introduces a learnable 3D feature space that operates on intermediate features of a pre-trained model. The purpose of this module is to explicitly capture, store, and re-purpose the distinctive facial details of a target identity during the adaptation process.\n\nTo validate their method, the authors show through quantitative and qualitative examples that their approach outperforms standard LoRA and a meta-learning baseline in terms of both visual quality and identity preservation.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n**Well-Motivated Problem**: The paper targets a clear and significant problem. As generic avatar models become more common, the need for high-fidelity, efficient personalization is crucial. The observation that vanilla LoRA fails to capture high-frequency details is an excellent and practical motivation for this work.\n\n**Simple and Intuitive Technical Contribution**: The proposed \"Register Module\" is a clever idea. Adapting the concept of \"registers\" from 2D Transformers to a 3D feature space for identity preservation is an insightful contribution. The intuition that LoRA's low-rank updates might be too coarse to handle fine details, thus requiring a dedicated \"memory\" or \"cache\" for them, is compelling.\n\n\n**Weaknesses**:\n\n**Lack of Architectural Detail**: The paper is surprisingly light on the specific architectural details of the Register Module. While the high-level concept is explained, it is unclear how this 3D feature space is implemented, how it interacts with the intermediate features of the DINOv2 backbone, and how information is \"stored\" and \"re-purposed\". This lack of clarity makes it difficult to fully assess the technical novelty and replicate the work from the paper alone.\n\n**Unsubstantiated Efficiency Claims**: The abstract and introduction claim the method requires \"a small number of parameters\" and retains \"the original inference speed.\" These are important claims for a personalization method but are not substantiated with data. The paper needs to report the parameter count of the Register Module and provide concrete numbers on the impact on inference time (e.g., FPS) compared to the vanilla LoRA baseline.\n\n**Limited Discussion on Generalizability**: The method is presented and validated on a single backbone architecture (GAGAvatar). While effective, this raises questions about the generalizability of the Register Module. A discussion on whether this module is a general-purpose \"plug-in\" for other 3D-aware generative models would significantly broaden the paper's impact.\n\nQUESTIONS:\n**Clarification of the Register Module**: Could you please provide a more detailed architectural description of the Register Module? The CNN-based encoder is mentioned but not illustrated. A small diagram illustrating how it takes intermediate features as input, processes them through the 3D learnable space, and outputs the enhanced features would be immensely helpful. \n\n**Quantify Efficiency Claims**: You claim the method is efficient in terms of parameters and speed. Could you please update the paper to include:\n\n(a) The number of trainable parameters for the vanilla LoRA baseline versus your method (LoRA + Registers).\n\n(b) The impact on inference speed (e.g., frames per second or milliseconds per frame) for both methods.\n\n\n**Temporal Consistency**: It seems that the register module rely heavily on the 3DMM prediction, does the video-level prediciton is done frame-wise? How does the authors improve temporal consistency?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript proposes a personalized head modeling approach for the domain of personalized avatar modeling, combining low-rank adaptation (LoRA) with a three-dimensional register module. The method aims to enhance the modeling of high-frequency, personalized facial details with minimal parameter updates, effectively addressing the limitations of traditional general-purpose models and existing efficient fine-tuning methods in detail preservation.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The overall logic of the paper is fairly clear, and the writing is well-structured.\n- The core innovation lies in embedding and projecting additional vertex features of the 3D facial mesh through a register module, enabling a deep integration of geometric and visual features. Furthermore, detail interpolation and feature fusion are achieved using a convolutional network.\n\n### Weaknesses\n- Its strong dependency on the Register Module's 3DMM parameter accuracy means that errors in pose estimation (from occlusions or extreme angles) cascade, severely degrading the geometric and detailed accuracy of the final image. Another significant drawback is its inability to preserve clothing details, a capability present in earlier keypoint-based methods like FOMM.\n- Specific lighting effects from the source image, such as highlights and shadows, are highly prone to being \"baked\" into the model's texture map or implicit appearance representation. This makes them part of the character's intrinsic identity, which severely limits the model's realism and generalization capability under different lighting conditions.\n- During training, the reconstruction branch receives a highly processed feature, $f_{reg}$; however, during inference, it receives a more \"raw\" DINOv2 feature, $f_{dense}^{src}$. This distribution gap between the training and inference inputs is a potential cause for the model's sub-optimal performance at test time.\n\nQUESTIONS:\n1. The regularization loss $L_{reg}$ in Equation (9) imposes a similarity constraint across all vertex embeddings. While this is effective for preventing overfitting, it may also lead to over-smoothing of features, thereby suppressing the inherent distinctiveness of local regions such as the lips and eyes. The weight for this loss, $\\lambda_{reg}$, is set to 20, which is significantly higher than the weight of 2 for the feature alignment loss. Could the authors provide a justification for this choice? I recommend adding an ablation study on the selection of the hyperparameter $\\lambda_{reg}$.\n2. Following up on question 1, what specific information do the embeddings corresponding to each vertex actually learn? Are there any qualitative conclusions that can be drawn about the learned representations?\n3. Could the authors clarify the rationale for summing the `source` features with the features that have been projected by the Register Module? It seems these two sets of features are not aligned to the same viewpoint.\n4. I have a question regarding the inference process. The paper states that `$f^{src}_{dense} can be used alone for inference. Could the authors elaborate on the reasoning behind this? Specifically:\n- Is the underlying assumption that the LoRA module inherits the capability to render fine-grained details?\n- Given that the input to `$f^{src}_{dense} does not seem to contain sufficient high-frequency information, how does the model generate these details effectively?\n5. Regarding the quantitative experiments, why were significantly fewer evaluation metrics used compared to those in GAGAvatar?\n\nI will consider raising my score after the authors have addressed my concerns.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}