SurFhead: Affine Rig Blending for Geometrically Accurate 2D Gaussian Surfel Head Avatars

Authors: Jaeseong Lee, Taewoong Kang, Marcel Buehler, Min-Jung Kim, Sungwon Hwang, Junha Hyung, Hyojin Jang, Jaegul Choo
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: dynamic head avatars, rigging, inverse-graphics

Abstract:
Recent advancements in head avatar rendering using Gaussian primitives have achieved significantly high-fidelity results. Although precise head geometry is crucial for applications like mesh reconstruction and relighting, current methods struggle to capture intricate geometric details and render unseen poses due to their reliance on similarity transformations, which cannot handle stretch and shear transforms essential for detailed deformations of geometry. To address this, we propose SurFhead, a novel method that reconstructs riggable head geometry from RGB videos using 2D Gaussian surfels, which offer well-defined geometric properties, such as precise depth from fixed ray intersections and normals derived from their surface orientation, making them advantageous over 3D counterparts. SurFhead ensures high-fidelity rendering of both normals and images, even in extreme poses, by leveraging classical mesh-based deformation transfer and affine transformation interpolation. SurFhead introduces precise geometric deformation and blends surfels through polar decomposition of transformations, including those affecting normals. Our key contribution lies in bridging classical graphics techniques, such as mesh-based deformation, with modern Gaussian primitives, achieving state-of-the-art geometry reconstruction and rendering quality. Unlike previous avatar rendering approaches, SurFhead enables efficient reconstruction driven by Gaussian primitives while preserving high-fidelity geometry.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730375644523

thread_type:
official_review

<thread_object>
creator: reviewer_cr49

<content>
summary:
The paper proposes a method that handles stretch and shear transforms essential for detailed deformations of geometry utilizing intricate deformations driven by the affine Jacobian gradient instead of similarity transformation and corresponding normal adjustments.

strengths:
1) Jacobian Blend Skinning (JBS): The use of Jacobian Blend Skinning (JBS) enables natural interpolation of affine transformations across adjacent deformations, effectively reducing discontinuities in transitions.

2) Cornea Opacity Constraint: To address the specular highlights in the eyeball region, the method constrains the corneal regions to remain opaque by regularizing the opacity of the respective Gaussians.

weaknesses:
1) Detail Representation: In Figure 5 (bottom row), there seems to be a lack of finer details, such as wrinkles. Adding more visual comparisons or details on addressing such high-frequency features could strengthen the analysis.

2) Rendering Speed and FPS: Given that methods like 3DGS/2DGS achieve real-time rendering, the speed of deformable-driven methods may be a limitation for applications requiring real-time animation. Could you report the FPS compared to other methods to clarify performance in time-sensitive scenarios?

questions:
1) Related Works: I think some related works of static reconstruction could be further discussed, such as H3DS [1], deformable model-driven approaches [2], and Implicit Neural Deformation methods [3]. These methods leverage 3D points from SfM, multi-scans, or multi-view data from various identities to enhance reconstruction under sparse-view conditions, although they primarily focus on static human face or head geometry reconstruction.

Ref:

[1] Ramon, Eduard, et al. "H3d-net: Few-shot high-fidelity 3d head reconstruction." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[2] Xu, Baixin, et al. "Deformable model-driven neural rendering for high-fidelity 3D reconstruction of human heads under low-view settings." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[3] Li, Moran, et al. "Implicit Neural Deformation for Sparse‐View Face Reconstruction." Computer Graphics Forum. Vol. 41. No. 7. 2022.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730571780401

thread_type:
official_review

<thread_object>
creator: reviewer_rtvj

<content>
summary:
This paper contributes a novel representation for geometrically accurate head avatars within the 3D Gaussian Splatting framework, a method for natural interpolation of affine transformations across adjacent deformations, and enhancements to the realism of corneal representations in head avatars. These contributions advance the state of the art in personalized head avatar construction and have the potential to improve various applications in computer graphics, virtual reality, and beyond.  The key contributions include:

* Introduction of SurFhead Model: The paper introduces SurFhead, the first geometrically accurate head avatar model within the Gaussian Splatting framework. This model is designed to capture the deformation of head geometry using intricate affine rigging that combines Gaussians and their normals solely from RGB videos.

* Jacobian Blend Skinning Algorithm: To address the issue of discontinuities between adjacent triangles in head deformations, the paper proposes the Jacobian Blend Skinning (JBS) algorithm. This algorithm blends adjacent transformations while avoiding geometric distortions by linearizing the non-linear matrix interpolation space, leveraging classical matrix animation techniques and geometrically smooth polar decomposition.

* Enhancement of Corneal Convexity and Specularity: The paper addresses the hollow illusion in the cornea by regularizing corneal convexity and enhancing specularity using computationally efficient Anisotropic Spherical Gaussians (ASGs). This improvement ensures a more realistic representation of the cornea in the head avatar.

strengths:
The paper introduces SurFhead, a novel model within the Gaussian Splatting framework that captures geometrically accurate head deformations. This representation utilizes intricate affine rigging combined with Gaussians and their normals, solely based on RGB videos, which is a significant advancement in achieving realistic and detailed head avatars.  The proposed Jacobian Blend Skinning (JBS) Algorithm is technically sound.  The paper tackles the problem of the hollow illusion in the cornea, where a concave surface appears convex due to the prioritization of photometric losses during training. 

The methods presented in the paper are demonstrated to achieve superior results across a variety of subjects, including real and synthetic data. They excel in challenging scenarios such as sharp reflections on convex eyeballs, fine geometric details, and exaggerated deformations, showcasing the robustness and effectiveness of the proposed approach.

weaknesses:
The method is built upon the foundation of 2D Gaussian Splatting, and I believe that the proposed method's success in recovering a superior surface geometry owes much to this solid groundwork. Indeed, the authors introduced several improvements on this basis, such as intricate affine rigging, but I consider these innovations to be more incremental improvements rather than groundbreaking advancements.

The geometric accuracy of the experimental results appears to exhibit significant variation: some achieve hair-level geometric detail, while others fail to recover the structure of the hair. Consequently, whether this variation stems from instability in the algorithm or differences in the data quality of various training datasets arises. I hope the author can provide more analysis and discussion on this issue.

questions:
As elaborated in the "weaknesses", the geometric accuracy of the experimental results exhibits considerable variation,  and we hope that the authors can further enhance their analysis and discussion on this crucial point.

In terms of experimental design, the NeRSemble dataset, while containing accurate 3D mesh models, lacks sufficient detail, and obtaining such precise models can often be challenging in practical applications.     In this regard, we eagerly inquire whether the proposed method heavily relies on such relatively accurate 3D mesh models, and how it would perform in their absence.     To validate this, the authors could consider using videos they have shot or sourced from the internet as input, employing monocular facial 3D reconstruction algorithms (such as DECA) to obtain mesh sequences, or directly bypassing the use of 3D mesh models altogether.     We are anticipating and curious about the results of such experimental setups.    We encourage the authors to actively explore and propose potential strategies for adapting their proposed approach to scenarios where detailed 3D mesh models are unavailable.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730587798199

thread_type:
official_review

<thread_object>
creator: reviewer_u9qy

<content>
summary:
This paper aims at better geometry estimation in head modeling. It replaces the 3D Gaussian splatting with 2D Gaussian splatting to better model the surface. Moreover, it addresses three issues in existing works with three novel components: 1) To compensate for the incorrect deformation of 2D Gaussians due to the triangle's shear and stretch, it proposes the Jacobian deformation; 2) To mitigate the discontinuities in adjacent triangles, it improves linear blend skinning with Jacobian blend skinning; 3) To resolve hollow illusion in eyeballs, it replaces Spherical Harmonics with Anisotropic Spherical Gaussians. It demonstrates that it outperforms the state-of-the-art regarding normal similarity and remains comparable in rendering quality.

strengths:
* It motivates each contribution properly.
* The use of Jacobian deformation and Jacobian blend skinning in the context of head modeling looks novel to me.
* The qualitative results and normal similarity evaluation demonstrate better geometry compared to baselines.
* The effectiveness of each component is well studied.

weaknesses:
* It is unclear how the deformation gradient $J$ in Sec 2.2 and the blended Jacobian $J_b$ in Sec 2.3 connect. In Line 258 to 261, it mentions that it replaces the original deformation in GaussianAvatar with a new deformation $J_b$, and $J$ appears only as a parameter of JBS in Eq. 2. What is the relationship between $J$ and $U_i, P_i$ in Eq. 2? Which transformation, $J$ or $J_b$, is used in the final method?  
* The paper measures the normal similarity between ground-truth normals and rendered normals from 2D Gaussians. However, since this work claims to achieve a better geometry, evaluating metrics that apply to meshes, such as Chamfer distance or normals rendered from the mesh is more informative when judging the geometry. Although previous methods use normals rendered from 2D Gaussians as proof of geometry, the link between it and the mesh quality still looks vague to me.
* As the rendering quality is only on par with state-of-the-art, e.g., GaussianAvatar, the comparison of training and rendering speed is missing.

questions:
* It is unclear to me how Jacobian blend skinning improves upon linear blending skinning. It seems to me that $J_b$ introduces spatial smoothness to Gaussians' deformation matrices. Is the vertices of the mesh still transformed by linear blending skinning before the Jacobian blend skinning is applied? Also, could the author clarify Fig. 2(b)? What are the meanings of the green and yellow lines and the weights? How do they connect to the deformation of the triangle meshes?
* The paper mainly focuses on improving the geometry. Since 2D Gaussian splatting is known to produce better geometry than 3D Gaussians, how much does 2D Gaussians help improve the geometry, compared to the components proposed in the paper?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731083413296

thread_type:
official_review

<thread_object>
creator: reviewer_d2nv

<content>
summary:
This paper presents a method for learning head avatars based on 2D Gaussian splatting. To make the Gaussian surfels better handle the stretch and shear deformation under extreme poses and facial expressions, this paper introduces affine transformation derived from Jacobian deformation gradient of the surface. Normal orientations are calculated accordingly. Moreover, the authors propose Jacobian blend skinning to interpolate these affine transformations to ensure surface smoothness. Results show that the proposed method is able to reconstruct drivable head avatars with high-quality geometry.

strengths:
* This paper introduces better deformation modeling techniques for Gaussian surfels. Compared to existing works, the proposed method is more reasonable and can handle more extreme deformations such as stretching and shearing. The proposed technique could be useful in other related research topics beyond head avatar modeling. 

* The proposed method is able to reconstruct fine geometric details, outperforming existing baselines by a large margin. 

* The paper is overall well-written and easy to follow.

weaknesses:
* Missing comparison against Gaussian Head Avatar (GHA) [Xu et al. 2023], which is a state-of-the-art head avatar method in terms of image synthesis quality. Although the authors have already compared with SplattingAvatar and GaussianAvatar, I think an additional comparison against GHA is also necessary because GHA demonstrates high-resolution image synthesis with the assistance of a super-resolution module. 

* It would be better if the authors report the training time and rendering speed. One important advantage of Gaussian splatting is its efficiency. I wonder whether the proposed techniques (such as Jacobian blend skinning) hinders this advantage or not. 

* It is not clear how the proposed method performs for subjects wearing eye-glasses. NeRSemble dataset contains cases with eye-glasses, but they are suspiciously skipped in the experiments.

questions:
See [Weaknesses].
</content>
</thread_object>

</thread>
<thread>
creation date:
1732114016990

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We would like to thank all reviewers for their constructive feedback and insightful comments. We especially appreciate that reviewers consider our work novel and sound *(R u9qy, R RtVj)*, effectiveness proved *(R D2NV, R u9qy, R RtVj, R Cr49)*,  thoroughly evaluated *(R D2NV, R u9qy, R RtVj)*, and versatile *(R D2NV)*.

We have responded to the comments of each reviewer and subsequently revised both our manuscript and Appendix, as outlined below.

> ### To R D2NV
#### 1. We compared SurFhead with an additional baseline, GHA [1].
#### 2. We trained an additional subject wearing eyeglasses (to be conducted as soon as possible).

> ### To R u9qy
#### 1. We clarified the role of $\mathbf{J}_{b}$.
#### 2. We adopted another metric, Chamfer Distance, to evaluate geometric quality.
#### 3. We provided additional qualitative results in extreme scenarios.
#### 4. We added further explanation of Fig. 2(b).
#### 5. We analyzed the effect of changing 3D Gaussians to 2D Gaussians.

> ### To R RtVj
#### 1. We highlighted the innovation of SurFhead.
#### 2. We analyzed the robustness of the geometric quality of hair strands.
#### 3. We trained SurFhead on monocular datasets with coarse meshes.

> ### To R Cr49
#### 1. We analyzed the fine-detailed reconstruction quality.
#### 2. We addressed the missing related work.

We sincerely thank our reviewers and look forward to further discussions.


Below is the training time and test time comparison *(R D2NV, R u9qy, R Cr49)* to highlight the efficiency of our method.

>
> ## **Training and Rendering Time Complexity Measurement**
>

The table below summarizes the rendering speed and training time for our method and GaussianAvatars **[2]**. The *Base* configuration refers to replacing the 3D Gaussian Splatting rasterizers in GaussianAvatars with their 2D counterparts. Our method incurs only a **17\% drop in rendering speed** and an **additional 25 minutes of training time**, while still achieving **3$\times$ real-time rendering speeds** (generally over 30 FPS) and maintaining efficient training. A detailed analysis attributes the minimal training and testing overhead to our GPU-level CUDA kernel implementation for Jacobian computations.

For rendering, the primary factor behind the speed reduction is the *Jacobian Blend Skinning (JBS)*, where the overhead mainly arises from the Polar Decomposition step. As described in Section A.3 of the Appendix, our current implementation utilizes PyTorch's SVD, which relies on the cuSOLVER backend. To further investigate this bottleneck, we conducted additional experiments using RoMA **[3]**  's specialized Procrustes routine, which is designed to efficiently compute the $3\times3$ unitary matrix $\mathbf{U}$ of the Jacobian $\mathbf{J}$. Notably, replacing `torch.svd` with `roma.special_procrustes` **yielded a performance gain of approximately 4–5 FPS.**

Although this demonstrates the potential of alternative approaches, there is still room for further improvement. Higham's routine **[4]** , specifically tailored for $3 \times 3$ matrices, offers a promising direction to address this overhead and is well-suited for CUDA-based implementations.

| Method           | GA   | Base   | +Jacobian | +JBS  | +eyeballs (=Ours) |
|------------------|-------|--------|-----------|--------|-------------|
| **Rendering Speed (FPS)**                                                                 |
| `torch.svd`        | 71.18 | 109.36 | 107.59    | 92.62  | 90.13       |
|  `roma.special_procrustes`           | N/A   | N/A    | N/A       | 97.29  | 94.72       |
| **Training Time (hours)** | 1.65   | 1.68    | 1.73       | 1.98   | 2.11       |



> *Reference*

**[1]** Xu, Yuelang, et al. "Gaussian head avatar: Ultra high-fidelity head avatar via dynamic gaussians." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

**[2]** Qian, Shenhan, et al. "Gaussianavatars: Photorealistic head avatars with rigged 3d gaussians." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

**[3]** Brégier, Romain. "Deep regression on manifolds: a 3d rotation case study." 2021 International Conference on 3D Vision (3DV). IEEE, 2021.

**[4]** Higham, Nicholas J., and Vanni Noferini. "An algorithm to compute the polar decomposition of a 3× 3 matrix." Numerical Algorithms 73 (2016): 349-369.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a new model within the Gaussian Splatting framework designed to capture realistic and detailed head deformations from RGB videos. The model enhances geometric accuracy and detail, particularly in complex scenarios like sharp reflections and exaggerated deformations, with good efficiency. The authors provide a substantial improvement over existing methods in terms of geometric detail reconstruction.

***Strengths:***
- The introduction of Jacobian deformation and blend skinning is novel and effectively enhances the model's ability to handle complex - deformations like stretching and shearing.
- The proposed method reconstructs fine geometric details more accurately than existing baselines.
- The paper is clearly written and easy to follow, aiding in its comprehension and potential replication.

***Weaknesses:*** 
The lack of results like comparison with GHA, glasses wearing cases, robust demonstrations etc.
Missing details on training time and rendering speeds.

Despite some concerns, the reviewers are generally positive, highlighting the paper’s innovative approach and its effectiveness in handling complex scenarios. The authors have also adequately addressed the issues with added experimental results.  

After careful discussion and consideration, we are pleased to inform this paper is accepted. The paper is accepted based on its contributions to geometric detail reconstruction in head avatars, its novel methodological advancements, and the overall positive reception from reviewers.