SurFhead: Affine Rig Blending for Geometrically Accurate 2D Gaussian Surfel Head Avatars

## Abstract

Abstract Recent advancements in head avatar rendering using Gaussian primitives have achieved significantly high-fidelity results. Although precise head geometry is crucial for applications like mesh reconstruction and relighting, current methods struggle to capture intricate geometric details and render unseen poses due to their reliance on similarity transformations, which cannot handle stretch and shear transforms essential for detailed deformations of geometry. To address this, we propose SurFhead , a novel method that reconstructs riggable head geometry from RGB videos using 2D Gaussian surfels, which offer well-defined geometric properties, such as precise depth from fixed ray intersections and normals derived from their surface orientation, making them advantageous over 3D counterparts. SurFhead ensures high-fidelity rendering of both normals and images, even in extreme poses, by leveraging classical mesh-based deformation transfer and affine transformation interpolation. SurFhead introduces precise geometric deformation and blends surfels through polar decomposition of transformations, including those affecting normals. Our key contribution lies in bridging classical graphics techniques, such as mesh-based deformation, with modern Gaussian primitives, achieving state-of-the-art geometry reconstruction and rendering quality. Unlike previous avatar rendering approaches, SurFhead enables efficient reconstruction driven by Gaussian primitives while preserving high-fidelity geometry.

## 1 Introduction

Figure: Figure 1: SurFhead reconstructs photo-realistic head avatars and high-fidelity surface normals, depth, and meshes from RGB videos alone. These avatars are represented through affine rigging of 2D surfel splats bound to a parametric morphable face model. SurFhead can fully control poses, expressions, and viewpoints, enhancing both appearance and geometry.
Refer to caption: x1.png

The construction of personalized head avatars has seen rapid advancements in both research and industry. Among the most notable developments in this field is the Codec Avatar family (Ma et al., ; Saito et al., ), which aims to reconstruct highly detailed, movie-quality head avatars using high-cost data captured from head-mounted cameras or studios. This approach has spurred significant research efforts to bridge the gap between high-cost and low-cost capture systems by utilizing only using RGB video setups. Neural Radiance Fields (NeRFs) (Mildenhall et al., ) have further accelerated these efforts with their topology-agnostic representations. As a result, numerous NeRF-based methods (Gafni et al., ; Athar et al., ; Zielonka et al., ) for constructing head avatars from RGB videos have emerged, demonstrating potentials of improving high-cost systems (Ma et al., ; Yang et al., ; Saito et al., ).

Recently, 3D Gaussian Splatting (3DGS) (Kerbl et al., ) has been used to create photo-realistic head avatars. However, there has been no attempt to create geometrically accurate head avatars within the 3DGS framework. While other representation-based methods (Bharadwaj et al., ; Zheng et al., ; ; Grassal et al., ) demonstrate plausible geometry using implicit or explicit representations, they still suffer from suboptimal results;
over-smoothed geometry from implicit neural representations or inherently limited explicit representations such as 3D Morphable Face Model (3DMFM) learned meshes or inflexible points.

In response, we introduce SurFhead, the first geometrically accurate head avatar model within the Gaussian Splatting framework (Kerbl et al., ), designed to capture deformation of head geometry. Our method integrates intricate affine rigging by combining Gaussians and their normals using only RGB videos. Building on 2DGS (Huang et al., ) with 2D surfel disks and adopting 3DMFM mesh binding from prior works (Qian et al., ; Shao et al., ; Zielonka et al., ; Lombardi et al., ), we address limitations of previous Gaussian-based methods that rely solely on rigid, isotropic transformations. This strategy often leads to undesirable deformations during pose extrapolation when 3DMFM meshes stretch in certain directions (Fig. 2(a)). To address these issues, we extend affine transformations beyond simple similarity transformations to handle shear and stretch, which often occur in extreme poses. However, applying the same affine transformation to normals can lead to incorrect deformations, violating the orthogonality of their primitive-normals. To correct this, we introduce the inverse-transpose of the affine transformation. In previous similarity transformations, this step was unnecessary because rotations and scaling were handled separately, and the rotation matrix inherently preserved normal directions due to its invariance under the inverse-transpose operation.

Our method is fundamentally based on 3DMFM mesh binding inheritance, similar to GaussianAvatars (Qian et al., ), which uses 3DMFM-based mesh triangle deformation. However, if we only consider local deformation, discontinuities can occur between adjacent triangles (Zielonka et al., ), and the method becomes limited in capturing deformations beyond the parametric head model. To address these discontinuities, previous methods (Zielonka et al., ; Shao et al., ) blend the transformations of adjacent triangles using element-wise summation. However, as illustrated in Fig. 2(b), this naive element-wise blending results in unnatural geometric interpolation within the matrix space. To overcome this issue, we propose the Jacobian Blend Skinning algorithm, which blends adjacent transformations while avoiding geometric distortions. This algorithm leverages the concept of linearizing the non-linear matrix interpolation space, drawing on classical matrix animation techniques (Shoemake & Duff, ) and employing geometrically smooth polar decomposition. Since the interpolation space of affine transformations is inherently non-linear, simple element-wise operations can lead to distortions (Fig. 2(b)).

Last but not least, we have witnessed a hollow illusion in the cornea, where a concave surface appears convex. This occurs during training as the model prioritizes photometric losses, constructing a concave retinal surface filled with low-bandwidth Spherical Harmonics (SHs) color to partially maintain specularity. However, this approach misleads the loss function by simulating mirror-like specularity with SHs. To address this, we regularize corneal convexity and enhance specularity using computationally efficient Anisotropic Spherical Gaussians (ASGs) (Xu et al., ).

Figure: (a)
Refer to caption: x2.png

To summarize our key contributions,

- •
We introduce a novel representation for geometrically accurate 2D Gaussian primitive-based head avatars, utilizing intricate deformations driven by the affine Jacobian gradient instead of similarity transformation and corresponding normal adjustments.
- •
We propose a method called Jacobian Blend Skinning (JBS) to naturally interpolate affine transformations across adjacent deformations, effectively mitigating discontinuities, such as those arising from unseen pose extrapolation cases due to variations in a local space.
- •
We demonstrate the advantages of our methods across a variety of subjects captured with real and synthetic data, achieving superior results in challenging scenarios, such as sharp reflections on convex eyeballs, fine geometric details, and exaggerated deformations.

## 2 Proposed Method

This section highlights the key technical contributions of our work. We introduce a novel representation for geometrically accurate 2D Gaussian primitive-based head avatars, alongside a geometrically precise Jacobian deformation gradient and corresponding principled normal deformation (Sec. 2.2). Our proposed Jacobian Blend Skinning (JBS) approach (Sec. 2.3) allows for the interpolation of affine transformations, effectively addressing the discontinuities caused by piece-wise deformation in meshes. Additionally, we tackle the concavity (hollow-illusion) issue observed in the cornea by regularizing the modern 3DMFM model, FLAME (Li et al., )’s eyeball model, to better manage high specularity through the use of Anisotropic Spherical Gaussians (ASGs) (Xu et al., ) (Sec. 2.4). We begin with preliminary on Gaussian Splatting and GaussianAvatars in Sec. 2.1.

### 2.1 Preliminary

The two key building blocks of our approach are 2D Gaussian Splatting (2DGS) (Huang et al., ) and GaussianAvatars (Qian et al., ). The former provides intricate geometric properties, such as depth and normal. The latter employs a mesh-based rule-governed rigging strategy, allowing primitives to be rigged directly according to mesh triangles.

#### 2.1.1 Gaussian Splatting

3D Gaussian Splatting (Kerbl et al., ) (3DGS) reconstructs a scene with anisotropic 3D Gaussian primitives. Each Gaussian is defined by a positive semi-definite covariance matrix $\Sigma$ that is centered at a position $\mu$. The covariance $\Sigma$ is decomposed as $\Sigma=RSS^{T}R^{T}$, where $R$ is a rotation matrix and $S$ is a diagonal scaling matrix. This covariance matrix is used when estimating each Gaussians’s function value at 3D point $x$ such as $G(x)=\text{exp}({-\frac{1}{2}(x-\mu)^{T}\Sigma^{-1}(x-\mu)})$.
Besides these geometrical properties, each Gaussian has appearance properties opacity $\alpha$ and color $c$.
To render the scene, the final color $C$ is computed by alpha-blending Gaussians after projecting them to the image plane: $C=\sum_{i=1}c_{i}\alpha_{i}G^{proj}(x)\prod_{j=1}^{i-1}(1-\alpha_{i}G^{proj}(x
)).$
$G^{proj}$ is given by evaluating the function value of 2D projection of the 3D Gaussian in image space by EWA volume splatting algorithm (Zwicker et al., ).

An extension to 3DGS, 2DGS (Huang et al., ), modifies 3D primitives to 2D “flat” surfels embedded in 3D space for reconstructing high-fidelity geometry. 2D surfels have some advantages compared with 3DGS in the respect of geometry; deriving intricate depth by ray-splat intersection algorithm (Weyrich et al., ), and closed-form modeling of the normal $n$ as cross-product of tangents $r_{1}$ and $r_{2}$. Thus, $R:=[r_{1};r_{2};n]$, $n:=r_{1}\times r_{2}$, and each scales $s_{1}$, $s_{2}$, and 1 composes the scale matrix $S$. Details are provided in 2DGS (Huang et al., ) paper.

Figure: Figure 3: Overall pipeline of SurFhead. Only from RGB videos, SurFhead constructs geometrically accurate head avatars, equipped with our intricate deformations. The Jacobian $\mathbf{J}$ covers stretch and shear deformations avoiding surface distortion. Moreover, the blended Jacobian $\mathbf{J}_{\text{b}}$ alleviates inherent local deformations’ discontinuity. Finally, elaborated modeling of eyeballs such as preservation of specularity and convexity achieves more realistic appearance and geometry.
Refer to caption: x4.png

#### 2.1.2 GaussianAvatars

GaussianAvatars (Qian et al., ) (GA) is an efficient head avatars model with FLAME (Li et al., ) mesh binding inheritance. Namely, each Gaussian has a single parent triangle in a canonical (local) space. When they are rendered in a deformed space, each Gaussian is transformed with their parent’s current state such as relative rotation matrix $R_{\textit{p}}$, relative triangle’s area volume as isotropic scale $s_{p}$, and relative barycenter position of triangle $T_{p}$ in the world space. Namely,

$$ $R=R_{p}R_{c},\mu=s_{p}R_{p}\mu_{c}+T_{p},S=s_{p}S_{c},$ (1) $$

where $R_{c}$ is canonical rotations, $\mu_{c}$ is canonical position, and $S_{c}$ is canonical scaling in parent’s triangle. Although this deformation approach is frequently utilized in certain head avatar research (Qian et al., ; Shao et al., ), we emphasize that it falls short in capturing the stretch and shear deformations that are essential for accurately extrapolating extreme expressions.

### 2.2 Affine transformation of 2D surfels

Our goal is to reconstruct geometrically accurate head avatars using the 2D surfel splatting regime, which employs a normal consistency energy (Huang et al., ). This approach implies that high-fidelity normals originate from high-fidelity depths. Previous methods only adopt rigid deformation to maintain the positive semi-definite (PSD) property of Gaussian. However, as shown in Fig. 2(a), using only rigid deformation for rigging can result in over- or under-occupying phenomena and deformation distortion.

Local Geometry Descriptor.
Let us consider a triangle in the canonical space as the matrix formed by its vertices ${v_{i}}\in\mathbb{R}^{3}$, and its edge matrix as $\mathbf{E}=[v_{1}-v_{0},v_{2}-v_{0},v_{3}-v_{0}]\in\mathbb{R}^{3\times 3}$, where $v_{3}:=v_{0}+\frac{(v_{1}-v_{0})\times(v_{2}-v_{0})}{||(v_{1}-v_{0})\times(v_{
2}-v_{0})||_{2}}$. $\tilde{\mathbf{E}}$ is the deformed version with the deformed triangle’s vertices $\tilde{v_{i}}$ following Eq. 1. Note that in GaussianAvatars, the edge matrix is defined with a normalized base, height, and normal direction. This constructs an orthogonal matrix and approximates the shear or stretching-related deformations with isotropic scaling from the average of the base and height length. Please find a detailed derivation in GaussianAvatars (Qian et al., ).

Jacobian Deformation Gradient for Surface. The deformation gradient $\mathbf{J}$ is defined as $\tilde{\mathbf{E}}\mathbf{E}^{-1}$ following (Sumner & Popović, ). We introduce affine rigging with 2D surfels. The new parameterization is $\Sigma^{1/2}=\mathbf{J}R_{c}S_{c},$
instead of $s_{p}R_{p}R_{c}S_{c}$, where $\Sigma=\Sigma^{1/2}(\Sigma^{1/2})^{T}$. We note that the Jacobian deformation gradient $\mathbf{J}$ can correct inaccuracies in deformation, such as the lack of stretch or shear awareness, which can lead to incorrect surface reconstruction. It is also important to keep in mind that the Gaussian primitives retain their physical meaning only when the positive semi-definite is kept (Kerbl et al., ). We prove whether the covariance matrix $\Sigma$ remains PSD after applying affine transformations in Appendix A.2.2.

Both depth and normals are essential for constructing high-fidelity head geometry. While recent works on 3DGS-based full-body avatars (Zielonka et al., ) and physical simulations (Xie et al., ; Jiang et al., ) have shown that Jacobian deformation gradients accurately describe deformations, no prior work has addressed normal deformation with Jacobians. We found that the inverse-transpose of the Jacobian, $\mathbf{J}^{-T}$, is the correct deformation matrix for normals, based on the principle that ”Orthogonality must be maintained in any space.” In canonical space, if there is a tangent vector $r_{c}$ in a surfel, the normal $n_{c}$ is orthogonal to tangent ($n_{c}^{T}\cdot r_{c}=0$). Then, the deformed tangent is defined as $r_{d}=\mathbf{J}r_{c}$. Let $\mathbf{A}$ be the matrix that transforms a normal vector from the canonical to the deformed space such as $n_{d}=\mathbf{A}n_{c}$. Trivially, the normal and tangent must preserve the orthogonality in the deformed space: $n_{d}^{T}\cdot r_{d}=(\mathbf{A}n_{c})^{T}\cdot\mathbf{J}r_{c}=n_{c}^{T}
\mathbf{A}^{T}\mathbf{J}r_{c}=0$. We already know that $n_{c}^{T}r_{c}=0$, then $\mathbf{A}^{T}=\mathbf{J}^{-1}$ (since $\mathbf{J}$ is non-singular). Therefore, $(\mathbf{A}^{T})^{T}=\mathbf{A}=\mathbf{J}^{-T}\qed$.
Consequently, we use the principled transformation of normal as inverse-transposed form of Jacobian, $n_{d}=\mathbf{J}^{-T}n_{c}$.

### 2.3 Jacobian Blend Skinning (JBS)

Only with the local affine transformations might introduce discontinuities among adjacent triangles. This subsection introduces an improved technique to alleviate such discontinuities:
Jacobian Blend Skinning. Specifically, we blend the local deformation with adjacent triangles’ deformations as done in previous arts (Zielonka et al., ; Shao et al., ).

Degenerate Solution of Linear Blend Skinning in Matrix Space. Linear Blend Skinning (LBS) (Badler & Morris, ) is widely used for dynamic modeling of the human body and head (Li et al., ; Loper et al., ), but it struggles with the non-linearity of rotation matrix interpolation in $SO(3)$, causing distortions in rotational transformations. Our toy experiments (Fig. 2(b)) show how element-wise linear interpolation distorts shapes. Previous works (Zielonka et al., ; Shao et al., ) also adopt this sub-optimal approach to handle local deformations. Techniques like matrix-logarithm interpolation or quaternion’s SLERP can address this but are limited to rotations.

Introducing Jacobian Blend Skinning (JBS). The Jacobian Blend Skinning (JBS) algorithm overcomes the limitations of LBS by focusing on the interpolation of Jacobian gradients, which encapsulate not only rotations but also shear and stretch, both residing in the broader $GL(3)$ (General Linear) field. The key to JBS is leveraging Polar Decomposition (PD) (Shoemake & Duff, ) to break down transformations into two meaningful components: rotation (orthogonal matrix $\mathbf{U}$) and stretch/shear (symmetric positive semi-definite matrix $\mathbf{P}$). This decomposition is unique and coordinate-independent (proof in Appendix A.2.1), making it geometrically sound for interpolation. By blending the rotation in matrix-logarithm space and the stretch/shear in linear matrix space, JBS ensures that the resulting transformations remain geometrically valid, avoiding the distortions seen with LBS (Fig. 2(b)).

JBS is applied to the Gaussian deformations. As aforementioned in Eq. 1, each Gaussian has its own parents’ deformation parameters which correspond to the current expression and pose, such as isotropic scale $s_{p}$ and rotation $R_{p}$. These parameters compose the deformed square root covariance and position of Gaussian, $\Sigma^{1/2}=s_{p}R_{p}R_{c}S_{c}$ and $\mu=s_{p}R_{p}\mu_{c}+T_{p}$, each. We replace the deformation of covariance term $s_{p}R_{p}$ to the blended Jacobian, $\mathbf{J}_{b}$, which is computed by blending adjacent triangles’ deformations.

The blending itself is performed separately for the rotational ($\mathbf{U}_{b}$) and stretch/shear ($\mathbf{P}_{b}$) components. For rotations, the blending weights are applied in the matrix-logarithm space $\mathfrak{so}(3)$, followed by an exponential mapping back into the $SO(3)$ space. For stretch/shear, the weights are applied directly in the linear matrix space, ensuring positive semi-definiteness, which avoids geometrical distortions. Mathematically, this Jacobian Blend Skinning is defined as:

$$ $\mathbf{J}_{b}:=\text{JBS}(\mathbf{J},w)=\underbrace{\text{exp}(\sum_{i\in adj .}w_{i}\text{log}(\mathbf{U}_{i}))}_{\mathbf{U}_{b}}\cdot\underbrace{\sum_{i \in adj.}w_{i}\mathbf{P}_{i}}_{\mathbf{P}_{b}}.$ (2) $$

The blending weights, $w_{i}$, in JBS are learnable convex weights that control how much influence each adjacent triangle has on the resulting blended Jacobian $\mathbf{J}_{b}$. Note that since $w_{i}$ is activated by Sigmoid function, the blended stretch/shear component $\mathbf{P}_{i}$ is also guaranteed positive semi-definite property. The exponential and logarithm mappings for rotations are performed using Rodrigues’ formula. Finally, the blended Jacobian $\mathbf{J}_{b}$ is then used to deform the canonical Gaussian’s covariance and mean position and also transform the normal.

$$ $\Sigma^{1/2}=\mathbf{J}_{b}R_{c}S_{c},\mu=\mathbf{J}_{b}\mu_{c}+T_{p}$ (3) $$

$$ $n_{d}=\mathbf{J}_{b}^{-T}n_{c}$ (4) $$

In summary, JBS improves on LBS by using Polar Decomposition to separately blend rotation and stretch/shear transformations. This approach ensures more accurate and geometrically meaningful deformations, avoiding the issues caused by linear interpolation. By operating in the appropriate spaces—matrix-logarithm for rotations and linear for stretch/shear—JBS preserves the integrity of transformations, making it a more reliable solution for complex deformations.

### 2.4 Resolving Hollow-Illusion in Eyeballs

[Uncaptioned image]: extracted/5928922/figures/hollow_illusion.png

Many studies (Park et al., ; Li et al., ; ) have witnessed that volumetric approaches often fail to produce a good approximation of the eyeball geometry and yield hollow eyes. We also have empirically found that the cornea of the human eye often appears as concave geometry without proper constriction. Since the cornea exhibits high specularity due to its multiple membranes (Dua et al., ), the limited representation capability Spherical Harmonics (SHs) can distort the geometry, unnaturally struggling to satisfy the photometric losses. This deceptive phenomenon is harmful to achieving accurate geometric representation as can be seen in the inset.

To address the issue of concave geometry around the eyeball, we eliminate the geometrical gradient on the eyeball-bound Gaussians, excluding Gaussians’ cloning and splitting. We use the FLAME eyeball mesh as an approximation for the eyeball geometry. Additionally, we regularize the opacity of the eyes to approach 1, following the energy $\mathcal{L}_{eye}$ in Sec. 3. Although this improves the geometry, SHs still fall short of capturing eye specularity.

In mitigation, we employ Anisotropic Spherical Gaussians (ASGs) (Xu et al., ). Since the cornea and sclera often reflect their light environment, ASGs excel at capturing sharp reflections. Implementation details of ASGs can be found in Appendix A.3. To preserve computational efficiency, we leverage common knowledge from casual data capture. First, eyeballs aren’t affected by light from the back of the head, so we limit the sampling range to the frontal hemisphere of the world space, reducing ASGs by 50% compared to previous methods (Han & Xiang, ). Second, given that captured data is often in environments with ample white light, representing specular reflections as a monochrome intensity channel is sufficient.

## 3 Training Strategy

### 3.1 Optimization

We supervise the rendered images with photometric loss $\mathcal{L}_{\text{photo}}$ which is a combination of L1
term $\mathcal{L}_{l1}$ and a D-SSIM term $\mathcal{L}_{\text{ssim}}$ following 3DGS (Kerbl et al., ).
Moreover, since we aim to reconstruct high-fidelity geometry, we followed the geometric energies in 2DGS (Huang et al., ), depth-distortion $\mathcal{L}_{\text{depth}}$ and normal consistency energies $\mathcal{L}_{\text{normal}}$.
Toward the better alignment between Gaussians and their parent triangles, we utilize two regularization energy terms $\mathcal{L}_{\text{scaling}}$ and $\mathcal{L}_{\text{position}}$ from GA (Qian et al., ).

Eyeball Regularization. We have empirically found that the highly specular eyeball region, especially the cornea, tends to yield overly transparent Gaussians to satisfy the photometric losses. This leads to an incorrectly shaped, concave cornea, although the genuine surface should be roughly convex (Li et al., ). To alleviate this issue, we constrain the cornea regions to be opaque by regularizing the opacity of the respective Gaussians: $\mathcal{L}_{\text{eye}}=\sum_{i\in E}(1-\alpha_{i})^{2},$ where the $E$ denotes the set of Gaussian in the eyeball region. Since we utilize the 3DMFM parametric mesh, this set can be easily extracted. We provide more details about the entire energies with coefficients in Appendix A.3.

### 3.2 Adaptive Density Control

We enable adaptive density control with binding inheritance (Qian et al., ). Moreover, to avoid degenerate transparent solutions of eyeballs, we stop the gradient of eyeball-bound Gaussians’ rotation $R_{c}$ and position $\mu_{c}$ by blocking the proliferation, initialized identity and zero. We describe the proliferation of Gaussians, so-called Adaptive Density Control (ADC) (Kerbl et al., ) below.

Occlusion Gradient Amplification. The original adaptive density control (ADC) strategy (Kerbl et al., ) relies on view-space gradients. Occluded regions such as the lower tooth tend to suffer from less frequent updates. To remedy this, we amplify the view-space gradients of the tooth by $20\times$. This amplifier is a hyper-parameter that depends on the extent of the occluded parts shown.

## 4 Experiments

As shown in Fig. 3, the input to our pipeline is a tracked RGB video. For head tracking, we adopt the same preprocessing approach used in GaussianAvatars (Qian et al., ) (GA) for the multiview RGB video dataset, NeRSemble (Kirschstein et al., ). Furthermore, since the strength of our method lies in accurately reconstructing the geometry of dynamic head avatars, we validate our approach using a synthetic dataset FaceTalk (Zheng et al., ) that includes ground truth normal.

### 4.1 Evaluation Protocol

#### 4.1.1 Baselines

For evaluation, we train our model with five baselines: IMAvatar (Zheng et al., ), FLARE (Bharadwaj et al., ), PointAvatar (Zheng et al., ), SplattingAvatar (Shao et al., ), and GaussianAvatars (Qian et al., ). IMAvatar employs a neural occupancy field, FLARE uses mesh-based avatars with intrinsic material decomposition, and PointAvatar adopts a point-based explicit model. GaussianAvatars and SplattingAvatars, utilizing mesh-binding inheritance, represent 3D Gaussian splatting. We exclude 3D Gaussian splatting models for the synthetic dataset due to the lack of surface normals and omit IMAvatar for the real dataset due to training instability.

#### 4.1.2 Datasets

Monocular Synthetic Dataset (FaceTalk (Zheng et al., )).
FaceTalk, rendered from the FLAME (Li et al., ) model, includes diverse expressions and poses at a resolution of $512\times 512$. With ground truth normals available, we assess geometry fidelity. For experiments, we selected five identities, using 49 sequences for training and 3 with extreme poses for testing.

Multi-view Real Dataset (NeRSemble (Kirschstein et al., )).
NeRSemble, a real human head dataset with 16 cameras, follows the protocol of GaussianAvatars, using 11 video sequences (four emotions, six expressions, and one free performance). One sequence is reserved for testing, and the free performance sequence is used for cross-identity reenactment. To assess generalization for extreme poses, we manually constructed a new held-out set (detailed in Appendix A.3). LPIPS (Zhang et al., ) is excluded for efficiency but included in Tab. 2 only for fair comparison.

#### 4.1.3 Metrics and Tasks

To evaluate self-reenactment and dynamic novel-view synthesis, we use PSNR, SSIM, perceptual LPIPS (Zhang et al., ), and normal cosine similarity (NCS). As the FaceTalk dataset (Zheng et al., ) lacks multi-view data, we only evaluate NVS on the NeRSemble dataset (Kirschstein et al., ). For NCS on NeRSemble, we use pseudo-ground truth normals from the Sapiens model (Khirodkar et al., ). Alongside quantitative evaluations, we report qualitative cross-reenactment results in Fig. 5. Each table highlights Best and Second best scores.

### 4.2 Head Avatar Reconstruction and Reenactment

Figure: Figure 4: Qualitative results on the FaceTalk dataset (Zheng et al., ). Distinguished from other baselines, SurFhead simultaneously achieves convex eyeballs, and highly-detailed ears and nasal line. Best viewed when zoomed in.
Refer to caption: x5.png

Synthetic Dataset. Fig. 4 presents qualitative comparisons on FaceTalk synthetic dataset. IMAvatar (Zheng et al., ) demonstrates plausible rendering and geometry, but it often misses geometrical details with over-smoothing, particularly in the ears and nasal line, and displays concave artifacts in the pupil region. Furthermore, this method is quite slow, as training requires numerical searches to locate the surface, making it approximately 200$\times$ slower than PointAvatar (Zheng et al., ). While PointAvatar offers faster performance, it suffers from dotted noise in geometry due to its fixed, isotropic point size, making it challenging to represent extreme poses. FLARE (Bharadwaj et al., ) also shows mangled rendering and geometry, particularly in cases of extreme facial expressions, as it involves decomposing intrinsic material from a single environment, an ill-posed problem. This trend is reflected in the quantitatives in Tab. 2.

**Table 1: Qualitative comparison on FaceTalk (Zheng et al., ).**
|  | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | NCS$\uparrow$ |
| --- | --- | --- | --- | --- |
| IMAvatar | 32.23 | 0.983 | 0.037 | 0.931 |
| PointAvatar | 34.93 | 0.984 | 0.065 | 0.756 |
| FLARE | 31.64 | 0.968 | 0.027 | 0.943 |
| Ours | 40.15 | 0.992 | 0.020 | 0.983 |

Figure: Figure 5: Qualitative results on NeRSemble dataset (Kirschstein et al., ). Thanks to Jacobian and their blending, our method produces high-quality geometry with intricate details, visible in the normal maps. Please be aware of red boxes.
Refer to caption: x6.png

Real Dataset. For the NeRSemble real dataset, Fig. 5 shows qualitative comparisons. FLARE (Bharadwaj et al., )’s mesh involving unstable remeshing technique leads both over-smoothed rendering and normal with inferior quality. PointAvatar (Zheng et al., ) excels in fine detailed normals such as wrinkles, owing to their drastic pruning strategy to avoid the ambiguity of normal volumetric rendering, but they show salt-and-pepper artifacts from inflexible point representation. We also regard these two baselines’ degradations stem from their inherent representations intractability of their personalized blendshape space. SplattingAvatar (SA) (Shao et al., ) and GaussianAvatars (GA) (Qian et al., ), which utilize 3DGS (Kerbl et al., ), aim for high-quality renderings. Note that the normal from SA and GA is derived from the shortest axis of the Gaussians, a method commonly used in recent relighting research (Jiang et al., ). SA lacks explicit regularization of Gaussian positions, allowing them to drift far from their parent triangles, leading to inferior normal quality and floating artifacts. In both quantitative (Tab. 2) and qualitative evaluations, GA produces results comparable to ours, but its coarse deformation strategy, similarity transformation, does not account for triangle stretching and deformation discontinuity, resulting in semi-transparent and blob-like artifacts during extreme pose extrapolation. Additionally, SA and GA’s normal quality suffer due to the absence of geometry-level optimizations. Otherwise, thanks to our design of capturing high-fidelity geometry and accurate deformations, ours show superior quality of normal with detailed geometry. Not only the geometry, ours outperforms other state-of-the-art methods in terms of rendering quality. The qualitative observation is also proved in Tab. 2. Notably, GA shows better PSNR and LPIPS than ours, owing to 3DGS (Kerbl et al., )’s representations. This phenomenon in rendering quality gap is also reported in 2DGS (Huang et al., ) in view of the absence of representation dimensionality. Although this numerical scores, as can be seen in red boxes Fig. 5, ours result is more robust than GA on extreme expression scenarios. However, ours with LPIPS achieves the best LPIPS with preserving other metrics. Ours scores best in the NCS which indicate its high fidelity for dynamic geometry reconstruction with a large margin. In summary, ours outperform other baselines in terms of reconstruction capability with appearance and geometry, both.

### 4.3 Ablation Study

Jacobian and JBS help intricate rigging.

**Table 3: Ablations. +/- indicate incremental changes from the previous.**
|  | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | NCS $\uparrow$ |
| --- | --- | --- | --- | --- |
| Vanilla | 22.35 | 0.901 | 0.093 | 0.809 |
| + Jacobian | 22.38 | 0.902 | 0.091 | 0.812 |
| + JBS (= Ours) | 23.09 | 0.931 | 0.082 | 0.845 |
| - ASGs | 23.07 | 0.922 | 0.089 | 0.846 |
| - eyeballs | 23.03 | 0.925 | 0.087 | 0.820 |

We further validate the effectiveness of our method’s Jacobian and Jacobian Blend Skinning (JBS) in Tab. 3
and Fig. 6(a). Here,“Vanilla” refers to the combination of 2DGS (Huang et al., ) and Gaussian Avatars (Qian et al., ), with eyeball convexity regularization and ASGs as described in Sec. 2.4.

When incorporating the Jacobian deformation gradient, we observe a noticeable improvement in normal quality, particularly in reducing artifacts near the jaw and nasal lines, resulting in more coherent normal representations.

Rendered images show better alignment between floating artifacts and surface geometry, creating a more structured and accurate overall appearance. With JBS applied, the normal achieves even higher quality, showing smoother and more distinct separation. Pop-out artifacts are nearly eliminated, leading to refined representations of facial features, especially around the nasal area, surpassing the coarse expressions captured by the FLAME mesh. JBS significantly enhances the fidelity and detail of the normal maps by compensating for deformation discontinuities.

Effects of ASGs and Eyeball Regularization. To model the cornea of the eyeballs more accurately, we propose the use of Anisotropic Spherical Gaussians (ASGs) with a new, efficient implementation and eyeball regularization. To validate these improvements, we present qualitative results in Fig. 6(b) and quantitative results in Tab. 3. Without ASGs, the rendered color of the eyeball regions shows low-frequency reflections, resulting in a matte appearance. This occurs because, during the optimization of the eyeball region with convexity regularization, Spherical Harmonics alone are insufficient to capture the specular highlights. Moreover, when eyeball regularization is further reduced, the eyeballs appear to retain specularity, but the normals exhibit concave shapes, creating a hollow illusion. These observations suggest that the intricate modeling of both the geometry and appearance of the eyeballs is not feasible without either our ASGs or eyeballs regularization.

## 5 Conclusion and Discussion

SurFhead introduces a novel method for reconstructing dynamic head avatars that strikes a balance between photorealism and geometrically accurate rigging. By integrating Jacobian deformation with detailed normal adjustments and Jacobian Blend Skinning (JBS), our approach enables precise control over both appearance and geometry. As a result, SurFhead surpasses existing state-of-the-art dynamic head models, excelling in accurate geometry reconstruction, pose and expression extrapolation, and demonstrating its applicability in areas such as relighting (see Appendix A.5) and dynamic mesh reconstruction by leveraging the efficient, rule-governed rigging regime of 3DMFM meshes.

However, there is still room for improvement in the near future. First, similar to other 3DMFM and Gaussian-based methods, certain challenges persist, particularly regarding the bounded representation of the expression space in 3DMFM. Additionally, expressions that are grossly exaggerated and fall outside the span of 3DMFM’s expression space, even cannot be head tracked in the preprocessing stage. Furthermore, elements like the tongue and individual hair strands are still missing in modern 3DMFMs. Besides, one of the strengths of our rule-governed deformation from 3DMFM meshes is efficiency. Therefore, we also discuss the improvement room for computational efficiency of polar decomposition, which, unlike other Gaussian primitive-based methods relying on black-box learning, is deterministically governed in SurFhead. These further discussions on these limitations are in Appendix A.5.

Figure: (a) Jacobian and JBS help to reduce render artifacts and messy normal with extreme poses.
Refer to caption: x7.png

## Appendix A Appendix

### A.1 Related Work

Static to Dynamic Radiance Field Reconstruction. NeRF (Mildenhall et al., ) and NV (Lombardi et al., ) have opened the era of photorealistic renderings with undertaking novel-view synthesis task. Optimization and rendering efficiency could be improved by hash encoding  (Müller et al., ) and tensor decomposition (Chen et al., ). Later, Mixture of Volumetric Primitives (Lombardi et al., ) proposes efficient methodology with surface-aligned cuboid primitives which boost only ray marching around surfaces. Few years later, 3D Gaussian Splatting (3DGS) (Kerbl et al., ) utilizes anisotropic 3D Gaussians by rasterizing them without expensive ray-marching like previous strategies. We leverage the 3DGS, benefiting from the expressiveness and efficiency.

Reconstructing and Animating Head Avatars. Existing approaches for reconstructing and animating avatars mainly differ in two fundamental aspects: implicit or explicit models. Implicit models reconstruct the face by neural radiance field in combination with volumetric rendering or using implicit surface functions. (Gao et al., ), (Zielonka et al., ), (Zheng et al., ), (Xu et al., )
With explicit models (Grassal et al., ),(Zheng et al., ),(Khakhulin et al., ), the seminal work of 3DMFM uses principal component analysis (PCA) to model facial appearance and geometry on a low-dimensional linear subspace. 3DMFM and its variants have been widely applied in optimization-based and deep learning-based head avatar creation.
Recently, due to its efficient rendering and topological flexibility, there are many works utilizing 3D Gaussian Splatting (Kerbl et al., ).
These works can be further categorized based on how they define and use Gaussians. Some approaches bind Gaussians directly to the mesh (Shao et al., ; Qian et al., ), while others map them onto UV coordinates (Xiang et al., ). In some cases, Gaussians are extracted as features (Giebenhain et al., ),(Xu et al., ), or a neural parametric model replaces the traditional 3DMFM (Giebenhain et al., ). However, in most cases, the deformation is often handled by the mesh itself with little additional consideration.

### A.2 Proofs

#### A.2.1 Polar Decomposition’s Uniqueness

Lemma 1. There exists a unique polar decomposition of an arbitrary invertible matrix $\mathbf{A}$ such that

$$ $\mathbf{A}=\mathbf{U}\mathbf{P},$ $$

where $\mathbf{U}$ is an orthogonal matrix and $\mathbf{P}$ is a positive semidefinite symmetric matrix.

Proof (Lemma 1). Suppose that there are two polar decompositions of the matrix $\mathbf{A}$:

$$ $\mathbf{A}=\mathbf{U}_{1}\mathbf{P}_{1}=\mathbf{U}_{2}\mathbf{P}_{2},$ $$

where $\mathbf{U}_{1},\mathbf{U}_{2}$ are orthogonal matrices and $\mathbf{P}_{1},\mathbf{P}_{2}$ are positive semidefinite symmetric matrices.

Now, multiply both sides of $\mathbf{U}_{1}\mathbf{P}_{1}=\mathbf{U}_{2}\mathbf{P}_{2}$ by $\mathbf{U}_{1}^{T}$ on the left:

$$ $\mathbf{P}_{1}=\mathbf{U}_{1}^{T}\mathbf{U}_{2}\mathbf{P}_{2}.$ $$

Let $\mathbf{Q}=\mathbf{U}_{1}^{T}\mathbf{U}_{2}$, so that

$$ $\mathbf{P}_{1}=\mathbf{Q}\mathbf{P}_{2}.$ $$

Since $\mathbf{Q}$ is an orthogonal matrix and both $\mathbf{P}_{1}$ and $\mathbf{P}_{2}$ are positive semidefinite symmetric matrices, we conclude that $\mathbf{Q}=\mathbf{I}$, the identity matrix. Hence,

$$ $\mathbf{U}_{1}^{T}\mathbf{U}_{2}=\mathbf{I}\quad\Rightarrow\quad\mathbf{U}_{1} =\mathbf{U}_{2}.$ $$

Thus, $\mathbf{P}_{1}=\mathbf{P}_{2}$, and therefore the polar decomposition $\mathbf{A}=\mathbf{U}\mathbf{P}$ is unique. ∎

#### A.2.2 Preservation of Positive Semidefinite with Jacobian Gradients

Lemma 2. Consider an arbitrary matrix $M$. The matrix multiplication $MM^{T}$ is positive semidefinite (PSD) because for any vector $\mathbf{x}$,

$$ $\mathbf{x}^{T}MM^{T}\mathbf{x}\geq 0\Leftrightarrow\left\|M^{T}\mathbf{x} \right\|^{2}\geq 0.$ $$

The matrix $\Sigma$ also follows this form. Therefore, $\Sigma$ retains its positive semidefinite property after the affine transformation. ∎

### A.3 Implementation Details

Optimization Specifications.
We use Adam (Kingma & Ba, ) optimizer for learnable Gaussian parameters and translation, joint rotations, and expression parameters of FLAME (Li et al., ). We set the learning rates for all parameters same as with GA (Qian et al., ), except for the blending weights $w$. For the $w$, we set the learning rate as 1e-3. We train for 300,000 iterations, and exponentially decay the learning rate for the $\mu$ until the final iteration, where it reaches 0.01$\times$ the initial.

Finally, the entire energies are defined as:

$$ $\mathcal{L}=\mathcal{L}_{photo}+\lambda_{\text{depth}}\mathcal{L}_{depth}+ \lambda_{\text{normal}}\mathcal{L}_{normal}+\lambda_{\text{eye}}\mathcal{L}_{ eye}.$ (5) $$

The energy balances are $\lambda_{depth}=100$, $\lambda_{normal}=0.05$, and $\lambda_{eye}=0.1$.

Calculation Jacobian with Covariance.
Since there is no support for precomputation of covariance with original 2DGS (Huang et al., ) rasterizer implementation (^1^11https://github.com/hbb1/diff-surfel-rasterization.git), we calculate the Jacobian deformation gradient $\mathbf{J}_{\text{b}}$ combining with 3DGS’s original covariance as NVIDIA CUDA kernel implementation.

Color Change with Geometrical Transformations.

[Uncaptioned image]: x9.png

Most previous dynamic head avatars almost neglect the color change with deformations (Zielonka et al., ; Qian et al., ; Shao et al., ; Lombardi et al., ). When the bases of SHs do not rotate, they just query the same direction with facing identical side of Gaussians in any deformed space. This phenomenon is illustrated in the inset of the upper right branch. To mitigate this, we suggest a simple solution like the lower right of the inset, we inversely rotate the view direction $\mathbf{d}$ with the inverse of Gaussian’s blended rotation part $\mathbf{U}_{b}^{T}$. Namely, we rewrite the rotated input view direction: $\mathbf{d_{\text{rot}}}=\mathbf{U}_{b}^{T}\mathbf{d}$.

Anisotropic Spherical Gaussians for Eyeballs’ specularity. Comparing with Spherical Gaussians (SGs), ASGs (Xu et al., ) have been demonstrated to effectively represent anisotropic scene with a relatively small number.

ASGs are theoretically defined as:

$$ $ASG(\nu|[x,y,z],[\lambda,\mu],\xi)=\xi\cdot max(\nu\cdot z,0)\cdot e^{-\lambda (\nu\cdot x)^{2}-\mu(\nu\cdot y)^{2}},$ (6) $$

where z is lobe-axis, x and y are each tangent and bi-tangent of z, $\{\lambda,\mu\}\in\mathbb{R^{+}}$ are sharpness parameters, $\nu$ is the unit direction function input, and the $max$ term implies the smooth term.

Given that most specular BRDFs feature a lobe aligned with a specific reflection direction, we calculate the reflection vector to serve as the input direction: $\bm{\omega}_{o}=2(\mathbf{d}_{\text{rot}}\cdot n)n-\mathbf{d}_{\text{rot}}$,
where $\mathbf{d}_{\text{rot}}$ is the rotated input view direction, $n$ is a normal direction computed for each Gaussian, and $\bm{\omega}_{o}$ is the reflect direction. Lastly, since a simple summation of ASG output limits its representative ability, we utilize tiny two-hidden layered MLP $\mathcal{F}$ to obtain the final specular color, following  (Yang et al., ):

$$ $\displaystyle c_{s}(\bm{\omega}_{o};\mathbf{x})=\mathcal{F}\left(\bigoplus_{i= 0}^{N-1}ASG(\omega_{o}|[\bm{\omega}_{i}^{\lambda},\bm{\omega}_{i}^{\mu},\bm{ \omega}_{i}],[\lambda_{i},\mu_{i}],\xi_{i}),\gamma(\mathbf{d}_{\text{rot}}), \mathbf{n}\cdot\mathbf{d}_{\text{rot}}\right)$ (7) $$

where $\bigoplus$ indicates the concatenation operation, $\gamma$ denotes the positional encoding, and N is the number of basis SGs.

Not to compromise the computational efficiency, we leverage the common prior knowledge from the casual data capture setting. First, the eyeballs could not be effected by lights from backside of heads. Therefore, we sample $\bm{\omega}_{i}$ with limited range from frontal hemisphere of world space; $N=4\times 4$ relatively $50\%$ reduced ASGs compared with  (Han & Xiang, ; Yang et al., ). Second, since the captured data typically lie under environments with abundant white light, representing the MLP $\mathcal{F}$ output color as a monochrome channel for intensity is sufficient.

For a sampled direction $\bm{\omega}_{i}=(\theta,\phi)$ in the spherical coordinate system, we set $\bm{\omega}_{i}^{\lambda}=(\theta+\pi/2)$. Using quaternion operations, $\bm{\omega}_{i}^{\lambda}$ is then rotated around $\bm{\omega}_{i}$ by $\pi/2$ to derive $\bm{\omega}_{i}^{\mu}$. Then, we finally obtain the color with $c=c_{d}+c_{s}$ used in volume rendering equation, where $c_{d}$ is obtained by SHs from 3DGS (Kerbl et al., ).

NeRSemble Real Dataset Division
We conduct experiments on video recordings of 9 subjects from the NeRSemble  (Kirschstein et al., ) dataset as described on Tab. 4. As we focus on deformation, we select extreme expression set for self-reenactment to emphasize the difference. Therefore, among the emotion (EMO) and expression (EXP) sequences, we hold out EMO-1 for self-reenactment evaluation and use the rest nine for training as shown in Tab. 5.

Figure: Figure 7: Dataset example of custom self-reenactment held-out.
Refer to caption: x10.png

**Table 4: The held-out sequence of each subject for self-reenactment evaluation.**
| Subject ID | 074 | 140 | 175 | 210 | 253 | 264 | 302 | 304 | 306 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |

**Table 5: The train and test held-out sequences.**
| Train Sequences | EMO-2 | EMO-3 | EMO-4 | EXP-2 | EMO-3 | EMO-4 | EMO-5 | EMO-8 | EMO-9 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Test Sequences | EMO-1 |  |  |  |  |  |  |  |  |

### A.4 Additional Ablation Study

Adaptive Density Control (ADC) Amplification on Teeth. We posted qualitatives with the occlusion amplification on the teeth part in Fig. 8. This technique is very simple yet efficient with boosting Gaussian proliferation to overlooked parts in terms of optimization.

Figure: Figure 8: Ablations for ADC amplification on teeth.
Refer to caption: x11.png

### A.5 Limitation and Seminal Future Work

Bounded Representation from 3DMFM. We utilized a 3DMFM model, FLAME (Li et al., ) to enable our model to be equipped cross-reenactment and rigged compactly. However, the 3DMFMs’ expression space has only PCA-based coarse rigging-ability such that cannot easily describe the dynamics of facial muscles such as wrinkles and extreme pose or expression (Fig. 9).

To end this, we believe that the image-based latent expression representations (Burkov et al., ; Wang et al., ; Drobyshev et al., ; ) can alleviate this issue some extent. However, naively applying the latent expression to dynamic head avatars is non-trivial. Because only designing dynamics of the geometrical deformation is hard to handle ambient-occlusion (such as shadow of neck and jaw from head and lips, each) without intricate physical-based rendering (PBR). Therefore, we guess that the future work should be conducted as learning the deformation dynamics with those latent expression spaces, also simultaneously covering the deformation-aware color changes.

Second, the inherent limitations of 3DMFM’s representation, particularly the absence of hair strands and the tongue, can degrade the render quality. Since 3DMFM is fundamentally a bald model for human heads, it lacks the capability to represent the high-frequency details of human hair (Fig.  10). We believe that integrating our model with parametric strand models (Sklyarova et al., ; Zakharov et al., ; Rosu et al., ) could help alleviate these issues.

Figure: Figure 9: Limitation#1. Example for bounded representation of expression space from 3DMFM.
Refer to caption: x12.png

Computational Efficiency for Calculating Polar Decomposition with Numerical Views. Our Jacobian Blend Skinning (JBS) algorithm is based on polar decomposition, which is indirectly computed using byproducts of singular value decomposition (SVD). Specifically, $\mathbf{J}=\mathbf{W}\mathbf{\Sigma}\mathbf{V}^{T}$, where $\mathbf{W}$ and $\mathbf{V}$ are orthogonal matrices, and $\mathbf{\Sigma}$ is a diagonal matrix with singular values. This can be rewritten as $\mathbf{J}=(\mathbf{W}\mathbf{V}^{T})(\mathbf{V}\mathbf{\Sigma}\mathbf{V}^{T})
=\mathbf{U}\mathbf{P}$, where $\mathbf{U}$ is the rotation matrix, and $\mathbf{P}$ is the positive semidefinite symmetric matrix.

However, as reported in RoMa (Brégier, ), PyTorch (Paszke et al., )’s SVD routine scales linearly in time as the batch size increases. In the Gaussian Splatting (Kerbl et al., ; Huang et al., ) regime, over 10K operations for Gaussians are performed simultaneously to render an image, which requires the SVD routine to be executed in parallel. The polar decomposition process involves finding the nearest rotation matrix given a matrix. We believe that adopting RoMa (Brégier, )’s Procrustes process or Higham’s (Higham & Noferini, ) routine, which is specifically designed to solve problems for $3\times 3$ matrices, could significantly improve efficiency.

Extending SurFhead to Relighting Task. Normal is one of the key intrinsic material properties. We demonstrate the possibility of using the normals generated by SurFhead for avatar relighting tasks in Fig. 12, utilizing GaussianShader (Jiang et al., ). The relit results exhibit high plausibility, with high-fidelity reflections of the environment maps. We present objects reconstructed using our method, including relighting scenarios under various lighting conditions, featuring both warm and cool tones, as well as indoor and outdoor environments. Our renderings, across three diverse sets, convincingly demonstrate that the relit scenes maintain realism, exemplifying our method’s proficiency in relighting applications.

Figure: Figure 10: Limitation #2: The lack of hair modeling in 3DMFM can result in blurry hair appearance.
Refer to caption: x13.png

### A.6 Conceptual comparison with previous method

Figure: Figure 11: Conceptual illustration of stretching and shearing: (Left) rigid and isotropic scaling and (Right) affine deformation (ours)
Refer to caption: x14.png

Previous methods,  (Qian et al., ; Shao et al., ), utilize isotropic scaling for deforming mesh-bound Gaussians, with scalar scaling parameter $s_{p}$. GaussianAvatars computed $s_{p}$ using the mean length of one of the edges and its perpendicular, while SplattingAvatars computed $s_{p}$ based on the ratio of canonical and deformed triangle areas. However, we find that these approaches can result in undesirable Gaussian deformations when the 3DMFM meshes undergo stretch or shear in specific directions. When the deformed and canonical triangle areas or the sum of an edge’s length and its perpendicular are the same but the shape is differ, the Gaussians become misaligned and fail to cover the necessary regions accurately. In contrast, our method uses affine deformation, allowing Gaussians to more precisely reflect the deformation of the triangles and cover the regions as intended.

Figure: Figure 12: Results for relighting with GaussianShader (Jiang et al., ).
Refer to caption: x15.png

Figure: Figure 13: SurFhead can reconstruct high-fidelity meshes from Truncated Signed Distance Fucntion (TSDF) alongside diverse pose and expression.
Refer to caption: x16.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See [Weaknesses].
soundness: 3
strengths: * This paper introduces better deformation modeling techniques for Gaussian surfels. Compared to existing works, the proposed method is more reasonable and can handle more extreme deformations such as stretching and shearing. The proposed technique could be useful in other related research topics beyond head avatar modeling. 

* The proposed method is able to reconstruct fine geometric details, outperforming existing baselines by a large margin. 

* The paper is overall well-written and easy to follow.
summary: This paper presents a method for learning head avatars based on 2D Gaussian splatting. To make the Gaussian surfels better handle the stretch and shear deformation under extreme poses and facial expressions, this paper introduces affine transformation derived from Jacobian deformation gradient of the surface. Normal orientations are calculated accordingly. Moreover, the authors propose Jacobian blend skinning to interpolate these affine transformations to ensure surface smoothness. Results show that the proposed method is able to reconstruct drivable head avatars with high-quality geometry.
weaknesses: * Missing comparison against Gaussian Head Avatar (GHA) [Xu et al. 2023], which is a state-of-the-art head avatar method in terms of image synthesis quality. Although the authors have already compared with SplattingAvatar and GaussianAvatar, I think an additional comparison against GHA is also necessary because GHA demonstrates high-resolution image synthesis with the assistance of a super-resolution module. 

* It would be better if the authors report the training time and rendering speed. One important advantage of Gaussian splatting is its efficiency. I wonder whether the proposed techniques (such as Jacobian blend skinning) hinders this advantage or not. 

* It is not clear how the proposed method performs for subjects wearing eye-glasses. NeRSemble dataset contains cases with eye-glasses, but they are suspiciously skipped in the experiments.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: * It is unclear to me how Jacobian blend skinning improves upon linear blending skinning. It seems to me that $J_b$ introduces spatial smoothness to Gaussians' deformation matrices. Is the vertices of the mesh still transformed by linear blending skinning before the Jacobian blend skinning is applied? Also, could the author clarify Fig. 2(b)? What are the meanings of the green and yellow lines and the weights? How do they connect to the deformation of the triangle meshes?
* The paper mainly focuses on improving the geometry. Since 2D Gaussian splatting is known to produce better geometry than 3D Gaussians, how much does 2D Gaussians help improve the geometry, compared to the components proposed in the paper?
soundness: 3
strengths: * It motivates each contribution properly.
* The use of Jacobian deformation and Jacobian blend skinning in the context of head modeling looks novel to me.
* The qualitative results and normal similarity evaluation demonstrate better geometry compared to baselines.
* The effectiveness of each component is well studied.
summary: This paper aims at better geometry estimation in head modeling. It replaces the 3D Gaussian splatting with 2D Gaussian splatting to better model the surface. Moreover, it addresses three issues in existing works with three novel components: 1) To compensate for the incorrect deformation of 2D Gaussians due to the triangle's shear and stretch, it proposes the Jacobian deformation; 2) To mitigate the discontinuities in adjacent triangles, it improves linear blend skinning with Jacobian blend skinning; 3) To resolve hollow illusion in eyeballs, it replaces Spherical Harmonics with Anisotropic Spherical Gaussians. It demonstrates that it outperforms the state-of-the-art regarding normal similarity and remains comparable in rendering quality.
weaknesses: * It is unclear how the deformation gradient $J$ in Sec 2.2 and the blended Jacobian $J_b$ in Sec 2.3 connect. In Line 258 to 261, it mentions that it replaces the original deformation in GaussianAvatar with a new deformation $J_b$, and $J$ appears only as a parameter of JBS in Eq. 2. What is the relationship between $J$ and $U_i, P_i$ in Eq. 2? Which transformation, $J$ or $J_b$, is used in the final method?  
* The paper measures the normal similarity between ground-truth normals and rendered normals from 2D Gaussians. However, since this work claims to achieve a better geometry, evaluating metrics that apply to meshes, such as Chamfer distance or normals rendered from the mesh is more informative when judging the geometry. Although previous methods use normals rendered from 2D Gaussians as proof of geometry, the link between it and the mesh quality still looks vague to me.
* As the rendering quality is only on par with state-of-the-art, e.g., GaussianAvatar, the comparison of training and rendering speed is missing.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: As elaborated in the "weaknesses", the geometric accuracy of the experimental results exhibits considerable variation,  and we hope that the authors can further enhance their analysis and discussion on this crucial point.

In terms of experimental design, the NeRSemble dataset, while containing accurate 3D mesh models, lacks sufficient detail, and obtaining such precise models can often be challenging in practical applications.     In this regard, we eagerly inquire whether the proposed method heavily relies on such relatively accurate 3D mesh models, and how it would perform in their absence.     To validate this, the authors could consider using videos they have shot or sourced from the internet as input, employing monocular facial 3D reconstruction algorithms (such as DECA) to obtain mesh sequences, or directly bypassing the use of 3D mesh models altogether.     We are anticipating and curious about the results of such experimental setups.    We encourage the authors to actively explore and propose potential strategies for adapting their proposed approach to scenarios where detailed 3D mesh models are unavailable.
soundness: 3
strengths: The paper introduces SurFhead, a novel model within the Gaussian Splatting framework that captures geometrically accurate head deformations. This representation utilizes intricate affine rigging combined with Gaussians and their normals, solely based on RGB videos, which is a significant advancement in achieving realistic and detailed head avatars.  The proposed Jacobian Blend Skinning (JBS) Algorithm is technically sound.  The paper tackles the problem of the hollow illusion in the cornea, where a concave surface appears convex due to the prioritization of photometric losses during training. 

The methods presented in the paper are demonstrated to achieve superior results across a variety of subjects, including real and synthetic data. They excel in challenging scenarios such as sharp reflections on convex eyeballs, fine geometric details, and exaggerated deformations, showcasing the robustness and effectiveness of the proposed approach.
summary: This paper contributes a novel representation for geometrically accurate head avatars within the 3D Gaussian Splatting framework, a method for natural interpolation of affine transformations across adjacent deformations, and enhancements to the realism of corneal representations in head avatars. These contributions advance the state of the art in personalized head avatar construction and have the potential to improve various applications in computer graphics, virtual reality, and beyond.  The key contributions include:

* Introduction of SurFhead Model: The paper introduces SurFhead, the first geometrically accurate head avatar model within the Gaussian Splatting framework. This model is designed to capture the deformation of head geometry using intricate affine rigging that combines Gaussians and their normals solely from RGB videos.

* Jacobian Blend Skinning Algorithm: To address the issue of discontinuities between adjacent triangles in head deformations, the paper proposes the Jacobian Blend Skinning (JBS) algorithm. This algorithm blends adjacent transformations while avoiding geometric distortions by linearizing the non-linear matrix interpolation space, leveraging classical matrix animation techniques and geometrically smooth polar decomposition.

* Enhancement of Corneal Convexity and Specularity: The paper addresses the hollow illusion in the cornea by regularizing corneal convexity and enhancing specularity using computationally efficient Anisotropic Spherical Gaussians (ASGs). This improvement ensures a more realistic representation of the cornea in the head avatar.
weaknesses: The method is built upon the foundation of 2D Gaussian Splatting, and I believe that the proposed method's success in recovering a superior surface geometry owes much to this solid groundwork. Indeed, the authors introduced several improvements on this basis, such as intricate affine rigging, but I consider these innovations to be more incremental improvements rather than groundbreaking advancements.

The geometric accuracy of the experimental results appears to exhibit significant variation: some achieve hair-level geometric detail, while others fail to recover the structure of the hair. Consequently, whether this variation stems from instability in the algorithm or differences in the data quality of various training datasets arises. I hope the author can provide more analysis and discussion on this issue.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1) Related Works: I think some related works of static reconstruction could be further discussed, such as H3DS [1], deformable model-driven approaches [2], and Implicit Neural Deformation methods [3]. These methods leverage 3D points from SfM, multi-scans, or multi-view data from various identities to enhance reconstruction under sparse-view conditions, although they primarily focus on static human face or head geometry reconstruction.

Ref:

[1] Ramon, Eduard, et al. "H3d-net: Few-shot high-fidelity 3d head reconstruction." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[2] Xu, Baixin, et al. "Deformable model-driven neural rendering for high-fidelity 3D reconstruction of human heads under low-view settings." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[3] Li, Moran, et al. "Implicit Neural Deformation for Sparse‐View Face Reconstruction." Computer Graphics Forum. Vol. 41. No. 7. 2022.
soundness: 3
strengths: 1) Jacobian Blend Skinning (JBS): The use of Jacobian Blend Skinning (JBS) enables natural interpolation of affine transformations across adjacent deformations, effectively reducing discontinuities in transitions.

2) Cornea Opacity Constraint: To address the specular highlights in the eyeball region, the method constrains the corneal regions to remain opaque by regularizing the opacity of the respective Gaussians.
summary: The paper proposes a method that handles stretch and shear transforms essential for detailed deformations of geometry utilizing intricate deformations driven by the affine Jacobian gradient instead of similarity transformation and corresponding normal adjustments.
weaknesses: 1) Detail Representation: In Figure 5 (bottom row), there seems to be a lack of finer details, such as wrinkles. Adding more visual comparisons or details on addressing such high-frequency features could strengthen the analysis.

2) Rendering Speed and FPS: Given that methods like 3DGS/2DGS achieve real-time rendering, the speed of deformable-driven methods may be a limitation for applications requiring real-time animation. Could you report the FPS compared to other methods to clarify performance in time-sensitive scenarios?

Meta-review: The paper introduces a new model within the Gaussian Splatting framework designed to capture realistic and detailed head deformations from RGB videos. The model enhances geometric accuracy and detail, particularly in complex scenarios like sharp reflections and exaggerated deformations, with good efficiency. The authors provide a substantial improvement over existing methods in terms of geometric detail reconstruction.

***Strengths:***
- The introduction of Jacobian deformation and blend skinning is novel and effectively enhances the model's ability to handle complex - deformations like stretching and shearing.
- The proposed method reconstructs fine geometric details more accurately than existing baselines.
- The paper is clearly written and easy to follow, aiding in its comprehension and potential replication.

***Weaknesses:*** 
The lack of results like comparison with GHA, glasses wearing cases, robust demonstrations etc.
Missing details on training time and rendering speeds.

Despite some concerns, the reviewers are generally positive, highlighting the paper’s innovative approach and its effectiveness in handling complex scenarios. The authors have also adequately addressed the issues with added experimental results.  

After careful discussion and consideration, we are pleased to inform this paper is accepted. The paper is accepted based on its contributions to geometric detail reconstruction in head avatars, its novel methodological advancements, and the overall positive reception from reviewers.

Decision: Accept (Poster)