SurFhead: Affine Rig Blending for Geometrically Accurate 2D Gaussian Surfel Head Avatars
Authors: Jaeseong Lee
Taewoong Kang
Marcel Buehler
Min-Jung Kim
Sungwon Hwang
Junha Hyung
Hyojin Jang
Jaegul Choo
Venue: ICLR 2025 Poster (2025)
Keywords: dynamic head avatars
rigging
inverse-graphics
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Recent advancements in head avatar rendering using Gaussian primitives have achieved significantly high-fidelity results. Although precise head geometry is crucial for applications like mesh reconstruction and relighting, current methods struggle to capture intricate geometric details and render unseen poses due to their reliance on similarity transformations, which cannot handle stretch and shear transforms essential for detailed deformations of geometry. To address this, we propose SurFhead, a novel method that reconstructs riggable head geometry from RGB videos using 2D Gaussian surfels, which offer well-defined geometric properties, such as precise depth from fixed ray intersections and normals derived from their surface orientation, making them advantageous over 3D counterparts. SurFhead ensures high-fidelity rendering of both normals and images, even in extreme poses, by leveraging classical mesh-based deformation transfer and affine transformation interpolation. SurFhead introduces precise geometric deformation and blends surfels through polar decomposition of transformations, including those affecting normals. Our key contribution lies in bridging classical graphics techniques, such as mesh-based deformation, with modern Gaussian primitives, achieving state-of-the-art geometry reconstruction and rendering quality. Unlike previous avatar rendering approaches, SurFhead enables efficient reconstruction driven by Gaussian primitives while preserving high-fidelity geometry.

Published as a conference paper at ICLR 2025
SURFHEAD: AFFINE RIG BLENDING FOR GEOMET- RICALLY ACCURATE 2D GAUSSIAN SURFEL HEAD
AVATARS
Jacseong Lee Taewoong Kang-*
Marcel C. Bühler²
Min-Jung Kim²
Sungwon Hwang' Junha Hyung! KAIST 2ETH Zürich
Hyojin Jang' Jaegul Choo
https:// https:://sumnetighht www.tiigggghhhhhg..gggg..gh 1teii///Suua SurFhead
ABSTRACT
Recent advancements in head avatar rendering using Gaussian primitives have achieved significantly high-fidelity results. Although precise head geometry is crucial for applications like mesh reconstruction and relighting, current meth- ods struggle capture intricate geometric details and render unseen poses due to their reliance on similarity transformations, which cannot handle stretch and shear transforms essential for detailed deformations of geometry. To address this, we propose SurFhead, a novel method that reconstructs riggable head geometry from RGB videos using 2D Gaussian surfels, which offer well-defined geometric properties, such precise depth from fixed ray intersections and normals derived from their surface orientation, making them advantageous over 3D counterparts. SurFhead ensures high-fideling rendering of both normals and images, treme poses, by leveraging classical mesh-based deformation transfer and affine transformation interpolation. SurFhead introduces precise geometric deforma- tion and blends surfels through polar decomposition of transformations, including those affecting normals. Our key contribution lies in bridging classical graphics techniques, such as mesh-based deformation, with modern Gaussian primitives, achieving state-of-the-art geometry reconstruction and rendering quality. Unlike previous avatar rendering approaches, SurFhead enables efficient reconstruction driven by Gaussian primitives while preserving figh-iidelity geometry.
in ex-
INTRODUCTION
The construction of personalized head avatars has seen rapid advancements in both research and industry. Among the most notable developments in this field is the Codec Avatar family (Ma et al., 2021; Saito et al., 2024), which aims to reconstruct highly detailed, movie-quality head avatars using high-cost data captured from mounted cameras or studios. This approach has spurred signifi- cant research efforts to bridge the gap between high-cost and low-cost capture systems by utilizing only using RGB video setups. Neural Radiance Fields (NeRFs) (Mildenhall et al., 2021) have fur- ther accelerated these efforts with their topology-agnostic representations. As a result, numerous NeRF-based methods (Gafni et al., 2021; Athar et al., 2022; Zielonka et al., 2023b) for construct- ing head avatars from RGB videos have emerged, demonstrating potentials of improving high-cost systems (Ma et al., 2021; Yang et al., 2023; Saito et al., 2024).
Recently, 3D Gaussian Splatting (3DGS) (Kerbl et al., 2023) has been used to create photo-realistic head avatars. However, there has been no attempt to create geometrically accurate head avatars within the 3DGS framework. While other representation- Zheng et al., 2022; 2023; Grassal al., 2022) demonstrate plausible geometry using implicit or explicit representations, they still suffer from suboptimal results; over-smoothed geometry from implicit neural representations or inherently limited explicit representations such as 3D Morphable Face Model (3DMFM) learned meshes or inflexible points.
methods (Bharadwaj et al., 2023;
In response, we introduce SurFhead, the first geometrically accurate head avatar model within the Gaussian Splatting framework (Kerbl et al., 2023), designed to capture deformation of head ge-
"quual contribution

---

Published as a conference paper at ICLR 2025
Figure SurFhead reconstructs photo-realistic head avatars and high-fidelity surface normals, depth, and meshes from RGB videos alone. These avatars are represented through affine rigging of 2D surfel splats bound to a parametric morphable face model. SurFhead can fully control poses, expressions, and viewpoints, enhancing both appearance and geometry.
ometry. Our method integrates intricate affine rigging by combining Gaussians and their normals using only RGB videos. Building 2DGS (Huang al., 2024) with 2D surfel disks and adopting 3DMFM mesh binding from prior works (Qian et al., 2024; Shao et al., 2024; Zielonka et al., 2023b; Lombardi et al., 2021), we address limitations of previous Caussian-based methods that rely solely on rigid, isotropic transformations. This strategy often leads to undesirable deformations during pose extrapolation when 3DMFM meshes stretch in certain directions (Fig. 2a). To address these issues, we extend affine transformations beyond simple similarity transformations to handle shear and stretch, which often occur extreme poses. However, applying the same affine transformation to normals can lead to incorrect deformations, violating the orthogonality of their To correct this, we introduce the inverse-transpose of the affine transformation. In previous simi- larity transformations, this step was unnecessary because rotations and scaling were handled sepa- rately, and the rotation matrix inherently preserved normal directions due to its invariance under the transpoee operation.
normals.
Our method is fundamentally based on 3DMFM mesh binding inheritance, similar to GaussianA- vatars (Qian al., 2024), which uses 3DMFMM-bsed mesh triangle deformation. However, if only consider local deformation, discontinuities can occur between adjacent triangles (Zielonka al., 2023b), and the method becomes limited in capturing deformations beyond the parametric head model. To address these discontinuities, previous methods (Zielonka et al., 2023b; Shao et al., 2024) blend the transformations of adjacent triangles using element-wise summation. However, as illustrated in Fig. 2b, this naive element-wise blending results in unnatural geometric interpolation within the matrix space. To overcome this issue, we propose the Jacobian Blend Skinning algo- rithm, which blends adjacent transformations while avoiding geometric distortions. This algorithm leverages the concept linearizing the non-linear matrix interpolation space, drawing on classical matrix animation techniques (Shoemake & Duff, 1992) and employing geometrically smooth po- lar decomposition. Since the interpolation space of affine transformations inherently non-linear, simple element wise operations can lead to distortions (Fig. 2b).
Last but not least, we have witnessed a hollow illusion in the comea,, where concave surface appears convex. This occurs during training the model prioritizes photometrie losses, constructing a concave retinal surface filled with low-bandwidth Spherical Harmonics (SHs) color to partially maintain specularity. However, this approach misleads the loss function by simulating mirror-like
2

---

Published as a conference paper at ICLR 2025
(1. (0.75, 0.26) (0.5. 0.5) (0.25, (0.25,0.75) (0.
Direct Interpolation
(1.0) (0.75, 0.25) 0.5) (0.25,0.) 0.75) (0.
Expansion
exin)
Carovical
Polar decomposition
(a) conceptual illustration of how triangle shear and stretch deformation are applied to 2D surfels. (Mid- dle) shows rigid and isotropic scaling deformation, while (Right) depicts our Jacobian deformation. Jacobian de- scribes deformations precisely.
(b) Comparison of deformation interpola- tions hetween direct element-wise interpola- tion and polar decomposition interpolation. The above tuple label indicates the weight of each left and right-most matrix.
Figure 2: Toy examples Jacobian deformation and Polar Decomposition.
specularity with SHs. To address this, we regularize corneal convexity and enhance specularity using computationally efficient Anisotropic Spherical Gaussians (ASGs) (Xu et al., 2013).
To summarize our key contributions,
We introduce novel representation for geometrically accurate 2D Gaussian primitive- based head avatars, utilizing intricate deformations driven by the affine Jacobian gradient instead of similarity transformation and corresponding normal adjustments.
We propose method called Jacobian Blend Skinning (JBS) to naturally interpolate affine transformations across adjacent deformations, effectively mitigating discontinuities, such those arising from unseen pose extrapolation cases due variations in local space.
We demonstrate the advantages of methods across variety of subjects captured with real and synthetic data, achieving superior results in challenging scenarios, such as sharp reflections convex eyeballs, fine geometric details, and exaggerated deformations.
PROPOSED METHOD
This section highlights the key technical contributions of our work. We introduce novel representa- tion for geometrically accurate 2D Gaussian primitive-based head avatars, alongside a geometrically precise Jacobian deformation gradient and corresponding principled normal deformation (Sec. 2.2). Our proposed Jacobian Blend Skinning (JBS) approach (Sec. 2.3) allows for the interpolation of affine transformations, effectively addressing the discontinuities caused by piece-wise deformation in meshes. Additionally, we tackle the concavity (hollow-illusion) issue observed in the cornea by regularizing the modern 3DMFM model, FLAME (Li al., 2017)'s eyeball model, to better man- age high specularity through the use of Anisotropic Spherical Gaussians (ASGs) (Xu et al., 2013) (Sec. 2.4). We begin with preliminary on Gaussian Splatting and AaassianAvatars in Sec. 2.1.
2.1 PRELIMINARY
The two key building blocks of our approach are 2D Gaussian Splatting (2DGS) (Huang et al., 2024) and GaassianAvatars (Qian et al., 2024). The former provides intricate geometric properties, such as depth and normal. The latter employs mesh-based alle-governed rigging strategy, allowing primitives to be rigged directly according to mesh triangles.
2.1.1 GAUSSIAN SPLATTING
3D Gaussian Splatting (Kerbl et al., 2023) (3DGS) reconstructs scene with anisotropic 3D Gaus- sian primitives. Each Gaussian is defined by positive semi-definite covariance matrix Σ that is centered at position μr. The covariance Σ is decomposed as Σ = RSS.T R where R is rotation matrix and S is a diagonal scaling matrix. This covariance matrix used when estimating each Gaussian's 's function value at 3D point such as G(z) exp( } μ1) T -- μ)). Besides these geometrical properties, each Gaussian has appearance properties opacity a and color c. To render the scene, the final color C' is computed by alpha-blending Gaussians after projecting them to the image plane: C Σi=l Ghooj (x) [ (1 agroro (x)). Gpros is given by evaluating
3

---

Published as a conference paper at ICLR 2025
Square root Counturce
J.R.5.
J.H.
S4 coeffrient
J
J Re
Woot
Σ aP.. Hell
Eyebali
F
Canonical space
Deformed space
Blerded deformed space
Figure 3: Overall pipeline of SurFhead. Only from RGB videos, SurFhead constructs geometri- cally accurate head avatars, equipped with our intricate deformations. The Jacobian J covers stretch and shear deformations avoiding surface distortion. Moreover, the blended Jacobian J, alleviates inherent local deformations' discontinuity. Finally, elaborated modeling of eyeballs such as preser- vation of specularity and convexity achieves more realistic appearance and geometry.
the function value of 2D) projection of the 3D Gaussian in image space by EWA volume splatting algorithm (Zwicker al., 2001).
An extension to 3DGS, 2DGS (Huang et al., 2024), modifies 3D primitives to 2D "flat" surfels embedded in 3D space for reconstructing high-fidelity geometry. 2D surfels have some advantages compared with 3DGS in the respect of geometry; deriving intricate depth by ray-splat intersection algorithm (Weyrich et al., 2007), and closed-form modeling the normal n as cross-prodiee of tangents r1 and 72. Thus, R = [m r2; m). n = 71 22, and each scales $1, $2, and composes the scale matrix Details are provided in 2DGS (Huang al., 2024) paper.
2.1.2 GAUSSIANAVATARS
GaussianAvatars (Qian et al., 2024) (GA) is an efficient head avatars model with FLAME (Li al., 2017) mesh binding inheritance. Namely, each Gaussian has single parent triangle canonical (local) space. When they rendered in a deformed space, each Gaussian is transformed with their parent's 's current state such as relative rotation matrix Rp, relative triangle's 's area volume as isotropic scale 8p, and relative barycenter position of triangle Tp in the world space. Namely,
RR = SyRpPe Tp, Sp.
where Rs is canonical rotations, Me canonical position, and S. canonical scaling par- ent's triangle. Although this deformation approach is frequently utilized in certain head avatar research (Qian et al., 2024; Shao et al., 2024), we emphasize that it falls short capturing stretch and shear deformations that are essential for accurately extrapolating extreme expressions.
2.2 AFFINE TRANSFORMATION OF 2D SURFELS
Our goal is to reconstruct geometrically accurate head avatars using the 2D surfel splatting regime, which employs normal consistency energy (Huang et al.. 2024). This approach implies that high- fidelity normals originate from high-fidelity depths. Previous methods only adopt rigid deformation to maintain the positive semi-definite (PSD) property of Gaussian. However, as shown in Fig. 2a, using only rigid deformation for rigging can result over- or under-occupying phenomena and deformation distortion.
Local Geometry Descriptor. Let us consider triangle in the canonical space as the matrix formed by its vertices v; E R3, and its edge matrix as E [m] Wo, 02 wo, 03 vol E R3x3 where E is the deformed version with the deformed triangle's vertices 03 Vi v [(v] - ((1
(v) (002-0))))-
de following Eq. 1. Note that in GaassianAvatars, the edge matrix is defined with a normalized base, height, and normal direction. This constructs orthogonal matrix and approximates the shear or stretching-relane related deformations with isotropic scaling from the average of the base and height length. Please find detailed derivation in Gaussian (Qian et al., 2024).

---

Published as a conference paper at ICLR 2025
Jacobian Deformation Gradient for Surface. The deformation gradient J is defined EE following (Sumner & Popović, 2004). We introduce affine rigging with 2D surfels. The new pa- rameterization is 5/1/2 Sc.c, instead of Sp Rc where 5 = E ////(/) We note that the Jacobian deformation gradient J can correct inaccuracies in deformation, such as the lack of stretch or shear awareness, which can lead to incorrect surface reconstruction. It is also important to keep in mind that the Gaussian primitives retain their physical meaning only when the positive semi-definite is kept (Kerbl et al., 2023). We prove whether the covariance matrix Σ remains PSD after applying affine transformations in Appendix A.2.2.
Both depth and normals are essential for constructing high-fidelity head geometry. While recent works 3DGS-based full-body avatars (Zielonka et al., 2023a) and physical simulations (Xie et al., 2024; Jiang et al., 2024a) have shown that Jacobian deformation gradients accurately describe de- formations, no prior work has addressed normal deformation with Jacobians. We found that the inverse-transpose of the Jacobian, J J-" is the correct deformation matrix for normals, based on the principle that "Orthogonality must be maintained in any space." In canonical space, if there is a tan- gent vector re in surfel, the normal ne is orthogonal to tangent (n² -re 0). Then, the deformed tangent is defined as ra Jr.. Let be the matrix that transforms a normal vector from the canon- ical to the deformed space such as nid An,- Trivially, the normal and tangent must preserve the orthogonality in the deformed space: n Td = (An.)* Jr. = n A Jr. = O. We already know that n. rc = 0, then A J- (since J is oon-singl,.r...... Therefore, (AT)T A J-T D. Consequently, we use the principled transformation of normal as inverse-transposed form of Jaco- bian, Rd J JTT Ne.
2.3 JACOBIAN BLEND SKINNING (JBS)
Replacing the scaled-rotation component spRtp of the similarity transform with the proposed J en- ables better representation of shear and stretch. However, ensuring the continuity of adjacent de- formations is crucial for achieving smoother transformations. To address this, we introduce im- proved technique to mitigate such discontinuities by substituting J again with a blended Jacobian, Jg. We refer to this approach as Jacobian Blend Skinning (JBS), which builds on the principles of Linear Blend Skinning (LBS) (Badler & Morris, 1982) while addressing its inherent limitations.
Degenerate Solution of Linear Blend Skinning in Matrix Space. Linear Blend Skinning (LBS) is widely used for dynamic modeling of the human body and head (Li et al., 2017; Loper et al., 2015), but it struggles with the non-linearity of rotation matrix interpolation in SO(3), causing distortions in rotational transformations. Our toy experiments (Fig. 2b) show how element-wise linear interpolation distorts shapes. Previous works (Zielonka al., 2023b; Shao et al., 2024) also adopt this sub-optimal approach to handle local deformations. Techniques like matrix-logarithm interpolation or quaternion's ' SLERP can address this but are limited to rotations.
Introducing Jacobian Blend Skinning (JBS). The Jacobian Blend Skinning (JBS) algorithm over- comes the limitations of LBS by focusing on the interpolation of Jacobian gradients, which encap- sulate not only rotations but also shear and stretch, both residing in the broader GL(3) (General Linear) field. The key to JBS is leveraging Polar Decomposition (PD) (Shoemake & Duff, 1992) to break down Jacobian gradient J into two meaningful components: rotation (orthogonal matrix U) and stretch/shear (symmetric positive semi-definite matrix P): J = UP. This decomposition is unique and coordinate- coordinate-indeeenneen (proof in Appendix A.2.1), making it geometrically sound for interpolation. By blending the rotation in matrix-logarithm space and the stretch/shear in linear ma- trix space, JBS ensures that the resulting transformations remain geometrically valid, avoiding the distortions seen with LBS (Fig. 2b).
JBS is applied to the Gaussian deformations considering not only its parents' triangle, but its adja- cent triangles. The blending itself is performed separately for the rotational (Us) and stretch/shear (P&) components. For rotations, the blending weights are applied in the matrix-logarithm space 53)3), followed by an exponential mapping back into the SO(3) space. For stretch/shear, the weights are applied directly in the linear matrix space, ensuring positive cemi-definiteness, which avoids ge- ometrical distortions. Mathematically, this Jacobian Blend Skinning is defined as:
J3 = JBS(J, w) exp( Σ
(.,))(U.))
Σ wrP. 1
(2)
(Eadi
tEady.
Up
5

---

Published as a conference paper at ICLR 2025
where adj. implies the set of adjacent triangles. The blending weights, Wy, in JBS learnable convex weights that control how much influence each adjacent triangle has on the resulting blended Jacobian J., Note that since w; is activated by Sigmoid function. the blended stretch/shear compo- nent P; is also guaranteed positive semi-definite property. The exponential and logarithm mappings for rotations are performed using Rodrigues' formula. Finally, the blended Jacobian Jg is then used to deform the canonical Gaussian's 's covariance and mean position and also transform the normal.
g = J....Se. = JoHe +
(3)
11d - TJ==Jo" nce
(4)
In summary, JBS improves on LBS by using Polar Decomposition to separately blend rotation and stretch/shear transformations. This approach ensures more accurate and geometrically meaningful deformations, avoiding the issues caused by linear interpolation. By operating in the appropriate spaces -logarii- for rotations and linear for stretch/shear-JBs. preserves the integrity of transformations, making it a more reliable solution for complex deformations.
2.4 RESOLVING HOLLOW-ILLUSION IN EYEBALLS
Many studies (Park et al., 2021: Li et al., 2022: 2024) have witnessed that volumetric approaches often fail to produce good approximation of the eye- ball geometry and yield hollow eyes. We also have empirically found that the cornea of the human eye often appears concave geometry without proper constriction. Since the cornea exhibits high specularity due to its multiple membranes (Dua al., 2013), the limited representation capability Spherical Harmonics (SHs) can distort the geometry, unnaturally struggling to satisfy the photometric losses. This deceptive phenomenon harmful to achieving accurate geometric representation as can be seen in the inset.
To address the issue of concave geometry around the eyeball, we eliminate the geometrical gradient on the eyeball-bound Gaussians, excluding Gaussians' cloning and splitting. We use the FLAAE eyeball mesh as an approximation for the eyeball geometry. Additionally, we regularize the opacity of the eyes to approach 1, following the energy Leye in Sec. 3. Although this improves the geometry, SHs still fall short of capturing eye specularity.
In mitigation, we employ Anisotropic Spherical Gaussians (ASGs) (Xu et al., 2013). Since the cornea and sclera often reflect their light environment, ASGs excel at capturing sharp reflections. Implementation details of ASGs can be found in Appendix A.3. To preserve computational effi- ciency, we leverage common knowledge from casual data capture. First, eycballs aren' affected by light from the back of the head, space, reducing ASGs by 50% compared to previous methods (Han & Xiang, 2023). Second, given that captured data is often in environments with ample white light, representing specular reflections as monochrome intensity channel is sufficient.
limit the sampling range to the frontal hemisphere of the world
3 TRAINING STRATEGY
3.1 OPTIMIZATION
We supervise the rendered images with photometric loss Cphoto which is combination of LI term Ln and a D-SSIM term Lsim following 3DGS (Kerbl et al., 2023). Moreover, since we aim to re- construct high-fidelity geometry, we followed the geometric energies in 2DGS (Huang et al., 2024), depth-distortion Liepth and normal consistency energies L.ormal- Toward the better alignment be- tween Gaussians and their parent triangles, we utilize two regularization energy terms Locaating and Epasition from GA (Qian et al., 2024).
Eyeball Regularization. We have empirically found that the highly specular eyeball region, es- pecially the cornea, tends to yield overly transparent Gaussians satisfy the photometric losses. This leads to an incorrectly shaped, concave cornea, although the genuine surface should be roughly convex (Li et al., 2022). To alleviate this issue, constrain the comea regions to be opaque by regularizing the opacity of the respective Gaussians: Ceye = Σieg (1 az)², where the E de- notes the set of Gaussian in the eyeball region. Since we utilize the 3DMFM parametric mesh, this

---

Published as a conference paper at ICLR 2025
set can be easily extracted. We provide more details about the entire energies with coefficients in Appendix A.3.
3.2 ADAPTIVE DENSITY CONTROL
We enable adaptive density control with binding inheritance (Qian et al., 2024). Moreover, avoid degenerate transparent solutions of eyeballs, we stop the gradient of breball-bound Gaussians' rota- tion R. and position P.. by blocking the proliferation, initialized identity and zero. We describe the proliferation of Gaussians, so-called Adaptive Density Control (ADC) (Kerbl et al., 2023) below.
Occlusion Gradient Amplification. The original adaptive density control (ADC) strategy (Kerbl et al., 2023) relies on view-space gradients. Occluded regions such as the lower tooth tend to suffer from less frequent updates. To remedy this, we amplify the view-space gradients of the tooth by 20x. This amplifier is hyper-parameter that depends on the extent of the occluded parts shown.
4 EXPERIMENTS
As shown in Fig. 3, the input to our pipeline is tracked RGB video. For head tracking, we adopt the same preprocessing approach used in GaussianAvatars (Qian et al., 2024) (GA) for the multiview RGB video dataset, NeRSemble (Kirschstein al., 2023). Furthermore, since the strength of our method lies in accurately reconstructing the geometry of dynamic head avatars, we validate our approach using synthetic dataset Face'Talk (Zheng et al., 2022) that includes ground truth normal.
4.1
EVALUATION PROTOCOT.
4.1.1 BASELINES
For evaluation, train model with five baselines: IMAvatar (Zheng et al., 2022), FLARE (Bharadwaj et al., 2023), PointAvatar (Zheng et al., 2023), SplattingAvatar (Shao et al., 2024), and CoussianAvatars (Qian et al., 2024). IMAvatar employs neural occupancy field, FLARE uses mesh-based avatars with intrinsic material decomposition, and PointAvatar adopts a point-based explicit model. Avatass and SplattingAvattas, utilizing mesh-binding inheritance, represent 3D Gaussian splatting. We exclude 3D Gaussian splatting models for the synthetic dataset due to the lack of surface normals and omit IMAvatar for the real dataset due training instability.
4.1.2 DATASETS
Monocular Synthetic Dataset (FaceTalk (Zheng et al., 2022)). FaceTalk, rendered from the FLAME (Li et al., 2017) model, includes diverse expressions and poses at a resolution of 512 x 512. With ground truth normals available, assess geometry fidelity. For experiments, we selected five identities, using 49 sequences for training and 3 with extreme poses for testing.
Multi-view Real Dataset (NeRSemble (Kirschstein et al., 2023)). NeRSemble, real human head dataset with 16 cameras, follows the protocol of GaussianAvatars using 11 video sequences (four emotions, six expressions, and one free performance). One sequence reserved for testing, and the free performance sequence is used for cross-identity reenactment. To assess generalization for ex- treme poses, we manually constructed new held-out (detailed in Appendix A.3). LPIPS (Zhang et al., 2018) is excluded for efficiency but included in Tab. 2 only for fair comparison.
4.1.3 METRICS AND TASKS
To evaluate self-reenactment and dynamic novel-view synthesis, we use PSNR, SSIM, perceptual LPIPS (Zhang al., 2018), and normal cosine similarity (NCS), As the FaceTalk dataset (Zheng et al., 2022) lacks multi-view data, we only evaluate NVS on the NeRSemble dataset (Kirschstein al., 2023). For NCS on NeRSemble, we use aseudo-ground truth normals from the Sapiens model (Khirodkar et al., 2024). Alongside quantitative evaluations, we report qualitative cross- reenactment results in Fig. 5. Each table highlights Best and Second best scores.

---

Published as a conference paper at ICLR 2025
Table 1:
Quantitative comparison on Fac-
Quantitative comparison on NeRSem- ble (Kirschstein al., 2023).
Table 2:
eTalk (Zheng et al., 2022).
PSNR
SSIM
LPIPS
NCS1
Nivel-Vicw Synlasis
Self-Romaatinenr
PSNRT
SSIM1
LPMPS,
NCS1
PSNRT
SSIM1
[.PIS]
NCS1
IMAvatar
32.23
0.983
0.037
0.931
PoentAvattr
20.56
0.844
0.206
0.410
20.59
0.834
0.190
0.611
PointAvatar
34,93
0.984
0.065
0.756
Fiam
21.91
0.814
0.228
0.817
0.892
0:227
0.752
FLARE
31.64
0.968
0.027
0.943
SybatinnpArottom
0.838
0.067
0.333
0.891
0.828
0:365
0,093
0.716
Ours
40.15
0.992
0.020
0.983
Durs
3030
0.934
0.099
0.886
23.53
0,892
0.00
0812
Ours - LMMPS
29.94
0,935
0.01
0
25.78
0.896
00089
0.87
4.2
HEAD AVATAR RECONSTRUCTION AND REENACTMENT
Synthetic Dataset. Fig. 4 presents qualita- tive comparisons on FaceTTakk synthetic dataset. IMAvatar (Zheng et al., 2022) demonstrates plausible rendering and geometry, but it of- ten misses geometrical details with over- smoothing, particularly in the cars and nasal line, and displays concave artifacts in the pupil region. Furthermore, this method is quite slow, as training requires numerical searches to locate the surface, making it approximately 200x slower than PointAvatar (Zheng al., 2023). While PointAvatar offers faster perfor- mance, suffers from dotted noise in geome- try due to its fixed, isotropic point size, mak- ing challenging to represent extreme poses. FLARE (Bharadwaj et al., 2023) also shows mangled rendering and geometry, particularly in cases of extreme facial expressions, it in- volves decomposing intrinsic material from single environment, ill-posed problem. This trend is reflected in the quantitatives in Tab. 1.
Medium
Strong
M
Zoom
RGB
PointAvatar
Qualitative results on the Face'Talk dataset (Zheng et al., 2022). Distinguished from other baselines, SurFhead simultaneously achieves convex eyeballs, and highly-detaile, ears and nasal line. Best viewed when zoomed in.
Figure 4:
Real Dataset. For the NeRSemble real dataset, Fig. 5 shows qualitative comparisons. FLARE (Bharadwaj et al., 2023)'s mesh involving unstable remeshing technique leads both over- smoothed rendering and normal with inferior quality. PointAvatar (Zheng al., 2023) excels in fine detailed normals such as wrinkles, owing to their drastic pruning strategy to avoid the ambiguity of normal volumetric rendering, but they show salt-and-pepper artifacts from inflexible point represen- tation. We also regard these two baselines' degradations stem from their inherent representations intractability of their personalized blendshape space. Splatting (SA) (Shao et al., 2024) and CaussianAvatar (GA) (Qian et al., 2024), which utilize 3DGS (Kerbl et al., 2023), aim for high- quality renderings. Note that the normal from SA and GA is derived from the shortest axis of the Gaussians, a method commonly used in recent relighting research (Jiang et al., 2024b). SA lacks explicit regularization of Gaussian positions, allowing them to drift far from their parent triangles, leading to inferior normal quality and floating artifacts. In both quantitative (Tab. 2) and qualitative evaluations, GA produces results comparable to ours, but its coarse deformation strategy, similar- ity transformation, does not account for triangle stretching and deformation discontinuity. resulting in semi- cemi-transpprent and blob-like artifacts during extreme pose extrapolation. Additionally, SA and GA's normal quality suffer due the absence of geometry-level optimizations. Otherwise, thanks to our design of capturing high-fidelity geometry and accurate deformations, ours show superior qual- ity of normal with detailed geometry, Not only the geometry, ours outperforms other state-of-the-an methods in terms of rendering quality. The qualitative observation is also proved in Tab. 2. Notably, GA shows better PSNR and LPIPS than ours, owing to 3DGS (Kerbl et al., 2023)'s representations. This phenomenon in rendering quality gap is also reported in 2DGS (Huang et al., 2024) in view of the absence representation dimensionality. Although this numerical scores, as can be seen in red boxes Fig. 5, ours result is more robust than GA on extreme expression scenarios. However, ours with LPIPS achieves the best LPIPS with preserving other metrics. Ours scores best in the NCS which indicate its high fidelity for dynamic geometry reconstruction with large margin. In summary, ours outperform other baselines in terms of reconstruction capability with appearance and geometry, both.
8

---

Published as a conference paper at ICLR 2025
RGB
Normat
Self Self-reenactment
RGB
Normal
Novel-view Synthesis
RGB
Normal
GT
FLARE
PointAvatar
SA
GA
Ours
Source Actor
Cross-reenactment
RGB
Normal
Figure 5: Qualitative results on NeRSemble dataset (Kirschstein et al., 2023). Thanks to Jacobian and their blending, method produces high-quality geometry with intricate details, visible in the normal maps. Please aware of red boxes.
4.3 ABLATION STUDY
Jacobian and JBS help intricate rigging.
We further validate the effectiveness our method's Jacobian and Jacobian Blend Skinning (JBS) in Tab. 3 and Fig. 6a. Here, "Vanilaa refers to the combination of 2DGS (Huang et al., 2024) and Gaussian Avatars (Qian et al., 2024).
Table 3: Ablations. +/- in- dicate incremental changes from the previous.
NCS
When incorporating the Jacobian deformation gradient, we observe a noticeable improvement in normal quality, particularly in reducing tifacts near the jaw and nasal lines, resulting in more coherent normal representations.
Coneinávatoon
22.40
0.036
01089
0.777
eyetalls Vanilla
22.32
0.907
0.093
0.903
03801
0.999

---

Published as a conference paper at ICLR 2025
Dender
nambbrr
Norres
Normal
GT
Jacobian
(a) Jacobian and JBS help to reduce render artifacts and messy normal with extreme poses.
(b) Both ASGs and eyeball regularization needed for accurate geometry and sharp specular rendering.
Figure 6: Qualitative ablation for proving the proposed method. The left side show how Jacobian deformations and Jacobian Blend Skinning alleviate artifacts in extreme poses. The right side shows degradations in the eye region when training without eyeballs or without anisotropic spherical Gaus- sians (ASG).
Rendered images show better alignment between floating artifacts and surface geometry, creating more structured and accurate overall appearance. With JBS applied, the normal achieves even higher quality, showing smoother and more distinct separation. Pop-out artifacts are nearly eliminated, leading to refined representations facial features, especially around the nasal area, surpassing coarse expressions captured by the FLAME mesh. JBS significantly enhances the fidelity and detail of the normal maps by compensating for deformation discontinuities.
Effects of ASGs and Eyeball Regularization. To model the cornea of the eyeballs more accurately, we propose the use of Anisotropic Spherical Gaussians (ASGs) with a new, efficient implementa- tion and eyeball regularization. To validate these improvements, Fig. 6b and quantitative results in Tab. 3. Without ASGs, the rendered color of the eyeball regions shows frequencyyn reflections, resulting optimization of the eyeball region with convexity regularization, Spherical Harmonics alone are insufficient to capture the specular highlights. Moreover, when eyeball regularization is further re- duced, the eyeballs appear to retain specularity, but the normals exhibit concave shapes, creating hollow illusion. These observations suggest that the intricate modeling of both the geometry and appearance of the eyeballs is not feasible without either our ASGs or eyeballs regularization.
present qualitative results in
matte appearance. This occurs because, during the
5 CONCLUSION AND DISCUSSION
SurFhead introduces novel method for reconstructing dynamic head avatars that strikes a balance between photorealism and geometrically accurate rigging. By integrating Jacobian deformation with detailed normal adjustments and Jacobian Blend Skinning (JBS), our approach enables precise control over both appearance and geometry. As a result, SurFhead surpasses existing fate-of-the-an dynamic head models, excelling in accurate geometry reconstruction, pose and expression extrapola- tion, and demonstrating its applicability in areas such as relighting (sce Appendix A.6) and dynamic mesh reconstruction by leveraging the efficient, rule-governed rigging regime of 3DMFM meshes.
However, there is still room for improvement in the near future. First, similar to other 3DMFM and Gaussian-basse based methods, certain challenges persist, particularly regarding the bounded represen- tation of the expression space in 3DMFM. Additionally, expressions that are grossly exaggerated and fall outside the span of 3DMFM's expression space, even cannot be head tracked in the prepro- cessing stage. Furthermore, elements like the tongue and individual hair strands are still missing in modern 3DMFMs. Besides, one of the strengths our rule-governed deformation from 3DMFM meshes is efficiency. Therefore, we also discuss the improvement room for computational effi- ciency of polar decomposition, which, unlike other Gaussian primitive-base methods relying on black-box learning, is deterministically governed in SurFhead. These further discussions on these limitations are in Appendix A.6.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
This research was supported by the Institute for Information & Communications Technology Plan- ning Evaluation (IITP) grant, funded by the Korean government (MSIT) (RS-2019-11190075, Artificial Intelligence Graduate School Program at KAIST). We also acknowledge the support of the National Research Foundation of Korea (NRF) grant, funded by the Korean government (MSIT) (No. RS-2025- AS-2025-005555 000555611.. Additionally, would like to express sincere gratitude to Jungwoo Ahn for conducting the baseline experiments.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of method, provide the anonymous GitHub link. Moreover, the details on implementation and the information of datasets can be found in the appendix A.3 and Sec. 4.1.2, respectively. For NeRSemble (Kirschstein et al., 2023) dataset, we used pre-processed dataset from GaussianAvatars (Qian et al., 2024). Together with provided code and publicly avail- able datasets, we believe our paper contains sufficient information for reimplementation.
ETHICS STATEMENT
The creation of highly accurate and geometrically detailed head avatars raises significant ethical and security concerns, particularly regarding privacy violations. The realistic nature of these avatars can lead to unauthorized manipulation, where personal likenesses may be misused without consent. This includes altering appearances in virtual environments or generating misleading content, which infringes privacy and may result in serious consequences.
One critical issue is the potential for creating deepfake videos, which could spread false information, harm reputations, and manipulate public opinion. The increased realism of these avatars makes it difficult to discern authentic content from fabricated material, amplifying risks of misinformation and defamation. Additionally, the possibility of identity theft through the exploitation of digital avatars presents a serious threat to personal security and public trust in digital media.
To mitigate these risks, will strictly limit the of method academic research. This restriction aims to prevent misuse and ensure the technology is applied an ethical and constructive manner.
REFERENCES
ShahRukh Athar, Zexiang Xu, Kalyan Sunkavalli, Eli Shechtman, and Zhixin Shu. Rignerf: Fully controllable neural 3d portraits. Proceedings of the IEEE/CVE conference on Computer Vision and Pattern Recognition, pp. 20364-20377 2022. (Cited on page 1)
Norman Badler and Mary Ann Morris. Modelling flexible articulated objects. In Proc. Computer Graphics' 82, Online Conf. pp. 305-314, 1982. (Cited on page 5)
Shrisha Bharadwaj. Yufeng Zheng, Otmar Hilliges, Michael J Black, and Victoria Fernandez- Abrevaya. Flare: Fast learning of animatable and relightable mesh avatars. arXiv preprint arXiv:2310.172 17519, 2023. (Cited page 7,
Romain Brégier. Deep regression on manifolds: 3d rotation case study. In 2021 International Conference on 3D Vision (3DV), pp. 166-174. IEEE, 2021. (Cited on page 20, 21)
Egor Burkov, Igor Pasechnik, Artur Grigorev, and Victor Lempitsky. Neural head reenactment with latent pose descriptors. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13786 13795, 2020. (Cited on page 21)
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European conference on computer vision, pp. 333-350. Springer, 2022. (Cited on page 17)
11

---

Published as a conference paper at ICLR 2025
Nikita Drobyshev, Jenya Chelishev, Taras Khakhulin, Aleksei Ivakhnenko, Victor Lempitsky, and Egor Zakharov. Megaportraits: One-shot megapixel neural head avatars. In Proceedings of the 30th ACM International Conference Multimedia, pp. 2663-2671, 2022. (Cited on page 21)
Nikita Drobyshev, Antoni Bigata Casademunt, Konstantinos Vougioukas, Zoe Landgraf, Stavros Petridis, and Maja Pantic. Emoportraits: Emotion-cchanced multimodal one-shot head avatars. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2498-8507, 2024. (Cited on page 21)
Harminder Dua, Lana A Faraj, Dalia G Said, Trevor Gray, and James Lowe. Human corneal anatomy redefined: novel pre-descemet's layer (dua's layer). Ophthalmology, 120(9)11778- 1785, 2013. (Cited on page 6)
Guy Gafni, Justus Thies, Michael Zollhofer, and Matthias Nießner. Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 8649-8652 8658, 2021. (Cited on page 1)
Xuan Gao, Chenglai Zhong, Jun Xiang, Yang Hong. Yudong Guo, and Juyong Zhang. Recon- structing personalized semantic facial nerf models from monocular video. ACM Transactions on Graphics (TOG), 1-12 2022. (Cited on page 17)
Simon Giebenhain, Tobias Kirschstein, Markos Georgopoulos, Martin Rünz, Lourdes Agapito, and Matthias Nießner. Learning neural parametric head models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 21003-21012, 2023. (Cited on page 17)
Simon Giebenhain, Tobias Kirschstein, Markos Georgopoulos, Martin Rünz, Lourdes Agapito, and Matthias Nießner. Mononphm: Dynamic head reconstruction from monocular videos. In Pro- ceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10747- 10758, 2024a. (Cited on page 17)
Simon Giebenhain, Tobias Kirschstein, Martin Rünz, Lourdes Agapito, and Matthias Nießner. Npga: Neural parametric gaussian avatars. arXiv preprint 1.XX::24055.933, 2024b. (Cited on page 17)
Philip- William Grassal, Malte Prinzler, Titus Leistner, Carsten Rother, Matthias Nießner, and Justus Thies. Neural head avatars from monocular rgb videos. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 18653-18664, 2022. (Cited on page 1, 17)
Kang Han and Wei Xiang. Multiscale tensor decomposition and rendering equation encoding for view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4232-4241, 2023, (Cited on page 6, 19)
Nicholas J Higham and Vanni Noferini. An algorithm compute the polar decomposition of 3x matrix. Numerical Algorithms, 73:349-369, 2016. (Cited on page 20, 21)
Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accurate radiance fields. In SIGGRAPH 2024 Conference Papers. Association for Computing Machinery, 2024. doi: 10. 00.114//264155996654422 (Cited on page 2, 3, 4, 6, 8, 9, 17, 18, 21)
Ying Jiang. Chang Yu, Tianyi Xie, Xuan Li. Yutao Feng, Huamin Wang, Minchen Li, Henry Lau, Feng Gao, Yin Yang, et al. Vr-gs: a physical dynamics-aware interactive gaussian splatting system in virtual reality. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-1, 2024a. (Cited on page 5)
Yingwenqi Jiang, Jiadong Tu, Yuan Liu, Xifeng Gao, Xiaoxiao Long, Wenping Wang, and Yuexin Ma. Gaussianshader: 3d gaussian splatting with shading functions for reflective surfaces. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5322- 5332, 2024b. (Cited on page 8, 21, 25)
12

---

Published as a conference paper at ICLR 2025
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023. URL (Cited on
https: : / repo- sam inria. fr hta.a//a.../uggaaaa/////aaaaa
page 3, 5, 6, 7, 8, 17, 20, 21)
Taras Khakhulin, Vanessa Sklyarova, Victor Lempitsky, and Egor Zakharov. Realistic one-shot mesh-based head avatars. European Conference on Computer Vision, pp. 345-362. Springer, 2022. (Cited page 17)
Rawal Khirodkar, Timur Bagautdinov, Julieta Martinez, Su Zhaoen, Austin James, Peter Selednik, Stuart Anderson, and Shunsuke Saito. Sapiens: Foundation for human vision models, 2024. URL atttps://azan org/ abs 2408 12569. (Cited on page 7)
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. (Cited on page 18)
Tobias Kirschstein, Shenhan Qian, Simon Giebenhain, Tim Walter, and Matthias Nießner. Nersem- ble: Multi-view radiance field reconstruction of human heads. ACM Transactions on Graphics (TOG), 1-14) 2023. (Cited on page 7, 9, 11, 19)
G. Li, K. Sarkar, A. Meka, M. Buchler, Mueller, P. Gotardo, O. Hilliges, and Beeler. Shell- nerf: Learning controllable high-resolution model of the eye and periocular region. Computer Graphics Forum, 43(2)ee15041, 2024. (Cited on page 6)
Gengyan Li, Abhimitra Meka, Franziska Mueller, Marcel C. Buchler, Otmar Hilliges, and Thabo Beeler. Eyenerf: A hybrid representation for photorealistic synthesis, animation and relighting of human eyes. ACM Trans. Graph., (4), jul 2022. ISSN 0730-0301. doi: 10.1145/3528223, 4 5/3522223322222 3530130. (Cited on page 6)
3530130. URL https: / doii org/ /
Tianye Li, Timo Bolkart, Michael. J. Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4D scans. ACM Transactions on Graphics, (Proc. SIG- GRAPH Asia), 19::1 2017. URL https:/ http://bbb..rrr0......111111.... 100011111111000000 3130813. (Cited on page 3, 4, 7, 17, 18, 21)
Zhaoshuo Li, Thomas Müller, Alex Evans, Russell H Taylor, Mathias Unberath, Ming- Yu Liu, and Chen-Hsuan Lin. Neuralangelo: High-fidelity neural surface reconstruction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8456 8465, 2023. (Cited on page 17)
Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images. arXiv preprint 1rX1vv1906.07751, 2019. (Cited on page 17)
Stephen Lombardi, Tomas Simon, Gabriel Schwartz, Michael Zollhoefer, Yaser Sheikh, and Jason Saragih. Mixture of volumetric primitives for efficient neural rendering. ACM Transactions on Graphics (ToG), 40(4):1-13, 2021. (Cited on page 2, 17, 18)
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6):248:1-248111 October 2015, (Cited on page 5)
Shugao Ma, Tomas Simon, Jason Saragih, Dawei Wang, Yuecheng Li, Fernando De La Torre, and Yaser Sheikh. Pixel codec avatars. In Proceedings of the IEEF/CVE Conference on Computer Vision and Paltern Recognition, pp. 64-73, 2021. (Cited on page
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1)999-106, 2021. (Cited on page 1, 17)
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with a multiresolution hash encoding. ACM transactions on graphics (TOG), 41(4):1-15, 2022. (Cited page 17)
13

---

Published as a conference paper at ICLR 2025
Michael Oechsle, Songyou Peng, and Andreas Geiger. Unisurf: Unifying neural implicit surfaces and radiance fields for multi- view reconstruction. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5589-5599, 2021. (Cited page 17)
Keunhong Park. Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp.
2021. (Cited on
page 6)
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
(Cited on page 21)
Pascal Paysan, Reinhard Knothe, Brian Amberg. Sami Romdhani, and Thomas Vetter. A 3d face model for pose and illumination invariant face recognition. In 2009 sixth IEEE international conference on advanced video and signal based surveillance, pp. 296-301. Ieee, 2009. (Cited on page 17)
Shenhan Qian, Tobias Kirschstein, Liam Schoneveld, Davide Davoli, Simon Giehenhain, and Matthias Niefnner Gaussianavatars: Photorealistic head avatars with rigged 3d gaussians. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20299- 20309, 2024. (Cited on page 2, 3, 4, 6, 8, 9, 11, 17, 18, 24)
Eduard Ramon, Gil Triginer, Janna Escur, Albert Pumarola, Jaime Garcia, Xavier Giro-i Nieto, and Francese Moreno-Noguer. H3d-net: Few-shot high-fidelity 3d head reconstruction. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5620-5622 2021. (Cited on page 17)
Radu Alexandru Rosu and Sven Behnke. Permutosdf: Fast multi-view reconstruction with implicit surfaces using permutohedral lattices. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8466-8475, 2023. (Cited on page 17)
Radu Alexandru Rosu, Shunsuke Saito, Ziyan Wang. Chenglei Wu, Sven Behnke, and Giljoo Nam. Neural strands: Learning hair geometry and appearance from multi-view images. In European Conference on Computer Vision, pp. 73-89. Springer, 2022. (Cited on page 21)
Shunsuke Saito, Gabriel Schwartz, Tomas Simon, Junxuan Li, and Giljoo Nam. Relightable gaussian codec avatars. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 130-141, 2024. (Cited on page I)
Zhijing Shao, Zhaolong Wang, Zhuang Li, Duotun Wang. Xiangru Lin, Yu Zhang, Mingming Fan, and Zeyu Wang. Splattingavatar: Realistic real-time human avatars with mesh-embedded gaus- sian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1606-1616, 2024. (Cited on page 2, 4, 5, 7, 8, 17, 18, 24)
Ken Shoemake and Tom Duff. Matrix animation and polar decomposition. In Proceedings of the conference on Graphics interface, volume 92, pp. 258-264, 1992. (Cited on page 2)
Vanessa Sklyarova, Jenya Chelishev, Andreea Dogaru, Igor Medvedev, Victor Lempitsky, and Egor Zakharov. Neural haircut: Prior-guided strand-based hair reconstruction. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 19762-19773, 2023. (Cited on page 21)
Robert W Sumner and Jovan Popović. Deformation transfer for triangle meshes. ACM Transactions on graphics (TOG), 23(3):399-405, 2004. (Cited on page 5)
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint Xrxiv:2106 10689, 2021. (Cited on page 17)
Yaohui Wang, Di Yang, Francois Bremond, and Antitza Dantcheva. Latent image animator: Learn ing to animate images via latent space navigation. arXiv preprint arXiv:2203.0900 090443 2022. (Cited page 21)
14

---

Published as a conference paper at ICLR 2025
Tim Weyrich, Simon Heinzle, Timo Aila, Daniel B Fasnacht, Stephan Oetiker, Mario Botsch, Cyril Flaig, Simon Mall, Kaspar Rohrer, Norbert Felber, et al. hardware architecture for surface splatting, ACM Transactions Graphics (TOG), 26(3):00 2007. (Cited page 4)
Jun Xiang, Xuan Gao, Yudong Guo, and Juyong Zhang. Flashavatar: High-fidelity head avatar with efficient gaussian embedding. In Proceedings of the IEEE/CVF Conference Computer Viston and Pattern Recognition, pp. 1802-1812, 2024. (Cited on page 17)
Tianyi Xie, Zeshun Zong, Yuxing Qiu, Xuan Li, Yutao Feng, Yin Yang, and Chenfanfu Jiang. Physgaussian: Physics-integrated 3d gaussians for generative dynamics. Proceedings of the JEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4389 4399, 2024. (Cited on page
Baixin Xu, Jiarui Zhang. Kwan-Yee Lin, Chen Qian, and Ying He. Deformable model-driven neural rendering for high-fidelity 3d reconstruction of human heads under low-view settings. In Proceed- ings of the IEEE/CVF International Conference Computer Vision, pp. 17924-17934, 2023a. (Cited on page 17)
Kun Xu, Wei-Lun Sun, Zhao Dong, Dan-Yong Zhao, Run-Dong Wu, and Shi-Min Hu. Anisotropic spherical gaussians. ACM Transactions on Graphics (TOG), 32(6):1-11, 2013. (Cited on page 3, 6, 18)
Yuelang Xu, Lizhen Wang, Xiaochen Zhao, Hongwen Zhang, and Yehin Liu. Avatarmav: Fast 3d head avatar reconstruction using motion-aware neural voxels. In ACM SIGGRAPH 2023 Confer- ence Proceedings, pp. 1-10, 2023b. (Cited on page 17)
Yuelang Xu, Benwang Chen, Zhe Li, Hongwen Zhang, Lizhen Wang, Zerong Zheng, and Yebin Liu. Gaussian head avatar: Ultra high-fidelity head avatar via dynamic gaussians. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1931-1941, 2024. (Cited on page 17, 20, 22)
Haotian Yang, Mingwu Zheng, Wanquan Feng, Haibin Huang, Yu-Kun Lai, Pengfei Wan, Zhongyuan Wang, and Chongyang Ma. Towards practical capture of high-fidelity relightable avatars. In SIGGRAPHI Asia 2023 Conference Pupers, pp. 1-11, 2023. (Cited on page 1)
Ziyi Yang, Xinyu Gao, Yangtian Sun, Yihua Huang, Xiaoyang Lyu, Wen Zhou, Shaohui Jiao, Xi- aojuan Qi, and Xiaogang Jin. Spec-gaussian: Anisotropic view-dependent appearance for 3d gaussian splatting. arXiv preprint arXiv:2402. 15870, 2024. (Cited on page 19)
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lip- man. Multiview neural surface reconstruction by disentangling geometry and appearance. Ad- vances in Neural Information Processing Systems, 33:2492-25555 2502, 2020. (Cited on page 17)
Egor Zakharov, Vanessa Sklyarova, Michael Black, Giljoo Nam, Justus Thies, and Otmar Hilliges. Human hair reconstruction with wrand-aligned 3d gaussians. ArXiv, Sep 2024. (Cited on page 21)
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018. (Cited page 7)
Yufeng Zheng, Victoria Fernández Abrevaya, Marcel C Bühler, Xu Chen, Michael Black, and Otmar Hilliges. Im avatar: Implicit morphable head avatars from videos. In Proceedings the JEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 13545-13555. 2022. (Cited on page 1, 17)
Yufeng Zheng, Wang Yifan, Gordon Wetzstein, Michael J Black, and Otmar Hilliges. Pointavatar: Deformable point-based head avatars from videos. In Proceedings of the IEEF/CVF conference on computer vision and patlern recognition, pp. 21057-21067, 2023. (Cited on page 1, 7, 8, 17)
Wojciech Zielonka, Timur Bagautdinov, Shunsuke Saito, Michael Zollhöfer, Justus Thies, and Javier Romero. Drivable 3d gaussian avatars. arXiv preprint (.Xiv:2311.0888) 2023a. (Cited on page 5)
15

---

Published as a conference paper at ICLR 2025
Wojeiech Zielonka, Timo Bolkart, and Justus Thies. Instant volumetric head avatars. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4574 4584, 2023b. (Cited on page 1, 2, 5, 17, 18)
Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross. Ewa volume splatting. In Proceedings Visualization, 2001. VIS"O1,, pp. 29-538. IEEE, 2001. (Cited on page 4)
16

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 RELATED WORK
Static to Dynamic Radiance Field Reconstruction. NeRF (Mildenhall et al., 2021) and NV (Lom- bardi et al., 2019) have opened the етa of photorealistic renderings with undertaking novel-view synthesis task. Optimization and rendering efficiency could be improved by hash encoding (Müller et al., 2022) and tensor decomposition (Chen et al., 2022). Later, Mixture of Volumetric Prim- itives (Lombardi et al., 2021) proposes efficient methodology with surface-aligned cuboid prim- itives which boost only ray marching around surfaces. Few years later, 3D Gaussian Splatting (3DGS) (Kerbl et al., 2023) utilizes anisotropic 3D Gaussians by rasterizing them without expensive ray-marching like previous strategies. We leverage the 3DGS, benefiting from the expressiveness and efficiency.
Neural Surface Reconstruction. Emerging from the success of NeRF (Mildenhall et al., 2021), neural surface reconstruction (Oechsle et al., 2021; Yariv et al., 2020; Wang et al., 2021) has garnered significant attention from researchers, leveraging Occupancy Fields and Signed-Diin Deante--isttace Functions (SDFs). Notably, subsequent works (Rosu & Behnke, 2023; Li et al., 2023) have introduced method- radiance fields (Müller et al., 2022) with SDFs. This development has inspired research into human head geometry reconstruction, moving beyond mesh-based topology-invariant 3DD Morphable Face Models (3DMFMs) (Li et al., 2017; Paysan et al., 2009). The first major work in this area, H3D-Net (Ramon et al., 2021), proposed an SDF-based approach for few-shot human head geometry reconstruction using an auto-decoder- trained model. Building on this, subsequent study (Xu et al., 2023a) targeted a similar setting as H3D-Net, focusing on more detailed human head geometry reconstruction with single-person refinement. Recently, MonONPHM (Giebenhain et al., 2024a) introduced a few-shot head recon- struction framework based on pretrained neural parametric head model (Giebenhain et al., 2023), which combines the topology-fre advantages of SDFs with the manipulability of parametric mod- els. Departing from these SDF-based approaches, we are the first to propose reconstructing dynamic head geometry from RGB videos, including monocular setups, utilizing Gaussian primitives (Huang et al., 2024).
ologies that integrate fast and efficient hash-encoding-
Reconstructing and Animating Head Avatars. Existing approaches for reconstructing and animat- ing avatars mainly differ in two fundamental aspects: implicit or explicit models. Implicit models reconstruct the face by neural radiance field in combination with volumetric rendering or using im- plicit surface functions. (Gao et al., 2022), (Zielonka et al., 2023b). (Zheng et al., 2022), (Xu et al., 2023b) With explicit models (Grassal et al., (022),(Zhenp et al., 2023)) (Khakhulin al., 2022), the seminal work of 3DMFM uses principal component analysis (PCA) to model facial appearance and geometry on a low-dimensional linear subspace. 3DMFM and its variants have been widely applied in optimization-base rendering and topological flexibility, there are many works utilizing 3D Gaussian Splatting (Kerbl et al., 2023). These works can be further categorized based on how they define and use Gaussians. Some approaches bind Gaussians directly to the mesh (Shao el al., 2024; Qian et al., 2024), while others map them onto UV coordinates (Xiang et al., 2024). In some cases, Gaussians are extracted as features (Giebenhain et al., 202 et al., 2024), or a neural parametric model replaces the traditional 3DMFM (Giehenhain al., 2024b). However, in most cases, the deformation is often handled by the mesh itself with little additional consideration.
and deep learning-based head avatar creation. Recently, due to its efficient
A.2 PROOFS
A.2.1 POLAR DECOMPOSITION'S UNIQUENESS
Lemma 1, There exists a unique polar decomposition of an arbitrary invertible matrix A such that
A = UP,
where is an orthogonal matrix and P is a positive semidefinite symmetric matrix. Proof (Lemma 1). Suppose that there are two polar decompositions of the matrix A:
A U,P UpP2, where U1 U2 are orthogonal matrices and P Ps are positive semidefinite symmetric matrices.
17

---

Published as a conference paper at ICLR 2025
Now, multiply both sides of U,PP U2P by U[ on the left:
Pi UfU,P.
Let Q U2 that
P1 = P
Since Q is orthogonal matrix and both P1 and P2 are positive semidefinite symmetric matrices, we conclude that Q = I, the identity matrix. Hence,
U
U1 U.
Thus, P1 = P2, and therefore the polar decomposition = UP unique.
A.2.2 PRESERVATION OF POSITIVE SEMIDEFINITE WITH JACOBIAN GRADIENTS
Lemma 2. Consider an arbitrary matrix M. The matrix multiplication M MM" is positive semidefinite (PSD) because for any vector
x MM + | 0.
The matrix also follows this form. Therefore, E retains its positive semidefinite property after the affine transformation.
A.3 IMPLEMENTATION DETAILS
Optimization Specifications. We use Adam (Kingma Ba, 2014) optimizer for learnable Gaussian parameters and translation, joint rotations, and expression parameters of FLAME (Li et al., 2017). We set the learning rates for all parameters same as with GA (Qian et al., 2024), except for the blending weights For the w, set the learning rate 1e-3. We train for 300,000 iterations, and exponentially decay the learning rate for the μ until the final iteration, where it reaches 0.01 x the initial.
Finally, the entire energies are defined as:
L = Lphoto + Adepth + Apormal noommal + Aaye Leye-
(5)
The energy balances Adepth 100, Anormal 0.05, and Aeye 0.1.
Calculation Jacobian with Covariance, Since there is no support for precomputation of covariance with original 2DGS (Huang et al., 2024) rasterizer implementation deformation gradient Jn combining with 3DGS's original covariance as NVIDIA CUDA kernel implementation.
calculate the Jacobian
Color Change with Geometrical Transformations.
Most previous dynamic head avatars almost neglect the color change with deformations (Zielonka et al., 2023b; Qian et al., 2024; Shao et al., 2024; Lombardi et al., 2021). When the bases of SHs do rotate, they just query the same direction with facing identical side of Gaussians in any deformed space. This phenomenon is illustrated in the inset of the upper right branch. To mitigate this, suggest simple solution like the lower right of the in- set, we inversely rotate the view direction with the inverse of Gaussian's s blended rotation part U/" Namely, rewrite the rotated input view direc- tion: d d.'..
Anisotropic Spherical Gaussians for Eyeballs' specularity. Comparing with Spherical Gaussians (SGs), ASGs (Xu et al., 2013) have been demonstrated to effectively represent anisotropic scene with a relatively small number.
ASGs are theoretically defined as:
ASG(v/lz.v.),, 1A. max(v 0) e- -M(ue))--((u1))
(6)
18

---

Published as a conference paper at ICLR 2025
self-reenact set
Our self-reenact set
GA seif-reenact set
Our self-reenact set
SUBIECT
SUBJECT 264
Figure 7: Dataset example custom self-reenactment held-out.
Subject
074
140
175
210
253
264
302
304
306
Table 4: The held-out sequence of each subject for self-reemactmeet evaluation.
where is lobe-axis, x and y are each tangent and bi-tangent of z, {), R+ are sharpness parameters, v is the unit direction function input, and the mar term implies the smooth term.
Given that most specular BRDFs feature lobe aligned with specific reflection direction, cal- culate the reflection vector to serve as the input direction: wo = 2( n)n d where d is the rotated input view direction, is normal direction computed for each Gaussian, and wa the reflect direction. Lastly, since a simple summation of ASG output limits its representative ability, utilize tiny two-hidden layered MLP F to obtain the final specular color, following (Yang al., 2024):
NN1 A
c=(woi F
ASG(wollw) wf, wi). Dr (.) dios 1=0
where ① indicates the concatenation operation, 7 denotes the positional encoding, and N is the number of basis SGs.
Not compromise the computational efficiency, we leverage the common prior knowledge from the casual data capture setting. First, the eyeballs could not be effected by lights from backside of heads. Therefore, we sample w, with limited range from frontal hemisphere of world space; relatively 50% reduced ASGs compared with (Han & Xiang, 2023; Yang et al., 2024). Second, since the captured data typically lie under environments with abundant white light, representing the MLP F output color as monochrome channel for intensity is sufficient.
For sampled direction wi (0, Φ) in the spherical coordinate system, we set w/ = (0 + T/2). Using quaternion operations, w/ is then rotated around w; by x x/2 derive w.t Then, we finally obtain the color with ca + c, used in volume rendering equation, where cæ is obtained by SHs from 3DGS (Kerbl et al., 2023).
NeRSemble Real Dataset Division We conduct experiments on video recordings of 9 subjects from the NeRSemble (Kirschstein et al., 2023) dataset as described on Tab. 4. As we focus on deformation, select extreme expression set for relf-reenattment to emphasize the difference. Therefore, among the emotion (EMO) and expression (EXP) sequences, we hold out EMO-1 for self-reenactment evaluation and the rest nine for training shown in Tab. 5.
19

---

Published as a conference paper at ICLR 2025
Train Sequences, EMO-2 EMO-3 EMO-4 EXP-2 EMO-3 EMO-4 EMO-5 EMO-8 EMO-9
Test Sequences
EMO-1
Table 5: The train and test held-out sequences.
Method
GA
Base
+ Jacobian +JBS
+SGs (= Ours)
+PyTorch 71.18
109.36
107.59
92.62
90.13
Rendering Speed (FPS)
+RoMA N/A
N/A
N/A
97.29
94.72
Training Time (hours)
1.65
1.68
1.73
1.98
2.11
Table 7: Comparison of training and rendering (test) computational costs.
A.4 ADDITIONAL EXPERIEMENT
Comparsion with Gaussian Head Avatars (GHA) (Xu et al., 2024) GHA serves strong base- line in rendering quality, leveraging an additional super-resolution model in screen space. As shown in Tab. 6, GHA outperforms our method in terms of PSNR. However, in other metrics such as SSIM and LPIPS, GHA falls behind.
Fig. 8 reveals potential reasons for this discrepancy. Notably, arti- facts such as over-saturation are visible, which we attribute GHA's screen-space refinement. This refinement struggles when rendering extreme poses that fall outside the distribution expected by the super- resolution model. Additionally, GHA faces challenges in reconstruct- ing high-fidelity geometry and handling extreme expressions, both of which are key strengths of our approach. We would like emphasize the importance of capturing the specular highlights of the eyes, particularly the cornea. GHA renders the pupils with a matte appearance, neglecting the -rehffreeency specular reflections in the corneal region. We argue that these specular details are critical for enhancing the realism of the eyes, which play pivotal role creating immersive and lifelike head avatars.
Table 6: Quantitative com- parsion with GHA (Xu et al., 2024)
PSNR1
SSIMI
LPIPS)
NCS
GHA
27.25
0.909
0.155
0.505
Ohirs
26.20
0.932
0.052
0,37
A.5 ADDITIONAL ABLATION STUDY
Adaptive Density Control (ADC) Amplification on Teeth. We posted qualitatives with the oc- clusion amplification the teeth part in Fig. 9. This technique is very simple yet efficient with boosting Gaussian proliferation to overlooked parts in terms of optimization.
Training and Rendering Time Complexity Measurement Tab. summarizes the rendering speed and training time for our method and GaussianAAvtars (Kerbl et al., 2023). The Base configuration refers to replacing the 3D Gaussian Splatting rasterizers in CaussianAvatars with their 2D coun- terparts. Our method incurs only a 17% drop in rendering speed and an additional 25 minutes of training time, while still achieving 3x real-time rendering speeds (generally over 30 FPS) and main- taining efficient training. A detailed analysis attributes the minimal training and testing overhead to our GPU-level CUDA kernel implementation for Jacobian computations.
For rendering, the primary factor behind the speed reduction is the Jacobian Blend Skinning (JBS), where the overhead mainly arises from the Polar Decomposition step. Our current implementation utilizes PyTorchhss ' SVD, which relies on the cuSOLVER backend. To further investigate this bot- tleneck, we conducted additional experiments using RoMA (Brégier, 2021)'s specialized Procrustes routine, which is designed to efficiently compute the 3 x 3 unitary matrix U of the Jacobian J. No- tably, replacing torch. svd with roma. special weeeccal-procruse proccrustes yielded a performance gain of approximately 4-5 FPS.
Although this demonstrates the potential of alternative approaches, there is still room for further improvement. Higham's 's routine (Higham & Noferini, 2016), specifically tailored for 3 x 3 matrices, offers a promising direction to address this overhead and is well-suited for CUDA-based implemen- tations.
20

---

Published as a conference paper at ICLR 2025
A.6 LIMITATION AND SEMINAL FUTURE WORK
Bounded Representation from 3DMFM. We utilized a 3DMFM model, FLAME (Li et al., 2017) to enable our model be equipped cross-reenactment and rigged compactly. However, the 3DMFMs' expression space has only PCA-based coarse rigging-ability such that cannot easily de- scribe the dynamics of facial muscles such as wrinkles and extreme pose or expression (Fig. 10).
To end this, we believe that the image-based latent expression representations (Burkov et al., 2020; Wang et al., 2022: Drobyshev al., 2022; 2024) can alleviate this issue some extent. However, naively applying the latent expression to dynamic head avatars is non-triival.. Because only designing dynamics of the geometrical deformation hard handle ambient-occluue neck and jaw from head and lips, each) without intricate physical-basee rendering (PBR). Therefore, we guess that the future work should be conducted as learning the deformation dynamics with those latent expression spaces, also simultancously covering the deformation-awes -warr color changes.
(such as shadow of
Second, the inherent limitations of 3DMFM's representation, particularly the absence of hair strands and the tongue, can degrade the render quality. Since 3DMFM is fundamentally bald model for human heads, it lacks the capability to represent the high-frequency details human hair (Fig. model with parametric strand models (Sklyarova et al., 2023;
11). We believe that integrating Zakharov et al., 2024; Rosu et al., 2022) could help alleviate these issues.
Computational Efficiency for Calculating Polar Decomposition with Numerical Views. Our Jacobian Blend Skinning (JBS) algorithm is hased on polar decomposition, which is indirectly com- puted using byproducts of singular value decomposition (SVD). Specifically, J = WEVT, where W and V are orthogonal matrices, and Σ is a diagonal matrix with singular values. This can be rewritten as
(VEV')) UP, where U is the rotation matrix, and P is the positive semidefinite symmetric matrix.
However, as reported in RoMa (Brégier, 2021), PyTorch (Paszke et al., 2019)'s SVD routine scales linearly in time as the batch size increases. In the Gaussian Splatting (Kerbl et al., 2023; Huang et al., 2024) regime, over 10K operations for Gaussians are performed simultaneously to render an image, which requires the SVD routine to be executed in parallel. The polar decomposition process involves finding the nearest rotation matrix given matrix. We believe that adopting RoMa (Brégier, 2021)'s Procrustes process Higham's 's (Higham & Noferini, 2016) routine, which is specifically designed to solve problems for 3 matrices, could significantly improve efficiency.
Extending SurFhead to Relighting Task. Normal is one of the key intrinsic material properties. We demonstrate the possibility of using the normals generated by SurFhead for avatar relighting tasks in Fig. 13, utilizing GaussianShader (Jiang et al., 2024b). The relit results exhibit high plausi- bility, with high-fidelity reflections of the environment maps. We present objects reconstructed using method, including relighting scenarios under various lighting conditions, featuring both warm and cool tones, as well as indoor and outdoor environments. Our renderings, across three diverse sets, convincingly demonstrate that the relit scenes maintain realism, exemplifying our method's proficiency in relighting applications.
21

---

Published as a conference paper at ICLR 2025
Render
Normal
GT
GHA
Ours
Figure 8: Qualitative Comparison with GHA (Xu et al., 2024)
teeth
Ours
GT
Figure 9: Ablations for ADC amplification on teeth.
22

---

Published as a conference paper at ICLR 2025
GT
FLAME Mesh
Overlay
Figure 10: Limitation#1. Example for bounded representation of expression space from 3DMFM.
GT
RGB
Normal
Figure 11: Limitation #2: The lack of hair modeling 3DMFM can result in blurry hair appearance.
23

---

Published as a conference paper at ICLR 2025
A.7 CONCEPTUAL COMPARISON WITH PREVIOUS METHOD
Previous methods, (Qian et al., 2024; Shao al., 2024), utilize isotropic scaling for de- forming mesh-bound Gaussians, with scalar scaling parameter Sy. GaussianAvatars com- puted Sp using the mean length of one of the edges and its perpendicular, while SplattingA- vatars computed Sp based on the ratio of canon- ical and deformed triangle areas. However, we find that these approaches can result in undesir- able Gaussian deformations when the 3DMFM meshes undergo stretch shear in specific di- rections. When the deformed and canonical tri- angle areas the sum of an edge's length and its perpendicular are the same but the shape is differ, the Gaussians become misaligned and fail cover the necessary regions accurately. In con- trast, our method uses affine deformation, allowing Gaussians to more precisely reflect the deforma- tion of the triangles and cover the regions as intended.
Stretching
14, aoda)
Shearing msain))
Antotospe Soaling Delamred (Oun)
Deformad
Figure 12: Conceptual illustration of stretching and shearing: (Left) rigid and isotropic scaling and (Right) affine deformation (ours)
24

---

Published as a conference paper at ICLR 2025
RGB
Envmap
Firepalce
Night
Studio
Figure 13: Results for relighting with Shader (Jiang et al., 2024h).
Figure 14: SurFhead can reconstruct high-fidelity meshes from Truncated Signed Distance Fuen- tion (TSDF) alongside diverse pose and expression.
25


Reviews:
[{"review_id": "pxO9S7LfeZ", "reviewer": "Reviewer_D2NV", "summary": "This paper presents a method for learning head avatars based on 2D Gaussian splatting. To make the Gaussian surfels better handle the stretch and shear deformation under extreme poses and facial expressions, this paper introduces affine transformation derived from Jacobian deformation gradient of the surface. Normal orientations are calculated accordingly. Moreover, the authors propose Jacobian blend skinning to interpolate these affine transformations to ensure surface smoothness. Results show that the proposed method is able to reconstruct drivable head avatars with high-quality geometry.", "questions": "See [Weaknesses].", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This paper introduces better deformation modeling techniques for Gaussian surfels. Compared to existing works, the proposed method is more reasonable and can handle more extreme deformations such as stretching and shearing. The proposed technique could be useful in other related research topics beyond head avatar modeling. \n\n* The proposed method is able to reconstruct fine geometric details, outperforming existing baselines by a large margin. \n\n* The paper is overall well-written and easy to follow.", "weaknesses": "* Missing comparison against Gaussian Head Avatar (GHA) [Xu et al. 2023], which is a state-of-the-art head avatar method in terms of image synthesis quality. Although the authors have already compared with SplattingAvatar and GaussianAvatar, I think an additional comparison against GHA is also necessary because GHA demonstrates high-resolution image synthesis with the assistance of a super-resolution module. \n\n* It would be better if the authors report the training time and rendering speed. One important advantage of Gaussian splatting is its efficiency. I wonder whether the proposed techniques (such as Jacobian blend skinning) hinders this advantage or not. \n\n* It is not clear how the proposed method performs for subjects wearing eye-glasses. NeRSemble dataset contains cases with eye-glasses, but they are suspiciously skipped in the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vK1Ej5H8Bp", "reviewer": "Reviewer_u9qy", "summary": "This paper aims at better geometry estimation in head modeling. It replaces the 3D Gaussian splatting with 2D Gaussian splatting to better model the surface. Moreover, it addresses three issues in existing works with three novel components: 1) To compensate for the incorrect deformation of 2D Gaussians due to the triangle's shear and stretch, it proposes the Jacobian deformation; 2) To mitigate the discontinuities in adjacent triangles, it improves linear blend skinning with Jacobian blend skinning; 3) To resolve hollow illusion in eyeballs, it replaces Spherical Harmonics with Anisotropic Spherical Gaussians. It demonstrates that it outperforms the state-of-the-art regarding normal similarity and remains comparable in rendering quality.", "questions": "* It is unclear to me how Jacobian blend skinning improves upon linear blending skinning. It seems to me that $J_b$ introduces spatial smoothness to Gaussians' deformation matrices. Is the vertices of the mesh still transformed by linear blending skinning before the Jacobian blend skinning is applied? Also, could the author clarify Fig. 2(b)? What are the meanings of the green and yellow lines and the weights? How do they connect to the deformation of the triangle meshes?\n* The paper mainly focuses on improving the geometry. Since 2D Gaussian splatting is known to produce better geometry than 3D Gaussians, how much does 2D Gaussians help improve the geometry, compared to the components proposed in the paper?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* It motivates each contribution properly.\n* The use of Jacobian deformation and Jacobian blend skinning in the context of head modeling looks novel to me.\n* The qualitative results and normal similarity evaluation demonstrate better geometry compared to baselines.\n* The effectiveness of each component is well studied.", "weaknesses": "* It is unclear how the deformation gradient $J$ in Sec 2.2 and the blended Jacobian $J_b$ in Sec 2.3 connect. In Line 258 to 261, it mentions that it replaces the original deformation in GaussianAvatar with a new deformation $J_b$, and $J$ appears only as a parameter of JBS in Eq. 2. What is the relationship between $J$ and $U_i, P_i$ in Eq. 2? Which transformation, $J$ or $J_b$, is used in the final method?  \n* The paper measures the normal similarity between ground-truth normals and rendered normals from 2D Gaussians. However, since this work claims to achieve a better geometry, evaluating metrics that apply to meshes, such as Chamfer distance or normals rendered from the mesh is more informative when judging the geometry. Although previous methods use normals rendered from 2D Gaussians as proof of geometry, the link between it and the mesh quality still looks vague to me.\n* As the rendering quality is only on par with state-of-the-art, e.g., GaussianAvatar, the comparison of training and rendering speed is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tFa8GJHH00", "reviewer": "Reviewer_RtVj", "summary": "This paper contributes a novel representation for geometrically accurate head avatars within the 3D Gaussian Splatting framework, a method for natural interpolation of affine transformations across adjacent deformations, and enhancements to the realism of corneal representations in head avatars. These contributions advance the state of the art in personalized head avatar construction and have the potential to improve various applications in computer graphics, virtual reality, and beyond.  The key contributions include:\n\n* Introduction of SurFhead Model: The paper introduces SurFhead, the first geometrically accurate head avatar model within the Gaussian Splatting framework. This model is designed to capture the deformation of head geometry using intricate affine rigging that combines Gaussians and their normals solely from RGB videos.\n\n* Jacobian Blend Skinning Algorithm: To address the issue of discontinuities between adjacent triangles in head deformations, the paper proposes the Jacobian Blend Skinning (JBS) algorithm. This algorithm blends adjacent transformations while avoiding geometric distortions by linearizing the non-linear matrix interpolation space, leveraging classical matrix animation techniques and geometrically smooth polar decomposition.\n\n* Enhancement of Corneal Convexity and Specularity: The paper addresses the hollow illusion in the cornea by regularizing corneal convexity and enhancing specularity using computationally efficient Anisotropic Spherical Gaussians (ASGs). This improvement ensures a more realistic representation of the cornea in the head avatar.", "questions": "As elaborated in the \"weaknesses\", the geometric accuracy of the experimental results exhibits considerable variation,  and we hope that the authors can further enhance their analysis and discussion on this crucial point.\n\nIn terms of experimental design, the NeRSemble dataset, while containing accurate 3D mesh models, lacks sufficient detail, and obtaining such precise models can often be challenging in practical applications.     In this regard, we eagerly inquire whether the proposed method heavily relies on such relatively accurate 3D mesh models, and how it would perform in their absence.     To validate this, the authors could consider using videos they have shot or sourced from the internet as input, employing monocular facial 3D reconstruction algorithms (such as DECA) to obtain mesh sequences, or directly bypassing the use of 3D mesh models altogether.     We are anticipating and curious about the results of such experimental setups.    We encourage the authors to actively explore and propose potential strategies for adapting their proposed approach to scenarios where detailed 3D mesh models are unavailable.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces SurFhead, a novel model within the Gaussian Splatting framework that captures geometrically accurate head deformations. This representation utilizes intricate affine rigging combined with Gaussians and their normals, solely based on RGB videos, which is a significant advancement in achieving realistic and detailed head avatars.  The proposed Jacobian Blend Skinning (JBS) Algorithm is technically sound.  The paper tackles the problem of the hollow illusion in the cornea, where a concave surface appears convex due to the prioritization of photometric losses during training. \n\nThe methods presented in the paper are demonstrated to achieve superior results across a variety of subjects, including real and synthetic data. They excel in challenging scenarios such as sharp reflections on convex eyeballs, fine geometric details, and exaggerated deformations, showcasing the robustness and effectiveness of the proposed approach.", "weaknesses": "The method is built upon the foundation of 2D Gaussian Splatting, and I believe that the proposed method's success in recovering a superior surface geometry owes much to this solid groundwork. Indeed, the authors introduced several improvements on this basis, such as intricate affine rigging, but I consider these innovations to be more incremental improvements rather than groundbreaking advancements.\n\nThe geometric accuracy of the experimental results appears to exhibit significant variation: some achieve hair-level geometric detail, while others fail to recover the structure of the hair. Consequently, whether this variation stems from instability in the algorithm or differences in the data quality of various training datasets arises. I hope the author can provide more analysis and discussion on this issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lDvtQmHiDE", "reviewer": "Reviewer_Cr49", "summary": "The paper proposes a method that handles stretch and shear transforms essential for detailed deformations of geometry utilizing intricate deformations driven by the affine Jacobian gradient instead of similarity transformation and corresponding normal adjustments.", "questions": "1) Related Works: I think some related works of static reconstruction could be further discussed, such as H3DS [1], deformable model-driven approaches [2], and Implicit Neural Deformation methods [3]. These methods leverage 3D points from SfM, multi-scans, or multi-view data from various identities to enhance reconstruction under sparse-view conditions, although they primarily focus on static human face or head geometry reconstruction.\n\nRef:\n\n[1] Ramon, Eduard, et al. \"H3d-net: Few-shot high-fidelity 3d head reconstruction.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n\n[2] Xu, Baixin, et al. \"Deformable model-driven neural rendering for high-fidelity 3D reconstruction of human heads under low-view settings.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[3] Li, Moran, et al. \"Implicit Neural Deformation for Sparse‐View Face Reconstruction.\" Computer Graphics Forum. Vol. 41. No. 7. 2022.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) Jacobian Blend Skinning (JBS): The use of Jacobian Blend Skinning (JBS) enables natural interpolation of affine transformations across adjacent deformations, effectively reducing discontinuities in transitions.\n\n2) Cornea Opacity Constraint: To address the specular highlights in the eyeball region, the method constrains the corneal regions to remain opaque by regularizing the opacity of the respective Gaussians.", "weaknesses": "1) Detail Representation: In Figure 5 (bottom row), there seems to be a lack of finer details, such as wrinkles. Adding more visual comparisons or details on addressing such high-frequency features could strengthen the analysis.\n\n2) Rendering Speed and FPS: Given that methods like 3DGS/2DGS achieve real-time rendering, the speed of deformable-driven methods may be a limitation for applications requiring real-time animation. Could you report the FPS compared to other methods to clarify performance in time-sensitive scenarios?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)