Perm: A Parametric Representation for Multi-Style 3D Hair Modeling

## Abstract

Abstract. We present Perm , a learned parametric model of human 3D hair designed to facilitate various hair-related applications. Unlike previous work that jointly models the global hair shape and local strand details, we propose to disentangle them using a PCA-based strand representation in the frequency domain, thereby allowing more precise editing and output control. Specifically, we leverage our strand representation to fit and decompose hair geometry textures into low- to high-frequency hair structures. These decomposed textures are later parameterized with different generative models, emulating common stages in the hair modeling process. We conduct extensive experiments to validate the architecture design of Perm , and finally deploy the trained model as a generic prior to solve task-agnostic problems, further showcasing its flexibility and superiority in tasks such as 3D hair parameterization, hairstyle interpolation, single-view hair reconstruction, and hair-conditioned image generation. Our code, data, and supplemental can be found at our project page: https://cs.yale.edu/homes/che/projects/perm/ .

## 1. Introduction

Figure: Figure 2. An application demo of Perm, where we first fit Perm parameters to reconstruct 3D hair from the input image, then edit the parameters to change its style from straight to wavy, and finally use the edited hairstyle and certain viewing direction as a condition for image generation with Adobe Firefly [Adobe, ].
Refer to caption: extracted/5780499/fig/img/demo/bbj-000015.png

Hair, as a crucial component in the realm
of digital humans, has been studied for decades in computer graphics. With the recent availability of high-quality 3D hair data [Hu et al., ; Shen et al., ], numerous methods have emerged in the field of hair modeling, demonstrating their effectiveness in applications such as hair manipulation [Chai et al., ; Weng et al., ], hair dynamics modeling [Yang et al., ; Wang et al., ], novel hairstyle synthesis [Zhou et al., ; Sklyarova et al., ], and 3D hair reconstruction either from images [Wu et al., ; Zheng et al., ; Takimoto et al., ; Kuang et al., ] or monocular videos [Wu et al., ; Luo et al., ; Sklyarova et al., ]. Despite their achievements, these methods often overlook the inherent *biscale* nature of hair, where hairstyles with similar global shapes may exhibit variations in local details. For instance, blowout hairstyles with a similar shape can have variations ranging from a sleek and smooth straight look to more modern styles such as beach waves or bouncy curls, depending on the blow-drying techniques employed(^1^11[https://classpass.com/blog/blowout-types/#](https://classpass.com/blog/blowout-types/#)).
Neglecting these biscale variations limits the generation and editing capability,
hindering applications such as changing a hairstyle from straight to curly while maintaining a similar haircut. Moreover, most of the models proposed in existing methods are *task-specific*, lacking the generalization capability to different down-stream tasks. In contrast, the field of human body modeling has well-accepted parametric models like SMPL [Loper et al., ] and MANO [Romero et al., ], which designed disentangled coefficients for pose and shape and are widely applied as a generic prior in deep learning applications such as human reconstruction and animation. To address this gap, we propose Perm, a parametric model of 3D human hair that can serve as an effective prior for various hair-related applications.

Perm is formulated as $\mathcal{M}(\vec{\theta},\vec{\beta},\mathcal{R})$, where the output hair is conditioned on the guide strand parameter $\vec{\theta}$ for its global shape, the hair styling parameter $\vec{\beta}$ for its local details, and a pre-defined root set $\mathcal{R}$ for the hair density.
To independently control the global shape and local details, we propose a strand representation based on principal component analysis (PCA) in the frequency domain, which fits the nearly periodic behavior of hair strands.
The learned principal components form an *explainable* space for hair decomposition, where the initial components capture low-frequency information, such as rough shape and large deformation, and the subsequent components encode high-frequency details like curliness.
Based on this strand representation, we fit geometry textures for each hairstyle and decompose them into *guide textures* for guide strands and *residual textures* for local strand details, facilitating precise editing and output control.
With these decomposed texture maps, we design different generative models to emulate common modeling stages in existing hair modeling software like Maya XGen [Autodesk, ]. Specifically, we train a StyleGAN2 [Karras et al., ] for guide textures, serving as an automated guide strand authoring system conditioned on parameters. We train a Variational Autoencoder (VAE) [Kingma and Welling, ] for residual textures, replacing those procedures with learned but not manually designed parameters. Finally, we introduce a U-Net [Ronneberger et al., ] super resolution module to upsample guide textures, replicating the guide interpolation stage.

After training Perm,
we demonstrate its capability across multiple applications, including 3D hair parameterization, hairstyle interpolation, and single-view hair reconstruction. Despite not being trained specifically for any of these tasks, Perm achieves performance equivalent or superior to state-of-the-art task-specific alternatives in our experiments. Moreover, we introduce a novel application of using Perm-generated hair for conditional image generation. A demo of these applications is provided in Fig. 2.

Our code and data will be publicly available after review process.

## 2. Related Work

Figure: Figure 3. Overview of Perm, where a guide strand function $\mathcal{G}(\vec{\theta})$ takes the parameter $\vec{\theta}$ to synthesize the guide texture $\hat{\mathbf{T}}_{\text{guide}}$ and guide mask $\hat{\mathbf{M}}_{\text{guide}}$. They are then super-resolved with the function $\mathcal{F}(\cdot)$ to obtain the upsampled texture $\hat{\mathbf{T}}_{\text{low}}$. For the high-frequency part, a hair styling function $\mathcal{D}(\vec{\beta})$ maps the parameter $\vec{\beta}$ to the residual texture $\hat{\mathbf{T}}_{\text{res}}$. These two textures are concatenated to from the geometry texture $\hat{\mathbf{T}}$, which can finally be sampled and decoded with the function $\mathcal{S}(\cdot)$ to obtain the strand geometry.
Refer to caption: x3.png

The unstructured nature of hair makes its modeling a long and tedious process. For decades people have been studying advanced ways to model hair geometries. Here we review this literature.

##### Geometric Hair Modeling

Techniques for geometric hair modeling typically involve explicit descriptions of hair structure. Observing the similarity among neighboring hair strands, early approaches utilize representations such as wisps [Watanabe and Suenaga, ], bundles of cylinders [Chen et al., ], or coarse polygonal meshes [Yuksel et al., ].
Similar strip representations have also been employed in reconstructing hair from single [Chai et al., ] or multiple
images [Luo et al., ], as well as in interactive hair modeling and immersive control [Xing et al., ].
However, as these low-resolution geometric-based models often lack fine-grained details, subsequent works shifted their focus to model each strand as a sequence of connected 3D points, resulting in methods utilizing 3D vector fields
[Fu et al., ; Chai et al., , ; Zhang et al., ; Shen et al., ].
Furthermore,
hair can be modeled with multi-resolution [Kim and Neumann, ] or multi-scale geometric representations [Wang et al., ; Weng et al., ].
Similar to these approaches, we decompose hair models into multi-scale features and use a scalp texture to record strand features.
However, we aim at generating a variety of hairstyles, rather than variations of a single style.

##### Volumetric Hair Modeling

To reconstruct 3D hair from real-world observations, several pioneering methods have been proposed with specialized hardware devices [Nam et al., ; Shen et al., ; Hu et al., ; Jakob et al., ; Herrera et al., ; Xu et al., ; Paris et al., ; Grabli et al., ; Wei et al., ],
which are often expensive and unsuitable for wide application.
Alternatively, some methods use single or multiple captured images to retrieve hair models from a pre-stored database [Hu et al., , ; Zhang et al., ; Liang et al., ],
offloading the burden of hardware setup to the computational cost of database search and result refinement.
To mitigate the high cost of database retrieval, deep learning techniques have been proposed
to non-linearly parameterize the large variations of 3D hair datasets [Saito et al., ; Zhang and Zheng, ].
Existing approaches often rely on volumetric fields as an efficient spatial-aware intermediate representation to encode hair, spanning from sketch-based hair modeling [Shen et al., ], video-based dynamic hair modeling [Yang et al., ], to 3D hair reconstruction from either images [Wu et al., ; Zheng et al., ; Takimoto et al., ; Kuang et al., ] or monocular videos [Wu et al., ; Luo et al., ; Sklyarova et al., ].
However, entangled volumetric fields are not well-suited for representing curly and kinky hair.
To better capture hair, Wang et al. [; ] learn fine-grained details of hair implicitly encoded in multiple neural volumetric primitives, which are attached to pre-obtained sparse guide strands. They also developed a neural volumetric renderer to model dynamic hair [Wang et al., ].

##### Hair Geometry Textures

Organizing strand features within the scalp texture [Wang et al., ] provides a 2D representation of hair that can be learned using 2D convolution networks [Zhou et al., ; Rosu et al., ], commonly referred to as geometry textures [Sklyarova et al., ]. This texture-based representation also facilitates the application of diffusion models [Ho et al., ] for hair modeling. For instance, Neural Haircut [Sklyarova et al., ] leverages a pre-trained diffusion prior to generate hair geometry textures, which serve as cues for reconstructing high-fidelity hairstyles from smartphone video captures. Similarly, HAAR [Sklyarova et al., ] introduces a text-conditioned diffusion model for 3D strand-based hairstyle generation, utilizing geometry textures as input to encode latent hairstyle representations.
While these diffusion-based methods produce impressive results, they typically involve a heavy denoising process and lack a compact latent space for hair reconstruction and editing.
The work most related to our approach is GroomGen [Zhou et al., ], which constructed hierarchical hair latent spaces represented in the scalp texture space.
Unlike previous work, we do not use a single hair texture map to directly record strand latent features. Instead, we represent hair strands with multiple frequency components to generate hierarchical hair geometry textures from low- to high-frequency. As a result,
our work offers independent control over the global hair shape and local strand details.

##### Parametric Modeling

In human digitalization, Blanz et al. [] first proposed a 3D statistical morphable face model using PCA to parameterize the face modeling space. After that, subsequent works have leveraged PCA-based parametric modeling for 3D facial expressions [Cao et al., ] and whole body reconstruction [Allen et al., ].
SMPL [Loper et al., ] extended this PCA-based idea to model a whole range of body shapes and poses, with its follow-up work SMPL-X [Pavlakos et al., ] capturing the body, hand, and face together by integrating SMPL with the FLAME [Li et al., ] head model and the MANO [Romero et al., ] hand model. Similar ideas can be applied to model animals [Zuffi et al., ], infants [Hesse et al., ], and cartoon characters [Luo et al., ].
For hair modeling, guide strands are commonly used as a reduced parameterization. Chai et al. [] presented a reduced hair model by parameterizing the dynamic hair space with selected guide strands.
They extended this work with an adaptive skinning method for interactive hair-solid simulation [Chai et al., ].
Following a similar idea, Lyu et al. [] employ a neural interpolator to generate adaptive dynamic weights on-the-fly to interpolate guide strands.
However, due to the absence of a known principal basis for hair geometries that generalizes to all hairstyles, there is no parametric model for hair styling. To tackle this, we explore the frequency domain of hair strands and develop a hybrid hierarchical parametric hair representation, incorporating both linear and non-linear parameterizations across different hairstyles.

## 3. Model Formulation

Figure: Figure 4. With our PCA-based strand representation, we can smooth a hairstyle by truncating its strand PCA coefficient from $64$ (a) to $10$ (b). By blending the low-rank and high-rank PCA coefficients from different hairstyles, we can create diverse composite hairstyles (d and f). Please zoom-in for details.
Refer to caption: x4.png

The architecture of Perm is illustrated in Fig. 3, which imitates the current hair modeling pipeline in industrial software like XGen.
Starting from a PCA-based strand representation in the frequency domain (Sec. 3.1), we design a function, $\mathcal{S}(\vec{\gamma}):\mathbb{R}^{|\vec{\gamma}|}\mapsto\mathbb{R}^{L
\times 3}$, that maps the strand PCA coefficients $\vec{\gamma}$ back to 3D polylines with $L$ points (where $L$ is set to $100$ in our experiments). These PCA coefficients are then employed to fit 2D geometry textures to represent the 3D hair, which are further decomposed and parameterized with a guide strand function $\mathcal{G}(\vec{\theta})$, a super resolution function $\mathcal{F}(\cdot)$, and a hair styling function $\mathcal{D}(\vec{\beta})$ with controlling parameters $\vec{\theta}$ and $\vec{\beta}$ (Sec. 3.2). The resulting Perm model can thus be expressed as:

$$ (1) $\mathcal{M}(\vec{\theta},\vec{\beta},\mathcal{R};\Phi)=\mathcal{S}\Big{(}\text {Sample}\big{(}\mathcal{R};\mathcal{F}(\mathcal{G}(\vec{\theta}))\oplus \mathcal{D}(\vec{\beta})\big{)}\Big{)},$ $$

where $\text{Sample}(\cdot)$ denotes nearest neighbor interpolation used to sample the synthesized texture with 2D root coordinates in the pre-defined root set $\mathcal{R}$, $\oplus$ represents the concatenation operation along the feature channels, and $\Phi$ specifies the full set of trainable parameters of Perm. Once trained, parameters in $\Phi$ are held fixed, and novel 3D hairstyles are synthesized by varying $\vec{\theta}$ and $\vec{\beta}$ respectively, while the hair density can be adjusted by manipulating the number of roots in $\mathcal{R}$.
The dimension of parameters $\vec{\theta}$ and $\vec{\beta}$ are both set to $512$ in our model ($|\vec{\theta}|=|\vec{\beta}|=512$). In the following, we delve into the details of each term in our model.

### 3.1. Strand Parameterization

Inspired by the nearly periodic behavior of hair strands, we define a low-dimensional parameter space based on PCA in the frequency domain.
Formally, given hair strands represented with a fixed number of points $\mathbf{S}=\{\mathbf{p}_{1},\mathbf{p}_{2},\dots,\mathbf{p}_{L}\}\in\mathbb{R}
^{L\times 3}$,
we parameterize them with a linear function $\mathcal{S}$ in the frequency domain:

$$ (2) $\mathcal{S}(\vec{\gamma};\mathbf{X})=\text{iDFT}(\bar{\mathbf{S}}+\vec{\gamma} ^{\top}\mathbf{X})=\text{iDFT}(\bar{\mathbf{S}}+\sum_{n=1}^{|\vec{\gamma}|} \vec{\gamma}_{n}\mathbf{X}_{n}),$ $$

where $\bar{\mathbf{S}}=[\bar{\mathbf{S}}_{\text{real}},\bar{\mathbf{S}}_{\text{imag}
}]\in\mathbb{R}^{k\times 3\times 2}$ denotes the mean phase vector of strands, with $k=\lfloor L/2\rfloor+1$ representing the number of frequency bands. $\mathbf{X}=[\mathbf{X}_{1},\mathbf{X}_{2},\dots,\mathbf{X}_{|\vec{\gamma}|}]^{
\top}\in\mathbb{R}^{|\vec{\gamma}|\times 6k}$ forms a matrix of orthonormal principal components that capture phase variations in different strands, and $\text{iDFT}(\cdot)$ represents the inverse Discrete Fourier Transform that maps strands back from the frequency domain to the spatial domain. Therefore, a 3D strand $\mathbf{S}$ is represented as a vector of linear coefficients $\vec{\gamma}=[\gamma_{1},\dots,\gamma_{|\vec{\gamma}|}]^{\top}$. In other sections, we will omit $\mathbf{X}$ in Eq. 2 for brevity.

To obtain quantities in Eq. 2, we first apply the Discrete Fourier Transform (DFT) along the $x$, $y$, $z$ axes to all strands in our collected hair models, and then compute the mean phase vector $\bar{\mathbf{S}}$ and solve the principal components $\mathbf{X}$ by performing PCA on the computed Fourier bases, which is similar to the prevalent PCA-based modeling methods in digital humans [Blanz and Vetter, ; Loper et al., ; Li et al., ]. We set the number of PCA coefficients $|\vec{\gamma}|=64$, which explained almost $100\%$ of the variance in the training set.

Our strand representation differs from previous work [Rosu et al., ; Sklyarova et al., ; Zhou et al., ] that
uses a VAE to learn the strand latent space. Throughout our experiments, this simple yet effective formulation attained comparable or superior performance compared to those carefully designed VAEs. It also
significantly reduces training time and GPU memory consumption due to its analytical computation. Additionally, our frequency-based formulation offers an improved preservation of strand curvatures, which aligns with the findings in GroomGen [Zhou et al., ].

Our PCA-based representation also facilitates a meaningful decomposition of strand geometry by forming an *explainable* parameter space. Our analysis reveals that the first $10$ PCA coefficients are enough to effectively capture the rough shape of each strand, and the remaining $54$ coefficients encode high-frequency details, such as strand curliness. Leveraging this intuitive decomposition, we can effortlessly smooth a given hairstyle by truncating its strand PCA coefficients (Fig. 4a, b), or create diverse styles by blending the low-rank and high-rank portions of PCA coefficients from different hairstyles (Fig. 4c-f).

### 3.2. Hairstyle Parameterization

With our proposed strand parameterization, we define a 2D parameterization of hairstyles on the scalp surface as a regular $uv$ texture map, where each texel stores the corresponding strand PCA coefficients $\vec{\gamma}$. These textures are referred to as *hair geometry textures*, denoted as $\mathbf{T}\in\mathbb{R}^{256\times 256\times 64}$. Similar to GroomGen [Zhou et al., ], we separately model the baldness area as a baldness map $\mathbf{M}\in\mathbb{R}^{256\times 256}$. Both $\mathbf{T}$ and $\mathbf{M}$ are downsampled to $32\times 32$ to represent *guide strands* (denoted as the guide texture $\mathbf{T}_{\text{guide}}$ and guide mask $\mathbf{M}_{\text{guide}}$), where only the low-rank PCA coefficients ($|\vec{\gamma}_{\text{guide}}|=10$) are kept in $\mathbf{T}_{\text{guide}}$, akin to those guide strands created by real users in professional hair modeling software. As not all texels will be decoded to meaningful strands, $\mathbf{T}_{\text{guide}}$ typically accommodates around $400$ strands, as visualized in Fig. 1. More technical details are given in Supplemental A.1. Motivated by deep learning-based methods [Zhou et al., ; Saito et al., ], we further decompose and parameterize our hair geometry textures with separate neural networks to align with existing hair modeling stages, namely guide strand synthesis, guide interpolation, and hair styling.

#### 3.2.1. Guide Strand Synthesis

Formally, for guide strands, we train a neural network to approximate the function, $\mathcal{G}(\vec{\theta};\phi_{1})$: $\mathbb{R}^{|\vec{\theta}|}\mapsto\mathbb{R}^{32\times 32\times(10+1)}$, which synthesizes $\mathbf{T}_{\text{guide}}$ and $\mathbf{M}_{\text{guide}}$ from the guide strand parameter $\vec{\theta}$, where $\phi_{1}$ denotes the trainable network parameters.
In our formulation, we choose StyleGAN2 [Karras et al., ] as the backbone of our model, which has been proved to be a powerful generator in both 2D images and 3D feature tri-planes [Chan et al., ]. It also ensures our guide strand parameter to follow the normal distribution $\vec{\theta}\sim\mathcal{N}(\bm{0},\bm{1})$, making it easy to sample. Moreover, the intermediate $\mathcal{W}$ and $\mathcal{W}+$ spaces introduced in StyleGAN allow for more faithful latent embedding [Abdal et al., ], which in our case facilitates hair reconstruction either from 3D strands or 2D orientation images.
Since our generator synthesizes both the guide texture $\mathbf{T}_{\text{guide}}$ and guide mask $\mathbf{M}_{\text{guide}}$, we use a similar dual discrimination method as EG3D [Chan et al., ], where we concatenate these two images and feed them into the discriminator. This operation ensures consistency between the generated texture and mask, which helps the generator to place zero-length strands in bald areas.
Our training objective is identical to StyleGAN2, which consists of a non-saturating GAN loss [Goodfellow et al., ] with $R_{1}$ regularization [Mescheder et al., ] on both the texture and mask, where the regularization strengths are set to $5$ and $1$, respectively.

#### 3.2.2. Guide Interpolation

Since guide strands only serve as a sparse representation of the full hair, an upsampling process is necessary to obtain the final hair strands. However, as illustrated in GroomGen, this process entails a *one-to-many* mapping [Zhou et al., ], wherein the same guide strands can be upsampled to yield diverse hairstyles with varying strand randomness and curliness. Achieving this property proves challenging, necessitating an orthogonal decomposition of the global hair shape and local strand details. Previous methods thus either downgraded to a deterministic one-to-one mapping [Sklyarova et al., ] or resorted to a manual post-processing step [Zhou et al., ].
To address this challenge, we further decompose our hair geometry texture $\mathbf{T}$ into $\mathbf{T}_{\text{low}}\in\mathbb{R}^{256\times 256\times 10}$ and $\mathbf{T}_{\text{res}}\in\mathbb{R}^{256\times 256\times 54}$ by splitting along the feature channels. Given the explainability of our strand PCA coefficients, we discern that $\mathbf{T}_{\text{low}}$ encapsulates the global hair shape, while $\mathbf{T}_{\text{res}}$ embodies the local strand details, denoted as the *residual texture*. Leveraging this decomposition, we employ different neural networks to parameterize them, simulating the automated processes of guide interpolation and hair styling. In this section, we will introduce the guide interpolation process, which generates $\mathbf{T}_{\text{low}}$ from the texture $\mathbf{T}_{\text{guide}}$ and mask $\mathbf{M}_{\text{guide}}$, and the subsequent stage will be elaborated in Sec. 3.2.3.

With our texture-based representation for 3D hair, guide interpolation becomes an image super resolution problem, which can be defined as the function, $\mathcal{F}(\mathbf{T}_{\text{guide}},\mathbf{M}_{\text{guide}};\phi_{2}):
\mathbb{R}^{32\times 32\times 11}\mapsto\mathbb{R}^{256\times 256\times 10}$, with trainable network parameters $\phi_{2}$. Note that this function operates as a deterministic mapping without parameter control, which aligns with both previous deep learning-based methods [Zhou et al., ; Sklyarova et al., ] and the adaptive interpolation algorithm used in current hair modeling software.

To approximate $\mathcal{F}(\cdot)$, we train a U-Net [Ronneberger et al., ] on the bilinearly upsampled textures, which translates them to $14$-channel weight maps, where the first $4$ channels represent the weight vector $\vec{\omega}$ for the $4$ neighboring guide strands, and the last $10$ channels represent a residual vector $\vec{\delta}$ to correct the blended coefficients. Therefore, the final coefficient $\vec{\gamma}_{\text{low}}$ can be calculated as:

$$ (3) $\vec{\gamma}_{\text{low}}=\vec{\omega}^{\top}\bm{\gamma}_{\text{guide}}+\vec{ \delta}=\sum_{n=1}^{4}\omega_{n}\vec{\gamma}_{\text{guide}}^{n}+\vec{\delta}.$ $$

Our experiments demonstrate that this weight-based blending output allows the network to converge to a sharper result compared to predicting coefficients directly.

We train the U-Net in a supervised manner, where we employ both an $L_{1}$ loss for the blended texture $\hat{\mathbf{T}}_{\text{low}}$ and a geometric loss $\mathcal{L}_{\text{geo}}$ for the decoded strand geometry. Specifically, the geometric loss $\mathcal{L}_{\text{geo}}$ encompasses an $L_{1}$ loss for the point position $\hat{\mathbf{p}}_{n}$, a cosine distance for the orientation $\hat{\mathbf{d}}_{n}=\hat{\mathbf{p}}_{n+1}-\hat{\mathbf{p}}_{n}$[Rosu et al., ], and an $L_{1}$ loss for the curvature, represented as the $L_{2}$ norm of binormal vector $\hat{\mathbf{b}}_{n}=\|\hat{\mathbf{d}}_{n}\times\hat{\mathbf{d}}_{n+1}\|_{2}$[Sklyarova et al., ]:

$$ (4) $\mathcal{L}_{\text{geo}}=\sum_{n=1}^{L}\|\hat{\mathbf{p}}_{n}-\mathbf{p}_{n}\| _{1}+(1-\hat{\mathbf{d}}_{n}\cdot\mathbf{d}_{n})+\|\hat{\mathbf{b}}_{n}- \mathbf{b}_{n}\|_{1}.$ $$

A regularization term is included as well to constrain the residual vector $\vec{\delta}$ towards 0 0. The overall loss function, denoted as $\mathcal{L}_{\text{superres}}$, then can be expressed as:

$$ (5) $\mathcal{L}_{\text{superres}}=\lambda_{\text{tex}}\|\hat{\mathbf{T}}_{\text{ low}}-\mathbf{T}_{\text{low}}\|_{1}+\lambda_{\text{geo}}\mathcal{L}_{\text{geo }}+\lambda_{\text{reg}}\|\vec{\delta}\|_{2}^{2},$ $$

where the weighting factors $\lambda_{\text{tex}}$, $\lambda_{\text{geo}}$ and $\lambda_{\text{reg}}$ are set to $1$, $1$ and $0.1$, respectively.

#### 3.2.3. Hair Styling

To capture local details such as high-frequency curliness and randomness, artists employ various procedures to synthesize them from manually designed parameters. In our formulation, we approximate this artistic process by training a neural network. This network can be defined as the function, $\mathcal{D}(\vec{\beta};\phi_{3}):\mathbb{R}^{|\vec{\beta}|}\mapsto\mathbb{R}^
{256\times 256\times 54}$, where we learn to synthesize the residual texture $\mathbf{T}_{\text{res}}$ from the hair styling parameter $\vec{\beta}$, and $\phi_{3}$ represents the trainable network parameters.

Although StyleGAN2 performs well in generating textures for guide strands, we found it struggles with residual textures, as they contain more data than high-resolution RGB images. Therefore, we opted for VAE [Kingma and Welling, ] as the backbone, where the encoder adopts an architecture similar to pSp [Richardson et al., ], and the decoder takes the same architecture as StyleGAN2 generator. Essentially, the encoder $\mathcal{E}$ projects the residual texture $\mathbf{T}_{\text{res}}$ and baldness map $\mathbf{M}$ into the latent space, where the latent vectors will then be used to modulate the decoder to reconstruct the input. A similar loss function as Eq. 5 is defined as the training objective for the VAE, which can be expressed as:

$$ (6) $\mathcal{L}_{\text{res}}=\lambda_{\text{tex}}(\|\hat{\mathbf{T}}_{\text{res}}- \mathbf{T}_{\text{res}}\|_{1}+\|\hat{\mathbf{M}}-\mathbf{M}\|_{1})+\lambda_{ \text{geo}}\mathcal{L}_{\text{geo}}+\lambda_{\text{KL}}\mathcal{L}_{\text{KL}},$ $$

where the reconstruction terms are computed on the residual texture, baldness map, and decoded strand geometry. The weighting factors $\lambda_{\text{tex}}$, $\lambda_{\text{geo}}$, and $\lambda_{\text{KL}}$ are set to $10$, $1$, and $1e^{-4}$, respectively.

After synthesizing residual textures, they can be concatenated with the upsampled guide textures, thereby composing the final hair geometry textures. These textures can then be sampled and decoded to produce the final strand geometry.

## 4. Experiments and Results

We train Perm on USC-HairSalon [Hu et al., ], a dataset comprising $343$ 3D hair models collected from online game communities. After augmentation, our training set contains a total of $21,054$ data samples. For evaluation, we compiled a dataset consisting of $17$ publicly available and $100$ artist-groomed hair models. Details are provided in Supplemental B.1.

### 4.1. Evaluation and Comparison

#### 4.1.1. Strand Representation

We evaluate our PCA-based strand representation by comparing it to alternative deep learning-based methods [Zhou et al., ; Rosu et al., ; Sklyarova et al., ] and a simpler PCA-based formulation without DFT. To quantitatively assess their reconstruction capabilities, we present the mean position error (pos. err.), computed as the average Euclidean distance between corresponding points on the reconstructed strands and the ground truth. Additionally, we report the mean curvature error (cur. err.), defined as the $L_{1}$ norm between the curvatures of reconstructed and ground truth strands. Please refer to Supplemental B.3 for details of the metrics.

**Table 1. Reconstruction errors reported on different strand representations.**
|  | Pos. VAE | Freq. VAE | Pos. PCA | Freq. PCA |
| --- | --- | --- | --- | --- |
| # params. | $15.67$M | $16.51$M | $19.31$K | $19.50$K |
| pos. err. | $0.075$ | $0.726$ | $\bm{0.004}$ | $\bm{0.004}$ |
| cur. err. | $0.860$ | $\bm{0.304}$ | $1.229$ | $0.569$ |

Quantitative results are reported in Table 1, where all configurations compress strands to $64$-dimensional vectors. Our PCA-based representations demonstrate a remarkably small position error, achieving this with a significantly reduced number of parameters compared to strand VAEs trained either in the spatial [Rosu et al., ; Sklyarova et al., ] or frequency domain [Zhou et al., ]. While performing PCA on strand positions results in a relatively high curvature error, performing PCA in the frequency domain effectively mitigates this issue, demonstrating superior preservation of strand curvatures. Note that, although the strand VAE trained in the frequency domain attains the lowest curvature error, it does not preserve the overall shape, thus leading to a high position error (see Fig. 5).

Figure: Figure 5. Comparison of our PCA-based strand representation (Freq. PCA) with other VAE-based representations (Pos. VAE [Rosu et al., ; Sklyarova et al., ], Freq. VAE [Zhou et al., ]) and a simpler PCA-based formulation in the spatial domain (Pos. PCA). Reconstructed strands are color-coded by their position error. More results are available in Fig. 3 of our supplemental.
Refer to caption: x5.png

#### 4.1.2. Guide Strand Synthesis

**Table 2. Reconstruction errors reported on different model representations.**
|  | Guide Strands | Full Hair |  |  |
| --- | --- | --- | --- | --- |
| PCA | StyleGAN2 (Ours) | GroomGen | Ours |  |
| pos. err. | $3.734$ | $\bm{0.491}$ | $2.584$ | $\bm{0.805}$ |
| cur. err. | $4.212$ | $\bm{0.281}$ | $3.442$ | $\bm{0.179}$ |

To assess the performance of StyleGAN2 in guide strand synthesis, we create a PCA-based representation for guide textures with a similar formulation as our strand parameterization, and set the subspace dimension to $512$ to align with Perm’s setup. We then embed guide textures in the testing set into the PCA subspace or the latent space learned by StyleGAN2, followed by a decoding step to reconstruct them.
Strands are further decoded from the reconstructed textures to evaluate quantitative errors, which are reported in Table 2 ($1$st and $2$nd columns). It is evident that results from StyleGAN2 exhibit both smaller position and curvature errors, indicating that StyleGAN2 learns a more expressive latent space than PCA. Fig. 6 displays examples of reconstructed guide strands, illustrating that StyleGAN2 more faithfully preserves the structure of the ground truth. Reasons for the failure of PCA are straightforward: USC-HairSalon contains a vast number of strands, specifically $3,150,559$, for us to learn the PCA subspace, with a compression rate of approximately $78.7\%$. However, for guide textures, we can only create $21,054$ samples after data augmentation, which is merely around $0.6\%$ of the total number of available strands. Moreover, the compression rate increases to $95\%$.
These two factors lead to the poor performance of PCA in this case.

Figure: Figure 6. Comparison of PCA and StyleGAN2 in reconstructing given guide strands.
Refer to caption: x11.png

#### 4.1.3. Guide Interpolation

In Fig. 7 we compare the strands decoded from textures upsampled with different interpolation methods. Nearest neighbor interpolation ($2$nd column) produces aliased strands as it involves simple repetition. Bilinear interpolation ($3$rd column) leads to smoother strands, but introduces unwanted flyaway fibers, particularly in the forehead area. Our method ($5$th column) achieves the most natural result, which resembles the shape of guide strands and forms reasonable hair partitions. In addition, we employ the same U-Net architecture but configure it to directly predict strand PCA coefficients rather than the blending weights. The output ($4$th column) contains more flyaway strands and a reduced fidelity in capturing the overall hair shape, particularly noticeable in the lower part of the hair.

Figure: Figure 7. Comparison of strands upsampled with different interpolation methods.
Refer to caption: extracted/5780499/fig/img/guide_interpolation/Blowout_guide.png

#### 4.1.4. Hair Styling

For the hair styling module, though StyleGAN2 performed well in synthesizing guide strands, we found it collapsed to produce blurry residual textures when embedding them into the latent space, as shown in Fig. 8 (left). We conjecture that this issue may be attributed to the large size of residual textures ($256\times 256\times 54$), as they contain more data than high-resolution RGB images ($1024\times 1024\times 3$). Moreover, our dataset is smaller than
those image datasets used for StyleGAN training (e.g., FFHQ [Karras et al., ], which contains $70$K high-quality portrait images), thus further suppressing the expressiveness of StyleGAN2. Considering these constraints, we devised our network architecture as a VAE, trading some sampling diversity for higher reconstruction fidelity, which reconstructs sharper residual textures (Fig. 8 middle).

Figure: Figure 8. Reconstruction quality of embedding residual textures into the latent space of different network architectures.
Refer to caption: extracted/5780499/fig/img/restex/wCurly_stylegan.png

#### 4.1.5. Comparison with GroomGen

The most relevant previous work is GroomGen [Zhou et al., ] that learns hierarchical representations of 3D hair. Since there is no publicly available GroomGen code, we implemented and trained it on the same augmented USC-HairSalon dataset. Unlike image synthesis tasks, there are no standard metrics like FID to evaluate the quality of randomly synthesized 3D hair. Therefore, we focus our quantitative comparison on reconstruction tasks, specifically examining the reconstruction capabilities of different strand representations and model representations. In Table 1 ($2$nd and $4$th columns), we present quantitative comparison between our PCA-based strand representation and GroomGen’s VAE-based strand representation, indicating that our representation achieves a significantly better reconstruction with slightly higher curvature errors. To compare model representations, we embed hair geometry textures into the latent spaces, and report quantitative reconstruction errors in Table 2 ($3$rd and $4$th columns), where our method achieves both smaller position and curvature errors.
Note that we do not evaluate reconstruction errors on guide strands, as our guide strand definition is different (we model *smoothed* guide strands without high-frequency curliness).
For qualitative comparison, we randomly synthesize $10$ hairstyles by sampling our parameter space and GroomGen’s latent space with the same Gaussian noise, and visualize the results in Fig. 5 of our supplemental. In this comparison, most of our synthesized hairstyles look more natural compared to GroomGen’s results. More details of random hairstyle synthesis can be found in Supplemental C.2.

### 4.2. Applications

After training Perm, we can deploy it as a generic hair prior to solve different down-stream tasks.

#### 4.2.1. 3D Hair Parameterization

The first application we explored is to fit Perm parameters to target 3D hair models through differentiable optimization, which we term as *3D hair parameterization*. Technical details are provided in Supplemental C.3.
In Fig. 9 we show a subset of fitted results.
Even if the target hair model has no guide strands, our model can generate reasonable guide strands to depict their overall shapes, which can be obtained from $\mathcal{S}\big{(}\mathcal{G}(\vec{\theta}^{\ast})\big{)}$.

Figure: Figure 9. A subset of 3D hair models fitted by Perm. More results are available in Fig. 6 of our supplemental.
Refer to caption: extracted/5780499/fig/img/perm_fitting/out-1.jpg

#### 4.2.2. Hairstyle Interpolation

As we introduce separate parameters to control the global hair shape and local strand details, we can edit the hairstyle with varying granularity by interpolating the corresponding parameters.
In the top row of Fig. 10, we linearly interpolate the guide strand parameter $\vec{\theta}$, resulting in hairstyles with different global shapes. Since the hair styling parameter $\vec{\beta}$ is kept fixed, local strand details, such as curls at the tips of strands, remain consistent.
In the middle row, we fix the parameter $\vec{\theta}$ and linearly interpolate the parameter $\vec{\beta}$, thereby generating novel hairstyles ranging from straight to curly while maintaining a nearly identical global shape.
In the bottom row, we jointly interpolate $\vec{\theta}$ and $\vec{\beta}$, demonstrating the transition from a short wavy hairstyle to a long straight hairstyle.
The comparison with [Weng et al., ] and [Zhou et al., ] are provided in Supplemental C.4.

Figure: Figure 10. Hairstyle interpolation with different granularity. Top: interpolate $\vec{\theta}$ only and keep $\vec{\beta}$ fixed; Middle: interpolate $\vec{\beta}$ only and keep $\vec{\theta}$ fixed; Bottom: jointly interpolate $\vec{\theta}$ and $\vec{\beta}$.
Refer to caption: extracted/5780499/fig/img/hair_morphing/out-1.jpg

#### 4.2.3. Single-view Hair Reconstruction

With 2D observations as input, we can reconstruct 3D hair by optimizing Perm parameters to minimize the energy of generated hair when projected to the 2D images. Technical details are provided in Supplemental C.5.
In Fig. 11 and Fig. 9 of our supplemental, we qualitatively compare our single-view reconstruction results with HairStep [Zheng et al., ], the state-of-the-art open-source method for single-view hair reconstruction.
Our method not only excels in reconstructing the global shape of the reference hairstyle but also preserves local details better compared to HairStep.
We also observed that HairStep occasionally generates unintended bald areas, while ours do not.

Figure: Figure 11. Qualitative comparison on single-view hair reconstruction with HairStep [Zheng et al., ]. More comparisons are provided in Fig. 9 and Fig. 10 of our supplemental.
Refer to caption: extracted/5780499/fig/img/single_view/bbj-000015-rgb.png

#### 4.2.4. Hair-conditioned Image Generation

Latest text-to-image (T2I) models (e.g., Adobe Firefly [Adobe, ]) can generate high-quality portrait images. However, the text embedding of simple prompts like “wavy and short hair” is not precise enough to represent a specific hairstyle. To tackle this issue, we show the use of Perm in facilitating conditional image generation. Specifically, We could use Perm to sample and edit hairstyles in 3D, or reconstruct 3D hairstyles from images, and then feed the depth and edge information extracted from the hair geometry to the T2I models to generate the final images. As shown in Fig. 11 of our supplemental, pure text prompts often lead to images with different hairstyles, while combined with the input hair reference, the generated images show a much more consistent hairstyle. More examples are available in Fig. 12 of our supplemental.

## 5. Limitations and Future Work

First, while our model adeptly fits and samples a diverse range of hairstyles, it currently faces challenges with intricate styles like buns and braids, as these complex hairstyles do not exist in the training data. Addressing this issue necessitates a systematic framework to efficiently capture a diverse set of real data.
To this end, Perm can serve as a pre-trained prior for efficient data capture, as it can fill invisible parts of the hair, such as the interior and occluded regions, and accommodate more sparse image inputs. The captured hairs could also be used to fine-tune Perm to narrow the modal gap from real hairs.
Moreover, aligning with trends in other 2D or 3D generative tasks, controllable 3D hair synthesis with multi-modal input signals (e.g., semantic attributes, natural languages, face parsing images, depth or normal maps, etc.) would be another promising direction to explore in our scenario.

## Acknowledgments

The authors would like to thank Yujian Zheng for providing the source code and model of HairStep and for discussing the HairStep results in the paper with us. We also extend our gratitude to Yao Feng for assisting us in setting up DELTA for face and shoulder reconstruction.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: Is  S = {p1 , p2 , . . . , pL }   the output of Equation 2? The way it is drafted in line 212 makes it seem like it should be an input to the subsequent equation.

Line 230 states that “Our analysis reveals that the first 10 PCA coefficients are enough to effectively capture the global structure of each strand, and the remaining 54 coefficients encode high-frequency details, such as curl patterns”. Where is this analysis? This is another place where I am frustrated by no-referencing.
soundness: 3
strengths: The strengths of the paper include its use of a PCA-based strand representation for 3D hair that efficiently disentangles global and local hair details, enabling precise control and intuitive editing. The PERM model’s architecture is computationally lightweight and achieves high fidelity with reduced memory and training requirements. It showcases versatility by performing well across various applications like single-view reconstruction and hairstyle editing, often matching or surpassing specialized, task-specific methods. Additionally, the framework’s modularity makes it adaptable for different hair modeling tasks, enhancing its practical value in digital human applications.
summary: The paper presents PERM, a parametric model for representing 3D hair, designed to efficiently handle hair synthesis, editing, and reconstruction. PERM disentangles global hair structure and local curl patterns using a PCA-based strand representation in the frequency domain, enabling intuitive editing and precise control. The model architecture combines StyleGAN2 and VAE networks to generate guide and residual textures, capturing both broad and detailed features of hair. PERM performs well across tasks like single-view hair reconstruction and hairstyle editing, often surpassing task-specific methods. The approach is lightweight, computationally efficient, and achieves high fidelity, providing a robust framework for diverse hair modeling applications.
weaknesses: While the motivation of this paper is clear and the introduction is enjoyable to read, I am extremely frustrated with the forward-referencing (or no-referencing) writing style in Section 3. I often have to hold many questions until much later in the text, making it so easy to lose track of the mathematical flow. For example, Figure 4 starts with StyleGAN, but it is not properly introduced until Section 3.2. The same issue occurs with guide textures, residual textures, and many other components. The strand parameterization function S() , which is the last step in the diagram of Figure 4, is introduced first in the subsections, while preceding steps are explained in a recursive manner. The PCA coefficients, \gamma, are not explicitly present on the RHS of Eq 1. I understand that they are outputs of the guide textures, but I was so clueless when first looking at it until very late in Section 3. Additionally, the function  M()  on the LHS of Equation 1 was never properly introduced. I am concerned that this paper requires a major revision.

The paper presents the method as a parametric model, but unlike deterministic models such as SMPL or FLAME, it lacks a clear deterministic nature as it involves stochastic elements (e.g., generative processes or randomness in texture synthesis). This ambiguity raises questions about the consistency and reproducibility of the model’s outputs. You won’t be able to encode hair geometry into a fixed set of parameters and then decode it back accurately.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 4
questions: The two questions are listed above.
soundness: 3
strengths: - The frequency domain PCA for hair strands is neat. Also, 10 coeffs for coarse strands and 54 coeffs for detailed curls is reasonable and straightforward.

- The proposed framework is meticulously designed. The model can be applied to differentiable optimization/rendering. I did not find an obvious fallback of the proposed framework.

- The evaluation of the paper is comprehensive, including comparisons with other methods, ablation studies on different key components, and exemplar downstream tasks. From my perspective, the authors successfully demonstrate the effectiveness/SOTA performance of the proposed representation.
summary: The paper proposed perm, a learning-based parametric representation for hair modeling. A PCA-based strand representation in the frequency domain is proposed to jointly model the global hair structure and local curl patterns, enabling hair editing and other downstream tasks. The authors have conductive comprehensive experiments, including comparisons to other representations, ablation studies, and demonstrations on downstream tasks, showing the great potential of the proposed hair parameterization model.
weaknesses: I am not an expert in hair modeling and I did not find obvious fallbacks of the proposed framework. Below I listed the questions raised after reading the manuscript:

- From single view reconstruction results in Fig. 10, I find that the current SOTA is a good fit for the hair in the input image. However, the hair details are still misaligned. Is this restricted by model capacity given the current parameterization? (I understand hair modeling is very difficult). Are there any suggestions to improve the parameterization (e.g., modeling the random strands in the input image)?

- Is it possible (or how) to combine the proposed method with NeRF/3DGS for realistic full-head/full-body reconstruction?

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: As explained in the "weaknesses", my primary concerns lie in the experiments section.      I hope that the authors can provide detailed responses and explanations to the following specific issues:

* In the comparison with GroomGen, it is evident that all random sampling results generated by GroomGen exhibit markedly unrealistic strand structures.      How can this phenomenon be explained?

* In Figure 5, the rendering results of the PCA method and the authors' proposed Freq. PCA (Ours) methods are similar, to the extent that they are almost indistinguishable by visual inspection.      Could the authors further elaborate on the differences and merits of these two methods in terms of specific rendering details?

* What is the extent of the impact of Guide Texture Upsampling on the final rendering results?  Incorporating an ablation study on the upsampling module is crucial for comprehensively evaluating its significance and contribution.

* I suggest that the authors showcase failure cases encountered in random generated and single-view reconstruction tasks, and provide a thorough analysis and discussion of such cases.      This not only helps to uncover potential limitations of the current method but also offers valuable insights and references for future research endeavors.
soundness: 3
strengths: The paper focuses on the hot topic of 3D hair parametric modeling and innovatively proposes a strand representation method based on Principal Component Analysis (PCA) in the frequency domain.  The paper is clearly articulated, with a rigorous technical approach and strong innovation.  The rendering results presented in the experimental stage effectively demonstrate the effectiveness of the method.

The authors have conducted extensive and in-depth experimental validations.  At the strand representation level, they have fully demonstrated the superiority of the PCA-based strand representation method compared to other representation approaches through comparative experiments.  In terms of full hair representation, the authors have compared the effectiveness with the Groom-Gen method.  Furthermore, the authors have showcased the broad application effects of the proposed parametric model in multiple tasks, including single-view hair reconstruction, perm parameter editing, and hair-conditioned image generation, fully proving the method's promising application prospects.
summary: The paper proposes PERM, a lightweight and generic parametric representation of 3D human hair.  The proposed PERM method contributes to the field of 3D hair modeling by providing a lightweight, generic, and editable parametric representation that addresses the limitations of existing methods and enables versatile applications in various hair-related tasks.

Specifically, a novel parametric representation of 3D human hair, PERM, is introduced.   To independently control the global structure and curl pattern, a strand representation based on principal component analysis (PCA) in the frequency domain is proposed.  The learned principal components form an interpretable space for hair decomposition, where initial components capture low-frequency information, and subsequent components encode high-frequency details.  The paper follows previous literature to compute the UV of the scalp and map the geometry features of each strand onto a 2D texture based on its root position.  With the strand PCA representation, dominant coefficients fit and are stored in guide textures, and higher-order coefficients are stored in residual textures for detailed strand patterns.
weaknesses: The paper only compared Groom-Gen and presented a limited number of samples, making it difficult to analyze the strengths and weaknesses of the two approaches comprehensively. To more accurately evaluate the methods, it is suggested that the number of samples in the comparison experiments be increased.

Furthermore, the paper did not discuss any failure cases of the proposed method. I am particularly interested in the failure cases of the method in random synthesis and single-view reconstruction tasks. Presenting these failure cases would help to elucidate the limitations of the proposed approach more clearly.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Regarding the overview pipeline, is the residual branch necessary, since the StyleGAN model with U-Net SR module can already generate pretty fine-grained results (intuitively). If this branch is removable, I believe the proposed pipeline will be more neat.
soundness: 3
strengths: 1. The proposed pipeline, as illustrated in Fig. 4, looks reasonable and straightforward.
2. The paper is well polished, and the experiments are sound.
summary: The paper introduces **PERM**, a parametric representation for 3D human hair designed to improve a variety of hair-related applications. Unlike previous models that combine global hair structure and local curl patterns, PERM separates these elements using a **PCA-based strand representation** in the frequency domain. This allows for more precise control and editing by decomposing hair textures into different frequency levels, from low to high. Each frequency level is then parameterized with distinct generative models to mimic typical hair grooming stages. Experiments validate PERM's architecture, and the trained model is applied as a flexible prior for various tasks, including single-view hair reconstruction, hairstyle editing, and hair-conditioned image generation.
weaknesses: N/A since I am not an expert in this area. A potential improvement direction, from my perspective, is to demonstrate the proposed representation on 3D generation tasks / other amortized pipelines, such as X-conditioned 3D avatar (with controllable hair) generation. E.g., as in Gaussian3Diff (ECCV 24'), the UV space can be used for 3D diffusion training for full head synthesis, and I think that pipeline is applicable to this proposed method.

Meta-review: The paper proposes a parametric representation for 3D hair using a PCA-based strand representation in the frequency domain. This approach enables precise control and editing by decomposing hair textures into different frequency levels. The proposed representation facilitates various applications, including hair synthesis, editing, and reconstruction. 

The main strength of the paper lies in its novelty. While the reviewers raised several issues requiring improvement or clarification, there is no significant weakness. These concerns were effectively addressed in the rebuttal. 

In summary, the paper makes a significant contribution by proposing a novel representation of 3D hair and demonstrating its effectiveness through extensive experiments.

Decision: Accept (Spotlight)