HairDiffusion: Vivid Multi-Colored Hair Editing via Latent Diffusion
Authors: Yu Zeng
Yang Zhang
Jiachen Liu
Linlin Shen
Kaijun Deng
Weizhao He
Jinbao Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Image Editing
Diffusion Model
Image Inpainting
Hair Editing
Primary area: diffusion_based_models

Abstract: Hair editing is a critical image synthesis task that aims to edit hair color and hairstyle using text descriptions or reference images, while preserving irrelevant attributes (e.g., identity, background, cloth). Many existing methods are based on StyleGAN to address this task. However, due to the limited spatial distribution of StyleGAN, it struggles with multiple hair color editing and facial preservation. Considering the advancements in diffusion models, we utilize Latent Diffusion Models (LDMs) for hairstyle editing. Our approach introduces Multi-stage Hairstyle Blend (MHB), effectively separating control of hair color and hairstyle in diffusion latent space. Additionally, we train a warping module to align the hair color with the target region. To further enhance multi-color hairstyle editing, we fine-tuned a CLIP model using a multi-color hairstyle dataset. Our method not only tackles the complexity of multi-color hairstyles but also addresses the challenge of preserving original colors during diffusion editing. Extensive experiments showcase the superiority of our method in editing multi-color hairstyles while preserving facial attributes given textual descriptions and reference images.

HairDiffusion: Vivid Multi-Colored Hair Editing via Latent Diffusion
Yu Zeng's Yang Zhang's Jiachen Liu', Linlin Shen 1:2.3 Kaijun Deng'. Weizhao Jinbao Wang
'Computer Vision Institute, School of Computer Science & Software Engineering, Shenzhen University Shenzhen Institute of Artificial Intelligence and Robotics for Society
Nattional Engineering Laboratory for Big Data System Computing Technology, Shenzhen University
*Guangdong Provincial Key Laboratory of Intelligent Information Processing
Lcengyu,luujiachen, jungkkijjun>ee 20230emall szu en, lewei2hha20228mmall szu. www..du.uucco....o cn
<yangzhang, 11shen, wangjnnaanna edu.
Irpul imneae
red
kabrrle
Tule Indentioo
"EROC wiiTE dE Orea PLLER THOBEAA
10. 1. noy hea
Figure 1: Our framework supports individual or collaborative editing of hairstyle and color, utilizing text, reference images, and stroke maps. With exceptional performance, particularly evident editing multiple hair colors.
Abstract
Hair editing is critical image synthesis task that aims edit hair color and hairstyle using text descriptions or reference images, while preserving irrelevant attributes (e.g., identity, background, cloth). Many existing methods are based on StyleGAN to address this task. However, due to the limited spatial distribution of StyleGAN, it struggles with multiple hair color editing and facial preservation. Con- sidering the advancements diffusion models, we utilize Latent Diffusion Models (LDMs) for hairstyle editing. Our approach introduces Multi-stage Hairstyle Blend (MHB), effectively separating control of hair color and hairstyle in diffusion latent space. Additionally, we train a warping module align the hair color with the target region. To further enhance multi-color hairstyle editing, we fine-tuned a CLIP model using a multi-color hairstyle dataset. Our method not only tackles the complexity of multi-color hairstyles but also addresses the challenge of preserv- ing original colors during diffusion editing. Extensive experiments showcase the superiority of our method in editing multi-color hairstyles while preserving facial attributes given textual descriptions and reference images.
1 Introduction
Hair editing is one of the most challenging and interesting facets of semantic face editing. The essence of this task is edit hair attributes such color, and hairstyle from the reference image or
"Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

text to the input image while preserving irrelevant attributes. Disentangling these attributes is the key to a quality solution to the problem. This task has many applications among professionals and amateurs during work with face editing programs, virtual reality, and computer games 20 43 37
In recent years, numerous methods leveraging Generative Adversarial Networks (GANs)
3 35
|4 36 17] have emerged in this field. These methods often involve mapping images into StyleGAN [6] latent space to disentangle hair features or utilizing Contrastive Language-Image
Pre-Training (CLIP) model 127] to translate textual descriptions into relevant semantic vectors, facilitating manipulation via text or reference images.
However, previous methods have overlooked the hair color structure. As shown in Figure I " "ombre hair" (the last column of the second row) means hair color structure with gradient transition from top-to-bottomy while "split color hair" (the last column of the first row) exhibits a hair color
structure with a left-to-right transition. In traditional StyleGAN-based approaches [45] 35 36 17],
the scarcity of multi-colo hair in the facial datasets 15 16 used to train StyleGAN [66] coupled
with the limited distribution of latent space in StyleGAN (16]. gives rise to two primary challenges: 1) difficulty in generating intricate hair color and hairstyle due to the insufficient diversity in the training data's 's multi-color hair distributions: 2) challenges in preserving the original facial information when editing the latent code after mapping images into latent space, leading to difficulties in editing images while preserving irrelevant attributes.
In recent years, with the advancement of diffusion models 39 generative capabilities have surpassed those of GANs in many aspects 6 In particular, Latent Diffu- sion Models (LDMs)) 12 40 47, 28) have demonstrated exceptional generative capabilities, notably in image inpainting tasks. However, the application of diffusion models to hair editing encounters three challenges: 1) lack of tailored masks for hairstyle inpainting, necessitating consideration of hairstyle regions while preserving irrelevant attributes: 2) difficulty in providing sufficient control for the hair editing task, which requires faithful transfer of hair color from another image or retaining the original hair color of the image; 3) limitations in text and semantic understanding related to hair color and hairstyle. hinder the precision of CLIP- -ulP-guided diffusion processes.
their robust and stable
To address these limitations, propose novel bascline approach based LDMs, enabling the automatic generation of edited regions while allowing separate control of hair color and hairstyle via text and reference images individually collaboratively. Our pipeline can be roughly divided into two stages. The first stage begins with obtaining hair-agnostic masks to acquire facial representations combined with facial keypoint information and
independent of hair. These representations processed through textual descriptions to perform the LDMs process, generating a style proxy (i.c., an image used to guide the hairstyle transfer process). The second stage involves transferring hair color using a reference image. Initially, the hair of the reference image is aligned with the source image using a pre- pre-trained warping module. The aligned hair serves as a color proxy (i.e., an image used to guide the color and color structure transfer process), which is then input into the LDMs process to transfer or retain the hair color. Simultancously, the Canny edge from the original hair or style proxy is employed to retain or edit the hairstyle. The key to the pipeline is to use different hair-agnostic masks for both the color proxy and style proxy at different stages and blend them in the diffusion latent space by Multi-stage Hairstyle Blend (MHB). By training the warping model to obtain aligned hair color prior information in targeted hair regions, our method effectively addresses multi-hair color editing tasks. Quantitative and qualitative evaluations, along with user studies, demonstrate the efficacy of our approach in hair editing tasks, particularly in color transformation and preservation of unrelated attributes within the diffusion model framework. Furthermore, by designing different stages of agnostic masks, we can maximize the retention of hairstyle- -eideeenndd features such as background information and facial features. In summary, our contributions outlined as follows:
We present warping module designed for hair warping, allowing the alignment of the target hair mask with precision and enabling comprehensive hair color structure editing through reference images.
The MHB method is proposed within LDMs, which enables the decoupling of hair color and hairstyle, thereby effectively achieving high-quality hair color and hairstyle editing.
Through extensive qualitative and quantitative evaluations, we showcase the superior per- formance of our method in text-based hairstyle editing, reference image- -aaee hair color editing, and preservation of facial attributes.
2

---

The application of LDMs to address the challenge of text and image-based hair editing is pioneered through the introduction of hair-agnostic facial representation masks, reframing hair editing as an inpainting task and representing novel approach. To the best of our knowledge, this method has not been previously explored in this domain.
2 Related Work
Diffusion Models. Diffusion models have established themselves as robust class of generative models capable of producing high-quality images through a process of iteratively denoising data [
12 [1 28]. Specifically, SDDInpainting (29) is based on the large-scale pre-trained text-to-image model, i.c., Stable Diffusion (28). By utilizing random masks as repair masks and employing image- text prompts, diffusion models can fill areas with content consistent with textual descriptions while retaining context awareness in other regions. Despite the success of diffusion models in the inpainting task, their potential has not been fully exploited yet. For instance, describing fine details like hair color directly through text remains challenging. Some methods combine different interaction strategies to achieve satisfactory inpainting results, such as CentrolNee-Inpaaniie to guide image restoration based on additional inputs like Canny edge, depth maps, etc. Additionally, some methods provide an additional reference image or involve drawing rough color strokes. In the hair editing task, it is essential to develop framework that facilitates multimodal conditions for image inpainting. Therefore, we leverage ControlNet to control hairstyle attributes and stroke maps, managing color information and color structure.
42 which utilizes ControlNet
Hair Editing. As essential component of the face, there have been numerous works on hairstyle editing and synthesis [33] 45
[9 35 36). Some works decouple facial attributes by using image- level masks and then splice hairstyle images onto facial images through generative network [33 30] However, this approach often results in inconsistencies in lighting or artificial shadows along the edges of the hairstyle region. Some approaches utilize the e4e [4] to map the target image and hairstyle reference image into W++ latent space of StyleGAN [16), enabling manipulation of hairstyle and hair color via vector operations, followed by image synthesis using StyleGAN
45
1T. HairCLIP [35 integrates CLIP 27 into hair-editing tasks by leveraging StyleGAN to decouple hair color and hairstyle features, enabling text-driven. Additionally, some methods attempt to enhance the generative capabilities of StyleGAN through various transformations of its latent space 46 45
35 36
26
Despite these advancements, challenges persist due to the limited data distribution inherent in StyleGAN [16), which is trained predominantly facial datasets [16] 23 This constraint leads to difficulties in preserving specialized facial features like earrings and eyeshadow, as well as in introducing novel attributes such as multi-colored hairstyles. To tackle the constraints of hair color diversity, this paper introduces stroke maps (24] into the denoising process of diffusion models as the color proxy prior. Utilizing CLIP text encoder and image encoder fine-tuned with images matching the hair color to achieve editing of hair color structure through text. For reference images, we finetune the model with textual inversion.
Image Deformation. Image deformation, or image warping, is a technique computer vision and graphics for manipulating and transforming images 7 32 22 The goal is to deform an input image while preserving its essential content and structure. In recent years, deep learning techniques have revolutionized the field of image deformation, particularly GANs [8] have revolutionized the field of image deformation 77 32 9]. GANs are employed to learn complex deformation mappings directly from data, capturing intricate image transformations and generating realistic deformations with minimal artifacts. Conditional GANs (cGANs) 25 represent an evolution image deformation by incorporating additional contextual information during deformation. These networks learn to map input images to desired target outputs based on specific conditions or auxiliary information [32]. For example, integrating DensePose 10), detailing pixel-level semantic information enables precise and controlled image warping. In this paper, we employ multiple conditioning inputs to train a warping network capable of aligning hair color while accounting for the relationship between hair and facial
features.
3 Method
we propose novel solution based on LDMs for the first time in this field. In particular, our work employs the Stable Diffusion architecture starting point for performing
As shown in Figure 2
3

---

(a) Hairstyle Editing
(b) Hair Color Editing
"aa at
laut
v" hairstyle"
*black white dia dyyed" 1
Tokenizet
7x
y1
"Suter halrstyle"
L-net
Za
(c) Hair Warping Module
(d)Multi
Stage Hairstyle Blend
Warp
Sampling
BiSeNET
D.
2 P1-2
Tor
Uncond
Figure 2: Overview of HairDiffusion: (a) Using a hairstyle description T or reference image 1; as conditional input, coupled with the hair-agnostic mask M, and source image Lares style proxy ps. (b) Leveraging the color proxy and style proxy, along with the hair-agnostic mask Mc and source image Ire. enables individual or collaborative editing of hair color and hairstyle. (c) Given series conditions driven from the input image Ic, the hair color reference image I is used to obtain the color proxy pa through warping module. In the of changing only the hairstyle while preserving the original hair color, 1 Jare- (d) The color proxy P and the style proxy ps are blended different stages of the diffusion process.
get the
the hairstyle editing task. We obtain the style proxy, denoted as Ps, to capture the hairstyle prior as shown in Figure (a). Concurrently, we acquire the color proxy pe, using a warping module as
shown in Figure 2 (c). Finally, we incorporate these two proxies into the diffusion process through Multi-stage Hairstyle Blend (MHB) to blend them effectively as shown in Figure 2 (d). Therefore, our framework supports individual or collaborative editing of hairstyle and single-color or multi-color, utilizing text, and reference images.
3.1 Preliminaries
Stable Diffusion. consists of an autoencoder encoder and an autoencoder decoder D, a text- conditional U-Net denoising model cg. CLIP text encoder Tg, which takes text T input. The encoder compresses an image / into the latent space of diffusion, while the decoder D performs the inverse operation and decodes latent variable into the pixel space. For clarity, convolutional input as the spatial input y since convolutions preserve the spatial structure, and the attention conditioning input as v. The training of the denoising network eæ is performed by minimizing the following loss function:
refer the
L RE.D [Ie ca (7, v)1
(1)
where t represents the diffusing time step, y 24, is the encoded image E(I) where stochas- tically add Gaussian noise ~ (O,, 1), and 1/ = [1; [T)). Our goal is to generate a new image user- user-proviin provided reference images textual descriptions of hairstyle or hair color, while preserving the unrelated features of the facial region outside the hair. This task special kind of inpainting, specifically replacing the hair information in face
Tout. editing the hair based
be viewed image based on user-provided conditions. Therefore, we use the Stable Diffusson inpainting pipeline the starting point of our approach. It takes as spatial input 7 the channel-wise concatenation of encoded masked image E ((Ma), resized binary inpainting mask m E {0, 1) 1x and the denoising network input zt. IM is the input image Lero masked according to the inpainting mask

---

Mc E 1) 13 and the binary inpainting mask m is the resized version according to the latent space spatial dimension of the original inpainting mask Mc. The spatial input of the inpainting denoising network is y = [ze; m. {(M)) [
ControlNet. is an extension of the diffusion model that incorporates conditional control to generate high-quality images with specific attributes. The main objective is to leverage additional conditional information с (e.g.. Canny edge, Openpose, Depth Map) to steer the image generation process. Combined with spatial imput y (e.g.. 2r), the diffusion process can be expressed as:
pe (2 2,, c) = (2(1111 (o t, c), Ee (2z, t, c)),
(2)
Both foo and Eg depend on the conditional information We introduce the pose image Recy obtained from are using a pose keypoint extraction model 4 during the hairstyle editing stage to align the generated hair with the face in P In the hair color editing stage, we use edge detection model to obtain Le from Ps or Irc to guide the generation of the hairstyle.
CLIP. It is a vision- vision-language language model [27], which aligns visual and textual inputs in a shared embedding space. CLIP consists of an image encoder VE and text encoder Te that extract feature representations (e( E Id and T3 (T)) E R for an input image I and its corresponding text caption T, respectively. Here, d the size of the CLIP embedding space, and E,, is the embedding lookup layer which maps each T tokenized word to the token embedding space W. To tackle the text-multi-cole hair editing task, we utilize hair color structure text Tm in comparison with the multi-color hairstyle image m. scraped from the internet to fine-tune the CLIP. The fine-tuning process involves aligning the texts with the corresponding images. To enable the model to learn the color structure of hair, we perform data augmentation through rotation and symmetry operations, introducing variety of directional patterns. The fine-tuning objective is given by: ming E(Im,Am)>-D. [C(T i Im; 0)] where the augmented dataset Daug is defined as: Dang {Augment(x:) X4 E D}. 0 represents the parameters of the CLIP text encoder TE.
For reference image- hairstyle editing, we employ textual inversion. Initially, we construct textual prompt that guides the diffusion process. This prompt is tokenized and each token is mapped into the token embedding space using the CLIP embedding lookup module, resulting in V Next, we encode the reference image I using the CLIP visual encoder Vg, feeding the features extracted from the last hidden layer into the textual inversion adapter Fg. This adapter maps the input visual features to the CLIP token embedding space W. The final prompt embedding vectors, combined with the predicted pseudo-word token embeddings, are formulated as follows:
F = Concat(V (V.()))))
The concatenated embedding E is then fed into the CLIP text encoder TE, and the output is used to condition the denoising network by leveraging the existing Stable Diffusion textual cross-attention mechanism. We train F6 with the input of a agir-agnostic masked face EIIM), compared with face- agnostic mask Ma. and the latent variable 2. During the training of the adapter Fe, the parameters of both CLIP and the U-net are kept frozen.
3.2 HairDiffusion
Data Preparation. We define the hair-agnostic masks for inpainting in the two stages. In the hair editing stage, the M2 retains facial information (excluding the forehead area) and neck information while removing other irrelevant (bakkousba hair, etc). In the color editing stage, the mask Me is used remove the hair information, Mc Mn. U Mp. Mn, denotes Lure hair region mask, Mp denotes ps hair region mask. The detailed mask design is provided in the Appendix A.
Style Proxy. Given style reference image 1; text prompt T of hairstyle. Our goal is obtain the style proxy ps to inpaint the source image 4are hair region with the desired hairstyle. We use the hair-agnostic mask M, define the inpainting region of the hair. To ensure that the generated hairstyle aligns with the orientation of the face in Jare, we employ pose key points image Tuey driven from Ire using a 3D key points extractor model Ep to guide the hairstyle inpainting process. The process can be formulated as:
p D(Co(E(TE O Ma),t, (((E,(TT.)) Ma))
(4)
where v represents the attention conditioning input (T=( (T))), C represents the control input from ControlNet.
5

---

Color Proxy. Given color reference image I or a text prompt T" of hair color. We use the warping module W to align the I with L.. The process to get the color proxy operates through the following process:
= - Ie) 3 M
(5)
Where M represents the hair region mask of L.. The entire framework can be described by:
I = HairDiffusion Aue. [M Mol. _T" T I,),
(6)
where [-] denotes optional conditional inputs.
3.3 Multi-Stage Hairstyle Blend
We employ MHB to achieve separate control of hairstyle and hair color. Given an input image Tare and the corresponding inpainting mask Mc for the hair region, we obtain the masked image /M. removing the original hair information. We then encode the color proxy 8e and Iy to the latent space of the diffusion model through an encoder E, obtaining the latent vectors = and zth respectively. We use the hyperparameter T to divide the denoising process into two stages. In the initial stage, we spatially blend z with zt to obtain modified latent z/ at a certain intermediate timestep T during the sampling, the blending operation B is formulated as follows:
{ 2f m. + 27" @ (1 me), i 2
4 = B(g m t)
(7)
otherwise,
where of is noised color latent at time t and m. is down sampled from the inpainting mask Mc. Owing the blend is in the early sample stage, thereby utilizing P to guide hair color generation and avoiding artifacts at the hair boundaries caused by the latent mask mixing. To prevent affecting irrelevant features during denoising, blending =t ensures that parts outside me remain unchanged. Alternatively, hair color can be edited using textual description T by guiding the denoising steps within the U-Net, with text-encoded information via CLIP's text encoder. In the latter stage, leverage the context-aware capability of inpainting to unconditionally generate the parts of the face and background occluded by the hair. Throughout the entire denoising process, we incorporate 1a or Lare via ControlNet to guide the hairstyle generation.
3.4 Hair Color Aligning
To transform the IE into color proxy pe for hair color editing, we introduce a warping module
inspired by the virtual try-on task 5 22), which is being employed in the domain of hair editing for
the first time best of our knowledge. We adopted the network architecture of HR- IR-VITON [22] incorporates DensPose D [10] and hair-agnostic segmentation map Sa driven from facial image 4t as priors to enable warping module W to account for facial poses. However, there are two main gaps between the two tasks. Virtual try-on datasets typically consist of two types of images: standalone clothing images and compared clothing images worn on models. These images are jointly used to train models so that they can transfer clothing to images of models. Existing facial datasets lack individual representations for hair. To reduce the gap, we first get the semantic segmentation M of facial image I using facial segmentation network, and the hair mask denotes M,, while the hair-agnostic mask represented as My. Then, we obtain the hair representation J, = T O Mn and hair-agnostic facial representation, defined as T, T @ My. We adopt several data augmentation techniques, denoted as A, including flipping, rotation, trigonometric distortion, and scaling, on the hair representation image In to break down the connection with the facial image 4,. Formally, the condition input for training the warping module is then given by:
(8) This strategy enables the warping module align the hair region of 1, to the hair region of the target hairstyle Ic, thereby obtaining In as shown in Figure c). When large warping or significant differences in hair length cause image tearing, gaps, incomplete filling of the target hairstyle areas, we leverage PatchMatch [29] to sense surrounding pixels and seamlessly fill the missing hair color, ensuring natural restoration.
((A(A))
The second gap is to extract the color features while filtering out the fighfrreyency details of the original hair. To address this, we applied bilateral filtering eliminate texture details from the hair,

---

which effectively preserved the original hair color and color structure information intact. We denote the bilateral filtering operation as G, and its application the image can be represented as:
p
where / represents the parameters of g.
Input
Ours
HairCLIPy2
HairCLIP
TediGAN
Control-SD
PP(Control)
DiffCLIP
Pl
p
blond
ie
lo
Io
purple ombre black dip dyed
white bair
red bair
jewiro
chignon
bob cut green
Figure 3: Visual comparison with HairCLIPV2 (6), HairCLIP 35 TediGAN B8), Power- Paint(" "CContooollee""" version) (47), ControlNet-Impaintinn text descriptions (editing hairstyle, hair color, or both them) are listed the leftmost side. Our approach demonstrates better editing effects and irrelevant attribute preservation (e.g.. identity, back- ground).
[2). and DifICLIP (18]. The simplified
Experiments
Implementation Details. We train and evaluate the warping module using a CelebAMaskk-HC 21 dataset and obtain paired data corresponding to the hair region through segmentation and transforma- tion processes. For the hairstyle text descriptions and hair color descriptions, we follow the HairCLIP methodology. Additionally, we incorporate additional hair color text descriptions for multi-color commonly used hair color scenarios. The CelebA-HQ [ dataset is used to provide reference images hairstyles. Additionally, as the dataset lacks multi-color hair, we supplement with images soureed from the internet, obtaining multi-hair-cot hair-color data with total of 12 categories of hair color structure. Detailed data collection is provided in Appendix A.2. We set the batch size for the training warping module to and trained the module for 100,000 iterations. The learning rates for both the generator and the discriminator in the warping module are set to 0.0002.
4.1 Quantitative and Qualitative Comparison.
Comparison with Text-Driven Hair Editing Methods. [shloss the quantitative results on
IDS, PSNR, and SSIM with the leading text-driven hair editing methods on the CelebA-HQ []] testset(2,000 images), following the evaluation setting of HairCLIPv2. For Diffusion-CI.n [18). we finetune a model for each text description. For the TediGAN 3S the number of optimization iterations is set to 200. We set the image generation size to 1024xx x 1024, consistent with the compared

---

methods. Our method accomplishes maximizing the preservation of irrelevant attributes. Figure shows the qualitative results StyleGAN- based and advanced SD-inpainting methods. The three diffusion-based methods are based on Stable Diffusion v1.5 in 50 steps. As shown in Figure 3 our method accomplishes satisfactory hair editing effects with hair color structure text. In text editing hairstyle scenarios (lines 4 and 5), compared to the diffusion based method, we can faithfully preserve the original hair color. The StyleGAN- methods are unsatisfactory in the preservation of irrelevant attributes. Even though HairCLIP22 performs well visually, it still struggles to preserve details, as shown in Figure
Methods
IDSt
PSNR
SSIM
Ours
0.94
33.1
0.95
HairCLIPv2
36
0.84
29.5
0.91
HairCLIP
135
0.45
21.6
0.74
TediGAN
38
0.16
22.5
0.74
DiffCLIP
[8
0.71
26.8
0.86
Table 1: Quantitative comparison for irrelevant attributes preservation. IDS [3] denotes identity
similarity, PSNR, and SSIM are calculated at the intersected non-hain regions before and after editing.
Ours
HairCLIPv2
HairCLIPv2
red jewfro hairstyle
blond ringlets
hairstyle
Figure 4: Comparison with HairCLIPv2 (36 in detail. Our approach shows better preservation of irrelevant attributes.
Comparison with Hair Color Transfer Methods. Due to the existing facial datasets lack of multi-color data, we randomly selected 666 images from collected multi-color dataset color reference images. The first 666 images from the CelebA-HQ I test set as input images. As shown in Figure our result excels transferring color structures while remaining faithful the hair color of the reference image. Although MichiGAN [3 demonstrate the ability to achieve multiple color structure, however, it results in poor hairstyle quality. On the other hand, HairCLIP, HairCLIPv2, HairFastGAN and Barbershop are constrained by the data distributions of StyleGAN, limiting their ability generate novel colors within the latent space.
User Study. As shown in Table our method outperformed other methods in terms of accuracy. In terms of preservation, method superior to traditional hair editing methods but slightly inferior to PowerPaint [47], which is specifically designed for inpainting using diffusion techniques. In the color transfer comparison, due to hair color feature blending mechanism, when editing hair color via color proxies that significantly differ from the target image, the overall image color balance shifts, resulting preservation score lower than MichiGAN 33) Regarding naturalness,
method was slightly inferior to SYH [19], which performs optimization entirely the latent space StyleGAN. The detailed user study setting is provided in Appendix A.3. Visual comparison on cross-model is provided in Appendix A.4.
Text-Driven
Color Transfer
Cross-Model
Metrics
Ours
36
[35]
[38
[18]
42
47
Ours
36
[35]
45
[9
33
[1
Ours
[36]
[35
Accuracy
42.9
21.3
13.2
1.5
1.3
5.0
14.8
64.5
8.0
4.7
10.8
1.8
6.8
3.5
68.3 21.7 10.1
Preservation
24.5
20.5
2.1
1.5
3.3
23.3
25.1
21.3
20.5
2.8
10.5
16.0
23.8
5.3
48.5 38.5 13.0
Naturalness
27.8
24.8
6.3
2.3
0.3
21.5
26.3
26.3
7.3
9.8
3.8
28.3
2.3
22.3
55.3
36.8 8.0
Table 2: User study on text-driven image manipulation, color transfer, and cross-modal hair editing methods. Accuracy denotes the manipulation accuracy for given conditional inputs, Preservation indicates the ability to preserve irrelevant regions and Naturalness denotes the visual realism of the manipulated image. The numbers represent the percentage votes. Bold and underline denote the best and the second best result, respectively.
8

---

Input
Example
Ours
HairCLIPv2
HairCLIP
Barbershop
CtrlHair
MichiGAN
FastGAN
EP
EV
2
Figure 5: Visual comparison with HairCLIPv2 36). HairCLIP [35] Barbershop
45)) CtrlHair [1].
MichiGAN 33
and HairFastGAN 226 on hair color transfer.
4.2 Ablation Study
To demonstrate the effect of each step in our pipeline, we conducted ablation experiments by incrementally adding conditions. As shown in Figure [6] the first row illustrates hairstyle editing based on a given hairstyle text. In the original image, the absence of an agnostic mask results in the failure to generate a bob-cut hairstyle with bangs, leading to poor hairstyle generation quality, In the second step, the lack of an Openpose ControlNet causes misalignment between the generated hairstyle and the face. The third step, without color proxy, results in uncontrollable hair color. In the fourth step, the introduction of an unwarped color proxy causes random color generation in regions lacking hair color guidance from the original image and alters the background color due to the color proxy's influence. The fifth step aligns the color proxy with the target hairstyle area, ensuring the hair color matches the Source Image. In the second column, reference hair color image is used for hair color editing. In the original image, the absence of a Canny edge ControlNet results in an uncontrollable hairstyle. In the third image, the Canny edge ControlNet controls the hairstyle structure but not the color. In the fourth image, using the reference image directly as stroke map for the color proxy results in the region without hair in the reference image lacking color, with the hair color also appearing in the background. In the fifth image, the absence of Bilateral Filtering in the hairstyle causes the warped hairstyle's 's structural features adversely affect the original hairstyle structure, leading to poor hairstyle structure.
To better assess the individual contribution of the warping module, observed that after performing the warp operation, significant discrepancies in hairstyle length or complex edges can lead to alignment challenges, as illustrated in F However, by incorporating the patch match method for inpainting, we can effectively reconstruct the corresponding hair color. Additionally, the removal of texture information through blurring further enhances the results. The influence of each configuration on hairstyle generation quality is presented in Table
Model
FIDU
FIDGue
SSIM
w/o warping module
33.17
12.53
0.62
w/o patch match
27.74
8.51
0.70
w/o bilateral filtering
20.85
6.02
0.74
HairDiffusion
20.83
5.96
0.76
Table 3: Quantitative comparison of different variants of warping module with various conditions removed. We achieve the best performance by leveraging the remaining techniques.

---

Source Image
Original Imagge
Hair-Agnostic Mask
Pose Control
*Color Procy(uzwarr))
Warping Module
bob zut hairstyle
Source Image
Original Image
Canny Control
Color Proxy(unwarp)
+Warping Modale
+Bilaterra Filtering
Figure 6: Ablation studies on text-guided hairstyle editing and reference image- gegeeeggueeddddee hair color editing.
Source
Target
Original
Waperd
Patch Match
Bilateral Filter
Combine
Output
Figure 7: Visualizations of the ablation studies the warping module and corresponding post- processing.
5 Conclusion
In this work, first propose the latent diffusion-basse approach for hair editing. We introduce the MHB module and hair-agnostic masks, which enable the diffusion model to effectively control hairstyle and hair color independently while preserving unrelated attributes. Additionally, employ a warping module for the first time in this task to ensure alignment hair color, demonstrating its capability in hair color manipulation and preservation. Furthermore, by collecting image-text pairs focused on hair color structure, we further enhance our model's s ability to finely control hair color using both text and reference images.
6 Acknowledgements
This work is supported by the National Natural Science Foundation of China under Grant 62176163 82261 138629, and 62320106007, the Science and Technology Foundation of Shenzhen under Grant JCY/202103240446222 tion under Grant 2023A151501066 and the Guangdong Provincial Key Laboratory under Grant 2023B1212000076, and the Shenzhen Higher Education Stable Support Program General Project under Grant 20231 120175215001.
the Guangdong Basic and Applied Basic Research Founda-
References
[1] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How embed images into the stylegan latent space? In Proceedings of the IEEE/CVF international conference on computer vision, pages 4432-4441, 2019.
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
10

---

[3] Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. ACM Transactions on Graphics (TOG), 42(4):1-11, 2023.
[4] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7291-7299, 2017.
[5] Seunghwan Choi, Sunghyun Park, Minsoo Lee, and Jaegul Choo. Viton-hd: High-resolutinn vir- tual try-on via misalignment- normalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14131-14140, 2021.
[6] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[7] Haoye Dong, Xiaodan Liang, Ke Gong, Hanjiang Lai, Jía Zhu, and Jian Yin. Soft-gated warping- gan for pose-guided person image synthesis. Advances in neural information processing systems,
31, 2018.
[8] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farleyy Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gan. Advances in neural information processing systems, 27, 2014.
[9] Artur Grigorev, Artem Sevastopolsky, Alexander Vakhitov, and Victor Lempitsky. Coordinate- based texture inpainting for pose-gute guided image generation. arXiv preprint 181.11981..1145,, 2018.
[10] Riza Alp Güler, Natalia Neverova, and Iasonas Kokkinos. Densepose: Dense human pose estimation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7297-7306, 2018
[11] Xuyang Guo, Meina Kan, Tianle Chen, and Shiguang Shan. Gan with multivariate disentangling for controllable hair editing. In European Conference on Computer Vision, pages 655-670 Springer, 2022.
[12] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 6855840-6851, 2020.
[13] Yuge Huang, Yuhan Wang, Ying Tai, Xiaoming Liu, Pengcheng Shen, Shaoxin Li, Jilin Li, and Feiyue Huang. Curricularface: adaptive curriculum learning loss for deep face recognition. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5901-5910, 2020.
[14] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125-11344 1134, 2017.
[15] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. growing of gans for improved quality, stability, and variation. arXiv preprint
2017.
[16] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401 2019.
[17] Sasikarn Khwanmuang, Pakkapon Phongthawee, Patsorn Sangkloy, and Supasorn Suwajanakorn. Stylegan salon: Multi-view latent optimization for pose-invariant hairstyle transfer. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8609-8618, 2023.
[18] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2426 2435, 2022.
[19] Taewoo Kim, Chaeyeon Chung, Yoonseo Kim, Sunghyun Park, Kangyeol Kim, and Jaegul Choo. Style your hair: Latent optimization for pose-invariant hairstyle transfer via local-style-aware hair alignment. In European Conference on Computer Vision, pages 188-203. Springer, 2022.
11

---

[20] Taewoo Kim, Chaeyeon Chung, Sunghyun Park, Gyojung Gu, Keonmin Nam, Wonzo Choe, Jaesung Lee, and Jaegul Choo. K-hairstyle: large-scale korean hairstyle dataset for virtual hair editing and hairstyle classification. In 2021 IEEE International Conference on Image Processing (ICIP), pages 1299-1303. IEFE, 2021.
[21] Cheng-Han Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 55585,5,, 2020.
[22] Sangyun Lee, Gyojung Gu, Sunghyun Park, Seunghwan Choi, and Jaegul Choo. High-resolution virtual try-on with misalignment and occlusion- haandle conditions. In European Conference on Computer Vision, pages 204-20 119. Springer, 2022.
[23] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference computer vision, pages 7730-3738, 2015.
[24] Chenlin Meng. Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.00 01003, 2021.
[25] Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv: 14 /784, 2014.
[26] Maxim Nikolaev, Mikhail Kuznetsov, Dmitry Vetrov, and Aibek Alanov. Hairfastgan: Realistic and robust hair transfer with a fast encoder-based approach. arXiv preprint vrXiv:2444.010%4
2024.
[27] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748-8763. PMLR, 2021.
[28] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pages 10684-10695, 2022.
[29] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[30] Rohit Saha, Brendan Duke, Florian Shkurti, Graham W Taylor, and Parham Aarabi. Loho: Latent optimization of hairstyles via orthogonalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1984-1993, 2021.
[31] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang. Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:46479-36499, 2022.
[32] Aliaksandr Siarohin, Enver Sangineto, Stéphane Lathuiliere, and Nicu Sebe. Deformable gans for pose-based human image generation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3408-38 1008-3416,00 2018
[33] Zhentao Tan, Menglei Chai, Dongdong Chen, Jing Liao, Qi Chu, Lu Yuan, Sergey Tulyakov, and Nenghai Yu. Michigan: multi-input-condiiineee hair image generation for portrait editing. arXiv preprint arXiv:2010.16417, 2020.
[34] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation. ACM Transactions on Graphics (TOG), -014) 2021.
12

---

[35] Tianyi Wei, Dongdong Chen, Wenbo Zhou, Jing Liao, Zhentao Tan, Lu Yuan, Weiming Zhang, and Nenghai Yu. Hairclip: Design your hair by text and reference image. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18072-18081, 2022.
[36] Tianyi Wei, Dongdong Chen, Wenbo Zhou, Jing Liao, Weiming Zhang, Gang Hua, and Nenghai Yu. Hairclipv2: Unifying hair editing via proxy feature blending. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2023.
[37] Keyu Wu, Yifan Ye, Lingchen Yang, Hongbo Fu, Kun Zhou, and Youyi Zheng. Neuralhdhair: Automatic high-fidelity hair modeling from single image using implicit neural representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition, pages 1526-1555 2022.
[38] Weihao Xia, Yujiu Yang, Jing-Hao Xue, and Baoyuan Wu. Tedigan: Text-guided diverse face image generation and manipulation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2021.
[39] Haotian Xue, Alexandre Araujo, Bin Hu, and Yongxin Chen. Diffusion-basee adversarial sample generation for improved stealthiness and controllability. Advances in Neural Information Processing Systems, 36, 2024.
[40] Shiyuan Yang. Xiaodong Chen, and Jing Liao. Uni-paint: A unified framework for multimodal image impainting with pretrained diffusion model. In Proceedings of the 31st ACM International Conference on Multimedia, pages 3190-3199, 2023.
[41] Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. Bisenet: Bilateral segmentation network for real-time semantic segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 925-3411 2018.
[42] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[43] Yujian Zheng, Zirong Jin, Moran Li, Haibin Huang, Chongyang Ma, Shuguang Cui, and Xiaoguang Han. Hairstep: Transfer synthetic to real using strand and depth maps for single- view 3d hair modeling. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12726-12735, 2023.
[44] Yi Zhou, Liwen Hu, Jun Xing, Weikai Chen, Han- Wei Kung, Xin Tong, and Hao Li. Haireet: Single-view hair reconstruction using convolutional neural networks. Proceedings of the European Conference on Computer Vision (ECCV), pages 235-251, 2018.
[45] Peihao Zhu, Rameen Abdal, John Femiani, and Peter Wonka. Barbershop: Gan-based image compositing using segmentation masks. arXiv preprini urXiv:2106.01500 2021.
[46] Peihao Zhu, Rameen Abdal, Yipeng Qin, John Femiani, and Peter Wonka. Improved stylegan embedding: Where are the good latents? arXiv preprint arXhv:2012.09036. 2020.
[47] Junhao Zhuang, Yanhong Zeng, Wenran Liu, Chun Yuan, and Kai Chen. A task is worth one word: Learning with task prompts for high-quality versatile image inpainting. arXiv preprint arXiv:2312.03594, 2023.
13

---

A Appendix supplemental material
A.1 Hair-Agnostic Masks Design
To transform the hair editing task into an inpainting task, we utilize hair-agnostic masks designed for hair editing tasks different stages, as shown Figure
Hairstyle Editing stage. As depicted in Figure 2 a), starting with a source image Lare. we obtain the hair-agnostic region mask M, transform the task into inpainting task. Aiming to obtain satisfactory style proxy pa, we focus on retaining facial and neck information while removing other irrelevant details (background, hair, etc). Additionally, for hairstyles with bangs, forehead information is also removed as part of this process. To get the Ma. The first step is to select reference images, (automatically) segment them, and select regions in the reference images that should be copied the target image. Let M1 SEGMENT (Trre) indicate the segmentation of reference image sre. where SEGMENT facial semantic segmentation network such as BiSeNET 41 The hair region of it is MP For the same, we also segment the Tc to get segmentation Mc, and the hair region of it represents M To ensure effective bangs generation, we obtained series of key point coordinates using a facial key point detection model 4. These key points are represented K (k. (kf,K k.),, where 1. ... nnn = 68) denotes the index of the key points. Assuming that the key points within i == 1,2, the eyebrow region are (kg, k.,)) for b n, 1, ...; m(m (m = 9), we employed Bézier curve fitting approach to identify dividing line beneath the eyebrows, splitting the face into two parts. Let B(t) represent the equation of this curve. For any pixel (x, y) located on Mf. falls below the curve defined by (t),, it is classified part of the area M below the forehead. To preserve other facial attributes such as ears and neck, we superimpose relevant attribute labels segmentation M2. The final hair-agnostic mask can be represented
Ma MP U MP
(10)
Hair Color Editing stage. Illustrated in Figure b). editing the hairstyle, the hair region of style proxy P and the target image Iare are not aligned, the purpose of the agnostic mask M, in this stage serves two functions: 1)lt removes the hair information from Iaee while preserving information outside of the hair area, such as the background, face, and neck. 2) prepares the mask to be suitable for editing with the ps_ This involves removing information in Jee corresponding to the hair region in the style proxy ensure effective hairstyle editing. So in this stage, the hair-agnostic mask can be represented as:
Mc M U
(11)
Notably, the region Mn, defined as M, M MM be inpainted using the context-aware capabilities of stable diffusion to achieve a reasonable background restoration.
Ms
M
M
Iair Color Edting
M 3
M
M
Mi 1
Hairstyle Edting
M
M
K
B(t)
Figure 8: Utilizing facial semantic segmentation and facial key points, we generate the hair-agnostic masks.
A.2 Data Collection
We crawled tens of thousands of multi-color hair images using keywordsand conducted a thorough data cleaning process, resulting in approximately 6,000 valid images. The cleaning process involved

---

Hair Color
Hair Color Structure Ombre. Dip-Dyed. Two- Sloge, and Hall, Stecked..
blond, blue. pink red. sliver. orange. green. purple...
Expert-guided Screening
Question Templates
Select words that describing the hair color from
www Wo
block, blonde
Select al words that describing the hair color structure from
Internet
"a photo of block, blonde, embre hairstyle" Datasels
Upload imago
Ombre
Figure 9: Pipeline for collecting multi-color hairstyle data and corresponding text annotations.
removing duplicates, www-usooutionn or blurry images, and filtering out irrelevant content (e.g., images without hair or those not fitting the multi-color criteria). We also manually and automatically excluded images with watermarks or copyright markings. To ensure privacy and ethical use, we applied blurring techniques to remove facial information since only hair information was necessary. These images were then input into GPT-4 2 with predefined hair color and structure categories generate text annotations like "a photo of [colors], [color structure hairstyle." We employed 10 professional annotators who rated the annotations through multiple rounds, discarding low-rated images and retaining 4,625 high-quality multi-color text-image pairs. This dataset was subsequently used to fine-tune the text encoder of the CLIP model, enhancing its performance in multimodal tasks.
A.3 User Study Setting
For the above three comparisons, we recruited 20 volunteers with backgrounds computer vision to conduct comprehensive forming a total of 60 test samples. During the testing process, the order of different methods was randomly shuffled. For each test sample, volunteers were asked to choose the best option.
study. We randomly selected 20 sets of results from each experiment,
A.4 Comparison with Cross-Modal Hair Editing Methods.
Input Image
HairCLIPY2
HairCLIP
Ours
HairCLIPv2
HaiPCLIP
cace aod onttr
Figure 10: Qualitative comparison with HairCLIP and HairCLIPV2 on cross-modal conditional input. Our approach shows better editing effects and excellent preservation of irrelevant attributes.
A.5 Examples of Reconstruction
We have supplemented self-transfer experiments with state-of-the-art methods HairFastGAN [26 and StyleGAN Salon [17]. The comparison illustrates the retention of accessories (D, H. J, M. O): multi-colon hair preservation (N, Q): hand preservation (F, K. L, U, W); unique facial features retention (I); hairstyle preservation (A, B, C, T, S. R. V); background retention (P). Our method also has limitations, such as color discrepancies in attributes other than the hairstyle in some cases, like hair color in the second column and skin color in the seventh column. This due merging the noisy latent, masked image latent, stroke map latent, and mask at the initial convolution layer of UNet architecture, where they are collectively influenced by the text embedding- Consequently, subsequent layers in the UNet model struggle to obtain pure masked image features due to the influence the text and stroke map. Quantitative comparisons of the reconstructions are shown on the right side of the Table 4
15

---

GT
HairFastGAN
StyleGAN Salon
10
Ours
Figure 11: Visualization of the reconstruction comparison with StyleGAN Salon and HairFastGAN.
A.6 Limitations
Our method also has multiple limitations while have trained warping network coupled with inpainting to align the target hair color with reference images, achieving direct color transfer remains challenging for reference hairstyles that differ significantly structure or facial pose. For instance, with styles like ombre hairstyle, it becomes difficult to produce plausible outcomes if the reference image features short hair while the target image features long hair. Similarly, when only a profile view is available, transferring hair color to frontal view becomes problematic.
Given the method's reliance segmentation networks, its performance is constrained by the segmen- tation network's 's ability to accurately delineate facial regions, especially the hair area. For intricate styles like cornrows hairstyle, which pose challenges for segmentation, achieving satisfactory results becomes notably difficult. In the task of single-color transfer, as shown in Table [ the metrics for this method are not satisfactory. However, this can be addressed by utilizing more advanced diffusion models.
More Examples of the Limitations. Figure 15 shows the limitations of the warping module in extreme cases of hair color transfer, including significant pose differences, complex textures, and large discrepancies in hairstyle regions. To visually illustrate the limitations of the warping module method, the warped results have not undergone the post- post-procsssing mentioned in the paper. The first and second rows demonstrate cases with significant differences in hair length. While the hairline region aligns well, the hair ends do not; The second row on the left, where the hair lengths are similar, performs well. This is due to the consistent inclusion of hair end information when generating the paired hair dataset used to train the warping module. The third, fourth, and fifth rows showcase significant pose differences. The warped result on the right side of the second row shows that the hair orientation aligns well with the facial orientation of the Source Image, indicating some robustness in the model across different poses. However, the right side of the second row and the third row still show misalignment in hair parting, leading to color inconsistencies in the output. The left side of the fourth row shows the model discarding the hair ends while supplementing the bangs area of the Source Image, though there is still some deviation in the hair color's s centerline. The last row depicts complex texture scenarios where the Output hair color does not match the Target Image, due compression of images required for diffusion input and the bilateral filtering operation removing high-frequency color details. For cases with missing target region hair color, post- post-processing with patch match can fill in the blank areas, as shown in Figure]]
16

---

Orignal
Hairstyle Edited
Figure 12: For complex hairstyles, existing face segmentation models struggle to accurately identify intricate hair patterns. Additionally, even though measures are taken to preserve the face and background during the multiple stages of image editing, the segmentation map indicates that there might be issues with correctly segmenting the facial regions.
M00
stroke Map
Hair Color Edited
Figure 13: By using stroke map encoder in the latent space, it inevitably overlooks details of hair color, as shown in the second column. And it does not completely match the provided hair color areas, as shown in the fourth column.
A.7 Social Impact
Negative Impact. Hair editing tasks can also carry negative implications and potential risks:
1) Social Bias: Similar to image inpainting models, hair editing models rely on internet-collected training data, which may contain social biases. Certain hairstyles could be incorrectly associated with specific races or genders, reinforcing existing social stereotypes. 2) Privacy Concerns: Hair editing involves modifying personal photos, which, if done without consent, can infringe individual privacy. 3) Misleading Information: The technology can be used to alter photos misleadingly. potentially spreading false information. This can be used maliciously to modify images of public figures or ordinary people, leading to misinformation or cyberbullying. 4) Aesthetic and Identity photo
Issues: Hairstyles are an important part of personal identity. Altering someone's hairstyle without their consent can negatively affect their self-image and identity. To address these concerns, it crucial emphasize responsible use and establish ethical guidelines when utilizing hair editing technology. This is also key focus for our future model releases.
17

---

Input
Ref
Warped
Input
Ref
Warped
BIV
EP
Figure 14:
The result of the warping module, ensures alignment of hair color with the target hair region, demonstrating its capability in hair color manipulation and preservation. To demonstrate the direct effect of the warping model, this shows the results without bilateral filtering processing. In cases where the hairstyles differ significantly, some areas may appear empty.
Positive Impact. Hair editing tasks also offer several positive impacts: 1) Creative Expression: Hair editing allows users to experiment with different hairstyles and colors, promoting creativity and self-expression. enables people to visualize new looks before making real-life changes. 2) Fashion and Beauty Industry: Professionals in the fashion and beauty industry can use hair editing tools for styling consultations, marketing campaigns. and virtual makeovers, enhancing customer engagement and satisfaction. 3) Accessibility: Hair editing technology can help individuals who may not have access professional styling services explore and enjoy different hairstyles virtually.
A.8 Detailed information
We perform the inference of different diffusion-based methods in the same setting unless specifically clarified, i.e., NVIDIA A40 following their opensource code with base model of Stabe Diffusion v1.5 in 50 steps, with a guidance scale of 7.5. We use an Adam optimizer with a batch size of 16. The initialized learning rate for the generator is set to 0.0002 and for the discriminator also to 0,0002.
18

---

Source
Target
Warped
Combine
Ourput
Source
Target
Warped
Combine
Output
CCC
Figure 15: Extreme Cases of the warping module, warped is the unprocessed output of the warping module. Combine is a visualization of the alignment between the warped output and the corresponding region of the ground truth image. Output treats the warped image color proxy for the source image.
Single Color transfer
Reconstruction
Model
FID
FIDeuP
LPIPS
PSNRT
FID
FIDeLIP
HairCLIP
40.08
10.94
0.36
14.08
35.49
10.48
HairCIIPV2
20.21
6.55
0.16
19.71
10.09
4.08
CtrlHair
19.65
3.62
0.15
19.96
8.03
1.25
YourHar
0.14
21.74
10.69
2.73
Barbershop 20.54
3.89
0.11
21.18
13.37
2.61
HairFastGAN 20.17
3.00
0.08
23.45
9.72
0.97
HairDiffusion 20.83
5.96
0.07
31.66
5.41
0.68
Table 4: Quantitative comparison of single color transfer and self-transfer reconstruction metrics. Bold and underline denote the best and the second best result, respectively.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims in the abstract and introduction are consistent with the paper's contributions and scope. They accurately outline the key findings and methodologies presented in the study.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
19

---

The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We discuss the limitations of our work in the main text as well as in the supplementary materials.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
The paper provides comprehensive set of assumptions for each theoretical result. Additionally, it includes complete and correct proofs, ensuring the validity and reliability of the results.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch provide intuition.
20

---

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
The paper fully discloses all the necessary information required to reproduce the main experimental results. This ensures that the main claims and conclusions of the paper can be independently verified and validated.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
The paper does not provide open access to the data due to privacy concerns, as the data was collected from the internet and may involve personal information. And the paper does not provide open access the code due to the complexity of the baseline process, 638 which requires time to organize. However, efforts may be made in the future to release the 639 code once it is properly documented and prepared for public use.
Justification:
Guidelines:
21

---

The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
CodeeSumeesieeeee ieipomisssooooooo ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The paper specifies all the training and test details, including data splits, hyperparameters, optimizer types, and how they chosen. This provides comprehensive understanding the experimental setup and ensures reproducibility of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The paper specifies all the training and test details, including data splits, hyperparameters, optimizer types, and how they were chosen. This provides a comprehensive understanding of the experimental setup and ensures reproducibility of the results.
Guidelines: In the Diffusion inpainting task, it is difficult to quantitatively assess the quality of hair editing due to the lack of ground truth for the inpainted parts. Most of these visual tasks are evaluated through user studies and the visual quality of the images.
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
22

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eтror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
The paper provides detailed information on the computer resources required for each experiment, including the type of compute workers, memory usage. and exccution time. This allows for the reproducibility of the experiments under similar computational conditions.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httttt:// httpc//pbblcccccccccc.....eeeee.
Answer: [Yes]
Justification: The research conducted in the paper adheres to the NeurIPS Code of Ethics, ensuring ethical standards in all aspects of the research process, including data collection, experimentation, and reporting of results.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses both potential positive and negative societal impacts of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
23

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We applied blurring to the scraped facial data to remove facial information.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original paper that produced the code package or dataset.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
24

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code//modll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: We conducted experiments involving human subjects part of the User Study, but these experiments are not main contribution the paper. We adhered to the NeurIPS Code of Ethics, ensuring that all participants were fully informed and participated voluntarily.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: ensure that these data are used solely for research purposes have been approved by the IRB. Additionally, since our method does not require facial information but only hairstyle information, we processed these images to remove any personally identifi- able information and confirmed that the data was obtained from public resources with appropriate usage permissions.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
25

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "Bi1l94Gxbd", "reviewer": "Reviewer_8NSD", "summary": "This paper presents a novel approach called HairDiffusion for editing hair in images using latent diffusion models. The main contributions of the work are:\n\n1. Introduction of the Multi-stage Hairstyle Blend (MHB) method for effectively separating control over hair color and hairstyle in the latent space of the diffusion model. MHB divides the diffusion process into two stages, allowing for precise guidance of hair color generation and context-aware generation of the rest of the image.\n\n2. Development of a warping module to align hair color with the target area. This module adapts the HR-VITON architecture, using DensePose and segmentation maps to account for facial poses.\n\n3. Utilization of hair-agnostic masks to transform the hair editing task into an inpainting task. The authors developed two types of masks for different editing stages, which effectively preserve necessary information and remove unnecessary information.\n\n4. Fine-tuning of the CLIP model on a dataset with multi-colored hairstyles to improve editing of complex hair color structures. The authors applied data augmentation to increase pattern diversity.\n\nThe authors demonstrate that their method outperforms existing approaches in hairstyle and hair color editing tasks, especially for complex multi-colored styles. The method allows for editing hairstyle and hair color separately or together, using textual descriptions or reference images. Experiments show that HairDiffusion better preserves relevant image attributes (e.g., facial features, background) compared to existing methods.\n\nThe integration of these components into a single pipeline enables HairDiffusion to work effectively with both simple and complex multi-colored hairstyles while preserving other image attributes.", "questions": "1. As a strong point of your method, you indicate excellent quality in preserving facial details, but you compare it with HairCLIP v2, which works in a relatively low-dimensional FS space in StyleGAN. Could you provide more examples of facial detail preservation on very complex images in comparison with methods like StyleGANSalon and HairFastGAN?\n\n2. Could you provide more examples of the limitations of your method? We would like to see how the method works with large pose differences, complex textures, and very different hair shapes in the image domain. With these images, we would like to understand how well the concept preservation works in the CLIP space, how well the warp module works, and how well the entire method functions overall.\n\n3. Could you provide information on how you collected the additional dataset with hair colors?", "limitations": "The paper does not adequately address the limitations of the proposed method. The authors dedicate only two brief paragraphs to limitations, and these are relegated to the supplementary materials. This treatment is insufficient for a comprehensive understanding of the method's constraints. A more appropriate approach would be to include a detailed discussion of limitations in the main body of the paper. Furthermore, the authors should present visual examples demonstrating scenarios where each module, as well as the overall method, underperforms. These examples should be accompanied by in-depth analysis to provide insights into the reasons for these limitations and potential avenues for future improvements. Such transparency would significantly enhance the paper's scientific value and reproducibility.", "rating": 4, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "A major strength of this method is the quality of complex hair color transfer, as well as the preservation of hair texture, which other methods typically don't pay much attention to. The approach demonstrates exceptional ability in handling multi-colored hairstyles. Additionally, the method's capacity to preserve relevant attributes such as facial features and background elements sets it apart from existing techniques. The flexibility to edit hairstyle and color separately or together, using either text or reference images, adds to its versatility and practical applicability.", "weaknesses": "The paper exhibits several weaknesses. Firstly, it employs confusing notations, some of which are not properly introduced in the text. Secondly, the ablation study lacks clarity and sufficient detail. There is also a notable absence of comprehensive information regarding the method's limitations. For instance, the paper fails to provide examples demonstrating how the style transfer functions when transferring from long to short hair, how it handles cases with significant pose differences, or how it manages complex textures and hairstyles. Moreover, there is insufficient information about the limitations of the warping module. All presented images showcasing this module display identical hair shapes, suggesting that the warping module's functionality is not truly tested. This makes it impossible to accurately evaluate the quality of its performance.\n\nFurthermore, the paper lacks sufficient information for its reproduction, as it wasn't explained how exactly and on what data the CLIP model was fine-tuned. In general, there is very little information throughout the work about additional datasets, where and how the data was scraped, and about hyperparameters.\n\nThe user study is very poorly described; there is no information anywhere about what exactly the questions looked like and in which domains these experiments were conducted. There are also questions about the statistical significance of the results.\n\nThe scientific novelty of this work is questionable. As demonstrated in Figure 3, the Control-SD method produces hairstyles that are virtually identical to those generated by HairDiffusion. This similarity suggests that the main contribution of the work is specifically in the transfer of hair color rather than the creation of the hairstyle. The color transfer component primarily consists of two stage: the warping module (which is based on HR-VITON, published in 2022) and the Multi-stage Hairstyle Blend (MHB) method (which utilizes ControlNet and simple blending techniques). Given that these core components are largely derived from existing methods, the overall originality of the approach is limited.\n\nOverall, the paper suffers from a lack of comprehensive evaluation. It presents a limited range of visual comparisons and metrics, which hinders a comprehensive evaluation of the method's performance. Specifically, it lacks crucial metrics such as realism after editing (FID), realism with pose differences (FID), and full-image reconstruction metrics (PSNR, SSIM, LPIPS, FID, RMSE) as used in [1]. Moreover, the authors have not included comparisons with recent relevant works like StyleGANSalon [1] and HairFastGAN [2]. This omission of key comparisons and metrics significantly limits the reader's ability to fully assess the performance of the proposed method in relation to the current state-of-the-art approaches.\n\n[1] Sasikarn Khwanmuang, Pakkapon Phongthawee, Patsorn Sangkloy, Supasorn Suwajanakorn. StyleGAN Salon: Multi-View Latent Optimization for Pose-Invariant Hairstyle Transfer. arXiv preprint arXiv:2304.02744, 2023.\n\n[2] Maxim Nikolaev, Mikhail Kuznetsov, Dmitry Vetrov, Aibek Alanov. HairFastGAN: Realistic and Robust Hair Transfer with a Fast Encoder-Based Approach. arXiv preprint arXiv:2404.01094, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback.\n1. We will correct the typos and missing text, and supplement the finetune hyperparameter settings in the future version.\n2. We have showed the limitations of our method's dependency on masks in the supplementary materials. We have supplemented the limitations and ablation experiments proving the effectiveness of the warping module (Figure 1, Table 2).\n3. The results of the ablation experiments in our paper clearly demonstrate the considerations at each step of our pipeline, and the reasons have been analyzed.\n4. User Study strictly follows HairCLIPv2.\n5. We have supplemented the experiments with HairFastGAN metrics and detailed visual comparisons with HairFastGAN and StyleGANSalon in the global response sections G3, G4 and G5. The following outlines reasons for not making certain comparisons:\n   - Quantitative comparisons with StyleGANSalon: StyleGANSalon is based on EG3D pre-trained on the FFHQ dataset for pose transfer, and EG3D does not provide pre-trained models on CelebA-HQ. Hence, we do not conduct quantitative metric comparisons with StyleGANSalon. Moreover, StyleGANSalon performs comparisons on two subsets of the FFHQ dataset without disclosing the partitioning method.\n   - Quantitative self-transfer comparisons with StyleGANSalon: Unlike HairFastGAN and our method, which can decouple and transfer hair color and hairstyle, StyleGANSalon transfers the entire hairstyle. Thus, we believe that comparisons would be unfair.\n   - Qualitative color transfer comparisons with StyleGANSalon: StyleGANSalon transfers the entire hairstyle and cannot decouple and transfer hair color independently, making hair color transfer visual comparisons infeasible.\n\nRegarding the concern that \"the Control-SD method produces hairstyles that are virtually identical to those generated by HairDiffusion\": The advantage of our method over diffusion-based text2img lies in multi-color text control and the maintenance of the original hair color when only using text to control the hairstyle.\nOur work is the first to introduce a warping module for color alignment in the field of hairstyle editing, differing from previous StyleGAN-based methods. This approach faithfully aligns with the target hair color without being constrained by the limitations of the StyleGAN latent space. To achieve color alignment through the warping module, we undertook the following steps:\n1. Addressing the lack of paired datasets: We used semantic segmentation and data augmentation to obtain hair regions deviating from the original face.\n2. Preserving hairstyle details: We employed bilateral filtering to eliminate low-frequency details, improving the quality of hairstyle generation.\nOverall, our approach can guide future researchers in applying the warping module to hairstyle editing.\nThe hair-agnostic masks we proposed is crucial for enabling hairstyle editing tasks with diffusion methods. It maximizes the preservation of hairstyle-independent attributes while retaining the editability of the hairstyle region. Although our pipeline borrows certain components from existing techniques, we focus on integrating and optimizing these techniques to address specific issues in practical applications. For example, achieving a balance between hair color and hairstyle blending while minimizing artifacts, filling in blank regions without hair, improving the quality of generated hairstyles, and achieving more flexible multi-condition editing through the multi-step decoupling of hairstyle and hair color are all considerations in practical applications.\n\nResponses to specific questions are as follows:\n\n**A1:** In the global response sections G3, G4 and G5, we have supplemented experiments with HairFastGAN and StyleGANSalon, including detailed visual comparisons in self-transfer methods with HairFastGAN and StyleGANSalon. \nAs shown in Figure 3, our method demonstrates better visual effects in preserving multi-color hairstyles and hairstyle details under facial detail attributes and occlusion conditions. Since whole hair transfer is not the focus of our work and StyleGANSalon is a pose transfer model pre-trained on EG3D with the FFHQ dataset, we do not conduct quantitative metric comparisons with StyleGANSalon.\n\n**A2:** \n- 1. **Limitations of the Warping Module**: We have supplemented discussions on the limitations of the warping module in extreme cases of hair color transfer in Figure 1 of the supplementary PDF, including significant pose differences, complex textures, and large differences in hairstyle regions.\n- 2. **Effectiveness of the Warping Module**: We have added an ablation experiment where 2000 images from the CelebA-HQ validation set are divided into source images and target hair color images for mutual color transfer experiments. The results are shown in Table 1 of the supplementary PDF.\nAs for Effectiveness of CLIP Fine-tuning, the results are shown in section G6 of the supplementary PDF.\n\n**A3:**\nDataset Details: Number : 4,625; \n\nAverage size: 224.93 x 264.25; \n\nCategories: \n- Hair colors: purple, red, green, blue, brown, silver, blonde, pink, burgundy; \n- Color structures: ombre, dip dyed, streaked, half and half, split color.\n\nData acquisition: We crawled tens of thousands of multi-color hair images using keywords, cleaned the data, and obtained approximately six thousand multi-color hair images. These images were input into GPT with predefined hair color and structure categories to generate text annotations like \"a photo of {colors}, {color structure} hairstyle.\" We then had 10 professional annotators rate the annotations in multiple rounds, discarding low-rated images, resulting in 4,625 multi-color text-image pairs.  If the paper is accepted, we will make this dataset publicly available.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "glpVnCfUDq", "reviewer": "Reviewer_Ubef", "summary": "The paper introduces an approach for hair editing using Latent Diffusion Models (LDMs). A warping module ensures precise alignment of the target hair mask and enables hair color structure editing using reference images. The proposed Multi-Hair Basis (MHB) method within LDMs decouples hair color and hairstyle. The authors showcase the performance of their method through GAN-based methods via qualitative and quantitative evaluations,", "questions": "1) Most examples in the results are front-facing subjects. How does the method perform when the reference and source images have diverse or significantly different poses?\n\n2) GAN-based approaches often overfit images for high-fidelity reconstruction, limiting further edits. How does the current method handle this issue?\n\n3) Can your method support additional edits post-hairstyle transfer, such as shortening hair, making it wavy or curly, or changing the subject's pose to view from different angles?\n\n4) What specific advantages does the current method offer over existing GAN frameworks, especially regarding editability and handling diverse poses?\n\nAlso, check the weakness section.", "limitations": "The authors have discussed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces a warping module ensuring precise alignment of the target hair mask. This helps in handling some small mismatches between the images.\n\nThe method separates hair color from hairstyle. This allows flexibility and control over the hair transfer task.\n\nThe paper also shows results of text-based hairstyle editing, reference image-based hair color editing, and claims to preserve facial attributes.", "weaknesses": "1) Most of the examples shown in the results are front-facing subjects in the case of the hairstyle transfer. In real-world applications, there are cases where the reference and source images can have diverse/different poses. The lack of such results raises the question if this is a limitation of the method. Such problems are addressed in the paper HairNet[1]. \n\n2) A limitation of some GAN-based approaches compared in the paper is that to achieve high-fidelity reconstruction, the images are overfitted into the networks. As such most of these methods do not support further edits, for example shortening the transferred hair, making it wavy or curly or changing the pose of the subject to view the subject from a different angle. The current method does not seem to solve this problem either. What are the advantages of these GAN frameworks? Some of these methods support further editing as well. (check video associated with HairNet). https://www.youtube.com/watch?v=WBB43cgCFZM&t=153s\n\n3) Some of the GAN based approaches can also control the degree of hairstyle transfer (Barbershop). Does this method control this efficiently?\n\n[1] Zhu, Peihao, et al. \"Hairnet: Hairstyle transfer with pose changes.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you to the reviewers for pointing out the weaknesses.\n1. We have reviewed the HairNet video and paper. The issue of transferring hair across diverse/different poses is not the problem addressed in our paper. HairNet can effectively control different facial angles, which would be highly beneficial for our method to achieve multi-angle and multi-color hair transfer in the future. We will cite this work in our revised version. However, it is challenging for GAN-based methods to excel in both editability and retention of original features. In Figure 5 of our parper, we compare various space mapping methods based on StyleGAN (including Barbershop based on FS space), and it is evident that they struggle to reconstruct such niche features as multi-color hairstyles. In the global response,  we present some examples of our Warping module handling cases with significantly different poses.\n2. **Advantages of GAN-based methods**: \n  1. The speed of image generation is very fast. Benefiting from the decoupled facial features in StyleGAN, it is almost possible to change certain feature vectors and quickly generate the corresponding images.\n  2. They retain editability, allowing operations in the latent space, where feature vectors in the StyleGAN latent space can be edited to control different features such as hair length and face orientation.\n3. Barbershop cannot control multi-color hair transfer. Although the FS space based on optimization can retain facial feature details, it is difficult to decouple the two features of hair color and hair color structure, because these two features are highly coupled (Figure 5 in the parper compares Barbershop).\n\nBelow are the responses to the questions:\n\n**A1:** The global response section G1  discussion on the Warping module's ability to transfer hair color under extreme conditions, including significant posture differences, complex textures, and large differences in hair regions.\n\n**A2:** Diffusion models generate images through a multi-step denoising process, from pure noise to clear images. This allows the model to better capture image details and semantic information at different levels during the generation process, rather than generating the entire image at once. This gradual generation approach helps avoid overfitting, as each step improves and adjusts different levels of the image. This editing capability makes diffusion models more flexible and controllable for different editing tasks. Our paper adopts several common Diffusion Inpainting methods to ensure the masked region remains unchanged. For example, we input the Hair Agnostic Mask and Source Image together into the model, allowing it to distinguish between known and unknown areas, thus only filling in the unknown areas without affecting the known ones. During the generation process, noise can be injected only into the masked areas while keeping the unmasked areas unaffected. This means the unmasked areas always retain their original pixel values during the diffusion process, ensuring these areas are not influenced by the generation process. For the hairstyle region needing editing, we remove information from the original image other than the face, use sparse Controlnet's openpose and textual information to control the hairstyle generation StyleProxy, then further obtain the hairstyle region mask of StyleProxy. As mentioned above, we input these together into the model, thereby preserving both the editability of the hairstyle and the high fidelity of the unrelated regions.\n\n**A3:** Our method does not support additional edits post-hairstyle transfer. Using StyleGAN, direct operations in the latent space can achieve quick and direct feature editing. However, the process of generating images via Diffusion is a multi-step gradual denoising process, where the latent space is not directly editable but involves a series of denoising steps to achieve image generation. Therefore, direct editing in the latent space is more challenging. The generation process of diffusion models requires multiple iterative steps, making real-time editing as in StyleGAN's HairNet impractical. Although some degree of editing can be achieved through conditional generation, such editing usually needs pre-set conditions before generation, making it difficult to achieve real-time, dynamic feature adjustments.\n\n**A4:** \n- **Editability**:\n1. For text-based hairstyle editing, compared to the current HairCLIP and HairCLIPv2 based on the combination of CLIP and StyleGAN, their editing capabilities are limited by the dimensions of the StyleGAN latent space and the GAN training process. They cannot capture niche and complex facial features or information outside the face, such as background and arms. In contrast, diffusion models can combine text conditions multiple times during the generation process, ensuring that each step of image generation aligns with the text description. This method better captures the details and complexity in the text description.\n2. Our method can integrate multiple conditions for additional control, such as stroke maps to control hair color, controlnet to introduce canny maps to control hair structure, and denspose maps to ensure the hairstyle generation aligns with facial posture. These capabilities significantly enhance the editability of our method.\n\n- **Handling pose diversity**: Compared to methods specifically dealing with multi-pose hair transfer, our paper focuses more on color transfer while maintaining the original hairstyle. The warping module aligning the target hair color and region with different facial poses is a conditional GAN method, as it is currently challenging to decouple hairstyle and hair color directly in the latent space of diffusion models. There are also related articles exploring multi-view generation in diffusion models, such as DiffPortrait3D, which achieves multi-view facial effects.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XpvBV3BVIc", "reviewer": "Reviewer_X6RQ", "summary": "This paper presents a new framework for hair editing tasks, which includes editing hair color and hairstyle using text descriptions, reference images, and stroke maps. The proposed approach leverages Latent Diffusion Models (LDMs) and introduces the Multi-stage Hairstyle Blend (MHB) technique to effectively separate the control of hair color and hairstyle. Additionally, the method incorporates a warping module to align hair color with the target region, enhancing multi-color hairstyle editing. The approach is evaluated through extensive experiments and user studies, demonstrating its superiority in editing multi-color hairstyles while preserving facial attributes.", "questions": "Please check Weaknesses.", "limitations": "NA", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The proposed method demonstrates impressive performance in multi-colored hair editing, showcasing the ability to handle complex hair color structures while preserving facial attributes effectively. The integration of Latent Diffusion Models (LDMs) and the Multi-stage Hairstyle Blend (MHB) technique provides a novel approach to decoupling hair color and hairstyle, enhancing the quality of the edited images.", "weaknesses": "1. The overall pipeline primarily consists of two stages:\n   Altering the Hairstyle: This involves using a combination of control net and diffusion models with a hair-agnostic mask, 2D body pose keypoints, prompts, and a reference image as conditions. These modules are commonly found in previous methods.\n     Editing the Hair Color: This stage blends information between the hair mask and source image, along with the Canny image of the stylized image and a prompt, to achieve the final result. The main contribution lies in the Multi-stage Hairstyle Blend (MHB) method, which warps the reference hair to the source image to better maintain the hairstyle. However, warping modules are also commonly used in previous methods [4, 21].\n  The overall technical contribution does not meet the bar for this conference.\n\n2. The hair structure of the source image is not well-maintained during color editing. As demonstrated in Fig. 5, fourth row, the strand direction of the hairline in the generated image is different from the input image. The results of HairClipv2 better preserve the curliness structure of the original input hair.  Additionally, the generated image in the third row of Fig. 5 shows noticeable brightness differences and a distinct color discrepancy compared to the input image.\n\n3. The paper's writing lacks clarity. It would be beneficial to specify the difference between \\( I_c \\) and \\( I_i \\) in Fig. 2, within the caption of this figure. Additionally, using \\( I_i \\) to denote the style proxy creates confusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you to the reviewers for pointing out the weaknesses.\n\n**W1:** This study is the first to propose a diffusion-based pipeline in the hair editing field, addressing the issue of **multi-color hair structure** in **text2img** and **img2img** scenarios. Previously, no methods specifically focused on the transfer of multi-color hairstyles, and our visual effects, whether in text2img or img2img, are among the most effective to date. We introduced the Warping module into the hair editing domain to align hair colors, resulting in a Color Proxy that enables the model to faithfully align the target hair color while maintaining the hairstyle structure. To achieve color alignment through the warping module, we undertook the following work: \n1. Addressing **the lack of paired datasets**: we used semantic segmentation and data augmentation to obtain the hair region deviating from the original face.\n2. **Preserving hairstyle structure** by removing low-frequency details: bilateral filtering was employed to remove low-frequency details, enhancing the quality of hairstyle generation. Overall, the Warping Module can provide guidance for future researchers applying it in the hair editing field.\nOur proposed hair-agnostic mask is also crucial for enabling hair editing tasks using diffusion methods, as it maximizes the retention of hairstyle-independent attributes while preserving the editability of the hairstyle region. Although our pipeline incorporates components from existing technologies, we focus on integrating and optimizing these techniques to address specific issues in practical applications. For instance, how to blend hair color and hairstyle to achieve balance while reducing artifacts, filling blank areas without a hairstyle, and improving the quality of generated hairstyles. Additionally, how to achieve more flexible editing by decoupling hair color and hairstyle through multiple steps are all considerations for practical application.\n\n**W2:1. Color discrepancy in facial images**\n\nThe issue of color discrepancy outside the inpainting region is a common challenge in diffusion-based inpainting tasks. For example, in the ControlNet-Inpainting model used in our paper, the initial convolution layer of the UNet architecture merges the noisy latent, masked image latent, and mask, all of which are collectively influenced by the text embedding. Consequently, subsequent layers in the UNet model struggle to obtain pure masked image features due to the text's influence. The paper \"BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion\" proposed a blending operation to address this issue. It presents a simple pixel space solution by first blurring the mask and then performing copy-and-paste using the blurred mask. It is important to emphasize that this is a common challenge for diffusion inpainting and is beyond the scope of this paper.\n\n**2. Preserving hairstyle details** \n\nThere are indeed slight changes in the generated hairstyle structure compared to the original, but the overall structural control is good. HairCLIPv2 does not faithfully reproduce the target image's hair color at all. This paper primarily focuses on multi-color transfer. ControlNet, as a sparse control matrix, relies on canny maps for hairstyle structure retention. However, Canny maps cannot achieve pixel-level control, resulting in minor differences between the generated and original hairstyles. These differences do not affect the overall hairstyle structure. Training a more refined canny map with a hairstyle dataset could potentially improve the quality of hairstyle retention, which is a direction for future improvements. As shown in Figure 3 and Table 2 of the supplementary PDF, we supplemented qualitative and quantitative experiments to demonstrate our method's ability to preserve hairstyle details.\n\n**W3:** When the target hairstyle is the one generated by the Hairstyle Editing Stage,I_c is P^s, while retaining the original hairstyle means I_i. This aspect was not clearly explained. The style proxy is intended to guide images with the target hairstyle, as mentioned in the introduction. Therefore, when only changing the hair color while retaining the original hairstyle, P^s in Figure 2 will be replaced by I_i. We will address this in future versions. Thank you for your help!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x2qrRCev0X", "reviewer": "Reviewer_GkwL", "summary": "### NeurIPS Review\n\n#### Summary:\nThis submission presents an innovative 2D hairstyle editing pipeline leveraging latent diffusion models (LDM). The key component of this method is the Multi-Stage Hairstyle Blend (MHB), which facilitates separate control over hairstyle and hair color. By integrating structural information from non-hair regions such as facial masks and keypoints, the method effectively preserves non-hair attributes in the editing results. Additionally, a hair warping module enables natural transfer of hair color from the source image to the generated target image. Extensive experiments on public datasets like CelebA-HQ demonstrate improved results compared to previous state-of-the-art methods.\n\n#### Strengths:\n\n\n#### Weaknesses:", "questions": "See weakness.", "limitations": "Yes.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Clarity and Writing: The paper is well-written and relatively easy to understand.\n- Technical Contributions: The submission presents a comprehensive system with solid and relatively novel technical contributions.\n      * Hair Warping Module: The inclusion of a warping module that transfers the color pattern from the source hairstyle to the target hairstyle is notable.\n      * Multi-Stage Hairstyle Blend (MHB) Module:** This module is effective in preserving non-hair attributes in the generated image.\n- Extensive Experiments:\n  - The method produces more natural hair editing results compared to previous approaches, maintaining consistent hair color with the input text/image (control source) and better preserving non-hair regions.\n  - A user study validates the human preference for the presented method over other methods.\n  - The ablation study includes visualizations demonstrating the unique contributions of each module/control signal to the final results, along with detailed discussions.", "weaknesses": "- More discussions on the failure cases: Most results are shown with a near-frontal head pose. Including and discussing more results with non-frontal head poses would enhance the completeness and understanding of the work. Additionally, as mentioned in the limitation section, the presented method might not work well with the color transfer between different hairstyles. It would also be great if more failure cases are included on that end.\n- Manuscript Quality: There are some typos and missing text. A thorough proofreading is necessary to refine the manuscript.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your suggestions. We have included additional failure cases in Figure 1 of the PDF, where we comprehensively demonstrate the effects of multi-view hair color transfer and discuss the reasons for poor performance.\n\nIn Figure 1 of the PDF, we have added discussions on the limitations of the warping module in extreme cases of hair color transfer, including significant pose differences, complex textures, and large discrepancies in hairstyle regions. To visually illustrate the limitations of the warping module method, the warped results in the figure have not undergone the post-processing mentioned in the paper.\nThe first and second rows demonstrate cases with significant differences in hair length. While the hairline region aligns well, the hair ends do not; however。the second row on the left, with similar hair lengths, performs well, which is related to not randomly removing hair end information when generating the paired hair dataset. The third, fourth, and fifth rows show cases with significant pose differences. The warped result on the right side of the second row shows that the hair orientation aligns well with the facial orientation of the Source Image, indicating some robustness in the model across different poses. However, the right side of the second row and the third row still show misalignment in hair parting, leading to color inconsistencies in the Output. The left side of the fourth row shows the model discarding the hair ends while supplementing the bangs area of the Source Image, though there is still some deviation in the hair color's centerline. The last row depicts  complex textures scenarios where the Output hair color does not match the Target Image, due to the compression of images required for diffusion input and the bilateral filtering operation removing high-frequency color details.\nFor cases with missing target region hair color, post-processing with PatchMatch can fill in the blank areas, as shown in Figure 4. The effectiveness is demonstrated in the ablation experiments in Table 1.\n\nWe will correct typos and missing text in the future version. Thank you for your assistance!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**G1: More Examples of the Limitations:** In Figure 1 of the PDF, we have added discussions on the limitations of the warping module in extreme cases of hair color transfer, including significant pose differences, complex textures, and large discrepancies in hairstyle regions. To visually illustrate the limitations of the warping module method, the warped results have not undergone the post-processing mentioned in the paper.
The first and second rows demonstrate cases with significant differences in hair length. While the hairline region aligns well, the hair ends do not; The second row on the left, where the hair lengths are similar, performs well. This is due to the consistent inclusion of hair end information when generating the paired hair dataset used to train the warping module. The third, fourth, and fifth rows show cases with significant pose differences. The warped result on the right side of the second row shows that the hair orientation aligns well with the facial orientation of the Source Image, indicating some robustness in the model across different poses. However, the right side of the second row and the third row still show misalignment in hair parting, leading to color inconsistencies in the Output. The left side of the fourth row shows the model discarding the hair ends while supplementing the bangs area of the Source Image, though there is still some deviation in the hair color's centerline. The last row depicts  complex textures scenarios where the Output hair color does not match the Target Image, due to the compression of images required for diffusion input and the bilateral filtering operation removing high-frequency color details.
For cases with missing target region hair color, post-processing with PatchMatch can fill in the blank areas, as shown in Figure 4. The effectiveness is demonstrated in the ablation experiments in Table 1.


**G2: More Examples of Reconstruction:** In Figure 3, we have supplemented self-transfer experiments with state-of-the-art methods HairFastGAN and StyleGAN Salon. The comparison illustrates the retention of accessories (G, M, O); multi-color hair preservation (N, Q); hand preservation (F, K, L, W); unique facial features retention (I); hairstyle preservation (A, B, C, E, T, S, R, N); background retention (P). Our method also has limitations, such as color discrepancies in attributes other than the hairstyle in some cases, like hair color in the second column and skin color in the seventh column. This is due to merging the noisy latent, masked image latent, stroke map latent, and mask at the initial convolution layer of the UNet architecture, where they are collectively influenced by the text embedding. Consequently, subsequent layers in the UNet model struggle to obtain pure masked image features due to the influence of the text and stroke map.


**G3: Hair Color Transfer Metrics:** In Table 2 left, we referenced HairFastGAN for **single color transfer** metrics testing, showing that our method performs comparably to GAN-based methods on the CelebA-HQ dataset, which predominantly features single-colored hair. However, the quality of the multi-colored hairstyles we crawled supports the fine-tuning of the CLIP model, but the majority do not reach the size of CelebA-HQ (1024x1024), and the data volume is insufficient for quantitative experiments. Thus, we only provide visual comparisons in the paper.


**G4: Hair Reconstruction Metrics:** In Table 2 right, we have referenced HairFastGAN to include more comprehensive metrics for Reconstruction quality, demonstrating the superiority of our method in overall image quality.


**G5: Comparison to HairFastGAN:** In Figure 2, our method outperforms HairFastGAN in both facial preservation and hair color transfer.


**G6:Effectiveness of CLIP Fine-tuning:** In Figure 5, We conducted a simple text-to-image alignment experiment. It demonstrates that our fine-tuned CLIP model performs better in hair color and multi-color structure compared to the original model. Some colors show overfitting, which we plan to address by balancing color proportions in the fine-tuning dataset in future work.

Meta-review: The submission received diverse scores, two positive and two negative. The authors provided detailed rebuttal and discussions with the reviewers. After carefully reading all comments, the AC found only one major concern is remaining: the lacking of novelty. After taking a look at the paper and the AC think the proposed method, though not that novel, is indeed a new thing and it really contributed new performance on hair editing. Given one negative reviewer did not give final responses on the authors' rebuttal and another negative reviewer said he will not argue for acceptance. The AC finally recommended an acceptance.
Decision: Accept (poster)