HairDiffusion: Vivid Multi-Colored Hair Editing via Latent Diffusion

Authors: yu  zeng
yang  zhang
jiachen  liu
linlin  shen
kaijun  deng
weizhao  he
jinbao  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 15:02:59

Abstract: Hair editing is a critical image synthesis task that aims to edit hair color and hairstyle using text descriptions or reference images, while preserving irrelevant attributes (e.g., identity, background, cloth). Many existing methods are based on StyleGAN to address this task. However, due to the limited spatial distribution of StyleGAN, it struggles with multiple hair color editing and facial preservation. Considering the advancements in diffusion models, we utilize Latent Diffusion Models (LDMs) for hairstyle editing. Our approach introduces Multi-stage Hairstyle Blend (MHB), effectively separating control of hair color and hairstyle in diffusion latent space. Additionally, we train a warping module to align the hair color with the target region. To further enhance multi-color hairstyle editing, we fine-tuned a CLIP model using a multi-color hairstyle dataset. Our method not only tackles the complexity of multi-color hairstyles but also addresses the challenge of preserving original colors during diffusion editing. Extensive experiments showcase the superiority of our method in editing multi-color hairstyles while preserving facial attributes given textual descriptions and reference images.

Main paper: 
 introduction: 
 Hair editing is one of the most challenging and interesting facets of semantic face editing. The essence of this task is to edit hair attributes such as color, and hairstyle from the reference image or text to the input image while preserving irrelevant attributes. Disentangling these attributes is the key to a quality solution to the problem. This task has many applications among professionals and amateurs during work with face editing programs, virtual reality, and computer games [20, 43, 37] .In recent years, numerous methods leveraging Generative Adversarial Networks (GANs) [33, 35, 14, 16, 36, 45, 17] have emerged in this field. These methods often involve mapping images into StyleGAN [16] latent space to disentangle hair features or utilizing Contrastive Language-Image Pre-Training (CLIP) model [27] to translate textual descriptions into relevant semantic vectors, facilitating manipulation via text or reference images.However, previous methods have overlooked the hair color structure. As shown in Figure 1 , "ombre hair" (the last column of the second row) means a hair color structure with a gradient transition from top-to-bottom, while "split color hair" (the last column of the first row) exhibits a hair color structure with a left-to-right transition. In traditional StyleGAN-based approaches [45, 35, 36, 17] , the scarcity of multi-color hair in the facial datasets [15, 16] used to train StyleGAN [16] , coupled with the limited distribution of latent space in StyleGAN [16] , gives rise to two primary challenges: 1) difficulty in generating intricate hair color and hairstyle due to the insufficient diversity in the training data's multi-color hair distributions; 2) challenges in preserving the original facial information when editing the latent code after mapping images into latent space, leading to difficulties in editing images while preserving irrelevant attributes.In recent years, with the advancement of diffusion models [39, 40, 47, 42, 3] , their robust and stable generative capabilities have surpassed those of GANs in many aspects [6] . In particular, Latent Diffusion Models (LDMs) [12, 40, 47, 28] have demonstrated exceptional generative capabilities, notably in image inpainting tasks. However, the application of diffusion models to hair editing encounters three challenges: 1) lack of tailored masks for hairstyle inpainting, necessitating consideration of hairstyle regions while preserving irrelevant attributes; 2) difficulty in providing sufficient control for the hair editing task, which requires faithful transfer of hair color from another image or retaining the original hair color of the image; 3) limitations in text and semantic understanding related to hair color and hairstyle, hinder the precision of CLIP-guided diffusion processes.To address these limitations, we propose a novel baseline approach based on LDMs, enabling the automatic generation of edited regions while allowing separate control of hair color and hairstyle via text and reference images individually or collaboratively. Our pipeline can be roughly divided into two stages. The first stage begins with obtaining hair-agnostic masks to acquire facial representations independent of hair. These representations are combined with facial keypoint information and processed through textual descriptions to perform the LDMs process, generating a style proxy (i.e., an image used to guide the hairstyle transfer process). The second stage involves transferring hair color using a reference image. Initially, the hair of the reference image is aligned with the source image using a pre-trained warping module. The aligned hair serves as a color proxy (i.e., an image used to guide the color and color structure transfer process), which is then input into the LDMs process to transfer or retain the hair color. Simultaneously, the Canny edge from the original hair or style proxy is employed to retain or edit the hairstyle. The key to the pipeline is to use different hair-agnostic masks for both the color proxy and style proxy at different stages and blend them in the diffusion latent space by Multi-stage Hairstyle Blend (MHB). By training the warping model to obtain aligned hair color prior information in targeted hair regions, our method effectively addresses multi-hair color editing tasks. Quantitative and qualitative evaluations, along with user studies, demonstrate the efficacy of our approach in hair editing tasks, particularly in color transformation and preservation of unrelated attributes within the diffusion model framework. Furthermore, by designing different stages of agnostic masks, we can maximize the retention of hairstyle-independent features such as background information and facial features. In summary, our contributions are outlined as follows:• We present a warping module designed for hair warping, allowing the alignment of the target hair mask with precision and enabling comprehensive hair color structure editing through reference images.• The MHB method is proposed within LDMs, which enables the decoupling of hair color and hairstyle, thereby effectively achieving high-quality hair color and hairstyle editing.• Through extensive qualitative and quantitative evaluations, we showcase the superior performance of our method in text-based hairstyle editing, reference image-based hair color editing, and preservation of facial attributes.• The application of LDMs to address the challenge of text and image-based hair editing is pioneered through the introduction of hair-agnostic facial representation masks, reframing hair editing as an inpainting task and representing a novel approach. To the best of our knowledge, this method has not been previously explored in this domain.To transform the hair editing task into an inpainting task, we utilize hair-agnostic masks designed for hair editing tasks at different stages, as shown in Figure 8 .Hairstyle Editing stage. As depicted in Figure 2 (a), starting with a source image I src , we obtain the hair-agnostic region mask M a to transform the task into an inpainting task. Aiming to obtain a satisfactory style proxy P s , we focus on retaining facial and neck information while removing other irrelevant details (background, hair, etc). Additionally, for hairstyles with bangs, forehead information is also removed as part of this process. To get the M a , The first step is to select reference images, (automatically) segment them, and select regions in the reference images that should be copied to the target image. Let M I = SEGMENT(I src ) indicate the segmentation of reference image I src , where SEGMENT is a facial semantic segmentation network such as BiSeNET [41] . The hair region of it is M h I . For the same, we also segment the I c to get segmentation M c , and the hair region of it represents M h c . To ensure effective bangs generation, we obtained a series of key point coordinates using a facial key point detection model [4] . These key points are represented as K : (k x i , k y i ), where i = 1, 2, . . . , n(n = 68) denotes the index of the key points. Assuming that the key points within the eyebrow region are (k x b , k y b ), for b = n, n + 1, ..., m(m -n = 9), we employed a Bézier curve fitting approach to identify a dividing line beneath the eyebrows, splitting the face into two parts. Let B(t) represent the equation of this curve. For any pixel (x, y) located on M 1 I , if it falls below the curve defined by B(t), it is classified as part of the area M 3 I below the forehead. To preserve other facial attributes such as ears and neck, we superimpose relevant attribute labels segmentation M 2 . The final hair-agnostic mask can be represented as:EQUATION)Hair Color Editing stage. Illustrated in Figure 2 (b). If editing the hairstyle, the hair region of style proxy P s and the target image I src are not aligned, the purpose of the agnostic mask M c in this stage serves two functions: 1)It removes the hair information from I src while preserving information outside of the hair area, such as the background, face, and neck. 2)It prepares the mask to be suitable for editing with the P s . This involves removing information in I src corresponding to the hair region in the style proxy to ensure effective hairstyle editing. So in this stage, the hair-agnostic mask can be represented as:EQUATIONNotably, the region M n , defined as M n = M h I ∩ ¬M h s , can be inpainted using the context-aware capabilities of stable diffusion to achieve a reasonable background restoration.  
related work: 
 Diffusion Models. Diffusion models have established themselves as a robust class of generative models capable of producing high-quality images through a process of iteratively denoising data [6, 12, 31, 28] . Specifically, SD-Inpainting [29] is based on the large-scale pre-trained text-to-image model, i.e., Stable Diffusion [28] . By utilizing random masks as repair masks and employing imagetext prompts, diffusion models can fill areas with content consistent with textual descriptions while retaining context awareness in other regions. Despite the success of diffusion models in the inpainting task, their potential has not been fully exploited yet. For instance, describing fine details like hair color directly through text remains challenging. Some methods combine different interaction strategies to achieve satisfactory inpainting results, such as ControlNet-Inpainting [42] which utilizes ControlNet to guide image restoration based on additional inputs like Canny edge, depth maps, etc. Additionally, some methods provide an additional reference image or involve drawing rough color strokes. In the hair editing task, it is essential to develop a framework that facilitates multimodal conditions for image inpainting. Therefore, we leverage ControlNet to control hairstyle attributes and stroke maps, managing color information and color structure.Hair Editing. As an essential component of the face, there have been numerous works on hairstyle editing and synthesis [33, 44, 45, 19, 35, 36] . Some works decouple facial attributes by using imagelevel masks and then splice hairstyle images onto facial images through generative network [33, 30] . However, this approach often results in inconsistencies in lighting or artificial shadows along the edges of the hairstyle region. Some approaches utilize the e4e [34] to map the target image and hairstyle reference image into W+ latent space of StyleGAN [16] , enabling manipulation of hairstyle and hair color via vector operations, followed by image synthesis using StyleGAN [35, 36, 26, 11] .HairCLIP [35] integrates CLIP [27] into hair-editing tasks by leveraging StyleGAN to decouple hair color and hairstyle features, enabling text-driven. Additionally, some methods attempt to enhance the generative capabilities of StyleGAN through various transformations of its latent space [1, 46, 45] . Despite these advancements, challenges persist due to the limited data distribution inherent in StyleGAN [16] , which is trained predominantly on facial datasets [16, 23] . This constraint leads to difficulties in preserving specialized facial features like earrings and eyeshadow, as well as in introducing novel attributes such as multi-colored hairstyles. To tackle the constraints of hair color diversity, this paper introduces stroke maps [24] into the denoising process of diffusion models as the color proxy prior. Utilizing a CLIP text encoder and image encoder fine-tuned with images matching the hair color to achieve editing of hair color structure through text. For reference images, we finetune the model with textual inversion.Image Deformation. Image deformation, or image warping, is a technique in computer vision and graphics for manipulating and transforming images [7, 32, 22] . The goal is to deform an input image while preserving its essential content and structure. In recent years, deep learning techniques have revolutionized the field of image deformation, particularly GANs [8] , have revolutionized the field of image deformation [7, 32, 9] . GANs are employed to learn complex deformation mappings directly from data, capturing intricate image transformations and generating realistic deformations with minimal artifacts. Conditional GANs (cGANs) [25] represent an evolution in image deformation by incorporating additional contextual information during deformation. These networks learn to map input images to desired target outputs based on specific conditions or auxiliary information [32] . For example, integrating DensePose [10] , detailing pixel-level semantic information enables precise and controlled image warping. In this paper, we employ multiple conditioning inputs to train a warping network capable of aligning hair color while accounting for the relationship between hair and facial features.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss the limitations of our work in the main text as well as in the supplementary materials. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 As shown in Figure 2 , we propose a novel solution based on LDMs for the first time in this field.In particular, our work employs the Stable Diffusion architecture as a starting point for performing the hairstyle editing task. We obtain the style proxy, denoted as P s , to capture the hairstyle prior as shown in Figure 2 (a) . Concurrently, we acquire the color proxy P c , using a warping module as shown in Figure 2 (c ). Finally, we incorporate these two proxies into the diffusion process through Multi-stage Hairstyle Blend (MHB) to blend them effectively as shown in Figure 2 (d) . Therefore, our framework supports individual or collaborative editing of hairstyle and single-color or multi-color, utilizing text, and reference images.Stable Diffusion. It consists of an autoencoder encoder E and an autoencoder decoder D, a textconditional U-Net denoising model ϵ θ , a CLIP text encoder T E , which takes text T as input. The encoder E compresses an image I into the latent space of diffusion, while the decoder D performs the inverse operation and decodes a latent variable into the pixel space. For clarity, we refer to the ϵ θ convolutional input as the spatial input γ since convolutions preserve the spatial structure, and to the attention conditioning input as ψ. The training of the denoising network ϵ θ is performed by minimizing the following loss function:EQUATIONwhere t represents the diffusing time step, γ = z t , z t is the encoded image E(I) where we stochastically add Gaussian noise ϵ ∼ N (0, 1), and ψ = [t; T E (T )]. Our goal is to generate a new image I out , editing the hair based on user-provided reference images or textual descriptions of hairstyle or hair color, while preserving the unrelated features of the facial region outside the hair. This task can be viewed as a special kind of inpainting, specifically replacing the hair information in a face image based on user-provided conditions. Therefore, we use the Stable Diffusion inpainting pipeline as the starting point of our approach. It takes as spatial input γ the channel-wise concatenation of an encoded masked image E(I M ), a resized binary inpainting mask m ∈ {0, 1} 1×h×w , and the denoising network input z t . I M is the input image I src masked according to the inpainting maskM c ∈ {0, 1}1×H×W , and the binary inpainting mask m is the resized version according to the latent space spatial dimension of the original inpainting mask M c . The spatial input of the inpainting denoising network is γ = [z t ; m; E(I M )] ∈ R (4+1+4)×h×w .ControlNet. It is an extension of the diffusion model that incorporates conditional control to generate high-quality images with specific attributes. The main objective is to leverage additional conditional information c (e.g., Canny edge, Openpose, Depth Map) to steer the image generation process. Combined with spatial input γ (e.g., z t ), the diffusion process can be expressed as:EQUATIONBoth µ θ and Σ θ depend on the conditional information c. We introduce the pose image I key , obtained from I src using a pose keypoint extraction model [4] , during the hairstyle editing stage to align the generated hair with the face in P s . In the hair color editing stage, we use an edge detection model to obtain I c from P s or I src to guide the generation of the hairstyle.CLIP. It is a vision-language model [27] , which aligns visual and textual inputs in a shared embedding space. CLIP consists of an image encoder V E and a text encoder T E that extract feature representations V E (I) ∈ R d and T E (E L (T )) ∈ R d for an input image I and its corresponding text caption T , respectively. Here, d is the size of the CLIP embedding space, and E L is the embedding lookup layer which maps each T tokenized word to the token embedding space W. To tackle the text-multi-color hair editing task, we utilize hair color structure text T m in comparison with the multi-color hairstyle image I m , scraped from the internet to fine-tune the CLIP. The fine-tuning process involves aligning the texts with the corresponding images. To enable the model to learn the color structure of hair, we perform data augmentation through rotation and symmetry operations, introducing a variety of directional patterns. The fine-tuning objective is given by: min θ E (Tm,Im)∼Daug [L(T m , I m ; θ)] where the augmented dataset D aug is defined as:D aug = {Augment(x i ) | x i ∈ D}, θ represents the parameters of the CLIP text encoder T E .For reference image-based hairstyle editing, we employ textual inversion. Initially, we construct a textual prompt q that guides the diffusion process. This prompt is tokenized and each token is mapped into the token embedding space using the CLIP embedding lookup module, resulting in V * . Next, we encode the reference image I s r using the CLIP visual encoder V E , feeding the features extracted from the last hidden layer into the textual inversion adapter F θ . This adapter maps the input visual features to the CLIP token embedding space W. The final prompt embedding vectors, combined with the predicted pseudo-word token embeddings, are formulated as follows:EQUATIONThe concatenated embedding E is then fed into the CLIP text encoder T E , and the output is used to condition the denoising network by leveraging the existing Stable Diffusion textual cross-attention mechanism. We train F θ with the input of a hair-agnostic masked face E(I M ), compared with faceagnostic mask M a , and the latent variable z. During the training of the adapter F θ , the parameters of both CLIP and the U-net are kept frozen.Data Preparation. We define the hair-agnostic masks for inpainting in the two stages. In the hair editing stage, the M a retains facial information (excluding the forehead area) and neck information while removing other irrelevant details(background, hair, etc). In the color editing stage, the mask M c is used to remove the hair information, M c = M h ∪ M p , M h denotes I src hair region mask, M p denotes P s hair region mask. The detailed mask design is provided in the Appendix A.1.Style Proxy. Given a style reference image I s r or a text prompt T s of hairstyle. Our goal is to obtain the style proxy P s to inpaint the source image I src hair region with the desired hairstyle. We use the hair-agnostic mask M a to define the inpainting region of the hair. To ensure that the generated hairstyle aligns with the orientation of the face in I src , we employ pose key points image I key driven from I src using a 3D key points extractor model E p to guide the hairstyle inpainting process. The process can be formulated as:EQUATIONwhere ψ represents the attention conditioning input [T E (T s ), I E (I s r )], C represents the control input from ControlNet.Color Proxy. Given a color reference image I c r or a text prompt T c of hair color. We use the warping module W to align the I c r with I c . The process to get the color proxy operates through the following process:EQUATIONWhere M h c represents the hair region mask of I c . The entire framework can be described by:EQUATIONwhere [•] denotes optional conditional inputs.We employ MHB to achieve separate control of hairstyle and hair color. Given an input image I src and the corresponding inpainting mask M c for the hair region, we obtain the masked image I M , removing the original hair information. We then encode the color proxy P c and I M to the latent space of the diffusion model through an encoder E, obtaining the latent vectors z c and z m respectively. We use the hyperparameter τ to divide the denoising process into two stages. In the initial stage, we spatially blend z c with z m to obtain modified latent z ′ t at a certain intermediate timestep τ during the sampling, the blending operation B is formulated as follows:EQUATIONwhere z c t is noised color latent at time t and m c is down sampled from the inpainting mask M c . Owing the blend is in the early sample stage, thereby utilizing P c to guide hair color generation and avoiding artifacts at the hair boundaries caused by the latent mask mixing. To prevent affecting irrelevant features during denoising, blending z m ensures that parts outside m c remain unchanged. Alternatively, hair color can be edited using a textual description T c by guiding the denoising steps within the U-Net, with text-encoded information via CLIP's text encoder. In the latter stage, we leverage the context-aware capability of inpainting to unconditionally generate the parts of the face and background occluded by the hair. Throughout the entire denoising process, we incorporate P s or I src via ControlNet to guide the hairstyle generation.To transform the I c r into color proxy P c for hair color editing, we introduce a warping module inspired by the virtual try-on task [5, 22] , which is being employed in the domain of hair editing for the first time to best of our knowledge. We adopted the network architecture of HR-VITON [22] incorporates DensPose D [10] and hair-agnostic segmentation map S a driven from facial image I f as priors to enable warping module W to account for facial poses. However, there are two main gaps between the two tasks. Virtual try-on datasets typically consist of two types of images: standalone clothing images and compared clothing images worn on models. These images are jointly used to train models so that they can transfer clothing to images of models. Existing facial datasets lack individual representations for hair. To reduce the gap, we first get the semantic segmentation M of facial image I f using a facial segmentation network, and the hair mask denotes M h , while the hair-agnostic mask is represented as M ′ a . Then, we obtain the hair representation I h = I f ⊙ M h and hair-agnostic facial representation, defined asI a = I f ⊙ M ′ a .We adopt several data augmentation techniques, denoted as A, including flipping, rotation, trigonometric distortion, and scaling, on the hair representation image I h to break down the connection with the facial image I f . Formally, the condition input for training the warping module is then given by:EQUATION)This strategy enables the warping module to align the hair region of I c r to the hair region of the target hairstyle I c , thereby obtaining I w as shown in Figure 2(c ). When large warping or significant differences in hair length cause image tearing, gaps, or incomplete filling of the target hairstyle areas, we leverage PatchMatch [29] to sense surrounding pixels and seamlessly fill the missing hair color, ensuring natural restoration.The second gap is to extract the color features while filtering out the high-frequency details of the original hair. To address this, we applied bilateral filtering to eliminate texture details from the hair, which effectively preserved the original hair color and color structure information intact. We denote the bilateral filtering operation as G, and its application to the image can be represented as:EQUATIONwhere θ represents the parameters of G.Figure 3 : Visual comparison with HairCLIPv2 [36] , HairCLIP [35] , TediGAN [38] , Power-Paint("ControlNet" version) [47] , ControlNet-Inpainting [42] , and DiffCLIP [18] . The simplified text descriptions (editing hairstyle, hair color, or both of them) are listed on the leftmost side. Our approach demonstrates better editing effects and irrelevant attribute preservation (e.g., identity, background).4 Experiments Implementation Details. We train and evaluate the warping module using a CelebAMask-HQ [21] dataset and obtain paired data corresponding to the hair region through segmentation and transformation processes. For the hairstyle text descriptions and hair color descriptions, we follow the HairCLIP methodology. Additionally, we incorporate additional hair color text descriptions for multi-color commonly used hair color scenarios. The CelebA-HQ [1] dataset is used to provide reference images for hairstyles. Additionally, as the dataset lacks multi-color hair, we supplement it with images sourced from the internet, obtaining multi-hair-color data with a total of 12 categories of hair color structure. Detailed data collection is provided in Appendix A.2. We set the batch size for the training warping module to 8 and trained the module for 100,000 iterations. The learning rates for both the generator and the discriminator in the warping module are set to 0.0002.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: The paper provides a comprehensive set of assumptions for each theoretical result. Additionally, it includes complete and correct proofs, ensuring the validity and reliability of the results. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.• Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
quantitative and qualitative comparison.: 
 Comparison with Text-Driven Hair Editing Methods. Table 1 shows the quantitative results on IDS, PSNR, and SSIM with the leading text-driven hair editing methods on the CelebA-HQ [1] testset(2,000 images), following the evaluation setting of HairCLIPv2. For Diffusion-CLIP [18] , we finetune a model for each text description. For the TediGAN [38] , the number of optimization iterations is set to 200. We set the image generation size to 1024×1024, consistent with the compared methods. Our method accomplishes maximizing the preservation of irrelevant attributes. Figure 3 shows the qualitative results in StyleGAN-based and advanced SD-inpainting methods. Comparison with Hair Color Transfer Methods. Due to the existing facial datasets lack of multi-color data, we randomly selected 666 images from our collected multi-color dataset as color reference images. The first 666 images from the CelebA-HQ [1] test set as input images. As shown in Figure 5 , our result excels at transferring color structures while remaining faithful to the hair color of the reference image. Although MichiGAN [33] demonstrate the ability to achieve multiple color structure, however, it results in poor hairstyle quality. On the other hand, HairCLIP, HairCLIPv2, HairFastGAN and Barbershop are constrained by the data distributions of StyleGAN, limiting their ability to generate novel colors within the latent space.User Study. As shown in Table 2 , our method outperformed other methods in terms of accuracy. In terms of preservation, our method was superior to traditional hair editing methods but slightly inferior to PowerPaint [47] , which is specifically designed for inpainting using diffusion techniques. In the color transfer comparison, due to our hair color feature blending mechanism, when editing hair color via color proxies that significantly differ from the target image, the overall image color balance shifts, resulting in a preservation score lower than MichiGAN [33] . Regarding naturalness, our method was slightly inferior to SYH [19] , which performs optimization entirely in the latent space of StyleGAN. The detailed user study setting is provided in Appendix A. , HairCLIP [35] , Barbershop [45] , CtrlHair [11] , MichiGAN [33] and HairFastGAN [26] on hair color transfer.To demonstrate the effect of each step in our pipeline, we conducted ablation experiments by incrementally adding conditions. As shown in Figure 6 , the first row illustrates hairstyle editing based on a given hairstyle text. In the original image, the absence of an agnostic mask results in the failure to generate a bob-cut hairstyle with bangs, leading to poor hairstyle generation quality. In the second step, the lack of an Openpose ControlNet [42] causes misalignment between the generated hairstyle and the face. The third step, without a color proxy, results in uncontrollable hair color.In the fourth step, the introduction of an unwarped color proxy causes random color generation in regions lacking hair color guidance from the original image and alters the background color due to the color proxy's influence. The fifth step aligns the color proxy with the target hairstyle area, ensuring the hair color matches the Source Image. In the second column, a reference hair color image is used for hair color editing. In the original image, the absence of a Canny edge ControlNet results in an uncontrollable hairstyle. In the third image, the Canny edge ControlNet controls the hairstyle structure but not the color. In the fourth image, using the reference image directly as a stroke map for the color proxy results in the region without hair in the reference image lacking color, with the hair color also appearing in the background. In the fifth image, the absence of Bilateral Filtering in the hairstyle causes the warped hairstyle's structural features to adversely affect the original hairstyle structure, leading to poor hairstyle structure.To better assess the individual contribution of the warping module, we observed that after performing the warp operation, significant discrepancies in hairstyle length or complex edges can lead to alignment challenges, as illustrated in Figure 7 . However, by incorporating the patch match method for inpainting, we can effectively reconstruct the corresponding hair color. Additionally, the removal of texture information through blurring further enhances the results. The influence of each configuration on hairstyle generation quality is presented in Table 3 . Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: The paper fully discloses all the necessary information required to reproduce the main experimental results. This ensures that the main claims and conclusions of the paper can be independently verified and validated.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
conclusion: 
 In this work, we first propose the latent diffusion-based approach for hair editing. We introduce the MHB module and hair-agnostic masks, which enable the diffusion model to effectively control hairstyle and hair color independently while preserving unrelated attributes. Additionally, we employ a warping module for the first time in this task to ensure alignment of hair color, demonstrating its capability in hair color manipulation and preservation. Furthermore, by collecting image-text pairs focused on hair color structure, we further enhance our model's ability to finely control hair color using both text and reference images.A Appendix / supplemental material A.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: The paper does not provide open access to the data due to privacy concerns, as the data was collected from the internet and may involve personal information. And the paper does not provide open access to the code due to the complexity of the baseline process, 638 which requires time to organize. However, efforts may be made in the future to release the 639 code once it is properly documented and prepared for public use.Guidelines: • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: The paper specifies all the training and test details, including data splits, hyperparameters, optimizer types, and how they were chosen. This provides a comprehensive understanding of the experimental setup and ensures reproducibility of the results. Guidelines: In the Diffusion inpainting task, it is difficult to quantitatively assess the quality of hair editing due to the lack of ground truth for the inpainted parts. Most of these visual tasks are evaluated through user studies and the visual quality of the images.• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: The paper provides detailed information on the computer resources required for each experiment, including the type of compute workers, memory usage, and execution time. This allows for the reproducibility of the experiments under similar computational conditions.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: The paper discusses both potential positive and negative societal impacts of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: We applied blurring to the scraped facial data to remove facial information. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cite the original paper that produced the code package or dataset. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We crawled tens of thousands of multi-color hair images using keywordsand conducted a thorough data cleaning process, resulting in approximately 6,000 valid images. The cleaning process involved removing duplicates, low-resolution or blurry images, and filtering out irrelevant content (e.g., images without hair or those not fitting the multi-color criteria). We also manually and automatically excluded images with watermarks or copyright markings. To ensure privacy and ethical use, we applied blurring techniques to remove facial information since only hair information was necessary. These images were then input into GPT-4 [2] with predefined hair color and structure categories to generate text annotations like "a photo of {colors}, {color structure} hairstyle." We employed 10 professional annotators who rated the annotations through multiple rounds, discarding low-rated images and retaining 4,625 high-quality multi-color text-image pairs. This dataset was subsequently used to fine-tune the text encoder of the CLIP model, enhancing its performance in multimodal tasks.For the above three comparisons, we recruited 20 volunteers with backgrounds in computer vision to conduct a comprehensive user study. We randomly selected 20 sets of results from each experiment, forming a total of 60 test samples. During the testing process, the order of different methods was randomly shuffled. For each test sample, volunteers were asked to choose the best option. Our approach shows better editing effects and excellent preservation of irrelevant attributes.We have supplemented self-transfer experiments with state-of-the-art methods HairFastGAN [26] and StyleGAN Salon [17] . The comparison illustrates the retention of accessories (D, G, H, J, M, O); multi-color hair preservation (N, Q); hand preservation (F, K, L, U, W); unique facial features retention (I); hairstyle preservation (A, B, C, E, T, S, R, V); background retention (P). Our method also has limitations, such as color discrepancies in attributes other than the hairstyle in some cases, like hair color in the second column and skin color in the seventh column. This is due to merging the noisy latent, masked image latent, stroke map latent, and mask at the initial convolution layer of the UNet architecture, where they are collectively influenced by the text embedding. Consequently, subsequent layers in the UNet model struggle to obtain pure masked image features due to the influence of the text and stroke map. Quantitative comparisons of the reconstructions are shown on the right side of the Table 4 . Our method also has multiple limitations while we have trained a warping network coupled with inpainting to align the target hair color with reference images, achieving direct color transfer remains challenging for reference hairstyles that differ significantly in structure or facial pose. For instance, with styles like ombre hairstyle, it becomes difficult to produce plausible outcomes if the reference image features short hair while the target image features long hair. Similarly, when only a profile view is available, transferring hair color to a frontal view becomes problematic.Given the method's reliance on segmentation networks, its performance is constrained by the segmentation network's ability to accurately delineate facial regions, especially the hair area. For intricate styles like cornrows hairstyle, which pose challenges for segmentation, achieving satisfactory results becomes notably difficult. In the task of single-color transfer, as shown in Table 4 , the metrics for this method are not satisfactory. However, this can be addressed by utilizing more advanced diffusion models.More Examples of the Limitations. Figure 15 shows the limitations of the warping module in extreme cases of hair color transfer, including significant pose differences, complex textures, and large discrepancies in hairstyle regions. To visually illustrate the limitations of the warping module method, the warped results have not undergone the post-processing mentioned in the paper. The first and second rows demonstrate cases with significant differences in hair length. While the hairline region aligns well, the hair ends do not; The second row on the left, where the hair lengths are similar, performs well. This is due to the consistent inclusion of hair end information when generating the paired hair dataset used to train the warping module. The third, fourth, and fifth rows showcase significant pose differences. The warped result on the right side of the second row shows that the hair orientation aligns well with the facial orientation of the Source Image, indicating some robustness in the model across different poses. However, the right side of the second row and the third row still show misalignment in hair parting, leading to color inconsistencies in the output. The left side of the fourth row shows the model discarding the hair ends while supplementing the bangs area of the Source Image, though there is still some deviation in the hair color's centerline. The last row depicts complex texture scenarios where the Output hair color does not match the Target Image, due to the compression of images required for diffusion input and the bilateral filtering operation removing high-frequency color details. For cases with missing target region hair color, post-processing with patch match can fill in the blank areas, as shown in Figure 7 . Negative Impact. Hair editing tasks can also carry some negative implications and potential risks: 1) Social Bias: Similar to image inpainting models, hair editing models rely on internet-collected training data, which may contain social biases. Certain hairstyles could be incorrectly associated with specific races or genders, reinforcing existing social stereotypes. 2) Privacy Concerns: Hair editing involves modifying personal photos, which, if done without consent, can infringe on individual privacy. 3) Misleading Information: The technology can be used to alter photos misleadingly, potentially spreading false information. This can be used maliciously to modify images of public figures or ordinary people, leading to misinformation or cyberbullying. 4) Aesthetic and Identity Issues: Hairstyles are an important part of personal identity. Altering someone's hairstyle in a photo without their consent can negatively affect their self-image and identity. To address these concerns, it is crucial to emphasize responsible use and establish ethical guidelines when utilizing hair editing technology. This is also a key focus for our future model releases.Figure 14: The result of the warping module, ensures alignment of hair color with the target hair region, demonstrating its capability in hair color manipulation and preservation. To demonstrate the direct effect of the warping model, this shows the results without bilateral filtering processing. In cases where the hairstyles differ significantly, some areas may appear empty.Positive Impact. Hair editing tasks also offer several positive impacts: 1) Creative Expression: Hair editing allows users to experiment with different hairstyles and colors, promoting creativity and self-expression. It enables people to visualize new looks before making real-life changes. 2) Fashion and Beauty Industry: Professionals in the fashion and beauty industry can use hair editing tools for styling consultations, marketing campaigns, and virtual makeovers, enhancing customer engagement and satisfaction. 3) Accessibility: Hair editing technology can help individuals who may not have access to professional styling services to explore and enjoy different hairstyles virtually.We perform the inference of different diffusion-based methods in the same setting unless specifically clarified, i.e., on NVIDIA A40 following their opensource code with a base model of Stabe Diffusion v1.5 in 50 steps, with a guidance scale of 7.5. We use an Adam optimizer with a batch size of 16. The initialized learning rate for the generator is set to 0.0002 and for the discriminator is also set to 0.0002.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: The research conducted in the paper adheres to the NeurIPS Code of Ethics, ensuring ethical standards in all aspects of the research process, including data collection, experimentation, and reporting of results.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The submission received diverse scores, two positive and two negative. The authors provided detailed rebuttal and discussions with the reviewers. After carefully reading all comments, the AC found only one major concern is remaining: the lacking of novelty. After taking a look at the paper and the AC think the proposed method, though not that novel, is indeed a new thing and it really contributed new performance on hair editing. Given one negative reviewer did not give final responses on the authors' rebuttal and another negative reviewer said he will not argue for acceptance. The AC finally recommended an acceptance.

Reviews: review: paper_summary: paper_summary: This paper presents a novel approach called HairDiffusion for editing hair in images using latent diffusion models. The main contributions of the work are:

1. Introduction of the Multi-stage Hairstyle Blend (MHB) method for effectively separating control over hair color and hairstyle in the latent space of the diffusion model. MHB divides the diffusion process into two stages, allowing for precise guidance of hair color generation and context-aware generation of the rest of the image.

2. Development of a warping module to align hair color with the target area. This module adapts the HR-VITON architecture, using DensePose and segmentation maps to account for facial poses.

3. Utilization of hair-agnostic masks to transform the hair editing task into an inpainting task. The authors developed two types of masks for different editing stages, which effectively preserve necessary information and remove unnecessary information.

4. Fine-tuning of the CLIP model on a dataset with multi-colored hairstyles to improve editing of complex hair color structures. The authors applied data augmentation to increase pattern diversity.

The authors demonstrate that their method outperforms existing approaches in hairstyle and hair color editing tasks, especially for complex multi-colored styles. The method allows for editing hairstyle and hair color separately or together, using textual descriptions or reference images. Experiments show that HairDiffusion better preserves relevant image attributes (e.g., facial features, background) compared to existing methods.

The integration of these components into a single pipeline enables HairDiffusion to work effectively with both simple and complex multi-colored hairstyles while preserving other image attributes.
strength_weakness: strength_weakness: The paper exhibits several weaknesses. Firstly, it employs confusing notations, some of which are not properly introduced in the text. Secondly, the ablation study lacks clarity and sufficient detail. There is also a notable absence of comprehensive information regarding the method's limitations. For instance, the paper fails to provide examples demonstrating how the style transfer functions when transferring from long to short hair, how it handles cases with significant pose differences, or how it manages complex textures and hairstyles. Moreover, there is insufficient information about the limitations of the warping module. All presented images showcasing this module display identical hair shapes, suggesting that the warping module's functionality is not truly tested. This makes it impossible to accurately evaluate the quality of its performance.

Furthermore, the paper lacks sufficient information for its reproduction, as it wasn't explained how exactly and on what data the CLIP model was fine-tuned. In general, there is very little information throughout the work about additional datasets, where and how the data was scraped, and about hyperparameters.

The user study is very poorly described; there is no information anywhere about what exactly the questions looked like and in which domains these experiments were conducted. There are also questions about the statistical significance of the results.

The scientific novelty of this work is questionable. As demonstrated in Figure 3, the Control-SD method produces hairstyles that are virtually identical to those generated by HairDiffusion. This similarity suggests that the main contribution of the work is specifically in the transfer of hair color rather than the creation of the hairstyle. The color transfer component primarily consists of two stage: the warping module (which is based on HR-VITON, published in 2022) and the Multi-stage Hairstyle Blend (MHB) method (which utilizes ControlNet and simple blending techniques). Given that these core components are largely derived from existing methods, the overall originality of the approach is limited.

Overall, the paper suffers from a lack of comprehensive evaluation. It presents a limited range of visual comparisons and metrics, which hinders a comprehensive evaluation of the method's performance. Specifically, it lacks crucial metrics such as realism after editing (FID), realism with pose differences (FID), and full-image reconstruction metrics (PSNR, SSIM, LPIPS, FID, RMSE) as used in [1]. Moreover, the authors have not included comparisons with recent relevant works like StyleGANSalon [1] and HairFastGAN [2]. This omission of key comparisons and metrics significantly limits the reader's ability to fully assess the performance of the proposed method in relation to the current state-of-the-art approaches.

[1] Sasikarn Khwanmuang, Pakkapon Phongthawee, Patsorn Sangkloy, Supasorn Suwajanakorn. StyleGAN Salon: Multi-View Latent Optimization for Pose-Invariant Hairstyle Transfer. arXiv preprint arXiv:2304.02744, 2023.

[2] Maxim Nikolaev, Mikhail Kuznetsov, Dmitry Vetrov, Aibek Alanov. HairFastGAN: Realistic and Robust Hair Transfer with a Fast Encoder-Based Approach. arXiv preprint arXiv:2404.01094, 2024.
questions: questions: 1. As a strong point of your method, you indicate excellent quality in preserving facial details, but you compare it with HairCLIP v2, which works in a relatively low-dimensional FS space in StyleGAN. Could you provide more examples of facial detail preservation on very complex images in comparison with methods like StyleGANSalon and HairFastGAN?

2. Could you provide more examples of the limitations of your method? We would like to see how the method works with large pose differences, complex textures, and very different hair shapes in the image domain. With these images, we would like to understand how well the concept preservation works in the CLIP space, how well the warp module works, and how well the entire method functions overall.

3. Could you provide information on how you collected the additional dataset with hair colors?
limitations: limitations: The paper does not adequately address the limitations of the proposed method. The authors dedicate only two brief paragraphs to limitations, and these are relegated to the supplementary materials. This treatment is insufficient for a comprehensive understanding of the method's constraints. A more appropriate approach would be to include a detailed discussion of limitations in the main body of the paper. Furthermore, the authors should present visual examples demonstrating scenarios where each module, as well as the overall method, underperforms. These examples should be accompanied by in-depth analysis to provide insights into the reasons for these limitations and potential avenues for future improvements. Such transparency would significantly enhance the paper's scientific value and reproducibility.
paper_summary: paper_summary: The paper introduces an approach for hair editing using Latent Diffusion Models (LDMs). A warping module ensures precise alignment of the target hair mask and enables hair color structure editing using reference images. The proposed Multi-Hair Basis (MHB) method within LDMs decouples hair color and hairstyle. The authors showcase the performance of their method through GAN-based methods via qualitative and quantitative evaluations,
strength_weakness: strength_weakness: 1) Most of the examples shown in the results are front-facing subjects in the case of the hairstyle transfer. In real-world applications, there are cases where the reference and source images can have diverse/different poses. The lack of such results raises the question if this is a limitation of the method. Such problems are addressed in the paper HairNet[1]. 

2) A limitation of some GAN-based approaches compared in the paper is that to achieve high-fidelity reconstruction, the images are overfitted into the networks. As such most of these methods do not support further edits, for example shortening the transferred hair, making it wavy or curly or changing the pose of the subject to view the subject from a different angle. The current method does not seem to solve this problem either. What are the advantages of these GAN frameworks? Some of these methods support further editing as well. (check video associated with HairNet). https://www.youtube.com/watch?v=WBB43cgCFZM&t=153s

3) Some of the GAN based approaches can also control the degree of hairstyle transfer (Barbershop). Does this method control this efficiently?

[1] Zhu, Peihao, et al. "Hairnet: Hairstyle transfer with pose changes." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.
questions: questions: 1) Most examples in the results are front-facing subjects. How does the method perform when the reference and source images have diverse or significantly different poses?

2) GAN-based approaches often overfit images for high-fidelity reconstruction, limiting further edits. How does the current method handle this issue?

3) Can your method support additional edits post-hairstyle transfer, such as shortening hair, making it wavy or curly, or changing the subject's pose to view from different angles?

4) What specific advantages does the current method offer over existing GAN frameworks, especially regarding editability and handling diverse poses?

Also, check the weakness section.
limitations: limitations: The authors have discussed the limitations.
paper_summary: paper_summary: This paper presents a new framework for hair editing tasks, which includes editing hair color and hairstyle using text descriptions, reference images, and stroke maps. The proposed approach leverages Latent Diffusion Models (LDMs) and introduces the Multi-stage Hairstyle Blend (MHB) technique to effectively separate the control of hair color and hairstyle. Additionally, the method incorporates a warping module to align hair color with the target region, enhancing multi-color hairstyle editing. The approach is evaluated through extensive experiments and user studies, demonstrating its superiority in editing multi-color hairstyles while preserving facial attributes.
strength_weakness: strength_weakness: 1. The overall pipeline primarily consists of two stages:
   Altering the Hairstyle: This involves using a combination of control net and diffusion models with a hair-agnostic mask, 2D body pose keypoints, prompts, and a reference image as conditions. These modules are commonly found in previous methods.
     Editing the Hair Color: This stage blends information between the hair mask and source image, along with the Canny image of the stylized image and a prompt, to achieve the final result. The main contribution lies in the Multi-stage Hairstyle Blend (MHB) method, which warps the reference hair to the source image to better maintain the hairstyle. However, warping modules are also commonly used in previous methods [4, 21].
  The overall technical contribution does not meet the bar for this conference.

2. The hair structure of the source image is not well-maintained during color editing. As demonstrated in Fig. 5, fourth row, the strand direction of the hairline in the generated image is different from the input image. The results of HairClipv2 better preserve the curliness structure of the original input hair.  Additionally, the generated image in the third row of Fig. 5 shows noticeable brightness differences and a distinct color discrepancy compared to the input image.

3. The paper's writing lacks clarity. It would be beneficial to specify the difference between \( I_c \) and \( I_i \) in Fig. 2, within the caption of this figure. Additionally, using \( I_i \) to denote the style proxy creates confusion.
questions: questions: Please check Weaknesses.
limitations: limitations: NA
paper_summary: paper_summary: ### NeurIPS Review

#### Summary:
This submission presents an innovative 2D hairstyle editing pipeline leveraging latent diffusion models (LDM). The key component of this method is the Multi-Stage Hairstyle Blend (MHB), which facilitates separate control over hairstyle and hair color. By integrating structural information from non-hair regions such as facial masks and keypoints, the method effectively preserves non-hair attributes in the editing results. Additionally, a hair warping module enables natural transfer of hair color from the source image to the generated target image. Extensive experiments on public datasets like CelebA-HQ demonstrate improved results compared to previous state-of-the-art methods.

#### Strengths:


#### Weaknesses:
strength_weakness: strength_weakness: - More discussions on the failure cases: Most results are shown with a near-frontal head pose. Including and discussing more results with non-frontal head poses would enhance the completeness and understanding of the work. Additionally, as mentioned in the limitation section, the presented method might not work well with the color transfer between different hairstyles. It would also be great if more failure cases are included on that end.
- Manuscript Quality: There are some typos and missing text. A thorough proofreading is necessary to refine the manuscript.
questions: questions: See weakness.
limitations: limitations: Yes.

Comments: title: The discussion the challenges and limitations of the current method in hairstyle transfer, color alignment, and retaining facial details, while suggesting future improvements and comparisons with other approaches.
comment: Thank you very much for taking the time to review my work amidst your busy schedule. 

**Innovativeness and Effectiveness** : Our approach has achieved notable results in the novel task of multi-color hair transfer. Previously, no method applied the warp model to hair color transfer. We believe that if existing methods in the hairstyle domain had utilized the warp module, it would be necessary to compare our approach with them and demonstrate that our method surpasses them in metrics. However, as this is our first introduction of this module, we have validated the effectiveness of our method through the following:

- Ablation Study (pdf Table 1): We have conducted ablation studies to verify the effectiveness of each step and the integration of the warp module.
- Evaluation Metrics (pdf Table 2): As shown in Table 2 of the supplementary materials, our method, based on a general hair editing self-transfer approach, outperforms previous methods in the task of multi-color hair transfer. While our metrics in the task of swapping monochrome hairstyles are not the best, they are comparable to earlier methods. Considering our focus on the multi-color task, we believe this is justified.


**Preservation of Facial Details and Irrelevant Attributes:**  We have not overstated our effectiveness in preserving irrelevant attributes. In fact, in most cases, our method excels in maintaining facial details, accessories, clothing, and background. Although there are some changes in skin tone on the left side of Figure 4, the retention of facial details and the background is excellent. For example, in Figure 5 of the paper, despite changes in skin tone, the clothes of the child behind are preserved much better than in the text2img comparison method. Therefore, from the overall quantitative metrics and the preservation of irrelevant attributes, our method is superior to the comparison methods. We are willing to provide additional image examples to further demonstrate our preservation of details in the background, clothing, and other aspects.

I greatly value your opinions and hope to improve my research under your guidance.

Thank you for your assistance.
Thank you for the rebuttal and the detailed explanations provided. While I appreciate the clarifications, I still have concerns regarding the overall novelty of the work. Specifically, the points raised about 'Addressing the lack of paired datasets' and 'Preserving hairstyle structure' do not seem to significantly advance the state of the art.

Moreover, in your rebuttal, you mentioned that 'color discrepancy outside the inpainting region is a common challenge in diffusion-based inpainting tasks.' However, in Figure 4, the claim that 'Our approach shows better preservation of irrelevant attributes' appears somewhat overstated given the context. This discrepancy raises questions about the extent of improvement your method offers.

Given these concerns, I have decided to maintain my original rating. That said, I am open to reconsideration if other reviewers strongly advocate for the acceptance of this paper.
Due to the upcoming deadline for the discussion, I would like to confirm if you have any further questions or if there are areas where you need further clarification. If there are no more issues with my submission, I hope to receive your feedback and proceed.

Thank you very much for taking the time to review my work amidst your busy schedule. I greatly value your opinions and hope to improve my research under your guidance.

Thank you for your assistance, and I look forward to your reply.
Due to the upcoming deadline for the discussion, I would like to confirm if you have any further questions or if there are areas where you need further clarification. If there are no more issues with my submission, I hope to receive your feedback and proceed.

Thank you very much for taking the time to review my work amidst your busy schedule. I greatly value your opinions and hope to improve my research under your guidance.

Thank you for your assistance, and I look forward to your reply.
Due to the upcoming deadline for the discussion, I would like to confirm if you have any further questions or if there are areas where you need further clarification. If there are no more issues with my submission, I hope to receive your feedback and proceed.

Thank you very much for taking the time to review my work amidst your busy schedule. I greatly value your opinions and hope to improve my research under your guidance.

Thank you for your assistance, and I look forward to your reply.
Thank you for pointing this out.

We acknowledge that in our initial submission, we mistakenly selected the "NA" option for the IRB review requirement due to a misunderstanding of the scope of "research on human subjects." We apologize for this oversight. In the final version, we will correct this and clearly state that our user studies have been approved by the IRB.

Regarding our use of internet images, we based our approach on the methodology described in [1](ECCV 2024). We internally collected an In-the-Wild dataset from social media platforms such as Instagram to evaluate and train our model's performance in multi-color hairstyle transfer. These data are used solely for research purposes and have been approved by the IRB. Additionally, since our method focuses only on hairstyle information and does not require facial recognition, we have processed these images to remove any personally identifiable information. We confirmed that the data was obtained from public resources or with appropriate usage permissions. We will add a section in future versions detailing our specific methods for de-identifying sensitive information and ensuring copyright compliance.

We appreciate the feedback and will ensure that our data collection and processing methods align with ethical standards in future research.

[1] Choi Y, Kwak S, Lee K, et al. Improving diffusion models for virtual try-on[J]. arXiv preprint arXiv:2403.05139, 2024.
Thank you for your valuable feedback.

We followed a paper from [1](ECCV 2024) and internally collected an In-the-Wild dataset from social media platforms such as Instagram to evaluate the model's performance in multi-color hairstyle transfer. We ensure that these data are used solely for research purposes and have been approved by the IRB. Additionally, since our method does not require facial information but only hairstyle information, we processed these images to remove any personally identifiable information and confirmed that the data was obtained from public resources or with appropriate usage permissions.

We will add a section in future versions detailing our specific methods for de-identifying sensitive information.

**To address the potential unfair usage** of generating hairstyles for different groups, we propose the following solutions:
1. Diversity and Representative Datasets:
  When collecting datasets, including both testing and training stages, we strive to include samples from various races, genders, and age groups to minimize the risk of bias and discrimination.
2. Responsible Use and Ethical Guidelines:
 We emphasize the importance of responsible use and the establishment of clear ethical guidelines when utilizing these models, ensuring that their use aligns with ethical standards.
3. Output Detection:
  We have implemented a detector to identify and filter content that is not suitable for public viewing, such as explicit or violent images, thereby protecting the user's visual experience.
4. Model Generalization: We utilize the powerful generative capabilities of diffusion models to achieve generalization and diversity:
  - Generalization Capability: The diffusion model is designed to learn the universal patterns and features of various hairstyles, not just those of specific groups present in the training dataset. This generalization capability allows the model to recognize and generate hairstyles suitable for different individuals.
  - Generating Diversity: The diffusion model has the ability to generate novel and unseen data, which indicates that the model can not only replicate the hairstyles from the training data but also apply the learned features to generate hairstyles suitable for different groups. This diversity is achieved through the model's extensive coverage of data distribution during the generation process.

This aspect will be discussed in future versions.

[1] Choi Y, Kwak S, Lee K, et al. Improving diffusion models for virtual try-on[J]. arXiv preprint arXiv:2403.05139, 2024.
Thank you for your suggestions, which have been very helpful in improving our paper.
Discussion on the Performance Compared to StyleGAN-Based Methods
1. Reasons for Subpar Performance Compared to StyleGAN-Based Methods:
  - In cases where hairstyles differ significantly, our color alignment method, which is designed to generate colors more consistent with adjacent hues, may produce colors that deviate from the original hairstyle color. This discrepancy can lead to inconsistencies in the generated results.
  - The sparse matrix control provided by ControlNet's Canny map does not offer pixel-to-pixel control over hairstyle features. Alternatively, we could attempt to use a step-by-step approach that mixes multiple ControlNet models, allowing the model to focus on additional information.
  - When editing hairstyles based on reference images while retaining Diffusion text editing, setting the text to "hair" by default may introduce additional information that degrades the quality of the generated results. We could try separating the control of the text and ControlNet to balance the control over hairstyle structure and color in the future.
  We will include examples of these issues in future versions.
Impact on Facial Details
2. Transfers and Facial Details:
  - As illustrated in Figure 3, StyleGAN-based methods show suboptimal retention of accessories (G, M, O) and unique facial features (I) when the training dataset contains few or no samples of these details. The preservation of multi-color hair (N, Q) is compromised because HairFastGAN's approach of decoupling hair color and hairstyle in the feature space makes it challenging to maintain the color structure. StyleGAN Salon employs post-processing enhancements that improve results for multi-color hair to some extent, but retaining hairstyle details remains challenging. Hairstyle preservation (A, B, C, E, T, S, R, N) suffers from the lack of masks to retain irrelevant information. Additionally, maintaining the background (P) and hand preservation (F, K, L, W) in cases where the hand obscures the hairstyle is also difficult without optimization. StyleGAN-based methods struggle to preserve detail for accessories or hands, particularly when the training dataset contains few examples of such features.
  - Conversely, the ControlNet Canny map, which provides line-based control, shows better performance in preserving hairstyle flow and individual hair strands.
Discussion on Hairstyle Transfer Limitations
3. Limitations of Hairstyle Transfer:
  - Hairstyle transfer is not the primary focus of our work, but it is important to address the method's shortcomings. We use a straightforward approach of converting hairstyles into text vectors, which often fails to capture the complete details and local features of hairstyles, leading to noticeable discrepancies between the generated and original hairstyles. Recent studies suggest using encoders trained on face datasets to better capture facial details, which is a promising direction for future work. We will include additional figures and textual explanations to discuss these limitations in future versions.
  - We have also noted the paper on Stable-Hair, which differs from our approach by not decoupling hairstyle features, thereby preserving both hair color and overall hairstyle during the transfer, and not focusing on text-based editing of hairstyles. This paper has not yet released code or demos; we will conduct a comparison once these resources are available.
Thank you for your responses. However, there are still some concerns regarding the limitations of your method that were not fully addressed in the rebuttal.

Regarding your answer A1, you demonstrate an improved reconstruction by referencing Table 2 and Figure 3. While this shows that your method outperforms in reconstruction tasks, it was already evident that Stable Diffusion's reconstruction is superior to StyleGAN's. However, Table 2 indicates that your method underperforms significantly compared to StyleGAN-based methods in single color transfer. It would be beneficial to explain why this occurs and provide examples to illustrate this point. Additionally, the impact of such transfers on facial details remains unclear and should be elaborated upon.

Your answer A2 did not directly address my original question. I inquired about how your method performs when transferring hairstyles from the image domain and its general limitations. Instead, you focused on the limitations of your method when recoloring hair.

In summary, the current presentation of the work lacks transparency, and there are several issues that need to be addressed. If the final paper is permitted to incorporate these important responses from the rebuttal, I believe it would be suitable for publication. Additionally, given the recent publication of the Stable-Hair paper [1], it would be valuable to include a comparison with this work in the future version of your paper.

[1] Yuxuan Zhang, Qing Zhang, Yiren Song, Jiaming Liu, Stable-Hair: Real-World Hair Transfer via Diffusion Model, arXiv:2407.14078