Title: HairFastGAN: Realistic and Robust Hair Transfer with a Fast Encoder-Based Approach
Venue: NeurIPS 2024
Accepted: yes
Keywords: Generative Model

StyleGAN

HairSwap
Abstract: Our paper addresses the complex task of transferring a hairstyle from a reference image to an input photo for virtual hair try-on. This task is challenging due to the need to adapt to various photo poses, the sensitivity of hairstyles, and the lack of objective metrics. The current state of the art hairstyle transfer methods use an optimization process for different parts of the approach, making them inexcusably slow. At the same time, faster encoder-based models are of very low quality because they either operate in StyleGAN's W+ space or use other low-dimensional image generators. Additionally, both approaches have a problem with hairstyle transfer when the source pose is very different from the target pose, because they either don't consider the pose at all or deal with it inefficiently. In our paper, we present the HairFast model, which uniquely solves these problems and achieves high resolution, near real-time performance, and superior reconstruction compared to optimization problem-based methods. Our solution includes a new architecture operating in the FS latent space of StyleGAN, an enhanced inpainting approach, and improved encoders for better alignment, color transfer, and a new encoder for post-processing. The effectiveness of our approach is demonstrated on realism metrics after random hairstyle transfer and reconstruction when the original hairstyle is transferred. In the most difficult scenario of transferring both shape and color of a hairstyle from different images, our method performs in less than a second on the Nvidia V100.


Peer reviews:
comments: SUMMARY:
This paper proposes a novel framework for hairstyle transfer from single images. As previous works have either suffered from long optimization times or low generation quality, this work introduces a new encoder-based solution that balances both efficiency and quality. The solution decomposes the pipeline into four stages: pose alignment, shape alignment, color alignment, and refinement alignment, with a specialized encoder trained separately for each stage. Detailed experiments demonstrate that this approach outperforms previous methods both quantitatively and qualitatively.

STRENGTHS:
1. The results exhibit a high quality, both qualitatively and quantitatively. Compared to related works, HairFast generates results with a more consistent identity and natural hairstyle.
2. I particularly appreciate the decomposition of hair shape and color, which enhances the flexibility of this method. 
3. The experiments conducted are comprehensive, with several baseline methods compared, and the reported results are promising. 
4. The write-up is thorough, including all necessary details (as well as the source code) required to reproduce this work.

WEAKNESSES:
Overall, I enjoyed reading this paper and did not identify any significant weaknesses. The proposed method might be somewhat complex, but with the shared source code, reproduction should not pose a major issue. There are a few typos in the paper, but these should be easy for the authors to fix.

QUESTIONS:
N/A

LIMITATIONS:
Limitations and failure cases are well discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces HairFast, a model designed to tackle the task of transferring hairstyles from reference images to input photos for virtual hair try-on. This task is notably challenging due to the diverse poses in photos, hairstyle intricacies, and the absence of standardized metrics for evaluation. Existing state-of-the-art methods often rely on slow optimization processes or low-quality encoder-based models operating in StyleGAN's W+ space or using other low-dimensional generators. HairFast addresses these shortcomings by leveraging a new architecture operating in the FS latent space of StyleGAN, enhancing inpainting techniques, and incorporating improved encoders for better alignment and color transfer.\n\nSTRENGTHS:\n1) HairFast achieves near real-time performance, processing hairstyle transfers in less than a second.\n\n2) The model produces high-resolution results, maintaining quality while operating in the FS latent space of StyleGAN\n\n3) Unlike existing approaches, HairFast effectively addresses challenges related to pose variations and color transfer in hairstyle transfer tasks. \n\n4) The paper compares with the competing methods based on StyleGAN.\n\nWEAKNESSES:\n1)  For StyleGAN-based hairstyle editing methods, one prominent limitation is the further editability of the images beyond hair. Methods like Barbershop, HairNet etc overfit on some part of the face to limit further editability. It is difficult to assess if the properties of editability of the underlying GAN are preserved. Can the current method edit the length of the hair, hairstyle (wavy, curly), and pose of the face after the hairstyle transfer is performed?\n\n2) While the paper claims identity preservation, there is less evidence in terms of quantitative analysis. A score based on the Arcface model may help in providing a robust analysis of the method.\n\nQUESTIONS:\nCan your method edit hair length, hairstyle type (e.g., wavy, curly), and facial pose post-hairstyle transfer, while preserving the editability of other facial features without overfitting?\n\nHave you used ArcFace or similar models to quantitatively assess identity preservation? \n\nHow does your method ensure continued editability of facial features after hairstyle transfer, and what measures prevent degradation of initial edits?\n\nCan your method effectively handle varying attributes like hair length, waviness, and curliness? How does its flexibility compare with existing methods?\n\nLIMITATIONS:\nThe authors have discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces HairFast, a model that addresses the challenge of transferring hairstyles from a reference image to an input photo in near real-time with high resolution and superior reconstruction. Existing methods either suffer from slow optimization processes or low quality due to operating in low-dimensional spaces. HairFast utilizes a new architecture in the FS latent space of StyleGAN, enhanced inpainting, and improved encoders to efficiently handle pose differences and transfer hairstyle shapes and colors in less than a second.\n\nSTRENGTHS:\n+1. The HairFast model achieves both high resolution and near real-time performance, outperforming optimization-based methods that are typically slow. This enables virtual hair try-on experiences that are more fluid and responsive for users.\n\n+2. The model demonstrates superior reconstruction compared to optimization-based hairstyle transfer methods, indicating a higher level of accuracy and fidelity in transferring hairstyles from one image to another. This leads to more realistic virtual hair try-on results.\n\n+3.  The HairFast model uniquely addresses the issue of pose differences between the source and target images, which has been a challenge for existing hairstyle transfer methods. By including enhanced inpainting, improved encoders for better alignment and color transfer, and a post-processing encoder, the model can effectively transfer hairstyles even when poses are significantly different, enabling a wider range of virtual hair try-on scenarios.\n\nWEAKNESSES:\n-1. The HairFast model operates within the FS latent space of pre-trained generative models like StyleGAN. This reliance restricts its performance when StyleGAN fails to represent certain hairstyles or facial features adequately. Additionally, this dependency limits the model's ability to generalize to different generative models or datasets.\n\n-2. While the HairFast model excels at transferring common hairstyles, it may struggle with extremely complex or unconventional styles (e.g. highly individualized, extremely curly, or straight hair patterns). \n\n-3. Although the HairFast model achieves near-real-time performance on high-performance hardware like the Nvidia V100 GPU, its performance may suffer on devices with limited computational resources (e.g. mobile devices or low-end PCs).\n\nQUESTIONS:\n-1. Since \"Fast\" is a key highlight of this work, why is there a lack of in-depth computational complexity comparisons, such as FLOPs and parameter count?\n\n-2. How does the HairFast model compare to other state-of-the-art hairstyle transfer methods based on the diffusion model in terms of accuracy, speed, and user-friendliness?\n\n-3. In line 173, what is the objective basis for setting the value of \\alpha to 0.95?\n\n-4. How does the HairFast model handle hairstyles that are not well-represented in its training dataset?\n\nLIMITATIONS:\n-1. The HairFast model is likely trained on a limited set of similar datasets. As a result, its performance may degrade when applied to diverse datasets with a wider range of hairstyles, ethnicities, or facial features.\n\n-2. The HairFast model operates within the FS latent space of pre-trained generative models like StyleGAN. This reliance restricts its performance when StyleGAN fails to represent certain hairstyles or facial features adequately. Additionally, this dependency limits the model's ability to generalize to different generative models or datasets.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}