ACFun: Abstract-Concrete Fusion Facial Stylization

Authors: jiapeng  ji
kun  wei
ziqi  zhang
cheng  deng
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-06 03:33:03

Abstract: Owing to advancements in image synthesis techniques, stylization methodologies for large models have garnered remarkable outcomes. However, when it comes to processing facial images, the outcomes frequently fall short of expectations. Facial stylization is predominantly challenged by two significant hurdles. Firstly, obtaining a large dataset of high-quality stylized images is difficult. The scarcity and diversity of artistic styles make it impractical to compile comprehensive datasets for each style. Secondly, while many methods can transfer colors and strokes from style images, these elements alone cannot fully capture a specific style, which encompasses both concrete and abstract visual elements. Additionally, facial stylization often alters the visual features of the face, making it challenging to balance these changes with the need to retain facial information. To address these issues, we propose a novel method called ACFun, which uses only one style image and one facial image for facial stylization. ACFun comprises an Abstract Fusion Module (AFun) and a Concrete Fusion Module (CFun), which separately learn the abstract and concrete features of the style and face. We also design a Face and Style Imagery Alignment Loss to align the style image with the face image in the latent space. Finally, we generate styled facial images from noise directly to complete the facial stylization task. Experiments show that our method outperforms others in facial stylization, producing highly artistic and visually pleasing results.

Main paper: 
 introduction: 
 Style transfer is a long-standing research topic that aims to generate a new artistic image from an arbitrary input pair of natural images and painting images by combining the content of the natural image and the style of the painting image. With the rapid development of deep learning, optimizationbased methods [1] [2] [3] and feed-forward based methods [4, 2, 5, 6] continually emerged. These methods have made progress in visual quality and computational efficiency. However, these style transfer techniques can only modify low-level image features such as strokes and colors, and they are unable to adjust high-level semantics such as shape and content. It's like using the style image as a template to paint the content image instead of truly creating a new image based on the style. Style encompasses not only basic visual elements like strokes and colors but also deeper aspects such as structure, layout, and shape. These elements combine to create a unique visual language and form of expression. In the context of facial stylization, the challenges of style transfer are even more significant. Facial recognition relies on distinct and recognizable individual characteristics, which means that style transfer must capture the visual features of the style image and seamlessly integrate them into the target image while preserving facial recognizability.With the rapid development of diffusion model-based image generation methods, several style transfer methods [7] [8] [9] have emerged, but these methods usually require a set of images belonging to the same Figure 1 : As shown in the figure, the ideal result of facial stylization aims to ensure the faithful restoration of facial information while also significantly deforming the facial image according to the given style image. To achieve high-quality results, it is essential to ensure that the production process not only meets the standards for good generation quality but also encompasses a compelling and evocative artistic atmosphere. style for training. However, due to the scarcity of high-quality stylized images, it is difficult to obtain them. Moreover, the diversity and complexity of artistic styles make compiling a comprehensive dataset for each style impractical. Subsequently, there are also some Textual-Inversion methods [10, 11] that can use a single image, but this method only fine-tuned the CLIP model as the conditional input, which limited its ability to generate models and often caused it to be heavily affected by style degradation, resulting in simpler results and reduced image quality. Moreover, its style transfer method of adding noise and denoising to the target image reduces the flexibility of the generated results and further limits its final generation performance.As depicted in Figure 1 , the most prominent challenge in facial stylization tasks is that images with artistic styles often display significant differences from real-world facial photos. These differences are evident not only in visual features but also in high-level semantics, such as shape and distribution. In particular, as shown in the lower part of Figure 1 , there are notable disparities between abstract style portraits and real portraits, leading to substantial changes in facial features during the stylization process. On the other side of the coin, it is essential for stylized facial images to maintain consistency in facial information so that they remain recognizable to people. In summary, for stylized facial images, balancing the ability to maintain facial images with the drastic semantic changes brought about by stylization is the key to achieving good facial stylization results.To this end, we propose a facial stylization method that allows deep neural networks to fuse abstract visual elements (deformation, distribution, atmosphere, etc.) from style and facial image with concrete visual elements (color, brushstrokes, lines, etc.) into an imagery visual feature just like artists, to achieve facial image stylization, rather than simply transferring texture and color information. Specifically, we introduced an Abstract-Concrete Fusion Facial Stylization Model (ACFun) to fuse, consisting of an Abstract Fusion Module(AFun) and a Concrete Fusion Module (CFun). The method ACFun uses for facial stylization involves learning abstract and concrete features separately and then combining style images with facial images by aligning them in an imagery feature space formed by the fusion of abstract and concrete features. We first extract abstract features of style and face through AFun by encoding style images into the CLIP latent space and combining it with the facial description text prompts we use; then, we optimize it using a Textual-Inversion-based method to make it recognizable by CLIP for subsequent generation processes. In addition, by introducing face images during the training process, we implicitly fuse facial information with style abstract information. However, CLIP has limitations in distinguishing between specific style and facial images at the fine-grained level. To address this, we adopt the CFun module, which is a set of trainable parameters embedded in a pre-training generator to enhance its ability to extract concrete features from specific styles and facial images, which greatly improves the quality of the generated results. Finally, the above process is constrained by the Face and Style Imagery Alignment loss we proposed. We demonstrate the effectiveness and excellent facial stylization effect of our method through massive experiments. 
related works: 
 Text-to-Image Generation. The method based on generating adversarial networks [12] has been widely applied in text-to-image generation by training on paired images and text samples. [13] improves condition generation [14] by changing the input conditions from a single class to more complex text descriptions. However, due to the difficulty of training GAN models and the significant progress of large language models [15, 16] and diffusion models, the diffusion models have gradually replaced the position of GAN. The diffusion model [17, 18] can generate realistic images through multiple rounds of iterative refinement. It has been widely used in the image generation task and has achieved significant results [19] , attracting the attention of many researchers. Among all these diffusion-based methods, Stable Diffusion [20] is the most typical work. It achieves awe-inspiring results through training on large-scale text image datasets. Afterward, SDXL [21] was introduced with larger model parameters and larger datasets. However, these methods all focus on learning text prompts, meaning that the model can only fit the given reference style through existing knowledge rather than train the generated model to have the ability to generate specific style images. We fine-tune the model to make it more intuitive to learn the target style straightly. Thanks to these powerful pre-trained generative models, we can flexibly apply them to various downstream task applications.Personalization of Generative Models. How to use a pre-trained text-to-image large model to complete personalized image synthesis tasks has attracted widespread attention [22] . The widely used method in image generation tasks is the inversion-based method, which aims to find a corresponding noise or conditioning embedding to a generated image. Textual Inversion [9] and Hard prompt made easy (PEZ) [23] convert a set of images of an object into corresponding textual representations (such as embedding, token) instead of changing the parameters of the text-to-image model. Dreambooth [7] learns target concepts by fine-tuning the pre-trained model. At the same time, when learning new class concepts, Dreambooth transforms this concept into a special sub-concept of a specific class. Dreambooth achieves the goal of data augmentation by generating original classes, preventing new concept classes from overwriting existing original classes, leading to catastrophic forgetting and affecting the diversity of generated models, leading to overfitting. Lora [8] has also achieved widespread application by transferring the strategy of fine-tuning pre-trained LLM to image generation methods [24] , allowing it to save computational and storage costs without the need to train the entire generation network. Although the above methods have achieved good results, they all learn the description of specific concepts through a set of images, which makes it difficult to meet our needs. The method we adopt only requires a pair of images to learn the abstract and concrete features of style images and facial images and ultimately generate a stylized facial image.Facial Stylization. Facial stylization is currently a widely recognized task in the field of style conversion. Typical style transfer methods such as [25, 26, 4, 6, 27] can effectively transfer the concrete visual features such as texture and color contained in the given style image. However, when facing facial images, especially when transferring real facial images to styles with strong geometric distortions (such as comics, anime, abstract paintings, etc.), the effect is often poor. Although some methods [28, 29] use large-scale paired image data for image-to-image translation training, in practical applications, due to the diversity and scarcity of high-quality style images, collecting such a large number of paired images is not realistic, and some images do not even exist, which makes the above methods difficult to apply in practice. StyleGAN [30] achieves facial stylization through image inversion [31, 29, [32] [33] [34] , but due to the presence of its latent space, it suffers from problems such as image degradation and limited deformation. Although the method of textual inversion [10, 11] can form certain deformations, the problem of style degradation cannot be ignored. Our method solves the above problem by learning abstract and concrete features separately and aligning images. Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss the limitations in Section 6.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
overview: 
 In this paper, we propose a novel method named ACFun for image style transfer; the structure and process are shown in Figure 2 . Facial style transfer aims to preserve facial identity information while being as close to the target style as possible. Inspired by different artistic styles of expression, we divide visual features into more complex and difficult to describe abstract features, such as shape or atmosphere, and more specific concrete features, such as color or strokes. And ultimately, by combining abstract features and concrete features into imagery features and utilizing them to generate stylized facial images.Therefore, we first propose an AFun module. By using a CLIP encoder to encode the style image and combine it with a facial description text prompt using the attention mechanism to fully use the information of the reference style image. Then, we optimize it into an embedding e * that can be recognized by the generated model in the subsequent generation process. However, simply using the aforementioned Inversion-based method often leads to style degradation, resulting in the loss of some concrete visual content in the final result, which is often fatal for facial stylization tasks. So, both facial and style images are introduced simultaneously during the training process. To this end, we adopt the CFun module, to improve the flexibility of the network and enable it to have contract generation capabilities, we insert a set of trainable parameters into a pre-trained U-net and adopt the Face and Style Imagery Alignment loss to constrain the above process.To fully utilize the pre-trained large-scale model, we hope to learn the style in the target style image as a specific intermediate representation, which is suitable for the pre-trained model. This method uses CLIP as a text encoder for text-to-image generation in stable diffusion. It inputs the text embedding, which is embedded by the CLIP text encoder τ t as a condition. We use the pseudo-word 'S * ' as a placeholder for the image style and introduce the concept of specific image styles into the pre-trained model through a learnable corresponding embedding e * t . However, using only a single pseudo-word S * as a placeholder, this simple and vague text prompt inevitably leads to us only being able to extract the content of the entire style image, regardless of whether it is an abstract style feature or a specific content feature. This will bring problems due to the content of the style image in our subsequent Figure 3 : Left: The specific insertion method and structure of our CFun. Right: Our proposed imagery latent space, where the red line represents the VQ visual space, the green line represents the imagery latent space, and the blue line represents the process of mapping the original image from the VQ space into the imagery latent space through the introduction of abstract features e a . facial stylization process, such as style leakage or style coverage. To address this issue, we used facial description text prompts S * f . The style and content of the style image can be automatically separated by the description of the facial image, such as "A painting of a woman in S * style." This is because in the subsequent training process, style images I s and facial images I f are introduced, and CLIP naturally implicitly receives information from facial images I f during the feature update process. We embed the style image I s using CLIP image encoder τ i as embedding e * is . Then, we fuse it with facial description text prompts embedding e * t f to obtain abstract features e a . This process can be expressed as:EQUATIONwhere ϵ stands for an initial noise map ϵ ∼ N (0, 1), εθ is the denoising network, t = 1 . . . T stands for the time step for the diffusion model. Since our method only requires a pair of images to complete training, it is difficult to thoroughly learn the style in the background image using a vanilla textual inversion method, and overfitting is prone to occur. To alleviate this problem, we also introduced image information, allowing the model to learn the style and facial abstract visual elements in the image more fully. Benefiting from the CLIP model that has already aligned text with images in its latent space as a multi-modal model, the fusion process of style image embedding e * is with facial description text prompts embedding e * t f using an attention mechanism can be represented as:EQUATIONEQUATIONand set dropout in the attention mechanism to alleviate overfitting problems.Even if we have obtained an abstract feature e a that blends style and facial information, relying solely on this feature to generate images means that we can only fit the target style and facial image using the prior knowledge already present in the pre-trained model. However, in real life, there are numerous style images and facial images, many unseen by pre-trained models, making it challenging to generate the desired results. To solve the above problem, we propose the CFun module for generating concrete visual features. By inserting a set of learnable parameters into U-net, we enhance the flexibility of the network and enable it to learn the visual features of input images. For its insertion position, considering that we do not want to damage the style and information in the facial image excessively. It is too risky to directly insert parameters into the Attention Block for training, as this may involve overly advanced semantics. Directly changing the Attention Block may result in too drastic advanced semantic changes, ultimately leading to excessively distorted stylized face images. So, we choose to insert the parameters into Resblock, which can make our parameters focus more on specific concrete visual features and indirectly affect the Attention module, thus achieving a balance between the information retention ability of facial images and the high-level semantic changeability brought by stylization. In addition, inspired by AdaIN [4] , our insertion module adopts an affine mechanism, which can ensure that the network parameters are not affected during initial training, further improving the stability during training. The specific structure and insertion method are shown on the left of Figure 3 . We separate content from the style by calculating mean-standard:EQUATIONAnd only the style information, i.e., standard σ(x), is fused with the network. This is mainly because the abstract features we use are only fused from style images and facial description text prompts at the beginning of training, and no facial image information has been obtained yet. To avoid excessive activation of abstract features and style images in cross-attention during the training process, causing the generated stylized facial images to irreversibly guide to style images, resulting in the final generation result being strongly covered by style.Finally, we use a Face and Style Imagery Alignment loss to constrain all the above processes. However, as shown on the right of Figure 3 , even if encoding using VQ space, the interpolated images represented by simple style images and facial images are meaningless, just a simple superposition of the two images. However, initially, we had already used AFun to fuse style images with facial description text prompts and provided an abstract feature e a for subsequent training processes. This allowed us to have abstract feature e a guidance in the subsequent optimization process. After combining the abstract feature with initial noise, the space it projects onto is not simply a VQ latent space composed of simple image features but a latent space composed of semantic features after the combination, we call this latent space, which combines abstraction and concreteness, the imagery space. This also makes our proposed Face and Style Imagery Alignment loss effective. Specifically, by aligning I s and I f in the imagery space, we obtain an image I i that integrates abstract and concrete features of facial image I f and style image I s to achieve the goal of facial stylization. We also use the two hyperparameters we establish, β and γ, to control the tendency of the model towards the relationship between facial image content and style during the optimization process. Finally, the pre-trained text-to-image diffusion model DM is used, and the initial noise map ϵ ∼ N (0, 1) and the abstract feature e * a obtained in the previous step are given as inputs to generate the image Îi = DM (ϵ, e * a ), and the diffusion model is fine-tuned using the squared error loss. The specific formula is as follows: min Îi E Ii,ϵ,t,e * a [βω t ∥DM (α t I i + σ t ϵ, e * a ) -I s ∥ 2Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Justification: The paper does not include theoretical results. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.• Theorems and Lemmas that the proof relies upon should be properly referenced. 
implementation details: 
 We trained on a single Nvidia A6000 graphics card, and in the case of a single pair of images, we set the batch size to 1. Each epoch took about 30 seconds, and the overall training process is 5 epochs, which takes only 3 minutes. We set the base learning rate to 1.0e -04, and the remaining hyperparameters are consistent with Stable Diffusion without changing. Through 40 steps of diffusion, our method can obtain stylized facial images with good results. We set the hyperparameters γ and β to 0.8 and 1.0, respectively, and all subsequent experiments will use this hyperparameter setting method.We compare our method with the state-of-the-arts image style transfer methods, including InstantStyle [35] , DreamStyler [10] , Inst [11] , AesPA [36] , StyTr2 [37] . As shown in Figure 4 , the generation results fully demonstrate the excellent performance of our method. Compared with the traditional two-column method on the far right, it can be seen that our method can modify the target image according to the high-level visual elements of the reference style image instead of traditional methods only focusing on low-level visual elements such as color or strokes. Compared to the DreamStyler and Inst inversion-based methods, it can be observed that both methods suffer from serious style degradation, which makes the stylized facial images they generate more sloppy and tend to generate simpler images. InstantStyle pursues fast generation due to its Tuning-Free mode. However, due to its use of ControlNets [38] to control the final generated face, the model only pieces together the elements in the style image according to the guidance by ControlNets rather than truly understanding the face. At the same time, due to the lack of new knowledge introduction, the generated results are not ideal. When using SDXL [21] , the generated results also show some competitiveness. However, when using a smaller network like SD1.5, it can be seen that due to the insufficient prior knowledge to support its generation of input images of any style, the results are quite struggling and even generate a lot of noise in some images. On the contrary, although our method only uses SD1.4, a more basic pre-trained generative network, as our backbone model, the injection of new knowledge makes our generation results more effortless. It can be seen that our results faithfully restore facial image information while being close to the reference style image, displaying a highly artistic facial stylization result. We show more quality results and quantitative results in the supplement material.In addition to focusing on facial stylization generation, our method also has the ability to generate by text guidance since the Stable Diffusion we take as our backbone is originally a text-to-image generation mode. We compare our method with InstantStyle [35] as shown in Figure 5 . It can be seen that, our method achieves better visual effects compared to Instantstyle. While ensuring that the style and facial features remain unchanged, we generated corresponding results based on the given text input and displayed good image quality. As Instantstyle is a method of tuning for free, it only relies on changing the attention parameter and subsequently using controllnet to control the final generation result. This also means that the Instantstyle method does not require additional knowledge injection, which makes it lack understanding of the given style and face. On the other hand, the presence of ControlNets can also interfere with the generation results of text participation guidance to some extent. As shown in the bottom line, Instantstyle hardly responds to the first four text inputs. Thanks to the injection of additional knowledge and the high compatibility between directly generating images from noise and text-to-image generation, our method can generate high-quality images that combine style, facial information, and fit text descriptions.We conducted ablation experiments on different modules to demonstrate how we extract the required features and how they work. As shown in Figure 6 , we first do not use any CFun modules but only use abstract features from Afun for generation. It can be seen that the generated result is only a blurry image without specific facial information. However, at an abstract level, it can be seen whether it is male or female, and it also provides the colors required for the final generated image. By introducing Cfun into the encoder, it can be seen that the specific features of CFun provide low-level visual information, such as stroke texture. In the decoder, CFun can see that the main concrete features are the structure and posture of the face. The results of the ablation experiment indicate that the feature extraction mechanism is consistent with what we described in the methods section, proving the effectiveness of our method and the rationality of the proposed method. We show more results in our supplement material.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes]Justification: We have provided a detailed introduction to the implementation details of our experiment in Section 4.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Justification: The paper provides comprehensive details of the training and testing settings in Section 4, including data splits, chosen hyperparameters, and the type of optimizer used. This ensures that the experimental results are fully understandable and reproducible.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 We propose a new facial stylization method called ACFun, which can achieve facial image stylization using only one image pair. To adapt to the task of facial stylization, we separate the learning of abstract features and concrete features through AFun and CFun and constrain the above process through a carefully designed Face and Style Imagery Alignment loss. We can transform any facial image into a specific style through short-term training. Numerous experiments have shown that our method exhibits excellent facial stylization and text-guided image generation results compared to state-of-the-art methods. A large number of visual results demonstrate that our method not only has good facial preservation ability but also generates stylized facial images with a strong artistic atmosphere. Meanwhile, the results of ablation research have demonstrated the effectiveness of our proposed method and demonstrated that our proposed AFun module and CFun module indeed learn abstract and concrete features separately. Our method provides excellent convenience for facial stylization generation. 
limitation: 
 Even if we use separate learning methods, style is always difficult to describe, which often leads to some low-level or advanced visual features being deeply bound to certain content, resulting in inevitable style leakage problems. A typical example is that stylized facial images will change with the posture in the style image. Furthermore, due to the ambiguity of the style, our method is still unable to accurately extract a few visual elements of the style we want to exist in the style image. Moreover, our method relies on the effectiveness of the pre-trained model. We found that the training data used in the pre-trained model is imbalanced, especially when facing certain specific character images or style images. This can lead to serious semantic binding phenomena, making it difficult to stylize or causing obvious style leakage, which can also impact the performance of our method. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Error bars are not reported as previous attribute and affordance benchmarks.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: There is no societal impact of the work performed. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: In Section 3 we cite including Stable Diffusion [20] Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We also conducted quantitative experiments based on user study, shown in Table 1 , providing 40 pairs of style facial images and their generated results, and collected voting results from 50 volunteers. The voting was conducted from three aspects: style consistency, facial consistency, and overall which image users prefer. The results are shown in the table. It can be seen that our method can achieve good results in style consistency and outstanding performance in facial consistency and overall evaluation.In addition, we also compared our method separately with Instantstyle SDXL, which is shown in Table 2 , and it can be seen that users favor our approach. In this section, we conduct a comparative analysis of the prompts within the AFun module discussed in Section 3.2, the two experimental implements within the CFun module detailed in Section 3.3, and the hyperparameters of the Face and Style Imagery Alignment Loss as outlined in Section 3.4.It can be seen in Figure 7 that the facial stylization results using facial description text prompts show better stability and generation quality, which also proves that facial description text prompts improve the accuracy of style and facial learning. Meanwhile, since facial description texts contain more information, we can obtain a better initial abstract feature, which reduces the difficulty of subsequent training. In addition, facial stylization results using only a single pseudo-word will always be affected by artifacts, especially in results with stronger styles. This further demonstrates the effectiveness of our facial description text prompts.We also set different γ and β parameters, which are 0.6:1, 0.8:1, and 1:1. It is obviously shown in Figure 7 that as a continues to increase, the style information contained in its facial stylization results will become more dense. At 0.6:1, the style information is almost completely ineffective, and its stylization ability is extremely limited, only staying at color changes or facial shifts. When set to 1:1, dense style information is introduced into the facial image, and even style leakage may occur, such as turning hair into green. This also proves the effectiveness of our proposed Imagery Alignment Loss and the existence of our proposed Imagery Latent Space. We can indeed generate stylized facial images by finding a point in the Imagery Latent Space that lies between the style image and the facial image. We mentioned our strategy of selecting the insertion parameter position n section 3.3. The experimental results in Figure 8 indicate that inserting training parameters into attention can lead to serious style leakage, resulting in overfitting the given style image. This is because the initial abstract features e a we provide are obtained by fusing the CLIP encoding e is of the style image I s with the CLIP encoding e t f of the facial description text prompt t f . This means that the initial abstract features e a do not contain facial information. If we use the method of inserting parameters into attention, the style information in the abstract features e a will be activated before the information in the facial image, leading to serious style leakage and overfitting. The use of Mean becomes even more complex, as shown in the figure. We found that the reverse direction of overfitting in the final result is uncertain, possibly due to the more complex information contained in the Mean and the higher degree of mixing of facial and style information. However, regardless, the excessive content information contained in Mean is not conducive to our facial stylization. By only using Standard, we can avoid this trouble and obtain more stable facial stylization results.In this section, we conduct an interesting experiment on splicing facial images and present an array of stylized experimental results, all of which demonstrate commendable efficacy.We have designed an extremely interesting task to demonstrate the superiority of our proposed method and why we believe that separating abstract and concrete features for learning and finally mixing them into imagery features can effectively complete the task of facial stylization. We use this image to stylize faces by concatenating different parts of different faces into a single face. As a comparison, InstantStyle, due to the presence of ControlNet, directly generates a fragmented stylized facial image based on the cropped image, indicating that it does not understand what a face is. Our images can generate complete and meaningful stylized facial images while ensuring the features contained in each facial feature as much as possible and harmoniously integrating them into one face. This is all due to our approach of abstract, concrete, and imagery features, which makes our model adopt a more knowledge-driven approach rather than a simple feature transfer facial stylization method. This also indicates that even if the given image is fragmented, it does not prevent our method from finding a meaningful feature corresponding to it in the imagery latent space, which proves that our method has a strong abstraction ability.We conducted experiments on more faces and styles to demonstrate the stability and generalization of our method, and we compared it with InstantStyle-SDXL, which is the best-performing InstantStyle method for comparison. in Figure 10 and Figure 11 . Even though we use SD1.4, which has a smaller CLIP and training on a smaller dataset than SDXL, our method appears more adept at facial stylization tasks due to the injection of additional knowledge. It can be seen that the stylized facial images generated by our method have a high artistic level, just like the creation of an artist. Of Is the Pre-trained Large Model a Natural Brain-like Mechanism? The fundamental idea of artificial intelligence is to simulate the operation of the human brain through artificial neurons. From a neurological perspective, human vision is not simply a peripheral sensory experience but a product of the coordination between the brain and the senses. So, the perceptual process of human "seeing" is not a simple bottom-up information transmission process but rather a top-down information construction process. The processing of facial information and style information by the human brain is unique. This also means that the action of "seeing" is not simply a visual drive but rather a knowledge-driven action.The human visual mechanism is quite complex, especially when facing faces and artistic-style images. For faces, humans not only receive bioelectric signals from edge sensors but also activate specific brain regions and use more advanced knowledge from the hippocampus to recognize faces jointly. And the same goes for style. As a visual art experience, neuroaesthetics points out that art works will stimulate specific brain regions through visual stimulation, while this process still involves knowledge from the hippocampus. This is why "There are a thousand Hamlets in a thousand peoples eyes". Different people have different understandings and feelings towards the same artistic image, as it involves their past life, experiences, and educational level. Therefore, facial stylization is an extremely complex task which involves both algorithm design and human visual perception. This is why it is difficult to obtain a standard evaluation standard.And does our method really possess mechanisms and abilities similar to the human brain? We can find a very interesting phenomenon. For the abstract and concrete features we learn, it is like the processing of visual information by the human brain. On the one hand, it is stimulated by edge sensors, and on the other hand, it is knowledge-driven cognition. In addition, concrete features also have vastly different characteristics in encoders and decoders, just like how the human left and right brains focus on different functions and tasks. However, the diffusion model is not a brain-like neural network designed concerning brain structures, which has also sparked our thinking about the relationship between large models and brain-like mechanisms. Justification: As in the Abstract, our claim is that we present an Abstract-Concrete Fusion Facial Stylization method, which is discussed in Section 3, and make contributions in Section 4Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.+ γω t ∥DM (α t I i + σ t ϵ, e * a ) -I f ∥ 2 2 ].(4)Where α t , σ t , ω t are parameters used to control the noise schedule and sampling quality and are functions of the diffusion process time t ∼ U ([0, 1]). Meanwhile, this is a win-win training strategy. On the one hand, the abstract features e a of AFun provide a good latent space for the training of the generator. On the other hand, through gradient backward, the abstract features of AFun can also extract concrete information from facial images, providing better quality guidance. Through this method, we can constrain the optimization process of abstract feature fusion for style images and face images in CLIP space, as well as the extraction of concrete features for style images and face images using learnable parameters inserted into the generator.4 ExperimentsIn this section, we conduct experiments using images collected from the Internet and provide visual comparisons. We also conducted experiments on both facial stylization and text-to-image generation tasks. Our method has achieved good visual effects in various experiments, demonstrating its effectiveness. 


Decision: This paper received mixed reviews, ranging from "Borderline reject" to "Accept". The reviewers noted that the paper's novel facial stylization, ACFun, achieves style extraction at both abstract and concrete levels using only one pair of images, calling it well-structured and easy to follow, and acknowledged its extensive experiments and visualizations.

Many of the reviewer's concerns related to the paper's experimental setup: The reviewers noted a lack of quantitative metrics and potential subjectivity in qualitative results, though this was partially addressed in the rebuttal  (5hdY, uyVB). The reviewers also called out limited comparison with existing methods, particularly for text-guided image generation (bo1j, 5hdY), inconsistent / unclear experimental results, particularly in ablation studies and style transfer quality (bo1j, uyVB), and the need for additional experiments to demonstrate generalization ability and address specific scenarios (6WJE). Other concerns related to insufficient novelty (5hdY) and unclear technical details (uyVB). 

The authors provided a detailed rebuttal, but it, and the reviewer discussion, did not change the distribution of recommendations. In such borderline cases, I am inclined to err on the side of acceptance and let the community decide the merits of this work.

Reviews: review: paper_summary: paper_summary: This article introduces a generative model ACFun for facial stylization, which designs two modules AFun and CFun to learn the abstract and concrete features of styles and faces. The authors design a Face and Style Imagery Alignment Loss to align the style image with the face image in the latent space, using these methods to extract more levels of style features and better balance facial changes caused by stylization and preserving facial information.
strength_weakness: strength_weakness: 1. Lack of Quality Analysis. Sec. 4.1 is missing.
2. The ability of text-guided image generation was introduced in the experiment but only compared with InstantStyle, which seems insufficient.
3. The role of CFun in the encoder and decoder was analyzed in the ablation experiment, and the results are shown in Fig.6. However, for Encoder, the result in the first line is only a blurry image with a rough structure, while the second line shows the specific facial structure and posture. It seems that the conclusions drawn from the two examples are different. For the decoder, the first line shows the specific facial structure and posture, while the second line's facial structure and posture appear to be inconsistent with the reference image. These two examples are insufficient to prove that “In the decoder, CFun can see that the main concrete features are the structure and posture of the face.”
questions: questions: 1. What is y in Equation 1?
2. The reviewer is interested in the image on the right side of Figure 3 and suggests adding some visualizations or experiments to demonstrate the role of the loss function in aligning style and face images.
limitations: limitations: Limitations are discussed.
paper_summary: paper_summary: This paper deals with the problem of facial stylization using one style image and one facial image. Specifically, the authors design an abstract fusion module and a concrete fusion module to learn the abstract and concrete features of the style and face separately. They further design a face and style imagery alignment loss for face and style image aligning. Experiments show that the proposed method outperforms other methods.
strength_weakness: strength_weakness: 1. There are no quantitive metrics, which can show the proposed method is better than the existing methods, only qualitative ones, which might be subjective and biased.
2. Some visualization results still have room to be improved. The most concern is that some style information is merged with the identity or object information of the style image, which is not well disentangled. For example, in the Fig.11, the first style image have purple and pink style, while the proposed method cannot learn the detailed style information very well.
3. The method is lack of novelty. To me, using the guidance of CLIP image and text features is somehow outdated. The concrete fusion module is an attention module to me, which is also well studied.
questions: questions: Could the authors provide more information about using CLIP, as it seems to me there are better vision language models which can extract better features than CLIP? How about using other models?
limitations: limitations: The authors have discussed the limitations of their paper.
paper_summary: paper_summary: This paper proposes a novel facial stylization method called ACFun, which achieves high-quality stylization effects by combining abstract and concrete visual elements. It contains the Abstract Fusion Module (AFun) and Concrete Fusion Module (CFun) to learn the abstract and concrete features. A new loss function is designed to align style and facial images in latent space to improve stylization accuracy. The effectiveness of the ACFun method is validated through extensive experiments, which can produce more artistic and visually appealing facial stylization results compared to existing methods.
strength_weakness: strength_weakness: Some experimental results still need to be supplemented to demonstrate the generalization ability of this method：
1)	Will there be any difference in the final results generated using detailed and vague text descriptions for the facial description prompts proposed in this paper?
2)	The experiment only showed the generation results of 40 diffusion steps. What are the differences between different diffusion steps?
3)	In the experiment, the gender of the given style image and face image is always the same. Does this method still work when faced with the gender of the given style image and the face image is different?
questions: questions: Shown as weaknesses.
limitations: limitations: Shown as weaknesses.
paper_summary: paper_summary: The paper introduces a new method for face stylization. The model uses a pretrained diffusion model (Stable Diffusion 1.4) and the proposed Abstract and Concrete Modules (AFun and CFun modules) to achieve the goal. The former extracts the style details to be imbued into the generative process. The latter conditions the diffusion process on those details. Notably, CFun leverages a technique known in AdaIN-like methods to modify the statistics in the generative process, while the diffusion keeps the general content of the face. Since the training happens every time a new image-style pair is introduced, it generalizes to new styles outside the used dataset. Additionally, the training lasts only 3 minutes.  This quick adaptation makes the method highly practical for users who want to efficiently apply different styles to faces.
strength_weakness: strength_weakness: - The authors tackle a highly subjective problem in terms of what makes "a good style transfer". The problem exists especially here, as only qualitative results are provided. For example, in many cases, InstantStyle SDXL provides (in my subjective opinion) better results than the proposed method. Additionally, Tuning Attention exhibits a better style transfer than CFun, as shown in Figure 8 in the supplementary. Figure 11 confuses me even more as the proposed method does not transfer the style correctly compared to InstantStyle.
- Although the introduction clearly states the problem and how the problem is being solved, the latter sections contain vague sentences that make the overall method hard to comprehend. For example, I do not understand how the alignment loss (Section 3.4) aligns the images as the diffusion process is tasked to reproduce both the style and reference images using the same input. The section does not say why the alignment is needed, and the ablation study does not explain it either. The right section of Figure 3. (especially the right bottom image) makes it even more ambiguous.
- Some parts need further explanation, for example:
    - (L65) What does making an image recognizable by the CLIP model exactly mean?
    - (L51-L53) The sentence is hard to understand what needs balancing
    - (L193) What is VQ space?
    - (L234-L237) What are those traditional methods, and what is the "two-column method"?
    - (L241-L243) What do the authors mean by "understanding the face" by a model? If ControlNet can reconstruct a face, does it not mean it understands the face, too? 
    - (L286-L287) What is the binding phenomenon, and what does the style leakage look like?
questions: questions: Questions:
- Why did the authors not apply "Tuning Attention" as the main component of the method instead of CFun? It seems to transfer the style more reliably and serves the same purpose as CFun?
- Could the authors elaborate on how using this method makes the stylization less effortless than using InstaStyle SDXL?

Suggestions:
- The authors need to evaluate the model in terms of the user preference. Given a high enough number of participants, the survey would objectively tell which stylization is better for an average person. 
- I suggest clarifying the parts mentioned in the weaknesses, which I attach below for self-consistency
- CFun resembles the commonly used AdaIN module
- Some parts need additional references. For example, the diffusion described in L228 seems to be a DDIM [2]

[1] Huang X, Belongie S. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE international conference on computer vision 2017 (pp. 1501-1510).
[2]  Jiaming Song, Chenlin Meng, Stefano ErmonDenoising Diffusion Implicit Models. In International Conference on Learning Representations 2021.
limitations: limitations: - The authors describe the limitations in a separate section. However, some additional visual examples would be necessary to explain what the authors mean by "extract a few visual elements of the style we want to exist in the style image" (L283) and "style leakage" in 287.

Comments: title: Response to the Authors' Rebuttal
comment: Thanks for the author's response. I will keep my original score.
Regarding novelty:
Reviewer 6WJE proposed: A novel facial stylization method called ACFun is proposed, which solves the limitations of existing stylization methods in processing facial images by combining abstract and concrete features. 
Reviewer uyVB proposed: The proposed approach generalizes to novel face-style pairs as it is "fine-tuned" each time. Splicing Facial Stylization is an exciting application of the method, which I have not seen in past works. It opens a new venue for artists.  

Regarding the motivation of CFun design:
Review uyVB proposed: I enjoyed reading the explanation behind the "abstract" and "concrete" features, which clearly show the necessity of such decoupling in the pipeline.
Reviewer 6WJE proposed: A novel facial stylization method called ACFun is proposed, which solves the limitations of existing stylization methods in processing facial images by combining abstract and concrete features.

Regarding the experiments:
Reviewer 5hdY proposed: Experiments show that the proposed method outperforms other methods. Many visualization results are shown in well-plotted figures.
Reviewer 6WJE proposed: Through extensive experiments, the author demonstrated the effectiveness of the ACFun method in facial stylization tasks, which can produce higher quality artistic results compared to other existing technologies. 
Reviewer uyVB proposed: The authors present sufficient qualitative results.
Thanks for your response. Firstly, unlike IP adapters, which aim to fuse a concept into a diffusion network using few shot images, our approach seeks to merge style images and facial images into one concept. In addition, we conducted comparative experiments comparing the results of the peer-reviewed papers Dreamstylist and Inst. We explained in the paper that we divide image features into abstract and concrete visual features for subsequent face stylization tasks. CFun was designed with this motivation and demonstrated its effectiveness, it is designed to transmit low-level visual features to make stylized faces more similar in specific strokes, textures, and other aspects. And simplicity itself is not a disadvantage. Simple and effective designs are often easier to understand and implement. Finally, regarding your subjective judgment on our method, as reviewer uyVB pointed out, this is a highly subjective issue, and a few individuals alone may have divergent judgments on the final result. That's why we cited a user study to demonstrate the superior performance of our method's results. Finally, up to now, the IP adapter you mentioned, like InstantStyle, is also not a  peer-reviewed paper, and even if InstantStyle is not a peer-reviewed paper, if it performs well in practice, it can still serve as an effective benchmark for comparison. In addition, we will add more visual results in the supplementary materials in future versions.
Thanks for the response. However, my concerns for not suggesting acceptance were not solved:
First, the method itself is not groundbreaking. The overall pipeline is similar to many popular solutions such as IPadaptor, which extract image and text features for a few shot samples and then fuses them to the pretrained network. The proposed CFun module is also rather simple without well-explained motivation.
Second, the results are not impressive. The authors claim that they only compared with InstantStyle for its excellent performance. However, InstantStyle is not even a peer-reviewed technical paper. Such a claim made the reviewer challenge the authors' basic technical skills. The newly added statistics are also not convincing. The face consistency of the proposed method is extremely high then the other method. However,  the visual results could not support such statistics. Also, the stylized effects are worse than the other selected methods.
I'm sorry, the answer to Weaknesses 3 and the question were ambiguous, which may cause confusion for you. I want to add that due to the use of backbone stable diffusion as a text-to-image generation model, the language model used is CLIP. As a pre-trained large model, it is necessary to retrain the diffusion model to replace CLIP with another model. However, the training cost of diffusion models is generally difficult to afford, so we choose to continue using CLIP in stable diffusion rather than choosing other new visual language models. In addition, the reason why we use CLIP as a visual extractor is also because the image embeddings encoded by CLIP's image encoder are more easily aligned with the text embeddings in the latent space they construct, which is also beneficial for subsequent fusion by cross-attention.
Thanks for taking the time to provide such a detailed response and acknowledging our work. 

For the stable diffusion model and VQGAN question you mentioned, the vq encoder is a module in the pre-trained stable diffusion model used to accelerate it. The stable diffusion training process first trains the VQ encoder and then the diffusion model itself. 

Also, thank you so much for your suggestions on our writing. We will follow your suggestions to improve the fluency of our paper, avoid long and difficult sentences as much as possible, and use clearer expressions to improve our writing.
Thank you for your detailed feedback and for providing the results in PDF format. I appreciate that most of your concerns were addressed in the response.

The primary issue affecting my initial score was the absence of quantitative results. Now that this has been rectified, I am pleased to increase my final rating. Additionally, I appreciate the design of the study and the decision to divide the evaluation into three independent aspects.

I have one remaining question (which, however, does not impact my score): Could you specify which components are pretrained? For instance, in Stable Diffusion, the VQGAN component is pretrained prior to training the diffusion model. Including such details in the implementation section would enhance the method's comprehensiveness.

My main remaining concern is the quality of the writing. There are several instances where the flow is disrupted or where there is repetitive use of words, which could potentially confuse the reader. For example:
- Line 133: The introduction of the problem with inversion-based models over two sentences causes the reader to lose track of the method's overview.
- Line 187: The sentence is overly long.
- Lines 197-201: These lines are difficult to comprehend on first reading.

Addressing these issues could significantly improve the clarity and readability of the text.