ZePo: Zero-Shot Portrait Stylization with Faster Sampling

Authors: jojogan  inst
stylegan-nada  dynagan
  cvt
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-01 05:11:08

Abstract: Diffusion-based text-to-image generation models have significantly advanced the field of art content synthesis. However, current portrait stylization methods generally require either model fine-tuning based on examples or the employment of DDIM Inversion to revert images to noise space, both of which substantially decelerate the image generation process. To overcome these limitations, this paper presents an inversion-free portrait stylization framework based on diffusion models that accomplishes content and style feature fusion in merely four sampling steps. We observed that Latent Consistency Models employing consistency distillation can effectively extract representative Consistency Features from noisy images. To blend the Consistency Features extracted from both content and style images, we introduce a Style Enhancement Attention Control technique that meticulously merges content and style features within the attention space of the target image. Moreover, we propose a feature merging strategy to amalgamate redundant features in Consistency Features, thereby reducing the computational load of attention control. Extensive experiments have validated the effectiveness of our proposed framework in enhancing stylization efficiency and fidelity.

Main paper: 
 introduction: 
 Portrait stylization involves the transfer of an art style from a reference portrait to a standard facial photograph. Traditional methods for portrait stylization [7, 9, 29, 51, 67] typically involve fine-tuning a pre-trained generative model [17, 46] , such as StyleGAN [23] or Stable Diffusion [46] , using various reference art portraits. However, these approaches necessitate considerable time for model finetuning and additional storage space to accommodate the models that have been fine-tuned for each distinct style image.To overcome the limitations mentioned above, recent studies have investigated a tuning-free method [10, 11, 31] that leverages self-attention features from both content and reference images during the generation process through Attention Control [5] , enabling zero-shot portrait stylization. Despite this advancement, the method struggles with slow image generation speeds. On the one hand, the diffusion model requires an extensive sampling process to iteratively denoise Gaussian noise. On the other hand, to accurately reconstruct the content and reference images, this method often depends on the protracted DDIM Inversion [52] process, which is necessary to obtain a sequence of intermediate anchors for image reconstruction. Additionally, the manually customized Attention Control [5] operation exacerbates the situation by involving excessive computations of the redundant self-attention mechanism, further impeding the image generation speed.In this work, we introduce ZePo, a Zero-shot Portrait Stylization framework, to address the aforementioned challenges. Regarding the issue of slow sampling speeds, one intuitive solution is to employ high-order numerical ODE solvers [2, 34, 70] to decrease the number of sampling steps required for image generation. However, these methods, which leverage high-order ODE approximations, necessitate multiple network function evaluations (NFEs) and achieve only a marginal reduction in actual sampling time. Moreover, these ODE solvers do not integrate well with the intermediate anchors established by DDIM Inversion, which restricts their applicability for this particular method. Therefore, rather than relying on high-order ODE samplers, we propose the use of accelerated distillation of pre-trained models, specifically Latent Consistency Models (LCMs) [35] , to expedite the image synthesis process. Additionally, to obviate the need for DDIM Inversion, our findings indicate that LCMs can directly extract representative consistency features from noised images. Building on this capability, we suggest a method to directly extract consistency features from noisy reference and content images. These features are then seamlessly incorporated during the generation process of the target image, resulting in a more efficient and streamlined stylization approach.To address the issue of speed reduction due to redundant computations in conventional Attention Control methods, we introduce the Style Enhancement Attention Control (SEAC). SEAC begins by integrating the redundant consistency features from both the source and reference images. Subsequently, it concatenates these merged features and maps them as key and value features within the self-attention space. To modulate the degree of image stylization, the key features of the reference image are multiplied by a Style Enhancement coefficient. Consequently, the attention map, calculated using the query features from the target image and the merged key features, can adaptively select the value features from both the content and reference images. This method not only increases the computational speed of Attention Control but also mitigates the issue of query confusion, enhancing the precision and efficiency of the stylization process.Ultimately, as illustrated in Figure 1 (left), our method demonstrates the capability to synthesize stylized portraits using no more than four sampling steps, significantly enhancing both the speed and practicality of portrait stylization using diffusion models. Through extensive experimentation, we have demonstrated the advantages of our ZePo framework in rapid stylized portrait synthesis. As illustrated in Figure 1 (right), ZePo does not require additional fine-tuning time, and it achieves the optimal CLIP-CIA score while reducing the inference time to just 0.6 seconds using a 4-step sampling process.To summarize, we make the following key contributions: (i) We introduce ZePo, a new inversion-free portrait stylization framework that requires as few as one sampling step to synthesize high-quality stylized portraits.(ii) We propose a novel attention control mechanism, termed Style Enhancement Attention Control, which leverages redundant feature fusion to enhance the speed of self-attention computations and can adaptively select value features from source and reference images.(iii) We demonstrate from both quantitative and qualitative perspectives that our method surpasses existing state-of-the-art baselines, achieving a significantly better balance between preserving source content information and enhancing image stylization. 
related works 2.1 few-shot face stylization: 
 Early methods [20, 26, 33, 57, 72, 73] of face stylization often required sampling a large volume of image data to train image-toimage translation models, consuming substantial training resources. To mitigate the high cost of training and capitalize on the prior knowledge embedded in pre-trained models, the few-shot face stylization approach has gained considerable attention. This method involves fine-tuning a pre-trained StyleGAN model [22] [23] [24] with a limited number of target image samples, a technique commonly known as GAN-adaptation [39, 45, 58, 59, 61, 69, 71] . Toonify [42] was among the pioneers in experimenting with face stylization using GAN-adaptation. They initially fine-tune a StyleGAN model using a limited set of cartoon samples and subsequently interpolated the weights of the fine-tuned model with those of the original model to generate cartoon-styled faces. [30, 40] introduced additional regularization terms in the latent space to mitigate the tendency of overfitting when fine-tuning pre-trained models with a small number of samples. AgileGAN [51] introduced an inversionconsistent transfer learning framework that effectively reduces the variance in the inversion distribution. [60] developed a method that introduces an intermediate domain between the source and animation domains to bridge the gap between the two. DualStyle-GAN [63] adds an additional style path to a pre-trained StyleGAN, enabling efficient modeling and adjustment of both intrinsic and extrinsic styles. However, it still requires hundreds of images for fine-tuning, which limits its applicability in scenarios with very few examples. JoJoGAN [9] advances this by proposing a one-shot face stylization method that utilizes a reference image to generate a style mixed paired dataset. The model is then fine-tuned using this dataset with pixel loss, enhancing its utility in limited-sample environments. [68] proposed a novel one-shot adaptation method for face stylization, which divides the process into style transformation and identity transformation. By effectively separating identity from style, their approach results in more natural and coherent transformation outcomes. StyleDomain [1] introduced a parameterefficient method that adapts pre-trained models to new domains by modifying style vectors within the Style Space, enhancing adaptability with minimal resource usage. [71] utilize a single real-style paired reference to provide style direction in the DINO-ViT [6] feature space, enabling precise fine-tuning of the generative model.Diffusion models [12, 18, 50, 52, 54] have increasingly become prominent in the field of generative models in recent years, particularly with the advent of pre-trained text-to-image (T2I) models [ 41, 46, 49] . These models have not only popularized AI-generated art but also spurred extensive research into style transfer methods leveraging diffusion models. Several methods are inspired by classifier guidance [12] , utilizing precisely formulated energy functions to deliver gradient information for guiding image generation [38, 64] . This allows unconditional diffusion models to produce images conditioned on specific content and style. For instance, [28] advocates for employing the style loss derived from a pre-trained DINO-ViT [6] to guide the generation of stylized images. Similarly, [62] utilizes style loss generated by CLIP [44] for guiding stylized image generation. Other techniques focus on personalizing diffusion models with specific styles by fine-tuning pre-trained T2I models using approaches like LoRA [19] , Textual Inversion [14] , or Dreambooth [48] . Subsequently, DDIM Inversion [43, 52] is used to derive a noise representation of content images, which the fine-tuned model denoises in the noise space, thereby generating stylized images [7, 25, 67] . Particularly, [7] combines this with null-text inversion [36] to achieve more precise content reconstruction, though this method tends to slow down the image synthesis process. Distinct from methods that merely fine-tune T2I models, some strategies [8] involve fine-tuning a pre-trained Diffusion Autoencoder [43] using optimized semantic latent codes to meticulously control both content and style. Furthermore, techniques such as ControlNet [65] and T2I-Adapter [37] train additional style adapters using extensive datasets of style images. These adapters are designed to adjust the style of images produced by pre-trained T2I models, offering a tailored approach to style management in image generation. Recent studies have begun to explore zero-shot stylization methods that utilize pre-trained T2I diffusion models [10, 11, 31] . These methods integrate content and style features within the attention space through meticulously engineered attention control modules [5] . However, they depend on DDIM Inversion to derive noise representations of content and style images, which consequently slows down the image synthesis process. 
preliminaries 3.1 latent diffusion models: 
 Latent Diffusion Models (LDMs) [46] employ a diffusion model within the latent space of a pre-trained Variational Autoencoder (VAE) [13] . The encoder E encodes images into latent codes 𝑧 0 = E (𝑥), while the decoder D reconstructs images 𝑥 = D (𝑧 0 ) from these codes. The forward process of diffusion models operates as a Markov chain, incrementally introducing noise into the initial latent code 𝑧 0 . Due to the additive nature of Gaussian noise, this process is generally modeled as a single-step addition of noise, directly yielding the noisy latent code 𝑧 𝑡 at any given step 𝑡:EQUATIONwhere 𝛼 𝑡 is a predefined diffusion schedule. The reverse process of diffusion models constitutes an approximate Markov chain, where progressively removing noise in 𝑧 𝑇 through the reverse process, ultimately restoring the noise-free latent code 𝑧 0 after 𝑇 iterative steps:EQUATIONwhere 𝝐 𝜃 is a time-conditioned U-Net [47] , tasked with predicting the noise component in 𝑧 𝑡 at each step 𝑡. The parameters 𝜃 within 𝝐 𝜃 are fine-tuned by minimizing a noise prediction loss:EQUATIONwhere 𝝐 denotes the noise introduced during the forward process as described in Eq. 1.Latent Consistency Models (LCMs) [35] , are a specialized form of Consistency Models (CMs) [53] that significantly accelerate the generation speed of LDMs. In LCMs, the consistency function 𝑓 (𝑧 𝑡 , 𝑡) ensures that each anchor point 𝑧 𝑡 in the sampling trajectory can EQUATIONwhere 𝑓 𝜃 denotes a consistency function parameterized by a trainable neural network, and 𝑓 𝜃 -is updated at a slow decay rate 𝑢 to adjust parameters within 𝑓 𝜃 . The variable ẑ𝜙 𝑡 represents a one-step estimate of 𝑧 𝑡 obtained through the sampler 𝜙 from 𝑧 𝑡 +1 . 
method: 
 In this section, we introduce ZePo, a zero-shot framework for portrait stylization that operates within four sampling steps. Our framework leverages Latent Consistency Models (LCMs), a variant of Stable Diffusion that distilled with the consistent objective (Eq. 5). ZePo capitalizes on the observation that LCMs not only significantly reduce the number of sampling time steps required for generating images but also efficiently extract representative features from noisy images, which we term Consistency Features. Utilizing the Consistency Features extracted from both source and reference images, we seamlessly integrate these features into the image generation process through our proposed Style Enhancement Attention Control module. This integration allows for subtle yet effective stylization adjustments. Ultimately, with just four sampling steps, our framework is capable of synthesizing high-quality stylized portraits that faithfully capture the style of the reference image. The overall architecture of our framework is depicted in Figure 2 .The primary purpose of employing DDIM Inversion [52] is to derive a series of anchor points {𝑧 𝑡 } that facilitate the reconstruction of the original image 𝑧 0 , where each anchor 𝑧 𝑡 is capable of recovering 𝑧 0 with better accuracy. As illustrated in Fig. 3 (a) (b), utilizing the noisy latent 𝑧 𝑡 derived from the forward process in Eq. 1, tends to yield a predicted 𝑧 0 that is blurry and lacks high-frequency details. In contrast, the noisy latent 𝑧 𝑡 post DDIM Inversion can estimate 𝑧 0 with enhanced accuracy. The optimization objective (Eq. 5) of LCMs is aims to minimize the disparity between the outputs of consistency function in adjacent samples, which corresponds to the distance between one-step predictions of the model for 𝑧 0 . It is observed that this objective endows LCMs with superior one-step predictive capabilities for 𝑧 0 . As depicted in Figure 3 (c ), the noise level during forward process is relatively low, particularly for time steps 𝑡 ≤ 300, the estimated ẑ0 by LCM exhibits clearer and more consistent details compared to the original 𝑧 0 . This suggests that LCMs can effectively extract representative features from a noisy image, which is referred to Consistency Features. Inspired by this capability, we propose leveraging the Consistency Features extracted from both source and reference images for portrait stylization. This method effectively replaces the time-consuming DDIM Inversion process, offering a more efficient pathway to achieving high-quality portraits stylization. Given a source image 𝐼 𝑠𝑟𝑐 and a reference image 𝐼 𝑟𝑒 𝑓 , initially, a pre-trained VAE encoder E encodes them into latent codes 𝑧 𝑠𝑟𝑐 and 𝑧 𝑟𝑒 𝑓 , respectively. Subsequently, a forward process (Eq. 1) is applied to introduce noise to these latent codes in a single step, defined as follows:𝑧 𝑠𝑟𝑐 𝑡 = √ 𝛼 𝑡 𝑧 𝑠𝑟𝑐 0 + √ 1 -𝛼 𝑡 𝜖, 𝑧 𝑟𝑒 𝑓 𝑡 = √ 𝛼 𝑡 𝑧 𝑟𝑒 𝑓 0 + √ 1 -𝛼 𝑡 𝜖, where 𝑡 represents a smaller time-step, and 𝜖 ∼ N (0, 𝑰 ). Finally, the noisy latent codes 𝑧 𝑠𝑟𝑐 𝑡 and 𝑧are inputted into the noise prediction network 𝜖 𝜃 of the LCMs, from which the consistency features {𝑓 𝑠𝑟𝑐 𝑙 } and {𝑓} of the source and reference images at each transformer layer 𝑙 of 𝜖 𝜃 are extracted. This process is formalized as:({𝑓 𝑠𝑟𝑐 𝑙 }, {𝑓 𝑟𝑒 𝑓 𝑙 }) = 𝜖 𝜃 ((𝑧 𝑠𝑟𝑐 𝑡 , 𝑧 𝑟𝑒 𝑓 𝑡 ), 𝑡, 𝑐), where 𝑐 denotes the textual condition. In contrast to previous approaches [5, 55] which necessitate feature injection to align with the current generation time step, our proposed consistency features exhibit flexibility in this regard. They are not bound by the requirement to match the current generation time step. Thus, the extracted consistency features can seamlessly integrate into the generation process at any time step, ensuring their consistent contribution throughout various stages of the generation process. We demonstrate the impact of feature extraction at different time steps on the generated results in Figure 7 .Attention Control Attention Control (AC) replaces the key and value features in the target image generation branch with those derived from the source image reconstruction branch. Leveraging the self-attention mechanism, AC adaptively aggregates features from the reference image, thereby preserving both semantic and structural information from the source image. However, the incorporation of AC significantly impacts the speed of image generation in existing methods. We conducted a comparative analysis, measuring the time required for image generation with and without AC under identical step settings. As presented in Table 1 Feature Merge In Vision Transformers, there exists redundancy in tokens, and pruning these redundant tokens during inference can lead to a model with faster inference speed [3] . Similar techniques have been investigated within the diffusion model framework, which extensively employs self-attention modules. Merging redundant features in diffusion models has been shown to significantly enhance the speed of image generation without compromising the quality of the generated images [4] . Building upon this observation, we propose leveraging the token merge technique to merge redundant feature sequences before to attention control, thereby reducing the length of features from 𝑁 to 𝑁 /2 or less. In contrast to the approach outlined in [4] , which necessitates the unmerging of merged token sequences to restore the original length of token sequences, our method exclusively merges the consistency features inputted into attention control. This targeted merging strategy helps circumvent errors that may arise during the un-merging process. ). Upon entering the Attention Control mechanism, these merged features are individually mapped to the key (𝐾 𝑠𝑟𝑐 , 𝐾 𝑟𝑒 𝑓 ) and value (𝑉 𝑠𝑟𝑐 , 𝑉 𝑟𝑒 𝑓 ) features within the self-attention module. In contrast to the conventional AC methods that directly replace key and value features, we introduce a Style Enhancement Attention Control (SEAC) mechanism. Specifically, we concatenate (𝐾 𝑠𝑟𝑐 , 𝐾 𝑟𝑒 𝑓 ) and (𝑉 𝑠𝑟𝑐 , 𝑉 𝑟𝑒 𝑓 ) from the source and reference images into a unified set of key and value features. Moreover, we enhance 𝐾 𝑟𝑒 𝑓 by multiplying it with a Style Enhancement coefficient 𝜆, yielding a new set of key and value features as follows:𝐾 𝑠𝑟 = Concat(𝐾 𝑠𝑟𝑐 , 𝜆 • 𝐾 𝑟𝑒 𝑓 ) ∈ R 𝐵,𝑁 ,𝐷 𝑉 𝑠𝑟 = Concat(𝑉 𝑠𝑟𝑐 , 𝑉 𝑟𝑒 𝑓 ) ∈ R 𝐵,𝑁 ,𝐷 .Subsequently, the key feature 𝐾 𝑠𝑟 and the query feature 𝑄 𝑡𝑔𝑡 ∈ R 𝐵,𝑁 ,𝐷 from the target image are utilized to compute a self-attention map 𝐴 given by:𝐴 = SoftMax 𝑄 𝑡𝑔𝑡 • 𝐾 𝑠𝑟 𝑇 √ 𝑑 ∈ R 𝐵,𝑁 ,𝑁 ,where 𝑑 represents the dimensionality of the query and key features. Finally, the self-attention map 𝐴 is applied to the value feature 𝑉 𝑠𝑟 to derive the final output 𝜙 𝑡𝑔𝑡 as follows:𝜙 𝑡𝑔𝑡 = 𝐴 • 𝑉 𝑠𝑟 .Hence, SEAC can effectively assess the similarity between the query features 𝑄 𝑡𝑔𝑡 and the combined key features (𝐾 𝑠𝑟𝑐 , 𝐾 𝑟𝑒 𝑓 ), enabling the adaptive aggregation of value features from (𝑉 𝑠𝑟𝑐 , 𝑉 𝑟𝑒 𝑓 ). Additionally, the lengths of the query, key, and value features utilized in the attention computation are all 𝑁 , ensuring consistency in the computational cost of attention control compared to the original self-attention mechanism. The comprehensive pipeline of the Style Enhancement Attention Control is illustrated in Fig. 2 5 EXPERIMENTS Implementation Details. Our experiments utilize Latent Consistency Models (LCMs), a variant of the acceleration-distilled Stable Diffusion, employing the LCM sampler. The synthesis of each stylized image proceeds through four sampling steps. We utilize the word "head" as the conditional text prompt, and specify the classifier-free guidance scale at 2. The style enhancement coefficient 𝜆 is set at 1.2. These experimental procedures are conducted using a single NVIDIA 4090 GPU. Reference images are sourced from the AFHQ dataset [32] , and content images are drawn from the CelebA-HQ dataset [21] . All images are processed at a resolution of 512 × 512 pixels. INST (e) InST [67] StyleGA N-Nada (f) NADA [15] (g) DynaGAN [27] JoJoGA N  
qualitative comparison: 
 Baselines. To evaluate our method's efficacy, we performed extensive comparative experiments against current state-of-the-art (SOTA) few-shot adaptation techniques. This analysis encompassed StyleGAN-based approaches, including JoJoGAN [9] , StyleGAN NADA (NADA) [15] , and DynaGAN [27] . We also compared our method with diffusion-based techniques such as InST [67] and VCT [7] . All stylized outputs were generated using the respective open-source implementations provided by the authors.Table 2 : Quantitative comparison with conventional portrait stylization baselines. The best and second best of each metrics will be highlighted in boldface and underline format, respectively. ↓ indicates the lower is better, and ↑ higher is better. The best and second best of each metrics will be highlighted in boldface and underline format, respectively. ↓ indicates the lower is better, and ↑ indicates higher is better. Our method achieved the best LPIPS and CLIP-CIA scores among all baselines, and the best Style score and fastest inference speed among all diffusion-based methods. Figure 4 provides a qualitative comparison among various methods. (a) presents reference artistic portraits, while (b) displays the original natural faces. And (c) illustrates the results of our ZePo, and the subsequent columns showcase outputs from various competing models. As illustrated in Figure 4 (f-h ), although StyleGANbased methods are effective in transferring the reference style to content images, they often lead to excessive stylization. This overstylization results in significant deviations from the original content images, particularly altering facial poses as shown in the fourth and fifth rows for NADA [15] . Moreover, while JoJoGAN [9] achieves superior stylization effects compared to other StyleGAN-based methods, it struggles with content consistency, especially in preserving background elements of the content images. Among diffusion-based methods, InST [67] shows tendencies of overfitting, resulting in less desirable outputs. Conversely, VCT [7] manages a better balance between style transformation and content retention, though it often introduces significant changes in expressions. Our method not only facilitates various style transformations but also excels in preserving local details, such as facial features and hair texture, thereby maintaining consistent facial characteristics between the source and output images. For instance, specific local details, such as earrings in the first and fourth rows, are meticulously preserved in our methond.To demonstrate the superior quality and efficiency of our method in portrait art synthesis, we conducted quantitative comparisons with existing state-of-the-art (SOTA) methods. Metric. To objectively assess the effectiveness of our proposed method, we employed LPIPS [66] for content preservation and VGG Style loss [16] for stylization evaluation. We observed that style loss predominantly focuses on external texture styles, which does not effectively capture the intrinsic style of images. Consequently, we propose the adoption of the non-referential evaluation metric, CLIP-IQA [56] , for a more comprehensive assessment of image quality. CLIP-IQA leverages the CLIP model [44] , pre-trained on a largescale text-image paired dataset, as an image feature extractor. Then, this method evaluates the overall image quality through different text prompts that relate to image quality and aesthetics. Evaluation. For quantitative assessment, we randomly selected 10 style images and 10 content images, generating a total of 100 stylized images for each baseline. The quantitative results are presented in Table 2 . Our method outperformed other techniques, achieving the best scores on both LPIPS and CLIP-IQA metrics. A lower LPIPS score indicates superior content preservation by our method, while a higher CLIP-IQA score reflects our method's ability to synthesize images with better overall quality and visual appeal. Additionally, our style score was the highest among methods based on diffusion models. In addition to evaluating the quality of the generated results, we assessed the fine-tuning and inference times required by each method, as presented in Table 2 . The results indicate that the previous approaches necessitate extended periods for fine-tuning. Moreover, diffusion-based methods exhibit prolonged inference times, for example, InST [67] requires approximately 7 seconds to synthesize one stylized image, whereas VCT [7] experiences an increase in inference time to 37 seconds due to the need for Nulltext text inversion [36] . Our framework, employing a zero-shot approach, eliminates the need for additional fine-tuning. By incorporating Style Enhancement Attention Control, we have reduced the inference time to approximately 0.6 seconds, thereby enhancing the practicality of our method. Attention Control. We conducted extensive ablation experiments to verify the effectiveness of the proposed Style Enhancement Attention Control (SEAC). Figure 5 presents the ablation results using different Attention Control (AC) methods. Excluding AC results in merely the reconstruction of content images, lacking any substantive stylization. Conversely, the use of conventional AC often leads to over-stylization and the loss of critical content details. In contrast, our proposed Style Enhancement Attention Control (SEAC) maintains the integrity of content information while imparting a more subtle stylization effect. Additionally, the Style Enhancement (SE) coefficient effectively controls the strength of stylization. By adjusting the SE coefficient to 1.2, the stylization effect is notably enhanced, thus affirming the capability of SEAC to maintain a balance between content preservation and the desired level of stylization.Inference Steps Figure 6 illustrates the results produced by our method at various sampling steps. Notably, our method can generate satisfactory stylized outcomes with just a single sampling step, and further increasing the number of sampling steps refines the detail of the synthesized images. As indicated in Table 3 , enhancing the number of sampling steps leads to higher CLIP-CIA scores. However, this increment also results in a slight decline in content preservation and inference speed. To strike an optimal balance among stylization quality, content preservation, and inference efficiency, we established the sampling steps at four for all experiments. Additionally, we validated the effectiveness of the Feature Merge (FM) technique. As depicted in Table 3 , implementing FM reduces the time required to synthesize images by 20%, without significantly compromising the quality of the generated images. This demonstrates that the feature merge technique not only enhances efficiency but also maintains high-quality stylization outcomes. Consistency Features. We performed an ablation study on the use of a fixed time step for consistent feature extraction in Figure 7 . Contrary to initial expectations, extracting features directly from the input image without the addition of noise results in the inability of the model to discern content and style features effectively. This phenomenon is consistent with the behavior of the consistency equation (Eq. 4) at 𝑡 = 0, where it merely outputs 𝑧 0 without any processing through the network. Consequently, during the consistency model distillation process, the noise prediction network 𝜖 𝜃  
conclusion: 
 In this study, we introduce ZePo, a framework capable of rapidly generating stylized portraits. Unlike previous methods, ZePo eliminates the need for fine-tuning on specific samples or DDIM Inversion for input images, thereby enabling high-quality stylization in just four sampling steps. This efficiency reduces the inference time to merely 0.6 seconds. We also introduce Consistency Features extraction strategy, which leverage a pre-trained diffusion model to extract multi-scale Consistency Features from both content and reference images. Through the proposed style Enhancement Attention Control module, Consistency Features are adaptively fused, allowing for adjustable stylization intensity via the style enhancement coefficient. Furthermore, we introduced a feature merge technique to merge redundant consistency features, significantly decreasing the computational cost of attention control and enhancing the model's sampling speed. Extensive experiments demonstrate that our method can synthesize high-quality stylized results while effectively preserving the content integrity of the source image, markedly surpassing the performance of existing advanced approaches. 
 
 Appendix:  


Decision: After the rebuttal, the reviewers unanimously recommend the acceptance of this paper. Therefore, we are happy to recommend the acceptance of this paper.

Reviews: review: paper_summary: paper_summary: This paper presents an inversion-free portrait stylization framework based on diffusion models that accomplishes content and style feature fusion in merely four sampling steps.
strength_weakness: strength_weakness: 1.Proposes an inversion-free portrait stylization framework, enhancing image generation speed.
2.Achieves content and style feature fusion in just four sampling steps.
3.Employs Latent Consistency Models with consistency distillation to effectively extract Consistency Features from noisy images.
4.Introduces a Style Enhancement Attention Control technique for meticulous content and style feature fusion.
limitations: limitations: 1.Unlike traditional style transfer methods that use VGG for feature extraction, the authors utilized VAE to obtain Consistency Features in section 4.1. I would like to understand the advantages of using VAE for the style transfer task or the reasons for not using models like VGG here. It is well-known that VAE sacrifices high-frequency features; therefore, I hope there are relevant experiments to validate this approach.

2.Is the Enhancement coefficient learnable? Is there any relevant experiment to prove the reliability of Enhancement coefficient?Why can SEAC effectively assess the similarity?

3.Can the models compared in Table 2 give more annotations? I don't know if this is the result of several sampling steps? According to Table 
3, different number of steps does not seem to have a direct effect.
paper_summary: paper_summary: The paper proposed ZePo, a framework for zero-shot portrait stylization that leverages diffusion models to synthesize stylized facial images without requiring model fine-tuning. ZePo achieves efficient content and style feature fusion in just four sampling steps, utilizing Latent Consistency Models and a novel Style Enhancement Attention Control mechanism to speed up the process. Extensive experiments demonstrate that ZePo significantly enhances the efficiency and fidelity of portrait stylization, achieving high-quality results with minimal inference time.
strength_weakness: strength_weakness: This paper is well-organized and straightforward. The experiments clearly demonstrate the advantages of the proposed pipeline. Leveraging the LCM improves the inference time significantly.
limitations: limitations: [1]. How does $\lambda < 1$​ impact the visual results?

[2]. Is the SEAC also available for text embedding guidance?
paper_summary: paper_summary: In this work, a zero-shot portrait stylization framework is proposed based on Consistency Models. The main component of this framework is a cross-image style feature injection module that utilizes self-attention features extracted from the Consistency Model during the inversion process as guidance. The primary contribution of this work is its leverage of the superior inference speed of Consistency Models compared to Diffusion Models, enabling more efficient portrait stylization.
strength_weakness: strength_weakness: As a test-time graphics problem, introducing Consistency Models in stylization for improved inference efficiency is a sound approach. Given that the output image’s style and content are determined by the reference and source images, the focus of this task should be on sample fidelity rather than mode coverage. Therefore, using Consistency Models is indeed a promising alternative to Diffusion Models. This approach offers valuable inspiration for future studies in image manipulation. I believe this work is near the acceptance bar for MM.
limitations: limitations: The technical contribution of this work is limited and incremental. The feature extraction and injection process using consistency models lacks an ablation study or insightful visualization to justify the current strategy. For instance, it is unclear in which part of the UNet (e.g., encoder layers, decoder layers, and different resolutions) the self-attention features contribute the most to stylization. Additionally, there is no analysis of redundant features from the self-attention layers used in the current setup. Why should Self-Attention K and V features be used for style injection instead of others? Further justification on this is needed.

Moreover, the Self-Attention feature injection strategy closely follows Prompt-to-Prompt (P2P) [3]. The use of self-attention K and V feature injection is the same as in [1]. The feature merging strategy has been employed by other works, such as RIVAL [2]. The style enhancement module mirrors the reweighting strategy used in P2P. The authors should modify the motivation extraction to place more emphasis on the Consistency Model itself, which is well-justified. Additional discussions on the technical setbacks encountered when adapting attention-based control strategies from Diffusion Models to Consistency Models would be beneficial.

[1]Alaluf, Yuval, et al. "Cross-image attention for zero-shot appearance transfer." arXiv preprint arXiv:2311.03335 (2023).

[2]Zhang, Yuechen, et al. "Real-world image variation by aligning diffusion inversion chain." Advances in Neural Information Processing Systems 36 (2024).

[3]Hertz, Amir, et al. "Prompt-to-prompt image editing with cross attention control." arXiv preprint arXiv:2208.01626 (2022).
paper_summary: paper_summary: This paper introduces ZePo, a portrait stylization framework that avoids DDIM inversion and relies on Latent Consistency Models (LCMs) to blend content and style features in just four sampling steps. Firstly, consistency features from both source and reference images are extracted using LCMs. These features are then utilized in the generation process of the final stylized portrait through the proposed style enhancement attention control module. Compared with current StyleGAN-based and diffusion-based methods, the proposed method shows better results, particularly in terms of LPIPS and CLIP-CIA scores.
strength_weakness: strength_weakness: 1. The main methodology is easy to read and understand.
2. This method directly extracts features from the source and reference images using LCMs instead of common DDIM inversion, reducing time consumption.
3. Quantitative and qualitive results demonstrate that the proposed method is superior to other compared approaches.
limitations: limitations: 1. It appears that CLIP-CIA is identical to CLIP-IQA in this manuscript; it is suggested to use one term consistently to avoid confusion. In addition, it is recommended to keep the names consistent such as x^{src} and I^{src} in Algorithm 1 and Figure 2. Moreover, there is a typo in Line 395: “is aims to” should be corrected.

2. The process of extracting consistency features at each transformer layer and the details of feature merging are not clearly explained. It would be beneficial to provide more detailed explanations for increased reproducibility. Specifically, clarifying whether the consistency features are the outputs of each self-attention layer would enhance understanding.

3. During the quantitative evaluation process, only 10 style images and 10 content images are utilized. Therefore, the numerical results in Table 2 are not very convincing.