Pick-and-Draw: Training-free Semantic Guidance for Text-to-Image Personalization

Authors: henglei  lv
jiayu  xiao
liang  li
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-31 15:06:51

Abstract: Diffusion-based text-to-image personalization has achieved great success in generating user-specified subjects in various contexts. However, finetuning-based methods often suffer from model overfitting, leading to reduced generative diversity, particularly when the provided subject images are limited. To address this issue, we introduce Pick-and-Draw, a training-free semantic guidance approach that enhances identity consistency and generative diversity. Our method comprises two key components: appearance-picking guidance and layout-drawing guidance. In the appearance-picking phase, we create an appearance palette from visual features of the reference image, selecting local patterns to maintain consistent subject identity. In the layout-drawing phase, we use a generative template from the base diffusion model to sketch the subject shape and scene outline, leveraging its strong image prior to produce diverse contexts based on various text prompts. Pick-and-Draw can be seamlessly integrated with any personalized diffusion model and requires only a single reference image. Both qualitative and quantitative evaluations demonstrate that our approach significantly improves identity consistency and generative diversity, establishing a new Pareto frontier in the balance between subject fidelity and image-text alignment.

Main paper: 
 introduction: 
 Recent large-scale diffusion models [7, 21, 22, 24] demonstrate remarkable capability in text-to-image generation. Trained on billions of image-text pairs collected from the Internet, these models are competent to synthesize high-quality and diverse images conditioned on textual inputs. Owing to unprecedentedly strong image priors, text-to-image diffusion models are successfully applied to various downstream tasks, including image editing [2, 8, 12] , inpainting [34, 37] , augmentation [9, 26] , style transfer [29, 35, 39] and controllable generation [31, 32, 38] .As a newly emerged task, text-to-image personalization aims to reason over specified subjects in assorted contexts. It requires the model to mimic the appearance of a subject given a reference image set and synthesize the same subject in different contexts. Many works [10, 16, 23] are proposed and have achieved impressive results. However, these methods still suffer from severe mode collapse in data-scarce scenarios where only a few reference images are available, and the diffusion network tends to simply memorize the few reference samples during the fine-tuning process. As a result, the model struggles to follow text instructions and synthesize subjects of different views, poses, and backgrounds. To mitigate this problem, some works [15, 23] leverage a regularization set to preserve the image priors of the original diffusion model, while others propose to fine-tune a subset of model parameters [15] or introduce extra low-rank adaptors [13] . These approaches help preserve the innate capabilities of the model, yet require extensive empirical hyperparameter tuning to obtain delicate results, and optimal hyperparameter configurations may vary across different subjects. Balancing the identity consistency and context diversity of generative outcomes remains a challenging problem.To this end, we propose Pick-and-Draw, training-free semantic guidance on text-to-image personalization methodologies, aiming to boost identity consistency while maintaining the ability of diverse context synthesis. In general, our approach consists of two components: appearance-picking guidance and layout-drawing guidance. (1) As for appearance-picking guidance, we feed the inverted latent of the reference image into the deep generative network and extract a visual feature set as a palette, from which we pick "color" for generating the specified subject. Specifically, we first adopt the layer-wise cross-attention maps corresponding to the specific subject, and threshold them to obtain salient binary masks. We then leverage the masks to extract the feature vectors within the object regions. Subsequently, we minimize the Unidirectional Relaxed Earth Mover Distance (UREMD) between the above feature vectors of the reference image and the generated image at each denoising step, to aid the model in better capturing the appearance cues of the new concept during the generation process.(2) As for layout drawing guidance, we borrow the subject shape and scene contour generated by the original diffusion model as a template. Then we imitate the outline to enable diverse posture and context synthesis of the new concept for a personalized model. This process aligns the subject's posture and scene context with the template to ensure diverse generated outcomes. The overall pipeline of our approach bears resemblance to the painting process of picking colors from a palette, drawing outlines based on a template, and subsequently applying colors to finalize the entire painting. In this sense, we term our method Pick-and-Draw.Pick-and-Draw is a training-free plug-and-play semantic guidance approach developed for boosting text-to-image personalization, applicable to various personalized models including Texual Inversion [10] , DreamBooth [23] , and BLIP-Diffusion [16] . Our method consistently improves personalized methods' identity consistency and generative diversity, pushing the trade-off between image fidelity and textual alignment to a new Pareto frontier. Moreover, we surprisingly find that directly applying Pick-and-Draw to vanilla Stable Diffusion [22] also yields favorable outcomes.To summarize, we make the following key contributions:• We propose Pick-and-Draw, a training-free semantic guidance approach to enhance identity consistency and generative diversity for text-to-image personalization models. EQUATIONStable Diffusion (SD) [22] is a powerful text-conditioned latent diffusion model that performs diffusion in the latent space 𝑍 instead of the pixel space 𝑋 and injects text condition into the diffusion process, allowing flexible conditional generation. 
energy functions in diffusion models: 
 From a score-based perspective, each step in the reverse process in a diffusion model can be seen as an estimate of a score function ∇ 𝑥 𝑡 log 𝑝 (𝑧 𝑡 ) [25] . Given an external condition 𝑦, diffusion models generate conditional samples from 𝑝 (𝑧 𝑡 |𝑦) ∝ 𝑝 (𝑧 𝑡 )𝑝 (𝑦|𝑧 𝑡 ). The first term 𝑝 (𝑧 𝑡 ) corresponds to the unconditional score function, and the second term 𝑝 (𝑦|𝑧 𝑡 ) is equivalent to an energy function E (𝑧 𝑡 ; 𝑡, 𝑦). Numerous energy functions have been proposed and used in various tasks, including classifier guidance [7] , CLIP scores [19] , and attention penalties [6, 8, 31, 33, 36] . In this sense, we propose two energy functions to boost identity consistency and maintain context diversity, respectively. We iteratively refine the generative outcomes via appearance picking and layout drawing, which is achieved by optimizing a designed score function. In appearance picking, we pick saliency-aware features from certain cross-attention decoder layers and transfer the appearance cues by minimizing the Unidirectional Relaxed Earth Movers Distance (UREMD), aiming to boost identity consistency. For layout drawing, we extract cross-attention maps in every cross-attention layer, smooth them with a Gaussian kernel, and then minimize the Frobenius norm to draw the subject outline and the scene contour. This localizes the appearance transfer within the subject-relative regions and introduces novel layouts from the vanilla Stable Diffusion, to improve generative diversity.Personalization aims to reason over specified subjects in assorted contexts. Textual Inversion [10] learns an embedding of a unique word to represent the specified subject. DreamBooth [23] fine-tunes the whole diffusion UNet to bind a unique identifier to the specified subject. Custom Diffusion [15] fine-tunes the key and value projection matrices in the cross-attention layers in the diffusion UNet. Encoder-based methods [11, 14, 30] fine-tunes the text encoder and potentially trains an image encoder or multi-modal transformer [16, 27] to encode example images of specified subjects into embedding that is leveraged in personalized generation. Our proposed Pick-and-Draw consistently improves identity reproduction of these personalization methods without harming the model diversity in a training-free one-shot manner.We share inspiration from DisenBooth [5] and DETEX [1] , which decouple the foreground and background regions in personalized generation. They aim to prevent the model from overfitting on a few samples during concept learning and fine-tune decoupled textual embeddings to separate subjects from irrelevant parts. In comparison, we impose constraints on intermediate features and cross-attention to correct the denoising path during inference time, which makes our approach training-free and model-agnostic. 
method: 
 Our purpose is to perform training-free surgery in personalized diffusion models to boost identity consistency and generative diversity. Given a reference image 𝐼 𝑟 that depicts a specified subject 𝑠 (e.g., dog), our aim is to generate a personalized picture 𝐼 𝑝 with a text prompt 𝑃 𝑝 (e.g. "A dog in water"), which is consistent with the instance 𝐼 𝑟 and conveys contextual semantics specified by 𝑃 𝑝 .Current personalized models fail to reconcile both identity consistency and generative diversity, since they are prone to overfit on the given few subject images, inevitably reducing the generative latent space of diffusion models to a lower dimension. To alleviate the above issues, our core idea is to inject appearance information of the reference image and contextual priors from the original diffusion into the personalized generative process. The overall architecture is shown in Fig. 2 . Our pipeline simulates a human painting process, we (1) adopt the intermediate feature set of the reference image as a palette, where we pick "colors" (i.e., representative feature vectors that convey appearance information) to blend the new subject on the canvas, and (2) draw the outlines based on a generative template. We develop an appearance picking guidance and a layout drawing guidance for the above two procedures, respectively, and iteratively update the noisy latent via optimizing a designed energy function. The appearance picking guidance helps preserve the subject identity, and the layout drawing guidance ensures the generative diversity. We first discuss cross-attention saliency map extraction and selection strategy in Section 3.1, then introduce our two types of semantic guidance in Section 3.2 and Section 3.3 respectively.Both the appearance and layout guidance rely on a saliency mask that highlights the subject-relative region. Previous works [12, 28, 31] show that cross-attention maps contain rich semantic and layout information. Similarly, we extract the subject-relevant crossattention maps at each layer 𝑙 of the diffusion UNet:EQUATIONwhere 𝑄 𝑙 is the query features projected from the image features, 𝐾 𝑙 is the key features projected from the textual embedding with corresponding projection matrices, and 𝑑 is a scaling factor. We perform min-max normalization on these maps to acquire a set of normalized cross-attention saliency maps Â0:𝐿 = { Â0 , Â1 , ..., Â𝐿 }.We omit the timestep 𝑡 for simplicity. Cross-attention maps of each layer contain different semantic information and highlight different regions of the image. Taking Stable Diffusion [22] as an example, the UNet consists of multiple up-blocks and down-blocks, with each block containing several cross-attention layers. We visualize the cross-attention maps corresponding to different layers in Fig. 3 . Maps from deeper blocks have smaller resolutions. Specifically, we observe that (1) maps of 64×64 resolution are fine-grained and tend to outline the edges of all salient objects. They capture the high-frequency attributes, yet contain much background noise; (2) maps of 32×32 resolution are better aligned with the subject and highlight different regions, entailing richer semantic information; (3) maps of 16×16 resolution are coarse and well aligned, reflecting the approximate layout information of objects; (4) maps from encoder layers are mostly blended with more background noise, while maps from decoder layers better align with the subject layout, which conveys richer semantic and structural information.Due to the disparate layout granularity and semantic information of cross-attention maps from each layer, we leverage different sets of maps for our proposed two types of semantic guidance. For appearance-picking guidance, we need to select the most representative activations to provide appearance cues. We choose the layers whose corresponding attention maps align well with the subject (or part of it). For layout drawing guidance, we need to introduce a novel image layout from an external prior. Note that the image layout not only includes the subject shape but also the context and background, and thereby we utilize attention maps from all layers.Intuitively, we regard the generative feature set associated with the object region of the reference image as a palette, which provides essential visual cues for object appearance. To generate subjects that are consistent with the reference image, we assign the closest element in the palette to each generative feature vector within the subject-related region. We then minimize a transport distance to facilitate the diffusion model to gradually capture the appearance essence from the reference image during the generation process.First, to obtain intermediate features of a reference image that depict the image content at each denoising step, we invert the reference image to the initial random noisy latent and feed it into the diffusion model to reproduce the denoising trajectory. We adopt Null-Text Inversion [17] as the inversion approach, which aligns the latent diffusion trajectory with the denoising trajectory by optimizing a null-text unconditional embedding in each step. Second, we spot the local regions corresponding to the specific subject of reference image and generative image, and accordingly extract the saliency-aware feature sets for appearance transfer. Specifically, we apply hard masks M 𝑙 derived from normalized attention maps Â𝑙 of each attention layer 𝑙 to highlight the subjectrelative regions of the dense feature Ψ 𝑙 , and acquire saliency-aware visual features V 𝑙 :EQUATIONwhere 𝜏 is a threshold parameter and tuple (ℎ, 𝑤) represents a spatial entry of the attention map. We then extract the nonzero channels of V 𝑙 ∈ R 𝐻 𝑙 ×𝑊 𝑙 ×𝐷 𝑙 to obtain a feature set V 𝑙 = {(𝑉 𝑙 ) 1 , ..., (𝑉 𝑙 ) 𝑛 }, where (𝑉 𝑙 ) 𝑖 is the 𝑖-th feature vector at the 𝑙-th layer and 𝑛 is the number of pixels within the salient region. Subsequently, we search for an effective way to transfer the appearance essence from the reference palette to the generative canvas during the denoising process. Previous works provide inspiration that leverages Earth Movers Distance (EMD) to model the divergence between two feature distributions. Specifically, let A = {𝐴 1 , ..., 𝐴 𝑛 } and B = {𝐵 1 , ..., 𝐵 𝑚 } be two sets of 𝑛 and 𝑚 feature vectors, respectively. The EMD is formulated as:EQUATION)where T is the transport matrix which defines partial pairwise assignments, and C is the cost matrix which defines the distance between an element in A and an element in B.The EMD measures the cost of bidirectional optimal transport between two sets of features. However, directly minimizing EMD brings undesired artifacts and distorted subject posture. We believe it is due to the redundant bidirectional constraints of EMD. In the context of appearance transfer, we aim to align the generative pixels with those of the reference subject. Specifically, we minimize the unidirectional optimal transport cost from the generated image featuresV gen 𝑙 = {(𝑉 gen 𝑙 ) 1 , ..., (𝑉 gen 𝑙 ) 𝑚 } to the reference image features V ref 𝑙 = {(𝑉 ref 𝑙 ) 1 , ..., (𝑉 ref 𝑙 ) 𝑛 }.We relax the EMD to a single constraint and define the Unidirectional Relaxed Earth Movers Distance (UREMD) as the appearance-aware loss:EQUATIONWe aim to assign the closest element in V ref 𝑙 to 𝑉 gen 𝑗 . In this manner, the aforementioned formulation is equivalent to:EQUATIONwhere we define the (𝑖, 𝑗)-th entry C 𝑖 𝑗 of cost matrix C as the pairwise cosine distance between two feature vectors:EQUATIONSimply put, at each step we find a one-to-one injection from the generated features to the reference features, and the mean UREMD between these two feature sets can be considered a metric evaluating overall subject appearance similarity. Optimizing the appearance-aware loss helps align the feature distributions of the reference image and generated image while avoiding excessive constraints on the generative layout, which is crucial for preserving the quality of the generated outcomes.Previous works have successfully utilized cross-attention layout control on image editing [8, 12] and grounded generation [6, 31, 33] . The key idea is that cross-attention maps highlight the salient objectrelated region, specifying the shape, posture, and position of objects within the canvas. Since vanilla Stable Diffusion which is trained on massive image-text pairing datasets has been proven of impressive generative diversity, we aim to perform layout guidance to borrow its image prior and guide personalized generation. We regard the layout generated by vanilla Stable Diffusion as a template and draw the outline by imitating. In this way, the personalized model inherits strong generative priors of SD, ensuring the diversity of generative outcomes.Unlike most works that aggregate the attention maps from each layer to a single saliency map, we leverage the cross-attention maps in all layers separately, as discussed in Section 3.1. Following Chefer et al. [4] , we first smooth the attention maps with a Gaussian kernel to eliminate noisy perturbations, then calculate the distance of layer-wise attention maps between the generated image and the template image as layout-aware loss:EQUATIONwhere 𝐺 is the Gaussian kernel, ∥ • ∥ 𝐹 is the Frobenius norm and 𝐿 is the number of cross-attention layers. By aligning the generative layout between the personalized model and the original diffusion model, we help the model inherit the strong generative priors, to synthesize diverse contexts according to different text conditions, and thus alleviate the model overfitting problem during generation. In practice, we also find that applying the layout-aware loss in the early steps of the denoising process helps pre-stabilizing the subject contour, providing a good initialization for performing the appearance guidance.By combining the appearance picking guidance and layout drawing guidance together, we generate new subjects that highly align with the reference image among various contexts, pushing the tradeoff between identity consistency and generative diversity to a new Pareto frontier. The overall loss function at step 𝑡 for personalized generation can be written as below:EQUATIONand the noisy latent 𝑧 𝑡 is iteratively updated at step 𝑡 byEQUATIONwhere 𝜂 𝑔 is the guidance ratio. 
experiments 4.1 experimental settings: 
 Dataset. We conduct experiments on DreamBench [23] , a dataset for text-to-image personalization performance evaluation. It consists of 30 subjects, including unique objects and pets such as backpacks, stuffed animals, dogs, cats, toys, etc. The subjects are separated into two categories, where 21 are objects and 9 are live subjects/pets. 25 text prompts are collected for testing generalization ability, including recontextualization and property modification. The evaluation suite requires generating 4 images for each subject and each prompt, amounting to a total of 3000 images. Evaluation Metrics. We follow DreamBooth [23] and employ evaluation metrics of DINO [3] , CLIP-I [20] , and CLIP-T scores. DINO and CLIP-I scores are utilized to evaluate subject fidelity, while CLIP-T score is utilized to evaluate image-text fidelity. The DINO score is the average pairwise cosine similarity between the ViT-S/16 DINO embeddings of the generated and real images. The CLIP-I score is the average distance of pairwise CLIP ViT-B/32 image embeddings of the generated and real images. DINO score is a preferred metric due to its sensitivity in capturing variations among subjects within the same class. These two combined reflect identity consistency. Lastly, the CLIP-T score represents the average cosine similarity between the CLIP embeddings of text prompt and image. CLIP-I measures image-text alignment among various contexts, thereby reflecting the model's generative diversity. Implementation Details. For our experiments, we set the guidance ratio to 10 and the binary mask threshold to 0.1. The weight of two losses alpha and beta are set to 10 and 100 respectively across all guidance steps. We perform guidance during the first 20 steps and"A V* dog with a blue house in the background" "dog" Subject Image w/o P&D w/ P&D w/o P&D w/ P&D w/o P&D w/ P&D "A cube shaped V* can" "can" "A V* bowl on top of a wooden floor" "bowl" "A V* clock on top of green grass with sunflowers around it" "clock" "A V* cat on top of a pink fabric" "cat" We provide qualitative comparisons of various text-to-image personalization methods including Textual Inversion [10] , Dream-Booth [23] and BLIP-Diffusion [16] with and without Pick-and-Draw in Fig. 4 . See the Appendix for a detailed description of the baselines. We observe that Pick-and-Draw consistently improves both subject fidelity and image-text fidelity on all three baselines. Textual Inversion falls short in preserving identity; DreamBooth and BLIP-Diffusion better preserve the subject identity, but tend to overfit and memorize the subject's pose and background, causing unsatisfactory alignment with the text prompt. Specifically, Dream-Booth fails to generate the blue house (1 st row) and the sunflowers (4 th row) and overfits the cat image (2 nd row), while BLIP-Diffusion memorizes the background of the forest (2 nd row) and the white fabric (3 rd row). All three methods fail to preserve appearance traits when performing geometric shape modification (5 th row). Furthermore, many synthesized subjects exhibit inconsistencies in certain details.In comparison, our proposed Pick-and-Draw (1) greatly enhances identity consistency, (2) improves generative diversity, and aligns better with the text prompt. can be attributed to the accurate semantics provided by appearancepicking guidance and layout-drawing guidance. The appearance picking guidance performs vigorous relaxed appearance transfer by calibrating the misalignment between the generated and reference appearance cues sets while the layout drawing guidance forces the personalized generation to an external layout, thereby mitigating the overfitting problem.In Tab. 2, we reproduce three personalization methods and study the impact of Pick-and-Draw quantitatively on the DreamBench dataset.In particular, we include vanilla Stable Diffusion as an additional baseline and report DINO and CLIP-I scores of real images as the The overall results are consistent with the qualitative findings, where Pick-and-Draw improves the performance of the three methods on all metrics. The remarkable performance gains on DINO and CLIP-I metrics can be attributed to the appearance picking guidance which performs accurate relaxed appearance transfer and greatly enhances identity consistency. The improved CLIP-T score can be attributed to the layout drawing guidance, which anchors the generated layout to an external prior and introduces more generative diversity, leading to better alignment with the text prompt. Additionally, in Fig. 5 we show per-subject metrics and observe that Pick-and-Draw significantly improves subject fidelity and improves image-text fidelity in most cases.Impact of activation selection. In Tab. 2, we quantitatively study the impacts of different activation selection strategies for the appearance picking guidance discussed in Section 3.2 for ablation. We choose Textual Inversion as the example baseline. We find that using activations of resolution 32 × 32 yields the best results on both subject fidelity and image-text fidelity. Specifically, activations of resolution 16 × 16 contain mainly layout information, thus unwanted layout leakage might happen and reduce the text prompt alignment. Activations of resolution 64 × 64 focus on fine-grained high-frequency details such as edges, which are not sufficient for local appearance transfer. In comparison, activations of resolution 32 × 32 encode rich semantic information and focus on different regions of the subject, facilitating the local appearance transfer and achieving the overall best result. This observation is consistent with the discussion in Section 3.1. Qualitative results are provided in the Appendix for intuitive visualization. Impact of two loss components. We show visual results for comprehension of our proposed appearance-aware loss ℓ app and layout-aware loss ℓ lay in Fig. 6 . We present two failure cases of DreamBooth, i.e. the attribute misbinding issue in the first row and the overfitting issue in the second row, and illustrate how the two losses work together to address these issues collectively. The ℓ app facilitates local appearance transfer, which makes the generated subjects (the 3-rd column) more aligned with the reference image. However, the appearance transfer may be unbounded and incorrect (the 1-st row) and novel layouts are not introduced (the 2-nd row). The ℓ lay (the 4-th column) constrains the appearance transfer within the subject region (the 1-st row) and introduces a novel layout (the 2-nd row), but without appearance picking guidance, the identity consistency of the new subject is not guaranteed. When combining the two losses for diffusion guidance, the generated outcomes (the last column) exhibit the best performance. Specifically, the appearance transfer is accurately bounded within the subject region, solving the attribute misbinding issue, while a novel layout is introduced with consistent subject identity, mitigating the overfitting problem. Choice between EMD and UREMD. We make a qualitative comparison between EMD and UREMD in the appearance-picking guide in Figure 8 . We notice that EMD loss may bring undesired artifacts and distorted subject posture, which confirms the discussion in Section 3.2. The intuition of our appearance-picking guidance involves treating the reference image as a palette and using its pixels to compose the synthesized subject. We aim to align the generative pixels with ones of the reference subject in a unidirectional manner, thus adopting UREMD instead of EMD.We also discuss the impact of different guidance timestep selections. Quantitative results and analysis are provided in Appendix.We directly apply Pick-and-Draw on vanilla Stable Diffusion and observe surprisingly favorable results in some cases. The visual results are presented in Fig. 7 . Without the strong subject prior of the fine-tuning-based personalization baselines, the appearance transfer still ensures identity consistency when the SD-generated Runtime cost. Pick-and-Draw uses energy function-based optimization during inference and has similar time complexity to other such methods, including Self-Guidance [8] , BoxDiff [33] , and Drag-onDiffusion [18] . In Table 3 , we compare the runtime between the methods mentioned above and our approach, tested on a single NVIDIA RTX-4090 GPU. Theoretically, the total number of forward and backward passes represents the runtime overhead. Our approach adopts a DDIM scheduler with 50 denoising steps, where we perform layout drawing guidance during the first 10 steps and appearance picking guidance between 10 and 20 steps. We optimize the noisy latent 3 times per step, which sums up to an extra 60 forward passes and 60 backward passes. Compared to the vanilla Stable Diffusion (50 forward passes), our method increases approximately two times overhead, which is worthwhile compared to repeated generation and cherry-picking (usually 10 times).SD Self-Guidance BoxDiff DragonDiffusion Pick-and-Draw  
conclusion: 
 In this paper, we propose Pick-and-Draw, a training-free semantic guidance approach for text-to-image personalization. We point out the prevalent overfitting issue of current methods: (1) they tend to memorize the few reference samples and struggle to generate diverse poses, views, and backgrounds of the subject; (2) they require careful hyperparameter tuning to achieve delicate results. To this end, we propose appearance-picking guidance and layout drawing guidance to boost performance for any personalized models with a single reference image. Qualitative and quantitative experiments demonstrate that Pick-and-Draw consistently improves identity consistency and generative diversity, pushing the trade-off between subject fidelity and image-text fidelity to a new Pareto frontier. 
 
 Appendix:  


Decision: This paper proposes Pick-and-Draw, a training-free semantic guidance approach for personalization methods, aiming to improve identity consistency and generative diversity. The approach consists of two components: appearance-picking guidance and layout-drawing guidance. It initially received 1 Weak Accept, 1 Borderline Accept, 1 Borderline Reject, and 1 Reject. Reviewers acknowledged the clarity of writing, flexible method design and the effectiveness of experiments. They also raised questions regarding comparative baselines, novelty, ablation studies, computational complexity, etc. The authors generally addressed those questions in the rebuttal. One reviewer raised score from Reject to Weak Reject, and the other three reviewers kept their scores. The AC finds the idea of this paper would be interesting to the community, and recommend to accept the paper.

Reviews: review: paper_summary: paper_summary: The paper proposes a training-free method to improve identity consistency and generative diversity in text-to-image models. The approach consists of two main components: appearance-picking guidance and layout-drawing guidance, which help generate diverse and contextually accurate images while preserving subject identity.
strength_weakness: strength_weakness: The paper is well-organized and written. The idea is novel and makes sense. The experiments are comprehensive and provide sufficient evidence to support the conclusions. The proposed method, Pick-and-Draw, effectively enhances identity consistency and generative diversity without requiring extensive retraining.
limitations: limitations: 1. To find the local region corresponding to a specific subject in the image, the authors set a threshold to filter the pixels and generate a binary image. How was the threshold of 0.1 selected? If the threshold is too large or too small, what impact will it have on appearance picking and the final results?
2.  Lack of sensitivity analysis for different thresholds in the Ablation study.
3.  Despite claiming to be "training-free," the method involves multiple optimization steps that require additional computational resources. The paper should provide specific training times and configurations.
paper_summary: paper_summary: The authors introduce Pick-and-Draw, an approach to text-to-image personalization that does not require training. They highlight a common problem with current methods: (1) they often memorize only a handful of reference samples and have difficulty generating a variety of poses, viewpoints, and backgrounds for the subject; (2) achieving nuanced results typically demands meticulous tuning of hyperparameters. To address these challenges, they present techniques for selecting appearances and drawing layouts to enhance the performance of personalized models using just one reference image.
strength_weakness: strength_weakness: 1. Pick-and-Draw is a training-free plug-and-play approach for boosting text-to-image personalization, which is flexible and applicable.

2. Pick-and-Draw leverages subject-relevant cross-attention maps to incorporate the appearance information from the reference image and contextual priors from the original diffusion into the personalized generative process.

3. This paper is well-organized, with clear grammar and expression, making it easy to understand.
limitations: limitations: 1. The novelty seems somewhat limited. Two loss function calculation methods are integrated into the diffusion process, which essentially constrains the attention maps to enhance generative capabilities.

2. How is generative diversity qualitatively manifested in Layout Drawing Guidance?

3. More state-of-the-art text-to-image personalization methods should be compared to demonstrate effectiveness, as only three are currently included.
paper_summary: paper_summary: This paper proposed Pick-and-Draw, a training-free semantic guidance approach to boost identity consistency and generative diversity for personalization methods. This method consists of two components: appearance-picking guidance and layout-drawing guidance and can be applied to any personalized diffusion model and requires as few as a single reference image.
strength_weakness: strength_weakness: 1. Good paper writing, solid work
2. The method is innovative and expandable to some extent, and can be combined with multiple personalized generation work to improve the original generation effect.
3. The details of the method are fully explored and analyzed in detail.
limitations: limitations: 1. The paper only optimizes the performance on the basis of related work, which is not innovative enough.
2. The test case list is of a single type, with little comparative work and do not include the latest performance related work. 
3. The complexity of the test text is not high enough
4. From the qualitative results, the ID fidelity of the generated image is still insufficient.
paper_summary: paper_summary: The author proposes Pick-and-Draw, a training-free semantic guidance approach to boost identity consistency and generative diversity for
personalization methods.
strength_weakness: strength_weakness: The method proposed by the author is effective.
The papaer is easy to follow.
limitations: limitations: 1、There are some shortcomings in the experiment. The baseline model used is too old. The authors need to add more comparative experiments.
2、Lack of innovation. The method proposed by the author is only a simple application of prompt2prompt[1].
[1] Hertz A, Mokady R, Tenenbaum J, et al. Prompt-to-prompt image editing with cross attention control[J]. arXiv preprint arXiv:2208.01626, 2022.