AP-Adapter: Improving Generalization of Automatic Prompts on Unseen Text-to-Image Diffusion Models

Authors: yuchen  fu
zhiwei  jiang
yuliang  liu
cong  wang
zexuan  deng
zhaoling  chen
qing  gu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 04:48:04

Abstract: Recent advancements in Automatic Prompt Optimization (APO) for text-to-image generation have streamlined user input while ensuring high-quality image output. However, most APO methods are trained assuming a fixed text-to-image model, which is impractical given the emergence of new models. To address this, we propose a novel task, model-generalized automatic prompt optimization (MGAPO), which trains APO methods on a set of known models to enable generalization to unseen models during testing. MGAPO presents significant challenges. First, we experimentally confirm the suboptimal performance of existing APO methods on unseen models. We then introduce a two-stage prompt optimization method, AP-Adapter. In the first stage, a large language model is used to rewrite the prompts. In the second stage, we propose a novel method to construct an enhanced representation space by leveraging inter-model differences. This space captures the characteristics of multiple domain models, storing them as domain prototypes. These prototypes serve as anchors to adjust prompt representations, enabling generalization to unseen models. The optimized prompt representations are subsequently used to generate conditional representations for controllable image generation. We curate a multi-modal, multi-model dataset that includes multiple diffusion models and their corresponding text-image data, and conduct experiments under a model generalization setting. The experimental results demonstrate the AP-Adapter's ability to enable the automatic prompts to generalize well to previously unseen diffusion models, generating high-quality images.

Main paper: 
 introduction: 
 Automatic Prompt Optimization (APO) in text-to-image generation aims to simplify the user input process while ensuring that the generated images exhibit both semantic consistency and aesthetic quality [9, 22, 24] . APO addresses the limitation of current text-to-image models, which often heavily rely on manually crafted prompts for generating high-quality images. The direct use of natural language descriptions as inputs often results in suboptimal outcomes.APO methods offer effective ways to directly utilize natural language descriptions for text-to-image diffusion models, which can be categorized into two paradigms: Text Optimization and Representation Optimization. Text Optimization entails generating refined prompts directly by fine-tuning the language model [8, 26] . On the other hand, Representation Optimization involves refining prompts within the feature space of the text encoder of the diffusion model [37] . As the advancements like LoRA [13] and DreamBooth [27] have simplified the training of diffusion models with personalized data, an increased number of diffusion models have emerged. The emergence of new models constantly requires us to conduct APO for new models and fresh prompt data. This process is overly labor-intensive and time-consuming. However, directly applying APO method specific to seen models to unseen models may result in poor generalization. As shown in the Figure 1a , Text optimization methods like MagicPrompt [7] and PromptPerfect [16] enhance image quality but sacrifice semantic consistency. Representation optimization methods, exemplified by SUR-adapter [37] maintain semantic consistency but show limited improvement in image quality.In this work, we address the model generalization scenario of Automatic Prompt Optimization (APO) and introduce the novel task of model-generalized automatic prompt optimization (MGAPO). The objective of MGAPO is to train an APO model on a limited set of known diffusion models and enable automatic prompt generalization for unseen diffusion models. MGAPO presents several challenges. First, implementing MGAPO requires incorporating extensive model information, which is absent in existing synthetic image datasets [37, 31] . Second, MGAPO must ensure both the semantic consistency and aesthetic quality of the generated images. Third, to achieve good generalization on unseen diffusion models, the model-related knowledge must be appropriately addressed during training, either retained or discarded.To train an APO model with generalization capabilities, we frame the task of MGAPO as a variant of domain generalization. We initially collected and annotated a multi-modal multi-domain dataset for training and testing. The dataset comprises natural language prompts, manually crafted prompts, high-quality images, and multiple stable diffusion (SD) checkpoints, serving as domain information. We then propose the Automatic Prompt Adapter (AP-Adapter), which is a two-stage framework. In the first stage, we leverage the ICL capability of large language models (LLMs) [23, 30] to generate automatic keyword prompts similar to manually crafted prompts [1, 34] . These prompts are then fed into the diffusion model's text encoder. In the second stage, we achieve model generalization using an adapter-based approach. The distinction between model-specific [37] and model-generalized adapters is illustrated in Figure 1b . We decode domain information from images [4, 35] and distill this information into domain prototypes. These prototypes serve as anchors to adjust the prompt representation, providing rich domain information for prompt adaptation. By further aligning with the representation specific to manual prompts, the adapted representation can simultaneously possess the ability for both semantically consistent and aesthetically pleasing generation.Our main contributions are as follows:• We explore model-generalized automatic prompt optimization (MGAPO), targeting the effectiveness of automatic prompts on unseen models, addressing a challenging aspect of domain generalization.• We propose AP-Adapter, the first method to address MGAPO, leveraging in-context learning of large language model for text optimization in the first stage and constructing an enhanced representation space by leveraging inter-model differences in the second stage. • We collect and annotate a multi-modal, multi-domain dataset for training and evaluation, including high-quality images, manually crafted prompts, natural language prompts, and model information. • Our extensive experiments on this dataset demonstrate Adapter's ability to enable the automatic prompts to generalize well to previously unseen diffusion models. 
related work: 
 Automatic Prompt Optimization for Text-to-Image Models.Prompt Optimization is essential for accurate semantic comprehension and high-quality image generation. Hao et al. [8] proposed a two-stage training approach. Initially, they supervised fine-tuned a language model using manual prompts. In the second stage, reinforcement learning, such as Proximal Policy Optimization (PPO), was employed to maximize the aesthetic and relevance scores of generated images. Rosenman [26] introduced the Neurologic Decoding method to generate more diverse and personalized prompts. Zhong et al. [37] improved image quality by aligning natural language descriptions with manually designed prompts in the feature space of CLIP's text encoder through the optimization of a lowparameter adapter. However, challenges in adapting pre-trained adapters to out-of-domain data and overlooking diversity across Stable Diffusion (SD) models persist. Noteworthy products, including Promptperfect and MagicPrompt, offer similar functionality, but our usage reveals limitations in their effective generalization across diverse SD models.Domain Generalization.Domain generalization aims to distill domain-invariant knowledge from a source domain, facilitating seamless model generalization to unknown domains. Advanced strategies leverage diversity within known domains to construct an enhanced feature space, aiding representation of unknown data using knowledge gained from known domains. DGSS [15] preserves style features from known domains as bases for generating new stylized images, promoting generalization to unknown domains. StyleAdv [5] introduces adversarial perturbations to image styles, enhancing model robustness to unknown image styles through adversarial training. Other works, such as [29, 3, 19] , fully exploit the adapter module for domain generalization on pre-trained models.3 MethodQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Limitations of our work are mentioned in Appendix 5Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
task definition: 
 Similar to domain generalization, MGAPO aims to achieve generalization to unseen models in the target domain by training the APO model on K source domain models. Formally, we introduce notations and formalize the task as follows. Given a diffusion model (i.e., Stable Diffusion 1.5)f (x; Θ) with K source checkpoints S = {Θ 1 , Θ 2 , ..., Θ K }, the k-th source checkpoint contains N k image-text pairs (n k i , m k i , I k i ) N k i=1, where n k i represents natural language descriptions, m k i denotes manually designed prompts, and I k i is the corresponding image. Each checkpoint has the same architecture as f (x; Θ) but with different parameters. Therefore each source checkpoint with its text-image pairs can be viewed as a source domain. In the testing phase, unseen target checkpoints T = {Θ K+1 , Θ K+2 , ..., Θ K+T } with natural language descriptions (n t i ) Nt i=1 are provided. Each target checkpoint can be viewed as a target domain. Our objective is to leverage source-domain checkpoints to create a pipeline capable of generalizing to any unseen target-domain checkpoints, generating coherent and aesthetically pleasing images for any given natural language descriptions.As illustrated in Figure 2 , our method comprises two stages of optimization for text prompts, corresponding to text-based optimization and representation-based optimization methods, respectively.In the first stage, we leverage the zero-shot reasoning capability of large language models f llm by employing in-context learning [21] to guide the generation of reliable prompts. Examples for in-context learning are obtained through similarity retrieval. In this step, natural language prompts are automatically translated into keyword prompts, which are then fed into the text encoder of the SD model f T clip to obtain text representations. In the second stage, we improve automatic keyword prompts by decoding domain-specific content from images and aligning it with manual prompts. To handle image unavailability during inference, we maintain prototypes in training to preserve domain-specific details. The adapter module then integrates information from text and prototypes, aligning its output representation with manual prompts in the feature space. Specifically, we freeze the parameters of the text encoder f T clip and denoiser f den in diffusion model. During training, for a specific domain's image-text pair (n k i , m k i , I k i ), we load the corresponding diffusion model parameters f (x; Θ k ) to extract features and denoise. The CLIP image encoder f I clip is introduced to extract image features, with the image encoder parameters also frozen.Previous work [21] has proved the demonstrations that are semantically similar to the test example exhibit higher performance in context learning. We employ a pre-trained sentence transformer [25] to calculate the cosine similarity between input and natural prompts in dataset. For each input natural prompts, we select the top k most similar demonstrations from dataset. These pairs of natural language prompts and manually crafted prompts, along with the input natural prompts, are embedded into the prompt template (depicted on the left side of Figure 2 ). The prompt template is then feed into GPT3.5 to obtain preliminary optimized keyword prompt. Compared to natural prompts, keyword prompts are closer in the semantic space of CLIP to manually crafted prompts. However, they are still insufficient for achieving multi-model generalization. Therefore, we introduce a prototype-based adaptive approach for automatic keyword prompts.In this section, we introduce how the adapter module leverages prototype learning to obtain model generalization capabilities.Alignment of Domain Prototypes.The parameter difference of models is the main factor leading to the domain shift, which limits the generalization ability of the optimized prompts. Parameter differences are encoded into the image during the process of generating images from text, resulting in the images exhibiting different domain variations. Therefore, we propose domain prototypes, which align with the domain-specific information decoded from the image.In detail, we employ intermediate features F k l ∈ R C×H×W with the l-th CLIP image encoder layer f I clip from image I k i . Here, C is the number of feature channels, while H and W denote the feature map's dimensions. Domain prototypes, denoted as {P dom k ∈ R Np×D } K k=1 , aim to align with the domain-specific information from image features to enhance automatic keyword prompt representations. N p denotes the number of prototypes, and D is the text token dimension.Identifying domain-specific information in image features is challenging due to the higher dimensionality of image features C compared to text features D. Inspired by DomainDrop [6] , we selectively drop non-informative channels in the feature space while retaining domain-specific information.To drop domain-insensitive feature channels, we use a domain discriminator f d (•) that predicts the image's domain based on intermediate layer features. This discriminator, comprising a global average pooling (GAP) layer and a fully-connected (FC) layer, outputs the domain label ŷ and computes cross-entropy loss for optimization:EQUATIONwhere K is the number of domains. Weights in the FC layer quantify each channel's contribution to domain discrimination. For the c-th channel in the feature map, its contribution is calculated as:EQUATIONwhere W ŷ ∈ R C is the weight in the fully connected layer when predicting the domain label ŷ correctly. A higher score indicates a greater contribution of the channel to domain discrimination. Subsequently, we can generate a binary mask M through the weighted random selection algorithm [12] .EQUATIONwherev c = r 1/scorec cis a computed by a random number r c ∈ (0, 1), Top(D) denotes the represents the top D items sorted by the values of v c , C and D are the dimensions of image features and text features, respectively.By dropping the domain-insensitive channels of the image feature channels, we enhance the representation of domain information in the features and ensure consistency in the number of feature channels between images and text. To prevent drastic changes in image domain during the training process, we employ momentum updates for smoothing:F drop k ′ = αF drop k + (1 -α)F drop k ′ (4)where α is a hyperparameter used to adjust the smoothness, F drop k and F drop k ′ are the image features after channel dropout and the features at the next step, respectively.We initialize the domain prototypes with standard normal distribution and then align the domain prototypes with the dropped image features:EQUATIONwhere KL(•, •) means KL divergence.Adaptation of Prompt Representation.We leverage all learned domain prototypes to achieve generalization to unseen models. We use the wasserstein distance dist(•, •) to measure the distance between prompt embeddings f T clip (f llm (n k i )) ∈ R L×D , where L is the prompt token length, and domain prototypes P dom s across all source domains:EQUATIONwhere d k represents the distance between the k-th domain prompt and the text token embedding. Then, we calculate the reciprocal of d k to determine the weight corresponding to each domain prompt:EQUATIONwhere the softmax operation ensures that the sum of all weights is 1. We performed a weighted sum of all domain prototypes across domains to obtain the final Prototype-Anchored Embedding:P a = K k=1 η k P dom k (8)Token embeddings of keyword prompts and mixed domain prototypes are concatenated for further training. We use aligned domain prototypes as anchors to adjust the representation of prompts, obtaining generalization capabilities on unseen domains.Alignment of Adapted Representation.We introduce a simple yet effective adapter g ada to aggregate and consolidate relevant information from all tokens, generating a conditional encoding representation for text-to-image generation. Comprising two transformer encoder layers and one linear layer, the adapter's linear layer parameters are initialized to 0 for stable training, following a proven approach [36] .The adapter's output is denoted as:EQUATIONOur goal is to make the representation of q s i as similar as possible to the representation of manually designed prompts. Therefore, we use mean squared error loss to align the representation of natural prompts and manual prompts.EQUATIONImage Denoising.During training phase, we need to ensure that the learned parameters do not compromise the denoising model's generation. We adopt the denoising loss from the diffusion model [28, 11] to guarantee this aspect.EQUATION)where ϵ θ represents the denoising model, ϵ represents the noise added to the input image x 0 , t represents the time step, and α t x 0 + β t ϵ is the result of adding noise to the image at time step t.Alternating Training.To optimize training efficiency, we employ two strategies. Firstly, considering the variation in diffusion model parameters during training, we streamline the process by sequentially training on all data from each domain within a single epoch. This minimizes the time spent on model switching. Secondly, to prevent interference between the learning of style prompts and adapters, we adopt an alternating training approach. Specifically, after completing an iteration across all data from each domain, we freeze either the adapter or domain prototypes parameters for the subsequent training epoch, repeating this cycle. Therefore, the loss used for training the domain prototypes is as follows:EQUATION) while the loss used for training the adapter is denoted as:EQUATION)where γ 1 , γ 2 , γ 3 , γ 4 are the loss coefficients, ranging from [0, 1].Our training pipeline is outlined in Appendix A. During the inference phase, we leverage the acquired domain prototypes and adapter for accurate predictions.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
dataset: 
 In this section, we introduce the process of collecting and creating the multi-modal multi-domain dataset for MGAPO. Data Cleaning.To ensure quality and semantic consistency, we employed Clipscore [10] and the aesthetic predictor 3 . Qualifying samples, totaling 25,395 image-text pairs, were selected based on aesthetic and clip scores surpassing their respective 25th percentiles.We enriched the dataset using LLaVA 1.5 [20] , a multimodal language model, to generate detailed natural language descriptions. We utilized instructions from [2] , guiding the generation with, "Describe this image and its style in a very detailed manner." Then, we selected the top 3 generated sentences with the highest relevance to the image content as natural language prompts.Our dataset contains a total of over 5000 checkpoints obtained through fine-tuning based on SD 1.5. However, most checkpoints have fewer than 10 corresponding image-text pairs. To ensure fairness in learning, we sorted the checkpoints based on the number of corresponding image-text pairs and selected the top 40 checkpoints as the source domain data and checkpoints ranked 41 to 100 as the target domain data. The source domain encompasses 7075 samples, whereas the target domain comprises 3064 samples. The remaining data is used as demonstrations for similarity retrieval in ICL-based prompt rewriting.During the training phase, we retrieve 5 pairs of natural language prompts and manually designed prompts as demonstrations for ICL from the dataset. We instruct GPT3.5 to generate 5 keyword prompts based on the input natural language example. By combining these 5 keyword prompts with their corresponding manual prompts and images, we augment the data used to train the modelgeneralized adapter.For the model's parameter settings, since the source domain data contains 40 checkpoints, the number of domain prototypes S is set to 40. The coefficients γ 1 , γ 2 , γ 3 , γ 4 for the loss functions are 0.01, 1.0, 0.001 and 1.0, respectively. Figure 9 shows the accuracy of the domain discrimination using the image features output from various layers of the CLIP image encoder. We use the image features from the 18-th layer, where the accuracy is the highest, as the input for the domain discriminator.The evaluation of our method on the target domain data encompasses both semantic consistency between text and images and image quality. For semantic consistency, we used Blipscore [17, 18] and followed the part on attribute binding in T2I-CompBench [14] . For image quality, we used aesthetic predictor, ImageReward [33] , and Human Preference Score(HPS) [32] . Furthermore, we recruited human evaluators to assess the aesthetics of the images and the consistency between text and images.For the metrics of color, shape, and texture, we use the model parameters in the target domain from our dataset and the text provided by T2I-CompBench as input. For other metrics, we use the model parameters and natural prompts in the target domain from our dataset. We compared various prompt optimization methods mentioned in Section 2. MagicPrompt [7] , a popular prompt generator for SD from Hugging Face, PromptPerfect [16] , a commercial tool for prompt optimization, Promptist [8] , and SUR-adapter [37] . Additionally, automatic keyword prompts and manual prompts are keyword based prompts generated by GPT-3.5 based on image descriptions and manually designed prompts in our dataset, respectively. Furthermore, we integrate our ICL-based prompt rewriting with the SUR-adapter for comparison in our experiments. Concretely, we augment the representation of automatic keyword prompts to the representation of natural language prompts from the SUR-adapter, using it as the generation condition.Table 1 shows that our proposed method outperforms most prompt optimization methods in semantic consistency and image quality. The results indicate that our proposed method exhibits good generalization performance in prompt optimization tasks. The APO method based on text optimization performs well on metrics that focus solely on image quality, such as AES scores and HPS, but falls short in terms of semantic consistency. In contrast, the SUR-adapter is able to maintain semantic consistency across different models, but does not perform well in terms of image aesthetics. Our method, however, can ensure both semantic consistency and aesthetics on unseen models. This conclusion is further strengthened by human evaluation results (Figure 4 ), where we see that our method consistently achieves higher win-rates than the other APO strategies.In Figure 3 , we showcase the generalization performance of different APO methods. The SD models in the figure are all out-of-domain models for our approach. It is evident that most APO methods produce satisfactory results within their training domains but struggle to generate good results on out-of-domain models. For instance, Promptist generated a "cat-human" image on the Dreamshaper model. PromptPerfect generates narrow-angled images on all of the out-of-domain models. In contrast, our method consistently produces accurate and aesthetically pleasing images across all out-of-domain models. We provide more comparative cases in Appendix C.3.To analyze the content contained in the trained domain prototypes, we artificially adjusted the weights η s in Equation 8 during the inference phase. We selected domain prototypes obtained from models with two different styles and treated P a as a different linear combination of the two. This is equivalent to adding domain information in different proportions to the generation conditions. In the Figure4a, we show the results of mixing animation style domain and the realistic style domain. From left to right, we gradually reduce the weight of anime and increase the weight of realism. In the Appendix C.4, we provide more cases for further analysis of domain prototypes.Effectiveness of Domain Prototypes.To evaluate the effectiveness of domain prototypes, we employed three settings. The first setting involves not using domain prototypes, only utilizing the adapter for fine-tuning. The second setting involves using a fixed-parameter fully connected layer to align the dimensions of image features and domain prototypes, which is equivalent to randomly selecting image feature channels. The third setting is the one where we drop domain-insensitive channel information. The evaluation results are presented in the Table 3 , showing that dropping domain-insensitive information effectively enhances the quality of the generated images.Effectiveness of Loss Functions.We conducted ablation experiments on the four loss functions we adopted. The results are shown in the Table 2 . The removal of the domain style distillation loss L a led to a significant decrease in the quality of the generated images. The denoising loss L d had a relatively minor impact on the generated results. Ultimately, the optimal performance emerged through the combination of all four losses, affirming the synergistic enhancement each proposed module provides. This underscores the significance of an integrated approach for superior model generalization, with each module playing a pivotal role. We visualize the ablation of loss functions in the Appendix C.5.We visualize the conditioned features used for text-to-image generation using the t-SNE algorithm. These include the original natural language descriptions of the images, keyword prompts optimized by the first-stage large language model, features output by the second-stage AP-adapter, and manually designed prompts.We use different colors to represent different domains, as shown in Figure 5 , reflecting the domain distinctiveness of various features. Both the natural language descriptions and keyword prompts have poor domain distinctiveness. We observe that the domain distinctiveness of the AP-adapter is even better than manually designed prompts. This is likely because we introduce image features that better represent domain differences. Therefore, the output of the AP-adapter performs better in domain distinctiveness than purely textual manual prompts.  
conclusion: 
 In this paper, we propose a new task, MGAPO, which identifies shortcomings of previous APO methods in generalization of unseen models. To address this challenge, we propose a two-stage approach called AP-Adapter that combines both text optimization and representation optimization methods. In the first stage, we leveraged the zero-shot reasoning capability of large language models to rewrite natural language prompts to keyword prompts. In the second stage, we introduce the domain prototypes, which decodes images to obtain representation for various domains. These prototypes are then combined with text representations to align with the representations of manually designed prompts. Rigorous experimentation and evaluation affirm the effectiveness of our proposed approach in optimizing prompts for out-of-domain models, achieving superior results without compromising the quality of the generated images. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: In the Section 1, we discuss the positive impact, and we believe this research has no negative social implications.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Although our data was collected from web crawling, we filtered out any unsafe data. Furthermore, our image data consists solely of synthetic images that do not contain any information that could pose harm to the real world.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All of the creators or original owners of assets used in our paper are cited properly.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: While our dataset offers valuable insights into the CIVITAI community, its relatively small size and focus on single-character images may limit the model's performance in handling more intricate and complex scenes. The necessity for an increasing number of domain prototypes as the dataset expands poses a challenge, leading to a growth in model parameters. Future work could explore leveraging parameter disparities among different models to tailor the APO optimization strategy, addressing this potential scalability concern.We illustrate our detailed training process in the Algorithm 1.Require: Get keyword prompt token embedding f T clip (f llm (n k i )) and manual prompt token embedding (f T clip (m k i )).Checkpoints S = {Θ 1 , Θ 2 , ..., Θ K }, datasets (n k i , m k i , I k i ) N k i=1 inGet domain prototypes P a by Equation 8.12:Concatenate domain prototypes and token embedding representations as adapter input.13:Obtain the output of the adapter g ada .14:if T is even then15:Calculate values of loss functions by Equation 13. 6 shows a sample from our dataset. We utilize the LLaVA-v1.5 13B model to generate natural language descriptions based on images. According to the query in the Figure 6 , LLaVa generates detailed descriptions about the image content and style, typically consisting of 5 to 6 sentences. To filter out the most relevant parts with the image, we calculate the CLIP similarity between each sentence and the image, selecting the top 3 sentences with the highest similarity as natural language prompts.Example of Stable Diffusion Models. In Figure 7 , we list some of the stable diffusion (SD) models we used. The models are ranked based on the number of samples they contribute to the dataset, with the top 20 models displayed. This ranking also reflects the popularity and generation quality of these models within the community. These models mainly come from two sources: one is models obtained by fine-tuning SD with custom data, and the other is models obtained by merging the weights of other models.Distribution of Sample Size. As shown in the Figure 8 , we present the distribution of samples for all models in the dataset. For clarity, we use a sample quantity of 35 as the threshold. It can be observed from the graph that the number of models with a large number of samples is relatively small, with many models having fewer than 10 samples. Therefore, we divide our dataset into three parts, as mentioned in Section 4.1. As stated, we assign models with a higher number of samples to the training and test sets, and the remaining samples were used as the demonstration library for Section 3.3. Due to the nature of our dataset, is entirely generated by diffusion models, it is unlikely to be used for systems that may violate personal privacy. Additionally, we have filtered out samples containing explicit and nude images from the data. Upon careful examination of our dataset, we believe it is unlikely to provide harmful information.In the ICL-Based Prompt Rewriting stage, we utilize the gpt-3.5-turbo API to generate automatic keyword prompts. It is noteworthy that, at this stage, the large language model generates automatic keyword prompts based on the demonstrations provided by ICL, including positive and negative prompts. Only positive prompts participate in training, while negative prompts serve as negative conditions for image generation during the testing phase. Here we provide the automatic keyword prompts generated from the natural language prompts in Figure 3 .Positive Prompts: masterpiece, best quality, vibrant colors, artistic atmosphere, black cat, yellow eyes, forest scene, visually striking, colorful background, unique appearance, eyecatching, animal focus, detailed, high resolution, solo, nature theme, visually appealing, beautiful, gorgeous Negative Prompts: lowres, bad anatomy, bad eyes, error, missing features, distorted features, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry, bad pictures, disconnected elements, unnatural colors, bad composition, overexposure, underexposure, monochrome, bad prompt, horror, easynegative. Different CLIP Image Encoder Layers for Domain Discrimination.To effectively acquire domain information, we tested the performance of the outputs of all layers in CLIP ViT-L/14 for domain classification, as shown in the Figure 9 . We found that as the layer index increases, the domain classification ability of image embeddings is not strictly monotonically increasing. This phenomenon may be due to domain information being more related to image style rather than semantic information.As the layers of the CLIP image encoder deepen, the output embeddings contain more semantic information.Different backbone of the CLIP image encoder.The evaluation results show the impact of different CLIP image encoders on the domain discriminator and the final generated results. Table 4 indicates that the choice of backbone has a minor influence on both domain classification and the quality of generated images. Additionally, it is observed that, across all backbones, features from deeper layers tend to be more advantageous for domain discrimination. We present more image generation results on out-of-domain models in Figure 10 and Figure 11 . The models in Figure 10 follow the setup from Figure 3 , while Figure 11 provides additional results from various models.According to the results, it can be observed that images generated using our method perform well in both semantic consistency and aesthetic quality across all models. In contrast, results generated by other methods are not stable across different models.We visualized the impact of domain prototypes on image generation in Figure 4a , Figure 12 , and Figure 13 . In Figure 4a and Figure 12 , we calculate P a using the following formula: In these cases, it is evident that domain prototypes possess clear semantic information, representing specific image styles such as cartoon, realistic, or 3D animation.However, domain prototypes do not always possess clear semantic information. As shown in Figure 13 , we gradually inject domain prototypes into the conditional information by adjusting the η i value of P dom i . It seems difficult to define the semantics represented by the domain prototypes in these cases. We believe this reflects a comprehensive factors such as image detail richness, photography techniques, lighting, image contrast, and image softness. In Figure 14 , we present additional results from the ablation experiments on the loss functions. Sequential ablation of each of the four loss functions reveals that the absence of any single loss function variably affects the image quality. A cat wearing a green jacket and sunglasses, giving it a stylish and unique appearance. The cat is positioned in the center of the image.A cartoon figure of a man wearing a suit and a hat. The man appears to be a caricature of Donald Trump, the former U.S. president.A plaster portrait from the Renaissance period. The plaster appears to be a man with long hair, facing the camera directly.w/ all ℒ Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.https://civitai.com/https://github.com/christophschuhmann/improved-aesthetic-predictor 


Decision: The final ratings are 5, 6, 6. However, it seems there are still some concerns as follow:

- **AP-Adaptor improvement over baselines are marginal**. Reviewer did mention that the performance is far from manual prompting. However, the AC's opinion is that the paper is on improving over automatic prompting so this is not such a big issue to him. Automatic prompt will not beat manual prompting at this time for sure but the research must still go on. So, the AC is not that concerned about achieving "commercial grade" for automatic prompting yet.
- **AP-Adaptor may not be truly out-of-domain capable**. This stems from the concerns that the approach is simply a result of a bunch of diversified checkpoints that allow AP-Adaptor to be multi-domain.
- **The proposed method is heavily dependent on the quality of the prompt generated by the LLM**. The authors tried to address these concerns by proposing two solutions but clearly the solutions are just hypothetical -- the AC don't find this very satisfying in his opinion.

Upon further discussions, reviewers remain a bit borderline on the second point and the third point. On (2), reviewers raised the point of whether the proposed approach is genuinely out-of-domain capable and whether the evaluation setting is entirely fair. 

After careful considerations, the AC urged the authors to address the second and third point strongly in the final manuscript and decided to accept the paper as the reviewers finally are slightly more positive than negative about the paper.

Reviews: review: paper_summary: paper_summary: 1. The authors propose model-generalized automatic prompt optimization (MGAPO), which targets the effectivenss of automatic prompts on unseen models.

2. The authors propose AP-Adapter which include in-context learning based prompt rewriting and prototype-based prompt adaptation.

3. The authors build a multi-modal, multi-domain dataset for training and evaluation.
strength_weakness: strength_weakness: 1. While the proposed method outperforms other baselines, its increased performance appears quite marginal, particularly considering the significant gap compared to the performance of manual prompts. I am curious about the main reasons for this substantial gap between automatic and manual prompts, and what fundamental limitations of automatic prompting hinder its improvement. Are there any promising directions to overcome these limitations? Can we consider the proposed method as addressing some of these limitations?

2. The proposed prototype-based adaptation utilizes the concept of DomainDrop in a reverse manner. DomainDrop aims to learn domain-invariant features by eliminating domain-sensitive information. Conversely, the proposed method removes domain-insensitive information to construct domain prototypes. However, in the context of domain generalization, domain-insensitive information is often considered highly valuable, as it contains information shared across different domains, which is crucial for generalizing to unseen domains. Therefore, I am curious if the process of removing domain-insensitive information in the proposed method could result in the loss of important information. Additionally, relying on domain prototypes may lead to situations where a model cannot properly handle data from completely new domains that cannot be adequately matched with existing domains. In this case, we need to consider the difficulty of continuously increasing the number of domain prototypes, as this would require an increase in model size as well.

3. While using different checkpoints, the data used for training and testing are consistent in that they are all collected from CIVITAI and fine-tuned based on SD1.5. However, the pre-trained data used for other baseline methods does not necessarily share these characteristics. I am curious if this could have introduced some biases in favor of the proposed method, leading to better results.
questions: questions: Please refer to the "Weaknesses" section.
limitations: limitations: This paper includes a specified section for limitations and the authors adequately addressed them.
paper_summary: paper_summary: The paper proposes model-generalized automatic prompt optimization (MGAPO), an automatic prompt optimization (APO) method which trains on a set of known models to enable generalization to unseen models during testing. MGAPO presents significant challenges, a perspective missing in previous methods. MGAPO includes a two-stage prompt optimization method. In the first stage, a LLM is used to rewrite the prompts and generate keyword prompts. In the second stage, the methods leverage inter-model differences (derived from varied checkpoints) to captures the characteristics of multiple domains and store them as domain prototypes. These prototypes serve as anchors to adjust prompt representations, enabling generalization to unseen models. The optimized prompt representations are subsequently used to generate conditional representations for controllable image generation.
strength_weakness: strength_weakness: - I believe the quality of writing can be improved. The language of paper seems unnecessarily complex and hard to comprehend at certain places especially the methodology section.
- The problem solved in the paper seems more of an incremental learning problem than domain generalization. The training and target domains differ only in terms of number of images seen. They do not differ in term of style or content as per paper. What are the views of authors about this?
- I can see in figure 2 that the UNET of diffusion model is freezed. If that's the case, then it renders the diffusion loss $\mathcal{L}_d$ in equation 11 useless. What exactly is being updated in equation 11? The $\theta$ parameters of UNET are freezed.
- In Fig 3 (row 3), the proposed method doesn't seem to produce optimal results with DreamShaper. The cat is supposed to be standing as per the prompt, however, the cat generated by the proposed method seems to be sitting. Compared to proposed method, the baselines seems to be working better in this case.
- Fig. 2 seems to be hastily drawn. There are many things which can improved in the figure (minor: "M" of LLM block is going out of the box). Also the font of in-context learning block is very small
- The gains in Table 1 seem to be average compared to baselines. In fact baselines are better in certain cases. Given the complexity of the proposed method, the gains are expected to be larger, which is not the case.

Overall, the authors are encouraged to answer the above queries and provide solid explanations
questions: questions: Please check the weaknesses section.
limitations: limitations: Authors have provided the limitations statement.
paper_summary: paper_summary: This paper proposes a new task called MGAPO, aimed at addressing the generalization problem of existing APO methods on unseen text-to-image generation models. To achieve this, the authors present a two-stage method called AP-Adapter. In the first stage, keyword prompts are generated through a large language model; in the second stage, an enhanced representation space is constructed by leveraging inter-model differences, and prompt representations are adjusted through domain prototypes to enable generalization to unseen models. Experimental results demonstrate that AP-Adapter can generate high-quality images on previously unseen diffusion models and outperform existing methods in both semantic consistency and aesthetic quality.
strength_weakness: strength_weakness: 1 The core of this paper lies in the computation of domain prototypes, which are influenced by CLIP. However, the authors did not conduct relevant ablation experiments.

2 Figure 4 includes human evaluation, which introduces a degree of subjectivity. Additionally, previous methods like sub-adapter did not conduct similar validation. Furthermore, the authors seem to lack detailed explanations regarding human validation in this paper.

3 The proposed method is relatively complex, introducing additional models such as a pre-trained LLM and CLIP for prototype computation, which increases overall complexity and training/inference time. The authors did not discuss these aspects.

4 AP-Adapter still requires a large number of manually designed prompts as references during training, which may increase the workload and complexity of data preparation.
questions: questions: see Weaknesses
limitations: limitations: The authors mentioned in the Limitation section that the main drawback of this paper is the small size of the dataset used, which is focused on single-character images. This limits the model's performance. Furthermore, the authors explained that as the dataset size expands, the number of domain prototypes might increase.

Comments: title: Please respond reviewer redB
comment: Thank you for the additional information. I will check and take it into account for the final recommendation.
Dear Reviewer,

We sincerely appreciate your valuable comments and the recognition of our rebuttal efforts. We would like to further clarify on your mentioned two issues:

1. **Regarding the good performance of manual prompting**, this can be attributed to three main factors: iterative selection of prompts (a repeated process of prompt adjustment and image generation), iterative selection of generated images (a repeated process of selecting images over multiple generations), and the possible use of auxiliary tools like ControlNet. In contrast, our method and other baselines do not involve such selection processes or the use of auxiliary tools.
 
2. **Regarding the fairness of the comparison**, this can be explained from three aspects:
   - Firstly, both our method and other baselines are trained on our collected data. For example, "Prompt Perfect" is trained with the data from the checkpoint DreamShaper.
   - Secondly, although all these data are collected from CIVITAI, they are uploaded by different users and generated using private fine-tuned checkpoints. Our experiments also show that data from different checkpoints exhibit significant domain differences and are not from the same distribution. Therefore, the data used for training and testing are not consistent.
   - Thirdly, our goal is to train an adapter on multiple checkpoints to achieve better generalization on unseen checkpoints. However, existing baseline methods cannot accomplish this and fail to produce an adapter with sufficient generalization capability. Therefore, the difference between the two training settings (as shown in Figure 1) also highlights how our method addresses the shortcomings of previous baseline methods, meeting the need to avoid retraining the adapter for new checkpoints.
Thank you for further clarification. I appreciate authors putting more work on explaining the things. I am increasing my score to weak accept. The paper is good however, the increments as compared to the baselines are not that great.
Dear Reviewer:

We sincerely appreciate your insightful comments and your willingness to increase the score. 
 
Regarding the issue of the "out-of-distribution nature of data samples" mentioned in Q2, we would like to provide some further clarification.

Our assumption that "relevant data from different checkpoints may have different data distributions" is not merely a conjecture but is based on our observations and experiments. The "out-of-distribution nature of data samples" can be summarized from three aspects:

1. **The prompts from different checkpoints have different distributions.** As shown in Figure 14, we attempted to map the prompts collected from various checkpoints into the same feature space and found that they are clustered according to the checkpoints, exhibiting domain distinctiveness.

2. **The images generated from different checkpoints exhibit domain differences.** As shown in Figure 8, we attempted to classify the source checkpoints of images collected from various checkpoints. We found that using only a simple linear classifier with pretrained CLIP encoder achieved around 95% accuracy. This indicates that images generated from different checkpoints have domain distinctiveness.

3. **The checkpoints fine-tuned with different data have shifted model parameters and should not be considered the same model.** As shown in Figure 3, the third column illustrates our attempt to apply "Prompt Perfect", which is specifically trained for the "DreamShaper" checkpoint, directly to four different checkpoints: "DreamShaper", "Realistic Vision", "SD V1.4", and "Lexica Aperture". We find that applying "Prompt Perfect" to the latter three unseen checkpoints can sometimes lead to semantic loss and worse performance. Other columns also show that the performance of APO methods on unseen checkpoints is sometimes worse. This demonstrates that checkpoints also exhibit domain distinctiveness.
Thank you to the authors for their responses. They have addressed some of my concerns. However, I still find the performance improvement of the proposed method over the baselines to be marginal, especially considering the significant difference when compared to manual prompting, particularly in terms of Blipscore and ImageReward. Most of all, I am concerned about the fairness of the comparison with other baselines, as the characteristics of effective prompts can be dependent on the choice of diffusion models and the content and style of the targeted images. In the present setting, the data used for training and testing are consistent in that they are all collected from CIVITAI and fine-tuned based on SD1.5. However, the proposed method is the only method optimized for this specific setting, whereas the other baselines are not. This could give a comparative advantage to the proposed method, making it difficult to assert that the evaluation is fair. For these reasons, I maintain my previous rating.
Reviewer redB:

Please let us know whether the authors have addressed your concerns.

Thanks.

-AC
Dear Reviewer,
  
We greatly appreciate your detailed review and insightful comments.
Thank you for your response. It addressed some of my concerns, and I am willing to increase my confidence level.
Dear Reviewer,
 
We sincerely appreciate your engagement in the discussion and your recognition of our rebuttal efforts. Thank you very much for your willingness to increase the score.
 
Since the rating in the original review has not yet been updated, we are unsure whether this is an oversight and would like to kindly confirm if the rating has been changed in the system.
 
Thank you once again for your time and valuable comments.
Thanks to authors for providing a detailed response to my queries. I am satisfied with most of the answers. However, regarding Q2, the out-of-distribution nature of data samples is based on the authors' assumptions which may or may not be true. Overall, I would like to increase my score to weak accept.
We appreciate your reply. We will address the concerns below.
 
  
Q2. How do authors ensure the style and content of target images are different?  
A2. In the dataset we collected, each image is annotated with the source checkpoint that generated it and the manually designed prompt used for its generation. The style and content of the image are determined by these two factors: the source checkpoint and the prompt, respectively.  
Regarding style, each checkpoint is uploaded to Civitai.com by users and is typically fine-tuned with users’ private data or data of particular interest. As a result, the fine-tuning data for different checkpoints can be viewed as coming from different data distributions. This variation in fine-tuning data leads to differing generative capabilities across checkpoints, resulting in differences in the styles of the images they generate.  
Regarding content, the prompt-image pairs we collected also come from different checkpoints, and these prompts are usually designed by users to highlight the generative capabilities of their respective checkpoints in specific domains. Figure 14 in the Appendix illustrates the differences among prompts, demonstrating that prompts from different checkpoints exhibit significant differences in content. As shown in Figure 14(d), different colors represent prompts from different checkpoints, and we can observe that the same colors tend to cluster together, indicating a clear semantic distinction between the prompts of different checkpoints.  
Therefore, in our dataset, the images generated from different checkpoints typically differ in both style and content and cannot be considered as coming from the same data distribution.
 
  
Q4. How do authors aim to improve or get rid of the dependency of the quality of automatic prompt? or is there any alternate way to get around such issues.  
A4. Our work primarily focuses on improving the generalization capability of automatic prompts across different checkpoints, with less emphasis on their generation. For the generation of automatic prompts, we only simply use a LLM without fine-tuning. To address the dependency of the quality of the automatic prompts, we can consider it from two perspectives:  
1. Use the natural language prompt as an extra input of the adapter. In this way, by introducing some specially-designed modules into adapter, the adapter can have additional capabilities for semantic completion of automatic prompts, thereby reducing its reliance on the first stage.  
2. Develop a more effective automatic prompt generation module. By jointly training this module with the adapter, we can enhance the module's ability to capture semantics, ensuring that the generated prompts have higher semantic quality while better aligning with the generalization requirements of the adapter.  
   
Q6. In terms of semantic consistency, the improvement of the method in this paper over SUR-adapter is modest.  
A6. We apologize for the earlier misreference; the correct figure is Figure 3.  
    Among the baselines, SUR-adapter mainly focuses on enhancing semantic consistency between prompts and images on a specific diffusion model by using an adapter. Our proposed method also employs an adapter but focuses on improving its generalization capability. Consequently, although our method offers only modest gains in semantic consistency compared to the SUR-adapter, it achieves a substantial improvement in image quality, as demonstrated in Table 1.  
   Among other baselines, PromptPerfect focuses on improving image quality and achieves an Aesthetic Score of 6.249, which is close to our score of 6.384. However, our method significantly outperforms PromptPerfect in semantic consistency.   
   In summary, our method achieves strong performance in both semantic consistency and image quality, whereas previous methods did not achieve both.
Thank you for your response. Based on the response of the authors, I have few more queries below:

- Q2: Authors mention that images sourced until checkpoint 40 are taken as source data and from checkpoint 41 as target data and they differ in content and style. How do authors ensure the style and content of target images are different. They are still from the same data distribution, hence more clarity is needed at this point.

- Q4:  Authors argue that LLMs inability to generate the correct pose for cat is the reason behind the incorrect pose of cat in the final image. This suggests that the proposed method is heavily dependent on the quality of the prompt generated by the LLM. How do authors aim to improve or get rid of this dependency? or is there any alternate way to get around such issues

- Q6: I think authors want to point out at Fig.3 (Fig 2 is a block diagram). Kindly request the authors to put correct references. The scores are still marginal. I do not see any steep jump in scores of semantic consistency. For example proposed method has a color score of 0.477 while SUR adapter has 0.472 which is very marginal increment. Same is with other cases as well.