AP-Adapter: Improving Generalization of Automatic Prompts on Unseen Text-to-Image Diffusion Models
Authors: Yuchen Fu
Zhiwei Jiang
Yuliang Liu
Cong Wang
Zexuan Deng
Zhaoling Chen
Qing Gu
Venue: NeurIPS 2024 poster (2024)
Keywords: text-to-image generation
diffusion model
Automatic Prompt Optimization
large language model
Primary area: diffusion_based_models

Abstract: Recent advancements in Automatic Prompt Optimization (APO) for text-to-image generation have streamlined user input while ensuring high-quality image output. However, most APO methods are trained assuming a fixed text-to-image model, which is impractical given the emergence of new models. To address this, we propose a novel task, model-generalized automatic prompt optimization (MGAPO), which trains APO methods on a set of known models to enable generalization to unseen models during testing. MGAPO presents significant challenges. First, we experimentally confirm the suboptimal performance of existing APO methods on unseen models. We then introduce a two-stage prompt optimization method, AP-Adapter. In the first stage, a large language model is used to rewrite the prompts. In the second stage, we propose a novel method to construct an enhanced representation space by leveraging inter-model differences. This space captures the characteristics of multiple domain models, storing them as domain prototypes. These prototypes serve as anchors to adjust prompt representations, enabling generalization to unseen models. The optimized prompt representations are subsequently used to generate conditional representations for controllable image generation. We curate a multi-modal, multi-model dataset that includes multiple diffusion models and their corresponding text-image data, and conduct experiments under a model generalization setting. The experimental results demonstrate the AP-Adapter's ability to enable the automatic prompts to generalize well to previously unseen diffusion models, generating high-quality images.

AP-Adapter: Improving Generalization of Automatic Prompts on Unseen Text-to-Image Diffusion Models
Yuchen Fu Zhiwei Jiang" Yuliang Liu Cong Wang
Zexuan Deng Zhaoling Chen Qing Gu
State Key Laboratory for Novel Software Technology, Nanjing University, China edu. en, jzw@nju edu.
yuchenfu@smail wjw..du..uu.....e... fyuliangliu, cw, dengzx, zhaol ingchnnn Mamaaln nju. edu. cn guq@nju guq@nju.edu.co edu. cn
Abstract
Recent advancements in Automatic Prompt Optimization (APO) for text-to-image generation have streamlined user input while ensuring high-quality image output. However, APO methods are trained assuming fixed text-to-image model, which is impractical given the emergence of new models. To address this, we propose novel task, gener--gnne automatic prompt optimization (MGAPO), which trains APO methods on a set of known models to enable generalization unseen models during testing. MGAPO presents significant challenges. First, we experimentally confirm the suboptimal performance of existing APO methods on unseen models. We then introduce two-stage prompt optimization method, AP-Adapter In the first stage, a large language model is used to rewrite the prompts. the second stage, we propose novel method to construct an enhanced repre- sentation space by leveraging inter-model differences. This space captures the characteristics of multiple domain models, storing them as domain prototypes. These prototypes serve as anchors to adjust prompt representations, enabling gener- alization to unseen models. The optimized prompt representations are subsequently used to generate conditional representations for controllable image generation. We curate a multi-modal, multi-model dataset that includes multiple diffusion models and their corresponding text- text-image data, and conduct experiments under model generalization setting. The experimental results demonstrate the AP-Adapter ability enable the automatic prompts to generalize well to previously unseen diffusion models, generating high-quality images.
1 Introduction
Automatic Prompt Optimization (APO) in text-to-image generation aims to simplify the user input process while ensuring that the generated images exhibit both semantic consistency and aesthetic quality [9 12 24). APO addresses the limitation of current text-to-image models, which often heavily rely on manually crafted prompts for generating high-quality images. The direct use of natural language descriptions as inputs often results in suboptimal outcomes.
APO methods offer effective ways directly utilize natural language descriptions for text-to-image diffusion models, which can be categorized into two paradigms: Text Optimization and Representation Optimization. Text Optimization entails generating refined prompts directly by fine-tuning the language model [8 26 On the other hand, Representation Optimization involves refining prompts within the feature space of the encoder of the diffusion model [37]
*Coresspndiigg author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Seon Medrik
Umaze Madel
horg
Texe
Le
Cxcodar
Adapner
Lee Quaky
herg
Dereicer
Denjiser
Poeze
(il Model-Specific Adapter for Prompt Representation
Natical
Senn Medch
Unsen Madal
Tout
Excodar
Aeepeer
Quality
Daroleer
Denglser
The recaci srligiaara of mag Capptaa
Representation
(a)
(b)
Figure 1: (a) Comparison of text-to-image generation among different APO methods on various unseen fine-tuned diffusion models. (b) Comparison between model- -specifc and generalend adapters under the model-gent model-generalized setting.
As the udvancements like LoRA [3 and DreamBooth [27] have simplified the training of diffu- sion models with personalized data, an increased number of diffusion models have emerged. The emergence of new models constantly requires us to conduct APO for new models and fresh prompt data. This process is overly labor-intensive and time-consuming. However, directly applying APO method specific to seen models to unseen models may result in poor generalization. As shown in the Figure Ta| Text optimization methods like MagicPrompt 77] and PromptPerfect [16] enhance image quality but sacrifice semantic consistency. Representation optimization methods, exemplified by SUR-adapter
37 maintain semantic consistency but show limited improvement in image quality.
In this work, we address the model generalization scenario of Automatic Prompt Optimization (APO) and introduce the novel task of model-generalizze automatic prompt optimization (MGAPO). The objective of MGAPO is to train an APO model on a limited set of known diffusion models and enable automatic prompt generalization for unseen diffusion models. MGAPO presents several challenges. First, implementing MGAPO requires incorporating extensive model information, which is absent in existing synthetic image datasets [37 31. Second, MGAPO must ensure both the semantic consistency and aesthetic quality of the generated images. Third, to achieve good generalization on unseen diffusion models, the model-related knowledge must be appropriately addressed during training, either retained or discarded.
To train APO model with generalization capabilities, frame the task of MGAPO as a variant of domain generalization. We initially collected and annotated a multi-modal multi-domain dataset for training and testing. The dataset comprises natural language prompts, manually crafted prompts, high-quality images, and multiple stable diffusion (SD) checkpoints, serving as domain information. We then propose the Automatic Prompt Adapter (AP-Adapter), which a two-stage framework. In
to generate automatic keyword prompts similar manually crafted prompts I 34 These prompts are then fed into the diffusion model's 's text encoder. In the second stage, we achieve model generalization using an adapter-based approach. The distinction between model-specific (37] and model-generalizze adapters is illustrated in Figure Tb We decode domain information from images 4 35 and distill this information into domain prototypes. These prototypes serve as anchors to adjust the prompt representation, providing rich domain information for prompt adaptation. By further aligning with the representation specific to manual prompts, the adapted representation can simultancously possess the ability for both semantically consistent and aesthetically pleasing generation.
the first stage, we leverage the ICL capability of large language models (LLMs) 23 30
Our main contributions are as follows:
We explore model-generalized automatic prompt optimization (MGAPO), targeting the effec- tiveness of automatic prompts on unseen models, addressing a challenging aspect of domain generalization.

---

We propose Ad-Adapeer, the first method to address MGAPO, leveraging in-context learning of large language model for text optimization the first stage and constructing an enhanced representation space by leveraging inter-model differences in the second stage.
We collect and annotate a multi-modal, multi-domain dataset for training and evaluation, in- cluding high-quality images, manually crafted prompts, natural language prompts, and model information.
Our extensive experiments on this dataset demonstrate Adapter's s ability to enable the automatic prompts generalize well previously unseen diffusion models.
Related Work
Automatic Prompt Optimization for Text-to-Image Models.Prompt Optimization is essential for
accurate semantic comprehension and high-quality image generation. Hao et al. [8] proposed two-stage training approach. Initially, they supervised fine-tuned a language model using manual prompts. In the second stage, reinforcement learning, such Proximal Policy Optimization (PPO),
was employed to maximize the aesthetic and relevance scores of generated images. Rosenman [26] introduced the Neurologic Decoding method to generate more diverse and personalized prompts. Zhong et al. (37] improved image quality by aligning natural language descriptions with manually designed prompts in the feature space of CLIP's text encoder through the optimization of low- parameter adapter. However, challenges in adapting pre-trained adapters to out-of-domain data and overlooking diversity across Stable Diffusion (SD) models persist. Noteworthy products, including Promptperfect and MagicPrompt, offer similar functionality, but our usage reveals limitations in their effective generalization across diverse SD models.
Domain pnteeraliaationnommae generalization aims to distill invariintn knowledge from a source domain, facilitating seamless model generalization to unknown domains. Advanced strategies leverage diversity within known domains to construct an enhanced feature space, aiding representation of unknown data using knowledge gained from known domains. DGSS (15) preserves style features from known domains as bases for generating new stylized images, promoting generalization to unknown domains. StyleAdv [5] introduces adversarial perturbations image styles, enhancing model robustness to unknown image styles through adversarial training. Other works, such as fully exploit the adapter module for domain generalization on pre-trained models.
29
3 Method
3.1
Task Definition
Similar to domain generalization, MGAPO aims to achieve generalization to unseen models in the target domain by training the APO model on K source domain models. Formally, we introduce notations and formalize the task as follows. Given a diffusion model (i.e., Stable Diffusion 1.5) [x;; with K source checkpoints S {O), O2,..., θx) ). the k-th source checkpoint contains N& image-text pairs (n: m. 11) No where nł represents natural language descriptions, mę denotes manually designed prompts, and Itt is the corresponding image. Each checkpoint has the same architecture as f(z; θ) but with different parameters. Therefore each source checkpoint with its text-image pairs can be viewed as a source domain. the testing phase, unseen target checkpoints
[0k+1, θ K+ ..., OK+T} } with natural language descriptions ) are provided. Each target checkpoint can be viewed as a target domain. Our objective is to leverage source-domain checkpoints to create a pipeline capable of generalizing any unseen target-domain checkpoints, generating coherent and aesthetically pleasing images for any given natural language descriptions.
3.2 Overview
As illustrated in Figure 2 method comprises stages of optimization for text prompts, corre- sponding to text-based optimization and representation- optimization methods, respectively.
In the first stage, we leverage the zero-shot reasoning capability of large language models fum by employing in-context learning [2] to guide the generation of reliable prompts. Examples for in-context learning are obtained through similarity retrieval. In this step, natural language prompts
3

---

Dicab Pridicka
Prototypa Alignnant
prico
Chaaed
Demnala Protstope
Aenerred Power
In- In-Context
Adapter
Natura Languge Prangt
LLM
Antuetatic Keyword Frocepe
Adapter
Texst Encoder
Masnal Kayward
+
Denciser of dowsh
Training Sumple the zomam
La
Figure 2: Illustration of our proposed AP-Adapeer
are automatically translated into keyword prompts, which are then fed into the text encoder of the SD model Se to obtain text representations.
In the second stage, we improve automatic keyword prompts by decoding domain-specific content from images and aligning it with manual prompts. To handle image unavailability during inference, we maintain prototypes in training to preserve fomain-speciion details. The adapter module then integrates information from text and prototypes, aligning its output representation with manual prompts in the feature space.
Specifically, we freeze the parameters of the text encoder Saip and denoiser Sden in diffusion model. During training, for specific domain's image-text pair (n.: m I we load the corresponding diffusion model parameters fa; θx) to extract features and denoise. The CLIP image encoder falip introduced extract image features, with the image encoder parameters also frozen.
3.3 ICL-Based Prompt Rewriting
Previous work 21 has proved the demonstrations that semantically similar the example exhibit higher performance in context learning. We employ pre-trained sentence transformer (25) calculate the cosine similarity between input and natural prompts dataset. For each input natural prompts, we select the top A most similar demonstrations from dataset. These pairs of natural language prompts and manually crafted prompts, along with the input natural prompts, are embedded into the prompt template (depicted on the left side of Figure 2). The prompt template is then feed into GPT3.5 to obtain preliminary optimized keyword prompt. Compared to natural prompts, keyword prompts are closer in the semantic space of CLIP to manually crafted prompts. However, they are still insufficient for achieving multi-model generalization. Therefore, we introduce prototype-based adaptive approach for automatic keyword prompts.
3.4 Prototype-Based Prompt Adaptation
In this section, introduce how the adapter module leverages prototype learning to obtain model generalization capabilities.
Aligament of Domain Prototypes.The parameter difference of models is the main factor leading to the domain shift, which limits the generalization ability the optimized prompts. Parameter differences are encoded into the image during the process of generating images from text, resulting in the images exhibiting different domain variations. Therefore, we propose domain prototypes, which align with the domain-specifie information decoded from the image.

---

In detail, we employ intermediate features Ff E RCxH with the 1-th CLIP image encoder layer faip from image 1 Here, C' is the number of feature channels, while H and W denote the feature map's dimensions. Domain prototypes, denoted {Prom [ Jk=1 with the domain-specific information from image features to enhance automatic keyword prompt representations. Nv denotes the number of prototypes, and D the text token dimension.
k== aim align
Identifying domain -emain-specific information in image features is challenging due to the higher dimension- ality of image features C' compared text features D. Inspired by DomainDrop [6. we selectively drop non-informative channels in the feature space while retaining domain-speciin information.
To drop cemmain-nssenstivve feature channels, we use a domain discriminator fa(-) that predicts the image's domain based on intermediate layer features. This discriminator, comprising global average pooling (GAP) layer and a fully-connected (FC) layer, outputs the domain label ý and computes cross-entropy loss for optimization:
K D logg(a(
Le
K
where K' is the number of domains. Weights in the FC layer quantify each channel's 's contribution to domain discrimination. For the e-th channel in the feature map, its contribution is calculated as:
ScoTE W GAP(F))
where W E RC is the weight in the fully connected layer when predicting the domain label 9 correctly, A higher score indicates greater contribution of the channel domain discrimination. Subsequently, we can generate a binary mask M through the weighted random selection algorithm
12
M. {o;
if e E "c)) otherwise
02,--,,
where U. = re scoree is a computed by random number r. E (0, 1). Top(D) denotes the represents the top D items sorted by the values of Ve. C and D are the dimensions of image features and text features, respectively.
By dropping the tation of domain information in the features and ensure consistency in the number of feature channels between images and text. To prevent drastic changes in image domain during the training process, employ momentum updates for smoothing:
channels of the image feature channels, enhance the represen-
Fddrop'
F. + (1 0.) Edrro'
where a is hyperparameter used to adjust the smoothness, Edrop and pdrop' are the image features after channel dropout and the features at the next step, respectively.
We initialize the domain prototypes with standard normal distribution and then align the domain prototypes with the dropped image features:
C. KL Edrop)
(5)
where KL(-.) means KL divergence.
Adaptation of Prompt Representation. We leverage all learned domain prototypes to achieve generalization to unseen models. We use the wasserstein distance dist(-.)) to measure the distance between prompt embeddings ((m((()))))) E IRLDD where L is the prompt token length, and domain prototypes pdom across all source domains:
dy = dist dist(Pdoom pdom SSr (fum (nt)))
where de represents the distance between the k-th domain prompt and the text token embedding. Then, we calculate the reciprocal of dę to determine the weight corresponding to each domain prompt:
7 = softmax 1 dr
(7)
5

---

where the softmax operation ensures that the sum of all weights is 1. We performed a weighted sum of all domain prototypes across domains obtain the final Prototype-Anchored Embedding:
K
Pa
Tl Pdom
(8)
Token embeddings of keyword prompts and mixed domain prototypes concatenated for further training. We use aligned domain prototypes as anchors to adjust the representation of prompts, obtaining generalization capabilities on unseen domains.
Alignment of Adapted Representation. We introduce a simple yet effective adapter Iada to aggregate and consolidate relevant information from all tokens, generating conditional encoding representation for text-to-image generation. Comprising two transformer encoder layers and one linear layer, the adapter's linear layer parameters are initialized to for stable training, following proven approach [36.
The adapter's output is denoted as:
Iada (cpnnca (n(T)m((())))) (n Pal)
Our goal is to make the representation of of as similar as possible to the representation of manually designed prompts. Therefore, mean squared error loss align the representation of natural prompts and manual prompts.
C,
(mi)) - )²²²²²
(10)
N
3.5 Model Training
Image Denoising.During training phase, we need to ensure that the learned parameters do not compromise the denoising model's generation. We adopt the denoising loss from the diffusion model
28
guarantee this aspect.
===Ble - eo (a + Bee, t, 2
(11)
where co represents the denoising model, c represents the noise added the input image zo. represents the time step, and ayTo + 3&e is the result of adding noise to the image at time step t.
Alternating Training.To To optimize training efficiency, employ two strategies. Firstly, considering the variation in diffusion model parameters during training, we streamline the process by sequentially training all data from each domain within single epoch. This minimizes the time spent on model switching. Secondly, to prevent interference between the learning of style prompts and adapters, we adopt an alternating training approach. Specifically, after completing an iteration across all data from each domain, we freeze either the adapter or domain prototypes parameters for the subsequent training epoch, repeating this cycle. Therefore, the loss used for training the domain prototypes as follows:
Epro La + 72h while the loss used for training the adapter is denoted as:
(12)
Lada Yahr + Ca.
(13)
where Yr, 72, 73, 74 are the loss coefficients, ranging from [0, 1].
Our training pipeline is outlined in Appendix A During the inference phase, we leverage the acquired domain prototypes and adapter for accurate predictions.
Experiments
4.1 Dataset
In this section, we introduce the process of collecting and creating the multi-modal multi-domain dataset for MGAPO.

---

Table Evaluation of the generated images on the target domain data using diverse prompt optimiza- tion methods. AKP denotes the automatic keyword prompt generated by ICL-based prompt rewriting.
Semantic Consistency
Image Quality
Methods
Color
Shape
Texture
Blipscore
Aesthetic Score
ImageReward
HPS
MagicPrompt
0.395
0.297
0.066
0.207
PromptPerfect
0.433
0.401
0,425
0.302
6.249
0.124
0.211
Promptist
0.439
0.398
0.427
0.292
6,000
0.089
0.202
SUR-adapter
0.472
0.413
0.449
0.325
6.009
0.286
0.198
AKP
0.456
0.401
0.437
0,305
6.113
0.253
0.213
AKP SUR-adapter
0.210
Ours
0.477
0.422
0.452
0.332
6.384
0.427
0.218
Manual Prompts (GT)
0.400
6.564
0.782
0.223
Data Collection. We sourced high-quality images and personalized SD checkponts from the CIVITAI ?
community We collected 47,695 image-text pairs gathered from various checkpoints, ensuring privacy protection. Further analysis of our dataset is provided in the Appendix B.1
Data Cleaning. To ensure quality and semantic consistency, we employed Clipscore [0] and the aesthetic predicto Qualifying samples, totaling 25,395 image-text pairs, were selected based on aesthetic and clip scores surpassing their respective 25th percentiles.
We enriched the dataset using LLaVA 1.5 [20]. a multimodal language model, to generate detailed natural language descriptions. We utilized instructions from (2] guiding the generation with, "De- scribe this image and its style in very detailed manner." Then, we selected the top 3 generated sentences with the highest relevance to the image content as natural language prompts.
Our dataset contains a total of over 5000 checkpoints obtained through fine-tuning based on SD 1.5. However, most checkpoints have fewer than 10 corresponding image-texI pairs. To ensure fairness in learning, we sorted the checkpoints based on the number of corresponding image-text pairs and selected the top 40 checkpoints
the source domain data and checkpoints ranked 41 to 100 as the target domain data. The source domain encompasses 7075 samples, whereas the target domain comprises 3064 samples. The remaining data used as demonstrations for similarity retrieval in ICL-based prompt rewriting.
4.2 Implementation Details
During the training phase, we retrieve 5 pairs of natural language prompts and manually designed prompts demonstrations for ICL from the dataset. We instruct GPT3.5 to generate 5 keyword prompts based on the input natural language example. By combining these 5 keyword prompts with their corresponding manual prompts and images, we augment the data used to train the model- generalized adapter.
For the model's parameter settings, since the source domain data contains 40 checkpoints, the number of domain prototypes S is set to 40. The coefficients Y1, 72, 73. 74 for the loss functions are 0.01, 1.0, 0.001 and 1.0, respectively. Figure/DJsove image features output from various layers of the CLIP image encoder. We use the image features from the 18-th layer, where the accuracy is the highest, the input for the domain discriminator.
the accuracy of the domain discrimination using the
4.3 Evaluation Metric
The evaluation of our method on the target domain data encompasses both semantic consistency between text and images and image quality. For semantic consistency, we used Blipscore [1]] and followed the part attribute binding in T21-CompBench [14]. For image quality, used aesthetic predictor, ImageReward [33), and Human Preference Score(HPS) [32]. Furthermore, we recruited human evaluators to assess the aesthetics of the images and the consistency between text and images.
For the metrics of color, shape, and texture, we use the model parameters in the target domain from our dataset and the text provided by T21-CompBench as input. For other metrics, we use the model parameters and natural prompts in the target domain from our dataset.
hharss//civviaa..comm
7

---

Natural Language
Magic Prompt
Prompt
Promptist
SUR-adapter
AKP
AKP+SUR- adapter
Ours
Realistic Visien
SD
Lexica Beexica Apertue DreemShaper
Natural Language Prompt: black with yellow yelloww colors, creating visually striking artistic atmosphere.
standing vibrant and folerfulforest The background
filled with variety
Figure 3: Results of diverse APO methods on MGAPO task. Each column displays images from the same APO method, and each row features images generated by the same SD model. APO methods with matching color and SD model share the same domain, indicating that the training data for the APO model was generated by that specific SD model.
4.4 Comparison to Other Methods
We compared various prompt optimization methods mentioned in Section 2] MagicPrompt 77. popular prompt generator for SD from Hugging Face, PromptPerfect [16). commercial tool for prompt optimization, Promptist [8, and SUR-adapter [37]. Additionally, automatic keyword prompts and manual prompts are keyword based prompts generated by GPT-3.5 based on image descriptions and manually designed prompts our dataset, respectively. Furthermore, we integrate our ICL-based prompt rewriting with the SUR-adapter for comparison in experiments. Coneretely, augment the representation of automatic keyword prompts to the representation natural language prompts from the SUR-ad UUd-adapeer, using it the generation condition.
Table T shows that our proposed method outperforms most prompt optimization methods in semantic consistency and image quality. The results indicate that our proposed method exhibits good gener- alization performance in prompt optimization tasks. The APO method based on text optimization performs well on metrics that focus solely on image quality, such as AES scores and HPS, but falls short in terms of semantic consistency. In contrast, the SUR-adapter able to maintain semantic consistency across different models, but does not perform well in terms of image aesthetics. Our method, however, can ensure both semantic consistency and aesthetics on unseen models. This conclusion is further strengthened by human evaluation results (Figure 4), where that our method consistently achieves higher win-rates than the other APO strategies.
4.5 Case Study
In F we showcase the generalization performance of different APO methods. The SD models in the figure are all out-of-domain models for our approach. is evident that most APO methods produce satisfactory results within their training domains but struggle to generate good results on out-of-domain models. For instance, Promptist generated "cat-human" image on the Dreamshaper model. PromptPerfect generates aarrow-aggled images all of the out-of-domain models. In contrast, our method consistently produces accurate and aesthetically pleasing images across all out-of-domain models. We provide more comparative cases in Appendix C
trained domain prototypes, we artificially adjusted the weights 7) in Equation 8 during the inference phase. We selected domain prototypes obtained from models with two different styles and treated Pa as different linear combination of the two. This equivalent to adding domain information in different proportions to the generation conditions. In the Figurd4a
To analyze the content contained

---

13
2in
(a)
(b)
(c)
Figure 4: (a) Linear combinations of domain prototypes from anime style models and domain prototypes from realism style models. The blending ratio changes from left to right. (b) Human Evaluation for Aesthetics. (c) Human Evaluation for Text-Image Alignment.
Table 2: Ablation study of the loss functions.
Table 3: Effectiveness of domain prototypes.
Cr Ca
BlipScore AcsScore ImageReward
Color BlipScore AesScore ImageReward
0,313
6.139
0.258
random prototypes
0,462
0.328
6,183
0.373
0.324
6.275
0.384
w/o prototypes
0.451
0.302
6.140
0.258
0.332
6,384
0.427
show the results of mixing animation style domain and the realistic style domain. From left to right, we gradually reduce the weight of anime and increase the weight of realism. In the Appendix provide more cases for further analysis of domain prototypes.
C.4
4.6
Ablation Study
Effectiveness of Domain Prototypes. To evaluate the effectiveness of domain prototypes, we em- ployed three settings. The first setting involves using domain prototypes, only utilizing the adapter for fine-tuning. The second setting involves using a exed-parameter fully connected layer align the dimensions of image features and domain prototypes, which equivalent to randomly selecting image feature channels. The third setting is the one where we drop comainninsenssiive channel information. The evaluation results are presented the Table 3 showing that dropping comain-insensieee information effectively enhances the quality of the generated images.
Effectiveness of Loss Functions. We conducted ablation experiments the four loss functions adopted. The results are shown the Table The removal of the domain style distillation loss fa led to a significant decrease in the quality of the generated images. The denoising loss La had a relatively minor impact on the generated results. Ultimately, the optimal performance emerged through the combination of all four losses, affirming the synergistic enhancement each proposed module provides. This underscores the significance of an integrated approach for superior model generalization, with each module playing a pivotal role. We visualize the ablation of loss functions in the Appendix C.5
4.7 Visualization of Conditioned Features
We visualize the conditioned features used for text-to-image generation using the t-SNE algorithm. These include the original natural language descriptions of the images, keyword prompts optimized by the first-stage large language model, features output by the second-stage AP-adapter, and manually designed prompts.
We use different colors represent different domains, as shown in Figure [5] reflecting the domain distinctiveness of various features. Both the natural language descriptions and keyword prompts have poor domain distinctiveness. We observe that the domain distinctiveness of the AP-adapter even better than manually designed prompts. This is likely because we introduce image features that better represent domain differences. Therefore, the output of the AP-adapter performs better in domain distinctiveness than purely textual manual prompts.

---

Aiyoiriigeeii nowt ACRUNIE
Iroe
And
(a) Natural language deseriptions.
(b) Keyword prompts.
Bor SALLES dlyooeeeeee
-20
(c) AP-adapter
(d) Manually designed prompts.
Figure 5: Visualization of text-conditioned domain distinctiveness. (a) Natural language description of the image.
(b) Keyword prompts output by the first-stage large language model. (c) Features output by the second-stage AP-adapter. (d) Manually designed prompts.
5 Conclusion
In this paper, we propose a new task, MGAPO, which identifies shortcomings of previous APO methods in generalization of unseen models. To address this challenge, we propose two-stage approach called AP-Adapter that combines both text optimization and representation optimization methods. In the first stage, we leveraged the zero-shot reasoning capability of large language models to rewrite natural language prompts to keyword prompts. In the second stage, we introduce the domain prototypes, which decodes images obtain representation for various domains. These prototypes are then combined with text representations to align with the representations of manually designed prompts. Rigorous experimentation and evaluation affirm the effectiveness of proposed approach models, achieving superior results without compromising
in optimizing prompts for the quality of the generated images.
Limitations
While our dataset offers valuable insights into the CIVITAI community, its relatively small size and focus on single-character images may limit the model's performance in handling more intricate and complex scenes. The necessity for an increasing number of domain prototypes as the dataset expands poses challenge, leading growth in model parameters. Future work could explore leveraging parameter disparities among different models to tailor the APO optimization strategy, addressing this potential scalability concern.
10

---

Acknowledgement
This work is supported by the National Natural Science Foundation of China under Grants Nos. 61972192, 62172208, 61906085. This work is partially supported by Collaborative Innovation Center of Novel Software Technology and Industrialization. This work is supported by the Fundamental Research Funds for the Central Universities under Grant No. 14380001.
References
[1] Betker, J., Goh, G., Jing, L., Brooks, T., Wang. Li, L., Ouyang, L., Zhuang, J., Lee, J., Guo, Y., al. (2023). Improving image generation with better captions. Computer Science. https://edn. openai. com/papers/dall-e. pdf, 2(3), 8.
[2] Chen, J. Yu, J., Ge, C., Yao, L., Xie, E., Wu, Y., Wang, Z., Kwok, J., Luo, P., Lu, H., et al. (2023). Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arX1v:2310.00426.
[3] Cui, Y., Yu, Z., Cai, R., Wang, X., Kot, A. C.,, and Liu, L. (2023). Generalized few-shot continual learning with contrastive mixture of adapters. arXiv preprint 1rXiv:2302.05936
[4] Ding, Y., Tian, C., Ding, H., and Liu, L. (2024). The clip model is secretly an image-to-pret -o-propptt converter. Advances Neural Information Processing Systems, 36.
[5] Fu, Y., Xie, Y., Fu, Y., and Jiang, Y.-G. (2023). Styleadv: Meta style adversarial training for cross-domain few-shot learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14575-24584
[6] Guo, J., Qi, L., and Shi, Y. (2023). Domaindrop: Suppressing domain-sensitive channels for domain generalization. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 191 19114499
[7] Gustavosta (2022). Magiccpompi-stible-dfffiuiiiii httper//hagginggaoo.ccoooootstss Paggiiprompp-ttabeeeeee Diffusion
[8] Hao, Y., Chi, Z., Dong. L., and Wei, (2022). Optimizing prompts for text-to-image generation. arXiv preprint 1rX.1:22/2.09611.
[9] He, Y., Robey, A., Murata, N., Jiang, Y.. Williams, Pappas, G. J., Hassani, H., Mitsufuji, Y., Salakhutdinov, R., and Kolter, J. Z. (2024). Automated black-box prompt engineering for personalized lext-to-image generation. arXiv preprint arXiv:2403. 19103.
[10] Hessel, J., Holtzaan, A., Forbes, M., Bras, I.,, and Choi, Y. (2021). Clipscore: A reference- free evaluation metric for image captioning. arXiv preprint arXiv: 104.0::800.
[11] Ho, J., Jain, A., and Abbeel, P. (2020). Denoising diffusion probabilistic models. Advances in neural information processing systems, 33, 6840-6851.
[12] Hou, S. and Wang, Z. (2019). Weighted channel dropout for regularization of deep convolutional neural network. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 8432.
[13] Hu, J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2021). Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0090
[14] Huang, K., Sun, K., Xie, E., Li, Z., and Liu, X. (2023a). 12i-compbench: compre- hensive benchmark for open-world compositional text-to-image generation. arXiv preprint arXiv:2307.06350.
[15] Huang. W., Chen, Y., Li, C., Song. F., Yan, Y., and Xiong, Z. (2023b). Style pro- jected clustering for domain generalized semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3061-3071.
[16] JinaAl (2023). Promptperfect. https://promptperrete wwt.aaaiaaaannn.... prompts

---

[17] Li, D., Xiong, C., and Hoi. S. (2022a). Blip: Bootstrapping language inguage-image pre- vre-iinining for unified vision-lane Saiin-la understanding and generation. In International Conference on Machine Learning, pages 12888-12900. PMLR.
(18) Li, J., Li, D., Savarese, S., and Hoi, S. (2023). Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. arXiv preprini urXiv:2301.12591
[19] Li, W.-H., Liu, X., and Bilen, H. (2022b). Cross-domain few-shot learning with task-specific adapters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pages 7161-7170.
[20] Liu, H., Li, C., Li, Y., and Lee, J. (2023). Improved baselines with visual instruction tuning. arXiv preprint 10..0374.
[21] Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L.,, and Chen, W. (2021). What makes good in-context examples for gpt-3? arXiv preprint 0rXiv:2101.06800
[22] Mañas, O., Astolfi, P.,, Hall, M., Ross, C., Urbanek, J., Williams, A., Agrawal, A., Romero- Soriano, A., and Drozdzal, M. (2024). Improving text-to-image consistency via automatic prompt optimization. arXiv preprint (1T::2203.17844
[23] Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. (2022). Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint 1282
[24] Mo, W., Zhang, T., Bai, Y., Su, B., Wen, J.-R., and Yang, Q. (2024). Dynamic prompt optimizing for text-to-image generation. arXiv preprint arXiv:2404.04095.
[25] Reimers, N. and Gurevych, I. (2019). Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908. 10084.
[26] Rosenman, S., Lal, V., and Howard, P. (2023). Neuroprompts: An adaptive framework to optimize prompts for text-to-image generation. arXiv preprint 22229.
[27] Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. (2023). Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500-22510.
[28] Song, J., Meng, C., and Ermon, S. (2020). Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502.
[29] Vu, T., Khadivi, S., Phung, D., and Haffari, G. (2022). Domain generalisation nmt: Fusing adapters with caaee-one-domai--ee training. In Findings of the Association for Computational Linguistics: ACL. 2022, pages 582-588.
[30] Wang, X., Thu, W., Saxon, M., Steyvers, M., and Wang, W. Y. (2023a). Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning. In seirth-e Conference Neural Information Processing Systems.
[31] Wang, Z., Montoya, E., Munechka, D.,, Yang. H., Hoover, B., and Chau, P. (2023b). Diffusiondb: A large-scale prompt gallery dataset for text-to-image generative models. In Annual Meeting of the Association for Computational Linguistics.
[32] Wu, X., Sun, K., 7hu, F., Zhao, R., and Li, H. (2023). Better aligning text-to-image models with human preference. arXiv preprint XxXi:23303.14422
[33] Xu, J., Liu, X., Wu, Y., Tong, Y., Li, Q., Ding, M., Tang, J., and Dong, Y. (2023). Imagere- ward: Learning and evaluating human preferences for text-to-image generation. arXiv preprint arXiv:2304.05911 059777
[34] Yang, I.,, Yu, Z., Meng, C., Xu, M., Ermon, S., and Cui, B. (2024). Mastering text-to- image diffusion: Recaptioning, planning, and generating with multimodal lms. arXiv preprint ar011:2401.11708.
12

---

[35] Ye, H., Zhang, J., Liu, S., Han, X., and Yang, W. (2023). Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721.
[36] Zhang, L., Rao, A., and Agrawala, M. (2023), Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1336-3847.
[37] Zhong, S., Huang, Z., Wen, W., Qin, J., and Lin, L. (2023). Sur-adapter: Enhancing text-to- image pre-trained diffusion models with large language models, arXiv preprint arXiv:2305.05/880
13

---

A Supplemental Algorithm Information
We illustrate our detailed training process in the Algorithm
Algorithm Training pipeline
Require: Checkpoints S = { 22, θ ), datasets (m m ) in the source domain, faip
and fdem. of the stable diffusion model, fais image encoder, fum large language model, domain prototypes {P& ]k= (61), adapter (=
Ensure: Trainable domain prompt parameters Φ1 and adapter parameters Φ2-
1: Initialize: Set ful and ddip to pretrained parameters.
2: for T in training epochs do
3: if T is even then
4: Freeze domain prototypes parameters d1-
5: else
6: Freeze adapter parameters Ø2. end if
7:
8:
for dataset of domain in datasets do
9:
Set feii and fden to O₂ parameters.
10:
Get keyword prompt token embedding faip (fum (n) and manual prompt token embedding
(Jaip (m²)).
11:
Get domain prototypes Pa by Equation 8
Concatenate domain prototypes and token embedding representations as adapter input.
13:
Obtain the output of the adapter Jada.
T is even then
15:
Calculate values of loss functions by Equation 13
else
17:
Calculate values of loss functions by Equation 12
18:
end if
19:
Update all learnable parameters.
20:
end for
21:
end for
B Supplemental Dataset Information
B.1 Dataset Analysis
Example of Data. T sample from our dataset. We utilize the LLaVA-v1.5 13B model to generate natural language descriptions based on images. According to the query in the Figure 6 LLaVa generates detailed descriptions about the image content and style, typically consisting of 5 to 6 sentences. To filter out the most relevant parts with the image, we calculate the CLIP similarity between each sentence and the image, selecting the top 3 sentences with the highest similarity as natural language prompts.
Example of Stable Diffusion Models. In Figure we list some of the stable diffusion (SD) models we used. The models are ranked based on number of samples they contribute the dataset, with the top 20 models displayed. This ranking also reflects the popularity and generation quality of these models within the community. These models mainly come from two sources: one is models obtained by fine-tuning SD with custom data, and the other is models obtained by merging the weights of other models.
Distribution of Sample Size. As shown in the Figure g we present the distribution of samples for all models in the dataset. For clarity, we use sample quantity of 35 as the threshold. It can be observed from the graph that the number of models with a large number of samples is relatively small, with many models having fewer than 10 samples. Therefore, we divide our dataset into three parts, as mentioned in Section 4.1 As stated, we assign models with higher number of samples to the training and test sets, and the remaining samples were used as the demonstration library for Section
3,3
14

---

"Describe this image and style in very detailed
LLaVA
High quality image
Natural Lauguage Prompt: a beautifully drawn illustration of'a woman dressed in a medieval outfi, possibly inspired by charucter Zelda from Legend Zelda video game series. She standing river, and appears he looking up. The woman is weuring long dress, and her attire includes headdress.
Mannually Designed Prompt:
Pusidiverr((Syylel))) (flat color, masterpicee:1.2. best quality) Compossttiioon { Igirl, solo, cute, mid shot from side, kneeling, looking up (Hair))) { long blonde hair, wavy hair Apppaararaacccll)) mature female, princess zelda, nintendo, the legend zeldd, oot, young zekla, slim, slender, fit, small breusts, flat_ chest, blue eyes (Locatioo)) { outdoors, dreamy autum forest on hills landscape with river in the valley, crystal blue sky. birds flying away }. 'nlo(orass)) <-orayoung zekda_ v1:8= }"
veggative: (worst quality, low quality:1.4) 4), by bad-artist-anime, by bad-artist, bad- bands-5:1.3, bud prompt version2, EasyNegatvve, ng deepnegutive vl___11_ verybadinagenggaaive.. v1.2
Stable Diffusion
Model name: ghostmix .12
Figure 6: A sample from the dataset, each containing natural language prompt generated by LLa LLaVa, a manually designed prompt, high-quality image, and model information.
B.2 Ethical Statement
Due to the nature of our dataset, which is entirely generated by diffusion models, it is unlikely to be used for systems that may violate personal privacy. Additionally, we have filtered out samples containing explicit and nude images from the data. Upon careful examination of our dataset, we believe is unlikely to provide harmful information.
C Supplemental Experiments
C.1 More Implementation Details
In the ICL-Based Prompt Rewriting stage, we utilize the gpt-3.5-turbo API to generate automatic keyword prompts. It is noteworthy that, at this stage, the large language model generates automatic keyword prompts based the demonstrations provided by ICL, including positive and negative prompts. Only positive prompts participate in training, while negative prompts serve as negative conditions for image generation during the testing phase. Here we provide the automatic keyword prompts generated from the natural language prompts in Figure
Positive Prompts: masterpiece, best quality, vibrant colors, artistic atmosphere, black cat, yellow cyes, forest scene, visually striking, colorful background, unique appearance, eye- catching, animal focus, detailed, high resolution, solo, nature theme, visually appealing, beautiful, gorgeous
Negative Prompts: lowres, bad anatomy, bad eyes, error, missing features, distorted features, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry, bad pictures, disconnected elements, unnatural colors, bad composition, overexposure, underexposure, monochrome, bad prompt, horror, easynegative.
15

---

revAnimated v122
majicmixRealistic. v6
ihilloutmix_iPPrueeeffi
deliberate .v2
CounterfeitV30_v45
ghostmixxv22Bakedvaa
endjourney_.. .1.1
majicmixRealistic, v5
yesmix v15
dreamshaper__
majicmixRRalistic. _4
photon_ v1
icbinpiCantBellevellss final
revAnimated, v11
ghostmix_vI2
majicmixSombre v20
lyriel_v16
dreamshaper_6a2 Baakeeddveeeeee
Xxmix9realistic.y.
Figure 7: The top 20 models by sample size, along with the respective proportions.
0.30
0.022
0.5
0.5
Frsquency
0.3
2.00
2.04
0.8
15
20
25
30
03002
200
400
600
200
1000
3000
Num of l'Smpples
Num SaSmples
(a) The distribution of model sample size within the range of 0 and 35.
(b) The distribution of model sample size with more than 35.
Figure 8: The distribution of model sample size.
In the Prototype-Basee -asee Prompt Adaptation stage, all models are trained on two NVIDIA RTX 3090 GPUs, with steps set 10000, batch size set to 16, and image resolution set 512. Standard data augmentation techniques, such as normalization, resizing, and horizontal flipping, are applied during training. As for the platform to implement our network, we use PyTorch 2.1.
C.2 Module Analysis
Different CLIP Image Encoder Layers for Domain Discrimination.Ts To effectively acquire domain information, tested the performance of the outputs of all layers in CLIP ViT-L/14 for domain
16

---

0.6
Arcuracy
0.5
0.4
0.3
0.2
12
13
14
15
16
17
18
10
20
Index of Luyer
Figure Domain discrimination accuracy at different layers of the CLIP image encoder.
classification, as shown in the Figure 9
We found that as the layer index increases, the domain classification ability of image embeddings is not strictly monotonically increasing. This phenomenon may be due to domain information being more related to image style rather than semantic information. As the layers of the CLIP image encoder deepen, the output embeddings contain semantic information.
Different backbone of the CLIP image encoder.The The evaluation results show the impact of different CLIP image encoders on the domain discriminator and the final generated results. Table 4 indicates that the choice of backbone has minor influence both domain classification and the quality of generated images. Additionally, it observed that, across all backbones, features from deeper layers tend to be more advantageous for domain discrimination.
Table 4: Evaluation results of the impact of different backbones of the CLIP image encoder the generated images. Acc refers to the classification accuracy of the domain discriminator.
Acc
BlipScore
AesScore
6.282
RN101
0.954
0.318
6.351
VIT-L/14
0.960
0.332
6,384
C.3 More Cases for Comparison
We present more image generation results on out-of-domaad models in Figure 10 and Figure II The models Figure 10 follow the setup from Figure while Figure II provides additional results from various models.
According to the results, it can be observed that images generated using method perform well in both semantic consistency and aesthetic quality across all models. contrast, results generated by other methods stable across different models.
C.4 More Cases for Domain Prototypes Analysis
We visualized the impact of domain prototypes on image generation in Figure 4m Figure and calculate P. using the following formula:
Figure 13 In Figure 4a and Figure I2
17

---

legug
May
Doaatll andodeeed Surnnas yocee wib seadi
sonea la usa Saniin foooss Sust uio (iul sy cormggaat gin ever androup..
Serail
Figure 10: More comparative results of generated images on both in-domain and out-of-domain models.
incorifarna
arytiing
eplexiin
raceiiane
Masuell
Febi
adaaa
Our
Shelt nerting w0l ahrdandd beadberd Thevores ac
Xome
Figure 11: More comparative results of generated images out-of-domain models.
P. p + pddo
(14)
which E [0, 1].
In these cases, it is evident that domain prototypes possess clear semantic information, representing specific image styles such as cartoon, realistic, or 3D animation.
However, domain prototypes do not always possess clear semantic information. As shown in Figure 13 we gradually inject domain prototypes into the conditional information by adjusting the 7j value of pdom. It seems difficult to define the semantics represented by the domain prototypes in these cases. We believe this reflects a comprehensive factors such as image detail richness, photography techniques, lighting, image contrast, and image softness.
18

---

Pl
pdmm
pdom
Figure 12: Linear combinations of domain prototypes from ith models and domain prototypes from jth models.
C.5 More Ablation Study
In Figure 14, we present additional results from the ablation experiments on the loss functions. Sequential ablation of each of the four loss functions reveals that the absence of any single loss function variably affects the image quality.
19

---

0
Figure 13: Injection of ith domain prototype information by adjusting the value of ni-
20

---

w/o c
w/o La
w/o r
w/o L
w/ L
A woman with long, dark hair, wearing white shirt, and eating slice of pizza, She is posing for the camera while enjoying her meal. The seene takes place on a street.
A cat wearing green jacket and sunglasses, giving a stylish and unique appcarance. The cat is positioned the center of the image.
cartoon figure of man wearing suit and hat. The man appears to be caricature Donald Trump, the former U.S. president.
plaster portrait from the Renaissance period. The plaster appears to be man with long hair, facing the camera directly.
Figure 14: Demonstration of Ablation Study of Loss Functions.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our abstract clearly reflects the contribution and scope of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations our work are mentioned Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: In the Section of the paper, we provide a detailed description of the model framework propose.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [No]
Justification: Our contributions include the dataset we collected and the code for model training and testing. We will release the data and code after the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
hetpolliicggiddee/ooooooe http:/uuddde/ooeSuubnrb ssicnlolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We describe these details in Section
4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Our experiments involve the generation of large number of images, which is consumin... Therefore, we did not perform coss-validation to calculate the error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide these information in Appendix C.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Our research conform with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In the Section I we discuss the positive impact, and we believe this research has no negative social implications.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system Iearns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Although our data collected from web crawling, filtered out any unsafe data, Furthermore, our image data consists solely of synthetic images that do not contain any information that could pose harm the real world.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All of the creators original owners of assets used in paper are cited properly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
26

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
We will release the data and code including documentation after the paper is accepted.
Justification:
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
27


Reviews:
[{"review_id": "YRUI0COIuA", "reviewer": "Reviewer_redB", "summary": "1. The authors propose model-generalized automatic prompt optimization (MGAPO), which targets the effectivenss of automatic prompts on unseen models.\n\n2. The authors propose AP-Adapter which include in-context learning based prompt rewriting and prototype-based prompt adaptation.\n\n3. The authors build a multi-modal, multi-domain dataset for training and evaluation.", "questions": "Please refer to the \"Weaknesses\" section.", "limitations": "This paper includes a specified section for limitations and the authors adequately addressed them.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The authors introduce the process of collecting and creating the multi-modal multi-domain dataset for training and evaluation.\n\n2. The experimental results show that the proposed method outperforms existing baseline models.\n\n3. The paper includes extensive further analyses and ablation studies to deepen the understanding of the proposed method.", "weaknesses": "1. While the proposed method outperforms other baselines, its increased performance appears quite marginal, particularly considering the significant gap compared to the performance of manual prompts. I am curious about the main reasons for this substantial gap between automatic and manual prompts, and what fundamental limitations of automatic prompting hinder its improvement. Are there any promising directions to overcome these limitations? Can we consider the proposed method as addressing some of these limitations?\n\n2. The proposed prototype-based adaptation utilizes the concept of DomainDrop in a reverse manner. DomainDrop aims to learn domain-invariant features by eliminating domain-sensitive information. Conversely, the proposed method removes domain-insensitive information to construct domain prototypes. However, in the context of domain generalization, domain-insensitive information is often considered highly valuable, as it contains information shared across different domains, which is crucial for generalizing to unseen domains. Therefore, I am curious if the process of removing domain-insensitive information in the proposed method could result in the loss of important information. Additionally, relying on domain prototypes may lead to situations where a model cannot properly handle data from completely new domains that cannot be adequately matched with existing domains. In this case, we need to consider the difficulty of continuously increasing the number of domain prototypes, as this would require an increase in model size as well.\n\n3. While using different checkpoints, the data used for training and testing are consistent in that they are all collected from CIVITAI and fine-tuned based on SD1.5. However, the pre-trained data used for other baseline methods does not necessarily share these characteristics. I am curious if this could have introduced some biases in favor of the proposed method, leading to better results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive comments. We will address the concerns below.\n\n\nQ1-1. What’s the the main reasons for this substantial gap between automatic and manual prompts?  \nA1-1. The effectiveness of manual prompts lies in their iterative nature, allowing humans to refine prompts based on the generated images until the desired quality is achieved. In contrast, automatic prompts aim to replicate this effectiveness by learning patterns from manual prompts.\n\n\nQ1-2. What fundamental limitations of automatic prompting hinder its improvement?  \nA1-2. The key limitation is that APO models struggle to evaluate whether an image meets specific requirements and to re-optimize the prompts accordingly.\n\n\nQ1-3. Are there any promising directions to overcome these limitations?  \nA1-3. One potential solution is to incorporate reinforcement learning by designing a reward model that simulates human judgment. This would enable the model to dynamically identify and correct deficiencies in the current prompts based on the generated images.\n\n\nQ1-4. Can we consider the proposed method as addressing some of these limitations?  \nA1-4. Our approach does not focus on generating superior automatic prompts but rather on enhancing the generalization of automatic prompts across different checkpoints, thereby avoiding the need for repetitive retraining of the prompt generation model for new checkpoints.\n\n\nQ2-1. If the process of removing domain-insensitive information in the proposed method could result in the loss of important information?  \nA2-1. As shown in Figure 2, in the adapter section, the final adapted embedding is obtained by concatenating the prompt embedding and the Prototype-anchored embedding, followed by transformation. During this process, domain-insensitive information remains in the prompt embedding and is not completely removed, ensuring that important semantic information is not lost.\n\n\nQ2-2. Is it necessary to continuously increase the number of domain prototypes to adapt to new data?  \nA2-2. We do not require the target domain to have perfectly matching domain prototypes. Instead, we assume that the target domains are entirely new and unseen, aiming to anchor their position using existing domain prototypes. To achieve this, we selected checkpoints with diverse styles as our training data to maximize the adaptability of the prompt representations. Consequently, we do not need to continually increase the number of domain prototypes.\n\n\nQ3. If data could have introduced some biases in favor of the proposed method, leading to better results.  \nA3. Our basic assumption is that all checkpoints must be with the same structure and fine-tuned based on the same base SD model using different data. Our method addresses the generalization of automatic prompts across different checkpoints. If the target model is with entirely different structure from the training model, such as SD XL or DALLE, the problem itself becomes unsolvable, as we cannot predict the behavior of an entirely unknown model. Generalization is only possible on checkpoints derived from the same model but fine-tuned with different data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9hag2jjzg1", "reviewer": "Reviewer_W76C", "summary": "The paper proposes model-generalized automatic prompt optimization (MGAPO), an automatic prompt optimization (APO) method which trains on a set of known models to enable generalization to unseen models during testing. MGAPO presents significant challenges, a perspective missing in previous methods. MGAPO includes a two-stage prompt optimization method. In the first stage, a LLM is used to rewrite the prompts and generate keyword prompts. In the second stage, the methods leverage inter-model differences (derived from varied checkpoints) to captures the characteristics of multiple domains and store them as domain prototypes. These prototypes serve as anchors to adjust prompt representations, enabling generalization to unseen models. The optimized prompt representations are subsequently used to generate conditional representations for controllable image generation.", "questions": "Please check the weaknesses section.", "limitations": "Authors have provided the limitations statement.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper presents a new perspective to the automatic prompt optimization at the feature level.\n- Authors have shown the effect of each loss component in Table 2.", "weaknesses": "- I believe the quality of writing can be improved. The language of paper seems unnecessarily complex and hard to comprehend at certain places especially the methodology section.\n- The problem solved in the paper seems more of an incremental learning problem than domain generalization. The training and target domains differ only in terms of number of images seen. They do not differ in term of style or content as per paper. What are the views of authors about this?\n- I can see in figure 2 that the UNET of diffusion model is freezed. If that's the case, then it renders the diffusion loss $\\mathcal{L}_d$ in equation 11 useless. What exactly is being updated in equation 11? The $\\theta$ parameters of UNET are freezed.\n- In Fig 3 (row 3), the proposed method doesn't seem to produce optimal results with DreamShaper. The cat is supposed to be standing as per the prompt, however, the cat generated by the proposed method seems to be sitting. Compared to proposed method, the baselines seems to be working better in this case.\n- Fig. 2 seems to be hastily drawn. There are many things which can improved in the figure (minor: \"M\" of LLM block is going out of the box). Also the font of in-context learning block is very small\n- The gains in Table 1 seem to be average compared to baselines. In fact baselines are better in certain cases. Given the complexity of the proposed method, the gains are expected to be larger, which is not the case.\n\nOverall, the authors are encouraged to answer the above queries and provide solid explanations", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive comments. We will address the concerns below.\n\nQ1. The quality of writing can be improved.  \nA1. Thank you for your suggestions. We will optimize the writing in the methodology section to make it more concise and easy to understand.\n\n\nQ2. What’s the difference between the training and target domains?  \nA2. In our dataset, each checkpoint is fine-tuned using images of different styles and content, resulting in varying generative capabilities. During experiments, we use a subset of these checkpoints as the training set and test on another subset. The fundamental difference between source and target domain checkpoints lies in the style and content of the fine-tuning data, not the number of the data.\n\n\nQ3. What exactly is being updated in equation 11?   \nA3. In Equation 11, the parameters of $ q_i^k $ will be updated. $ q_i^k $ is the output of $ g_{\\text{ada}} $ in Equation 9. Thus, the variables being updated are the parameters of the transformer layers and linear layers in the adapter.\n\n\nQ4. Compared to proposed method, the baselines seems to be working better in Figure 2.  \nA4. We found that the issue lies in the ICL-based prompt rewriting in the first stage. As shown in Appendix C.1, the prompts rewritten by the large language model do not include the word \"standing.\" This indicates that while the adapter in the second stage can enhance the generalization of the automatic prompts to new models, it cannot compensate for the information lost by LLM in the first stage. Therefore, the quality of the automatic prompts is also crucial.\n\n\nQ5. Fig. 2 need to be improved.  \nA5. Thank you for your advice. We will improve the quality of the figures and provide clearer figures in the final version.\n\n\nQ6. Given the complexity of the proposed method, the gains are expected to be larger, which is not the case.  \nA6. Compared to baseline methods, our approach requires significantly fewer parameters to be trained, limited only to the domain prototypes and adapter. Therefore, our method is not more complex. Additionally, our approach performs better than other methods in most cases, as shown in Table 1 and Figure 2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uFhu4gQwYy", "reviewer": "Reviewer_fgUM", "summary": "This paper proposes a new task called MGAPO, aimed at addressing the generalization problem of existing APO methods on unseen text-to-image generation models. To achieve this, the authors present a two-stage method called AP-Adapter. In the first stage, keyword prompts are generated through a large language model; in the second stage, an enhanced representation space is constructed by leveraging inter-model differences, and prompt representations are adjusted through domain prototypes to enable generalization to unseen models. Experimental results demonstrate that AP-Adapter can generate high-quality images on previously unseen diffusion models and outperform existing methods in both semantic consistency and aesthetic quality.", "questions": "see Weaknesses", "limitations": "The authors mentioned in the Limitation section that the main drawback of this paper is the small size of the dataset used, which is focused on single-character images. This limits the model's performance. Furthermore, the authors explained that as the dataset size expands, the number of domain prototypes might increase.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1 The introduction of the MGAPO task makes text-to-image model design more applicable to real-world scenarios.\n\n2 The related concepts and differences are clearly articulated, especially in Figure 1.", "weaknesses": "1 The core of this paper lies in the computation of domain prototypes, which are influenced by CLIP. However, the authors did not conduct relevant ablation experiments.\n\n2 Figure 4 includes human evaluation, which introduces a degree of subjectivity. Additionally, previous methods like sub-adapter did not conduct similar validation. Furthermore, the authors seem to lack detailed explanations regarding human validation in this paper.\n\n3 The proposed method is relatively complex, introducing additional models such as a pre-trained LLM and CLIP for prototype computation, which increases overall complexity and training/inference time. The authors did not discuss these aspects.\n\n4 AP-Adapter still requires a large number of manually designed prompts as references during training, which may increase the workload and complexity of data preparation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive comments. We will address the concerns below.\n\n\nQ1. The authors did not conduct relevant ablation experiments on CLIP.  \nA1. The computation of domain prototypes indeed relies on the image encoder. Moreover, domain prototypes are concatenated with text representations to serve as control conditions for image generation, so they should be as similar to text representations as possible. Given that stable diffusion 1.5 uses CLIP as the text encoder, we also use CLIP as the image encoder, ensuring that the image representations share a feature space with the text representations.\nIn the appendix, we explore the impact of CLIP on domain prototypes. The experiments include Figure 8 and Table 4 in section C.2.\n\n\nQ2. The human validation lack detailed explanations.  \nA2. We followed the approach presented in the SUR-adapter[1]. We collected 108 valid questionnaires. Participants were shown images generated by our method and baseline methods, along with their corresponding descriptions. They were asked to choose the better image based on the questions \"Which image do you think has higher quality?\" and \"Which image do you think better matches the text description?\" We then compiled and analyzed the survey results.\n\n\nQ3. The proposed method introduces additional models such as a pre-trained LLM and CLIP for prototype computation.  \nA3. The LLM is only used in the first stage for automatic prompt generation, employing the ICL strategy, and does not participate in subsequent training. The LLM’s model parameters are not updated. During training, the parameters of CLIP are frozen and do not get updated. During inference, CLIP is no longer used; only the trained domain prototypes are involved. Therefore, training and inference times are not significantly increased. \n\n\nQ4. The data collection process adds to the workload and complexity of the proposed method.  \nA4. All existing APO methods require a large number of manually designed prompts as references for training. \nMoreover, such data collection and retraining should be repeated for the APO on each newly-coming diffusion model.\nOur method is desined to address this issue by eliminating the need for new data preparation and retraining when encountering new diffusion models. Our trained adapter can be directly used on new models without retraining.\n\n\n[1] Zhong S, Huang Z, Wen W, et al. Sur-adapter: Enhancing text-to-image pre-trained diffusion models with large language models[C]//Proceedings of the 31st ACM International Conference on Multimedia. 2023: 567-578.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The final ratings are 5, 6, 6. However, it seems there are still some concerns as follow:

- **AP-Adaptor improvement over baselines are marginal**. Reviewer did mention that the performance is far from manual prompting. However, the AC's opinion is that the paper is on improving over automatic prompting so this is not such a big issue to him. Automatic prompt will not beat manual prompting at this time for sure but the research must still go on. So, the AC is not that concerned about achieving "commercial grade" for automatic prompting yet.
- **AP-Adaptor may not be truly out-of-domain capable**. This stems from the concerns that the approach is simply a result of a bunch of diversified checkpoints that allow AP-Adaptor to be multi-domain.
- **The proposed method is heavily dependent on the quality of the prompt generated by the LLM**. The authors tried to address these concerns by proposing two solutions but clearly the solutions are just hypothetical -- the AC don't find this very satisfying in his opinion.

Upon further discussions, reviewers remain a bit borderline on the second point and the third point. On (2), reviewers raised the point of whether the proposed approach is genuinely out-of-domain capable and whether the evaluation setting is entirely fair. 

After careful considerations, the AC urged the authors to address the second and third point strongly in the final manuscript and decided to accept the paper as the reviewers finally are slightly more positive than negative about the paper.
Decision: Accept (poster)