Automated Black-box Prompt Engineering for Personalized Text-to-Image Generation

## Abstract

Abstract Prompt engineering is effective for controlling the output of text-to-image (T2I) generative models, but it is also laborious due to the need for manually crafted prompts.
This challenge has spurred the development of algorithms for automated prompt generation.
However, these methods often struggle with transferability across T2I models, require white-box access to the underlying model, and produce non-intuitive prompts.
In this work, we introduce PRISM , an algorithm that automatically identifies human-interpretable and transferable prompts that can effectively generate desired concepts given only black-box access to T2I models.
Inspired by large language model (LLM) jailbreaking, PRISM leverages the in-context learning ability of LLMs to iteratively refine the candidate prompts distribution for given reference images.
Our experiments demonstrate the versatility and effectiveness of PRISM in generating accurate prompts for objects, styles, and images across multiple T2I models, including Stable Diffusion, DALL-E, and Midjourney.

###### Keywords:

## 1 Introduction

Figure: Figure 1: Given a set of reference images, our method, PRISM, is capable of creating human-interpretable and accurate prompts for the desired concept that are also transferable to both open-sourced and closed-sourced text-to-image models.

An important goal of generative modeling is to design algorithms capable of steering generative models to produce desired output images.
Early attempts, which often centered on particular architectures or tasks, were largely characterized by manually-curated data collection, fine-tuning, or retraining from scratch [, , , ].
These requirements are often costly, and the resulting solutions usually do not transfer well between models. Thus despite the promise of these methods, efficient and generalized algorithms for controllable generation remain sought after.

Today, perhaps the most popular approach for controllable generation is to guide the generation process with a piece of textual information, or prompt, that describes the properties of the desired output using text-to-image (T2I) generative models [, ]. Through text, T2I models allow users to quickly and easily describe a wide variety of concepts, and model designers can more efficiently explore the behavior of their model through a myriad of strategies [, ].
The predominant method for obtaining such input text is to manually design candidate prompts in an iterative, trial-and-error fashion, a process known as *prompt engineering*, based on what the user (prompt engineer) believes will lead to a desirable output.
Unfortunately, these practices are often sensitive to different phrasings [], require expert domain knowledge, and are notably inefficient as they necessitate a human in the loop.

Motivated by the drawbacks of manual prompt engineering, a recent line of work known as *personalized* or *subject-driven* T2I generation has sought to automate the controllable generation pipeline. Given a collection of reference images that capture specific concepts, such as artistic style or shared objects, personalized T2I algorithms are designed to produce images that reflect those concepts illustrated in the reference images.
While personalized T2I methods often involve fine-tuning or retraining the underlying T2I model [, , ], several approaches focus specifically on automating prompt engineering to generate effective prompts.
Unfortunately, existing algorithms in this spirit tend to require pre-collected, architecture-specific keywords(^1^11https://github.com/pharmapsychotic/clip-interrogator) or white-box, embedding-based optimization [, ], leading to non-interpretable prompts [] and precluding the possibility of directly generating prompts for closed-source T2I models (e.g., Midjourney or DALL-E).

In order to address these shortcomings, we propose *Prompt* *Refinement* *and* *Iterative* *Sampling* *Mechanism* (PRISM), a new automated prompt engineering algorithm for personalized T2I generation.
A key observation is that prompt engineers repeat the process of updating their “belief” of what makes an effective prompt based on the difference between their desired results and the generated images from previous iterations. Inspired by jailbreaking attacks on large language models (LLMs) [], we design an algorithm that operates with only limited human input, is capable of generating human interpretable and editable prompts, makes minimal assumptions about the underlying T2I generative model, and generalizes across different T2I models, including popular black-box models such as DALL-E and Midjourney.

Given a set of reference images, our method first generates an initial prompt and its corresponding image using a multimodal LLM and a T2I generative model. We then obtain a score indicating the visual similarity of the generated image and the reference image with respect to the targeting concept via another multimodal LLM. Leveraging LLMs’ in-context learning abilities [, , ], we instruct the LLM to update the candidate prompt distribution based on the previously generated prompt, images, and the evaluation scores. This processing is then repeated for a predetermined number of iterations. In the end, PRISM outputs the best-performing prompt by re-evaluating the top prompts generated from this process.

Experimentally, our results indicate that PRISM consistently outperforms existing methods, including Textual Inversion [], PEZ [], BLIP2 [] and CLIP-Interrogator^1, with respect to human-interpretability while maintaining high visual accuracy. Our method also shows significantly better generalizability and transferability as we achieve the best performance in almost all metrics when experimenting with closed-source models in comparison to baselines. Finally, we also show that because of the interpretability provided by our method, the prompts produced by PRISM are also easily editable, enabling a wide range of creativity possibilities in real life.

## 2 Related Works

#### 2.0.1 Controllable T2I generation.

Several methods tackle conditional image generation in a training-free manner by using pretrained diffusion models as priors for the data distribution [, , , , ], and analogous approaches exist for T2I diffusion models (e.g., StableDiffusion) [, , ]. In general, these methods assume that the controllability objective can be formulated as differentiable loss functions, although they require access to model parameters and involve complex hyperparameter tuning. Another class of approaches such as ControlNet [], IP-Adapter [], Dreambooth [], SuTI [] and InstantBooth [] also improve the controllability of pretrained T2I models, but they require expensive fine-tuning or re-training of the underlying model. Prompt tuning methods such as Textual Inversion [], PEZ [], and PH2P [, ] are in the same spirit as this paper, as they do not require fine-tuning or optimizing the underlying model and generate images that inherit the properties of a given reference image. However, unlike PRISM, each of these methods requires access to the underlying model parameters and produces non-interpretable prompts.

#### 2.0.2 Prompt engineering.

Manual prompt engineering is one of the most popular approaches to eliciting desired behaviors from large pre-trained models because it uses little or no data and does not require fine-tuning [, ]. However, major drawbacks of manual prompt engineering include its laborious nature, its reliance on domain expertise, and the fact that its performance can be highly sensitive to how the prompts are phrased [, ]. To address this issue, several methods have been proposed to construct the prompts in an automated manner [, , , , , ]. In particular, the field of LLM jailbreaking is concerned with automatically designing prompts that can elicit specific content (which is often objectionable or illicit) from a targeted LLM [, , , ]. A particularly relevant work is [], which uses an auxiliary LLM to iteratively construct jailbreak prompts that elicit harmful behaviors from a targeted LLM. Our method builds on this idea to generate prompts that will result in images that satisfy the desired criteria.

## 3 Method

Figure: Figure 2: An illustration of PRISM. The label “System” indicates the system prompts setups for the multimodal LLMs.

### 3.1 Problem Statement

First, let $x\in{\mathcal{X}}$ denote an image, and $y\in{\mathcal{Y}}$ denote a textual prompt. Given a collection of reference images $\{x_{i}\}_{i=1}^{M}$, a prompt engineer $\mathbf{F}:{\mathcal{X}}\rightarrow\Delta({\mathcal{Y}})$ samples a candidate prompt $y$ corresponding to each reference image $x$, i.e., $y\sim p_{\theta_{{\mathbf{F}}}}(y\mid x)$. A T2I generative model $\mathbf{G}:{\mathcal{Y}}\rightarrow\Delta({\mathcal{X}})$ then uses this candidate prompt to generate a new image, $x\sim p_{\theta_{\mathbf{G}}}(x\mid y)$, and a judge model $\mathbf{D}:{\mathcal{X}}\times{\mathcal{X}}\rightarrow[0,1]$ then scores the visual similarity between the images based on some criteria. Our goal is then to find the best prompt:

$$ $y^{\star}\left(\{x_{i}\}_{i=1}^{M}\right)=\operatorname*{arg\,max}_{y\in{ \mathcal{Y}}}\sum_{i=1}^{M}\texttt{Score}(x_{i},y),$ (1) $$

where $\texttt{Score}(x_{\text{target}},y)=\mathbb{E}_{x\sim p_{\theta_{\mathbf{G}}}(
x\mid y)}\left[{\mathbf{D}}(x,x_{\text{target}})\right]$.

The criteria can be any visual similarity metric that may or may not be easy to specify in a closed form, including “how similar are the main objects in the images” or “how similar are the styles of the image” or “how similar are the two images in general”. The resulting $y^{\star}$ should be able to generate an image that is very close to the reference images based on the criteria with some (possibly unseen) T2I models $p_{\theta}(x\mid y)$.

### 3.2 Algorithm

Figure: Algorithm 1 Prompt Refinement and Iterative Sampling Mechanism (PRISM)

Our method, Prompt Refinement and Iterative Sampling Mechanism  (PRISM), is an iterative process that repeats a prompt-refinement subroutine for $K$ iterations in $N$ parallel streams, where $N\times K$ is a predetermined compute budget. At iteration $k$, the $n$-th stream of PRISM  randomly selects a reference image $x_{k,n}$ from $\{x_{i}\}_{i=1}^{M}$ and uses ${\mathbf{F}}$ to sample a candidate prompt $y_{k,n}$ from $p_{\theta_{\mathbf{F}}}(y\mid x_{k,n})$. Then it queries ${\mathbf{G}}$ to generate a single $\hat{x}_{k,n}$ from $y_{k,n}$ with $p_{\theta_{\mathbf{G}}}(x\mid y_{k,n})$ and evaluate the prompt with ${\mathbf{D}}$ to obtain an in-iteration score $\texttt{Score}^{\prime}(x_{k,n},y_{k,n})=\mathbf{D}(x_{k,n},\hat{x}_{k,n})$. At the end of the iteration, we use the generated $y_{k,n}$ and its score to update $p_{\theta_{\mathbf{F}}}(y\mid x)$.
After the entire process, we collect the subset of $\{y_{c}\}_{c=1}^{C}$ generated throughout this process that has the $C$-best in-iteration scores. Then we re-evaluate this subset with the total score $\sum_{i=1}^{M}\texttt{Score}(x_{i},y_{c})$ and return the prompt with the best total score. If there is a tie, then we return the prompt with the highest log likelihood []. The pseudocode for the algorithm is outlined in Algorithm 1 and we also illustrate this algorithm in Figure 2.

The key difference between PRISM  and prior methods is that PRISM  updates the entire sampling distribution of prompts, whereas prior works [, , ] directly update the tokens of a single prompt or the embedding of the prompt.
We believe that maintaining the whole prompt distribution is beneficial as text-to-image generation is not a one-to-one operation, i.e. an image can be described by multiple different text prompts and the same text prompt can correspond to multiple different generated images. Having access to the whole distribution allows the method to sample a more diverse range of prompts without starting from scratch and may also help the optimization escape potential local optima.

Since PRISM only requires samples from ${\mathbf{G}}$ conditioned on the prompts, one may use any T2I generative model of their choice. On the other hand, more careful treatment is required for designing ${\mathbf{F}}$ and ${\mathbf{D}}$. We will elaborate on these design decisions below.

### 3.3 Designing and updating 𝐅 𝐅 \mathbf{F} bold_F and p θ 𝐅 subscript 𝑝 subscript 𝜃 𝐅 p_{\theta_{\mathbf{F}}} italic_p start_POSTSUBSCRIPT italic_θ start_POSTSUBSCRIPT bold_F end_POSTSUBSCRIPT end_POSTSUBSCRIPT

#### 3.3.1 What is p ⁢ ( y ∣ x ) 𝑝 conditional 𝑦 𝑥 p(y\mid x) italic_p ( italic_y ∣ italic_x ) ?

In general, it is not obvious what the joint or the conditional distribution of all text and images is, so some form of approximation is unavoidable.
In the context of image generation, a natural choice of the image-conditioned text distribution is an image captioning model.
Traditional captioning models, however, fall short in controlled image generation for two primary reasons: (1) The level of detail necessary for generating specific images far exceeds what generic captioning models provide []; (2) effective prompts for T2I models are often not grammatically correct sentences but rather collection of phrases that describe the details about the image, which generic captioning models are not trained to generate. For example, in Figure 5, the second reference image is generated by the prompt “A broken robot lays on the ground with plants growing over it, somber, HD, hyper realistic, intricate detail” with Stable Diffusion, but a caption for this image will not include components like “HD” or “hyper realistic”.
As a result, instead of “a good description of an image”, we wish to directly model “possible prompts that are used to generate this image”.

#### 3.3.2 Desiderata

A desirable $\mathbf{F}$ can sample from a distribution $p_{\theta_{\mathbf{F}}}(y\mid x)$ that models “the prompt that can be used to generate this image”, and it should also be easily updated if the current generation is suboptimal.
Ideally, such an update can be done without any retraining or fine-tuning since these operations are generally expensive and incompatible with black-box T2I models.

#### 3.3.3 Multimodal LLM

stands out as the ideal choice for ${\mathbf{F}}$ due to their ability to directly tailor the generation of prompts via system prompts and to adapt through in-context learning without requiring access to the model’s parameters.
Specifically, since the model can ingest both images and texts, we can incorporate the reference images, intermediate prompts and generated images, and the score associated with the generated images all in the context of the LLM. Then, the model can be prompted to jointly reason over all available information and perform in-context learning. The in-context learning facilitates iterative refinement of the prompt to update the posterior distribution based on feedback or even additional human instructions, without the need for model retraining. Concretely, the model would process how the image generative model is affected by different prompts, propose improvements, and create new prompts, much like a prompt engineer.
More precisely, in practice, we design system prompts that explicitly condition the LLM to generate improvements and new prompts given the results from the previous iterations, similar to the chain-of-thought [] technique.

### 3.4 Designing the judge model 𝐃 𝐃 \mathbf{D} bold_D

We have a wider range of choices for the judging model as long as it provides a notion of similarity between a pair of images.
A simple solution is to use pre-trained discriminative models such as CLIP [] and DINO [], and measure the distance of images in their embedding spaces.
These models have seen various degrees of success but come with inherent limitations – the discriminative objective (e.g., contrastive loss) does not incentivize the model to attend to fine-grained details since they do not improve the objective further, an issue similar to the shortcomings of using captioning models to generate prompts []. Moreover, in image generation, the criteria of success can be nuanced and difficult to quantify through traditional distance or similarity functions yet can be effortlessly described in human language. Lastly, the similarity we wish to measure may only involve some part of the visual features (e.g. color, painting stroke type, etc), and not all applications share the same notion of similarity. If we want to use pre-trained discriminative models, then we need to find a different model for each specific task, which can be impractical.

In light of these challenges, an ideal judge model should be maximally flexible for different kinds of criteria and can perform fine-grained analysis of the images.
Once again, a multimodal LLM emerges as the perfect candidate: using system prompts and in-context learning, we can easily specify metrics that may be otherwise difficult to describe or evaluate and even intervene in the reasoning chain if we want to, and, more importantly, the same model can be applied to a wide range of tasks.

## 4 Experiments

### 4.1 Experimental Settings

#### 4.1.1 Implementation Details

For all of our experiments, we choose GPT-4V [] as both the prompt engineer assistant model ${\mathbf{F}}$ and the judge ${\mathbf{D}}$. We also fix the T2I generator as SDXL-Turbo [] for all of our experiments. We design different system prompts for both ${\mathbf{F}}$ and ${\mathbf{D}}$ for each task and we provide details about the system prompts in the appendix.

We evaluate the prompts generated from PRISM and baselines with five different T2I models. In particular, we choose two open-sourced models, Stable Diffusion 2.1 (SD 2.1) and SDXL-Turbo, and two closed-sourced models, Dall-E 2 and Dall-E 3, to quantitatively measure the performance. We also qualitatively showcase results from Midjourney, which is another closed-sourced T2I platform. For SD 2.1 and SDXL-Turbo, we clip all prompt lengths to 77 due to their context length constraint.

We compare PRISM and baselines in two settings: personalized T2I generation and direct image inversion, and we will elaborate on the task definitions in their corresponding sections below. For personalized T2I generation, we use a maximum budget of $40$ and report the quantitative results from the setting $N=10,K=4$. For direct image inversion, we use a maximum budget of $30$ and report the quantitative results from the setting $N=6,K=5$. To simplify the implementation, we only keep a chat history length of $3$ and use the length of the prompt as an approximation of the prompt log-likelihood in the final prompt selection. For direct image inversion, we re-evaluate the top $5$ candidates twice and tally the score with the in-iteration scores to make the final decision. For personalized T2I generation, we re-evaluate once for each reference image and use the average score to select the output.

#### 4.1.2 Baselines

We choose Textual Inversion (TI) [], BLIP-2 (BLIP2) [], CLIP-Interrogator (CLIP-Int) and PEZ [] as the baselines.
Textual Inversion trains a “soft token” which cannot be directly translated into regular human language to represent the concepts in the reference images. BLIP-2 is the state-of-the-art image captioning model. CLIP-Interrogator${}^{\ref{clip-int}}$ combines BLIP-2 captions with a suffix which is created by searching a pre-collected bank of keywords using CLIP [] score. PEZ is a gradient-based optimization method that searches for the best combination of existing tokens in the vocabulary with CLIP similarity.
We use OpenCLIP-ViT-H-14 trained on LAION2B [] for both CLIP-Int and PEZ and use Blip2-Flan-T5-XL for both CLIP-Int and BLIP-2.
Notice that TI requires training on individual models and CLIP-Int requires a pre-collected keyword bank, both of which provides unfair advantages over our setting.

#### 4.1.3 Evaluation Metrics

We evaluate the prompt interpretability using mean negative log-likelihood (NLL) calculated from Mistral 7B []. For image quality evaluation, we mainly measure the CLIP image similarity score (CLIP-I) to quantify the difference between the generated images and the reference images. Following [], we also use DINO V2 [] embedding similarity to calculate the object-sensitive image similarity for the personalized T2I generation task. We chose CLIP-ViT-L-14 and DINO-V2-Base as the base models. For Dall-E 2 and Dall-E 3, we also compare the number of times each method fails to pass its black-box safeguard. More failures indicate a higher potential to produce unsafe prompts. For each prompt, we allow $5$ attempts before counting it as a failure.

### 4.2 Personalized Text-to-Image Generation

Figure: Figure 3: Qualitative results for personalized T2I generation on DreamBooth dataset.

Table: Table 1: Personalized T2I results on DreamBooth dataset. Bold fonts indicate the best score and underlines indicate the second best score.

We first demonstrate PRISM’s ability to find human-interpretable and accurate prompts to describe certain objects and styles in the task of personalized T2I generation. Given a set of reference images that depict the same concept (such as objects and style), personalized T2I tasks require the model to synthesize images in new contexts while maintaining the original concept.

#### 4.2.1 Datasets

We use the dataset collected by DreamBooth [] to quantitatively compare the performance in personalized T2I generation. The DreamBooth dataset contains 30 daily objects, and each subject has 4-6 images. For each subject, we adopt the 25 prompt templates curated by DreamBooth to create varying contexts and scenarios to test the fidelity of the subject representation in diverse settings.
We generate 4 images for each subject and template combination with open sourced T2I models, and 1 image for each combination with closed sourced T2I models.
For methods that directly find English prompts, we use the class noun to fill in the template and the output prompts that describe these concepts serve as suffixes. For Textual Inversion, we follow the original setting for the templates.

We also qualitatively demonstrate the ability to represent a certain artistic style using Wikiart dataset [].
We use three images from each artist as reference images.
To create diverse scenes, we follow [] and use descriptive prompts from PartiPrompts [] as prefixes to the output prompts similar to the previous setting.

#### 4.2.2 DreamBooth Dataset Results

Table 1 and Figure 3 respectively show the quantitative and qualitative results on the DreamBooth dataset. As we can observe, PRISM achieves the best performance across the board except for the image similarity metrics for SDXL-Turbo.
PRISM is the only method in our experiments that can produce fully human-readable prompts for these subjects. In particular, we can observe that PEZ renders completely indecipherable texts, BLIP-2 only describes the general scene but fails to mention any visual details and textual inversion is entirely not interpretable since it produces soft embeddings. Since CLIP-Interrogator combines the results from BLIP-2 and a CLIP search, it improves the interpretability over PEZ-like gradient search-only method. However, it still falls short in terms of human readability in comparison to our method.

In terms of image quality and object fidelity, we also find PRISM to constantly achieve accurate depiction of the target subject while the baselines sometimes struggle to capture all the details. Fine-grained details such as the color of the dog and cat fur and the shape of the shoe sole are better described and reflected with our method. And out of the four training-free methods we experiment with, PRISM is the only one that can tackle complicated objects such as the red monster toy and the dog-shaped backpack as shown in Figure 3 when all the other methods fail to generate similar objects. Due to the nature of their methodologies, BLIP-2 and CLIP-Interrogator also capture the background and other irrelevant elements in the scene when describing the objects. However, unlike our method, where we can directly specify the tasks and the judging criteria in the system prompts of the LLMs, there is no simple way to automatically filter out those irrelevant elements in BLIP-2 and CLIP-Interrogator’s outputs. Even though Textual Inversion obtains marginally higher CLIP-I scores and DINO scores with SDXL-Turbo, notice that Textual Inversion requires a lot more modeling assumptions than our method, and the new embeddings it learns are not transferable – not even to SD 2.1, because SDXL models use two text encoders whereas SD 2.1 model use only one.

When transferring the output prompts to black-box T2I models, our method shows even larger advantages over the baselines. We also observe that our method produces the fewest unsafe prompts judged by Dall-E safeguards, while the baselines can fail to pass the safeguard up to almost $16.7\%$ of the time.

#### 4.2.3 Wikiart Results

Figure: Figure 4: Qualitative results for personalized style T2I generation on Wikiart dataset.

In Figure 4, we also show a qualitative comparison between our method and the baselines on the Wikiart dataset.
We find that our method is capable of precisely identifying the genres, eras, and sometimes even the names of the artists when describing the style of the reference artworks. On the other hand, we observe that the baselines fail to recognize these crucial keywords, even when they have access to a pre-collected bank of words that are supposed to provide accurate descriptions of the style.
In addition, PRISM can also provide other fine-grained details such as the style of the pen strokes and color palettes in a human-interpretable way to better assist the generation of the target style.

### 4.3 Direct Image Inversion

Figure: Figure 5: Image inversion results for different methods on different T2I models.

**Table 2: Metrics for the image inversion results. old fonts indicate the best score and underlines indicate the second best score.**
| Method | Prompt | SD 2.1 | SDXL TUrbo | Dall-E 2 | Dall-E 3 |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| NLL $\downarrow$ | CLIP-I $\uparrow$ | CLIP-I $\uparrow$ | CLIP-I $\uparrow$ | Failed $\downarrow$ | CLIP-I $\uparrow$ | Failed $\downarrow$ |  |
| CLIP-Int | 4.193 | 0.800 | 0.783 | 0.761 | 17.0% | 0.719 | 0.0% |
| BLIP2 | 4.299 | 0.710 | 0.707 | 0.687 | 2.0% | 0.695 | 0.0% |
| PEZ | 6.736 | 0.746 | 0.726 | 0.616 | 3.0% | 0.635 | 0.0% |
| PRISM (Ours) | 2.762 | 0.749 | 0.776 | 0.741 | 2.0% | 0.767 | 0.0% |

To demonstrate the versatility of our method, we also compare PRISM with the baselines in the task of direct image inversion. In this task, the goal is to directly find the prompt that can exactly generate the input image. Here the number of reference images is $M=1$ and we aim to capture all aspects of the image, including the subjects, background, theme, style, and other details in the scene.

#### 4.3.1 Datasets

We use images from the DiffusionDB dataset [] for the direct image inversion task.
This dataset includes a wide variety of image pairs generated by Stable Diffusion and we choose a random sample of 100 images from the large_random_10k split on Huggingface.

#### 4.3.2 Results

As shown in Table 2, we can first obtain a similar observation in terms of human-interpretability to the previous experiment. For direct image inversion, we also immediately see a significant improvement in the readability of inverted prompts using PRISM. While expected for methods, such as PEZ, in which the process of image inversion has no language prior, we find that our method finds text that more closely aligns with a learned distribution of English language text (i.e. Mistral logits) than CLIP-Interrogator and BLIP2.

When comparing the image quality, we first note that because all images in DiffusionDB are generated by Stable Diffusion, which is exactly the model design space of CLIP-Interrogator and PEZ, it gives significant modeling assumption advantages to these baselines over our method when testing on Stable Diffusion models. This advantage enables relatively high performance for these baselines in Stable Diffusion models, but it does not transfer well into other closed-sourced models. In fact, we can even observe that CLIP-Interrogator generates the highest quality images with SD 2.1, which is the weakest model in this comparison and generates the lowest quality images with Dall-E 3, which is the strongest T2I model in this table. This phenomenon indicates that the design choices of CLIP-Interrogate and PEZ are heavily overfitted to achieve high performance on Stable Diffusion, but provide poor generalizability to other models. On the other hand, the prompts produced by our method generalize significantly better than the baselines and we achieve the best results on Dall-E 3.

Qualitatively, our method also provides prompts that are both semantically aligned with and can generate images that are visually similar to the reference.
In particular, Figure 5, shows that we can find text that aligns with the image, even when those images have particularly unique features. For example, in Figure 5 Dall-E3 generated a grid of images of animal faces. Not only does the PRISM’s prompt explicitly include a request for this grid structure, unlike our comparison methods, but it also takes into account the coloration of the background in the reference. In the second row of Figure 5, our method is also the only method that captures the small flowers in the grass, showcasing the capability of identifying and reflecting small fine-grained details from the reference.

### 4.4 Ablation Study

Figure: Figure 6: Comparison with GPT-4V in both personalized T2I generation and direct image inversion experiments.

Figure: Figure 8: Ablation study on the trade-off between N and K. All runs shown in this plot have the same budget $N\times K=30$, but each run operates a different number of iterations $K$.

#### 4.4.1 Comparison with GPT-4V.

PRISM  relies on having a strong multimodal LLM. In our case, we chose GPT-4V as the multimodal foundation model. While in principle we may use any multimodal LLM, it is nonetheless useful to understand what benefits PRISM  adds to an already capable foundation model.
To show the effectiveness of iterative prompt refinement and parallel search, we compare our method with GPT-4V’s zero-shot performance with the same system prompts for object and image inversion tasks on SDXL-Turbo. We see in Table 7 that PRISM  consistently outperforms GPT-4V’s zero-shot performance, although the latter is already compelling. In Figure 7, we show some examples of the generated results. We see that qualitatively GPT-4V can capture the high-level semantics of the reference images but still misses more fine-grained details.

#### 4.4.2 Trade-off between N and K

PRISM has two hyperparameters $N$ and $K$ which control the amount of parallel search and the depth of iterative refinement.
Figure 9 shows a trade-off between N and K with the same budget $N\times K=30$. Similar to the findings of [], we find that performance can degrade if the refinement is repeated too many times (i.e., $K$ is too large), and in general, we do not recommend practitioners with small budgets to go beyond $K=5$.
Unlike jailbreaking [], we observe that the optimal $N$ and $K$ can vary depending on the task: if the target concept is simple (e.g. a commonly seen dog), then small $N$ and $K$ are generally sufficient, and prioritizing $N$ tends to be more helpful. However, if the target concept is rarer and more complicated (e.g. a very specific toy), a larger reasoning depth (i.e., larger $K$) would be more helpful. In Figure 9, we show the distribution of iteration numbers at which the best prompt is found in the image inversion experiment. In practice, one may tune these hyperparameters further for specific use cases.

### 4.5 Prompt Editing

Figure: Figure 10: Prompt editing demonstration with Midjourney.

Because the prompts produced by PRISM is very human-interpretable, after obtaining a prompt from the reference images, one can easily modify the output prompts to change attributes in their desired generated images. Figure 10 demonstrates an examples of prompt editing with PRISM in Midjourney. With simple and intuitive prompt edit, we are able to change specific attributes of the images while keeping the other components in the scene unchanged.

## 5 Conclusion and Broader Impact Statement

In this paper, we propose PRISM, an algorithm that automatically creates human-interpretable and accurate text prompts for text-to-image generative models, based on visual concepts provided by reference images. Our method iteratively refines the sampling distribution of the text prompt via LLM in-context learning and is capable of creating prompts that are transferable to any T2I model, including black-box platforms like Dall-E and Midjourney. However, just as LLMs are suceptible to being jailbroken or adversarially manipulated by malicious actors [], our method may also be vulnerable to malicious intent, potential bias, or limitations in the base models. Therefore, we intent to implement necessary safeguards upon the public release of our code and are committed to keep up with future advancements in improving the safety of our method.

## Acknowledgements

This work was supported in part of the ONR grant N000142312368. We also thank Sony AI for sponsoring part of this work. A.R., H.H., and G.P. are supported by the NSF Institute for CORE Emerging Methods in Data Science (EnCORE). A.R. is also supported by an ASSET Amazon AWS Trustworthy AI Fellowship. Y.J. is supported by Google PhD Fellowship.

## Appendix 0.A Additional Experiment Details

In this section, we provide further details about the implementation of our experiments. For all quantitative analysis that uses Stable Diffusion based model, we generate four images for each combination of prefixes and prompts. For all experiments with Dall-E based model, we generate one image per combination. In the DreamBooth dataset experiment, we also replace the class noun for “stuffed animal” with “toy” to obtain fair comparisons with textual inversion, which can only take a single token as the initialization token.

During PRISM iterations, we allow a maximum of 5 generation attempts for each stream and each iteration in case of potential run time errors related to black-box API calls. We set the maximum number of tokens generated by the prompt engineer assistant at each iteration to be 500. This contains both the improvement and the new prompt for the target concept. We encourage the assistant to generate shorter prompts using system prompts (details in the next section) and at test time, when the testing T2I model has a shorter prompt length than the prompt generated, we clip the generated prompt to the maximum length of the respective T2I model.

When evaluating the judge scores $\mathbf{D}(x,\hat{x})$ in PRISM iterations, we shuffle the reference images when $M>1$. The judge score is rescaled into a range from 0 to 10. During re-evaluation, we re-evaluate each prompt once with each reference image when $M>1$ and re-evaluate each prompt twice and tally these scores with the in-iteration scores when $M=1$.

## Appendix 0.B Designing System Prompts

System prompting is the standard way to condition a general purpose LLM for specific tasks of request. The key idea is that, before the conversation starts, the LLM receives a tailored message, the system prompt, that provides the contexts, conversation scenario settings, formats and other guidelines as the prefix of the entire conversation ahead.
In this section, we elaborate on the design of the system prompts for the prompt engineer assistant ${\mathbf{F}}$ and the judge ${\mathbf{D}}$.
We also provide the full system prompts used in all of our experiments at the end of this paper in Section 0.F and in our demo code.

### 0.B.1 Prompt Engineer Assistant 𝐅 𝐅 {\mathbf{F}} bold_F

To design the system prompts for the prompt engineer assistant ${\mathbf{F}}$, we follow [] and include the following components in the system prompt of ${\mathbf{F}}$.

#### 0.B.1.1 Setting

We first set up the scenarios and assign a role for the LLM to better perform on the specific task of choice. The setting paragraphs start with “You are a helpful prompt engineer assistant. You are free to generate sentences that do NOT follow English grammar. You must obey all of the following instructions.” and continue with the specific description of the task and the objective. We also inform the assistant that it is expected to iterate and refine the prompts it generates throughout the conversation.

#### 0.B.1.2 Format

We then provide the guidelines for formatting the inputs and the outputs of the assistant. We describe what are expected in the inputs at each iteration and the content required in the outputs. We also provide descriptions of the meanings of each input and output components. More specifically, we inform the assistant that the inputs consist of three parts: a generated image, a reference images and a visual similarity score, and that the assistant is expected to generate both the improvement to refine the previous prompt and the next new prompt. All generated text are formatted in JSON.

#### 0.B.1.3 Examples

Finally, we provide some examples of the potential formatted inputs and outputs that the assistants may receive and produce. We also provide examples of potential improvements for the assistant. Optionally, we can also provide examples of prompts that can successfully generate the target concepts in these paragraphs.

### 0.B.2 Judge 𝐃 𝐃 {\mathbf{D}} bold_D

We follow the same strategy to design system prompts for the judge ${\mathbf{D}}$. More specifically, we set up the scene for the judge by stating “Please act as an impartial judge and evaluate …” in the system prompts and describe the visual similarity criteria based on the desired features for different tasks.
We then provide the instructions on the formatting and give an example of the expected output.

## Appendix 0.C Additional Results

In this section, we provide additional experimental results and further baselines comparisons with our method. We also showcase the flexibility of the PRISM framework by demonstrating the effectiveness of a different T2I model $G$ and a different judge $D$ in PRISM.

### 0.C.1 Additional Qualitative Results

In Figure 13, 14, 15, 16 and 17, we provide additional qualitative showcases for subject-driven personalized T2I generation, style-driven personalized T2I generation, direct image inversion and prompt editing. We also provide an example of the iteration and refinement process as a conversation between all three components in PRISM in Figure 18.

### 0.C.2 Flexible Model Choices

To further demonstrate the effectiveness and flexibility of PRISM, we also experiment a different T2I Generator ${\mathbf{G}}$ and showcase the transferability of the prompts generated by PRISM. Figure 19 shows qualitative examples of PRISM prompts with Dall-E 2 as the Generator ${\mathbf{G}}$ for personalized T2I generation and the images generated from those prompts using SDXL-Turbo, Dall-E 3 and Midjourney. Our method is capable of producing human-interpretable and accurate prompts for both subject-driven T2I personalization and style-driven T2I personalization with this new Generator ${\mathbf{G}}$.

## Appendix 0.D Additional Ablation Study

In this section, we provide a more detailed ablation study on each component of the PRISM framework. In particular, we demonstrate the effect of the existence of the Judge ${\mathbf{D}}$ and re-evaluation, different choices of the total budget, number of streams $N$ and number of iterations $K$, and also compare a non-LLM judge (a CLIP judge) against our choice of a LLM judge (GPT-4V Judge).

We first compare the performance of zero-shot GPT-4V, GPT-4V parallel search with budget 30 and the Judge to select the best resulting prompts, PRISM without re-evaluation, and two different PRISM settings with the same budget of 30. Table 3 shows the quantitative comparison among all settings using SDXL-Turbo as both the T2I Generator ${\mathbf{G}}$ and the testing T2I model on the direct image inversion task. We can observe that adding a judge, re-evaluation and more budget all have impact on the prompt accuracy improvement in PRISM, even though GPT-4V itself also demonstrates impressive performance. In Figure 20, we show qualitative comparisons on several challenging cases in the direct image inversion task using various settings of $N$ and $K$ with the same budget. These examples show that, although quantitatively all settings are able to achieve high scores, prompts generated by appropriately tuned $N$ and $K$ can produce images with higher qualitative visual alignments, especially with respect to features including finer details, overall scene layouts and the artistic styles which are more difficult to quantify with standard metrics.

Next we take a closer look at the effect of increasing the total budget in PRISM in small budget settings. Figure 12 and 12 show the effect of increasing the number of streams $N$ and the number of iterations respectively. We observe that when increasing the number of streams $N$ while keeping the number of iterations $K$ fixed, we can obtain steady performance improvements in both human readability and prompt accuracy. When increasing $K$ while keeping $N$ fixed, although we do not observe a monotonic relationship between the performance and $K$, we can still notice a general upward trend in prompt accuracy. Generally speaking, the optimal number of iterations vary case to case, and we encourage practitioners to experiment with different choices of $K$ to obtain the best resutls.

Finally, we demonstrate the importance of using a multimodal LLM as the Judge. When assessing image similarity, it is natural to default to existing metrics that do not involve LLM’s such as CLIP similarity. However, as we have mentioned in the main text, these metrics do not perform well outside of their trained notion of similarities and therefore is not very generalizable to custom tasks from users. Figure 21 demonstrates the qualitative difference between PRISM with a CLIP judge versus PRISM with a GPT-4V judge. We can observe that in subject-driven T2I personalization, CLIP judged PRISM often include irrelevant elements such as the environment (e.g. “on green grass”) and omits important details such as the color and the other distinctive features whereas GPT-4V judged PRISM can adhere better to object oriented details and ignores other unrelated factors. In style-driven T2I personalization, CLIP judged PRISM fails to capture the artistic styles and mainly focus on the general contents of the reference image. On the contrary, GPT-4V judged PRISM produces much more precise and focused prompts for the reference styles.

**Table 3: Ablation study on the effect of the existence of the Judge ${\mathbf{D}}$, re-evaluation, the budget, and different choices of $N$ and $K$. All methods use SDXL-Turbo as the T2I Generator ${\mathbf{G}}$ and also are tested with SDXL-Turbo on the direct image inversion task.**
| Method | N | K | Prompt NLL $\downarrow$ | CLIP-I $\uparrow$ |
| --- | --- | --- | --- | --- |
| GPT-4V | 1 | 1 | 2.356 | 0.756 |
| GPT-4V + Judge | 30 | 1 | 2.349 | 0.769 |
| GPT-4V + Judge | 6 | 5 | 2.615 | 0.771 |
| GPT-4V + Judge + Re-evaluation (PRISM) | 30 | 1 | 2.456 | 0.771 |
| GPT-4V + Judge + Re-evaluation (PRISM) | 6 | 5 | 2.739 | 0.776 |

Figure: Figure 11: Ablation study on different numbers of streams $N$ with the same number of iterations $K=5$.

## Appendix 0.E Limitations and Future Works

In this section, we discuss the current limitation of our PRISM framework and also potential future work directions that can help further improve the performance of our method.

Firstly, as we can observe in almost all of the qualitative examples, when the targeting concept is more challenging (e.g. a very particular toy), our method still fail to capture all the fine grained details in the image generation. Although this phenomenon is to some extent expected due to the fact that text-to-image generation is not a one-to-one function, there is still a long way to go in order to achieve the same performance as methods like DreamBooth [] that involve finetuning. Moreover, even with very accurate prompts, because of the limitation of the downstream testing T2I models, sometimes it still fail to generate the correct concepts. One potential direction is to combine gradient-based search methods like PEZ [] with PRISM to create model-specific prompts similar to CLIP-Interrogator.

Another drawback of our method is that, similar to real life prompt tuning, the optimal numbers of streams and iterations are very instance dependent. In other words, for different target concepts, depending on whether it is more commonly seen and better defined or more peculiar, the optimal budget required can vary drastically. An interesting question to answer will be how to better automaticallly decide the minimal budget required for a certain target concept.

Performance wise, although qualitatively the difference is very difficult to notice, we do find that our method marginally falls short in CLIP-T score, which is the score that measures the context-image alignment in the task of subject-driven T2I personalization (shown in Table 4). A potential solution is to have a stricter constraint on the length of the prompts generated by our method, and we leave this direction also to future work to explore.

Finally, we want to re-iterate the potential societal impacts of our work. Just like LLMs are prone to jail-breaking and leaking, we also do not guarantee complete protection against malicious use intent, underlying bias and other limitations inherent from the base models. We are committed to implement and constantly improve the safety precautions in our code base after its public release, and we encourage practitioners to also take preventative actions in order to mitigate these potential issues.

**Table 4: Quantitative comparison on CLIP-T scores.**
| Method | SD 2.1 | SDXL-Turbo | Dall-E 2 | Dall-E 3 |
| --- | --- | --- | --- | --- |
| Textual Inversion (SD 2.1) | 0.234 | - | - | - |
| Textual Inversion (SDXL) | - | 0.231 | - | - |
| CLIP-Interrogator | 0.225 | 0.229 | 0.219 | 0.218 |
| BLIP-2 | 0.241 | 0.259 | 0.252 | 0.250 |
| PEZ | 0.247 | 0.249 | 0.237 | 0.234 |
| PRISM (Ours) | 0.229 | 0.233 | 0.241 | 0.241 |

## Appendix 0.F Full System Prompts

### 0.F.1 Subject-Driven Personalized T2I Generation

#### 0.F.1.1 Prompt Engineer Assistant 𝐅 𝐅 {\mathbf{F}} bold_F

Below is the system prompt for ${\mathbf{F}}$ in subject-drive personalized T2I generation.

#### 0.F.1.2 Judge 𝐃 𝐃 {\mathbf{D}} bold_D

Below is the system prompt for ${\mathbf{D}}$ in subject-drive personalized T2I generation.

### 0.F.2 Style-Driven Personalized T2I Generation

#### 0.F.2.1 Prompt Engineer Assistant 𝐅 𝐅 {\mathbf{F}} bold_F

Below is the system prompt for ${\mathbf{F}}$ in style-drive personalized T2I generation.

#### 0.F.2.2 Judge 𝐃 𝐃 {\mathbf{D}} bold_D

Below is the system prompt for ${\mathbf{D}}$ in style-drive personalized T2I generation.

### 0.F.3 Direct Image Inversion

#### 0.F.3.1 Prompt Engineer Assistant 𝐅 𝐅 {\mathbf{F}} bold_F

Below is the system prompt for ${\mathbf{F}}$ in direct image inversion.

#### 0.F.3.2 Judge 𝐃 𝐃 {\mathbf{D}} bold_D

Below is the system prompt for ${\mathbf{D}}$ in direct image inversion.

Figure: Figure 13: Qualitative examples of the subject-driven T2I personalization task tested on open sourced T2I models.

Figure: Figure 14: Qualitative examples of the subject-driven T2I personalization task tested on closed sourced T2I models.

Figure: Figure 15: Qualitative examples of the style-driven T2I personalization task.

Figure: Figure 16: Qualitative examples of the direct image inversion task.

Figure: Figure 17: Qualitative examples of the prompt editing task with Dall-E 3.

Figure: Figure 18: An example of the iteration and refinement process as a conversation between the three components of PRISM. Only the system prompts (labeled as “system”) and the first two iterations are shown in this example.

Figure: Figure 19: Qualitative examples of the subject-driven T2I personalization task using Dall-E 2 as the T2I Generator ${\mathbf{G}}$.

Figure: Figure 20: Qualitative examples to showcase the effect of different numbers of streams $N$ and iterations $K$ on PRISM with the same budge $N\times K=30$.

Figure: Figure 21: Qualitative comparison between using the CLIP model as the Judge ${\mathbf{D}}$ in PRISM and using GPT-4V as the Judge.

## Peer Reviews

confidence: 5
contribution: 1
initial_rating: 5
presentation: 4
questions: Please comment on the mentioned weaknesses.
soundness: 3
strengths: 1. The paper is well written. The structure makes it clear and easy to follow.
2. The experiments are comprehensive and well executed
3. Higher interpretability and transferability compared to other personalization / invertion methods
summary: The paper proposes a training-free iterative algorithm based on in-context learning of Multimodal LLMs (MLLM) to refine prompts for text-to-image (T2I) generation that would increase alignment of the generated images with a given set of reference images. The algorithm, PRISM, makes use of a MLLM to first zero-shot guess a caption for the reference images, then it uses the caption to generate a new image, which is evaluated in terms of visual similarity to one of the reference images using another multimodal model (in practice, the same model is used). Then the new prompt and its generation are included in the metaprompt of the MLLM for proposing a new caption based on in-context learning. This pipeline is executed for multiple iterations and with different bootstrapped captions in parallel. Eventually, the prompt generating the most similar image to the entire reference set is selected. The experiments test PRISM for personalized image generation/editing, and image inversions. A comparison with other baselines from these tasks show on-par or better performance when testing with different T2I models.
weaknesses: 1. The idea is not novel. There exists similar attempts, .e.g., Manas et al 2024,  Liu et al 2024, Yang et al 2023. Manas et al propose a very similar iterative algorithm based on in-context learning. The only (small) difference lies in the tasks, which here is personalization / inversion, while in Manas et al is improving prompt-image consistency in general (with no reference images).  Yang et al work is similar as it relies in on GPT4V, which is used to evaluate and propose new candidates prompt based on an initial idea of the user (kinda personalization). Finally, Yang et al, explore in a very similar the use of GPT-4V for text to image generation and prompt invertion (see Section 6 of the paper). Moreover, despite less related, i.e., do not need to be compared, there exists a few work that proposed prompt improvement based on LLM fine-tuning, e.g., Prompist (Hao et al 2024). I think these works might be worth a mention in the related work section even tho clearly using a different approach.  
2. The improvements w.r.t. GP4V is marginal. Table 3 shows quantitatively that PRISM only improves 0.02 of ClipScore on images, while having higher (worse) negative log likelihood.


* _Mañas, Oscar, et al. "Improving text-to-image consistency via automatic prompt optimization." arXiv preprint arXiv:2403.17804 (2024)._
* _Yang, Zhengyuan, et al. "Idea2img: Iterative self-refinement with gpt-4v (ision) for automatic image design and generation." arXiv preprint arXiv:2310.08541 (2023)._
* _Liu, Shihong, et al. "Language models as black-box optimizers for vision-language models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024._
* _Hao, Yaru, et al. "Optimizing prompts for text-to-image generation." Advances in Neural Information Processing Systems 36 (2024)._

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: [Questions numbered according to each Weakness above]
1. How might known issues with VLMs affect the efficacy of PRISM (e.g. compositionality and counting), and what are possible mitigations?
2. How do quantitative improvements across iterations and associated budget correspond to visual improvements in images? 
3. How well are different kinds of visual similarity, and their potential mutual exclusivity, captured by PRISM in Experimentation? Is it possible to prioritize certain kinds of criteria via the meta-prompt for the judge or generator within the method? Are there potential gaps in achieving some kinds of visual similarity?
soundness: 3
strengths: * The PRISM method is presented with clarity, is intuitive, and straightforward to implement.
* The experimentation applies PRISM to two useful usecases - text-to-image personalization and image inversion - demonstrating its utility.
* The authors apply PRISM to three different families of models and several generations of models, demonstrating extensibility across model classes 
* The analysis includes good coverage over other methodologies, and the PRISM method shows strong performance compared to prior works, highlighting its efficacy.
* The paper includes useful qualitative discussions about patterns of the generated prompts and subsequent images, including highlighting strengths of the method such as maintaining structural consistencies or small visual details in images.
* The method usefully recommend alternatives to CLIP in the selection of the judge model, noting its weakness in the context of promp generation.
summary: This work presents the PRISM algorithm for producing prompts that can generate desired concepts given an input of images and (optionally) an original prompt. PRISM iteratively generates images with a text-to-image model, assesses the images with a vision-language model, and generates new prompts with a large-language model. The method is applied to two kinds of prompting situations - personalized text-to-image generation and prompt inversion from a given set of images. The work includes comparisons across existing methods, with both quantitative and qualitative analyses. The work also includes ablations over alternative vision-language models (GPT-4V) and the number of iterations used.
weaknesses: 1. The work would be strengthened with greater discussions about possible weaknesses of VLMs as judge models and how these concerns may be mitigated. For example, VLMs have known issues with compositionality and counting [1,2]. 
2. While the ablation focusing on budget provides useful insights, it would be useful to contextualize how the improvement in metrics translate to visually perceptible improvements to better contextualize the extent to which the iterations are required for human-perceived improvements. 
3. Section 3.1 and 3.4 usefully highlights that the visual similarity criteria may relate to different kinds of image qualities, such as a focus on the main objects in the image, style of the image, or general similarity. However, the Experimental Section 4.2 does not have much discussion about how these different similarity criteria are captured by the PRISM methodology and how it affects the visual representation of the newly generated images using the PRISM-generated prompts.
4. [minor] The maintenance of the whole prompt distribution in the prompt generation process (mentioned in Section 3.2) has been studied in other works [3]. It would be useful to contextualize among this prior work.

References:
[1] Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality
[2] WHEN AND WHY VISION-LANGUAGE MODELS BEHAVE LIKE BAGS-OF-WORDS, AND WHAT TO DO ABOUT IT?
[3] Improving text-to-image consistency via automatic prompt optimization

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: see weaknesses

+)
Have you explored other tasks in the PEZ paper, such as style transfer, prompt distillation, or multi-concept generation (prompt concatenation)? Since this task requires a much shorter prompt than the maximum text length allowed by CLIP (for concatenate the different prompts), if PRISM struggles with generating shorter prompts, it may not perform well in those tasks either.
soundness: 3
strengths: 1. Unlike previous techniques that generate only a bag of words, PRISM generates fully human-readable prompts for image generation.

2. PRISM has a straightforward implementation, as it does not require model training.

3. The paper is written in a clear and accessible manner.
summary: This paper proposes a prompting technique for text-to-image generation, termed PRISM, utilizing a Large Multimodal Model (LMM). PRISM iteratively generates image captions, evaluates the generated images, and refines the images based on the evaluation scores. By providing scores to the LMM in a manner similar to the chain-of-thought approach, PRISM guides the LMM to improve captions for enhanced image generation quality.
weaknesses: 1. In Line 187 and Line 9 of the algorithm, you state that$\text{P}_{\theta_F}$ is updated; however, the method for updating the distribution is not specified in the method section. In the Appendix, it appears that this process simply involves prompting the score generated by the discriminator. How, then, can this be interpreted as updating the distribution? This could lead to misunderstandings, as “updating the distribution” generally implies updating the model parameters.

2. In Figures 3,4, and 5, the length of the prompts used in the comparison techniques varies, so it is necessary to clearly indicate the settings for each comparison group. However, varying the prompt length in this way seems to make the experiment somewhat unfair.

3. There is no ablation study for the LMM model. The authors should conduct experiments with other LMMs (e.g., BLIP-2, LLaVA) to assess the generalization capability.

4. There is no ablation study on prompt length.

5. Iteratively prompting a large model is computationally expensive, as noted in the Appendix.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: * Is there a mechanism that strictly guarantees the candidates’ score (provided by the Judge) keep increasing? Or does the model itself implicitly increase the score? I am asking this question because I am not sure whether the “scoring” process is necessary, and whether the “score” actually increases through iteration.
* Does the concept of “stream” equal to the beam search candidates? There are not many details about generating diverse streams N.
soundness: 2
strengths: * PRISM is compatible with any black-box T2I models, Judge, and Prompt engineer models. This model-agnostic design and hard prompt generation seem to be a good combination. In addition, PRISM does not require any training.
* The paper demonstrates PRISM’s effectiveness using a variety of experiments. Quantitative evaluations include NLL, CLIP-I, and DINO scores. Qualitative demonstrations present various applications including personalized T2I, direct image inversion, and easy prompt editing.
* The paper thoroughly discusses the background and related works.
summary: The paper introduces PRISM, an automated hard prompt generation algorithm designed for text-to-image (T2I) generation. PRISM minimizes the needs for human intervention, avoiding laborious trial-and-error prompt creation processes. PRISM integrates T2I generator, Judge, and Prompt Engineer models, where the latter two are existing black-box Vision-Language models. Generated prompts are intended to be human-interpretable and transferable across various T2I models. Experimental results show that PRISM achieves lower NLL and higher image similarity scores compared to previous methods such as TI or PEZ.
weaknesses: * Although the automated process would reduce human effort, PRISM’s iterative process may require huge computational resources and time (as discussed by authors in the limitation section). On the other hand, there is no efficiency (latency, resource usage) comparison between PRISM vs. pervious approaches. For example, CLIP-Interrogator may not achieve the best performance; but given the same resources, does PRISM show better results than CLIP-Interrogator?
* N seems to be more critical than K. For example, in Table 5, using (N, K) = (30, 1) shows a much lower NLL than (6, 5) while maintaining a similar CLIP-I score. This implies that the quality of the initial prompt/candidate may be a key factor in the final performance, making the role of iterative refinement less important. To better illustrate this, it would be useful to show (1) the distribution of initial “scores” across N, and (2) the “score” improvements for all candidates through multiple iterations. These “scores” may be represented by the Judge score, or CLIP-I score, directly reflecting the image quality.

Meta-review: This paper introduces an in-context learning approach to iteratively update prompts for to improve text-to-image generation. The paper was reviewed by three knowledgeable reviewers, who acknowledged that the paper was well presented and clear (2eVw, 9NvK), and the method's design  simple (9NvK) and model agnostic (uivB). The main concerns raised by the reviewers were:
1. Unclear novelty of the proposed approach (7uVm)
2. Positioning w.r.t. prior art
3. Limited discussion of weaknesses (2eVw)
4. Unconvincing experimental evidence: marginal improvements (7uVm), missing ablations (9NvK) or resource-fixed comparisons with CLIP-Interrogator (uivB)

During rebuttal/discussion phase, the authors partially addressed the reviewers' concerns by arguing for the novelty of the proposed approach (targeting a different task than prior work) and their achieved improvements, adding comparisons with some existing methods pointed by the reviewers, analyzing the effect of iterations, adding ablations on the effect of prompt length, clarifying questions related to the method/experiments, and discussing resource and latency comparisons. After discussion, the reviewers remain unconvinced. In particular, concerns w.r.t. novelty persist (LLM as optimizers have been applied for T2I models, only the end task differs), and the experimental results, which should include more comparisons with prior work as well as CLIP Interrogator (the latter would need to be evaluated by adjusting the captioning model). For these reasons, the reviewers lean towards rejection during the discussion period. The AC agrees with the reviewers' assessment and recommends to reject. The AC encourages the authors to consider the feedback received to improve future iterations of their work.

Decision: Reject