Automated Black-box Prompt Engineering for Personalized Text-to-Image Generation
Authors: Yutong He
Alexander Robey
Naoki Murata
Yiding Jiang
Joshua Nathaniel Williams
George J. Pappas
Hamed Hassani
Yuki Mitsufuji
Ruslan Salakhutdinov
J Zico Kolter
Venue: Accepted by TMLR ()

Abstract: Prompt engineering is an effective but labor-intensive way to control text-to-image (T2I) generative models. Its time-intensive nature and complexity have spurred the development of algorithms for automated prompt generation. However, these methods often struggle with transferability across T2I models, require white-box access to the underlying model, or produce non-intuitive prompts. In this work, we introduce PRISM, an algorithm that automatically produces human-interpretable and transferable prompts that can effectively generate desired concepts given only black-box access to T2I models. Inspired by large language model (LLM) jailbreaking, PRISM leverages the in-context learning ability of LLMs to iteratively refine the candidate prompt distribution built upon the reference images. Our experiments demonstrate the versatility and effectiveness of PRISM in generating accurate prompts for objects, styles, and images across multiple T2I models, including Stable Diffusion, DALL-E, and Midjourney.

Published in Transactions on Machine Learning Research (08/2025)
Automated Black-box Prompt Engineering for Personalized Text-to-lmage Generation
Yutong Hel, Alexander Robeyl.2, Naoki Murata Yiding Jiang', Joshua N. Williams', George J. Pappas", Hamed Hassani', Yuki Mitsufuji Ruslan Salakhutdinov's J. Zico Kolter Carnegie Mellon University's ,University of Pennsylvania ,oony Af, ,Sony Group Corporation' ,osch Center for Af
Reviewed on OpenReview: https: // openreview. net/ net/forum? forum? id= JVYVDN6p66
Abstract
Prompt engineering is an effective but labor-intensive way to control text-to-image (T2I) generative models. Its time-intensive nature and complexity have spurred the development of algorithms for automated prompt generation. However, these methods often struggle with transferability across T2I models, require white-box access to the underlying model, or produce non-intuitive prompts. In this work, we introduce PRISM, an algorithm that automatically produces numan-interpreee generate desired concepts given only black-box access to T2I models. Inspired by large lan- guage model (LLM) jailbreaking, PRISM leverages the in-context learning ability of LLMs to iteratively refine the candidate prompt distribution built upon the reference images. Our experiments demonstrate the versatility and effectiveness of PRISM in generating acen- rate prompts for objects, styles, and images across multiple T2T models, including Stable Diffusion, DALL-E, and Midjourney, Our code is available via the project page here
and transferable prompts that can effectively
1 Introduction
An important goal of generative modeling is to design algorithms capable of steering generative models to pro- duce desired output images. Early attempts, which often centered on particular architectures or tasks, were
O
A shete wy dag the boach
Prompt Engineer Assistant F
Stable Diffusiin
fluffy
with golden-
Chow dog
+
A whate of wy dag sleeping
and blue tongue
oo
Dall-E
Output Prompt
A shote of dog skingg down movntain
T21 Generator G
Reference Images
Judge
PRISM
Midjourncc
Figure Given a set of reference images, our method, PRISM, is capable of creating human- iuman-interpretable and accurate prompts for the desired concept that also transferable to both open-sourced and closed- sourced text-to-image models. b denotes prompt concatenation.

---

Published in Transactions on Machine Learning Research (08/2025)
largely characterized by mamaaly--curatdd data collection, fine-tuning, or retraining from scratch (Srivastava
& Salakhutdinoy 2012 Mirza & Osindero 2014 Isola et al. 2017 Zhu et al.] 2017). These requirements are often costly, and the solutions usually do not transfer well between models. Thus despite the promise of
these methods, efficient and generalized algorithms for controllable generation remain sought after.
Today, perhaps the most popular approach for controllable generation is to guide the generation process with a piece of textual information, or prompt, that descríbes the properties of the desired output using text-to- image (T2I) generative models (Rombach et al. 2022: Yu ct al. 2022). Through text, T2I models allow users to quickly and easily describe wide variety of concepts, and users can more efficiently explore the behavior of their model through a myriad of strategios Chattogiss/Chao et al. (023);; Wen et al. (2023). The prodominant method for obtaining such input text is to manually design candidate prompts in an iterative, trial-and-error fashion, a process known as prompt engineering. based on what the user (prompt engineer) believes will lead to a desirable output. Unfortunately, these practices are often sensitive to different phrasings (Webson & Pavlick, 2022), require expert domain knowledge, and are notably inefficient as they necessitate a human in the loop.
Motivated by the drawbacks of manual prompt engineering, recent line of work known personalized or subject-driven T21 generation has sought to automate the controllable generation pipeline. Given a collection of reference images that capture specific concepts, such as artistic style or shared objects, personalized T2I algorithms are designed to produce images that reflect those concepts illustrated in the reference images. While personalized T21 methods often involve fine-tuning or retraining the underlying T21 model (Ruiz
et al. 2023 Chen et al., 2023 Shi et al.) 2023 several approaches focus specifically on automating prompt engineering to generate effective prompts. Unfortunately, existing algorithms in this spirit tend to require pre-collected, architecture-speeffic keyword or white-box, embedding-based optimization (Gal et al. 2023 Mahajan et al.) 2023 leading to non-interpretable prompts (Wen et al. 2023) and preclude the possibility of directly generating prompts for closed-source T2I models (e.g., Midjourney or DALL-E).
In order to address these shortcomings, we propose Prompt Refinement and Iterative Sampling Mechanism (PRISM), a new automated prompt engineering algorithum for personalized T21 generation. A key observation is that prompt engineers repeat the process of updating their "belief" of what makes an effective prompt based on the difference between their desired results and the generated images from previous iterations. Inspired by jailbreaking attacks on large language modcls (LLMs) (Chao et al. 2023) and LLMs optimizers (Pryzant 2023 we design an algorithm that operates with only limited human input, is capable of generaling human interpretable and editable prompts, makes minimal assumptions about the underlying T21 model, and generalizes across different T21 models, including popular black- box models such as DALL-E and Midjourney.
et al.
Given a set of reference images, our method first generates an initial prompt and ils corresponding image using a vision-language model (VLM) as "prompt engineer assistant" and a T2I generator. We then obtain a score indicating the visual similarity of the generated image and the reference image with respect to the targeting concept via another VLM as judge. Leveraging LLMs in-context learning abilities (Shin et al., 2020 Zhou et al., 2023 Yang et al. 2024), we instruct the prompt engineer assistant VLM to update the candidate prompt distribution based on the previously generated prompt, images, and the evaluation scores. This processing, shown in Figure 1 is then repeated for a predetermined number of iterations. In the end, PRISM outputs the best-performing prompt by re-evaluating the top prompts generated from this process. In this way, PRISM scamlessly integrates iterative reasoning into the image generation process, much like a real prompt engineer. Our approach can therefore go beyond basic inage-to-inage transformations and conventional single-shot methods, providing a more versatile and robust framework for generating images that are both visually precise and contextually relevant.
Experimentally, our method shows significantly better generalizability and transferability as we achieve the best performance in almost all metries when experimenting with closed-source models in comparison to baselines including Textual Inversion (Gal et al. 2022), PEZ (Wen et al.] 2023). BLIP2 (Li et al. 2023) and CLIP-Interoggatooll Our results also indicate that PRISM consistently outperforms existing methods with respect to human-interpretabiliyy while maintaining high visual accuracy. Finally, we demonstrate that the strong human interpretability makes the prompts generated by PRISM casily editable, unlocking a wide array of creative possibilities in real life.
2

---

Published in Transactions on Machine Learning Research (08/2025)
2
Related Works
Controllable T2I generation Several methods tackle conditional image generation in training-froo
manner by using pretrained diffusion models as priors (Meng al. 2022 Chung et al.
2023 Song et al.
2022
He et al.
2023 and analogous approaches exist for T2I diffusion models (Yu et al. 2023
Rout et al.
2023
He et al. 2024 However, these methods assume that the controllability objectives can be formulated
differentiable loss functions, require access to model parameters and involve complex hyperparameter tuning,
Another class of approaches (Zhang et al. 2023 Ye et al. 2023 Ruiz et al. 2023 Chen et al. 2023 Shi et al.
2023 also improve the controllability of pretrained T21 models, but they require expensive tin-tuuing or re- re-training of the underlying model every time they are applied to a new lask. Prompt tuning methods (Gal
et al. 2023 Wen et al. 2023; Mahajan et al. 2023) are in the same spirit as this paper, as they do not require training of the T2T model and condition generations on given reference images. However, unlike PRISM, these methods require access to the underlying model parameters produce non-interpretable prompts.
Prompt engineering Manual prompt enginecring is a popular approach to cliciting desired behaviors from large pre-trained models because it uses little or no data and does not require fine-tuning (Radford et al. 2021 Brown al. 2020). However, major drawbacks of manual prompt engineering include its laborious nature, its reliance on domain expertise, and its sensitivity to phrasings (Lu et al. 2022 Webson & Pavlick 2022). To address this issue, several methods have been proposed to construct prompts in an
automated manner (Shin et al., 2020 Gao et al.) 2021 Zhou et al. 2022 2023 Manikandan et al. 2023
Pryzant et al.} 2023 Yang et al., 2024). and some have applied similar techniques to various downstream
tasks (Mañas et al. 2024 Liu et al. 2024 Yang et al. 2023 Hao et al.) 2024). In particular, [Lu et al. (0024)
applied the algorithm they designed for image classification to image inversion. Moreoever, LLM jailbreaking
focuses automatically designing prompts that elicits specific content (often objectionable illicit) from
targeted LLM Zoou ct al.] 2023 Wei et al. 2024 Robey et al., 2023 Liu al. 2023). A particularly
relevant work is Chao et al. (2023). which uses an auxiliary LLM to iteralively construct jailbreak prompis.
Our method builds on this idea to create prompts to generate images satisfying the desired criteria.
3 Method
3.1
Problem Statement
First, let x E X denote image, and y E y denote textual prompt. Given collection of reference images M a prompt engineer F ::2 X' + samples candidate prompt y corresponding to each reference image 2, i.e., y ~ Pep (y) x). A T21 generative model G : y + X then uses this candidate prompt to generate a new image, I ~ Pag (I y), and a judge model D : X' + [0, 1] then scores the visual similarity between the images based on some criteria. Our goal is then to find the best prompt:
M
y² ((x)) = arg max DEY
s(1,,))
(1)
where s ((targgtt v) = Rampas (xll) D(r, Ptarget)]
The criteria can be any visual similarity metric that may or may not be easy to specify in a closed form, including "how similar are the main objects in the images" or "how similar are the styles of the image" or "how similar are the two images in general". The resulting y* should be able to generate an image that is very close to the reference images basod on the criteria with some (possibly unscen) T2I models po (x v).
3.2 Algorithm
Our method, Prompt Refinement and Iterative Sampling Mechanism (PRISM), is an iterative process that repeats a prompt refinement subroutine for K iterations in N parallel streams, where N K a predeter- mined compute budget. At iteration k. the n-th stream of PRISM randomly selects reference image In from (xx),M M and uses F to sample a candidate prompt yk,, from pep (y æk,n). Then it queries G to generate single &t.n from y with paa (x: yk,n) and evaluate the prompt with D to obtain in-iteration score
3

---

Published in Transactions on Machine Learning Research (08/2025)
I am a helpful prompt engineer assistant. generate great prompts for this object...
Sone of the colors and the design features are missing- I should improve prompts accordingly..
System
"a photo of a
Score: 5/10
@ Prompt Engineer Assistant F
Generated Image
"pastel-colored sneaker with purple toe, white laces, grayish-white vamp, yellow stripe, white midsole"
Reference
T2I Generutor G
Reference Images
Judge
am an impartial judge and will try my best to rate the similarity of the objects these images...
System
Generated Images
Figure 2: An illustration of PRISM. "System" indicates the system prompt setups for the VLMs.
s' (I (k,n) (z d. At the end of the iteration, we use the generated yk.n and its score to update pop (y z). After the entire process, collect the subset of {yele generated throughout this process that has the C-best in-iteration scores. Then we re-evaluate this subset with the total score ΣM s(2;, Vc) and return the prompt with the best total score. If there tie, then we return the prompt with the highest log likelihood. The pseudocode and illustration are outlined in Algorithm D and Figure Seggurerrsppree respectively
The key difference between PRISM and prior methods is that PRISM updates the entire sampling distri- bution of prompts, whereas prior works (Gal et al. 2023 Wen et al. 2023 Mabajan et al.) 2023) directly update the tokens or the embeddings of a single prompt. We believe that maintaining the whole prompt distribution beneficial as text-to-image generation not a one-to-one operation, i.e. an image can be de- scribed by multiple different text prompts and the same text prompt can correspond to multiple differently generated images. Having access the whole distribution allows the method sample more diverse range of prompts without starting from scratch and may also help the optimization escape potential local optima.
Since PRISM only requires samples from G, one may use any T2I model of their choice. However, careful consideration is needed when designing F and D, which we will elaborate on below.
3.3 Designing and updating F and p0p
What is p(yl x)? In general, is not obvious what the joint or the conditional distribution of all text and images is, so some form approximation is unavoidable. In the context image generation, a natural choice of the inage-conditioned text distribution an image captioning model. Traditional captioning models, however, fall short in controlled image generation for two primary reasons: (1) The level of detail necessary for generating specific images far exceeds what generic captioning models provide (Liang et al. 2023 (2) effective prompts for T2I models are often not grammatically correct sentences but rather collection of phrases that describe the details about the image, which generic captioning models are not trained to generate. For example, in Figure [] the second reference image is generated by the prompt "A broken robot lays on the ground with plants growing over it, somber, HD, hyper realistic, intricate detai" with Stable Diffusion, but caption for this image will not include components like "HD" "hyper realistic". Hence, instead of "a good description of image", we wish to directly model "possible prompts used generate this image".
4

---

Published in Transactions on Machine Learning Research (08/2025)
Algorithm Prompt Refinement and Iterative Sampling Mechanism (PRISM)
1: Input: N streams, K iterations, {ax) reference images
2: Output: Best prompt y" based on total score in parallel do for k= to K do
3: for n = 1
Randomly sample Tk,n from {x:) MM
6: F samples Me, ~ Por (y Tk,n)
7:
G samples $g,n P0c (xr Wk,.)
8:
D calculates an in-iteration score s' (Z Bkmn) (Z (kkn)
9: Update por based on Dk,n Rk.n nk s' (Zp,ns Vk,,a) and the chat history of stream n
10: end for
11: end for
12: Collect the subset {yeJem with the C-best in-iteration scores
13: Re-evaluate this subset with total score ΣM 8(F,; Ve))
14: Return the prompt with the best total score. In case of tic, return the prompt with the highest log likelihood.
Desiderata A desirable F can sample from distribution pop (y x) that models "the prompt that can be used to generate this image", and it should also be easily updated if the current generation is suboptimal. Ideally, such an update can be done without any retraining or fine-tuning since these operations are generally expensive and incompatible with black-bon T21 models.
Vision-Language Models as F VLMs stand out as the ideal choice for F due to their ability to directly tailor the generation prompts via system prompts and to adapt through in-context learning without roquiring access to the model's parameters. Specifically, since the model can ingest both images and texts, we can incorporate the history of reference images, intermediate prompts, generated images, and the evaluation scores all in the context of the LLM. Then, the model can be prompted jointly reason over all available information and perform in-context learning. The in-context learning facilitates iterative refinement of the prompt to update the posterior distribution based on feedback or even additional human instructions, without the need for model retraining. Concretely, the model would process how the image generative model is affected by different prompts, propose improvements, and create new prompts, much like a prompt engineer. This way, we can naturally incorporate iterative reasoning into the image generation process and go beyond simple image-to-image transformations and traditional single-shot generation, and thus offers a more robust and versatile framework for producing accurate and contextually relevant images. In practice, design system prompts that explicitly condition the LLM to generate improvements and new prompts given the results from the previous iterations, similar to the chain-of-thought (Wei el. al. 2022 and textual gradients Prryzant et al. 2023) technique.
3.4 Designing the judge model D
We have a wider range of choices for D as long it provides a notion of similarity between pair of images. A simple solution is to use pre-trained discriminative models such as CLIP (Radford et al. 2021), and measure the distance in their embedding spaces. While these models have seen various degrees of success, they also come with inherent limitations the discriminative objective (e.g., contrastive loss) does not incentivize the model to attend to fino-grained details, an issuc similar to the shortcomings using captioning models generale prompts (Liang al.] 2023). Moreover, in image generation, the criteria of success can be nuanced and difficult to quantify through traditional similarity metrics yet be effortlessly described in human language. Lastly, the similarity we wish to measure may only involve some part of the visual features (c.g. color), and not all applications share the same notion of similarity. If we want to use pretrained discriminative models, then we need to find a different model for each task, which can be inefficient.
In light of these challenges, an idcal judge model should be flexible for different kinds of criteria and can perform fine-grained analysis of the images. Once again, VLM emerges as the perfect candidate: using system prompts and in-context learning, can easily specify metrics that may be otherwise difficult to
5

---

Published in Transactions on Machine Learning Research (08/2025)
describe or evaluate and even intervene in the reasoning chain if we want to, and, more importantly, the same model can be applied to wide range of tasks.
4 Experiments
4.1 Experimental Settings
Implementation Details For all of our experiments, we choose GPT-4V (OpenAI, 2023) as both the prompt engineer assistant model F and the judge D. We also fix the T2T generator as SuXT-Turbo (Sauer) et al. 2023 for all of our experiments. We design different system prompts for both F and D for each task and wC provide details about the system prompts in the appendix.
We evaluate the prompts generated from PRISM and baselines with five different T2T models. In particular, we choose two open-sourced models, Stable Diffusion 2.1 (SD 2.1) and Turbo,urbo, and two closed-sourced models, Dall-E and Dall-E 3, to quantitatively measure the performance. We also qualitatively showcase results from Midjourney, which is another closed-sourced T2I platform. For SD 2.1 and SDXI-Turbo, we clip all prompt lengths to 77 due to their context length constraint.
We compare PRISM and baselines in two setlings: personalized T21 generation and direct image inversion, and we will elaborate on the task definitions in their corresponding sections below. For personalized T21 generation, we use a maximum budget of 40 and report the quantitative results with = 10, K = 4. For direct image inversion, we use a maximum budget of 30 and report the quantitative results with = 6, K 5.
CLIP- Interrogator (CLIP-Int) and PEZ (Wen et al. 2023) as the baselines. Textual Inversion trains a "soft token" which cannot be directly translated into regular human language to represent the concepts in the reference images. BLIP-2 is the state-of-tho-ae image captioning model. CLIP-Interrogatooooommbmiii combincs BLIP- captions with a sullix which created by searching a pre-collected bank of keywords using CLIP (Radfrdd let al.) 2021) score. PEZ is a gradient- braee optimization method that searches for the best combination of existing tokens in the vocabulary with CLIP similarity. For image inversion, we also include VLM- based baseline, Liu et al. (2024), to demonstrate the effectiveness of our algorithm. Notice that TI requires training on individual models and CLIP-Int requires a pre-collected keyword bank, both of which provides unfair advantages over our setting.
Baselines We choose Textual Inversion (TT) Gal et al. 2023 BLIP-2 (BLIP2) Li et al.
2023
We evaluate the prompt interpretability using mean negative likelioel (NLL) caleulated from Mistral 7B (Jiang et al., 2023). For image quality evaluation, we mainly measure the CLIP image similarity score (CLIP-I) to quantily ily the difference between the generated images and the reference images. Following Ruiz et al. (2023), we also use DINO V2 (Oquab et al., 2024) embedding similarity to calculate the object-sensitive image similarity for the personalized T21 generation task. We chose CLIP-ViT- L-14 and DINO-V2- V2-Base as the base models. For Dall-E and Dall-E 3, we also compare the number of times each method fails to pass its black-box safeguard. More failures indicate a higher potential to produce unsafe prompts. For each prompt, we allow 5 attempts before counting it as failure.
Evaluation Metrics
4.2 Personalized Text-to-lmage Generation
We first demonstrate PRISM's ability to find human-interpretable and accurate prompts to describe certain objects or styles in the task of personalized T2T generation. Given a set of reference images that depict the same concept (such as objects and style), this task requires the T21 model to synthesize images in new contexts while maintaining the original concept.
Datasets We use DrcamBooth datasct (Ruiz et al. 2023) to quantitatively compare the performance in personalized T2I generation. DreamBooth dataset contains 30 daily objects and each subject has 4-6 images. For each subject, we adopt the 25 prompt templates curated by DreamBooth to create varying contexts and scenarios to test the fidclity of the subject representation in diverse settings. We generate images for each subject and template combination with open sourced T2I models, and image for each combination with closed sourced T2T models. For Textual Inversion, we follow the original setting to fill the templates and for
6

---

Published in Transactions on Machine Learning Research (08/2025)
Reference
Context
Textual Inversion
CLIP CLII-Inierrogafor
BLIP2
PEZ
PRISM (Ours)
cogi peneeiliooed
Early Resledd
uith
SD2i
dog in the jungle
Nat Applicable
eller wili unch
on tongre
Eatrer kencu
have leree Hortiiae
resling otprosins
Aouir athidd
SDXL- Tuebo
with the Eirtel Tower the bockpround
Not Applicable
dietee lubles foc
pair belographic sneakess brick wall
ciler Baiedcoci
raginte Lntil herge
lelegrohiic
seatrral chando
pol Jesr lia light rike
Doll-E
cat in
Pecispelbllc sloqplictace auridaaia piofla
purple wizard outfit
Bowent peined
2014
with suturtat leaves the bockground
hore tet gebrgaa Ics caan
vilir araitedd avee trimin cefiemnn solle checr day
red stufted antmal sttiigg on of montoo
Dall-E
Not Applicable
pate
Midjounny
Ang luge
my backpack the river
dag backpack
mights comton 301.
laye hags can feeddyy
Figure 3: Qualitative results for personalized T2I generation on DrcamBooth dataset.
Table 1: Personalized T2T results on DreamBooth dataset. Bold fonts indicate the best score and underlines indicate the second best score.
Prompt
SD 2.1
SDXL Turba
Dall-E
Dall-F.
Method
NLL
CLIP-I f
DINO 1
CLIP-1 DINO CLIP-I
DINO f Failed CLIP-1
DINO
Failed
TI 2.1)
0.707
0.443
(SDXL)
0.771
0.504
CLIP-Int
4.361
0.733
0.446
0.756
0.490
0.711
0.464
13.3%
0.619
0.386
1.1%
BLIP2
4.378
0.706
0.408
0.729
0.456
0.707
0.430
6.9%
0.655
0.377
0.3%
PEZ
6.188
0.709
0.384
0.722
0.418
0.676
0.389
16.7%
0.618
0.344
1.1%
PRISM (Ours)
3.466
0.743
0.464
0.770
0.499
0.734
0.482
6.9%
0.734
0.464
0.1%
all the other methods, use the class noun to fill the template and the output prompts that describe these concepts serve suffixes.
We also qualitatively demonstrate the ability to represent a certain artistic style using Wikiart dataset Tan
et al. 2019). We use three images from each artist as reference images. To create diverse scenes, we
follow He et al. (2024) and use descriptive prompts from PartiPrompts Yuu et al. 2022 as prefixes to the output prompts similar to the previous setting.
DreamBooth Dataset Results Table I and Figure 3 respectively show the quantitative and qualitative results on the DreamBooth dataset. As we can observe, PRISM achieves the best performance across the board except. for the image similarity metrics for SDXT-Turbo.
In terms of object fidelity, we find PRISM to constantly achieve accurate depiction of the target subject while the baselines sometimes struggle to capture all fine-grained details like the colors of the animals and the shape of the shoe sole. And out of the four training-frt methods we experiment with, PRISM is the only one that can attempt to tackle complicated objects such as the red monster toy and the dog-shaped backpack when all the other methods fail to generate even remotely similar objects. Due to the nat ure of their methodologies, BLIP-2 and CLIP-I Interrooooo also capture the background and other irrelevant elements in the scone when describing the objects. However, unlike our mothod, where we can directly specify the tasks and the judging criteria in the system prompts of the VLMs, there no simple way automatically filter out those irrelevant elements in BLIP-2 and CI.II-Interrogatoo''s outputs. Even though Textual Inversion

---

Published in Transactions on Machine Learning Research (08/2025)
Reference
Context
Inte-noterrogator
BLIP2
PEZ
PRISM (Ours)
paeking wits anlioou dediofiaa Mas. backrnaak dasled repar tepikork, bedtese, liee....
madiaze otainaial basi generully
beprowienti stek, light brad strekos
painting of river with boats and trees
SD2.
coffee
colir pulatl 19b-
mug
whose gillab recomslfturr
cattrry Prenic ars tranpel
SDXL -urbo
Siberian husky playing the piano
pakiing on Mik backgmend shouls crening renpi wnth 1127 goady exse balcril...
painting of fruits on a blue background
mnkerraaaioonn frodenccarii niko
Elaccir ey iuproed tigh color eastas, disrggad for resurl colors expossivee sieytifel
matisse alexcttziral uitinscfarpeeuus
canagee soviet
Dull-E2
a small kitchen with white gout in
ponwe apecingala proput people picaus paaniag dark contral woudde
person's painting of group of people playing music
datlaz
Cobeis style bell coler conmant.. Exgauited prrpectos cuty Misccrtuyy tumpoan anarig
mece menil withers dyears
pileteaet dwiil uke dovie lacarallt Abrdiaa pollin light your detak,
ns lnsteremiennd ttilks
charocan Spaking becuns sech wicker latti, High cieernll shalows
Dall-E 3
a painting of girl sewing with needle
A bowl of Beef Pho
namoa viscmmkee peeeennghiinegh
lis puintia
Midjoumey
sunken ship becomes the homeland of fish
wilh
painting of field with grass and trees
oid defeadd Japen eetting
esabiis smelinad Suurenteed
cectaro,, 17E0 and enlvecaly unc. maverii
cpptomes cxepleminicy
Figure 4: Qualitative results for personalized style T21 generation Wikiart dataset.
obtains marginally higher CLIP-I scores and DINO scores with SDXI-Turbo, it requires a lot more modeling assumptions than our method, and the new embeddings it learns not transferable not even to SD 2.1.
PRISM is the only method in our experiments that can produce fully human-readable prompts while pro- viding enough relevant. details. In particular, we can observe that PEZ renders completely indecipherable texts, BLIP-2 only describes the general scene but fails to mention any visual details and textual inversion is combines the results
entirely not interpretable since produces soft embeddings. Since CLIP-Interrogen from BLIP-2 and CLIP search, improves the interpretability over PEZ-like gradient search-only method. However, it still falls short. in terms of human readability in comparison to method.
When transferring the output prompts to black-box T21 models, our method shows even larger advantages over the baselines. We also observe that our method produces the fewest unsafe prompts judged by Dall-E safeguards, while the baselines fail to pass the safeguard up to 16.7% of the time.
Wikiart Results In Figure 4 we show a qualitative comparison between our method and basclines on the Wikiart dataset. We find that our method is capable of precisely identifying the genres, eras, and sometimes even the names of the artists when describing the style of the reference artworks. On the other hand, the baselines fail to recognize these crucial keywords, even when they have access to a pro-collected bank of words that is supposed to provide accurate descriptions of art styles. In addition, PRISM can provide other fine-grained details such as pen strokes style and color palettes in human-interpretlalle way to better assist the generation of the target style.
4.3 Direct Image Inversion
To demonstrate the versatility of our method, we also compare PRISM the baselines in the task of direct image inversion. In this task, the goal is to directly find the prompt that can exactly generate the input image. Here the number of reference images is M = 1 and we aim to capture all aspects of the image, including the subjects, background, theme, style, and other details in the scene.
Datasets
We use images from the DiffusionDB dataset (Wang et al. 2022) for the direct image inversion task. This dataset includes wide variety of image pairs generated by Stable Diffusion and we choose 10k split Huggingface.
random sample of 100 images from the
8

---

Published in Transactions on Machine Learning Research (08/2025)
Reference
Cl.I--Interrogator
BI.IP2
PEZ
PRISM (Ours)
seetaaii aod arco playig clance instuaeet, joi hankd, camo hikling manree, modazn clueepp
yp chinese devepors culuraleccumpicc moon woree niulele puitaria gels bucns the
Blattaiion suuling Joker wounaa with short lair paysag tnaittonal strigee instrnment slocviion rec med-omage dross with flwser
a woman in dress playing chinese
SD
SDXL SDXI-Turbe Turbo
laying in sleminaan, gardenina, muntrate, vthei in 794, Hi hong iniei radar contrd,
robot laying in the grass with green griss
yp chinese diaspora coltralaacomplich mens illustation womeen olf uiulele guitaritt bucua
tmage wit
argular centerdd groee burne on tiro unalerciresbe grorn ryes, and light buely
busai loze take, iplans waljsper. kalil komutter, tockgmend
waskeavis bunasi suntisaale bahbypetensson users kif workplace finsart purtzali macro
Cizate ianas of meticaloadly pe
Dall-E
lusel lose will thick, tixuing creppy
bonsai table
Alcbuner Lernoner. bessiffl phoce
centored
wooden salle
st deftrent diffecett colors. anda ins, slalds, mazz kerzm utde, che iimme, caroe. ac Japienn
rodritalone waeehi divensity dapper tusedo vensiun android twehve suticnszmende
Lieati and
set of different animal different colors
Dall-E7 E
gog
each sqarr the shind niee which
Gaif Pauf
laca gray mihrn floe. Eads aquare
Volala
Midjourney
anybones dowe colvett lecigrional iskroll. 6x quar. pertaaeas checkzned flocr.
cotpmnaiiiies karmpiiee eiceforcil welsavilyy give aqaaca setindaggere
Enanca oyse po mulr dlanater pennterwllliidee bar sungheess enie The huckgrondd chackered...
sunglusses is shown on a colorful background
uniztkkpeizingg compaabble
Figure 5: Image inversion results for different methods on different T2I models.
Table 2: Metrics for the image inversion results. old fonts indicate the best score and underlines indicate the second best score.
Prompt
SD 2.1
SDXL TUrbo
Dall-E 2
Dall-E 3
Method
NLL
CLIP-I
CLIP-I
CLIP-I
Failed 4
CLIP-I1
Failed
CLIP-Int
4.193
0.800
0.783
0.761
17.0%
0.719
0.0%
BLIP2
4.299
0.710
0.707
0.687
2.0%
0.695
0.0%
PEZ
6.736
0.746
0.726
0.616
3.0%
0.635
0.0%
Liu et al. (2024)
2.520
0.713
0.720
0.689
0.0%
0.732
0.0%
PRISM (Ours)
2.762
0.749
0.776
0.741
2.0%
0.767
0.0%
Results As shown in Table 2 we immediately see a significant improvement in the human-interpretabiiie of inverted prompts using PRISM. While expected for methods, such as PEZ, which has no language prior, we also find that our method finds text that more closely aligns with a learned distribution of English language text (i.e. lower NL..) than CIIP--Ineerrogaor and BLIP2.
When comparing the image quality, we first note that because all images in DiffusionDB generated by Stable Diffusion, which is exactly the model design space of CLIP-Interrogien and PEZ, it gives significant modeling assumption advantages to these baselines over our method when testing on Stable Diffusion models. This advantage enables relatively high performance for these basclincs on Stable Diffusion models, but it does not transfer well into other closed-sourced models. In fact, we can even observe that CLIP-Interrogatorr generates the highest quality images with SD 2.1, which is the weakest model in this comparison, and the lowest quality images with Dall-E 3, which the strongest T2I model this table. This phenomenon indicates that the design choices of Interoate and PEZ are heavily overfitted on Stable Diffusion, and have poor generalizability to other models. On the other hand, the prompts produced by PRISM generalize
9

---

Published in Transactions on Machine Learning Research (08/2025)
Reference
GPT-4V
PRISM (Ours)
Table 3: Comparison with GPT-4V in both per- sonalized T2I generation and direct image inver- sion experiments.
Image
Object
Method
NLL
CLIP-T
NLL.
CL.IP-T
GPT-4V
2.356
0.756
3.393
0.757
PRISM
2.762
0.776
3.466
0.770
Figure 6: Qualitative comparison with GPT-4V.
-2.74
0.776
2.45
0.773
Prompt Log Likellhood
-275
0.775
Prompt Log Likellhood
-2.50-
0.772
Score
-2.76
0.774
-2.55
0.771
-2.77
Image CLIP-1
0.770
0.773
-2.60
-2778
0.769
00.722
2.65
Image
-2.79
0.768
-2.80
0.771
-2.70
0.767
-2.81
0.770
-2.75
0.766
6
i
3
5
Number of Streams
Number of Iterations
Figure 7: Ablation study on different numbers of streams N with fixed 5
Figure 8: Ablation study on different numbers of iterations K with fixed N= =3 3.
significantly better than the baselines and we achieve better results with more powerful T21 models. When compared with the other VLM baseline Liu et al. (2024), with which a thorough comparison is included in Section F PRISM performs it in almost all metrics, indicating a superior algorithmic design.
Qualitatively, our method also provides prompts that are both semantically aligned with and generate images that are visually similar to the reference. In particular, Figure 3 shows that we can find text that aligns with the image, even when those images have particularly unique features. For example, in Figure 5 Dall-E3 generated a grid of images of animal faces. Not only does the PRISM's prompt explicitly include a request for this grid structure, unlike our comparison methods, but also takes into account the coloration of the background that captures the small flowers in the grass, showcasing the capability of identifying and reflecting small fino-grained details from the reference.
the reference. In the second row of Figure our method is also the only method
4.4 Ablation Study
Comparison with GPT-4V While is nonetheless useful to understand what benefits PRISM adds to an already capable foundation model like GPT-4V. Therefore, we compare our method with GPT-4V's zero-shot performance with the same system prompts for both tasks on SDXL-Turbo. We can see in Table 3 that PRISM consistently outperforms GPT-4V's zero-shot performance, although the latter is already compelling. In Figure i we can also observe that qualitatively GPT-4V can capture the high-level semantics of the reference images but still misses fine-grained details.
any VLM (which we demonstrate in Appendix C.2 it
10

---

Published in Transactions on Machine Learning Research (08/2025)
Reference
Reconstruction
PRISM Generated Prompt
Generate hyper-realistic portrail of an Asian female with bright, vivid neon orange chin-length bob haircut, straight bangs above eyebrows. She looks to her left with strong, determined expression, wearing saturated green turtlencck outfit with bright yellow accents and distinct circular logo on the chest. Her left fist is prominently clenched in front of her, showing motivation. The background is deep purple with bright, well- defined diagonal white lens flares, high contrast lighting.
... distinct circular logo that says "PRISM"
Her left hand waving at the camera.
The background NYC time square many night.
blue
hoodie
Figure 9: Prompt editing demonstration with Midjourney.
Effect of Budgets Next we take a closer look at the effect of increasing the budget in PRISM. Figure 7 and show the effect of increasing the number of streams and the number of iterations K respectively. We observe that when incrcasing N and keeping K fixed, we can obtaín steady performance improvements both human readability and prompt accuracy. When increasing K and keeping N fixed, although we do not observe monotonic relationship between the performance and K, we still notice general upward trend in prompt accuracy. In Appendix D we discuss the trade-off between N and K to better inform practitioners how to choose these hyperparameters. Besides adjusting the budget, one can also use cheaper open-sourced VLMs to lower the cost. lypical PRISM run with GPT-4V and x = 30 budget costs around $1.5 USD and 11 minutes (as May 2024). However, in Appendix achieve high performance with GPT-40-mini (< 2% of the price of GPT-4V) and IDEFICS2 {Laurençon) et al. 2024) (free of charge), showing the cost-flexibility of our method. In Appendix E. we provide detailed comparison regarding cost and latency among baselines and PRISM with different VLMss.
C.2 we demonstrate that PRISM can still
4.5 Additional Applications
Prompt Editing Because the prompts produced by PRISM is very human-interpretablle after obtaining a prompt from the reference images, one can easily modify the output prompts to change attributes in their desired generated images. Figure demoosstetssss an examples of prompt editing with PRISM on Midjourney. With simple and intuitive prompt edits, we are able change specific attributes of the images while keeping the other components in the scene relatively unchanged.
Multi-Concept Generation PRISM is particularly well-suited for multi-concept generation due to the human readability of its generated prompts. This feature allows for easy identification and composition of different components within scene, enabling intuitive control over multi-concept results. Unlike PEZ, which does not provide explicit control over which part of the prompt corresponds to specific aspects of the image, PRISM allows for much clearer and more direct manipulation, which demonstrate in Figure 10
Prompt Distillation Unlike PEZ, which requires an ionalional optimization process to generate distilled prompts, PRISM leverages its highly interpretable prompts and VLMs to simplify prompts effectively. By directly instructing a VLM (in this case GPT-4o) with the prompt " "Here prompt to this image with text- to-image model, make it more concise (less than <length> tokens) but keep all the descriptive details". achieve concise, distilled prompts without additional computational overhead, shown in Figure 11
11

---

Published in Transactions on Machine Learning Research (08/2025)
ortaee tably cat with penstre expeeson, titting ontioors with. liste tlttrrig thaoush leaves, dusiact bi staped mariing on forehoad, white anond the mortt azd eyes, stset fur
adalt Russian fhort in light slatn gry, virid eyus looking Eorword strught pointed composed sitting pose
fluity Pembroke Corgi dog with a whith and golden- brown neat, upright cors, smiliin expresion, visiblle trngue, sitting postiio
Nutfy Chow dog golden-bron brown fur and blue tongue
PRISM Prompt
Concept References
Multi-concept Generation
Multi-concept Prompt
A dog and a cat on a beach. Concept PRISM Prompt < Concept PRISM Prompt >
Figure 10: Qualitative demonstration of multi-concept generation with PRISM.
Referunce
PRISM Generated Prumpt
Reconstraction
Distille PRISM Prompt
Distilled Recon.
Caserite iepos-ddiste portnit of as sias female with bright, visid noon crange chin lingt bab harout, stright bangs above eyshrows. She looks to her act with stoog coteminns expesssion, wearing satuitt trtleneck cutfe with yellow sooess and distrcct dirslar dirsalar loge on the chost, Hor left fix primmertly clenched frent her, shooiing motivation The bickymand isa daig purple with bright, wall- defined degural white lems flane. ligh contrnet lighting.
Hypes-maliss maliste portril an Asizz
woman with neon orange bod zairut, turtereck with yullow accents. clerchel fist, perple background, white lens flones.
child's backpaakk shaped like a dog's head with floppy ears, two round black tyrs, brige friendly smile with pink tongue, adged with blue piptnng, light gray body
Dog shrped chill'ss backpock, floppy ears, nose, plikk longue, pipling, gay body.
brown
Figure 11: Qualitative demonstration of prompt distillation with PRISM.
5 Conclusion
In this paper, propose PRISM, algorithm that automatically creates human-inteerpeetbble and accurate text prompts for text-to-image generative models, based on visual concepts provided by reference images. Our method iteratively refines the sampling distribution of the text prompt via VLM in-context learning and is capable of creating prompts that are transferable to any T2I models, including black-bor platforms like Dall-E and Midjourney. We hope our work also encourages researchers, particularly those non-LLM fields, consider how the advancements in LLMs can offer simple yet effective solutions to problems that pre-LLM methods have struggled to address.
Broader Impact Statement
Just as LLMs are susceptible to being jailbroken or adversarially manipulated by malicious actors Zou et al. 2023), our method may also be vulnerable to malicious intent, potential bias, limitations in the base models. Therefore, we will implement necessary safeguards upon the public release of our code and are committed to keep up with future advancements in improving the safety of our method.
12

---

Published in Transactions on Machine Learning Research (08/2025)
Acknowledgements
This work supported in part of the ONR grant N000142312365 We also thank Sony AI for sponsoring part of this work. A.R., H.H., and G.P. are supported by the NSF Institule for CORE Emerging Methods in Data Science (EnCORE). A.R. is also supported by an ASSET Amazon AWS Trustworthy AT Fellowship. Y.J. is supported by Google PhD Fellowship. We thank Murtaza Dalal, Martin Ma, Lawrence Jiang, Aakash Lahoti, Aviv Bick, Issac Liao, Sukjun Hwang, Kevin Li and Ricardo Buitrago for their feedback on this paper.
References
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. 33:1877-1901, 2020.
Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. Jail- breaking black box large language models in twenty queries. arXiv preprint or arXiv:2310.08/11 2023.
Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming- Wei Chang, and William W Cohen. Subject-driven text-to-image generation via apprenticeship learning. 96:30286-303488 2023.
Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Mare Louis Klasky, and Jong Chul Ye. Diffusion postcrior sampling for general noisy inverse problems. 2023. URL http::///ppparrreee.eottttfrrrrr
OnD9zGAGTO1
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or, An image is worth one word: Personalizing text-to-image generation using textual inversion. 2023. URL
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joinl Conference on Natural Language Processing, ACL-IJCNLP 2021, pp. 3816 3830. Association for Computational Linguistics (ACL), 2021.
Yaru Hao, Zewen Chi, Li Dong, and Furu Wei. Optimizing prompts for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Yutong He, Ruslan Salakhutdinov, and J. Zico Kolter. Localized texl-to-image generation for free via cross attention control. arXiv preprint arXiv:2306.14638 14636, 2023.
Yutong He, Naoki Murata, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei-Hsiang Liao, Yuki Mitsufuji, Zico Kolter, Ruslan Salakhutdinov, and Stefano Ermon. Manifold preserving guided diffusion. 2024. URL bttp::///pparerreeee.e////rraaad
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. taage-to-image translation with conditional adversarial networks. 2017.
Albert Q, Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision- language models?, 2024.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. JMg.. 2023.
13

---

Published in Transactions on Machine Learning Research (08/2025)
Paul Pu Liang Yun Cheng, Xiang Fan, Chun Kai Ling, Suzamne Nie, Richard J. Chen, Zihao Deng, Nicholas Allen, Randy Auerbach, Faisal Mahmood, Ruslan Salakhutdinov, and Louis-Philippe Morency. Quan- tifying & modeling multimodal interactions: An information decomposition framework. 2023. URL
https://openrevieg net pet//forum2iddJggiijjpp
Shihong Liu, Samuel Yu, Zhiqiu Lin, Deepak Pathak, and Deva Ramanan. Language models as black-box optimizers for vision-language models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 12687-12697, 2024.
Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.0442 2023.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 11//////112222AALLLL00
8086 8098. Association for Computational Linguistics, 2022. doi: URL https://dol org 10. 18653/11/222222.... htt.:/22222.cl11oogg. aww.allllll..5... 556
Shweta Mahajan, Tanzila Rahman, Kwang Moo Yi, and Leonid Sigal. Prompting hard or hardly prompting: Prompt inversion for text-to-image diffusion models. arXiv preprint arXiv:2312.1244 12416, 2023.
Oscar Mañas, Pietro Astolfi, Melissa Hall, Candace Ross, Jack Urbanek, Adina Williams, Aishwarya Agrawal, Adriana Romero-Soriano, and Michal Drozdzal. Improving text-to-image consistency via au- tomatic prompt optimization. Xiviv preprint urXie:2403. 10000.1.780. 17804, 2024.
Hariharan Manikandan, Yiding Jiang, and J Zico Kolter. Language models are weak learners. volume 36, pp. 50907-50931, 2023. URL https:://peerevviii mat.:/oormmmmm//d5550
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. 2022.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint 4r11:.4411.1884 2014.
OpenAL. Gpt-4 technical report, 2023.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Woj- ciech Galuba, Russell Howes, Po-Yao Huang, Shang-Waae Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bo- janowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learn- ing Research, 2024. ISSN 2835-8856. URL https:///penreevie wttp://wwwwweeeeeeeeouoo/eieeeee nett:/Eorrumiio--assStttEr
Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with' gradient descent" and beam search. arXiv preprint ar arXin:2305.03400 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishknnn Jack Clark, et al. Learning transferable visual models from natural language supervision. pp. 8748-8763. PMLR, 2021.
Alexander Robey, Eric Wong, Hamed Hassani, and George J Pappas. Smoothllm: Defending large language models against jailbreaking attacks. arXio preprint Xkio:2310.03665 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. pp. 10684 10695, 2022.
Litu Rout, Negin Raoof, Giannis Daras, Constantine Caramanis, Alexandros G Dimakis, and Sanjay Shakkottai. Solving linear inverse problems provably via posterior sampling with latent diffusion models. arXiv preprint ar X0x:222007.0011, 2023.
14

---

Published in Transactions on Machine Learning Research (08/2025)
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yacl Pritch, Michael Rubinstein, and Kfrr Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. pp. 22500-22022 22510, 2023.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation, 2023.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kun- durthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models, 2022.
Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image generation without test-time finetuning, 2023.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 4222-4235. Association for Computational Linguistics, 2020. doi: 10.18653/V1/20000EMMB WMN.E-MAINNNN..... https://doi. org/10..// 10. 18653/v1/202// emnlp-maine 346
URL
Jiaming Song, Arash Vahdat, Morteza. Mardani, and Jan Kautz. Pseudoinverse-guided diffusion models for inverse problems. 2022.
Nitish Srivastava and Russ R Salakhutdinov. Multimodal learning with deep boltzmann machines. In F. Percira, C.J. Burges, L. Bottou, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012. URL htttps://proceddii weraangg...euriig ceuurpp..ccpappe..
pdf
Wei Ren Tan, Chee Seng Chan, Hernan Aguirre, and Kiyoshi Tanaka. Improved artgan for conditional synthesis of natural image and artwork. IEEE Transactions on Image Processing, 2811):39,-409, 2019.
doi: 10.1109/TTIB TTT....1...86666 URL http::::///di..o/11..11111.1 11t./.......11.P... 2018.28666660
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. Tn Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9238-5248, 2022.
Zijie J. Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. DiffusionDB: A large-scale prompt gallery dataset for text-to-image generative models. arziv preprint arXiv:2210 14896, 2022.
Albert Webson and Ellie Pavlick. Do prompt-based models really understand the meaning of their prompts? In Marine Carpuat, Marie- Catherine de Marneffe, and Iván Vladimir Meza Ruíz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Sealile, WA, Uniled States, July 10-15, 2022, pp. 2300- 2344. Association for Computational Linguistics, 2022. doi: 10. 10.18653/ 10.18653/V1/2022.2222 N1.2222.NNAALLLMII.... URL https://doi. 11g/10.116633/1/00000 naacl -main. 167
Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbrokon: How docs Ilm safety training fail?
Advances in Neural Information Processing Systems, 36, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural in information processing systems, 35:24824 24837, 2022.
Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient- discrete optimization for prompt tuning and discovery. volume 36, pp. 51008 51025, 2023. URL
15

---

Yongchao Zhou, Andrei loan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy net/ Published in Transactions on Machine Learning Research (08/2025)
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. 2024. URL htttps://ppenneevev..eee. htt.://www....rrrrooaaab-DDOO...
Zhengyuan Yang, Jianfeng Wang, Linjie Li, Kevin Lin, Chung-Chine Lin, Zicheng Liu, and Lijuan Wang. Idea2ing: Tterative self-refinment with gpt-4v (ision) for automatic image design and generation. arXiv preprint arXit:2310.08441 2023.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Adapter:r: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arrie:2308. 06721, 2023.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hulchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL Featured Certification.
Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. FreeDoM: Training- freee energy- guided conditional diffusion model. arXiv:2303.0088 09833, 2023.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision- language models behave like bags-of-words, and what to do about it? arXiv preprint arXiv:2210.0100 01009996, 2022.
Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. 2023.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9)22337,2348, 2022.
Ba. Large language models are human-level prompt engineers. 2023. URL
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-inage translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-223,, 2017.
Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. Xrxiv preprint arXic:2307. 15043, 2023.
16

---

Published in Transactions on Machine Learning Research (08/2025)
A Additional Experiment Details
In this section, we provide further details about the implementation of our experiments. For all quantitative analysis that uses Stable Diffusion based model, we generate four images for each combination of prefixes and prompts. For all experiments with Dall-E based model, we generate one image per combination. In the DoothBooth dataset experiment, we also replace the class noun for "stuffed animal" with "toy" to obtain lair comparisons with textual inversion, which can only take a single token as the initialization loken. We use OpenCLIP-ViT-H-I trained on LAION2B (Schuhmann et al. 2022 for both CLIP-Int and PEZ and use Blip2-Flan-T5-XT for both CLIP-Int and BLIP-2.
During PRISM iterations, we allow a maximum of 5 generation attempls for each stream and each ileration in case of potential run time errors related to blx API calls. We set the maximum number of tokens generated by the prompt engincer assistant at cach iteration to be 500. This contains both the improvement and the new prompt for the target concept. We encourage the assistant to generate shorter prompts using system prompts (details in the next section) and at test time, when the testing T2T model has a shorter prompt length than the prompt generated, we clip the generated prompt to the maximum length of the respective T2I model.
To simplify the implementation, only keep a chat history length of 3 and use the length of the prompt approximation of the log-likelihood for the final prompt selection. When evaluating the judge scores D(z, 2) in PRISM iterations, we shuffle the reference images when M > 1. The judge score is rescaled into range from 0 to 10. For direct image inversion, we re-evaluate the top 5 candidates twice and tally the scores with in-iteration scores to make the final decision. For personalized T2I generation, we re-evaluate once for each reference image and use the average score to select the output.
B Designing System Prompts
System prompting the standard way to condition a general purpose LLM for specific tasks of roquest. The key idea is that, before the conversation starts, the LLM receives tailored message, the system prompt, that provides the contexts, conversation scenario settings, formats and other guidelines the prefix of the entire conversation ahcad. In this section, we claborate on the design of the system prompts for the prompt engineer assistant F and the judge D. We also provide the full system prompts used in all of our experiments
at the end of this paper in Section G
B.1 Prompt Engineer Assistant
To design the system prompts for the prompt engineer assistant F, we follow Chao et al. 2023 and include the following components in the system prompt of F.
Setting We first set up the scenarios and assign a role for the LLM to perform better on the specific task of choice. The setting paragraphs start with "You are a helpful prompt engineer assistant. You are free to generate sentences that do NOT follow English grammar. You must obey all of the following instructions." " and continue with the specific description of the task and the objective. We also inform the assistant that it is expected to iterate and refine the prompts it generates throughout the conversation.
Format We then provide the guidelines for formatting the inputs and the outputs of the assistant. We describe what are expected in the inputs at each iteration and the content required in the outputs. We also provide descriptions of the meanings of each input and output components. More specifically, we inform the assistant that the inputs consist of three parts: generated image, reference images, and a visual similarity score, and that the assistant is expected to generate both the improvement to refine the previous prompt and the next new prompt. All generated text is formatted in JSON.
Examples Finally, we provide some examples of the potential formatted inputs and outputs that the assistants may receive and produce. We also provide examples of potential improvements for the assistant.
17

---

Published in Transactions on Machine Learning Research (08/2025)
Table Additional experimental results with different VLMs as on DreamBooth dataset.
Prompt NLL4
SDXL Turbo
Dall-E 2
Method
CLIP-It
DINOT
CLIP-I1
DINOT
Failed
TI (SDXL)
0.771
0.504
CLIP-Int
4.361
0.756
0.490
0.711
0.464
13.3%
BLIP2
4.378
0.729
0.456
0.707
0.430
6.9%
PEZ
6.188
0.722
0.418
0.676
0.389
16.7%
PRISM (IDEFICS2)
3.047
0.739
0.468
0.721
0.453
6.7%
PRISM (GPT-4o-mini)
3.498
0.768
0.493
0.730
0.475
6.7%
PRISM (GPT-4V)
3.466
0.770
0.499
0.734
0.482
6.9%
Optionally, we can also provide examples of prompts that can succossfully generate the target concepts in these paragraphs.
B.2 Judge D
We follow the same strategy to design system prompts for the judge D. More specifically, we set up the scene for the judge by stating "Please act as an importial judge and evaluate ..." in the system prompts and describe the visual similarity criteria based on the desired features for different tasks. We then provide the instructions on the formatting and give an example of the expected output.
C Additional Results
In this section, we provide additional experimental results and further baselines comparisons with our method. We also showcase the flexibility of the PRISM framework by demonstrating the effectiveness of different T2I model G and different judge D in PRISM.
C.1 Additional Qualitative Results
16 we provide additional qualitative showcases for subject-driven personalized T2T generation, sty de-driven personalized T2I generation, direct image inversion and prompt editing. We also provide an example of the iteration and refinement process conversation between all three components in PRISM in Figure 18
In Figure 12
13 14
15 and
C.2 Flexible Model Choices
To further demonstrate the effectiveness and flexibility of PRISM, conduct further experiments with diverse model choices for prompt engineer assistant F, T2T generator G and judge D. In Table 4 we first provide additional validation of the flexibility for choosing different VLM as base models for F and D using (1) a significantly smaller open-source model IDFFICS2 (Laurençon et al. 2024h, with IDEFICS2-Sb-chatee as F and IDFFICS2-8 as and (2) a significantly smaller and cheaper closed-source model GPT-4o-mini as both F and D. While there some expected performance drop compared GPT-4V, PRISM still delivers very competitive results and notably maintains human-readabilii and generalizability, particularly with the closed-sourced model Dall-E 2. This aligns with our previous conclusion and underscores PRISM's adaptability across various computational environments.
We also experiment a different T2I Generator G to showcase the transferability of the prompts generated by PRISM. Figure [7 shows qualitative examples of PRISM prompts with Dall-E 2 as the Generator for personalizod T2I generation and the images generated from those prompts using SDXL-Turbo, Dall-E 3 and Midjourncy. Our method capable of producing human- comaan-nterpreeablle and accurate prompts for both subject-driven T2I personalization and style-driven T2T personalization with this Generator G.
18

---

Published in Transactions on Machine Learning Research (08/2025)
Reference & Context
Textual Inversion
CL CL-P-Intemmon Interrogator
BI .IP
PF7
PRISM (Ours)
dachshund dog grey
sentament risk dachshund
Golden long-haired miniature dachshund with floppy cars and soutful eyes
dog in the snow
couch, bediead, 0
dachshund dog on a grey couch
Not Applicable
photo,
colswsuggests apertment renevation Gjosica illly reliable metcoullgist
SD 211
new castle, copper and emerald. taken with
a leapot on of the sidewalk in crowded street
leapot cup lea on a table, inspired by Gong Xian, rich in listing imsag, covironmental
a teapot with cup of tea table
misty mocelight meditnic
sarrounds honoesble
brown clay tespet with flat base, small spout,
shaped
Not Applicable
anttent mzhogunt known
straw jiuqing
simple
lacs
looped handle sitting...
10
yellow ziam cleck on white bod. 1840572578, packshort yous old inppired by Peter Alexander Hay, mading: time, scboots, atmee ardliah.....
a yellow alarm clock on a white bed
plagaed sleap represeets coregiver auditinn yellow gall consistent eclectic
vintage
SDXL- SDXL-Turbe Turbo
wet clock
Not Applicable
clock with two bells and handle white face
numbers
hunds, set againss
pair betge inspized ivy' foost mane.
pair of beige
llinge style
cream colured mid mid--calf
with
purple boot
Not Applicable
fringe boots on the street
Libaddest handicus
smoooh from
sight
pointed tox, and a seall heel
appar
Figure 12: Qualitative examples of the subject-driven T2T personalization task tested open sourced T2T models.
C.3 Discussions on prompts rejected by T21 safeguards
Moreover, as previously demonstrated, while our method is generally safer than most baselines, there are still instances where the generated prompts fail to bypass the safeguards of the T2I model. For example, the prompt "single three-dimensional emoji-like object, stylized pile of poop, very smooth and uniform matte surface, solid light brown color without any pattern, cartoon eyes accurately sized, simple mouth with cheerful expression, isolated" is rejected by DALL-E 2's content filters. This failure is likely due to the presence of certain keywords (e.g, "poop") that trigger automated content moderation, regardless of the otherwise harmless intent of the prompt.
On the contrary, bascline methods like Cnte-Interrogator often generate prompts that include the names of specific artists or copyrighted entities. This issue arises from their use of pre-collected keyword bank that contains numerous proper nouns, including well-known artist names and intellect ual property (IP)
19

---

Published in Transactions on Machine Learning Research (08/2025)
Reference Context
Clle--oterrogttor
BL.PP
PEZ
PRISM (Ours)
Failed to Pass the Safe Guard
Dall-E2
a candle with a city in the background
wouden jar stllnng the soft white ruhher, photo degrees, candle, barycentrie subdivision, lush and green, top lid, just after rain,
a wooden jar sitting on the grass
internithlydrated mantessori yogurt container candle loire tearmsisd landseaping wood wailis torento genderequality
opaque cream-filled cylindrical glass jar with flat wooden lid
Dall-E
09
a poop toy is sitting the silicone skin, frowal, recyeled, di skcinoone, festivals, ripple. afficial producr haggy eves, small jaw, air shot,
single thrve-iimensioall emoji-lke object, styized pile of poop, very uniform matte surface, solid brown pottens, cartoon eyes ...
a toy on the beach
a poop toy sitting in the grass
emogii fidget to @AP@
T aachie ultimatefimuliintteiaa
any
Midjourney
golden retrieerr standing on the beach
joyful amber-eyed dog with
golden retriever standing the beach at sunset
sodium clive spanicl retriever represents atiilee gotcha almond spouse performed antiovidant !!!!!
deep reddich- booobbrww
dog wearing rainbow scarf
pomeranian mix, red head, in the winning pholo, breezy hackground, conturoy, highresohtton, tourist....
textured by movement medium-sised distinclly
Hoppy shiny cast mouth open happy expression.
Figure 13: Qualitative examples of the subject-driven T2I personalization task tested on closed sourced T2I models.
references like "Pokemon". Prompts generated in this way are not only more likely to be rejected by models such as DALL-E, which explicitly ban names of living artist and copyrighted IP, but also carry a higher risk of producing outputs that closely mimic copyrighted styles or identities. This substantially increases the potential for copyright infringement, making these methods less suitable for deployment in practical settings where safety and legal compliance critical.
20

---

Published in Transactions on Machine Learning Research (08/2025)
Reference
Context
CLIP-Interrogator
BLIP2
PEZ
PRISM (Ours)
SD 211
Christmas toy train
spiriing ioe centes keal fanou tnstakee, reathed verrin
painting of woman helping two girls
pelladepssedde
opane sadd
Imgressetil sole. out toaa emikes nanual Indain las canuarryyy Edger Degan
musdeffaram
cammane
SDXL. Turbo
un old
sanilaal hupprainng DELRE
Grili evaieerdk kaninlk dcalls
case creiting frou cronds andica commes cittng on arllann
city intersection
alearshhi Vadooersiiie
barcking encnodavtrati poliia 19isensay
slanding in front tent
chartng bigoad zdake fake
Wast trsettate cocur began dengrewen nicho
camtyy Notthem Rowinome
Dall-E2
chicken working out in gym
poople will hern.
a painting of'a group of people books
Graneeeidd godling cenos for live. hurving sklinia. Fonds
pirsing witl umuuied vere bocker,
exprovaes, saikeed evtly
Dall-E
lavender backpack with triceratops stuffed animal top
paitrig afto ire cartee lands Bhtas
a painting of two sitting table
baiilk-orakk pino zia songe evemaly hegas
Farly orevoraryy oyti thic welluil braal vamios maaee cattor pelcas doop blick penaml
Lunoea mos nostiaa
aepied by Gorgjo
presrseia
Midjouney
settecceeeeetti praning Iic, syck, nupeed Alleel
a watercolor painting of clothes drying on a line
abympiic aita dhopie was lagely boeve sorreoter reckiales
A picture of very tall stop sign
hight, tonokavet
watoralee gentl
undidf
Rargont
Figure 14: Qualitative examples of the style-driven T21 personalization task.
Reference
CLIP-Interrogate
BLIP2
PEZ
PRISM (Ours)
For pasdings with dileron eakooial dupec bequier sboteterim
ligherglala lantia conxaniaa corensire amros cannik saroesectrve
Gesazute zipto dloitat paintngo with expanddase
SD 2.1
paintings with different colors and shapes
Zand with
nall body. rhel veise
nathrvel gaaliti strcks
chaots while scriiila Back
SDXL Turbo
Srewing with
nerdy mue ut dictches smcr konu ales lampard atfleck univenal superman
Croace
skend male five clnck dledin bovnt intense maso Icoaaed ipsky sorrayled sprcods, dyshoos
ald amilane foos amos, mending on cedernntaciaa: appoermmmned
drawing of man with glasses
ursinilsmehsh
a-day alting an rock guita, tatans golass, iepered Geliniina Prandeeenne, clawy vasli, inepieed by Miae Moseeso
dog Sa Figute lusky colostial
Carooziia blocko: Luaky, munina aliing
Dall-E
a dag is sitting rock with guitar
violinist
skilele lotaclui coodfurrding scontihh testoe lustration
baldng srmple
zolsen hizlle Setgle gradion deep
kubrick eatepgychedelic classical vegesation sculptance psychedelic ilusttations shorttlim
Canerrae
Dall-E3 3
siting un sodu. besple and taan gloud. inviuse cumene pantinu. o'ld" Iemussi Anzrci Koloraiie....
fistao seand. criceaal hertanod
mossy creature is sitting rock
feed, sono, port
bork wilhomee sol.
Midjourney
with two eyess breppe and gloun orange skir. intricate, aleo grey cat don mumborr pater
Instance custody mitheld cat happnnt alustration vib unt pvcalloinn sua intricate sious sllwwingg
imause two cats, mericclly alinnd. Top eac seall we heed only directly gozing with kimnnus
cat with two eyes and rainbow background
Figure 15: Qualitative examples of the direct image inversion task.
21

---

Published in Transactions on Machine Learning Research (08/2025)
Reference
Reconstruction
PRISM Generated Prompt
child's backpack shaped like a dog's head with floppy cars, two round black eyes, a large beige snout with a brown nose, a big friendly smile with pink tongue, edged with blue piping, light grey body
handbag
bunny
pointy
closed mouth
pink
Figure 16: Qualitative examples of the prompt oditing task with Dall-E 3.
Reference
Context
PRISM w/ Dall-E 2
Reference
Context
PRISM w/ Dall-E 2
SDXL Turbo
cat in wizard outfit
orange tabby cat stripes, upright ears, and attentive gaze
A picture of very tall stop sign
style high cummes deralee liee work, sicm-lanchin.. mamochhomaticc
dramans lishting neltswar soone
white ceramic bowl filled with blueberries, grid pattern Bon Appétit text
Cublit style wi abstracted for mis multiic po petppctivss softoned color palette with less sotu soturation nuanoed inerplay of light and shaetne. carly 20th contury
Dall-E 3
a bowl the beach
a coffee mug
Midjoumey
a dog in the jungle
small white dog, wavy fur, looser curls, hright white color, dark eyes, pink nose, wearing collar
chicken working out in a gym
Inpressionst plem air sedniquu: liick impatto bradtwokk eeporee sicid calors, dificeed aiilight quality Piacaro infuencee late ocstury
Figure 17: Qualitative examples of the subject-driven T2I personalization task using Dall-E 2 the T2I Generator G.
22

---

Published in Transactions on Machine Learning Research (08/2025)
You are helpful pronpe engineer assistant. You are free to generate sentences that do MI follow English
System
PL9
Aurissnt
You are helpful promet enginear inages: You receive text-to- co-t--t--tt tmage generative
System
is generated by and the sacond ane is a real image.
PLease act an impartial. judge
Judge
[IMPROVEMENT]:
This the first iteration, will tzy my describe detafled as posstble.
image
PUB
Prongi Enginaer
[PROMPT]:
photo of one whire curly-hatin haaderrd smatl dog black collar and ID sitting patterned blanket
Beinnos
T21 Generator
[Score]:
Judge
[IMPROVEMENT]:
Received score of indicating the generated image is sinilar the obfactive generated dog't than the tazget, and its eyes are less proninent. It is essential to specify tur texture more clearly and emphasize eye features. Next prorpt will ll focus retining charseteristies.
PO9
The curly
appears
Purnpt Engenoce Amiitai
and
[PROMPT]:
close-up photo et the smatl white dog with distinct fur, expzessive eyes, visible black collax shiny ID tag
Refernzee
Oo
[Score]: :
Judge
Figure 18: An example of the iteration and refinement process a conversation between the three compo- nents of PRISM. Only the system prompts (labeled as "system") and the first two iterations are shown in this example.
23

---

Published in Transactions on Machine Learning Research (08/2025)
0.776
-2.55
0.25
Percentage of Final Prompts
Prompt Log Likelihood
-2.60
0.774
Image CLIP Score
-2.65
0.772
0.15
0.10
-2.70
0.770
0.05
-2.75
0.768
4
0.00
Number of Iterations
2
3 Iteration
5
Figure 19: Ablation study on the trade-off between N and K. All runs shown in this plot have the same budget N x K 30, but each run operates different. number of iterations K.
The distribution of the final selected prompts in each iteration for the image inversion ex- periment. Here N 6 and K =.
Figure 20:
D Additional Ablation Study
In this section, we provide more detailed ablation study on cach component of the PRISM framework. In particular, we retenstrate the trade-off between the number of streams and the number of iterations K, compare a non- VLM judge (a CLIP judge) against our choice of a LLM judge (GPT-4V Judge), and also the effect of the existence of the Judge and re-evaluation.
Trade-off between N and PRISM has two hyperparameters and K which control the amount of parallel search and the depth of iterative refinement. Figure [99 shows a trade-off between N and K with the same budget
K 30. Similar the findings of Chao al. (2023), we find that performance degrade if the refinement is repeated too many times (i.c., K is large), and in general, recommend practitioners with small budgets to go beyond K = 5. Unlike jailbreaking (Chao et al. 2023 we observe that the optimal N and K can vary depending on the task: if the target concept is simple (e.g. a commonly seen dog), then small and K are generally sufficient, and prioritizing N tends to be more helpful. However, if the target concept rarer and more complicated (e.g. a very specific toy), a larger reasoning depth (i.e., larger K) would be more helpful. In Figure 20 we show the distribution of ileration numbers which the best prompt is found in the image inversion experiment. In practice, one may tune these hyperparameters further for specific use cases.
do not
Comparison between CLIP Judge and VLM Judge Finally, we demonstrate the importance of using a VLM the Judge. When assessing image similarity, it is natural to default to existing metrics that do not involve LLM's such CLIP similarity. However, we have mentioned in the main text, these metrics do not perform well outside of their trained notion of similarities and therefore is not very generalizable to custom tasks from users. Figure 21 demonstrates the qualitative difference between PRISM with a CLIP judge versus PRISM with a GPT-4V judge. We can observe that subject-driven T2I personalization, CLIP judged PRISM often include irrelevant elements such as the environment. (e.g. "on green grass") and omits important details such as the color and the other distinctive features whereas GPT-4V judged PRISM can adhere better to object oriented details and ignores other unrelated factors. In style-driven T2I personalization, CLIP judged PRISM fails to capture the artistic styles and mainly focus on the general contents of the reference image. On the contrary, GPT-4V judged PRISM produces much more prociso and locused prompls for the reference styles. The drawbacks of using CLIP-based Judge can polentially attribute to its incapability of identifying fine grained details and distinctions in different contexts, as studied in (Thrush et al. 2022: Yuksckgonul al. 2022). Using an autorogressively trained VLM such as GPT-4V can mitigate this issue. However, these models are not perfect either. As future works, we can potentially introduce more rule-based reasoning in the iterative process similar to (aañas al. 2024).
24

---

Published in Transactions on Machine Learning Research (08/2025)
Table Ablation study on the effect of the existence the Judge D, re-evaluation, the budget, and different choices of N and K. All methods use EDXL-Turbe as the T2I Generator and also are tested with SDXL- Turbo on the direct image inversion task.
Method
N
K
Prompt NLL +
CLIP-I
GPT-4V GPT-4V + Judge
1
2.356
0.756
30
2.349
0.769
GPT-4V + Judge
6
5
2.615
0.771
GPT-4V + Judge + Re-evaluaaion (PRISM)
30
2.456
0.771
GPT-4V + Judge + Ro-ovaluation (PRISM)
6
2.739
0.776
Reference
CLIP Judge
GPT-4V Judge
single three- dimensional emoji-like object, stylized pile of poop, very smooth uniform matte surface, solid light brown color without any pattem, cartoon eyes
SDXL-Ture Turbo
cartoonish poop emoji toy with a smiling face and big white eyes sitting on grass
round white ceramic jar with cork lid and clear knob handle on top sitting on green grass
Dall-E 3
opaque cream- filled cylindrical glass jar with flat wooden lid
Modigliani-inspae clongation features, muted tones, distinct stylization of faces with almond eyes, small pursed mouths, and swan-like wian-like neeks, carly 20th-century modemnist, European art scene
Expressionist style, muted earthy color palette, broad brushstrokes, simplified forms, somber mood, early 20th century European art, hint of cubist influence
EDXL-Turbo
Northern Renaissance, 1500s, enhanced golden earthy tones, luminous layering. detailed fine brushwork, deep color with chiaroscuro, religious or historical context, aged oak panel texture, balanced harmonious composition
TOO
Early 16th century Northern Renuissance solemnity, aged painting crackled vamish texture, deep introspective expressions, subdued earthy tones with touches of rich color, detailed texture in religious garments, diffuse, soft lighling
Midjourney
Figure 21: Qualitative comparison between using the CLIP model as the Judge D PRISM and using
GPT-4V as the Judge.
Effect of the Judge D and Re-Evaluation We first compare the performance zero-shot GPT-4V, GPT-4V parallel search with budget and the Judge seloct the best resulting prompts, PRISM without ro-evaluation, and two different PRISM settings with the same budget of 30. Table Table the quantitative comparison among all settings using SDXL-TTurbo both the T2I Generator G and the testing T2T model
25

---

Published in Transactions on Machine Learning Research (08/2025)
Reference
N=30 K=1
N=15 K=2
N=10 K=3
N K=5
N=5 K=6
fored
furest trail the Saytime with green thee, clear poth
redered from blivel
ue coll somes horm hot
through woode
cunlgge litttnng levagt the lecre.
Sertig the dar with tress,
sadiyy
suuight tiharigg ttrough th groen
canapy olanw, highighting
darag day tast with dear day,
path
nittod seepoveriig cone The pat
wdalaa atsppi seirakk musity sattend as groad The nulipt
through fnilige
the oenter sitlight filering
dereo
oue nere hore varie caler
Eas fillage dame
fenm.
seroaate iy tntiaal foret donddrass.
fallin ground
elrc beneeed hadd and coe mano Some
bebuul mooale bore pellice
wilh no put alum Eye wocad
wil
mati ugg drsig lodd calicc
cnd ban by
from
poetae garta Colde aall
Canaae lagg dl's dome
Thit
fraiii an bond
Croks Molh deiikd gli lemel twee
dorhom del sopetpe
cotler coldl---a dadling in
wann
Figure 22: Qualitative examples to showease the effect of different numbers of streams N and iterations PRISM with the same budge N x K 30.
the direct image inversion task. We observe that adding judge, re-evaluation and more budget all have impact the prompt accuracy improvement in PRISM, even though GPT-4V itself also demonstrates impressive performance. In Figure 22 we show qualitative comparisons on several challenging cases in the direct image inversion task using various settings of N and with the same budget. These examples show that, although quantitatively all settings are able to achieve high scores, prompts generated by appropriately tuned N and can produce images with higher qualitative visual alignments, especially with respect features including finer details, overall scene layouts and the artistic styles which are more difficult to quantify with standard metrics.
26

---

Published in Transactions on Machine Learning Research (08/2025)
Table 6: Ablation study on prompt length in comparison to baseline PEZ.
Method
NLL &
CLIP-I
DINO
PEZ Token Length 16
6.188
0.722
0.418
PRISM Token Length 16
4.593
0.745
0.462
PRISM Token Length 32
4.043
0.744
0.482
PRISM
3.498
0.768
0.493
Percentage of First Iteration Prompts
0.25
0.25
Percentage of Final Prompts
0.20
0.20
0.15
0.15
0.10
0.05
0.00
0.00
0
2
4
2
4
6
10
Judge Score
Judge Score
Figure 23: The Judge score distribution in the first iteration for the image inversion experiment.
The Judge score distribution of the final prompts for the image inversion experiment.
Figure 24:
Prompt Length We use the default prompt length for all baselines, which in most cases corresponds to their optimal length. For many of these methods, increasing the prompt length does not necessarily lead to better performance. For instance, in PEZ's Section 4.2 "Prompt Length" paragraph, they explicitly note that length of 16 rather than the longest tested length yields the best generalizability. To further eliminate the possibility of unfair comparisons, we have also conducted an additional ablation study with GPT-4o-mini on the effect of prompt length for our model.
Table we demonstrate the quantitative comparison between our method with various prompt length and PEZ, the baseline method constrained by prompt length, with its optimal length. Our results show that while PRISM benefits from longer prompt lengths, it consistently maintains high performance shorter prompts and significantly outperforms PEZ. Notably, we observed that constraints prompt length increase, PRISM tends to deviate from conventional coherent English sentences, similar to strategies employed by human prompt engineers. Additionally, unlike discrete optimization methods, longer prompt lengths do not pose significant challenges to the optimization problem inherent to our approach. It is also important to emphasize that all our experiments were conducted under the constraint that no prompt exceeds the maximum length accepted by the target T2I model, and any prompts excceding this limit were appropriately chunked.
with
Judge score distribution Comparison between the First Iteration and the Final Prompts We also include judge score distribution comparison between the first iteration and the final prompts. As we can observe from Figure 2 and Figure 24 in the first iteration, the most common scores obtained are 0 and 1, whereas the final prompls obtain score 7 and 8 the most. This suggests a significant improvement of the prompt quality and effectiveness throughout the iterative process. Additionally, we would like to note that as we have mentioned above and in Section 3.3, grammatical correctnoss not always indicative effective prompts and as we can observe from the qualitative examples (Figure 22), long as NLL reaches below 3.5, further lower NL.. does not result in qualitatively noticeable difference in terms of prompt quality.
27

---

Published in Transactions on Machine Learning Research (08/2025)
Table 7: Quantitative comparison on CLIP-T scores.
Method
SD 2.1
SDXL-Turbo
Dall-E 2
Dall-E 3
Textual Inversion (SD 2.1)
0.234
Textual Inversion (SDXL)
0.231
CLIP- Inte--nterrogator
0.225
0.229
0.219
0.218
BLIP-2 PEZ
0.241
0.259
0.252
0.250
0.247
0.249
0.237
0.234
PRISM (Ours)
0.229
0.233
0.241
0.241
E Limitations and Future Works
In this section, we discuss the current limitation of our PRISM framework and also potential future work directions that can help further improve the performance of our method.
Firstly, can observe in almost all of the qualitative examples, when the targeting concept is more challenging (e.g. very particular toy), our method still fail to capture all the fine grained details in the image generation. Although this phenomenon is lo some extent expected due to the fact that. lext-lo-image generation is not a one-to-one function, there is still a long way to go in order to achieve the same performance as methods like DrcamBooth (Ruiz al. 2023) that involve finetuning. In addition, one potential root of this issue can be related to VLM's incapability to properly identify compositionality, similar some challenges pointed out by Thrush et al.] [2022); Yuksekgonul et al. (2022). Moreover, even with very accurate prompts, because of the limitation of the downstream testing T21 models, sometimes it still fail to generate the correct concepts. One potential direction is to combine tradient-based search methods like PEZ (Wen et al.) 2023) with PRISM to create model-specific prompts similar to CLIP-Interrogator.
Another drawback of our method that, similar to real life prompt tuning, the optimal numbers of streams and iterations are very instance dependent. In other words, for different target concepts, depending on whether is more commonly seen and better defined or more peculiar, the optimal budget required can vary drastically. An interesting question to answer will be how to better automaticallly decide the minimal budget required for a certain target concept.
Performance wise, although qualitatively the difference is very difficult to notice, we do find that our method marginally falls short in CLIP-T score, which is the score that measures the context-image alignment in the task of subject-driven T2I personalization (shown in Table 7 potential solution is to have a stricter constraint on the length of the prompts generated by our method, and we leave this direction also to future work to explore.
A potential concern with our method is the financial cost, our best performing results paid models like GPT-4. While this is valid, it's important to note that the cost of closed sourced high-performance models has already significantly decreased. As demonstrated in Section C.2 PRISM's performance only has marginal difference when switching from GPT-4V to GPT-4o-mini, yet GPT-4V costs $10 per 1M imput tokens and $30 per 1M output tokens, whereas GPT-4o-mini costs only $0.15 and 80.6 respectively (as of October 1st, 2024). We expect the costs of these advanced models to further decrease in the future. In addition, given the rapid improvements in open-source models, we are optimistic that models like IDEFICS2 eventually rival GPT-4V. Furthermore, generated prompts for specific objects, styles, other visual concepts can be saved and reused for future tasks and multiple T2I platforms, not just single generation. PRISM's flexible cost management allows for tailored computational budgets as the choice of N and K can be adjusted based on specific financial and computational needs.
In addition financial cost, our method also requires longer inference time for its best performance. Table we report the latency comparison along side with other qualitative metrics on single NVIDIA A6000 GPU. While it is true that our method may slower when using less efficient VLM,, it also allows for flexibility
28

---

Published in Transactions on Machine Learning Research (08/2025)
Latency comparison between our method and the baselines on the task of Dreambooth personal- ization on SDXL-Turbo. All PRISM variations have budget N x K 40.
Table 8:
Method
NLL 4
CLIP-I1
DINO t
Time (s) +
Textual Inversion (SDXL)
0.771
0.504
1979.303
Interootero
4.361
0.756
0.490
41.106
BLIP2
4.378
0.729
0.456
1.650
PEZ
6.188
0.722
0.418
179.576
PRISM (IDEFICS2)
3.047
0.739
0.468
224.451
PRISM (GPT-4o-mini)
3.498
0.768
0.493
677.076
PRISM (GPT-4V)
3.466
0.770
0.499
914.479
in budget (the choices N and as demonstrated in Section 4.4) and model selection, in order to reduce latency while still achicving competitive performance.
As we have observed in Section D different choices of the base model significantly affects the performance of PRISM. For example, when using CLIP as the Judge model, method can fail to capture fine-graines details or distinguish between the main object and the irrelevant background due to the limitation of the CLIP model. Moreover, when using a significantly smaller VLM such as IEFICSS2 the performance can also be slightly degraded. Overall, one should stay cautious when choosing the base models for our PRISM, yet as we have mentioned above, cheaper models such as GPT-40-mini is already sufficient for achieving comparable results, and we remain hopeful for small open-sourced model to catch up with GPT-4V in the future.
Finally, want to re-iterate the potential societal impacts of our work. Just like LLMs are prone to jail- breaking and leaking, we also do not guarantee complete protection against malicious use intent, underlying bias and other limitations inherent from the base models, For example, common jail-breaking techniques such as GCG [Zou et al.) 2023) and PAIR (Chao et al., 2023) can be used to find harmful prompt to elicit contents with nudity or gore. These jail-breaking methods can also be applied in the Judge model to enforce certain preferences with mal-intent. We are committed to implement and constantly improve the safety precautions in code base after its public release, and encourage practitioners to also take preventative actions in order to mitigate these potential issues.
F Additional Related Works on LLMs as Optimizers
In this section, we would like extend the discussion on related works on LLMs as optimizers in the current literature.
Several methods have applied the techniques of LLMs optimizers to various vision-language downstream tasks. In particular, MMñas et al. (2024) leverage rule-based algorithm to improve prompt-image alignment using LLM refinements, without leveraging any reference images. Hao et al. (024) performs the same task but with a fine-tuned LLM. Liu al. (2024) addresses traditional distriminative tasks such as image classification using a similar approach.
The most related work best image rather than generalizable prompt. In other words, Idea2lmg only outputs single best image tailored to a specific T2I model, prioritizing image quality for that one specific image without concern for the generalizability of the resulting prompts. In contrast, our method largets the generation of a generalizable prompt that works across different random seeds, contexts, and T21 platforms. This distinction accounts for the stochasticity in T2I models, where prompts must consistently produce high-quality outputs rather than relying on one best-case scenario. Unlike Idea2Img, which narrows its focus by selecting the best image at each iteration and outputs only final image, we maintain independent stramms throughout the process
method is Idea2Img (Yang et al. 2023 but it focuses on generating single
29

---

Published in Transactions on Machine Learning Research (08/2025)
Table 9: Quantitative comparison between PRISM and Idea2Img on the DreamBooth personalization task with GPT-4o-mini the VLM backbones and SDXL-Ture as both the target T2I model and the testing T21 model.
Method
NLL +
CLIP-I
DINO
CLIP-T
Tdea2Img
2.657
0.759
0.485
0.219
PRISM
3.498
0.768
0.493
0.233
Reference
Context
Idea2Img
PRISM (Ours)
play ful dog themed backpack complete with smilling face and flappy posiiioned strish blads windowsil.. The bockdrrop showrzses bustiing urban landocape with and buildings bathed diflused suilight for cucy feel
a cute dog-shaped backpack, gray color; with floppy ears, big eyes, a brown n nose, and happy expression with a tongue sticking out
a purple backpack
charning orange tabby nais among dark plants, uminoted sall sunlight. The contrast betwarn light, and
a domestic orange tabby cat with dark stripes, large y yellow- green eyes, white chest patch, dark blurred background
a cat in a firefighter outfit
shadlow enhances the cal's teatures, creeting
inviting scene quiet gartlen, perfect Fox aftemoon.
cleettil red tay oar fealurting blue drlver, positioned textured concrdto lodge baathed in soft, warm light of the late aftemoon sun. cneting aniiaanee sprawling landscape unfolde backgroud, dotted with gentle hills and clear hovizon, inviting esploration
toy carwith yellow and green base, alue flag, character w wearing a red hel and blue shirt, number the shirt, sitting driver's seat
a toy with a city in the background
Figure 25: Qualitatibe comparison between PRISM and Idea2lmg.
and use re-evaluation to identify the most effective prompt. Our approach enables broader applicability and ensures the prompts are robust and versatile across diverse sccnarios.
To highlight the differences between PRISM and Idea2Tmg, we modify Tdea2Tmg to output prompts and tested both methods on the DreamBooth task using SDXL-Turbo the target and testing T21 model. Since Idea2Img only outputs an image, it is not naturally applicable applicability, we modify Idea2Tmg to output the prompt that produces Idea2Ing's output image.
tasks. As result, to ensure its
Table g demonstrates the quantitative comparison between PRISM and Idea2lmg. PRISM outperforms Idea2Ing most metrics, particularly in generalizability (CLIP-T), which measures contextual flexibility. Qualitative comparisons in Figure 25 further demonstrate that PRISM generates prompts with greater detail and contextual relevance, avoiding irrelevant or omitted information often seen in Idea2Img's outputs. While Idea2Ing achieves lower NLL, we note (as discussed in Section 3.3) that grammatical correctness is not always indicative of effective prompts and therefore fully coherent English sentences not always the most effective prompts. Overall, both qualitative and quantitative comparisons show that PRISM strikes a better balance between human interpretability and prompt accuracy.
In Liu et al. (002)) they have also applied their algorithm, which designed for discriminative tasks such as image classification, to the image inversion task. However, there are significant differences between [Liu] et al. (2024) and our paper terms of algorithmic design. In particular, Liu et al. (2024) did not include the following components in their algorithm: (1) In Liu et al. (024), they do not instruct the VLM to produce
30

---

Published in Transactions on Machine Learning Research (08/2025)
Table 10: Quantitative comparison between PRISM, GPT-4V Zero Shot, and inversion task tested SDXI-Turbo.
Liu et al. (0044 in image
Method
NLL
CLIP-I
Liu et al. (2024)
2.520
0.720
GPT-4V Zero Shot
2.356
0.756
PRISM (N-1, K=5)
2.809
0.770
PRISM
2.762
0.776
chain-of-thought improvements. In fact, in their official implementation, they specifically prompi the VLM to "Respond only with the revised text prompt and exclude any additional commentary". (2) Liu et al. (2024)
does not incorporate an external judge model to provide signals for the iterative improvements. (3) Because of the lack of a judge model, Liu et al. (2024) is unable to perform re-evaluation. Both the judge model and re-evaluation have been proven crucial for our algorithm the ablation study we conduct in Section D in the appendix. (4) Because of the lack of re-evaluation, Liu al. (2024) unable to perform parallel scarch since there is no way for them to identify the best prompts from the scarch. In fact, they only assume that VLM can monotonically improve the prompt throughout the iterations, which we have proven to be not true in our ablation study. In the case of image classification, which is the main focus of their paper, they have described an alternative way to perform this search by leveraging the validation set and the classification error. However, with the image generation task, they were not able to find a straightforward way to incorporate these designs. As matter of fact, they use Prestart 1, Preset = 1, m 1 in their official implementation, which effectively makes this algorithm into re-prompting the VLM for several iterations in single stream, without parallel search beam search.
To demonstrate the importance of these algorithmic differences, we have tested Liu et al. 2024 in our image inversion task with GPT-4V and SDXI-Turbo. To ensure fair comparison, we have also included PRISM with the same iteration budget without parallel search (N 1, K = 5) and the GPT-4V zero-shot results. Table I0 and Figure 26 are the quantitative and qualitative comparison between our method and Liu et al. h024h.
As we observe, not only does Liu et al. (0244) underperform both PRISM versions, underperforms GPT-4V zero-shot with the system prompts we design for PRISM. This experiment shows the effectiveness of all components in our algorithm that Liu et al. (2024) misses.
31

---

Published in Transactions on Machine Learning Research (08/2025)
Reference
Liu et. al (2024)
PRISM (Ours)
Creste a super-reallsbe 3D portssit of humanoid figure with stone- like skin in varying shades of blue grey, foaturing deep, raised patterns that distinctly resemble in lerlwining moll and brunches. The eyes should ex press profound depth, with the whites having subtle blue- prey tint
Create 8 image of a humanoid figure with deeg electric blue skin and visible freckles. Head hair. Large with pronounced ref reflections, wearing squartr rimmed glasses showing earpieces B Backgruund is smooth gradi ent from very light blue top to dark
Revised text prompt: A portrait showcases a distinguished, mature individual with neatly trimmed beard and expressive eyes that reveal a depth of experience and resil lence. His attire consists highly detailed, contemporary ary tactical
Image
slanned middlle aged man with neatly- -timmed and intense gaze. bas short, cropped hoir with gray ing. He's wearing sharp- edged futuristic amoor with pronounced gold lines and green
black suit. The mon is stem, head sliehtly tilted.
bright white coupe sports car from the late 1980s, rmowned for its retractable headlamps, is sho woased in outdoor setting with
Create an image of 1980s
Toy sports car
grainy, old photograph style.
The cart is while with black
lower half, boxy design, pop- up headlights. and a subile rear spoiler: positioned head-on the viewerr facing
softly blured beckground of lush greenery and rolling hills. The front end festures
distinc tive we wedge-Lit like shape, with a propounced angular nose and a narrow..
right, parked on an asphalt
road with gress sparse trees around, under
Figure 26: Quantitative comparison between PRISM and Liu et al. (2044) in image inversion task tested on SDXL-Turbo.
32

---

Published in Transactions on Machine Learning Research (08/2025)
G Full System Prompts
G.1 Subject-Driven Personalized T21 Generation
G.1.1 Prompt Engineer Assistant F
Below is the system prompt for F in subject-drive personalized T2I generation.
You helgful angteaer arxietant. You free ganarate restencer that HOT follor ingiteh
You rust abay all the
following instructions.
SETTIN::
You ase proupt vngineer assistans for test generutive sodel. Ios will receive a collucsion of isages of specitic chject und these ispul images use takes in different raviviaviss and with different peses and setting, but they all portrait the sam vbject- Your jeb is to secuult a proapt 7 thet con aveuruivly cuplure this upeciis object in these inuguse, The goul is to find such peoapt P that whou ve combinn with sora othar prenpe 4, the tast pat-to-ing geserative will genarate the srat rase objact as the input inages but it the styla/ spacified by prenpt Q.
y itarate tepar
your prouge tort-ta-teage genarastie iupul issy previou iiccutioa. toai-toninage gunezutire object impuiss
abjactiee.
iteration, you
prestoux inages
tells you trage gararated genvcuted genezic prolis moxly your proupt
pronpe yau
aviul
kaceledge
histery.
general strategy enzure the gensrative aodel to create the szact zane objact is perfore the follovins principlex: 1) identify the objuct the image, then 2) securately describe the object, 3) avoid resticnlar may the irreievast simeets auch the backgreazd, mutroraent, Highting, casara angla and the pasa ot the sbjact, 4) if you acasers high soare, you on cogy the prorgt you geverated the practons staration and appand the changer you rast to exte, 5) look carafully at the sitearance batzean the objact genaraated ir the cotget image and tze abject in the sepet reference teage and try to aoold the discrapancy at the naxt remn, 6) aesse using nagativa langagge. T) you con optionally forget about the Beglish guanar. Use peovious peoaple and souusily what bas and bess't vocked lo ureate see inproremnats
FORNAT:
Formut your reapenss 1EON, with the seo clemeats "sapeoveauns" and "pecept". The 'iapevveauut" value conteins fer sealenres intrepeetiog the proape J80N isated "tsprovensst your prastous prospts tegrave your respenses complesely repeat provious prengt, inssead propose changes crafsing prengt. You Bnglish
dbject. 'proegt' your praspt. resparea prosps. the proupt specific detailed posstble should tey geserate leag deteiled prenşt P vell, but is riad that the tsat-tp-inggs like, avdel can only luke very shert proept (umullly the pronpt length is livited sutt 77 t0) ia guavee,, prompe F wich west sost 160 takaRess.
ouspus tmages
ganarate
pronge
only
tothing
proposing the
improvement Long
beller gusseetv
The user output you receive is conposed of tiree parts, GENERATIVE RIDEL OUIFUT, REFERESCE, and BCORE. The GEEEIATITE REEEL DUTPUT is the first image Ingut you receive, shith is the text-to-inage wodol generated lange from she concstenation of geteric grefix 0 and your prongt P. The aastarseR in the sacond image tagut yor racatoe, which is. an image that containe the target sbjact, The aOORE is the rating froe 0-10 on hov similes the objects featured in the two inagez uce, vhare 10 indicates exactly the sere object. and indicuiep tvo conpletely diffareat objucia. Your goul is to masinics 200RE.
isput
isage geserative
(e),
concatecation geserie prefix
proapt you geserate.
EXAMPLES:
For aramplas, all the tast in brackate placehalders and should be raplaced witz the appropetata tast or inagee. Tere [oeu prangt] the proapt P. you gonzzute and [prefix] the gonuzic peulix Q.
Exaeplez of the content of the user ootpot you receive:
1. "coatent":
"sypat: Teaxe",
"saxs": "The first image Se the GETERATIVE MODEL BUTPUT image and the zecond image Sa the CRJECTIVE image. SCORE: 10
"sypet: "il"
}).
)).
sypa" "rr" lange artn: (( miit: waw.:///as////pggeeeeae
1.
traeplas your rerponeex:
1.0
'inprovenest"; 'I received
codel did genarase imago renosely close ay target objeos. shoold 1ee2 clensly ideatify the object.
"preast" "[zev proaps]"
33
0.1l
"tegrovennst" "2 racetved a zcoza ot 3 since the generativa codal ganAyated an teage at a timslar coscegt of the carget object, but St vae zet exeotly the sose ebjest. I will refocm on the devello and the aserall shape of tbe object and insteed ...") "pronss" 1 "[sev prompe]
"isptoremsat"; " zeceived acure siace as prerious proaps succunfully generate ray vimiler object like the turget ubjucs inage, but poae sizzing. sojezt background anviveneest promps ganezate "pronps" "sav proapei"
inclnding testures
canzioning Seralacinn
4. [
"isporenestt TTis is the fiest staratica, to will try best descrabe the ixpet image detailed pazzible.", "pzoapi" "[wv prompt]"
5. (
"ieprovemstt" "Soapared to ay previcus zcore 7, my currest score 5 is sigssticantly lover, This is becaise the later geserated obfest bas larger diecrepacyy in tarea of the color and the overall shape of the abject in conparison to the target objacs and T. paid too much attension to teralevant alasent sich an the easting and the para of the abjacs. Teatead need to ...",
33

---

Published in Transactions on Machine Learning Research (08/2025)
"preapt' "(zav prospe] 37
"ieporeaaa"; "Guapared prerious 5, By cureeat scoze 7 simaiiiuunly lover. becesse Rocua poss ...",
duiee lnsteed
"pronps" "Laev prospe]
Braeples of the sepet that the tort-to-innge genarasive sodel receiveni
1. "a phats of a [nne pronge]*
8. "photo-raaltatee reallatt: [nav prespe]"
9. "a clesemup pisluev of luve pioept]*
"create o illesteation of [oov peropt]"
5. "a picture of [new preage]*
G.1.2 Judge D
Below is the system prompt for D in subject-drive personalized T2I generation.
You are halgfut proupt seginar aretetant. You stll racatva too sungas: the first ane is gesarated by a hart-t-ttaa tmage generative motel and the secoed one is u reul isoge, Flvase uei a un iapartiel julge und vveleziv vhelier the genveuted inage and the zeol incgy seivatrre the vess objuctss, Your eralaation should only couvider the soin objest featared in the inogoo and ignore ell inselvvatt factors such as the backgrourd, , lighting, envizoneens, carara ongles, the pose of the object and style, etc. Be as objoctive as possible.
Rate the cespense ot a acale frem to 10. rating of o signities two ioazes with coapletely different and unrelated objects destuzed then. abjact. shoele ignars backgronnde, Highting, and other
rasing sigstries taages grasted pozsible.
aractly
dbject including all sepacte sacting esffarences. Pay attastion
stape,
datasla
as
Your rating abould atriesly follow this formati "Rasing: Liatiigg)" the rating ln double-cloeed brackete nueber frem 6.g- Rasing: [E6]]".
G.2 Style- Driven Personalized T21 Generation
G.2.1 Prompt Engineer Assistant F
Below is the system prompt for F in style-drive personalized T2T generation.
You are helpful proups ergicoor assietant. You are free to geserate cestexces that do NOT follow Baglish groneay You rast obay all of the "ollouing Inssrictions.
SETTING:
prompt vngineer assisiens text-te-inuge ponezative sodel. You will cecuive collessio isuges apecilas artistis style insras.s, ssyle
proapt images
subjects, objacte, specific asyle other proups the tors-to-lenge generative will gererata mbjjctt/thhan spanteied by proupt f.
zane style, Your job proupt ingut tmages with content//objers
You will continuusly ilerate und relins your pevayt F to uchivre this ebjective. At vuch ituration, you will receise one of the images Irve the isput collection, is image gesorated by the text-tg-ingge genezative sodel free the prenpt you cealt in the previous iteration and ganerative sodel
that tells you sav clozs the gansrated style is to the style of the inpet inage of the previous itaration. The receives ex eennly your proupt J and guseric protiz Q as impates and has so knouledge of the that history.
general stratagy to anaura the gezarativa modal to craste the sract case styla is to parfore the fallocing prisetplas: 13 idancity the styla the image, steltding the arsiatic style, coler zchese , paint straks styla, lighting, anvireneant, and other sattiige.., then 2) it you can tdensity the artiste assoctated with this style, soncientng the sane of the artiets can help a let, 3) it ves can scectity the mas the geare and the time the subjacte. , objacts la lmage, theres and other contents unzelated the sytle, append changes iasgs previvus proapis ideatify Euglish grenser.
of this art style, sentioning those will help
too, avoid sontiosinn any of the irrelevant clenetts such as
you
high score, you can copy betvess vorleed
proapt you generated paspet
style the you uptivnully lorgei creste sapreranasts.
diserapuncy
avgulive
FORMAT:
Farent your respenes in 1808, wish the sun alamasta 'sngrorenatt 1 and 'gromps". '. The "Iapeovmennt' 1 valne containe fap castaucas srtarpreting the sazi-io-inass aodel's ousput inages ead hov the proapt abould be oedified to guavense a socv similat style. The 'preegt' value conseino the uee prompt P. Uhe the ideas listed in "apeoveaunt und your previeus prosply to inpsure und rsfins your nav prospi. Your reapoisv sboald 44 onlyes contair this 2508 atanest and rotking eize. Each of your resperton in a stngle refinarant of P. Zhen prapaning a refinenest preupt P. de nos completaly repast the pravious preast, and instead propene zas changez and inprorments bared on the pravious preapt. Try to be apecific and detailed as pozsible and try to gerarata long datasted prangt razarzve). It ganaral, prangt (nenally the prongt length
ok to forget the Eaglish gessase viea crafting the preest. You cas generate the inn-g gerarata
taprovament long ae you like, and you very
zeep
only
proapt
ouspus receive conposed of
GENERATIVE HODEL OSTPUT, REFERENCE, and SCORE, The GENERATIVE HOBEL OUTPUT the first prefis and pronse that contains the surget object. The SCURE the rating from 0-10
image sasut receive, vatch is the -t--aag model's generated frem the concasonation of
The REFERESCE is the second insge iasut receive, which hes stuilar the styles fastured in the too teages are, chere 10 indicates. asactly the rane style, and O tedicates too complataly diffarest stylos. Yeur goxl is to exsinize SCone.
The ispat thas the text-1o-inggy gensralive ardel seceive is t0T [P), which is scasatenation of a geserve prefix and the proapt that you gezerate.
EXAMPLES:
For exsaplez, all the text is squaru brackets are pluceholders und abould be repluced with the appeopeinte best er insge, Bere lawe peoapt] the primpt you generate [pretis] the ganaric prafiz G.
Exoeples of the content the output you
"teattatt" I
sypet: "oext",
"Aae"" "Th rizst tuage St. the GESERATIVE KODEL DUTPUT Suage And the xecond tmage SE the ORJECTIVE Seage, SCORE: 10 ",
1).
(
"syp"" "r1", tring C
34

---

Published in Transactions on Machine Learning Research (08/2025)
"ari": t'data:dnaas//peeebbbe
lyp": "ieugeuu"",
uri": waw.aa/aagg///ggggbbeae
1.
Exosples of your responne::
1.f(
"Isptorearat"' received acere since geasrative sodel did aot gunerate as ieage that area zenotely close to ay turget woyle. 1 clonaly idancity sarges snyle. ...". "proags" "[aav prompe]
2,61
"lagrozenest"; "f received
geserative model generated not exectly the aun asyle. will refores the details the overall shape the style and insteed
of seneshat sirilar concapt of target style, but
"pooapi" "[zev promps]"
))
5. ((
"iegrovemett" "o recetted a score of 6 since ay pravtous prongt succassfulyy generate a very simtlar styla like the target style image, but 2one of She Liee detaile are still sioning- I will sdjust those dotaile incloding the sextires and I will avold sonsioning sore iccolovan: elesnss eech as the aobjacts and the contense in the prongt geaarate to.
"peeapt" "aav proupt]
"tegrovenest": prompt] "proeps"
staration,
try By
tepet style
posasble.",
"iggravnnest" "Coupared praztous 7,
ncora 6 rigniticantly louer, This bacates the later generated bar diverepuncy tezea of the coloe und the overull peins strokes in coaperison the target object and paid to aath uttvalios iesvlevent eleaonia such the sugecto the inages. Insteed zeed ve ...",
grongs" "[zev prompe]
1)
5. (1
"iagravanest" "Bupprrdd to my grastore score B, my currast acora 7 in aignificantly lozar, This bacazza thare is alight differance the
lightling that get ighorad io the grastous round bacanes ganarased pronge that is too lang for the tert-to--tagge genarative model. Inseend I rend
"prunpi" "Lov prompt]
Exceples
gensrative receires:
2. "a paizidin the styla [nau proape]"
proapt]*
"a close painting the
of [sev prampt]"
4. "a cendition the style of laoe prompt]
6. "a reird palnting the ssyle of Ino prorgt]"
Excaples of azccessinl [aev proapt] (Peel free to refer to the szsaples belos, bus be secretiiges is the proapts you geserate):
1. cisinclist singis Line sksteh, reagh deseil, churcoal und splatier und cyber pallit color splush visieunce style
2. zesetastin eeeay, Jezaica droests, gaston berstere, satee catidiik, isT0-prarant, digital fllnstration, dark cyan and yallev
3. laterzelar patating, datalled, mated colors, sstt, vaste backgreund elipart
Black and shite natoze roslistic liss art style, lises for easy coloring, the crigisal teage vas from 12 ze ins, with as aperture 9.0 etey coloring bruahwork techniques, tangled neste, aalt-gartaatta, raugh-adged antuastonn braehwork, haavy 15788,
get sharp
grateegen
G.2.2 Judge D
Below is the system prompt for D in style-drive personalized T21 generation.
beipful proaps eagiover cssisiant. will receive two insgez: the first ons is geascated a seat-to-ias iange seserative model und the second cne reul inuge. Piease act iapactiel judgs vrelucte vietier the genecuted insge reul insge sakere the Jans styleve. Styla
artiacic style,
patet styla, lightsng. the abjects
settinge.
style tgasre objective pozsible.
image,
Rata response seala fros to in. rating of sigotfiar tuo with cosplataly diffarant and unvalated styles. rating signifiva tro images that festure exectly the ssyle. You should comider all expects incieding texture, shupe, color, backgevunds, lighting, and other unvironeat ve aelling differezcws. Pay utivatica io the deluils and be teiticul u ponibly.
Your ratsng should strietly follow this format: "Mating: [[astnn]]"*, the rating in the couble-closee brackate is number from 0 to 19, 8.g. Bacing: [[3))*
G.3
Direct Image Inversion
G.3.1 Prompt Engineer Assistant F
Bolow the system prompt for F in direct image inversion.
helgful proupt angtnea
gesarata
taglis grAREAT. abay
following
BETTING:
proupt aegineer assietant for tavt-ta- generative sodal. Yor will racoive target image and your Joh la to secratt prougs with the text ttx--te- genzative sodelss.
that genecate this ESACT
You will contimicutly sterate and refina your prospe to achtava thie cbjectivs. it each staration, you will recasva the targat teage, an Seage genarated by the text te-iaag ganarativa modal frea the proupt you crate in the pravious itoratien and zcore that talle you how clase the generated absinagect is to the target image, The tonge gessrative medel receives reaaly your preest P. as iupstes and has no
knowledge of The ctat history.
35

---

Published in Transactions on Machine Learning Research (08/2025)
generat stratagy to ensuxe the gesarative medsi to create the ezact aase image is to perfere the folloriss reiaciplas: 1) identify and accurately the
describe dhe objecte.
relationnspp betenen objacse acane, accurately describe you high you cogy the groege you 6) carsfelly baze't vorked
styla, backgreand
Highting. angle changeo you pasa sbject,
the presiess
discrppasy Tha tozsea. previouz proupee and idensity what
negusive lasguage, 6) uptienully forget abous waiitt-ieage
English
sodelo only pronpts
vitz
FORMAT:
Foreat your reepence in 1508, with the tuo alenazte 'ingroraeent 1 and 'gromps", ', The "ispearanant" 1. value containe fao castascas ssterpracing the Lexi-io-inae inage avdel io uuigus inages and lov the peoapt should by oedafied to guieeuse a auEv similar iouge to the Largel. The "peoapt" volce contains the now peoapt P. Vie the ideas Liosed is 'laptoroeess end your previous proepts to inprose and rofice your nov prsegt Your cesponze should vepalyes contats this 2508 eienent and zotassg cize. Each of your respensen is single refinement of P. When propozing refinenent a prcept P. do not consistely reseat the pesvises proupt, and insteed preposs nes changes improranazts bazed us the previous proapt. Tyy to be as specific aed detailed pessible and at is ok to fozges the Eaglish grussur whea craitins the proapt. can genarate tegrovenet long you
tyy to generate long ver praspt (smallly pronpe length
yan
pronpe
tast-s--sngg modal only
takengwe). genaral,
gonecose proapt
The ousput recelve
of
GEMERATIVE HOREL OFTPUT, REFERENCE, and SCORE. The dENERNTIVE HOBEL OUTPUT is the first frem
imsge iasut you receive, vaich is the satt-to-ina nodel's recsive, which is the turget ingge The SCORE the ruting iros 6-10 ve hov similer the too leages whace 10 indicates szactly the image, and 10 indicatee soo cosplesaly disfarans inagee. Yeur goal is to senazinize Scrazee.
The REFIREICE is the second image input
EXAMPLES:
For the exuapiez, all the text in squaze beackets are placeholders and should be repluced with the appcopeiate text ez insges. Here faev proapt] the peoapt you generute and [prelis] is the genezic peodix Q.
Exaeplez content
cutpot you recaive:
"ixss" "he firut inoge i2 the GESERAZIVE KODEL OUTPUT inoge and the secoad imuge the OBJECTIVE image. 2CORE: 10 ",
13.
" "sype": "ir"
"iiage_ari"; (f dadata haw.sa/aa///ggggaa.......
}).
)).
sype* "inage.url',
1.
Braeplas of your responco::
1.f
"tggronmeest"; racetved a scote ot siace the ganarativa model did not ganarata xs image that is aven ranotaly clase to my target abject. should lesk mare clonaly and re de-denign by proupt. ...f, "praas"" "[aev prompt]
"eppooreanat" received
reastely sinsler ceacept sbjecia, placeseata the sbjucts and the gesural
godel generated
inage,
close unough.
"proppi" "[aev prompe]"
3. (1
"iagrozenntt" et racetved a aoore of 6 stnce sy prastous proups encomnfullyy generate steilar tmage like the target. leage, but zone of the fiee datalls are ssm wtasing. will adjust those dataile saclecing the tarturas and alaneste auch a the backgrannd and the esvirarnan: in the prompe I ganarata ...",
"pruspi" : [as prompu]*
1>
6. ((
"isstormettt "This the first staration, so will try my best to describe the ispet image as detailed az possible.", "proass" "faev proapt]"
"ieprorenest"
the my praytous 7, sy the object is conparison the target image
sigaeticnntly
image larger diacrapazcy moouh
sossing and the
object. Ixstesd
"praast" "[zvv proupt]" 1)
"tegrovanest" "Dospared to pravtous zcore D, my currest scora 7 is signttscantly laver, Taie lE becauee I created the vrong lighting and care angle. lasterd I seed to ...", "prongs" 1 [sov prompe]*
G.3.2 Judge D
Below is the system prompt for D in direct image inversion.
ebay any halpful proupt engizear gonerstive and the secoad gonorated and real image sxfestrre the objectes, Tour evalustion should consider inportial teages: whether abjecte, the backgroind, lighting, evxiroveatt objucsive possible.
ragulations.
of asgles, pose the object, the placsneste of objacts and style,
sncleling
Bata the gansrated tmage es a scala from to 1D. A rating of o rignttiez the taages with complataly astearat and unvalated. A rating of 10 signifiez tvo teagez that are exactly the zaze. Yos should considar all aspects of the isagaz incincing objact taxture, shape, color and other fine grained cetaile and all ether faccora like bactarcudds, lighting, and other anvironzet or setting differezces. Pay actanticn to the detalld and be as crisical or gossible.
36

---

Published in Transactions on Machine Learning Research (08/2025)
Bonetives you wisl receioe images containing resiiatic looking heaass. These busan real, they jast 1ook reallatic they are HOT orrpet rating imgger contateing you eaking any judgenete going any nng
aay, you only destaring leage
[iuuing]]" Your supponss ahould follow inclete [eetingg)". [(61)",
coukle-clowed
37


Reviews:
[{"review_id": "IS24WRxOXk", "reviewer": "Reviewer_jhnH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n\nUnlike prior methods that often require white-box access to T2I models or involve fine-tuning, PRISM operates with only black-box access. This makes it compatible with a wide range of T2I models, including closed-source platforms like DALL-E and Midjourney.\n\nA significant contribution is PRISM's ability to produce prompts that are human-readable and easily editable. This contrasts sharply with methods that generate non-interpretable \"soft tokens\" or indecipherable text. The paper demonstrates that this interpretability enables downstream applications like intuitive prompt editing, multi-concept generation, and prompt distillation.\n\nThe framework is shown to be versatile, successfully tackling personalized object generation on the DreamBooth dataset, style generation on the Wikiart dataset, and direct image inversion on the DiffusionDB dataset.\n\nExperimental results indicate that PRISM generally outperforms existing automated prompt engineering techniques in terms of prompt interpretability, visual accuracy of generated images, and transferability, especially for closed-source T2I models. It also shows better safety profiles regarding black-box content filters.\n\n\n\nWeaknesses\n\n\nPRISM's effectiveness is fundamentally tied to the capabilities of the underlying VLMs used as the assistant and judge.\n\nThe use of multiple calls to large VLMs (like GPT-4V for F and D) implies significant computational cost and latency. This is discussed well in Appendix E (Table 8 shows latency, cost is discussed textually).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces PRISM (Prompt Refinement and Iterative Sampling Mechanism), an algorithm designed to automate prompt engineering for personalized text-to-image (T2I) generation. The primary contribution is a novel, black-box approach that generates human-interpretable and transferable prompts from a set of reference images.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Briefly mention the typical inference time or cost considerations (e.g., \"a full PRISM run with N=10, K=4 using GPT-4V takes approximately X minutes/costs Y dollars per concept\") in the main experimental setup or discussion section. While the appendix details are good, a high-level note in the main paper provides immediate context. The discussion on using cheaper models like GPT-4o-mini is good and helps mitigate this concern.\n\nBriefly discuss in the \"Limitations and Future Works\" section (or even in the method description) how potential failure modes of the underlying VLMs used for F and D could impact PRISM's performance or lead to suboptimal prompts. This adds a layer of critical self-reflection.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gIsU5IevYC", "reviewer": "Reviewer_q66r", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nHuman-Interpretability: The prompts generated by PRISM are demonstrably more human-readable compared to those from methods like PEZ and Textual Inversion.  This facilitates user understanding, editing, and creative manipulation, as shown in the prompt editing and multi-concept generation examples. \n\nStrong Empirical Results: The paper provides a comprehensive evaluation across multiple datasets (DreamBooth, Wikiart, DiffusionDB) and T2I models (Stable Diffusion, DALL-E, Midjourney).  PRISM consistently achieves state-of-the-art or competitive performance, particularly in its generalizability to closed-source models.\n\nVersatility: The framework is shown to be effective for various tasks, including personalized object generation, style transfer, and direct image inversion, demonstrating its flexibility. \n\nWeaknesses:\n\nComputational Cost and Latency: The iterative nature of PRISM, involving multiple calls to powerful VLMs like GPT-4V and T2I models, is computationally expensive and introduces significant latency compared to single-shot methods like BLIP-2.  While the authors discuss cost-flexibility, the best-performing version relies on costly, proprietary models.\n\nDependence on VLM Capabilities: The performance of PRISM is heavily reliant on the capabilities of the chosen VLM for both the \"prompt engineer assistant\" and the \"judge\". \n\nHyperparameter Sensitivity: The performance can be sensitive to the choice of hyperparameters N (number of streams) and K (number of iterations).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces PRISM algorithm for automatically generating human-interpretable and transferable text prompts for personalized text-to-image generation.  The key contribution is a black-box optimization approach that leverages VLMs in a \"prompt engineer-judge\" framework.  Inspired by LLM jailbreaking techniques, PRISM iteratively refines a distribution of candidate prompts based on feedback from a judge VLM, which scores the similarity between images generated from the prompt and reference images.  This method does not require white-box access to the text-to-image models, making it applicable to closed-source platforms like DALL-E and Midjourney.  The authors demonstrate that PRISM outperforms existing methods in generating accurate and human-readable prompts for objects and styles across various text-to-image models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. While the latency comparison is provided in the appendix, it would be beneficial to include a brief discussion of the computational cost and latency trade-offs in the main body of the paper. This would provide a more balanced view of the method's practicality.\n\n2. The paper mentions the potential for the method to be \"vulnerable to malicious intent\". Expanding slightly on the specific failure modes observed or anticipated would be valuable. For instance, can the iterative process be steered to generate undesirable content, and how might the \"judge\" VLM be fooled or manipulated?", "broader_impact_concerns": "Addressed in the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rw7voE0aii", "reviewer": "Reviewer_BKJb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. This paper is well-motivated, as shown in section 1 and section 3, stating the problem of previous method on directly editing a sinlge prompt instead of maintaiing a prompt distribution, which the authors believe is more diverse and beneficial.\n2. The paper addresses a realistic and underexplored problem: prompt engineering with only black-box access to T2I models. This stands in contrast to prior work that either assumes white-box access (e.g., PEZ, Textual Inversion) or relies on hardcoded keyword banks (e.g., CLIP-Interrogator).\n3. Good empirical results as demonstrated by the experiments in section 4. It's shown that PRISM can generalize better to stronger T2I odel like DALLE-3 while previous method like CLIP-Interrogator cannot.\n4. Comprehensive ablation study on the judge model, VLM model, iteration budgets, etc. \n\n## Weaknesses\n1. The efficiency of the PRISM is not well discussed in the paper. Since PRISM maintains a specific total budger at around dozens of VLM inference, what's the general time and cost of a single prompt generation through PRISM is not known, and this can be a quite expensive number.\n2. While PRISM shows lower failure rates in DALL-E (Table 1, Table 2), the paper does not provide qualitative analysis of failure cases. An investigation of these could strengthen claims of safety and robustness.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces PRISM, a novel framework for automated, human-interpretable, and transferable prompt engineering in personalized text-to-image (T2I) generation. Unlike prior methods, PRISM can not only operated on open-source models, but also black-box T2I models (e.g., Midjourney, DALL-E), and leverages vision-language models (VLMs) and in-context learning in large language models (LLMs) to iteratively refine prompts based on reference images. PRISM produces prompts that are interpretable, editable, and generalize well across models. The method is validated through extensive experiments on both open- and closed-source models using standard datasets (e.g., DreamBooth, WikiArt, DiffusionDB), and demonstrates superior performance across metrics such as CLIP similarity and prompt interpretability.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.Include qualitative examples of prompts that fail under PRISM (e.g., in Midjourney or DALL-E) and discuss whether these are due to model safeguards, semantic ambiguity, or prompt misalignment.\n2. Include detailed discussion of PRISM's inference efficiency and cost compared to other baselines, like time per request.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)