Visually Guided Decoding: Gradient-Free Hard Prompt Inversion with Language Models
Authors: Donghoon Kim
Minji Bae
Kyuhong Shim
Byonghyo Shim
Venue: ICLR 2025 Poster (2025)
Keywords: text-to-image
inversion
gradient free hard prompt inversion
language model guidance on latent diffusion model
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Visually Guided Decoding (VGD) improves prompt generation for text-to-image models by using language models and CLIP guidance to create coherent, human-readable prompts aligned with visual concepts.

Abstract: Text-to-image generative models like DALL-E and Stable Diffusion have revolutionized visual content creation across various applications, including advertising, personalized media, and design prototyping.
However, crafting effective textual prompts to guide these models remains challenging, often requiring extensive trial and error. 
Existing prompt inversion approaches, such as soft and hard prompt techniques, are not so effective due to the limited interpretability and incoherent prompt generation. 
To address these issues, we propose Visually Guided Decoding (VGD), a gradient-free approach that leverages large language models (LLMs) and CLIP-based guidance to generate coherent and semantically aligned prompts. 
In essence, VGD utilizes the robust text generation capabilities of LLMs to produce human-readable prompts. 
Further, by employing CLIP scores to ensure alignment with user-specified visual concepts, VGD enhances the interpretability, generalization, and flexibility of prompt generation without the need for additional training. 
Our experiments demonstrate that VGD outperforms existing prompt inversion techniques in generating understandable and contextually relevant prompts, facilitating more intuitive and controllable interactions with text-to-image models.

Published as a conference paper at ICLR 2025
VISUALLY GUIDED DECODING: GRADIENT-FREE HARD PROMPT INVERSION WITH LANGUAGE MODELS
Donghoon Kim Minji Bae', Kyuhhng Shim Byonghyo Shim 'Scuul National University "Sungkyunkwan University {chkim, mibae, bshim @iiimeisslab.. snu. lockk;; khshim@skku.ee
ABSTRACT
Text-lo-image generative models like DALL-E and Stable Diffusion have revo- lutionized visual content creation across various applications, including adver- tising, personalized media, and design prototyping. However, crafting effective textual prompts to guide these models remains challenging, often requiring ex- tensive trial and error. Existing prompt inversion approaches, such soft and hard prompt techniques, are not so effective due to the limited interpretability and incoherent prompt generation. To address these issues, we propose Visually Guided Decoding (VGD), a gradient-fre approach that leverages large language models (LLMs)) and CLIP-based guidance to generate coherent and semantically aligned prompts.
essence, VGD utilizes the robust text generation capabili- ties of LLMs produce human-readable prompts. Further, by employing CLIP scores to ensure alignment with user-specified visual concepts, VGD enhances the interpretability. generalization, and flexibility of prompt generation without the need for additional training. Our experiments demonstrate that VGD outperforms existing prompt inversion techniques in generating understandable and contextu- ally relevant prompts, facilitating more intuitive and controllable interactions with text-to-image models.
INTRODUCTION
In recent years, image generative models such as DALL-E and Stable Diffusion have shown remarkable success in gen- erating high-fidelity images (Ramesh et al.) 2022] Rombach al., 2022 Podell et al.) 20224). These models are widely used in a variety of applications, including visual content gen- eration (e.g., advertisement, movie, game), personalized con- tent generation (e.g., caricature, photo editing), and prototyp- ing (e.g., architecture and product design). Recent studies have shown that, just as humans can draw an image of an ob- ject solely based on detailed descriptions (e.g., criminal com- posite sketch), generative models can generate images of ob- jects using a well-crafted prompt, even if they have not been trained on those specific objects (Gal et al. 2023 Everaent al. 2023).
creste picture delicions
This the picture of food artinty French fising: Glimpoe hared gowemand Jérime interntiedd plat du jour specially created cand".
P
What it was mode by McDonald?
A well-known drawback of these approaches is the diffi- culty in finding textual description (a.k.a,, prompt) that effectively guides the generation of the desired visual con- tent (Hao et al. 2024). For example, to create a culinary mas- terpiece (see Fig. D.. one needs to have enough knowledge exquisite cooking styles such as French cuisine or molecu- lar gastronomy. Without the expertise to craft prompts, users have to rely on laborious trial and error to generate high- quality gourmet images.
Figure 1: Visually Guided Decod- ing (VGD) works with any LLM without extra training, making easy integrate into chat-based interface that offers interpretable and controllable text-to-image gen- eration.
Corresponding authors

---

---h-huum-adaaaaaa vector, users have no clue to Published as a conference paper at ICLR 2025
Target Image
Soft Prompt Inversion
Conventional Hard Prompt Inversion
VGD (Ours)
lovely perents admiration lips touch takelesson regnorrdeup abstractilluriios snimal discipline liots vulnerable stile ozin dtuciikeeeer cardinals serratimatefamiiinnn funedi sifawordccup stndes climate cancer sol oldders séhins
Linnee and her cué wniff closs: vihrent sun cose ahove them; them;they's they're against thicker, storter and
photo of Not Interpreiable
Lens Interpretable
Fully Interpretable
Figure 2: VGD generates fully interpretable prompts that enhance generalizability across tasks and models in text-to-image generation.
French
Trainable Prompt
CCLi
Food artistry most
French
0.4
Text2lin
LLM
P(Gμ)
Diffusion Model
OR
with
0.11
fLOW
gountect
0.21
Gradient-based
CLIP
p(1)))))
Update
style
0.05
&
QQantizze
(a) Training-based Prompt Inversion
(b) Gradieet-frre, Visually Guided Hard Prompt Inversion (Ours)
Figure 3: While conventional prompt inversion techniques update prompt embeddings through gradient-based optimization and quantization, VGD is a gradient-free technique that utilizes large language models and CLIP to generate relevant sentences.
To address the difficulty in finding the prompt, soft prompt inversion techniques have been pro- posed (Gal et al. 2023] Kumari et al. 2023 Ruiz et al.] 2023) Voynov et al.] 2023). In these ap- proaches, similar to the prompt learning technique in natural language processing (Li & Liang 2021 Lester et al. 2021 Liu et al. 2022, Kim et al.) 2024), an image is transformed into the embedding vector. Specifically, learnable input vector for the conditioning network (Radford al. [222] in Stable Diffusion is trained to reconstruct user-provided images through an iterative de-noising pro- cess (a.ka., reverse diffusion process). After training. the learned vector used as a soft prompt. Since the soft prompt is used as word, to generate the desired image, it should be combined with other words (see Fig. Unfortunately, handle it.
Recently, to mitigate the lack of interpretability, hard prompt inversion has been suggested (Mahajan et al. 2024| Wen et al., (2024). Essentially, training process of the hard prompt inversion similar to the soft-prompt learning but the trained input vector is projected to the nearest neighbor a predefined vocabulary to ensure hudaabieed (see Fig. (a)). The potential drawbacks of hard prompt inversion are: 1) the vanishing gradient problem caused by the long gradient path in the reverse diffusion process, and 2) the difficulty in identifying the essential words for generating the desired image. In fact, as described in Fig. 2 well useful words.
sequence of the hard prompts contain meaningless
An aim of this paper is to propose a novel prompt generation technique that generates fully inter- pretable prompts, thereby reducing the need for trial-and-error texl-to-image generation. Essence of our technique, henceforth referred to as visually guided decoding (VGD), is to generate contex- tually meaningful sentence for hard prompt through the token generation process of large language models (LLMs) (Ouyang et al. 2022 Touvron et al. 2023). In order to align the generated token in the provided images, we use the CLIP model in the token generation process (see Fig. 3 (b)). This process ensures that the resulting prompt is not only semantically relevant to the image but also easily understood by humans. The major benefit of VGD that one can fully understand the mean- ing of the prompt (interpretability) and synthesize images for various applications (generalizability) (e.g., style transfer, image editing, and image variation). Also, when users want to modify objects, background, and/or style,
easily change the generated prompt (flexibility).
In our experiments, we show that VGD qualitatively and quantitatively achieves state-of tate-of-the-as of-tth-art (SOTA) performance, demonstrating superior interpretability. generalizability, and flexibility in text- to-image generation compared to previous soft and hard prompt inversion methods. We also show that VGD compatible with combination of various L.Mss (i.e, LLaMA2, LLaMA3, Mistral) and
2

---

Published as a conference paper at ICLR 2025
image generation models (i.e., DALL-E 2. MidJoumey, Stable Diffusion 2). Since VGD seamlessly stitches LLMs and text-to-image models without additional training (i.e., gradient-free) VGD pro- vides flexible and efficient solution for chat interintefacee image generation services using LLM (e.g.,
DALL-E extension on ChatGPT) (see Fig. 1
2
BACKGROUND AND RELATED WORKS
2.1 CLIP MODEL
CLIP model aligns semantically related visual and textual content within shared representation space (Radford et al. 2021 Lee et al. 2025). It consists of an image encoder CLIP (ng)) (-) and text encoder CLIP n (-). The image encoder encodes an input image I into visual embedding fimg = CLIP img (.) The text encoder processes the input text T = [x1 22 ,... NNl to generate textual embedding fuu (T)).,(T). CLIP trained using contrastive learning approach such that the similarity between for and fing for matched sentence-image pair is maximized, while the similarity for unmatched pairs is minimized. The similarity defined as:
frx fing
Sfm fung)
(1)
foxt 2 fang 2
where s a scalar scaling factor. Then, the training objective CLIP formulated as:
Maximize CLIPPa
log (TT)T) + log P (7[T),
()
exp (Sim(fuu fing))
exp (Sim flng))
T)
P(I[T)
(3)
Σ exp Sim(foay fmng))
> exp (Sim(fur fime,)
where indicates the number text-image pairs the mini-batch.
2.2 TEXT-TO-IMAGE DIFFUSION MODEL
Text-to-image models generate image I that maximizes P(I[T) given textual description T. Modern text-to-image models, such as Stable Diffusion, are based on Latent Diffusion Model (LDM), which usually takes CLIP text embedding fox condition to generate an image. Our goal is to find the optimal text condition T that yields the image containing desired visual concepts.
2.3 PROMPT INVERSION
Soft Prompt Inversion Following prompt-tuning approach Li Liang (202I). Soft Prompt In- version techniques (Gal et al. 2023 Kumari al.) 2023) extend the vocabulary of the model with new special token S., which embeds the objects from the provided images. During the generation process, the continuous embedding vector of S is prepended the embeddings of the input text tokens (e.g., "A photo of S.", "A rendition of S ") and then used as an input to the LDM model. The soft prompt is nigh-dimensional vector with real elements so that it is hoe-hhmma-rredabll
Hard Prompt Inversion To mitigate the limitations of Soft Prompt Inversion, hard prompt tech- niques has been proposed (Wen al. 2024 Mahajan et al. 2024). In this scheme, the generated soft prompt is mapped to the nearest word token to produce the hard prompt. Often, the sequence of hard prompts consists of unrelated words, so that it is very difficult to identify words be modified in the desired image generation. Moreover, hard prompt inversion involves complex training process due to the discrete optimization and multi-stage pipelining.
Image Captioning One might think of an idea to generate an image directly from captions using LMM (Li et al., 2022, 2023; Alayrac et al.) 2022 Liu et al.) 2024).. This approach is conceptually simple but the generated captions lack the fine details necessary for detailed image synthesis control.
To complement the missing information of image captions, prompt generation services like CLIP- Interrogator Taaroogggorraaaee have been introduced (see Appendix A.4 for more details).
3

---

Published as a conference paper at ICLR 2025
VGD: Visually Guided Decoding (Ours)
LDM
Pourr (1)
CLIP Image Uncader
CLIP Toxt Eneeder
PANN Criie
User Input
Generated Image
figer
35
User's Textual Instruction
"Pless gonerate the pronpp
"I want to draw a riger with this drawing style"
shrough Diflsion jor tnage."
LLM Prompt
Generated Hard Prompt
Figure 4: Overview of the proposed hard prompt inversion method, VGD, which seamlessly inte- grates LLM, CLIP, and LDM for user-friendly image generation process.
GRADIENT FREE PROMPT INVERSION WITH LANGUAGE MODELS
3.1 INTERPRETABILITE DEGRADATION OF PREVIOUS HARD PROMPT INVERSION
The goal of hard prompt inversion to determine the text prompt T [ .x ...., ...,INN [X1] that max- imizes the LDM's probability P([[T) for a target image 1, with each a.! being chosen from the model's vocabulary. By applying Bayes' theorem, this objective can reformulated
P([ = P(I))(
(4)
P(T)
Since the prior probability of the image P(T) is independent of T,, finding the optimal 7' is equivalent finding T that maximizes P(T)T)/P(T). That is,
P(T
f argmax P(T)
(5)
As indicated in Eq. the inversion process maximizes P(T (a.ka, image captioning objective) while simultaneously minimizing P(T), the prior probability of text. In the P(T) minimization pro- cess, the hard prompt T might contain uncommon or awkward phrasing, resulting in a degradation of interpretability. We argue that this is why conventional hard prompt inversion techniques exhibit lower interpretability (see Section 4.3 for qualitative comparison).
3.2 VISUALLY GUIDED DECODING
Our goal is to find interpretable text prompt without training prompt embeddings, using the grudient-free approach. Inspired by the noisy channel model (Ju- rrfssy 2000 Brown al.) [199),, widely used in machine translation and speech recognition, model the prompt discovery process by integrating an LDM objective with a regularization term for language modeling. The objective is formulated
Step 1- Problem Formulation
f argmax P(TT))(TT)
(6)
Step Approximation with CLIP Score Computing P (7[T) using forward and reverse dif- fusion processes every token generation step is computationally intractable. To alleviate this computational burden, we approximate P (Z[T') using CLIP, such that P(! & P This approximation is justified by the fact that Stable Diffusion utilizes the frozen CLIP text encoder. Empirically. demonstrate that when CLIP model different from the aligned with Stable Diffusion used, the approximation no longer holds, leading to a degradation in performance (see Section 4.4 for the ablation study CL.P image encoders).
4

---

Published as a conference paper at ICLR 2025
For the prior probability of text P(T), leverage external language model (e.g., LLaMA, Mistral, GPT). Our main objective is then formulated as
argmax PCLIP (LM) (T)
where is hyperparameter.
Step 3 Token-by-Token Generation of the text probability PImM(T). That is,
We can express Eq. using a left-to-right decomposition
N II PLLM (11-)=1))1)
PCLP (MaTiN)
This decomposition facilitates boken-by-token text generation using a beam search decoding strat-
egy (Anderson et al. 2017; Post & Vilar 2018 Hu et al. 2019 Holtzman et al. 2020). Specifically,
we iteratively select the i-th token xr, that maximizes the objective in Eq. S In doing so, VGD
generates prompts that are semantically aligned with the target image (via CLIP) while ensuring linguistic coherence and interpretability (via LLM).
3.3 IMPLEMENTATION OF VISUALLY GUIDED DECODING
LLM Prompting Instead of fine-tuning LLM for each image generation task, design specific system and user prompts tailored to different tasks. Using the designed system and user prompt, L.MM is queried to generate tokens if it were creating text prompts for text-to-image models (see Appendix A.I for details).
Beam Initialization Providing LLM with image-related words the start of the generation pro- cess increases the probability of generating tokens that align with the image in subsequent steps. To do so, we first compare all text embeddings fę in CLIP vocabulary with the target image em- bedding fing. selecting the top-M tokens r f as the initial input prefix for L.M.. Empirically, we find that even M embeddings can be precomputed.
sufficient for VGD. This process is computationally efficient since all text
Beam Expansion and Pruning Since CLIP does not provide next-token probabilities, we use LLM's next-token predictions to select beam search candidates. Specifically, for beam width of K, each beam is expanded by appending the K highhst-probability next-token candidates based on PLLM(1)) [ [x|s) [+|]_) ]. The expanded set of K candidates is then pruned to retain only K beams according to Eq. 8 Unless otherwise specified, we set K 10 for the experiments.
Beam Search Termination The beam search terminates when either 1) beam expansion fails to improve the score (Eq.8) or 2) the predefined maximum prompt length is reached. The final result I't is then selected from the remaining candidates based on the highest score.
4 EXPERIMENTS
4.1 SETUP
Datasets We conduct experiments four datasets with diverse distributions: LAION- 400M (Schuhmann et al., 2021] 2022). MS COCO (Lin et al. (0014), Celeb-A (Liu et al. 2015 and Lexica.art art + Following PEZ (Wen et al. 2024), randomly sample 100 images from each dataset and evaluate prompt inversion methods across 5 runs using different random seeds. See Appendix A.3 for more details.
Baselines We compare the proposed method with the hard prompt inversion (PRZ (Wen et al,
20244), soft prompt inversion (Textual Inversion (Gal al. 2023)) LLaVA-1.5 (Liu al.) (024) generated caption, and LLaVA-1.5 combined with CLIP- Interrogorr....rr Images are generated with the Stable Diffusion 2.1-768 model across all comparisons (Podell et al. 2024).
5

---

Published as a conference paper at ICLR 2025
Table 1: Image quality (CLIP-I score) and prompt quality (BERTScore) comparison.
LAION
Caleb-A
COCO (BERTScore)
Lexicuan (BERTS
Method
#Tokens
CLIP-I
Precision
Recall
Precision
Recall
Textual Inversion
0.388
0.522
0.697
LLaVA-1.5
0.463
0.580
0.914
CLIP Interrogator
-77
0,540
0,685
0.511
0.762
0.794
0.898
0.843
0.818
0.819
0.819
32
0.573
0,718
0.546
0.769
0.831
0.879
0.854
0.835
0.793
0.813
L VGD
-77
0.544
0.785
0.800
0.810
PH2P
16
0.442
0.589
0.413
0.678
0.800
0.842
0.820
0.805
0.780
0.792
16
0.538
0.687
0.622
0.743
0.760
0.834
0.795
0.772
0.783
0.777
PEZ
32
0.530
0.685
0.619
0.745
0.736
0.830
0.780
0.752
0.784
0.768
0.584
16
0.484
0,650
0.482
0.700
0.833
0.862
0.847
0.827
0.779
0.802
32
0.493
0.670
0.506
0.735
0.818
0.868
0.842
0,816
0.786
0.801
VGD (Ours)
0.511
0.514
0,753
0.787
0.863
0.823
0.799
0.789
0.794
77
0,510
0.678
0.513
0.754
0.788
0.864
0.824
0.801
0.791
0.795
Target Image
Generated Prompt and Images
Target Image
Generated Prompt and Images
lighing: roon dees Smcd
sebral perpegi
Fiekick suatt ca the seat Jaby aokan fign en tacklu wakmn dibtad imaograngelon sat, bnd popli yolon
Pattanec, cracdid beallrdss Jem paw.aaa dOwc torwar
Sylidtma pobingmmpll pwwleopppppos.cccccc....
The meis riek sorn goltan duvie, llrrraate heartti nabogory soo wils kana fidt gurkk Be dat gat.lattta igl.inccoo cokoed
Mando
chilaaaa
Figure 5: Generated images using hard prompts produced by the proposed VGD,
Evaluation Metric We evaluate the quality of the prompt via image embedding similarity between the target image and image generated using the generated hard prompt. We call this similarity- based metric CLIP-I score. To ensure fairness, we utilize different CLIP models for generation and evaluation: CLIP-
CI.I--VTT---L VIT-H-14 for generation and larger CL.P-ViT--E for similarity evaluation. Following (Mahajan et al. 2024), we also compare the contextual similarity between the generated prompt and the ground-truth truth annotations (captions) of the target image, using BERTScore (Zhang et al. 2020) as metric.
4.2
IMAGE GENERATION WITH VGD PROMPTS
In Fig.
we qualitatively demonstrate that VGD generates realistic and diverse images without overfitting to the original target image.
Qualitative Evaluation As shown in Table the images generated by VGD the most similar to the original images, as measured by the CLIP-I score, even without using a gradient-based optimiza- tion process like PEZ. When using LLaVA-1.5 the language model, VGD achieves SOTA CLIP-I scores across all four datasets, surpassing the strong baseline that exploits an external database (LLaVA 1.5 + CLIP Interrogator) by considerable margin. Furthermore, unlike PEZ. VGD ex- hibits positive correlation between the CLIP-I score and the number of tokens, highlighting the effectiveness of VGD's loken-by-token generation process.

---

Published as a conference paper at ICLR 2025
Turget
Generated Prompt and Images
Generated Images with Fused Concept
lacss rumsing pictanofaa
colecfil joamcying theough spekling backgmand group
lovingly phonogmphad the Messed hitrgmees, fonns enubaacig snotber sunlight
borkzon: galden-hrad spproch beneuth scateredcloude
Figure 6: Multi-concept image generation with VGD. We show that the hard prompts obtained from two different images can be concatenated to fuse the semantic concepts.
Target Style
Generated Prompt
Target Style
Generated Prompt
anime di fustayy gointed virtage polliets shapp cyberp
Styles:
Decumyy Galactic
"A tiger"
"The streets of Paris"
"A rocket"
Figure 7: Decoded hard prompt for style transfer. Given several sample images with the same style, we can extract the style with a hard prompt and transfer it to other objects or scenes.
Multi-concept Generation VGD is capable of fusing multiple concepts from different images without additional effort. For each image with its distinct concept, we individually extract prompts and concatenate them to generate a single image that combines their concepts. Fig. 6 presents examples of multi-concept generation, illustrating that VGD accurately extracts the key elements from each image so that the generated image can contain multiple concepts without omitting any. We show the example of multi-coneept generation of 5 images with VGD and PEZ in Fig. 21 of the Appendix.
Style Transfer VGD can also be easily adapted to style transfer. Given several examples (typically images) that share the same style, we extract their shared style characteristics as a single hard prompt and use this prompt to upply the style to new objects scenes. Fig. 7 shows two examples of style transfer. demonstrating that VGD correctly embeds the shared style elements the prompt, which can be easily combined with new objects.
7

---

Published as a conference paper at ICLR 2025
4.3 QUALITY OF VGD PROMPTS
Quantitative Evaluation To assess the coherence of generated prompts, measure the BERTScore between the prompts generated by each method and the ground-truth -ouud captions. As shown in Table VGD achieves significant improvement in BERTScore compared to PEZ, espe- cially when using more than 64 tokens. Although LLaVA-based baselines show higher BERTScore, their CLIP-I scores are worse than those of VGD. This implies that the LLaVA captions fail to de- scribe the target image in detail, even though the text itself may be semantically similar to ground- truth captions.
Interpretability When prompt closely aligns with the image content and is interpretable by humans. it becomes easier for users to modify the image in the desired direction by making slight edits to the prompt. To evaluate the interpretability of the generated prompts, we make minimal modifications to the prompt and observe the effects. For example, we change "winter" to "spring" and see if the generated image successfully reflects the change. As illustrated in Fig, | compare VGD with previous gradient-based prompt inversion techniques, namely PEZ (hard) and Textual Inversion (soft).
The results indicate that VGD exhibits superior interpretability. When we adjust the number of objects, background elements, atmosphere (seasons), and species in the prompt, VGD successfully incorporates these changes. The relevant terms are already present the original prompt, enabling accurate image generation with the modified prompts. In contrast, the competitors struggle to pro- duce the desired outcomes. For instance, when we try to generate four wolves, both PRZ and Textual Inversion fail change the number of wolves included.
Prompt Distillation We can also utilize prompt inversion to re- duce the length of prompts while preserving their capability, a.k.a.
prompt distillation. Long prompts may contain redundant and unimportant information, especially when hand-crafted. This be-
score
CLIP CLIPscore
0.6
comes problematic, particularly when there exists a limit on the number of tokens that the model can process. Using VGD, we
0.4
generate shorter prompt that preserves the key aspects of the
0.2
original prompt. As illustrated in Fig. [0] the images generated with distilled prompts are very similar to the original image. In addition, images generated with distilled prompts do not show sig- nificant performance degradation on CLIP-I score compared the original prompt, as shown in Fig. g
0.0
0.1
0.3
Min
0.5
Median
0.7
Mox
0.9
Distillation Ratio
Figure 8: Prompt distillation Lexica.art dataset.
4.4
EFFECT OF CLIP AND LANGUAGE MODEL.
Table Comparison of VGD variants.
Table 3: Ablation on CLIP models.
BERTScore
CLIP Model
Model
#Tokens
CLIP Score
Model
CLIP Score
Precision
Recall
F1
laion/ViT-H--4 (original)
32
0.735
VGD
0.735
0.816
0.786
0.801
openai/ViT-B-33
32
0.655
only
0.487
0.773
0.791
openai/ViT-L-14
32
0.626
CLIP only
0.768
0.727
0.779
0.751
To investigate the effect of using both CLIP and LLM, we modify VGD in two ways and compare these variants on the Lexica.art. In the 'LLM only' variant, VGD selects the next token based solely on PLLM( ( without considering CLIP score, operating like typical LLM text decoding. the 'CLIP only' variant, VGD determines the next token using only the CLIP model. Table shows that the 'LLM only' variant suffers from drastic performance degradation the CLIP score because it does not consider the target image at all. On the other hand, the 'CLIP only' variant achieves the highest CLIP score but the lowest BERTScore, suggesting that the generated prompt is less coherent and harder to interpret. In comparison, the proposed VGD demonstrates strong and balanced performance the CLIP score and BERTScore.
Role of CLIP and LLM
8

---

Published as a conference paper at ICLR 2025
Target Image
VGD (Ours)
PEZ (Hard Prompt)
Textual Inversion (Soft Prompt
chomd timmiiemene jody riggs radgers
nepol wilmington idlide soaring hudkrphotogrppinn wils detriled lapland
Wolf pack enveloppings, large wolves, lighring moon
soswtoom destratic profound moon timberwolves highs adog adog guildtail thanlogiving wooslii SPPP
painting
Object Number
choms limmsizmenejjeet jody liars rigge codgers nepel wilmington wildirie soaring wolves, instepphoogmmphin cils detailed lopland timberwolves highs
Wall pock deep winter biiss etveloppings fintsdi
painting of
profound
february tlankagsiing velerinive
Background Object
chora timmsiemets jody liars repei wilmingron wildlife soaring budkerphuoogreehigg deseiled lapland secwstorn cramatic peofound sul timberwelves hight adoy guildhal arties lebruary thatiagiiiig till wonsh (P49 veterinive
Moon - Sun
Wall winter esveloppings,
lurge wolves, view. lighting lust diin;;
A paiming S. withaum
Season
Winter
clonal Limmsicmats jolly linrs riggs midgees nepal wilmingtton wildlife soaring Innkerphottgrsphing oils detriled lapland Blosam dnumatic profourd mose timberwolves highs ndog guildhall artists musch thanksgiving ing fill Wordtl
Spring
pack deep enveloppiags. sidk lighting moon shine, Sacah
pointing S
Species
Wolf Rabbit
choral timmeiemens jody lians riggs rodpers
nepal wilmington wildlife scering
Rubbit pucks Geop enoppppinss large lighting
hnuik--phetogappiin
detailed Iapland niblils
drumutic profound
paiming
hight adog guildhall artiets fahearry thanisgiving fall poosiiliApp vesertnive
Figure 9: Prompt interpretability. With the good interpretability of the prompt, we can modify them to edit the original image to the desired direction.
CLIP Model Ablation VGD leverages the same CLIP model that is used the text-to- -o--to--imgge model, Stable Diffusion 2.1. To assess the importance of this alignment. perform ablation study using two alternative CLIP models: openai/ViT-Bi /VVi----3222 and openai/ViT-I: ble shows that using misaligned CLIP model leads significant performance degradation. This
Ta-

---

Published as a conference paper at ICLR 2025
Target Image
Target Prompt
Prompt Distillation Results
(Distillation Rutio)
50 %
30 % 10 %
the cat karakl an energy drink, concept art. wlop, digital painting, trending detailed, epic composition, official media, uhd
highly
ArtStation cal drinks energy drink, concept wlop, digital painting, tr
Cat drinks energy drink concept wlop digital painting
Cat drinks energy
Cloudseage by Adam Paquette, nebala gasses background by Gene Raz Edler, fantasy magic angel concept art from deviantart Desato Giancola, Rendered in cinematic, Highly Detailed
Cloudscape rendering
Adam Paquetic, nebula background, fintasy
Cloudscape render by Adamn Paquelte, fantasy
Cloudscapp Render
magic angel
Figure 10: Prompt distillation results generated by the proposed VGD. With VGD, we can distillate the hard prompt that can still generate images similar in concept to the original.
Table 4: Ablation language models.
LLM
#Tokens
LAION
MS COCO
Celeb-A
Lexica.art
MS COCO
Lexicaart
CLIP Score
Recall
F1
Precision
Recall
0.443
0.615
0.422
0.706
0.808
0.850
0.828
0.815
0,780
0.797
Mistral-7B
0.463
0.656
0.466
0.858
0.829
0.787
0.797
64
0.497
0.680
0.476
0,751
0,783
0.858
0.818
0,791
0.789
0.790
16
0.484
0.650
0.482
0,700
0.833
0.862
0.847
0.827
0.779
0.802
LLaMA2-7B
0.868
0.842
0.786
0.801
0.511
0.678
0.514
0,753
0,787
0.863
0.823
0,799
0.789
0.794
16
0.480
0.648
0.521
0.731
0.820
0.861
0.840
0.829
0.784
0.806
LLAMA3-8B
0.506
0.661
0.540
0.752
0.805
0.864
0.833
0.817
0.790
0.803
0.515
0.685
0.543
0.769
0,769
0.858
0.811
0.796
0.794
0.795
highlights the importance of proper approximation P(T))) = (T))( When a different CLIP model used, the approximation no longer holds, leading to a drop performance.
Language Model Ablation We conduct an ablation study using three different LLMSs: Mistral- 7B (Jiang et al./ 2023), LLMM2--7B (Touvron et al. 2023). and I.LaMA3-8B (Dubey et al.) 2024). As shown Table 4 although LLaMA3-8B exhibits the best performance, the overall differences among the models are marginal. This indicates that VGD is not dependent on the specific language model used, allowing users to choose an LLM that best suits their system.
5 CONCLUSION
This paper tackles the difficulty of creating effective textual prompts for advanced text-to-image models like DALLL-E and Stable Diffusion. Traditional prompt inversion methods often lack inter- pretability and coherence, limiting their usefulness. We introduce VGD, gradient-free technique that combines LLMs with CLIP-based guidance to generate clear and semantically accurate prompts. VGD enhances prompt interpretability, generalization, and flexibility without requiring additional training. Our experiments show that VGD outperforms existing methods, producing more under- standable and contextually relevant prompts. By integrating seamlessly with various LLMs such as LLaMA2, LLaMA3, and Mistral, VGD provides an efficient and versatile solution for improving user interactions with text-to-image generative models.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea govermment(MSIT) 200--22--022-0025555555----22--- and RS-2024-00398155 The authors would like to thank Jinwoo Son for his valuable assistance in proofreading this manuscript.
REFERENCES
lean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716- 23736, 2022.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Guided open vocabulary image captioning with constrained beam search. In Proceedings of the 2017 Conference on Em- pirical Methods in Natural Language Processing, pp. 936-945, 2017.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392-18402, 2023.
Peter F Brown, Stephen A Della Pietra, Vincent J Della Pietra, and Robert L Mercer. The mathe- matics of statistical machine translation: Parameter estimation. Computational linguistics, 19(2): 263-311, 1993.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv preprint 24 2024.
Martin Nicolas Everaert, Marco Bocchio, Sami Arpa, Sabine Süsstrunk, and Radhakrishna Achanta. Vetim: Expanding the vocabulary of text-to-image models only with text. In The 34th British Machine Vision Conference (BMVC 2023). BMVA, 2023.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In The Eleventh International Conference on Learning Representations, 2023.
songwei Ge, Taesung Park, Jun- Jun-Yan Zhu, and iia-Bin Huang. Expressive text-to-image generation with rich text, In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 7545-7556, 2023.
Qin Guo and Tianwei Lin. Focus on your instruction: Fine-graine Sine-grained and multi-instruction image editing by attention modulation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6986-6996, 2024.
Yanu Hao, Zewen Chi, Li Dong, and Furu Wei. Optimizing prompts for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or. Prompt-to-promp image editing with cross-attention control. In The Eleventh International Con- ference on Learning Representations, 2023.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020.
J Edward Hu, Huda Khayrallah, Ryan Culkin, Patrick Xia, Tongfei Chen, Matt Post, and Benjamin Van Durme. Improved lexically constrained decoding for translation and monolingual rewriting. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 839-850, 2019.
11

---

Published as a conference paper at ICLR 2025
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Dong Jing, Xiaolong He, Yutian Luo, Nanyi Fei, Guoxing Yang, Wei Wei, Huiwen Zhao, and Zhiwu Lu. Fineclip: Self-distilled region-based clip for better fine- -raigga understanding, In The Thirty- eighth Annual Conference on Neural Information Processing Systems.
Daniel Jurafsky. Speech and language processing, 2000.
Donghoon Kim, Gusang Lee, Kyuhong Shim, and Byonghyo Shim. Preserving pre-trained rep- resentation space: On effectiveness of prefix-tuning for large multi-modal models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Com- putational Linguisties: EMNLP 2024, pp. 797-819, Miami, Florida, USA, November 2024. findinng---mnlp.44.. URL https: logy. wtt.////2222 fi f ndi ngs emn pp 44/
Association for Computational Linguistics. doi: 10.
Xianghao Kong, Ollie Liu, Han Li, Dani Yogatama, and Greg Ver Steeg. Interpretable diffusion via information decomposition. In The Twelfth International Conference on Learning Representa- lions, 2024.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. Proceedings of the IEEE/CVF Conference on Com- puter Vision and Paitern Recognition, pp. 1931-1941 2023.
Insu Lee, Jiseob Kim, Kyuhong Shim, and Byonghyo Shim. Learning primitive relations for com- positional zero-shot learning, arXiv preprint 25tt:25501.13300,, 2025.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficien prompt tuning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045-3059, November 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International conference on machine learning, pp. 12888-1200 PMLR, 2022.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference machine learning, pp. 19730-19742 PMLR, 2023.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. 2021.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022.
Ziwei Liu, Ping Luo. Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Shweta Mahajan, Tanzila Rahman, Kwang Moo Yi, and Leonid Sigal. Prompting hard or hardly prompting: Prompt inversion for text seext-to-imagee to-image diffusion models. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6808-6811 2024.
12

---

Published as a conference paper at ICLR 2025
Hadas Orgad, Bahjat Kawar, and Yonatan Belinkov. Editing implicit assumptions in text-to-imane diffusion models. In Proceedings of the IEEE/CVF Interaational Conference on Computer Vision, pp. 7053-7061, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal. Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching clip to count to ten. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 3170-3180, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference Learning Representations, 2024.
Matt Post and David Vilar. Fast lexically constrained decoding with dynamic beam allocation for neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Kanchana Ranasinghe, Brandon McKinzie, Sachin Ravi, Yinfci Yang, Alexander Toshev, and Jonathon Shlens. Perceptual grouping in contrastive vision- models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5571-5584 2023.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684 10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-lo-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500- 22510, 2023.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtedd fil-ffiteeed 400 million image-text pairs. arXiv preprint arXiv:2111.0211 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 25:25278-25285 2022.
Raphael Tang, Linqing Liu, Akshat Pandey, Zhiying Jiang, Gefei Yang. Karun Kumar, Pontus Stene- torp, Jimmy Lin, and Ferhan Ture. What the DAAM: Interpreting stable diffusion using cross attention. In Proceedings of the 61 Annual Meeting of the Association for Computational Lin- guistics, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288. 2023.
Andrey Voynov, Qinghao Chu, Daniel Cohen Or, and Kfir Aberman. p+: Extended textual condi- tioning in text-to-image generation. arXiv preprint arXiv:2303.09522 2023.
13

---

Published as a conference paper at ICLR 2025
Zhijie Wang, Yuheng Huang, Da Song, Lei Ma, and Tianyi Zhang. Promptcharm: text-to-imant generation through multi-modal prompting and refinement. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1-21, 2024.
Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. Advances in Neural Information Processing Systems, 36, 2024.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? arXiv preprint arXiv:2210.01956, 2022.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evalu- ating text generation with bert. In International Conference Learning Representations, 2020.
Yuechen Zhang, Shengju Qian, Bohao Peng, Shu Liu, and Jiaya Jia. Prompt highlighter: Interactive control for multi-modal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13215-13224, 2024.
Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-basee based language-ing eigguage-mmage pretraining. In Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition, pp. 16793- 16803, 2022.
14

---

Published as a conference paper at ICLR 2025
A DETAILS
A.1 PROMPT SETUP FOR LARGE LANGUAGE MODELS
We use different system and user prompts for different image generation tasks. Tables 5 and 6 show the prompts used for the experiments.
Table 5: LLM prompt setting for different image generation tasks.
System Prompt
You respectful and honest visual description generator for Stable Diffusion prompt. Answer in I sentence and do not mention anything other than the prompt. Do not mention 'description'.
User Prompt (Prompt Inversion)
Please generate the diffusion prompt on the given condition containing the objects, people, background. and the style of the image:
User Prompt (Style Transfer)
Please generate the diffusion prompt of the image style based on the given condition containing the painting style, color, and shapes of the image:
User Prompt (Prompt Distillation)
Please generate the diffusion prompt within $(max length) tokens so that you can generate same images with a given prompt: S(target prompt]
Model prompt
Answer: Sure, here is prompt for stable diffusion within S(model.max.Iengg) lengt) tokens:
Table 6: LLaVA prompt setting for different image generation tasks.
Prompt (Image Captioning)
USER: image Describe the in this image with sentence.
ASSISTANT:
Prompt (VGD)
USER: image Please generate the diffusion prompt containing the objects, people, background and the style of the image.
ASSISTANT: Sure, here is prompt for stable diffusion within m/ tokens:
A.2 HYPERPARAMETERS AND MODELS
The beam width is 10. The balancing hyperparameter a is set to 0.67. For VGD gen- eration, we used ation, we used hatp:///wip.----gggg----aaaa---- stabblleyee/sttaledddfffooo----- for text-to-image model.
For CLIP-I score evalu- We used Stable Diffusion
A.3 DATASET CHARACTERISTICS
LAION-400M (Schuhmann et al. 2021 2022) comprises 400 million diverse CLIP-filtere images.
MS COCO (Lin et al. 2014) mainly contains real-life photographs with multiple common objects,
15

---

Published as a conference paper at ICLR 2025
whereas Celeb-A (Liu et al., 2015) consists of celebrity portraits. Lexica.art contains Al-generated paintings with their prompts. While different datasets exhibit different characteristics, VGD works effectively regardless of the data type.
A.4 CLIP-INTERROGATOR'S APPROACII
Inte--ntero supplements missing details in generated image captions by extensive trial-and- error approach. Specifically, CLIP-Interrogator augments the caption with additional tokens selected from a pre-collected bank of 100K keywords (e.g., artist names, styles, mediums, movements), and compare each with the target image through CLIP model to filter out image-irrelevant keywords. While effective in generating the hard prompt, CLIP-Interroge inability to generate images for styles, artists, objects that are not registered the keyword bank, and 2) the computational burden of processing 100K phrases through CLIP encoder. These limitations highlight the challenges of relying solely on image captions for image synthesis, and underscore the need for efficient and flexible hard prompt generation techniques.
has several drawbacks: 1) an
B ADDITIONAL RELATED WORKS
B.1 PROMPT EDITING FOR TEXT-TO-IMAGE MODELS
To fully harness the capabilities of sott-to-image models, various approaches have been proposed to tailor text prompts to the user's specific intentions (Hertz et al. 2023 Brooks et al., 2023] Wang jet al. 2024) Notably, a rich text editor has been introduced, enabling users to design complex textual guidance to image generation models (Ge et al., 2023). Similarly, Prompt Highlighter, an interactive text-to-image interface utilizing multi-modal L.L,Ms, has been proposed (Zhang al. 2024). These studies underscore the necessity for user- wreenffiendly and controllable text editing methods. Our VGD can be seamlessly integrated into LLM-based user interfaces, as illustrated in Fig. [ VGD leverages the same token generation process LLMs, benefiting from the highly optimized inference process without requiring additional training.
B.2 INTERPRETABILITY OF TEXT-TO-IMAGE MODELS
As text-to-image applications become standard in the industry, interpretability studies for these mod- els have gained significant attention. For instance, research has focused on the cross-attention mech- anism identify the model's attention for each word (Tang et al.] 2023). In advance, the diffusion objective has been investigated to understand the internal behavior of text-to-image models (Kong al. 2024). Building on these findings, recent works modified the model's architecture to more faithfully reflect user's ' instructions (Orgad et al., 22033, Guo & Lin 2024). We emphasize that VGD aims to enhance the human interpretability without architectural modifications. We believe that previous works can also be also applied to VGD to further improve usability.
C ADDITIONAL EXPERIMENTAL RESULTS
C.1 GENERALIZATION OF GENERATED PROMPTS TO DIFFERENT T21 MODELS
We demonstrate how prompts generated by our method and other baseline methods behave when applied to different text-to-imags models (see Fig. 13 14 B and 16]. In this experiment, we did not generate any new prompt for the newly tested T21 model (i.e., MidJourney, DALL-EE2), meaning that we simply reused the prompts generated for SD in evaluating the effectiveness of producing high-quality images with various T2I models. In doing so, we can ensure a fair comparison and also highlight the generalizability of the prompts across models. Finally, would like to point out that this type of generalization cannot be done with soft prompt methods like Textual Inversion, as they are inherently tied to the specific model they are trained on. When compared to hard prompt inversion methods (such as PH2P and PEZ), VGD generates consistently better results in various text-to-image models, which underscores the robustness and flexibility of the proposed VGD.
16

---

Published as a conference paper at ICLR 2025
C.2 PROMPT GENERATION TIME
We further investigate the efficiency of VGD in comparison with other baseline methods, measured on single A100 80GB GPU. As shown in Table 7 the proposed VGD needs significantly less time to generate each image; 7x faster than Textual Inversion, 12x faster than PEZ, and more than 2200x faster than PH2P. This is because VGD does not require any training steps and only the decoding process is slightly modified.
Table 7: Prompt generation time of VGD, PEZ, PH2P, Textual Inversion, and CLIP Interrogator.
Method
PEZ
PH2P
Textual Inversion
CLIP Interrogator
VGD (ours)
Time (sec)
193.14
34264.74
103.45
24.85
15.33
C.3
MORE QUALITATIVE COMPARISON
As shown in Fig. 17 and 19 we conduct a qualitative comparison of image variation and style transfer performance across different methods, including PEZ. Textual Inversion, and CLIP Inter- rogator. The results demonstrate that, similar to the findings in Table I our method outperforms prior works, with CL.IP Intermogttor coming second.
While CLIP Interrogator captures objects effectively by generating captions using LLaVA (Liu et al., 2024) and predefined rich keyword bank, there are some critical weaknesses in prompts generated by the CLIP Interrogator (see Fig. 17p. First, the subsequent keywords extracted from the caption's 's key- word bank using CLIP similarity often suffer from redundancy and lack of interpretability. Second, CLIP Interrogator tends to generate prompts of full length (77 words), which limits its applicabil- ity to diverse tasks like multi-concept image generation and style transfer. Finally, as seen in the prompt example for the tiger image in Fig. 17 the quality of the generated images often decreases when using the extracted keywords, compared to images generated without attaching the keywords. In some cases, this leads to all extracted keywords being rejected entirely.
C.4 HUMAN EVALUATION
To evaluate whether the images generated by VGD are semantically aligned with the original image and to assess image quality, we conduct human preference evaluation. For this experiment, partici- pants were shown images generated for image variation and style transfer by five different methods (VGD, Textual Inversion, PEZ, PH2P, and CLIP Interrogator) and were asked to select the image most similar to the target image and style (Kumari et al. 2023) (see Fig, 12 The evaluation was conducted using 65 images randomly sampled from Lexica.art dataset. Fig. demonstrates that method consistently performed best in both image variation and style transfer tasks according to user preferences. Specifically, 60.5% and 52.6% of the participants selected VGD's results as the most similarly generated ones for image variation and most foauuctioo-following generations for style transfer tasks, respectively.
C.5
BAD CASES
While VGD generally shows decent results, our method sometimes struggles to capture fine- gra-ggained details of regional background objects in complex images. These cases occur when the image contains multiple objects (see Fig. 20). This is because VGD has difficulty in generating prompts for regional or local objects in the background. We believe this phenomenon is mainly due to the CLIP vision encoder's tendency to prioritize main objects in an image (Jing et al.) Paiss et al.) 2023
Ranasinghe al. 2023; Yuksekgonul et al. 2022 Zhong al. 2022
C.6 MULTI-CONCEPT GENERATION
We further explore the potential of VGD by combining five different concepts into single image. Fig. 21 demonstrates that the prompts generated by VGD can be concatenated to generate a complex image without missing concepts. However, generated prompts from PEZ suffers omits important details, such as the parliamentary, wolves, and the style of The Starry Night. This shows that VGD generates hard prompts that are compact and meaningful.
17

---

Published as a conference paper at ICLR 2025
2.5%
7,4%
23.3%
9.9%
+
PEZ
VGD (Ours)
60.5%
VGD
52.6%
Textual Inversion
(Ours)
Textual Inversion
29.6%
24.1%
VGD (Ours)
Textual Inversion
LLaVA CI
PEZ
PH2P
Figure 11: Human preference evaluation for image variation (left) and style transfer (right) tasks using different prompt generation methods. CI indicates CLIP Interrogator. Best viewed in color.
Original image
Style instruction
Target styles
12082
O Optin
(a) Image Variation Task
(b) Style Transfer Task
Figure 12: Webpage used for human preference evaluation.
D LIMITATION AND ETHICAL STATEMENT
D.1 LIMITATIONS
Although showcase the generalizability of VGD -e-egeeeeaedd prompts across multiple text-to- image models, our evaluations primarily focus on popular models such as Stable Diffusion and MidJoumey. However, since many other models are trained similar datasets and follow compara-
18

---

Published as a conference paper at ICLR 2025
Table 8: Prompt interpretability evaluation with Perplexity metric. performance.
Lower scores indicate better
Method
#Tokens
LAION-400M
MS COCO
Celeb-A
Lexica.art
PEZ
32
813.27
766.81
971.35
659.39
LLaVA-15
-77
102.62
80.97
67.59
76.04
PH2P
16
1274.14
947.29
892.04
1126.11
VGD
32
94.96
89.48
68.69
91.12
VGD+LLaVA
190.37
106.84
108.80
121.11
VGD+LLaVA
42,00
34.39
47.07
ble methods and architectures, we anticipate that the results would be consistent across these models as well.
D.2 ETHICS STATEMENT
The development of VGD raises important ethical considerations, particularly regarding potential misuse. While our method improves the interpretability and generalizability of text-to-image gener- ation, could also be exploited generate harmful or inappropriate content, such as deepfakes or offensive imagery. We strongly discourage such uses and emphasize that this method is intended for responsible applications creative, educational, and professional contexts.
19

---

Published as a conference paper at ICLR 2025
SD 2.1
Midjourney
Dall-E 2
SD 2.1
Midjourney
Dail-E 2
SD 2.1
Midjourney
Dull-E 2
VGD
LLav
Rejected by Dall-E safeguard
+C
SD 2.1
Midjourney
SD 2.1
Midjourney
SD 2.1
Midjourney
Dall-E
Rejected Dall-E safeguard
PEZ.
LLaV +C
Rejected by Dall-E safeguard
Midjourney
Midjourney
SD Midjourney
PEZ
Figure 13: Qualitative examples generated by Stable Diffusion, Midjourney and Dall-E 2 using prompts generated from VGD, PEZ, and LLaVA+CLIP-Imtemmiee
20

---

Published as a conference paper at ICLR 2025
SD 2.1
Midjourney
Dall-E
SD 2.1
Midjourney
Dall-E 2
SD
Midjourney
Dall-E
YGD
Rejected by Midjourney safeguard
Rejected by Dall-E safeguard
Kelli
PEZ
LLaV
+C
Figure 14: More qualitative examples generated by Stable Diffusson, Midjourney and Dall-E 2 using prompts generated from VGD, PEZ, and WWWVA-CCII-IInerroggall
21

---

Published as a conference paper at ICLR 2025
VGD
PEZ
LLaVA
VGD
PEZ
LLaVA
CI
VGD
LLaVA
CI
Figure More examples generated by MidJourney.
Target
Generated and Images
Generated Images with Fused Concept
Figure Multi-concept image generation by MidJourney.
22

---

Published as a conference paper at ICLR 2025
VGD
PEZ
TI
C
VGD
PEZ
T
LLaVA CI
Chool DOEIE
milgon
vithor by Lacorood Lewe Pay Rolent well lnd hg mow
AShr (icoper aopportiiccciooo atrenic
Worf pack day
right
mome ceweir heepet me monec Ll skil Vake
cahes,
inmve en
keretaa doy
Jeneny annmn
spcs.
lugirn g.
Albe Soowidd
VGD
PEZ
I1
LLaVA
VGD
PEZ
TI
LLaVA
Tristick, labsodfermaacchel
with a diran lie-
the cur oullyy
Mas
lightun
gosseli pá crien
choppurkndee spple
porpes
1iti se hout
evann dy
Zelaaaaa rajods pee so wend.
slwa best lend normttnn deluificc
pina allenhiphe polttoo
Abouatt
typorees brdc and ceveent
stabl unpost lodi bailin
mock de Mises
pruc yelin
walertin thecc wore, Sgan van Viciglle
click new led
pre alyyiieeee e
in mnmania sashing firtte stare anaunng pho gratke.
lauu capil and
VGD
PEZ
TI
+
VGD
PEZ
TI
LLaVA CI
encispion Sibile
Wetamlettion
cycllto nas ME3
eveties
south buily
pure smap sike
bcethrna chirke
sentical oralanos menical
sprands people speciaec
mionight alsoecia
doweward
woild
socul roaiiiggen dighttl Banui
upe galp
cenel loman fostin
VGD
PEZ
11
+
VGD
PEZ
11
LLavA +
CI
Bog
and rnd
Inndy parmo
Bersrdd love Katumal dalerty analemens lave lilal
Balkioy wrilec
tabckoue regrecatioo ou
Daa head's
Limoe ancher witting cheer cheee
oiee wamne-iblc
teckgaond,
vaterr eo saleice
two we
thens cedrrda
7th plomummttr Toeehor phons viiii:
Extere mnteii cauue claseeeeeeee ciaeeraaau
king Lasicaaa
baggy
Figure 17: Image variation comparison between VGD and baselines (generated by Stable Diffusion).
23

---

Published as a conference paper at ICLR 2025
VGD
PFZ
T1
+
VGD
PE7
TI
LLaVA
beane paom juee defos nul otpozeat lipad, anicuaallo caple gitiis terrell gres grem seds premiillos. sgrons prom
table lubli
rins batighhiiiee adtte adiitua
purpme
wetghtry Miinded
lady
lad booud Dona
arthill, seaci and Borierd crese
ville poaca dionaging for
sail mannaaeeaa feal
uue ev loaid
claua dis pert
pioigiallyy deallo
cxmpertie
fox woreh
vill
keng
lergh oend
dealted boike hamber
VGD
PEZ
LLaVA
VGD
PEZ
+
ledal the soddi odiin seilciedd
doe dideeson
disc baachai ovea
Micck
Hace hale racea of bodds corict
cresa on demame engo
citH Mepiagoe
Light of
daccy bal
andl k-sue daa
warmalogg
blac lege will
call lindl filiaer
bembhe
Figure 18: More image variation comparison between VGD and baselines (generated by Stable Diffusion).
Target Style
Target Style
Styles:
VGD
PEZ
TI
VGD
PFZ
TI
"A tiger"
"The streets of Paris"
"A rocket"
Figure 19: Style transfer comparison between VGD and baselines.
24

---

Published as a conference paper at ICLR 2025
Target Image
Generated Prompt and Images
Cardinals winter worderlard scenes with birds nostled snug agarissi buckground as they prepare for their hig day their hrihtll missing
Hears explore misty monntanns mune milload snaking through emerald pinc footsteps echoing through time. missing train
Nature photographer esptured mument: squirel twitch squirrel twilch buckground image surruunded by varioss objects 100%
2 missing bird, mushroom
Figure 20: Bad case examples of VGD.
25

---

Published as a conference paper at ICLR 2025
Target Images to Combine
VGD
PEZ
+
ateqamas legeraaninctaateinn vazclla fnsche seceas
Bos park ars dky ths, watd: do
Adel Masdon's stner aaaiing vale.
dopetasan contrnaden ganaee boketo Freas tegerepingnee
smega Heutedacooo proposess mensniiiiiuuu
belyouss Steoo sveiing
affection joyfal gutitiatice mnganficet
Wald
confastion geffcloonennicaae
sbmy riahuiizipp oratannioliaa
Vinee: Inckgoond
WamEE cxgnadileio conscian
heare
Figure 21: Five-concept image generation comparison between VGD and PEZ.
26


Reviews:
[{"review_id": "QGqwkRYkIq", "reviewer": "Reviewer_tBYJ", "summary": "In this paper, the authors tackle the problem of hard prompt inversion for text-to-image generative models, which is to find a piece of text input to these text-to-image models that can yield images that consist of similar visual concepts in some reference images. Unlike prior methods, which generate soft tokens or completely incomprehensible prompts, the authors propose to combine the language priors in LLMs and CLIP similarity objective to generate human-readable prompts. The authors provide some qualitative and quantitative experimental evidence to demonstrate their claims on the tasks of single image prompt inversion and multi-image concept combination.", "questions": "It would be great if the authors can address the weakness mentioned above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors propose a pretty creative way to conduct gradient-free text-to-image prompt inversion and incorporate language priors in the process. The qualitative results also show some obvious improvement on CLIP-I scores when used with Llava.", "weaknesses": "My main concern about this paper lies in the experiments. In general, I am not very convinced by their result that this method significantly improves upon the existing literature.\n1. The authors mainly conduct qualitative comparison with PEZ and textual inversion, and not with CLIP-Interrogator, which is very misleading given that CLIP-Interrogator is the best performing baseline based on Table 1 and it can also generate prompts that have similar human interpretability in comparison to the proposed method when used with Llava or BLIP.\n2. The authors mention PH2P in their literature review but did not use it as a baseline. From the PH2P paper it seems that they can also obtain similar human interpretability.\n3. PEZ also has a variation that incorporates language fluency objectives (Section 5 in PEZ paper). Since this is the main contribution of this paper, the authors should consider comparing it with this variation too.\n4. Authors should also consider (at least conceptually) compare with PRISM (https://arxiv.org/pdf/2403.19103), which is another prompt inversion method that uses VLM in their process and can achieve pretty good human interpretability.\n5. The main contribution of this paper is to generate human readable prompts. However, the authors fail to provide a principle and quantitative way to measure this contribution. Metrics like perplexity can be easily implemented here.\n6. The authors only compare performance on one text-to-image model and fail to demonstrate the generalizability of the inverted prompts on different text-to-image models.\n7. The qualitative comparisons provided in this paper are very limited.\n8. No limitation section or ethic statement. Given the potential malicious usage of this method (e.g. to generate inappropriate contents), I would encourage the authors to include these sections.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mhpJIJrCD3", "reviewer": "Reviewer_uCXB", "summary": "The paper presents a gradient-free method, Visually Guided Decoding, that integrates LLMs and CLIP-based guidance to generate coherent and interpretable prompts for text-to-image generation. Compared to traditional methods, VGD can create readable prompts by using LLMs and optimizing these prompts with visual cues by CLIP scores. Experiments indicate that VGD outperforms existing techniques in prompt quality and interpretability on several datasets.", "questions": "In what scenarios might the method generate inaccurate prompts?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- VGD produces coherent and human-readable prompts, facilitating user interaction and modification.\n- The training-free method allows easy integration with different LLMs, enhancing adaptability.\n- Demonstrates superior performance in generating contextually relevant prompts, as supported by both qualitative and quantitative results.", "weaknesses": "- This paper does not analyze bad cases. \n- The evaluation lacks depth, especially in semantic aspect evaluation. No human evaluation was conducted. Since image generation is a complex, semantically rich task, CLIPScore may not fully capture true image-prompt alignment, and its classification granularity is limited. Style transfer also requires human evaluation, but the paper only shows a few examples.\n- The paper evaluates only semantics without assessing image quality. There should be a discussion on whether using LLMs to create prompts could cause prompts to deviate from the training distribution, potentially lowering image quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k4wm8KvVVN", "reviewer": "Reviewer_jnTD", "summary": "This paper introduce Visually Guided Decoding (VGD), a gradient method that utilizes LLMs and CLIP-based guidance to generate coherent and semantically aligned texts. The experiment demonstrate that VGD outperforms existing prompt inversion techniques.", "questions": "1. When using different model architecture like SDXL and SD3 for image generation, how to deal with the problem mentioned in L220-222.\n\n2. I would appreciate if you could conduct more experimental analysis about comparison between VGD and previous methods.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.  VGD generates fully interpretable prompts that enhance generalizability across tasks.\n\n2. VGD is a gradient-free method which is more flexible", "weaknesses": "1. When apply to more complex open-source models with multiple text encoders like SDXL and SD3, as mentioned  in L220-222, the performance of the method would decline. What's more, when facing non-CLIP based models that utilize T5 as text encoders, the methods is quite limited.\n\n2.The current experimental analysis also appears insufficient.  While the method in the paper shows superior performance compare to previous method, it also should include the experiments about the time and cost for more comprehensive comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1j45ozLbfV", "reviewer": "Reviewer_CweT", "summary": "The paper introduces Visually Guided Decoding (VGD), a gradient-free approach that leverages large language models (LLMs) and CLIP based guidance to generate coherent and semantically aligned prompts. VGD further uses LLMs to produce human-readable prompts while employing CLIP scores to ensure alignment with user-specified visual concepts. Experiments demonstrate that VGD outperforms existing prompt inversion techniques in generating understandable and contextually relevant prompts.", "questions": "- Are baselines also evaluated across 5 runs like the proposed method?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Nice written paper\n- The proposed approach is gradient-free which can save time.\n- Results in Fig 7 are good.", "weaknesses": "- Comparisons are missing with existing relevant methods which improve T2I generation by refining prompts  such as [1] and [2].\n- The score in Table are not consistent as LLaVA 1.5 + CLIP Interrogator is outperforming baseline in many cases.\n- CLIP can overly large similarity scores as is proved by many existing works. To get a sense of fine-grained similarity, other metrics should be tried.\n- Authors have shown that when different CLIP model is used, the approximation no longer holds. This is obvious because Diffusion model uses CLIP-ViT-B/16 which is aligned with Diffusion latent space, Some other CLIP model will not be in the same space and will ultimately not give good results. While this ablation is good, however, this is not novel and not necessary.\n- Overall, the paper is good, however, there are some concerns regarding the quantitative evaluations, missing comparisons with other existing works etc.\n\n\n\n\n\nReferences\n\n[1] Hao, Yaru, et al. \"Optimizing prompts for text-to-image generation.\" NeurIPS 2024.\n\n[2] Zhan, J., Ai et al. Prompt Refinement with Image Pivot for Text-to-Image Generation. ACL 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)