Prompt Recovery for Image Generation Models: A Comparative Study of Discrete Optimizers

Authors: Joshua Nathaniel Williams, Avi Schwarzschild, Yutong He, J Zico Kolter
Primary area: datasets and benchmarks
Keywords: Discrete Optimization, Prompt Inversion, Benchmarking

Abstract:
Recovering natural language prompts for image generation models, solely based on the generated images is a difficult discrete optimization problem. In this work, we present the first head-to-head comparison of recent discrete optimization techniques for the problem of prompt inversion. Following prior work on prompt inversion, we use CLIP's (Radford et al., 2021) text-image alignment as an inexpensive proxy for the distribution of prompt-image pairs, and compare several discrete optimizers against BLIP2's image captioner (Li et al., 2024) and PRISM (He et al., 2024) in order to evaluate the quality of discretely optimized prompts across various metrics related to the quality of inverted prompts and the images that they generate. We find that while the discrete optimizers effectively minimize their objectives, CLIP similarity between the inverted prompts and the ground truth image acts as a poor proxy for the distribution of prompt-image pairs -- responses from well-trained captioners often lead to generated images that more closely resemble those produced by the original prompts. This finding highlights the need for further investigation into inexpensive methods of modeling the relationship between the prompts for generative models and their output space.

Decision: reject
Reviewer scores: 3, 3, 3, 3

Reviews and discussion:
<thread>
creation date:
1729345386217

thread_type:
official_review

<thread_object>
creator: reviewer_4nso

<content>
summary:
The paper investigates solutions for the task of prompt inversion, where the goal is to recover discrete text-to-image prompts that lead to the reconstruction of images from the same distribution as a given target image.

Specifically, they consider a set of existing discrete optimization techniques, as well as a feed-forward captioning model and an in-context learning based approach that leverages a VLM judge to iteratively refine prompts.

To evaluate these, the authors create a set of images with a fixed prompt benchmark, invert the prompts using each method, and then generate novel images using the inverted prompts. These new images are evaluated against the original generated images using standard quality metrics (FID, KID). Furthermore, CLIP-score is used to compare the new images to the original prompt set, and a text-to-text CLIP similarity metric is used to evaluate the inverted prompts.

The authors conclude that CLIP similarity between inverted prompts and the ground truth images is a poor proxy for the prompt inversion task, and that further research is needed in this direction.

strengths:
The paper’s main strength is the attempt at a more thorough evaluation and better benchmarking of methods in the prompt-inversion field. The metrics used are mostly reasonable, and are used to evaluate a range of baselines, which are clearly presented in the paper.

Overall, the paper was also fairly easy to read, the experiments easy to understand, and I believe this paper could be fairly easily replicated.

weaknesses:
Unfortunately, the paper is very limited in scope and grounds some of its choices in claims that are probably incorrect. For example:

1) 
The motivation for focusing on discrete prompts (L138) is probably wrong.
The findings of Khashabi do not seem relevant in this scenario since the text-to-image field uses soft prompting very extensively (Textual Inversion [Gal et al, ICLR 2023], Null-Text Inversion [Mokady et al, CVPR 2023], Custom Diffusion [Kumari et al, CVPR 2023], Imagic [Kawar et al, CVPR 2023], Reversion [Huang et al, 2023]). 

Similarly, the claim “As a consequence, most prompt inversion methods focus on strategies for discrete optimization within the embedding table E” is likely wrong. This is difficult to gauge, but prominent soft prompting techniques in the text-to-image setting are cited 10-20 times as often as their discrete counterparts.

2) 
The work focuses on CLIP-based inversion, where CLIP-space similarity is used to gauge the alignment between prompts and images. This is a very limiting restriction, and likely leads to much of the author’s conclusions. 

A lot of prior art does use the diffusion model to compute a loss, and the signal there is stronger and more aligned with the diffusion distribution itself (see e.g., all the papers cited in point (1) above, or the work of Mahajan et al (CVPR 2024) which outperforms PEZ across the board. 

If a range of methods is being benchmarked, it would be revealing to see how different families of approaches behave when compared with each-other, and specifically if and how the use of a diffusion loss changes the findings on CLIP-based optimizers.
This holds doubly true these days since most large-scale models are shifting away from using CLIP as the text encoder and its not clear if the findings here will even generalize to such models.

3) 
The authors demonstrate that CLIP alignment is decoupled from FID, KID improvements and conclude that CLIP is a poor proxy for the prompt inversion task (L374). However, most of the benchmarked methods use the same CLIP model for a loss and for the evaluation. I am not entirely convinced that this is this a matter of CLIP acting as a poor proxy, instead of the proposed methods simply finding solutions which are adversarial to CLIP (see e.g., EditBench [Wang et al, CVPR 2023] which arrive at an opposite conclusion regarding CLIP). When you consider AutoDAN with the prior, for example, you get improved results on the visual side, indicating that making it harder for the model to find adversarial solutions may help? 

It would be informative to see CLIP results also using a different CLIP model (e.g., OpenCLIP). This could help somewhat in disambiguating adversarialness vs a disconnect between CLIP and image alignment.

Additional concerns:

4) 
The evaluation is relatively basic. In a paper focused on this kind of benchmarking, I would have expected to see far more in-depth experiments. For example, does the performance of different approaches depend on the type of image being inverted (style reference, dense scene with lots of content, single vs multiple subjects etc.)? Do these findings generalize across different generative models? What about models not using CLIP as a text encoder?

5) 
I would have liked to see human evaluation to back up most of the core experiments in the paper (even at a smaller scale). 

6) 
There is only a single qualitative example of inverted prompts, and no examples of the reconstructed images for any of the methods. Please add some visual examples. 

I will note that my biggest concern is the extremely limited scope of the work, and I’m not sure this can be addressed in a rebuttal. However, I’m willing to discuss matters if the authors believe they can convince me otherwise.

questions:
See questions and suggestions in the weakness section.

Particularly, if the scope of the work remains just CLIP-based discrete inversion methods, I would have liked to see more extensive evaluation and insights into the specific scenarios where different methods excel or underperform. Moreover, I would have liked to see a more robust evaluation of the claims surrounding CLIP being a poor proxy (trying to control for adversarial solutions, adding human evaluations).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730457332690

thread_type:
official_review

<thread_object>
creator: reviewer_uzm8

<content>
summary:
The paper presents an empirical study of different techniques for prompt recovery in text-to-image diffusion model. The idea is to select 6 baseline methods (PRISM, Captioning, AutoDAN, PEZ, GCG) and assess their performance in recovering the input prompt for an input image. The study is conducted by sampling 10 images for each one of 100 prompts used as input in Stable Diffusion 2.1, where the prompt is reconstructed by each of the proposed methods. The quality of the reconstruction is then evaluated with FID/KID score among images of the original and reconstructed prompts, CLIP score among inverted prompts and original images, and text embedding similarity between the original and inverted prompts.

strengths:
I recognize the novelty of the proposed approach. To the best of my knowledge, this is the first work comparing different prompt inversion techniques for text-to-image models. The paper is overall well-written and easy to follow and understand.

weaknesses:
I have several concerns about this manuscript. Overall, I believe that unfortunately, the proposed study lacks the importance of the results to be at the level of ICLR. I will detail my viewpoint in separate sections.
1. My main concern is related to the impact of the proposed results on the community. In a comparative study like this, I believe the most important factor is providing the community with a new viewpoint on an important issue, potentially encouraging new research or raising new questions. I struggle to see this in the proposed manuscript. By reading the paper, it looks to me that results are just presented, without further interpretation that may give new perspectives. There is a Discussion section but it is quite rudimentary, stating that captioning is a good alternative to prompt inversion and re-assessing questions already known in the literature, properly cited.
2. I also believe that the experimental effort in this study is quite limited. There is only one diffusion model considered (SD v2.1), 1000 generated images for each method under analysis. This can be done in just a few days in any laboratory with sufficient computational power. The evaluation itself is quite limited. For instance, there is no comparison of the effect on fine-grained details of prompt inversion with respect to captioning, nor a proper analysis of the variability of results for the same prompt (variability is included in the experiments but not treated separately). For the qualitative evaluation, there is no user study allowing to trace of new conclusions following human judgment.
3. While the text is easy to follow, I believe that Section 3 of the paper can be completely removed and replaced with a paragraph, to allow for more space for experiments, which is highly needed in this case. Why describe all methods in detail when we can simply refer to the literature and quickly summarise them? The entire section feels unnecessary.

questions:
1. What new observation we can have, as a community, on different prompt inversion techniques, and why this is important?
2. How can we complement this analysis with proper results on multiple aspects of prompt inversion?
3. Why Section 3 is needed rather than other experiments?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730694752760

thread_type:
official_review

<thread_object>
creator: reviewer_uwcm

<content>
summary:
This work performs a comparative study on the different discrete optimization frameworks for hard prompt optimization. The methods compared are PEZ, PRISM, GCD, random search and captioning based models. The paper performs qualitative and quantitative analysis on different discrete optimization strategies for finding the best prompt for text-to-image generation. Experiments are performed on a set of 100 prompts from Stable diffusion and the inverted prompts along with the generated images are evaluated on the CLIP-scores, Cosine-similarity, FiD and KiD metrics.

strengths:
+ The paper is easy to read and self-contained.
+ The work performs a comparative analysis on different prompt optimization strategies for text-to-image generation across different metrics such as CLIP-scores, FiD, and KID.

weaknesses:
- The work considers Stable-diffusion 2.1 which uses a fixed pretrained CLIP-based text encoder. Therefore, the study will yield biased results towards approaches which minimize CLIP objective or have same distribution of image-prompt pairs. For example, PEZ optimizes in the CLIP space and therefore has better CLIP-scores in quantitative analysis.

- The work concludes captioning based models to yield better prompts for generating images. However, this is not necessarily true. The vocabulary and the distribution of a captioning model and the image generation model can be very different. 

- Prompt inversion directly from SD model as in Mahajan et al. 2024 is not considered. The diffusion objective with a PEZ -like optimizer would provide insights into the impact of loss in eq.1 and eq.2 of the paper.  

- For text-to-image generation, is the fluency of the prompt an important factor? 

Minor:
The formatting is different from the ICLR template (package times in latex).

questions:
- Are the results biased towards models which actually perform inversion (not captioning based) in the CLIP-space. Can an independent model be used to verify the validity of the claims? 

- As mentioned in the weaknesses, for the claim that captioning based models are better suited, the vocabulary of the pretrained text-to-image diffusion model should align with that of the captioning model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730724292949

thread_type:
official_review

<thread_object>
creator: reviewer_bept

<content>
summary:
This paper presents an analysis of existing prompt optimization methodologies, including PEZ, Greedy Coordinate Gradients, AutoDAN, Random Search, PRISM, and Captioning. It contains extensive quantitative analysis of 100 prompts and images from the Stable Diffusion model across image:image, text:image, and text:text metrics for overall perfrmance and performance throughout the optimization process (for relevant methods). In addition, it includes a discussion of a qualitative prompt example and possible concept relations between specific terms in the prompts and original image.

strengths:
* The paper is well-written, clear and easy to read with methodologies explained well and experimental set-up very clear
* The analysis contains broad coverage over six different prompt recovery methodologies 
* The provided qualitative analysis is very thorough for the given example, usefully highlighting connections between related concepts mentioned in prompts and rarity/imagniation of certain terminologies.
* The quantitative analysis covers a variety of evaluation types, including image:image, text:image, and text:text, including between original and updated images, new prompt and original image, and new prompt and original prompt.
* The work is well-motivated and highly contextualized among previous works, defining why prompt inversion techniques are useful compared to other methods.
* The work usefully adapts existing methods used in language models to allow extensibility for text-to-image generative models.

weaknesses:
1. While the provided analysis is insightful and sets the groundwork for improved methods, the work lacks thorough discussion or early experimentation of proposed solutions to the challenges observed with the existing prompt inversion methods. The contribution of the work is limited without studies or even specific proposals for improvements of existing methods contextualized by findings of the paper.
2. CLIP has known issues as a consistency metric, including bias in its representations [1,2] and issues with compositionality [3,4], that could make CLIPScore vulnerable. More robust alternatives such as DSG and VQAScore should be considered for the text:image analyses. 
3. The analysis is only used on images generated from Stable Diffusion, thus limiting its insights. The work would be strengthened with study of prompt inversion techniques for images that come from other models, e.g. DALLE.
4. The qualitative analysis, while very thorough and insightful, seems limited to only a single image. This limits its utility and the takeaways may not be extensible to other prompt styles or concepts.

[References]
1. Evaluating clip: towards characterization of broader capabilities and downstream implications
2. Vision-Language Models Performing Zero-Shot Tasks Exhibit Disparities Between Gender Groups
3. Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality
4. WHEN AND WHY VISION-LANGUAGE MODELS BEHAVE LIKE BAGS-OF-WORDS, AND WHAT TO DO ABOUT IT?

questions:
[Questions numbered according to each Weakness above]
1. The Discussion includes "details and quirks" for further investigation. Could there be more clarity about possible experimental directions to take steps towards improving the methods informed by these insights? Are there preliminary studies enabled by the existing analytical set-up that can provide initial guidance into these direction? 
2. Do quantitative analyses using CLIPScore for text:image analyses hold when extended to more robust consistency metrics such as DSG and VQAScore?
3. Should we expect these patterns to hold for other image generation models?
4. Do the patterns described in the qualitative analysis section hold for other kinds of prompts and concept?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents an empirical analysis of prompt recovery methods in text-to-image models, showcasing both qualitative and quantitative results.

The paper was reviewed by four experts in the domain who acknowledged that the paper was well written and easy to follow (BePT, uWcM, Uzm8, 4nSo), the quantitative analysis considered several evaluation types / metrics (BePT, uWcM), the work was well positioned in the literature (BePT), and the experiments were easy to understand (4nSo).

The main concerns raised by the reviewers were:
1. Missing sufficiently in-depth discussion on the findings and early experimentation of proposed solutions to the observed challenges, making the contribution rather limited (BePT, 4nSo)
2. The use of CLIPscore should be revised to increase the reliability of the observations (BePT)
3. Analysis performed on stable diffusion images only, and the model of choice relying on CLIP, thus limiting its insights and takeaways (BePT, uWcM, Uzm8, 4nSo)
4. Claims not well supported by experimental evidence -- e.g. differences in vocabulary in image captioning and generation models (uWcM)
5. Missing related works in the discussion (uWcM, 4nSo)
6. Unclear significance of the presented results (Uzm8)

During rebuttal and discussion, the authors acknowledge the relevance of some of the points raised by the reviewers and consider those to improve future iterations of their work. Unfortunately, the rebuttal does not take the opportunity to strengthen the analysis and back up claims with additional empirical evidence, as requested by the reviewers. After discussion, the reviewers unanimously recommend rejection. The AC agrees with the reviewers' assessment and recommends to reject. The AC encourages the authors to consider the feedback received to improve future iterations of their work.