Visually Guided Decoding: Gradient-Free Hard Prompt Inversion with Language Models

Authors: Donghoon Kim, Minji Bae, Kyuhong Shim, Byonghyo Shim
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: text-to-image, inversion, gradient free hard prompt inversion, language model guidance on latent diffusion model

Abstract:
Text-to-image generative models like DALL-E and Stable Diffusion have revolutionized visual content creation across various applications, including advertising, personalized media, and design prototyping.
However, crafting effective textual prompts to guide these models remains challenging, often requiring extensive trial and error. 
Existing prompt inversion approaches, such as soft and hard prompt techniques, are not so effective due to the limited interpretability and incoherent prompt generation. 
To address these issues, we propose Visually Guided Decoding (VGD), a gradient-free approach that leverages large language models (LLMs) and CLIP-based guidance to generate coherent and semantically aligned prompts. 
In essence, VGD utilizes the robust text generation capabilities of LLMs to produce human-readable prompts. 
Further, by employing CLIP scores to ensure alignment with user-specified visual concepts, VGD enhances the interpretability, generalization, and flexibility of prompt generation without the need for additional training. 
Our experiments demonstrate that VGD outperforms existing prompt inversion techniques in generating understandable and contextually relevant prompts, facilitating more intuitive and controllable interactions with text-to-image models.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730647472006

thread_type:
official_review

<thread_object>
creator: reviewer_cwet

<content>
summary:
The paper introduces Visually Guided Decoding (VGD), a gradient-free approach that leverages large language models (LLMs) and CLIP based guidance to generate coherent and semantically aligned prompts. VGD further uses LLMs to produce human-readable prompts while employing CLIP scores to ensure alignment with user-specified visual concepts. Experiments demonstrate that VGD outperforms existing prompt inversion techniques in generating understandable and contextually relevant prompts.

strengths:
- Nice written paper
- The proposed approach is gradient-free which can save time.
- Results in Fig 7 are good.

weaknesses:
- Comparisons are missing with existing relevant methods which improve T2I generation by refining prompts  such as [1] and [2].
- The score in Table are not consistent as LLaVA 1.5 + CLIP Interrogator is outperforming baseline in many cases.
- CLIP can overly large similarity scores as is proved by many existing works. To get a sense of fine-grained similarity, other metrics should be tried.
- Authors have shown that when different CLIP model is used, the approximation no longer holds. This is obvious because Diffusion model uses CLIP-ViT-B/16 which is aligned with Diffusion latent space, Some other CLIP model will not be in the same space and will ultimately not give good results. While this ablation is good, however, this is not novel and not necessary.
- Overall, the paper is good, however, there are some concerns regarding the quantitative evaluations, missing comparisons with other existing works etc.





References

[1] Hao, Yaru, et al. "Optimizing prompts for text-to-image generation." NeurIPS 2024.

[2] Zhan, J., Ai et al. Prompt Refinement with Image Pivot for Text-to-Image Generation. ACL 2024.

questions:
- Are baselines also evaluated across 5 runs like the proposed method?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730657723814

thread_type:
official_review

<thread_object>
creator: reviewer_jntd

<content>
summary:
This paper introduce Visually Guided Decoding (VGD), a gradient method that utilizes LLMs and CLIP-based guidance to generate coherent and semantically aligned texts. The experiment demonstrate that VGD outperforms existing prompt inversion techniques.

strengths:
1.  VGD generates fully interpretable prompts that enhance generalizability across tasks.

2. VGD is a gradient-free method which is more flexible

weaknesses:
1. When apply to more complex open-source models with multiple text encoders like SDXL and SD3, as mentioned  in L220-222, the performance of the method would decline. What's more, when facing non-CLIP based models that utilize T5 as text encoders, the methods is quite limited.

2.The current experimental analysis also appears insufficient.  While the method in the paper shows superior performance compare to previous method, it also should include the experiments about the time and cost for more comprehensive comparison.

questions:
1. When using different model architecture like SDXL and SD3 for image generation, how to deal with the problem mentioned in L220-222.

2. I would appreciate if you could conduct more experimental analysis about comparison between VGD and previous methods.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730863724303

thread_type:
official_review

<thread_object>
creator: reviewer_ucxb

<content>
summary:
The paper presents a gradient-free method, Visually Guided Decoding, that integrates LLMs and CLIP-based guidance to generate coherent and interpretable prompts for text-to-image generation. Compared to traditional methods, VGD can create readable prompts by using LLMs and optimizing these prompts with visual cues by CLIP scores. Experiments indicate that VGD outperforms existing techniques in prompt quality and interpretability on several datasets.

strengths:
- VGD produces coherent and human-readable prompts, facilitating user interaction and modification.
- The training-free method allows easy integration with different LLMs, enhancing adaptability.
- Demonstrates superior performance in generating contextually relevant prompts, as supported by both qualitative and quantitative results.

weaknesses:
- This paper does not analyze bad cases. 
- The evaluation lacks depth, especially in semantic aspect evaluation. No human evaluation was conducted. Since image generation is a complex, semantically rich task, CLIPScore may not fully capture true image-prompt alignment, and its classification granularity is limited. Style transfer also requires human evaluation, but the paper only shows a few examples.
- The paper evaluates only semantics without assessing image quality. There should be a discussion on whether using LLMs to create prompts could cause prompts to deviate from the training distribution, potentially lowering image quality.

questions:
In what scenarios might the method generate inaccurate prompts?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730914131563

thread_type:
official_review

<thread_object>
creator: reviewer_tbyj

<content>
summary:
In this paper, the authors tackle the problem of hard prompt inversion for text-to-image generative models, which is to find a piece of text input to these text-to-image models that can yield images that consist of similar visual concepts in some reference images. Unlike prior methods, which generate soft tokens or completely incomprehensible prompts, the authors propose to combine the language priors in LLMs and CLIP similarity objective to generate human-readable prompts. The authors provide some qualitative and quantitative experimental evidence to demonstrate their claims on the tasks of single image prompt inversion and multi-image concept combination.

strengths:
The authors propose a pretty creative way to conduct gradient-free text-to-image prompt inversion and incorporate language priors in the process. The qualitative results also show some obvious improvement on CLIP-I scores when used with Llava.

weaknesses:
My main concern about this paper lies in the experiments. In general, I am not very convinced by their result that this method significantly improves upon the existing literature.
1. The authors mainly conduct qualitative comparison with PEZ and textual inversion, and not with CLIP-Interrogator, which is very misleading given that CLIP-Interrogator is the best performing baseline based on Table 1 and it can also generate prompts that have similar human interpretability in comparison to the proposed method when used with Llava or BLIP.
2. The authors mention PH2P in their literature review but did not use it as a baseline. From the PH2P paper it seems that they can also obtain similar human interpretability.
3. PEZ also has a variation that incorporates language fluency objectives (Section 5 in PEZ paper). Since this is the main contribution of this paper, the authors should consider comparing it with this variation too.
4. Authors should also consider (at least conceptually) compare with PRISM (https://arxiv.org/pdf/2403.19103), which is another prompt inversion method that uses VLM in their process and can achieve pretty good human interpretability.
5. The main contribution of this paper is to generate human readable prompts. However, the authors fail to provide a principle and quantitative way to measure this contribution. Metrics like perplexity can be easily implemented here.
6. The authors only compare performance on one text-to-image model and fail to demonstrate the generalizability of the inverted prompts on different text-to-image models.
7. The qualitative comparisons provided in this paper are very limited.
8. No limitation section or ethic statement. Given the potential malicious usage of this method (e.g. to generate inappropriate contents), I would encourage the authors to include these sections.

questions:
It would be great if the authors can address the weakness mentioned above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732264998599

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Response

comment:
We sincerely appreciate the reviewers for their time and effort in reviewing our manuscript and providing valuable comments and insights. We are encouraged by the reviewers' assessment that:

- Our work addresses a crucial challenge in text-to-image generation: creating interpretable prompts for generative models in a gradient-free manner (tBYJ, uCXB, jnTD, CweT).
- Our proposed method, referred to as **Visually Guided Decoding (VGD)**, deliberately combines large language model (LLM) language priors with CLIP conditional probability to generate human-readable prompts (tBYJ, uCXB).
- VGD demonstrates qualitative and quantitative improvements, notably in creating contextually relevant and semantically aligned prompts (uCXB, jnTD, CweT).
- VGD performs comparable to or even better than existing approaches in terms of prompt quality and interpretability (tBYJ, uCXB, jnTD).
- The writing is easy to follow (uCXB), well-written, and concise (CweT).

Again, we thank the reviewers for recognizing the contributions and strengths of our work. We also appreciate the reviewers’ thoughtful comments and useful feedback, mainly on the experimental results. To address the reviewers’ suggestions, we have conducted additional experiments and provided detailed responses to each comment. We hope that our efforts will meet your expectations and further demonstrate the effectiveness of our approach (**Please see Appendix of the revised manuscript.**).
</content>
</thread_object>

</thread>
<thread>
creation date:
1753077001550

thread_type:
public_comment

<thread_object>
creator: ~donghoon_kim1

<content>
comment:
Official code implementation : https://github.com/DonghoonKim-1938/VGD
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper addresses the prompt inversion problem, with a gradient-free method that leverages LLMs and CLIP-based guidance to generate relevant prompts for text-to-image generation. The paper was reviewed by four experts in the field who acknowledged that the method proposed is creative (tBYJ) and adaptable / flexible (uCXB, jnTD), produces human-readable prompts which facilitate user interaction (uCXB, jnTD), and that the paper is well written (CweT).

The main concerns raised by the reviewers can be summarized as: 
- Experimental validation is not compelling: some baselines are missing (tBYJ, CweT); only one T2I model is considered (tBYJ, jnTD), and so the generalizability of the method remains unclear; qualitative comparisons are not performed against the best performing method (tBYJ); there is no assessment of image quality (uCXB); scores in tables appear inconsistent (CweT); it would be beneficial to consider metrics beyond CLIP similarity scores (CweT)
- It is not clear how to measure in a principled way the impact of the contribution (tBYJ)
- There is no limitations discussion (tBYJ)
- The method appears quite limited beyond CLIP-based encoder (jnTD), and many T2I models use other encoders such as T5.

During rebuttal and discussion period, the authors add results on Dalle and Midjourney, evaluate computational efficiency of the proposed approach, substantially expand qualitative comparisons, they also include a human evaluation user study, add metrics and baselines as suggested by reviewers, and discuss failure cases. There is is quite a bit of back and forth between the authors and the reviewers during the discussion period, which results in reviewers being satisfied with the updated paper and leaning unanimously towards acceptance. The AC acknowledges all the effort put by the authors during the discussion period, and agrees with the reviewers' assessment. Therefore, the AC recommends to accept.