Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty

Authors: Meera Hahn, Wenjun Zeng, Nithish Kannen, Rich Galt, Kartikeya Badola, Been Kim, Zi Wang
Primary area: interpretability and explainable AI
Keywords: Interpretable belief state, uncertainty estimation, information gathering, intelligent agents, question-asking under uncertainty

Abstract:
User prompts for generative AI models are often underspecified or open-ended, which may lead to sub-optimal responses. This prompt underspecification problem is particularly evident in text-to-image (T2I) generation, where users commonly struggle to articulate their precise intent. This disconnect between the user's vision and the model's interpretation often forces users to painstakingly and repeatedly refine their prompts. To address this, we propose a design for proactive T2I agents equipped with an interface to actively ask clarification questions when uncertain, and present their understanding of user intent as an interpretable **belief graph** that a user can edit. We build simple prototypes for such agents and verify their effectiveness through both human studies and automated evaluation. We observed that at least 90\% of human subjects found these agents and their belief graphs helpful for their T2I workflow. Moreover, we use a scalable automated evaluation approach using two agents, one with a ground truth image and the other tries to ask as few questions as possible to align with the ground truth. On DesignBench, a benchmark we created for artists and designers, the COCO dataset (Lin et al.,2014) and ImageInWords (Garg et al., 2024), we observed that these T2I agents were able to ask informative questions and elicit crucial information to achieve successful alignment with at least 2 times higher VQAScore (Lin et al., 2024) than the standard single-turn T2I generation.

Decision: reject
Reviewer scores: 6, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730580798019

thread_type:
official_review

<thread_object>
creator: reviewer_tosu

<content>
summary:
This paper introduces a proactive design for text-to-image (T2I) agents that enhances user interaction by prompting clarification questions when user intent is unclear. The approach enables T2I agents to create an interpretable 'belief state'—a representation of their current understanding—that users can review and adjust as necessary. By allowing edits to this belief state, the system promotes transparency and collaboration, enabling the agent to refine its responses to better match the user’s expectations. Overall, this design aims to make T2I agents more interactive, adaptable, and user-centric.

strengths:
The paper is well-written and easy to understand, presenting concepts and methodologies in a clear and accessible manner. Additionally, the results are strong, demonstrating the effectiveness and reliability of the proposed approach.

weaknesses:
However, many previous methods have proposed self-correction techniques to improve generated images, as demonstrated in works such as [1], [2], and [3]. This paper, however, does not include comparisons with these related methods. Given this omission, I question the technical contribution of the paper and am uncertain whether it would be a suitable fit for CVPR.

Previous related works have proposed iterative improvements to image quality. Therefore, it would be helpful if the authors could address the following points: 
1) What is the difference between the iterative procedure used in the proposed methods and that of related works? There is no direct comparison with all related works. 

2) It would be beneficial if the authors could compare their methods with these works and explain why the proposed method has a competitive edge. 

3) If users make errors when inputting their data, is the proposed model able to detect and correct these mistakes? 

4) Regarding reasoning ability, if a user requests advanced generation requirements, such as "a robot executes a task," is the proposed method capable of handling spatially-aware generation?

[1] Wu, Tsung-Han, et al. "Self-correcting llm-controlled diffusion models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Yang, Ling, et al. "Mastering text-to-image diffusion: Recaptioning, planning, and generating with multimodal llms." Forty-first International Conference on Machine Learning. 2024.

[3] Jiang, Dongzhi, et al. "CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matching." arXiv preprint arXiv:2404.03653 (2024).

questions:
It would be appreciated if the authors could elaborate on the technical challenges addressed, as well as the potential insights and future impact of this work in the field of computer vision.

========================= 

After reviewing the rebuttal, I have decided to keep my score unchanged.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730625787295

thread_type:
official_review

<thread_object>
creator: reviewer_jltn

<content>
summary:
This paper introduces a proactive approach to multi-turn text-to-image (T2I) generation, addressing the common problem of prompt underspecification, where users struggle to articulate their vision clearly, leading to suboptimal results. The proposed T2I agents actively engage with users by asking clarification questions and updating their understanding through an interpretable belief state that users can edit. The paper details the design of these agents, the use of a graph-based belief state to manage uncertainty, and the development of a new benchmark, DesignBench, for evaluation. Experimental results, including both automated evaluations and human studies, demonstrate that these proactive agents significantly improve image generation quality, achieving higher alignment with user intent compared to traditional single-turn T2I models.

strengths:
The paper's strength lies in its interactive approach to improving text-to-image (T2I) generation. By using proactive agents that ask clarification questions and utilize a graph-based belief state for transparency, the method addresses prompt underspecification effectively. The combination of user-centric design, comprehensive evaluation, and the introduction of the DesignBench benchmark demonstrates significant improvements over traditional T2I models.

weaknesses:
1. If the approach involves making prompt engineering more detailed than the initial prompt, it seems intuitive that the resulting image would be more accurately generated as intended. Could you elaborate on what aspects of this method are novel or distinct beyond providing more detailed prompts?
2. How do you determine which characteristics are crucial for accurately generating each image? A clearer and more logical explanation of the criteria or process used for this selection would be helpful.

questions:
1. Is there a validation process for the ground truth caption? What guarantees that it’s a well-generated caption?
2. I wonder if 15 turns are essential. The number of turns needed would likely vary for each case, so wasn’t this considered? Why 15 turns specifically?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730712899119

thread_type:
official_review

<thread_object>
creator: reviewer_zpfg

<content>
summary:
A new concept, build in a graph-based symbolic belief state for agents to understand its own uncertainty aboutpossible entities that might appear in the image.
It's a nice thing that  this article want to do, but  the solution is so boring. T2I + MLLM directly.

strengths:
The question does arise as to how to make the generation of T2I models more personalised and more responsive to the specific and potential needs of users.
1. Design and prototypes for T2I agents that adaptively ask clarification questions and present belief states.
2. An automatic evaluation pipeline with simulated users to assess the question-asking skills of T2I agents.
3. DesignBench:a new T2I agent benchmark.

weaknesses:
The solution is too simple and very uninventive. Put in a new concept and then come back with a self-explanatory statement.
Beliefstate It's a former concept, and there's nothing inherently innovative about it.

questions:
Focus on D: implementation details and show your novelty. The main text is too shallow.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730745925970

thread_type:
official_review

<thread_object>
creator: reviewer_m762

<content>
summary:
This work addresses the problem of sub-optimal image generation from text-to-image generators due to under-specified or open-ended user prompts. The work proposes building proactive T2I agents that could ask clarifications questions. Furthermore, the understanding is present as a belief state visible and editable by the user. The work also proposes a scalable automated evaluation benchmark. Experimental results suggest that 90% of the human subjects found agents and the belief states useful for the T2I workflow, and the proposed agents significantly improve the VQAScores of the generation.

strengths:
* In contrast to previous multi-turn T2I systems working on multi-turn user instructions, the proposed system asks question to the users for clarifications, which is a new form of interaction with the users orthogonal to previous works.
* This work proposes an evaluation pipeline that simulates users, which makes the evaluation of proactive agents human-free and much easier. This pipeline could also benefit the development of future agents that ask questions.
* The proposed DesignBench supplements COCO-Captions in the artistic images, which is a dataset that benchmarks the capabilities of text-to-image systems tailored to the needs of designers and artists.
* Writing: the paper has its messages clearly conveyed.

weaknesses:
* The work frames the problem as the agent updating beliefs according to a fixed world state in the user's mind. However, the user may not have a clear idea in mind (i.e., the user may not have a pre-defined world state). In contast, the user might want to get some inspirations from the system without constraints. This use case has not been considered in the system design. This limitation might affect users such as artists. This was discussed in L534-539, but no suggestions were proposed.
* The work uses LLMs in Ag2 and Ag3 without fine-tuning. However, this work does not explore trained LLMs (VLMs) with either image data or trajectories that include asking questions. This indicates that the LLM is purely exploring in text space. The exploration might be sub-optimal since exploration on text space might be different than exploration with images in mind.
* The output of the system still might not follow user's instructions. For example, one of the generated images in Fig. 1 does not have the rabbit chasing the dog.

questions:
The reviewer has a question regarding the failure case in Fig. 1:
* Is the system bottle-necked by the alignment and prompt-following capabilities of the text-to-image model or by the capabilities of the agent?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732066421187

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General reply

comment:
We thank the reviewers for time and feedback. We are encouraged that the reviewers found our work novel (m762 “new form of interaction”, zPFG “new concept” “new T2I agent benchmark”), effective (Tosu, JLtN), reliable (Tosu), significant (JLtN), well-written (Tosu) and clear (Tosu, m762), with comprehensive evaluation (JLtN) and strong results (Tosu). Reviewers m762, JLtN and zPFG also acknowledged that DesignBench and our automated evaluation are strengths of our contribution; they are both easy to use and beneficial for future agent development.

In the following, we would like to emphasize the novelty and contributions of this work. **We use “belief graph” as a shorthand for our graph-based T2I agent belief state.** Please note that the goal of this work is to enable better human-agent collaboration for T2I generation through clarification questions and controlling the belief graph. Improving T2I models themselves, such as text-image alignment, is not the goal of our work. Novelty and contributions:

1. System design of proactive T2I agents:
    * Novel human-agent interaction modalities: Prior to our work, human users typically interact with current T2I systems by giving additional instructions or refining the prompt. To the best of our knowledge, our work is the first to propose a proactive T2I agent system that is able to ask clarification questions and present its belief graph for the user to edit.
    * Novel human-agent interaction interface:  We designed a new interface to best enable the clarification and belief graph interaction modalities. We have not seen these features in any T2I, or other generative media apps that are publicly live to date, signifying to us total uniqueness. Our human studies showed that at least 85% of raters found each component of the interface useful for their workflow, for us proving that these are both novel and useful.
    * Novel design of different T2I agents that enable the proposed interaction modalities. Please see Section 4 of the paper for the full details of the design principles and construction of those T2I agent prototypes (Ag1, Ag2, Ag3).
2. Our **belief graph significantly differs from the classic belief state** in the following ways:
    * Hardcoded predicates v.s. Automatically-generated predicates: Traditionally, constructing classic symbolic belief states requires a pre-defined set of predicates such as “on(a, b)”, “is\_red(a)”, “at\_position(robot, x, y, z)” and it is non-trivial to learn new predicates that can be used and generalized to new tasks [1, 2]. Typically the pre-defined set of predicates are written by system developers and hardcoded into classic AI systems [3, 4]. 
 Our belief graph does not require any pre-defined set of predicates. Instead, we propose to construct symbolic beliefs using a sequential in-context learning (ICL) method with LLMs. This method first generates a list of entities together with their descriptions conditioned on the user description of image; then, we add each entity to the context, and let the LLM generate a list of attributes and values (this step is done in parallel across entities); and finally, we add all entities to the context and let the LLM generate relations and their attributes.  Our method can be generalized across a wide range of T2I tasks and achieve high performance (see our comprehensive results on Coco, Imageinwords, DesignBench). We will include a pseudo code for this method in the paper.
    * Application to planning v.s. T2I: To the best of our knowledge, classic symbolic belief states are mostly used for robot planning, and we are the first to use symbolic beliefs to assist T2I tasks.
    * Data structure for symbolic states / belief states – Set v.s. Graph: Because of the application to planning, a symbolic world state is usually implemented and stored as a set or list of literals (atoms or negation of atoms where atoms are instantiated predicates) [5, 6, 7] so that whenever an action is applied, the agent can apply transition by adding and deleting items in the set according to the precondition and effect of the action. 
For T2I tasks, it is more convenient to use a graph to represent the world state associated with an image, since entities and relations naturally form a set of nodes (entities) and edges (relations between entities). Each component of the graph can also have probabilities, making it easy to turn a world state into a belief state using the same data structure. Hence we represent T2I agent beliefs using graphs. The agent can directly update the graph for each transition instead of going through a set or list.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732750782032

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear reviewers and AC,

We would like to highlight some important updates we have made to the paper. Please see an anonymous video linked in the abstract for a demo of our agent.

Emphasis on novelty:
- clarified novelty in the introduction (last paragraph).
- added pseudo code for our new belief graph generation approach in Algorithm 1.
- clarified the difference between standard belief states and our belief graph in Section 4.2 and Appendix 1.

New experiments:
- We added results for validation for the ground truth caption.
- We added empirical results for ImageInWords to further enrich the evaluation.

We also further improved clarity throughout the paper according to the suggestions made by the reviewers.

-Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733260465590

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Anonymous link to video demo for further clarification

comment:
Dear reviewers and AC,

To ensure a thorough understanding, we have put together an anonymous short video to demonstrate the UI and behaviors of our proactive T2I agents: **https://youtu.be/OitADTsaqM0**.

We hope this video can further clarify the significant contribution of our work, and make this paper more accessible to all readers.

-Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
a) The paper introduces proactive T2I agents that clarify user intent via multi-turn interactions, asking questions and visualizing beliefs through an interpretable graph. Human studies and automated benchmarks show up to 2x higher VQAScore than single-turn T2I systems.

(b) Strengths:
1. Novel human-agent interaction with belief graph for transparency.
2. Comprehensive evaluation (DesignBench, COCO, ImageInWords).

(c) Weaknesses:
1. Limited exploration of fine-tuned vision-language models.
2. Bottlenecks from frozen T2I models.
(d) Decision: Reject. While the work explores a relevant issue, it fails to address modern image-editing advancements, and relies on incremental improvements.