Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty
Authors: Meera Hahn
Wenjun Zeng
Nithish Kannen
Rich Galt
Kartikeya Badola
Been Kim
Zi Wang
Venue: ICML 2025 poster (2025)
Keywords: Prompt Underspecification
Interpretability
Explainable AI
Dialog
Human-AI-Interaction
Agents
Primary area: social_aspects->accountability_transparency_and_interpretability
TL;DR: Proactive agents for multi-turn uncertainty-aware text-to-image generation with an interface to ask questions when uncertain and present agent beliefs so users can directly edit

Abstract: User prompts for generative AI models are often underspecified, leading to a misalignment between the user intent and models' understanding. As a result, users commonly have to painstakingly refine their prompts. We study this alignment problem in text-to-image (T2I) generation and propose a prototype for proactive T2I agents equipped with an interface to (1) actively ask clarification questions when uncertain, and (2) present their uncertainty about user intent as an understandable and editable belief graph. We build simple prototypes for such agents and propose a new scalable and automated evaluation approach using two agents, one with a ground truth intent (an image) while the other tries to ask as few questions as possible to align with the ground truth. We experiment over three image-text datasets: ImageInWords (Garg et al., 2024), COCO (Lin et al., 2014) and DesignBench, a benchmark we curated with strong artistic and design elements. Experiments over the three datasets demonstrate the proposed T2I agents' ability to ask informative questions and elicit crucial information to achieve successful alignment with at least 2 times higher VQAScore (Lin et al., 2024) than the standard T2I generation. Moreover, we conducted human studies and observed that at least 90% of human subjects found these agents and their belief graphs helpful for their T2I workflow, highlighting the effectiveness of our approach. Code and DesignBench can be found at https://github.com/google-deepmind/proactive_t2i_agents.

Proactive Agents for Multi-Tune Text-to-Image Generation Under Uncertainty
Meera Hahn
Wenjun Zeng'
Nithish Kannen'
Rich Galt
Kartikeya Badola
Been Kim Zi Wang"
Abstract
t Prosctive Agent
Prompt
Outputs
User prompts for generative Al models often underspecified, leading to misalignment be- tween the user intent and models' understanding. As a result, users commonly have to painstakingly refine their prompts. We study this alignment problem text-to-image (T2I) generation and propose prototype for proactive T2I agents equipped with an interface to (1) actively ask clarification questions when uncertain, and (2) present their uncertainty about user intent as an understandable and editable belief graph. We build simple prototypes for such agents and propose new scalable and automated evaluation approach using two agents, one with ground truth intent (an image) while the other tries ask as few questions possible to align with the ground truth. We experiment over three image-text datasets: Imageln Words (Garg et al., 2024), COCO (Lin et al., 2014) and DesignBench, benchmark we curated with strong artistic and design elements. Experiments over the three datasets demonstrate the proposed T21 agents' ability to ask informative questions and elicit crucial information to achieve successful alignment with at least 2 times higher VQAS- core (Lin et al., 2024) than the standard T21 generation. Moreover, we conducted human studies and observed that at least 90% of human subjects found these agents and their belief graphs helpful for their T2I workflow, highlighting the effectiveness of our approach. Code and DesignBench can be found at https: //github. com/ google deieee-deeendd/// proactive_t22 t2i _ggntt.
en image of lerge homemade breakfast. spread acroes wooden table, sunlight iluminates
Regenerote
Cierifications
1of
Which type cuisine should be pictured?
American
Europcan
Chinase
Graph
Photo style? 65% cont.
Room? 49% conf.
Table
Sunlight
Breakfast
Fork?
35% cont
Entity: Breakfast
"Chiees' (99)
100% Cont.
Aemerian 195)
Attributes
Beverage Uniknowe
Amouni
your
Custre CLoknown
Figure I. Our proactive T2I agent elarifies user prompt with questions, incorporates user feedback, and expresses its uncertainty and understanding editable belief graph.
1. Introduction
A fundamental challenge in the development of Al agents is how to foster effective and efficient multi-turn commu- nication and collaboration with human users to achieve user-defined goals, especially when faced with the common issue of vague incomplete instructions from humans.
As testbed for the communication problem. this work uses text-to-image (T2I) generation, where recent advance- ments (Baldridge et al., 2024; Betker et al., 2023; Podell et al., 2023; Yu al., 2023) have enabled the creation of stunning images from complex text descriptions. T2I ex- emplifies the challenge of moving from low-information (prompts) to high-information (images), forcing models to make implicit assumptions to resolve prompt uncertainty. Moreover, T2I offers well-defined metrics and clear prob- lem scope, enabling comprehensive automatic evaluation.
'qqual contribution 'Google DeepMind. Correspondence to: Zi Wang <wangzi @ google.com>>
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Interpretations of prompts can vary significantly across di-

---

Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty
verse populations, groups, and cultures, resulting in mis- alignment between user intents and model outputs. For example, the prompt "A breakfast plate" can evoke wide range of mental images depending on cultural background. While a typical T21 model might generate a generic break- fast plate, it is unlikely to produce a culturally appropri- ate cuisine without further information about the user's 's background and desired cuisine. These model assump- tions can lead to user frustration and dissatisfaction, re- peated prompt adjustments, while also potentially propagat- ing bias (Kannen et al., 2024; Basu et al., 2023) and causing offense (Bianchi et al., 2023).
items within the graph.
However, uncertainty alone is an insufficient trigger for clarification, as not all uncertain elements are equally conse- quential. For instance, the breakfast's cuisine type is likely more impactful on the generated images than the table's wood type. To address this, we embed importance scores alongside probabílities within the belief graph for each en- tity, attribute, and relation.
Based on both uncertainty and importance, the agent proac- tively asks clarification questions. It can, for instance, iden- tify the most uncertain attribute of an important entity (e.g. cuisine type of "breakfast"). ) An LLM is then used to formu- late a relevant question (e.g.. "Which type of cuisine should be pictured?"). To make answering easier, the agent pro- vides likely options for users select from (e.g., American, Indian, European, Chinese), though users can also type their own answers directly.
Beyond cultural contexts and differences, similar challenges arise with complex scenes. Consider the prompt "Image of rabbit near cat." Standard T2I models will likely generate an image containing both animals, however, it is unlikely the generated images will capture the specific details and spatial relationships the user envisions. These combined factors often result in a frustrating cycle of trial and error, as users repeatedly refine their prompts in an attempt to guide the model toward the desired output (Vodrahalli & Zou, 2024; Huang et al., 2024; Sun & Guo, 2023).
Conditioned on user answers and edits in the belief graph, the agent uses an LLM to update the prompt and transitions to a new belief graph by modifying the uncertainty of items clarified by the user. With the updated prompt, the agent calls an off-the-shelf T21 model to generate images.
In this paper we pursue quest for ageney in T2I generation with the aim to mitigate the issues that arise from passive T2I models. In the proposed setup, T2I agents actively engage with human users to provide a collaborative and interactive experience for image creation. We envision that these T2I agents will be able to (1) express and visualize their beliefs and uncertainty about user intents, (2) allow human users to directly control agents' beliefs beyond just text descriptions, and (3) proactively seek clarification from the human user to iteratively align their understanding with what the human user intends to generate.
The structure of our agent prototypes is highly modular, making it easy to improve each component individually, e.g., changing the strategy for asking questions, updating the belief graph construction method, and switching to better LLMs or T2I models when they become available.
To address the lack of evaluation standards for proactive T2I agents, we develop a scalable and automatic evalua- tion pipeline with a collection of metrics. The system first constructs a simulated user whose ground truth intent is a prefixed image, and translates the image to a short, under- specified text description. We then let a proactive agent and the simulated user play a multi-turn question answering game. The proactive agent aims collect as much informa- tion as possible to generate images close to the ground truth, while the simulated user provides concise answers to agent questions. We then use text and image similarity metrics to evaluate the generated image against the goal image.
At the core of these agents, we build in a graph-based sym- bolic belief state, termed the belief graph, that allows agents to express uncertainty about entities (e.g.. "breakfast", "ta- ble") that may appear in the image, attributes of entities (e.g., cuisine type) and relations between entities (e.g.. "breakfast" being on "table"). Given a prompt, we use an LLM and constrain its generation to adhere to the belief graph data structure, which includes probability estimates for entity presence and potential values for attributes and relations.
To better represent intents of diverse artists and designers (a key group of T2I users), we have created DesignBench, carefully hand-curated benchmark. This benchmark con- tains aesthetic scenes with multiple entities and interactions between them. It also provides both short and long captions that standardize simulated users in our evaluation pipeline. DesignBench features diversity between photo-realism, ani- mation and multiple styles, allowing a robust testing with the use case of artists and designers in mind.
For example, for prompt related to "homemade breakfast on wooden table" in Figure 1, the agent's belief graph might indicate a 9% probability that the cuisine is Ameri- can, while also recognizing other possibilities like Indian or Faroese. The belief graph also anticipates entities not explicitly mentioned but the user might want. Even though the user only specifies "homemade breakfast", the agent might be 35% confident that fork will be in the generated images. The expression of uncertainty enables the agent to inspire new ideas and to guide users in editing uncertain
We run both automatic and human evaluations on Design- Bench, the COCO dataset (Lin et al., 2014) and ImageIn-
2

---

Proactive Agents for Multi-Turn Text-to-Imane Image Generation Under Uncertainty
Words (Garg et al., 2024). Our proposed agents achieve at least 2 times higher VQAScore (Lin al., 2024) than traditional single-turn T21 generation within just 5 turns of interaction. Additionally we conduct a human user study in which over 90% human subjects expect proactive clarifi- cations to be helpful, about 85% find belief graphs helpful, and 58% think the question asking feature of agents could deliver value their work very soon, or immediately. Partic- ipants prefer images generated by our agents over those of single-turn T2I systems in more than 80% cases of the 550 prompt-image pairs used in the human study. This shows that our simple framework, which uses an off' the shelf model without fine-tuning, can provide significant gains.
represent an imaginary world. The image generation prob- lem can be viewed as taking a picture of the imaginary world. The world state should include all entities that are in the picture, together with their attributes and relations.
Belief states. Term "belief state" (Nilsson, 1986; Kaelbling et al., 1998) has been used to describe a distribution over states. E.g., for block a, we might have p(on table(a)) 0.5 and p(-oo_table(a)) unsure whether the block is on a table. To represent the T21 agent's 's belief on which image to generate, we need to consider the distribution over all possible "worlds" in which the picture can be taken. This distribution can be described by the probabilities that an entity appears in the picture, an attribute gets assigned a certain value, etc.
0.5, which means the agent is
We believe our approach, where agents actively gather infor- mation from the user to reduce uncertainty, will empower diverse range of users to leverage generative models for their needs and make these models safer and more responsible. By enabling direct control via interaction and understand- able belief graphs, we move beyond prompt crafting and towards a future where tools like T2I are more accessible to people without special knowledge of writing prompts.
In contrast classical belief states, our belief graphs in- corporate importance scores tailored to generative Al tasks and leverage L.Mss to automatically derive predicates from user prompts. This approach makes the belief graphs highly adaptable to new tasks, eliminating the need for the rigid, hardcoded rules in traditional AI systems.
3. Belief Graphs
2. Background
Instead of using hardcoded symbols in classic belief rep- resentations (Fikes & Nilsson, 1971) described in $2, We adopt a simple approach: leveraging L.MSs as a tool to adap- tively generate belief graphs from image descriptions of any length. While more sophisticated approaches, such as those using activations, can be explored as future work, this simple method allows for flexible belief graph generation across many types of images.
The belief graph in our work is closely related to symbolic world representations. See $B for other related works on human-computer interaction, interpretability, etc.
World states. Classical AI represents the world with sym- bols (McCarthy & Hayes, 1969; Minsky, 1974; 1988; Pa- sula et al., 2007; Kaelbling& & Lozano-Pérez, 2011). For example, in the blocks world (Ginsberg & Smith, 1988; Gupta & Nau, 1992; Alkhazraji et al.. 2020), state can be is (l)c A is_red(a) A (a) A is is_block (b) A is (be( A on a), describing that there are a red block and blue block, referred to as a and b, block a is on a table, and block is on a. Such world states include entities (c.g.. a and b), their attributes (e.g., position on_table, charac- teristies is block)) and relations (e.g.. on(b, a)) which are critical for enabling a robot to know and act in the world. The predicates like on table are pre-defined and hardcoded into robot systems.
In a belief graph, nodes are entities and edges are relations between entities. Each entity is composed of its name, de- scription, type, probability, importance score and attributes.
We have 3 types of entities: (a) explicit: entities mentioned in the prompt, (b) implicit: entities not mentioned in the prompt but likely to appear, e.g., "fork" in Figure and (c) buckground entities, such as image style, time of day, location, which play important roles in constructing the image.
In linguistics, Davidson (1965; 1967b;a) introduce logic- based formalisms of meanings of sentences. The semantics of sentence is decomposed such that no propositions can be added or removed from the set to represent the meaning of the sentence. Cho et al. (2023) propose Davidsonian Scene Graph (DSG) which rep- resents an image description as a set of atomic propositions (and corresponding questions about each proposition) to evaluate T2I alignment.
Each entity's probability reflects its likelihood of appearing in the generated image, given the user's prompt. This can be thought of as the output of classifier. Explicit and background entities typically have probabilities close to I or O, clearly indicating their presence or absence. Implicit entities, however, usually have probabilities between 0 and 1, reflecting their uncertain inclusion.
set of atomic propositions,
An entity's s importance score (normalized between 0 and 1) reflects its relevance and impact on the image. For example, in the "breakfast spread out across wooden table" prompt in Figure 1, we care more about the specific breakfast items
We borrow the same concept as symbolic world represen- tations and scene graphs, except that the agent needs to
3

---

Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty
than the type of wood the table is made of. Therefore, "breakfast" should have higher importance score than "ta- ble". It is worth noting that different people have different judgments on the relative importance of elements in an im- age. In this work, we make a simplifying assumption that the agent doesn't t consider the distribution over people's perceptions of importance scores.
4.1. Belief Graph Transitions
The agent's belief graph gets updated when the agent re- ceives new information through user feedback. This feed- back can come from agent-user dialog through user's direct manipulation of the belief graph on the agent interface (Figure 1). The transition process integrates information from the initial user prompt, the conversation history, inter- action and the previous belief. In an agent prototype, we use simple approach to implement the transition: Generate comprehensive prompt that summarizes all interactions and information gathered thus far. This merged prompt is then used to re-generate the belief, effectively incorporating the new information into refreshed representation. The implementation details can be found in $F.4.
An entity has multiple attributes, each with a name, list of malue-probabilie pairs (e.g., see the cuisine attribute of "breakfast" in Figure 1) and an importance score. An attribute's 's importance score shows its relevance to the image. For example, the cuisine of the breakfast might be more important than its exact size.
Relations are a special case of attribute that connects two entities, along with other information that an attribute can have. For each relation, the values in the list of value- probability pairs can be "part of", "under", "'vverlap", etc., describing the spatial relations between the two entities. Besides explicit relations (e.g., breakfast on table), belief graph also includes potential implicit relations. For instance, the spatial relation between "fork" and "table" might have name like "in the center of".
4.2. Asking Questions to Gather Information
We identify the following principles for an agent ask the user questions about the underspecified prompt and their intended image: (i) No Redundancy: The question should not collect information present in the history of interactions with the user. (ii) Uncertainty Reduction: The question should aim to reduce the agent's overall uncertainty about what contents should be included in the generated image, e.g., entities, attributes, spatial layout, style. (iii) Relevance: The question should be based the user prompt. (iv) Easy- to-Answer: The question should be as concise and direct as possible to ensure it is not too difficult for the user to answer. The No Redundancy and Relevance principles are self-explanatory, we detail the other principles below.
4. Proactive T2I Agent Design
We provide high-level principles and design that guide our agent how to behave and interact with users to generate de- sired images from text through multi-turn interactions. The goal of the agent to generate images that match the user's intended image as closely as possible with minimal back- and-forth, particularly in cases with underspecified prompts and the agent needs to gather information proactively. This requires a decision strategy on information gathering to trade off between the cost of interactions and the quality of generated images. The formal problem definition can be found in SC. Algorithm summarizes how the agent constructs an initial belief graph from a user prompt, takes actions (including asking questions and presenting belief graphs), incorporates user feedback during interaction, and updates the belief graph and prompt.
The Uncertainty Reduction principle aims to let the agent elicit information about various characteristies of the desired image, which the agent is unsure of.
First, the agent needs to know what characteristics of images are important. For the "Image of a rabbit near a cat" prompt, some examples include: (i) Attributes of the subjects, such as breed, size, or color, with questions like What kind of rabbit? What color the cat?: (ii) Spatial relationships between the subjects, such as proximity and relative position (Are the rabbit and cal close to each other? Are they facing each other?); (iii) Background information, such as location, style and time of day (Are they in a park or at home?); ?); and (iv) Implicit entities that might not be explicitly mentioned in the initial prompt but are relevant to the user's vision (Are there any other animals or people present?).
Algorithm Belief parsing and interaction
1: Input: Initial user prompt (p)
2: Initialization:
3: Parse entities from prompt p (E.5)
4: Parse attributes and relations from entities and (E.6,
Second, the agent needs to know its uncertainty about those characteristics. In the belief graph of the agent, the uncertainty is explicit. One strategy is to form questions about the image characteristics that the agent is most uncer- tain about. We can define an acquisition function that takes into account the entropy over entities, attributes, relations, as well as the importance scores. We discuss more in $E.2.
E.7)
5: Construct initial belief graph
6: for turn <- 1 to max_/urn do
7: Take action a < T (b.p) p)
8: Observe user feedback 0
9: Update b,p p + T(b, p. 0) (E.9)
10: end for

---

Proactive Agents for Multi-Turn Text-to- Image Generation Under Uncertainty
Third, the agent needs to update its own uncertainty once the user gives response to its question (a.k.a. transition in $4.1). Then, it can construct questions again based on its updated uncertainty estimates. This iterative clarification process allows the agent to progressively refine its under- standing of the user's intent and generate an image that more accurately reflects their desired output.
LLM is also guided with specific instructions on how con- struct ideal questions, such "The question should be as concise and direct as possible," to make questions less dif- ficult to answer, however they are not limited to multiple choice. Questions are often of the structure "What color is the rabbit in the image?"
Ag2 does not require the post-processing logic ($E.4) for the belief transition, since the LLM expected to use the conversation history to judge redundancy. Our experiments in $6.2 support this hypothesis, showing that Ag2 tends to achieve better performance than Agl with fewer turns of questions. However, Ag2 generates fewer easy-to- easy-to-answer questions than AgI, as shown in Figure 4.
The Easy- Easy-to-Answer principle aims to reduce the mental effort required from users when responding to questions. One strategy is for the agent to offer pre-defined answer choices. These choices are typically the ones the agent deems most probable. For example, What color is the cat? (a) Black (b) Brown (c) Orange (d) Other (please specify).
5.3. Ag3: Principle-prompted Agent
5. Implemented Agent Prototypes
The primary distinction between Ag2 and Ag3 is the in- formation they are prompted with. While Ag2 leverages a belief graph and merged prompt, Ag3 uses the AICQ prompt (SE.10), which exclusively relies on the conversation history and the question-asking principles from $4.2.
We propose and experiment with three agent prototypes (Agl, Ag2, and Ag3). All agents use the same belief graph construction approach in Algorithm 1. They differ in how they address the four question-asking principles (detailed in $4.2), specifically through their decision strategy т.
Ag3 tends to ask more open-ended questions such as "What else is present in the image?" and more complex questions like "Is the cake on plate or a stand, and what is its color and shape?" Figure 4, Figure and Table show that these types of questions are more difficult for users to answer but gain more information about user intent.
5.1. Agl: Heuristic Score Agent
Agl asks questions about an entity's existence, attribute val- ues, or relation types when these elements are both highly important and uncertain. It calculates a heuristic score in Equation (1) by multiplying importance scores and entropy of every belief graph element, then selects the highest- scoring question. We then prompt LLM (SE.12) to craft easy-to-answer, multiple-choice questions like, "What color of the rabbit do you have in mind? a. black, b. white, c. brown,& d. unknown, or something else?"
6. Experiments
We study the effectiveness of the agent design via both automatic evaluation, which uses a simulated user to con- verse with a T2I agent, and human study, which studies the efficacy of our framework with human subjects.
For belief graph transition, Agl re- e--genrrates the belief graph
based on the merged prompt (p in Line 9 of Algorithm 1), and applies the post-processing logic outlined in $E.4 to ensure consistency and prevent redundancy.
6.1. Automatic Evaluation
We simulate user-agent conversations using LLM self-play in Algorithm 2, where the conversation begins with goal image and a detailed ground truth prompt describing it. We use the algorithm similar to Ag2 to simulate the user, answering questions based on the ground truth prompt and the belief graph generated from the ground truth prompt. More details of the simulated user can be found in $E.3. We run each agent-user conversation for 15 turns and compute various metrics the end of each turn. Figure 2 part b shows the multi-turn set up that we use in our results.
Figure 4 shows Agl excels at constructing -o-y-to-nnswer questions. However, its hardcoded post-processing and heuristic function doesn't 't consistently prevent redundancy, largely due to potential LLM errors in parsing beliefs from prompts. For example in Figure 5, Ag1 asks whether "cake" exists, even when it was mentioned in the initial prompt. This can happen if the LLM assigns "cake" a less than 100% existence probability and a high importance score.
5.2. Ag2: Belief-prompted Agent
1.1.1. SETUPS FOR AGENTS AND BASELINE
Ag2 employs belief-based AICQa prompt in SE. in- stead of using heuristic score like Ag1, it directly prompts an L.M to produce a question. The L.LM generates each question using the belief graph, merged prompt, and conver- sation history. This comprehensive input allows it to craft highly relevant, informative and non-redundant queries. The
Baselines. We use standard T21 model as baseline, which directly generates image based prompt with- out asking any questions. We refer to this baseline as 'T2I'.
We tried prompting an LLM with only history or simple in- structions like "ask question", ", but the quality the clarification questions was noticeably worse than prompting with the principles.
5

---

Proactive Agents for Multi-Turn Text-to-Imape Image Generation Under Uncertainty
121-Only
Agl
Ag2
Ag3
Target
Original Prompe: A hand puts a coin inlo a piggy bank
agent verbel
fon
What the piggy bank made of?
00
Ceramic
agent, agent, transition state agent. se_ect acciion
Are there other coins in the image?
Agent
00
Yes, there lots coins the tahle
Ussrr
What color the piggy bank?
Black
Final State Transition
+
Final Image
a) generated outputs and target image
b) multi-turn Ag3 example real generated outputs
Figure 2. a) Each column displays the output of an agent after 15 turns the right most column shows target image, which belongs to DesignBench. b) A visualization user outputs turns 3, 10 and 15.
multi-turn evaluation set up in experiments. These are real generated outputs and simulated
Agents. We use Agl, Ag2 and Ag3 with question-asking strategies introduced in $5. Further implementation details of each agent can be found in $E.
for testing the robustness of the agents, evaluate over the Coco-Cas Coco-Captions dataset validation split (Chen et al., 2015) and Imageln Words dataset contains a diverse set realistic and cartoon images and has human annotators create dense detailed cap- tions that describe attributes and relationships between ob- jects in the image. In ImageIn Words evaluations we the long human annotation as the ground truth caption and auto- summarize the long caption with Gemini 1.5 Pro to create starting prompt po- For each image in Coco-Captions, the dataset provides multiple human generated captions which are short and describe the basic elements and the interac- tions in the image. We select the shortest of captions per image the starting prompt Pn- We obtain the ground truth caption by expanding Po via prompting Gemini 1.5 Pro with ground truth image and asking it to add more details of the attributes of the entities in the image as well as the style and image composition.
(Garg et al., 2024) dataset. Imageln-
Model Selection. We implement the agent belief parsing and interaction in Algorithm on top of the Gemini 1.5 (Gemini Team Google, 2024) using the default temperature and 32K context length. For T2I generation, gen 3 (Baldridge et al., 2024) across all baselines given its recency and the models served publicly using the Vertex 2
Ima-
capabilities. We used both
6.1.2. DATASETS.
The evaluation data consists of tuples: (I. Po, c, byt), where I represents the target image, Po is an initial prompt describ- ing only the primary elements of the scene, is ground truth caption providing detailed description of I, including spatial layout, background elements, and style, and byt is the ground truth belief graph constructed via parsing c. The initial prompt Pp is intentionally less detailed than c ne- cessitate multi-turn refinement. The evaluation framework we propose directly assesses the agent's ability to guide the user towards the target image I starting from Pn-
COCO- Cacoo-apsions and ImageIn Words datasets lack artistic and non-photorealistic imagery often desired by designers and artists. To better evaluate target for flexible use cases such by artists, we introduce and release DesignBench? novel dataset comprising 30 scenes specifically designed for this purpose. Each scene follows the (I, Po, c, byx) format described earlier. DesignBench includes a mix of human generated cartoon graphics, photorealistic yet improbable
Existing image-caption datasets primarily focus on simple scenes (Deng et al., 2009; Krizhevsky et al., 2009; Deng, 2012) or focus on very specific categories (Liu et al., 2016; Liao et al., 2022). With the aim for complex realistic images
6

---

Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty
scenes, and artistic photographic images. Initial prompt Po and ground truth caption in DesignBench are generated via prompting Gemini 1.5 Pro with ground truth image.
columns except 'human rating' show the mean and standard deviation of the similarity metric at the final turn. The blue row shows the baseline T21 method which generates an image directly from the starting prompt po-
We validate the quality of the Gemini 1.5 Pro generated captions, c, by performing Text to Image (VQA) Similarity between e and the For DesignBench the mean T2I VQA similarity between the ground truth caption and ground truth image is 0.999 with a median 1.0, and standard deviation of 4.5e-07, as expected of an accurate and well formed caption.
The results in Table I show that significant gains in performance come from using proactive multi- turliitturnnu agents. The blue row shows the simplest baseline which directly uses a T2I model and performs updates to the initial prompt pu- We see that all of the multi-turn agents far exceed the baseline T2I model on both datasets and all metrics. Ag3 (the LLM agent that does not explicitly utilize the belief graph to generate ques- tions) shows superior performance across all metrics. This confirms that the current T2I agents often produce less de- sirable images given ambiguity in prompts. In Figure 2 we see real outputs of the multi-turn set up with the Ag3 agent. Additionally, we see that the multi-turn agents generally improve in most metrics as they increase the number of interactions. Figure 3 and ?? show the 121, T21, T2T and NLL metrics, averaged across all images in the ImageIn- Words dataset, per turn for 15 turns. Interestingly we see the T2T and the T2I VQA similarity scores seem to plateau or decrease after about 10 interactions, while the 121 scores continue to increase. The NLL metric shows large perfor- mance gains of the Ag3 agent in comparison to all other methods. Figure 11 shows the T2T DSG metrics.
Multi- Turn agents show clear advantage:
6.1.3. METRICS
Each agent outputs included a generated image, a final cap- tion and a final belief graph. We evaluate each modality via their alignment to the ground truth image I, caption c and belief byt. using the following metrics.
Text-Text Similarity: We use the following metries to com- pare the ground truth caption and the merged prompt from the agent: 1) T2T- - saabeeddin-ssmillriiii computed using Gemini 1.5 Pro* and 2) DSG (Cho et al., 2024) adapted to parse text prompts into Davidsonian scene graph using the released code.
Image-Image Similarity (121): We compute cosine sim- ilarity between the groundtruth image and the generated image from the agent prompt. We use image features from DINOv2 (Oquab et al., 2024) model following prior works.
Text-Image Similarity (T2I): We compare the ground truth prompt with the generated image using VQAScore (Lin et al., 2024). We use the author released implementation of the metric and use Gemini 1.5 Pro as the underlying multi-modal large language model.
LLMs play a significant role: The best performers (Ag2 and Ag3) both query the LLM to generate a question based on contextual information such as the belief graph and con- versation history. They query the LLM to construct con- cise and clear question but don't 't impose further constraints on the question construction. Agl provides a programmatic template for how the L.LM should construct the question based on its belief graph and does not provide any conver- sation history information. Examples of dialogs and the generated questions produced by the three agents can be found in the Appendix in Figure 5. This figure demonstrates that the templated question creation sometimes leads to questions that gather minimal information in return. This can be an intrinsic limitation of hard coded question selec- tion strategy but also can be an issue of the heuristic scores we defined for question selection in Agl. In contrast, Ag2 and Ag3 generate questions that are more open-ended thus allowing the user to provide more nuanced details which in consequence enhance the agent's image knowledge.
Negative log likelihood (NLL): We construct the ground truth state of the image in the form of belief graph but with no uncertainty. We then approximately compute the NLL of the ground truth state given the belief of the agent at each turn, by assuming the independence of all entities, attributes and relations, and summing their log probabilities."
6.2. Automatic Metrics and Qualitative Results
The automatic evaluations across COCO-captions, Imageln- Words, DesignBench datasets show similar results and high- light the same patterns across the different agents.
Results in Table 1 show the I, c and byt against each agent's final generated image, merged prompt and belief graph. All
"Text embeddings are obtained from Embeddings API: https://ai.gnnggee http://aignoole.eeeee www.dd/eeemiiipppddeeeeeeeeeeeee
Question prompts with question-asking principles show advantage over those with beliefs: The Ag3 agent (which uses an LLM with question generation instructions about entity, attributes etc related to the belief) dominates across all datasets on almost every metric. Ag2 uses the belief explicitly construct questions by passing the belief into the LLM as information from which to generate the next
*This approximation does not account for potential similarities in the names of entities or attributes. This could lead to approxima- tion errors if, for example, the model confuses "Persian cat" with "Siamese cat" due their similar names. Addressing this limita- tion would require incorporating semantic similarity measures into the NLL computation.
7

---

Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty
15.00%
T21
0.75
Ag1
Ag2
Ag3
0.62
Ag3
T21 (VQA Score) Similarity (0 -1
0.70
Ag3
10.00%
Ag2
Ag2
5.00%
Image to Image Similarity
0.60
Ag1
0.65
Ag1
0.00%
(a)
Too long/complex
Unimportant attributes
Irrelevant to prompt
Unclear and ambiguous
No information gain
0.58
0.60
0.55
0.56
0.50
Very Close
Fairly Close
Medium
0.54
0.45
Fairly Different
Very Different
0.52
80.00%
0.40
60.00%
0.50
40.00%
10 Turns
15
10
15
20.00%
Turss
0.00%
Figure 3. 121 and T21 metrics per interaction tum on ImagelnWords. Out proactive agents outperform standard setups of T21.
(b)
Ag1
Ag2
Ag3
Figure 4. Human study results on the dialogues between agents and simulated users. (a) Issues cach agent's 's questions, as determined by human raters. (b) Ratings of how well the final generated image corresponds to the user prompt and dialogue.
question. When inspecting the reasoning steps of Ag2, we found that Ag2 excessively relies on importance scores in beliefs to ask questions, and if the importance scores not estimated properly, the quality of the questions decreases.
show Ag2 and Ag3 suffer from too long questions while Agl suffers from questions that do not gain new information. (2) Raters are asked to evaluate the correspondence of each image the agent-user dialog and original prompt. Approx- imately 1.5k image-dialog pairs are rated using human raters. Results in Figure 4 (b) show that for all of our agents, more than 96% of the 1.5k image-dialog pairs are rated as very close or fairly close with some differences. This high rating shows the viability of the T2I model employed by the agents, as well as the agents' ability to combine the dialogue into coherent prompt to feed the T2I model.
6.3. Human Studies Generated Images and Dialogues
We conducted human studies to provide more insights that complement the quantitative results from our automatic evaluations. See detailed design in Figure 19, Figure 20 and Figure 21. Information on human user recruitment and consent can be found in $I.
Table shows the human rater evaluations on the generated images in comparison to the ground truth image. Partici- pants asked to rank the images produced by the three proposed multi-turn agents and single-tura T2I model in terms of content and aesthetics/style against ground truth image for which the original prompt was derived and the answers to the agent's s questions were derived. The human ratings correspond to the I2I (DINO) metric. Approximately 550 amageedialoy pairs per agent are rated using? human raters. The generated unlabeled images were presented in random order and the human rater tasked with ranking the images from best to worst in terms of content correct- ness. The results are in Table 1, under the column Human Rating, showing that agentic systems are selected as the best generated image over the single-tum T2I in 80%+ of cases for both content and aesthetics/style.
6.4. Human Studies on the Agent Interface
We performed a human survey for understanding user frus- trations and validation. We gathered data from 143 partici- pants who all identified to be regular T21 users (at least once a month). Participants were presented with four hypothe- sized frustrations (prompt misinterpretation, many iterations, inconsistent generations, incorrect assumptions) and three potential mitigating features (clarifications, entity graph, relationship graph; more details in SH).
Table 4 in Appendix confirms the prevalence of hypothe- sized frustrations amongst users, with 83% experiencing occasional, frequent, or very frequent frustration due to prompt iterations, followed by 70% for misinterpretationss, 71% % for inconsistent generations, and 60% experiencing frustration due to incorrect assumptions. Most acutely 55% of participants reported frequent or very frequent frustration due to the prompt iteration frequency necessary. In Table 2, we report the mitigation features that are likely to help. Clar- ifications were reported as having the highest likelihood to help current workflows (91 could likely / very likely to
Figure 4 shows the human study results on the generated agent-simmn simulated user dialogues. (1) Per question raters are asked to mark issues each agent question contains that could pose a disturbance to the user, such as "question is too long and complex'. Approximately 8k questions per agent are rated. The results Figure 4 (a) show agentic systems have issues with their questions in 1 14% or less cases. Results
8

---

Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty
(EmbedSim) T2T 1
(DSG)
(DINO)
(VQAScore) T21
(Acsthetics) Human Eval*
(Content) Human Evalt
Dataset
Model
T2T
121
NLI
T21
0.87611003
0.590
0.5174.2
0.298
520,065 161
0.103
0.184
Ag1
0.944#.02
0.756#.1
0.627#.1
0.583
508,401 158
0.133
0.203
Coco-Captions
Ag2
0.946:2.2
0.8341.1
0.614
0.6631..5
481.722
0.253
0.316
Ag3
0,950++.02
0.900
0.658
0.775
446.568 151
0.511
0.297
T21
0.881#.02
0.681
0.515
0.371
459,905-2200
0.156
0.215
Ag1
0.9431.02
0.8161.1
0,5551.2
0.5058
449.893 196
0.201
0.235
ImageInWorte
Ag2
0.938
0.879
0.5654.2
0.570
444.523- 192
0.277
0.307
Ag3
0,942
0.912
0.588
0.662
429,464=195
0.365
0.244
T21
0.874
0.607#.1
0.544
0.353
320.890=144
0.032
0.127
Ag1
0.937+.02
0.829
0.594
0.685
195
0.175
0.190
DesignBench
Ag2
0.938+.02
0,918
0.642#:1
0.855
271.260=:2
0.159
0.317
Ag3
0.943:1.02
0.949
0.6921.1
0.95511.2
257.435-168
0.635
0.365
Table 1. Automatic evaluation results on Coco-Captions, ImageIn Words, and DesignBench. Our agents show large performance gains in all metrics over standard T2I model. The Human Eval columns display the % of times an image was ranked as closest to the ground truth image in terms of content or aesthetics. All other columns correspond to automatic evaluations using metrics described in 6.1.3.
Helpfulness Rating
Clarification Entity
Relation
Modular design. Our agent prototypes are highly modular: they use frozen T21 models to generate images based on prompts updated by the agents. This allows for scamless integration of improved off-the-shelf T2I models as they become available, boosting system performance without requiring further adaptation.
V. Likely (%)
21.7
20.3
20.3
Likely (%)
37.8
32.9
28.7
Could Help (%)
31.5
35
37.1
Unlikely (%)
5.6
7.7
7.0
V. Unlikely (%)
3.5
4.2
7.0
While T2I prompt-image alignment errors can limit the effectiveness of our proposed agents, our experiment in $F22 demonstrates that by leveraging agent-user QA pairs to evaluate image fidelity, we can filter out misaligned images across N seeds, thereby mitigating the impact of these errors. Note that the agents' T2T scores in Table I (92%+) ablate the T2I model, showing that they have already reached high alignment on the caption level, which is bound to increase with improved T2I capabilities.
Table 2. Perceived helpfulness of cach proposed feature. Shown as (% of users) out of 143 human raters.
be helpful), followed by entity graphs (88% could / likely / very likely to be helpful) and relationship graphs (86% could/ likely/ very likely to be helpful). Clarifications were expected to deliver value immediately very soon by 58%.
These findings highlight strong user desire for, and a high likelihood of success for, features that make T2I genera- tion more efficient and user- user-friendly by reducing iterations and mitigating misinterpretations. Full explanations of the hypothesized frustrations, mitigation and responses splits are in SH. All respondents were compensated for their time as per market rates, and were recruited by our vendor to ensure diversity across age, gender, and T2I usage in terms of models, frequency and purpose (work and non work).
Culturally competent models: There has been growing use of generative Al models by a broader section of society, which is only expected to continue to grow. With the varied preferences and intents across diverse user bases, this work is an example of steerable models that can adapt better to diverse user needs. Moreover, different groups of people may perceive harmfulness of content differently, so learning more about the user through clarification questions can mit- igate risks, make models safer, and pave the way more inclusive generative AI systems.
7. Discussion and Conclusion
This work introduces a design for agents that assist users in generating images through an interactive process of proac- tive question asking and belief graph refinement. By dy- namically updating its understanding of the user's intent, the agent facilitates a collaborative approach to image gen- cration with fewer user frustrations. Moreover, presenting the agent's 's belief graph can be a generalizable method for Al transparency, which is an important factor given the increasing complexity of modern Al models.
Future work. It would be interesting to explore generating images directly from belief graphs and fine-tuning VLMs on image-text interleaved multi-turn trajectory data. This may require a) collecting data such as gold-standard trajectories or annotations the quality of trajectories of human-agent conversations and b) approaches to fine-tune the model on multi-ture trajectories of images and text, which can poten- tially improve the performance of the agent.
9

---

Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty
Impact Statement
image generation with better captions. Computer Sci- ence. https://edn. openai. com/papers/dallle-3, pdf. 2(3):
Our proposed T2I agents are equipped with better tools (be- lief graphs) for interpretability and controllability, Present- ing the agent's belief graph can be generalizable method for Al transparency, which is an important factor given the increasing complexity of modern Al models.
8, 2023.
Bianchi, F., Kalluri, P., Durmus, E., Ladhak, F,, Cheng, M., Nozza, D., Hashimoto, T., Jurafsky. D., Zou, J., and Caliskan, A. Easily accessible text-to-image gen- eration amplifies demographic stereotypes large scale, In 2023 ACM Conference Faimess, Accountability, and Transparency, FAccT '23, pp. 1493-1504, ACM, June 2023. doi: 10.11445/5593301333540000 URL http: / //dx.doo org/ / 1 1111/11533/111 359499999931339399.9. 3594095.
By asking clarification questions, our proposed agents may enable a more customizable and personalized content cre- ation experience. Because different groups of people may perceive harmfulness of contents differently, learning more about the user through clarification questions can potentially mitigate risks of generating contents that can be offensive to each specific user.
Cai, C. Reif, E., Hegde, N., Hipp. J., Kim, B., Smilkov, D., Wattenberg, M., Viegas, F., Corrado, G. S., Stumpe, M. C.,, et al. Human-centered tools for coping with imper- feet algorithms during medical pecision-making In ACM Conference on Human Factors in Computing Systems (CHI), 2019.
Acknowledgements
We would like thank Jason Baldridge and Zoubin Ghahra- mani for insightful discussions on multi-turn T2I and belief states, Mahima Pushkarna for the help and consultation on user study. We would also like to thank Richard Song, Noah Fiedel, Susanna Ricco and anonymous reviewers for feedback on the paper.
Chaloner, K. and Verdinelli, Bayesian experimental de- sign: review. Statistical science, pp. 273-304, 1995.
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollár, P., and Zitnick, C. L. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: New::000550055 2015.
References
Alkhazraji, Y., Frorath, M.. Grützner, M., Helmert, M., Liebetraut, T., Mattmüller, R., Ortlieb, M., Seipp, J., Springenberg, T., Stahl, P. and Wülfing, Pyperplan. https: /doi. org/10. 5281/ aarodo.370081 9, 2020. URL https://ddo org/10. 5281/ /et.// 3700819,
Chen, Y., Zhong, R., Ri, N., Zhao, C., He, H., Steinhardt, J., Yu, Z.,, and McKeown, K. Do models explain themselves? counterfactual simulatability of natural language expla- nations, 2023. URL https: www.i.iivoogoo..ooo..o.oooooo abs/
Chen, Y., Wu, A., DePodesta, T., Yeh, C., Li, K., Marin, N. C., Patel, O., Riecke, J., Raval, S., Seow, O., et De- signing a dashboard for transparency and control of con- versational ai. arXiv preprint 2rXiv:2406.07882, 2024.
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P. Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. Guidelines for human-ai interaction. In ACM Conference on Human Factors in Computing Systems (CHI), 2019.
Cho, J. Hu, Y., Garg, R., Anderson, P., Krishna, R., Baldridge, J., Bansal, M., Pont-Tuset, J., and Wang, S. Davidsonian scene graph: Improving reliability in fine theeggriied evaluation for text-image generation. arXiv preprint arXiv:2310 18235, 2023.
Auer, P. Using confidence bounds for exploitation- exploration tradeoffs. Journal of Machine Learning Re- search (JMIR), 3:397-422, 2002.
Baldridge, J., Bauer, J., Bhutani, M., Brichtova, N., Bun- ner, A., Chan, K., Chen, Y., Dieleman, S., Du, Y., Faton-Rosen, Z., et al. Imagen 3. arXiv preprint arXiv:2408.07700) 2024.
Cho, J., Hu, Y., Garg, R., Anderson, P., Krishna, R., Baldridge, J., Bansal, M., Pont-Tuset, J., and Wang, S. Davidsonian scene graph: Improving reliability in fine- grained evaluation for text-to-image generation, 2024. URL https::tp//......... //arxiv//xvvvv/// .wwvvoorrgg .rgg /agg//bb//// 221 00.10..20 18235.
Basu, A., Babu, R. V., and Pruthi, D. Inspecting the ge- ographical representativenes of images from text-to- image models, 2023. URL httpthttp////c...t. htttps///axxiv.rkt
Cohn, D. A., Ghahramani, Z., and Jordan, M. I. Active learning with statistical models. Journal or Artificial Intelligence Research (JAIR), 4:129-145, 145, 1996.
abs/2305. 11080.
Betker, J., Goh, G., Jing, L., Brooks, T., Wang, J., Li, L., Ouyang, L.,, Zhuang, J., Lee, J., Guo, Y., et al. Improving
Datta, S., Ku, A., Ramachandran, D., and Anderson, P. Prompt expansion for adaptive text-to-image generation.
10

---

Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty
In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Pro- ceedings of the 62nd Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 3449-3476 Bangkok, Thailand, August 2024. As- sociation for Computational Linguisties. URL https: //aclanthology org/ 2024. acl- -ong.. 189.
Gupta, N. and Nau, D. S. On the complexity of blocks- world planning. Artificial Intelligence, 56(2-3):223-25 1992.
Hall, M., Ross, C., Williams, A., Carion, N., Drozdzal, M., and Soriano, A. R. Dig in: Evaluating disparities in image generations with indicators for geographic diversity, 2024. URL https::ttp.........t. wwwiv.vvoooo org abs 2308 061 98.
Davidson, D. Theories meaning and learnable languages. In Bar-Hillel, Y. (ed.), Proceedings of the 1964 Interna- fional Congress for Logic, Methodology, and Philosophy of Science, pp. 383-394, North-Holland Publishing, 1965.
Hennig. P. and Schuler, C. Entropy search for information- efficient global optimization. Journal of Machine Learn- ing Reseurch (JMLR), 13: 13:1809-1837, 2012.
Davidson, D. Truth and meaning. In Philosophy, Language, and Artificial Intelligence: Resources for Processing Nat- ural Language, pp. 93-111. Springer, 1967a.
Hessel, J., Holtzman, A., Forbes, M., Bras, R. L.,, and Choi, Y. Clipscore: A reference-free evaluation metric for im- age captioning. 2022.
Davidson, D. The logical form of action sentences. The Logic of Decision and Action, 1967b.
Höök, K. Steps take before intelligent user interfaces become real. Interacting with computers, 12(4):409-427 2000.
Deng, J., Dong, W., Socher, R., Li, L.-J, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 248-255 IEEE, 2009.
Houlsby, N., Huszár, F., Ghahramani, Z., and Lengyel, M. Bayesian active learning for classification and preference learning, arXiv preprint arXiv: 2011.
Deng, L. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 29(6):141-142, 2012.
Hu, Y., Liu, B., Kasai, J., Wang, Y., Ostendorf, M., Krishna, R., and Smith, N. A. TIFA: Accurate and interpretable text-to-image faithfulness evaluation with question an- swering. In International Conference on Computer Vision (ICCV), 2023.
Fikes, and Nilsson, N. J. Strips: new approach to
the application of theorem proving to problem solving. Artificial intelligence, 2(3-4):189 1971.
Huang, M., Long, Y., Deng, X., Chu, R., Xiong, J., Liang, X., Cheng, H., Lu, Q., and Liu, W. Dialoggen: Multi- modal interactive dialogue system for multi-turn text- to-image generation. arXiv preprint arXiv:2403.08855 2024.
Gal, Y., Islam, R., and Ghahramani, Z. Deep bayesian active learning with image data. In International Conference on Machine Learning (ICML), pp. 1183-1192. PMLR, 2017.
Hutchinson, B., Baldridge, J., and Prabhakaran, V. Under- specification in scene descrippioonttodepicciio tasks. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume I: Long Papers), 2022.
Garg, R., Burns, A., Ayan, B. K., Bitton, Y., Montgomery, C.. Onoe, Y., Bunner, A., Krishna, R., Baldridge, J., and Soricut, R. Imageinwords: Unlocking hyper-detailed im- age descriptions, 2024. URL https::///// /ttp:ii/////vvo/eaeeaeeee org/ abs/ 2405. 02793.
Garnett, R. Bayesian oplimization. Cambridge University Press, 2023.
Kaelbling, L. P. and Lozano-Pérez, T. Hierarchical task and motion planning in the now. In IEEE International Conference on Robotics and Automation (ICRA), 201
Gemini Team Google. Gemini 1.5: Unlocking multimodal understanding across millions of tokens context, 2024.
Kaelbling, L. P., Littman, M. L., and Moore, A. W. Re- inforcement learning: A survey. Journal or Artificial Intelligence Research (JAIR), 4:237-285, 1996.
URL https:////// aaraiiivvv.... org/ wbw.///////aaaa.ae..oooeaeeeee/a aba/a/ss///40 05530.
Ghavamzadeh, M., Mannor, S., Pineau, J., Tamar, A., al.
Bayesian reinforcement learning: A survey. Foundations and Trends in Machine Learning, 8(5-6):359-48,, 2015.
Kaelbling, L. P., Littman, M. L., and Cassandra, A. R. Plan- ning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1-2)999-134 1998.
Ginsberg, M. L. and Smith, D. E. Reasoning about action A possible worlds approach. Artificial Intelligence, 35 (2):165-195, 1988.
Kannen, N., Ahmad, A., Andrectto, M., Prabhakaran, V.,
Prabhu, U., Dieng, A. B., Bhattacharyya, P., and Dave,

---

Proactive Agents for Multi-Turn Text- Generation Under Uncertainty
S. Beyond aesthetics: Cultural competence in text-to- image models, 2024. URL httpp::/arxiiv..orr
McCarthy, J. and Hayes, P.J. Some philosophical problems from the standpoint of artificial intelligence. In Meltzer, B. and Michie, D. (eds.), Machine Intelligence 4, pp. 463-502. Edinburgh University Press, 1969. reprinted in McC90.
abs/2407. 06863.
Kim, S. S., Watkins, E. A., Russakovsky, O., Fong. R., and
A. "help me help the ai": Un- derstanding how explainability can support human-ai ai in- teraction. In ACM Conference on Human Factors in Computing Systems (CHI), 2023.
Mehrabi, N., Goyal, P., Verma, A., Dhamala, J., Kumar, V., Hu, Q., Chang, K.-W., Zemel, R., Galstyan, A., and Gupta, R. Is the elephant flying? resolving ambigui- ties in text-to-image generative models. arXiv preprint arXiv:2211.12550: 2022.
Kirk, R. F. Experimental design. The SAGE Handbook of Quantitative Methods in Psychology, pp. 23-45, 2009.
Minsky, M. A framework for representing knowl- edge. Technical report, A.I. Laboratory, Mas- sachusetts Institute of Technology, 1974. URL
Kirstain, Y., Polyak, A., Singer, U., Matiana, S., Penna,
J., and Levy, O. Pick-a-pic: An open dataset of user preferences for text-to-image generation, 2023.
attps://dapace.nnt mit. hetptt/tt/bbbbtteaeaaaeeee handle/1121.116609 /AIM- 306.. pdf..
Krizhevsky, A., Hinton, G., et Learning multiple layers of features from tiny images. 2009.
Minsky, M. The Society of Mind. Simon and Schuster, 1988.
Kushner, H. A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise. Journal of Fluids Engineering, 86(1):97-106, 1964.
Močkus, On Bayesian methods for seeking the extremum. In Optimization Techniques IFIP Technical Conference, 1974.
Moore, R. C. A formal theory of knowledge and action. Formal theories of the commonsense world, pp. 319-358, 1985.
Lanham, T., Chen, A., Radhakrishnan, A., Steiner, B., Deni- son, C., Hermande,, D., Li, D., Durmus, E., Hubinger, E., Kernion, J., Lukošiâté, K., Nguyen, K., Cheng, N., Joseph, N., Schiefer, N., Rausch, O., Larson, R., McCan- dlish, S., Kundu, S., Kadavath, S., Yang. S., Henighan, T., Maxwell, T., Telleen-Lawton, T., Hume, T., Hatfield- Dodds, Z., Kaplan, J., Brauner, J., Bowman, S. R., and Perez, E. Mcasuring faithfulness in offin-oo--hhught reasoning, 2023. URL /satps://arxiv... org/ abs/ 2307.1376 13702.
Nacem, M. F., Oh, S. J., Uh Y., Choi, Y., and Yoo, J. Re- liable fidelity and diversity metries for generative mod- els, 2020. URL htttps://arxiv.ot wbsp//////bbs.../// 09797.
Nilsson, J. Probabilistic logic. Artificial Intelligence, 28 (1):71-88 877, 1986.
Nilsson, N. J. The Quest for Artificial Intelligence. Cam- bridge University Press, 2009.
Liao, P., Li, X., Liu, X., and Keutzer, K. The artbench dataset: Benchmarking generative models with artworks. arXiv preprint arXiv:2206.1 1404, 2022.
Norman, D. A. How might people interact with agents. Communications of the ACM, 37(7))68-71, 1994.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D.,. Dollár, P., and Zitnick, C. L. Microsoft COCO: Common objects in context. Europ. Confer- ence on Computer Vision (ECCV), 2014.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F,, El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.-Y., Li, S.-W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jegou, H., Mairal, J., Labatut, P., Joulin, A., and Bojanowski, P. Dinov2: Learning robust visual features without supervision, 2024. URL https: / aaaaaaivvvvrr... www.wx.www...///e.//...
Lin, Z., Pathak, D., Li, B., Li, J., Xia, X., Neubig, G., Zhang, P., and Ramanan, D. Evaluating text-to-visual generation with image-to-text generation, 2024. URL https: bttps://arxe /atp///aavvvv//aa///a/eeeeaa.av org/ abs/2404 01291.
07193.
Liu, Z., Luo, P., Qiu, S., Wang, X., and Tang, X. Deepfash- ion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pp. 1096-1104, 2016.
Pasula, H. M., Zettlemoyer, L. S., and Kaelbling, L. Learn ing symbolic models of stochastic domains. Journal Artificial Intelligence Research (JAIR), pp. 309-352, 2007.
Podell, D.,, English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffiusion models for high- resolution image synthesis. arXiv preprint arXiv:2307.0/955 2023.
Lu, Y. Yang, X., Li, X., Wang, X. E., and Wang, W. Y. Limscore: Unveiling the power large language models in text-to-image synthesis evaluation, 2023.
12

---

Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty
Ren, P., Xiao, Y., Chang, X., Huang, P.-Y., Li, Z., Gupta, B. B., Chen, X., and Wang, X. A survey of deep active learning. ACM computing surveys (CSUR), 54(9):1-40, 2021.
Wang, Z., Ku, A., Baldridge, J., Griffiths, T., and Kim, B. Gaussian process probes (GPP) for uncertainty- probing, In Advances in Neural Information Processing Systems (NeurIPS), 2024c.
Russell, S. J. and Norvig, P. Artificial intelligence: A mod- ern approuch. Pearson, 2016.
Wei, F., Zeng, W., Li, Z., Yin, D., Duan, L., and Li, W. Powerful and flexible: Personalized text-to-image generation via reinforcement learning. arXiv preprint arXiv:2407.066/1 2024.
Settles, B. Active learning literature survey. Computer
Sciences Technical Report 1648, University of Wisconsin- Madison, 2009.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F,, Chi, E., Le, Q. and Zhou, D. Chain-of- thought prompting elicits reasoning in large language models, 2023a. URL https: http://axxvvv.rro 2201.11903
Srinivas, N., Krause, A., Kakade, S. M., and Seeger, M. Gaussian process optimization in the bandit setting: No regret and experimental design. In International Confer- ence on Machine Learning (ICML), 2010.
Wei, J., Wei, J., Tay, Y., Tran, D., Webson, A., Lu, Y., Chen, X., Liu, H., Huang, D., Zhou, D., and Ma, T. Larger language models do in-context learning differently, 2023b. URL https: weaxiiv.oo org/ abs/ / 2303 03846.
Sun, H. and Guo, Q. Dsg-gan: Multi-turn text-to-image syn- thesis via dual semantic-strean guidance with global and local linguistics. Intelligent Systems with Applications, 20:200271, 2023.
Wiles, O., Zhang, C., Albuquerque, I., Kajić, I., Wang, S., Bugliarello, E., Onoe, Y., Knutsen, C., Rashtchian, C., Pont-Tusets J., et al. Revisiting text-to-image evaluation with gecko: On metrics, prompts, and human ratings. arXiv preprint arXiv:2404. 16820, 2024.
Sutton, R. S. Reinforcement learning: An introduction. A Bradford Book, 2018
Viégas, F. and Wattenberg, M. The system model and the user model: Exploring ai dashboard design. arXiv preprint arXiv:2305.02440 2023.
Wu, X., Hao, Y., Sun, K., Chen, Y., Zhu, F., Zhao, R., and Li, H. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis, 2023.
Vodrahalli, K. and Zou, J. Artwhisperer: A dataset for char-
acterizing human-ai interactions in artistic creations. In International Conference on Machine Learning (ICML), 2024.
Xu, J., Liu, X., Wu, Y., Tong, Y., Li, Q., Ding, M., Tang, J., and Dong, Y. Imagereward: Learning and evaluating human preferences for text-to-image generation, 2023.
Wang, Q. Anikina, T. Feldhus, N., van Genabith, J., Hennig, L., and Möller, S. LImcheckup: Conversa- tional examination of large language models via inter- pretability tools and self-explanations, 2024a. URL /tttss://artie org/ abs/ 2201. 12576.
Yang, Q., Steinfeld, A., Rosé, C., and Zimmerman, J. Re- examining whether, why, and how human-ai interaction is uniquely difficult to design. In ACM Conference on Human Factors in Computing Systems (CHI), 2020.
Wang, Z. and Jegelka, Max-value entropy search for effi- cient Bayesian optimization. In International Conference on Machine Learning (ICML), 2017.
Ye, X. and Durrett, G. few-shot prompting for textual reasoning, 2022. URL https: http://axxvvvv....eee aat.///ss.22.//..///// 03401.
The unreliability of explanations in
Wang, Z., Garrett, C. R., Kaelbling, L. P., and Lozano-Pérre Pérez, T. Active model learning and diverse action sampling for task and motion planning. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4107-4114. 14. IEEE, 2018
Yin, G., Liu, B., Sheng, I.,, Yu, N., Wang, X., and Shao, J. Semantics disentangling for text-to-image generation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 2327-2336. 2019.
Wang, Z., Liu, L., Zhang, H., Liu, D., and Song, Y. Genera- tive adversarial text-to-ing text-to-image generation with style image 2023. Wang, Z., Dahl, G. F., Swersky, K., Lee, C., Nado, Z., Gilmer, J., Snock, J., and Ghahramani, Z. Pre-trained gaussian processes for bayesian optimization. Journal of Machine Learning Research (JMLR), 25(212):1-83, 2024b.
Yu, L., Shi, B., Pasunuru, R., Muller, B., Golovneva, O., Wang, T., Babu, A., Tang, B., Karrer, B., Sheynin, S., et al. Scaling autoregressive multi-modal models: Pretraining and instruction tuning. arXiv preprint arXiv:2309.02591, 2(3), 2023.
constraint. Multimedia Systems, (9)):32211333))
Zha, Y., Yang, Y., Li, R., and Hu, Z. Alignscore: Evaluating factual consistency with a unified alignment function, 2023.
13

---

Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty
A. Author contributions
All authors contributed to writing.
Meera Hahn (meerahahn @ wowglegooooogooooo com): Led automated evaluation and human evaluation of agents. Created DesignBench.
Contributed to agent design and development.
Wenjun Zeng (wenjunzeng @ google.com): Significant contribution to agent design and development, as well as open- sourcing.
Nithish Kannen (nitkan @ google.com): Significant contribution to experiments and early versions of agents. Led -pen-soureing.
Rich Galt (richgalt @ww.gggooooommio Led agent interface design and human studies.
Kartikeya Badola (kbadola @ google.com): Contributed to experiments and early versions of agents.
Been Kim (beenkim @ @google.com) Advised project direction with critical feedback.
Zi Wang (wangzi @ google.com): Proposed and initiated project. Led agent design and development. Advised project. Contributed to evaluation.
B. Related Work
From the very outset of artificial intelligence, a core challenge has been to develop intelligent agents capable of representing knowledge and taking actions to acquire knowledge necessary for achieving their goals (McCarthy & Hayes, 1969; Minsky, 1974; Moore, 1985; Nilsson, 2009; Russell & Norvig, 2016). Our work is an attempt address this challenge for intelligent T2I agents.
In machine learning and statisties, efficient data acquisition has been extensively studied for many problems, including active learning (Cohn et al., 1996; Settles, 2009; Houlsby et al., 201 1; Gal et al., 2017; Ren et al., 2021; Wang et al., 2018), Bayesian optimization (Garnett, 2023; Kushner, 1964; Močkus, 1974; Auer, 2002; Srinivas et al., 2010; Hennig & Schuler, 2012; Wang & Jegelka, 2017: Wang et al., 2024b), reinforcement learning (Kaelbling et al., 1996; Ghavamzadeh et al., 2015; Sutton, 2018) and experimental design (Chaloner & Verdinelli, 1995; Kirk, 2009). We reckon that T2I agents should also be capable of actively seeking important information from human users quickly reduce uncertainty (Wang al., 2024c) and generate satisfying images. In $E, we detail the implementation of action selection strategies for our T2I agents.
In human-computer interaction, researchers have been extensively studying how to best enable Auman-AI interaction especially from user experience perspectives (Norman, 1994; Höök, 2000; Amershi et al., 2019; Cai et al., 2019; Viégas & Wattenberg, 2023; Chen et al., 2024; Yang et al., 2020; Kim et al., 2023). Interface design for Al becoming increasingly challenging due to the lack of transparency (Viégas & Wattenberg, 2023; Chen al., 2024), uncertainty about Al capability and complex outputs (Yang al., 2020). We aim to build user-friendly agents, and an indispensable component their interface to enable them effectively act and observe, as detailed in $G.
Interpretability. Surfacing an agent's ' belief overlaps with interpretability as both aim to understand model or agent's s internal state. Some methods leverage LLM's natural language interface to surface their reasoning (e.g., chain of thought (Wei et al., 2023a)). sometime interactively (Wang et al., 2024a). While these approaches make accessible explanations, whether the explanations represent the truth has been questioned (Lanham et al., 2023; Wei et al., 2023b; Chen et al., 2023). Some studies indicate explanations generated by the LLMS may not entail the models' predictions nor be factually grounded in the input, even on simple tasks with extractive explanations (Ye & Durrett, 2022). In this work, the belief graph does not correspond to the distribution over outputs of the T2I model itself conditioned the underspecified prompt. Instead, the belief graph is designed to align with the distribution over paage-prompts generated by the agent, since the agent can construct detailed prompts according to its belief, and feed them into high-quality T2I model.
Text-to-Image (T2I) generation. Text-to-image prompts can be ambiguous, subjective (Hutchinson et al., 2022), or challenging to represent visually (Wiles et al., 2024). Different users often have distinct requirements for image generation, including personal preferences (Wei et al., 2024), style constraints (Wang et al., 2023), and individual interpretations (Yin et al., 2019). To create images that better align with users' specific needs and interpretations, it is essential to actively communicate and interact with the user to understand the user's intent. Other work (Mehrabi et al., 2022) proposes to use clarification questions to resolve ambiguities (a prompt has multiple meanings), but our aim is to use clarification questions to resolve under-specification (the prompt is not ambiguous, but lacks information to fully describe the image).
T2I models are evaluated by analyzing amagge-rropp alignment using metrics that can be embedding- based, such as
14

---

Proactive Agents for Multi-Turn Text- Generation Under Uncertainty
CLIPScore (Hessel al., 2022), ALIGNScore (Zha et al., 2023), VQA-based such TIFA (Hu al., 2023), DSG (Cho al., 2023) and VQAScore (Lin al., 2024), and captioning- based like LLMSCcre (Lu et al., 2023). Approaches such as PickScore (Kirstain et al., 2023), ImageReward (Xu et al., 2023) and HPS-v2 (Wu et al., 2023) finetune models on human ratings to devise metric that aligns with human preferences. Diversity of generated images (Naeem et al., 2020) is also used as a metric to track progress, especially in the geo-cultural context (Kannen et al., 2024; Hall et al., 2024). To evaluate our agent-user conversations and corresponding generated images, we develop an automatic approach using a variety of the metries mentioned as well as collect human ratings on the generated images and generated conversations.
Prompt expansion widely known technique improve image generation (Betker et al., 2023). Imagein Words (Garg et al., 2024) proposes to obtain high-quality hyper-detailed captions for images, which significantly improve the quality of image generation. Datta et al. (2024) present a generic prompt expansion framework used alongside Text-to-Image generation and show an increase in user satisfaction through human study. While our work can be viewed as a method to adaptively expand T21 prompt based on user feedback", evaluating our method as a prompt expansion tool is outside of our scope.
C. Formalism of the Agent and its Objective
We define interactive agent as a (B, A, O, T) tuple, where have
S: a state space where each state represents an intent,
B: a space of agent belief states,
A: a space of actions that the agent can take,
O: a space of agent observations of the user,
transition function т B x A x O ~ B for updating beliefs given new interactions,
action selection strategy æ : B ~ A, which specifies which action to take given a belief.
For each user-initiated interaction, we assume that there exists a specific intent 8 E S in hindsight, where $ is the space of all possible user intents. For a T2I task, this intent is the image the user would like to generate at the end of the interaction.
Each type of T2I agents can have a unique user intent representation, belicf representation, construction of the action space, and user interface design to obtain observations of users.
In $4, we show the examples for these components.
We score function, f B S + R, to evaluate the alignment between agent belief and user intent any turn of the interaction. Function f can only be evaluated in hindsight once the user intent is revealed. The agent does not have direct access to function f since the user intent is hidden from the agent. However, the agent may construct a probabilistic distribution over function f based on its belief about the user intent. The goal of the agent is to maximize function f with as few turns of interaction with the user as possible.
D. Visualization of Multi-Turn Agent-User Dialogs and Generated Images
In Figure 5, we show examples of multi-turn dialogs between simulated users and the three agents in Section 6. We also visualize the generated images in Figure 6, Figure 7, Figure 8 and Figure 9.
E. Implementation Details of Agent Prototypes
We propose three T2I agents, each characterized by a unique configuration of .T) tuples:
Agl: Heuristic Score Agent. This agent incorporates deffnndefined heuristic score based on the belief to guide
question generation. This heuristic score reflects the perceived importance of different aspects of the belief in driving the conversation forward:
Ag2: relief-prompter Agent. This agent leverages an LLM to generate questions by processing both the conversation history and a structured representation of the belief.
Ag3: Principle-prompte Agent. This agent generates questions directly from the conversation history based on the principles introduced in $4.2. The question asking strategy of Ag3 relies solely on the implicit knowledge and reasoning
"Samples from the agent belief can be used to construct expanded prompts.
15

---

Proactive Agents for Multi-Turn Text-to-Inee Image Generation Under Uncertainty
Agl
Ag2
Ag3
Question: entity "cake' present image?
Question: What type of berries would you like on the cake (c.g.. strawberries. blueberries, raspherries, mixed)?
Question: What types of berries are on the cake and how are they arranged? Answer: The cake decorated with fresh blueberries
Target Image
Answer: Yes
Answer: Blueberries and blackberries Question: What style photo are you imagining? (e.g. photorealistic, minimalist, rustic, painting-like)) Answer: Minimalist
blackberries. They are artfully arranged on the side the cake.
Question: Is the entity 'type cake' present the image?
Answer:
Question: Besides the berries, what other decorations or details are visible on the cake?
What type cake you thinking of? sponge cake, chocolate cake, c. cheesecake, d. vanilla cake, e. pound cake. f. unknown
Question:
Question:
What color would you like the
cake to be, and should it have any frosting ...?
Answer: Besides the berries, the cake is decorated with smooth white
Initial Prompt
Answer: Vamilla cake
Answer: The cake should be white and buttercream frosting.
buttercream frosting that has a Question: What the overall shape and color of the cake?
Question:
What should the image style
be? a. photorealistic, realistic, render, ...
Question: How many layers should the cake have, and what flavor should (e.g,, vanilla, lemon, etc.)?
photo cake that adomed with berries sitting
Answer: The cake round and white.
Question: Is there any visible text or writing on the cake?
Answer: Realistic
Answer: The cake should have two layers and be vanilla flavor.
Question: type frosting would
Answer: No, there is no visible text or
on a table.
you like the cake? buttercream,
Question: For minimalist
writing on the cake.
cream cheese, c. whipped cream,
background, would you prefer plain white background, minimalistic studio backdrop, blurred kitchen scene? Answer: A plain white background
Question: Is the cake on a plate or stand, what color and shape?
Auswer: Buttercream
cake stand.
white and likely round shape.
Turn Generated Image:
Turn 5 Generated Image:
Turn Generated Image:
Figure 5. Real multi-turn dialogs generated by the Agl, Ag2, and Ag3 agents on image from DesignBench. The figure additionally shows the image generated after the turn dialog per agent.
16

---

Proactive Agents for Multi-Turn Text-to-Imape Image Generation Under Uncertainty
T21-Only
Agl
Ag2
Ag3
Goal Image
Figure 6. Agent Generated Image Outputs DesignBench: chart of the generated image outputs of the four main Agent types in comparison to the goal image. Each column displays the output of a different agent and the right most column shows the goal image that the agents aimed recreate. Each agent was provided with the same starting prompt and iterated for turns, with the exception of the "T2I" agent column which produces an image from the starting prompt. Agl, Ag2 and Agã refer to the Agents deseribed in $E. Each agent uses same model produce final image. The goal images displayed here are from our DesignBench dataset described the experiments section.
17

---

Proactive Agents for Multi-Turn Text-to-Inie Image Generation Under Uncertainty
T21-Only
Agl
Ag2
Ag3
Goal Image
Figure Agent Generated Image Outputs on DesignBench (Continued): chart the generated image outputs of the four main Agent types in comparison to the goal image. Each column displays the output of different agent and the right most column shows the goal image that the agents aimed to recreate. Each agent was provided with same starting prompt and iterated for 15 turns, exception of the "T2I" agent column which produces an image from the starting prompt. Agl. Ag2 and Ag3 refer to the Agents described $E. Each agent uses the same T2I model produce the final image. The goal images displayed are from the BesignBench dataset described in the experiments section.
the
18

---

Proactive Agents for Multi-Turn Text-to-Inin Image Generation Under Uncertainty
T2I-Only
Ag l
Ag2
Ag3
Goal Image
10
Figure 8. Agent Generated Image Outputs types comparison to goal image. Fach column displays the output of different agent and right most column shows the goal image that the agents aimed to recreate. Each agent provided with the starting prompt and iterated for 15 turns, with the exception the "T21" agent column which produces an image from the starting prompt. Agl, Ag2 and Ag3 refer to the Agents described in ŞE. Each agent uses the same T2I model to produce the final image. The goal images displayed here are from the Coco- Coco-Captio Captions (Chen et al., 2015) dataset described in the experiments section.
Validation): chart of the generated image outputs of the four main Agent
19

---

Proactive Agents for Multi-Turn Text-to-Imane Image Generation Under Uncertainty
T21-Only
Ag
Ag2
Ag3
Goal Image
Figure 9. Agent Generated Image Outputs (Coco- Caco-Caatioons Validation): chart of the generated image outputs of the four main Agent types comparison to the goal image. Each column displays the output of different agent and the right most column shows the goal image that the agents aimed to recreate. Each agent was provided with the same starting prompt and iterated for 15 turns, with the exception the "T21" agent column which produces image from the starting prompt. Agl, Ag2 and Ag3 refer the Agent described in the methods section. Each agent uses the same T2I model to produce the final image. The goal images displayed here are from the Coco-Care Cappoinn (Chen al., 2015) dataset described the experimžtts section.

---

Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty
capabilities of the underlying LLM.
A summary of each agent is located in $5.
E.1. Implementation of Agent Beliefs
Technically, an agent belief b E B is represented in two complementary forms: (i) Merged prompt: This is natural language representation that summarizes the entire conversation history up to the current turn. It provides comprchensive textual overview of the user's requests, feedback, and any clarifications exchanged with the agent. (ii) Belief graph: This is a symbolic representation derived from the merged prompt. It parses the natural language text into a structured format, capturing key elements like entities, attributes, relationships, and associated probabilities. This structured representation facilitates more precise reasoning and decision-making by the agent.
Prompt Merging. An LLM ($E.8) summarizes the latest interaction, encapsulating the agent's question and the user's response into a concise textual representation. This step distills the essential information exchanged during the interaction. Another LLM (SE.9) merges the existing merged prompt (containing the accumulated information from previous interactions) and the summarized interaction at the current turn. This creates an updated prompt that reflects the evolving understanding of the user's intent.
Belief Parsing. See an example of the belief graph in Figure 10. We employ three specialized parsers trained via in-context learning (ICL): entity parser ($E.5) analyzes the user prompt to identify and extract a list of relevant entities.; attribute parser ($E.6) takes user prompt and an entity as the input to extract list of attributes associated with that entity; relation parser ($E.7) takes the user prompt and list of entities as input and identifies relationships between those entities. Each entity is associated with meta information like name, importance to ask score, description, probability of appearing, list of attributes like color, position, etc' 7. Each attribute contains meta information like name, importance to ask score, a list of possible values for the attribute along with their associated probabilities, etc. Each relation includes meta information such as: name, description, spatial relation, importance to ask score, entity 1 and entity 2, whether the relation is bidirectional, etc.
E.2. Implementation of Action Selection
From an information theoretic perspective, an optimal action is the one that maximizes the information gain between the observation and the belief, i.e. a = arg max, H(;;-1;;;-> b; a) (o;; by a). However, directly optimizing this objective can be computationally challenging. Therefore, we explore several heuristic strategies to effectively reduce uncertainty:
Maximize the overall heuristic importance score (MH))):: This strategy focuses on maximizing the overall importance score of the entities, attributes, and relations within the belief. We further ask question regarding attribute or relation by maximizing the overall heuristic importance score. The score can be modeled as:
maTe, (IS((e * IS(a) P(e)= * Ent(c), P(r) * Ent(c))
(I)
Here IS, P. Ent represent importance to ask score, probability of appearing, and entropy of the probabilities respec- tively and e, a, c, r represent entity, attribute, candidate list, relation respectively.
Ask Important Clarification Question based on belief (AICQB): This strategy leverages the structured information within the belief. We provide the LLM with the user prompt, conversation history, and the current belief, utilizing an ICL prompt ($E.11) 1) to guide question generation. The LLM then formulates a clarification question aimed at eliciting information about key features of the image, naturally prioritizing those with higher Importance to ask score within the belief.
Ask Important Clarification Question directly (ATCQuasee: This strategy relies on the LLM's inherent ability to identify important aspects of the clarification question based on its implicit understanding of the user's needs, without explicitly relying on the structured information in the belief.
user prompt and conversation history. The LLM (SE.10) generates an important
Agl employs M HHHS IS strategy for question generation. This strategy leverages the importance scores assigned to entities, attributes, and relations within the belief graph. It identifies the element with the highest heuristic importance score and
NNaee is unique identifier for the entity: Importance to ask score: A numerical value indicating the entity's perceived importance in satisfying the user's request. Entities with higher scores are prioritized during question generation, as they are likely to reduce uncertainty and contribute significantly to the final image; Deseription provides textual description of the entity: probability of appearing estimates likelihond of the entity being present in the generated image; Attributes are for understanding the detailed attributes the entities.
21

---

Proactive Agents for Multi-Turn Text-to-Inin Image Generation Under Uncertainty
Entities
Rabbit
Attribute Name: color, importance score: 0.9,
candidates: [brown: 0.25, white: 0.25, grey: 0.2, black: 0.15, ...]
Attribute Name: breed, importance score: 0.3,
candidates: [Dutch: 0.2, Mini Lop: 0.15, Netherland Dwarf: 0.15, ...]
Attribute Name: expression, importance score: 0.5,
candidates: [scared: 0.8, determined: 0.1, playful: 0.1]
Dog
Attribute Name: breed, importance score: 0.8,
candidates: [Labrador Retriever: 0.15, Golden Retriever: 0.15, German
Shepherd: 0.15, Bulldog: 0.1, Beagle: 0.1,
Attribute Name: coat color, importance score: 0.7,
candidates: [brown: 0.2, black: 0.2, white: 0.2, ...]
Attribute Name: coat style, importance score: 0.6,
candidates: [long: 0.3, short: 0.3, fluffyy: 0.2, shaggy: 0.1, wavy: 0.1]
Attribute Name: hat, importance score: 0.5,
candidates: [baseball cap: 0.2, bowler hat: 0.2, top hat: 0.2, ...]
Relations
Dog-Rabbit
importance_scom score: 0.9,
spatial relation: [chasing: 1.0]
Coat-Dog
importance score: 0.8,
adormennt_ relation: [wearing: 1.0]
Figure 10. An example of the belicf graph data structure for given prompt in Figure 1.
22

---

Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty
formulates a question aimed eliciting further information about that specific element. The question is then verbalized using the LLM described in Section $2.12.
Ag2 utilizes the parsed belief graph as the basis for question generation. It employs the AICQn n strategy, which leverages the structured information within the belief graph to generate targeted clarification questions.
Ag3 relies solely on the conversation history for question generation. It employs the ALCQ strategy, which leverages the LLM's ability to understand the ongoing dialogue and identify key areas requiring further clarification.
E.3. User Simulation
To simulate end-to-end agent-user interactions, we implement a user simulator that mimics human question-anse answering behavior. This simulator operates as follows:
It generates belief graph based on a ground truth prompt, representing the user's intended image. This serves as the simulator's internal representation of the desired image.
The simulator takes the ground truth prompt, conversation history, and its current belief graph as input. It then leverages an ICL prompt (see $E.11) to generate a response to the agent's 's question. This ensures that the simulator's 's answers are consistent with its internal belief graph and the ongoing conversation.
E.4. Implementation of Belief Transition
Both Ag/ and Ag2 require belief updating to incorporate new information gained during the interaction in order to compose clarification questions. At each turn, we perform prompt merging to create a comprehensive prompt that summarizes the conversation history. This merged prompt is then used for belief parsing to obtain an updated belief graph. For Ag2 (and Ag3), this updated belief graph directly informs the subsequent interaction. For Ag/, incorporates additional post-processing mechanisms to enhance memory and prevent redundant questioning: (i) Redundancy elimination: If attribute or relation has already been addressed in the conversation history, the corresponding user response is assigned as the sole candidate with probability of 1.0, and its importance score is set to O. This prevents the agent from repeatedly asking about the same information. (ii) Information retention: If an attribute or relation from the conversation history is absent in the updated belief graph, it is explicitly added. This ensures that the agent retains crucial information even if it's not explicitly present in the latest parsed belief graph.
E.5. Entity Parser Prompt Instruction
Given as lext -to-image prompt list out all the mitiiie that are mentioned in the prompt.
Explicit Entisies List all cleary stated emities within prompt (people. objects. , animul. locations. eto.). Implicat lanties Identity posental cstriies
inglaed strongly suggested by the pcompt, even it not explicitl mentioned. eeBaackround Lnusies:e cen Deduce relevant background elements which could impact the image generation from the peompt or coneest including: We If the scene moud suggess specific weather conditions (sunny, zanyy stormy, eic J. specific setting himed (indooes. oundoces, particuiar /landscuge, etc poompt implies certain time (dawn, midday, dusk, night).
ss If general
eoTime Duyoe aeMood or Ammogghere in If the prompt evokes particular emotion or ambiance (joyful. mysterious, peaceful, etc
10
11
12
The ouppt should be
each entry should be formated ISON "name": The name of the emity
following fields
13
14
15
importance.co to
of asking question about earry rsduce the uncertiinty what the given procuee Make number betweet and 1. higher means more importiat. Consider these facoors when assigning scores; increate score for entilies that are the primary of the prompt: 2. incress the score for entites that could strongly infience the layout of the image, such as the position or portrayal of other entities in the scene; 3. signfiiantlydsceesse the score for cutties that are alrendy well specrtied in the promut. 4. sguiftaantyineresee the scoce for implicit entitie that are likely significantly impact image.
importance
subject
appear in the image and their uppearance
16
deseription
description of the entity...
17
cannyy ". The type of this cattty... It could be either expleit. implicit, background. No other value is allowed.
18
probability of appeazing ": The prebibility of the eatity uppeasing the image. This is number between O and 1. You should assign probability with the foliowing rules in mind
If pronte says an entiry does not exist, assign 0.O peobability. Because the entry does not exst, you should also assign importunce score of entity... 20
19
2. IE the pronpt indicates an eatity definitely exists in the image, assign a 1.0 probability.
21
3. 1r the promett does not say asything about the existence of the eatity. assigo probuiility between O and 1. This probability is higher if the entity is more likely to appear
the image gaven the context specified
prompt.
22
the pronpt says entity exiss there indication. the entity likely appear the image, assign probubility between and 1. higher eatiyy
is more likely to appear in the image.
23
24
Below example injut anput pair:
25
Exumplel::
26
laput: [(
27
"user peampe" "generate on imaga of a liothead rabbit munning on grass with sun shining. There is no noes in the hackground."
28
29
Output:
30
{{
23

---

Proactive Agents for Multi-Turn Text-to-Inan Image Generation Under Uncertainty
"hame": "rabbii
32
importance coundk.coee 0.5.
33
"description" ": loohead rabbit
camay_yype explicit
35
probubility.u Catabbiittyyllagee aepporring 1.0
36
1).
37
{1
"grass"
39
importance Jounk soore 0.5.
40
description ": "grass"
41
entiy explicit
43
44
45
"naee:: "sun".
amportance wow.ck 0.1.
17
description ": "sun is shining"
48
canny_aype explicit
49
probubility appearing " 0.3
51
52
name":
"sun light
importacc
description ":
shining on grass rabbit
55
cotay type explicit
56
1.0
59
"ame"" "tree".
60
importance to ask scoce 0.
ddesription
background",
capey-oye 7; explicit
63
probubility appearing
64
angle"
67
importance to.nok score
B.8.
68
description ": "the comera angle of the image""
cntity rype "tackgroan".
70
probubility of agpearring
71
72
"weather".
importance aek sooce 0.8.
75
description "; "eeather" canny "; "hackgrooaid""
76
probuhility sepearing
78
79
80
"name": "image style"
impontace style image".
1.B.
description
83
entity type "luckgroand",
84
probahilit_.f oepeprrring 1.0
87
hame": "background color". ".
88
importance to.nsk score 0.8.
deseription ": "the background color image",
eutiyyrype "; "Buckgroond""
91
probability of sppooringg "; 0.5
92
}}
95
[[a few additional ecamples]]
96
98
Edeniify entities given the input given below. Strictly
99
Input: {f
100
"user peompt" "( "[nser_peompp""
Output:
E.6. Attribute Parser Prompt Instruction
Given prompt
cutity described the prompt, your goal ideatiyy list
possible amrilbutes
could describe
particular
entty...
Ousput Requiremeas:
if this attribute has already existed ss emity in other ctisting entity list. thes do not include it.
attibuze candidate could be maxed values like "caler and color
oupput
joon pans-aable format:
name (tr): The name of the attribute betees and higher more inportunt Consider these factoss when assigning socces: Increste soare atrihutes
inportance ask
importance
asking question
stribute roluce uncertainty what image given prompt.
primary aniibutes
important emity: 2. sigaifianily increase the score for ummiuies that could strongly influesce the generation or poctrayal of UTHER attiluuees in the scase;
Assereasc the soce for atrihutes that are alrendy well specified in the prompe, For example, becod a dog would impacr other attrihtes like color, size. dec So the
breed. atiibute
color. probubulttiss y: List possible values rcalistie for the given entity For cads auribate. returns the prbabbitay that the use wants this canditate hased on the user prompe. If it's already menticned by the
impostanee
Assign much sure generate caa
autribue value is already more possible values. should
prompt.
(List
24

---

Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty
eatr, only genente one canditate (the mentioned one) and assign the probability The sum probubilities over candidutes shall Also infor prebubility based on the prompt. For example, for dog with breed Samuyed, the color uttribute has very high probubility of whise.
10
examples ingut ourput paiss:
12
13
Example I:
14
Iaput:
"user_peumpp" generate image
running grass
16
emily "rubbil"
17
other.eusinggcoom cotities ": "grass"
18
Ourpur:
20
21
name": "color.
22
importacee
candidates" (("white" ]100)
24
25
26
"breed"
importace wou sooce 1.0,
28
"andiddates"
[[ (("Dusch": 0.20,
29
"Mini Lop": 0.15,
30
"Netherland Dwarf":
"Lionhead":
32
"Flemish Giant": 0.10.
33
"Mini Rex": 0.10,
34
"Laglish Angora"; 0.08,
0.05,
36
"Himalayua" B.8.
37
CCllffrmaan ": 0.02)]
38
40
bame" "age"
41
importace to.nsk osk score D.I,
42
"candidates (""cutt"
"abyy
44
"enior 0.2)
45
}}
48
Ila few additional examples]]
49
Genente attibutes given iaput given below. include entities the attribates Strictly
laput: {{
52
"nser "x "[mser prompe)".
53
"enity "; "[entity name)
54
evisingg
esisting entities
}}
56 Output:
E.7. Relation Parse Prompt Instruction
Given ai bext -to-image grumpi and of enitty deserieed prompt, your pool to identify a. list entity pains. that have rehsioms between them. Ignore entily without relations... The ourpt sbould be joon parse-able format (No comma after the last element of the last
pairs
Iaput:
userapromap:: the prompt from the wer. entinies list of eniites mensioned in the user, prompt.
Output:
mame (tr ): The name of the relation. . Use emityl =emity?' description str i: A shoet deserittons of the relation
format.
10
spatial relution probabllity The
from potential relation candidates probability Pussible spatial relations between prububilities
relation is mentioned
prompe,
candidates shall importatee.to.skl score float The importacce score of asking question regarding this relation to reduce entropy. This is number between and 1, higher means more importunt. important, the relation between them is very unclear. and the relation is very important for the layout of the image.
11
Assign higher score if the two entiies enity.1
12
entity
conitecnlyy_22 ):
eatry...
14
is bodirectional (bool): Whether the relation bidirectional.
15
16
Helow example input oupput
Examplel:
18
Iaput: [(
19
"see peompe generate an image liothead rabbit sitting on grass, and eagle is flying through the sky. There tree in the background.""
20
emtiy
"eagle"
)
22
Output:
23
{(
24
"name": rrabbttgraas sitting grass".
"dsccription" [ ("above": 0.8. "helow": 0.0, "in front of": D.D, "behind": D.,, "left of": D.I, "rigte of": 0.1)
26
reattal.
27
iimpottacc to.ask scoce U.I.
28
"name entity mibis
"name .nnitty "grass".
30
bidirectocal true
31
32
"eagle-grass"
34
"description" ". "caple Bying through the sky".
25

---

probubiiity agent estimated Proactive Agents for Multi-Turn Texte Generation Under Uncertainty
35
prelttlurelation (f"shove": 1.0. "below": 0.., front of": 0.0. "behind"" 0.0. left of": 00, "right of": 0.0)).
36
"importaace www.dk..oooooooooo ": 0.1.
37
"nume entity. eugll"""""
"name -enteenttty "grass".
39
is.didirctional fabs
40
41
"cree-grass" "dscrription ":
43
44
"patiilreelation {{ (("above": 0.5, "below": 0.0, "in tront of": 0.0, "behind": 0.0, lIft of": 0.2. "right ef": 0.253). 0.I,
45
importunce to.ask
"name .n tree
47
"aame antiyy 22 "grass"
18
is hidirectional fabe
-9
50
51
aIa few additional examples]]
52
53
55
Ldemify relationships between enticies gaven the inpur given below. Stricdly stick to the format.
56 Input: {{
geomgee""" "[user [user_prompe)"
58
emity "[entity names)"
59
}}
60
Ourput
E.8, Verbalization Prompt Instruction
The chatl history is as follows:
question: (action. vabulizasaaction and answer: [observaton):
the question action singlo declarative sentence that describes answer phrase as question. Example ourput:
image
E.9. Merge Prompt Prompt Instruction
You writing proppt
foadBook. The original pronge is (propp)). The pewided additional information: adLitionaljinf text to--ggg model. The new prompt information, Do not ald any new information that is not menioned in the prompt or the additional information. Make sure the information in the original prompt is not changed. Make sure the adtricoal information is included in the new prompt. Make sure the new pronpt deseription of an image. If the additional information the image, you should sure new prompt mentions this thing NOT generate rationale anything that is not part descriptio of the image.
Please wite new prompt lest-to-image model based
meaningful sontence porgraahh comtines the original poompt the additionl
original prompt specifically says that thing
exist the image.
E.10. AiCQe Prompt Instruction
II Instruction the question II
The original peompe was: [satr, osiginal. grompt] Hosed on the original prompt planse provide question ask about the image. The question should be concise and direct possible The the answer within <question > and <cqqusrion'> markers
should
leam about the attibuuees and conteats of the image, objects, the sputial layout,
Make you question
II Instraction for the following question| II
Based on chat history please provide new question ask image, history follows enclosed <chat history's and <c/ehatt </eehat.histrry > markers: [setr. chat history } <cicuut history > The question should he as concise and dinect is possible. The question should aim to teom more about the auributes and contents of the image. the objerss, the spatial layeur, and the style. Make sure that you question the answer within <question>> and </question>> markers"
E.11. AICQ& Prompt Instruction
You are intelligent agent that helps users generaee images. Before generating the image requested by the user. you abould ask the most impostans clarification questions to make you undesstand The user deseribes image as: (user_geompt).
features
image.
The following is your belief of what the image contains. including the entinies aumiutes of each entity and relations between eniities
Each entity his "name". descriptions
"Importance usk "impartace ask score" and "robubiiity of appearing" ". "Name" is the identifier of the entity "Descriptions" " is the deseription of the important image.
whether the enity exists. Probability appearing"
this entity exits
Each ensity has a list of amibutes Each atributt has "name", "impostance to ask score" and candidates"" "Name" is the identiier of the attribute "Importance to ask score" is ask about the value the autihut of enilyy "Candidates"
important
of possible values for atribute.
Each candidate value has probability that describes how likely this cundidate value should be assigned to the attribute
For example. Amsibute Name: color, Inspectance to ask Score: 1.9, Candidates: (white: U.S, black: 0.5 means the color eihter white Black, each with 0.5 probahility If you highest uncertainty.. Your uncestainty judged by the probabilities the peobabiliries 0.5
ask about attributes you should about the amilhut probabiities are and 0.9, you are fairly centain..
you are uncertain.
10
11 The agent belief is: 0)
12 bellef
14 Hased on the user prompt" [( [wer prompe)" " and the belief of the agent, plasse grovide a question to ask about the image. The question should be as concise and direct as possible The question should sim to oleain the most information shout the style, carniios attrilutes. spatial layout and other contents of the image. Remember to ask for information critezll
eritical detuils of the image
the The question should SDT question ure obvious to answer based on the user geomge [user prompt"" " DO any
your uncertainty
intest much
possible. DO NUT question that questicn about information presest in the following ueer aggnt diologue within <dialogue > and <iddnlogue's markess
answered by common
26

---

Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty
15
16
<diillogue>>
17
[comvensatio)
<idialouue>>
19
20
DO NOT ask question that. has been usked in the diologue above.
21
22
can construct any question make younelf more emtirely decided by image to genersee Make sure to ask unly one question. Make sure it is nol very diflicult for the user to answer. For Think step by step and ressun about your unceriainty
question
image
23
example, do not ask very very long question. which can tike the user a long time to read and answer.
24
Make that you question the answer within <<quesiion>> <cqquestion> markers.
E.12. HSA Question Prompt Instruction
You
text-to-imaa (121) prompt
details from the most impostant eatity
You have aek a question about
most impostant emtly the
We have cuzey types: (i) explicit: direcily ask question with options: (a) implicit ask whether this colity required for the image with yes or no as options: ; ( (ill) backgrouad:
ignore the anvibate value and directly ndt the value of the entity (iv) relation sái keywoed like rolition emphisiie this onilyy is a relution Constract simple questions directly asks this information from the
provides opticn
from. question follow with options
Exxmpiel:
antit: nbbit
attrittt:: color
10
candidates: black. white, brown
entity type: explicit
12 qusstion: What
do you have mind?a. black, white, d. nnkown. options, What color the do you have
13
14
[[a few adfitional examples]]
16
ExampleS:
17
antit: Semily
18
aitibute: Sattribute
candidstes: Scandidatess
20
eatity Jype: Seye
21
questio::
F. Automatic Evaluation
In Algorithm 2, we show the user-agent self-play procedures that we used to perform all automated evaluation.
F.. DSG Evaluations
The DSG (Cho et al., 2024) metric is used to compare the ground truth caption and the generated caption. DSG is adapted to parse text prompts into Davidsonian scene graph using the released code. The plots in Figure 11 show the T2T DSG metrics per turn. The results in Figure 11 show that significant gains performance come from using proactive multi-turn agents. The blue line on the graph represents the simplest baseline which directly uses T2I model and performs no updates the initial prompt pO. Note that all multi-turn agents far exceed the baseline T2I model all benchmarks for the DSG metric.
F22 Mitigation of T21 Errors
limitation of the proposed pipeline is that its overall capability constrained by the prompt-following abilities of the employed text-to-image (T2I) model. The proposed agent prototypes call off-the-shelf T21 models directly, treating T21 APIs as tools. This offers the benefit of seamlessly switching to better T2I models when they become available. In effort to mitigate T2I errors caused by the model not following the prompt, we perform experiment demonstrating that using agent-user QA pairs can improve T2I fidelity across a batch of N seeds.
The experiment design is as follows: Each QA pair from the agent-user dialogue is converted into a (yes/no) VQA question concerned with a single detail of the image. Then, using a VQA score metric with these new questions, we can remove erroneous images from a set of seeds by filtering out images with low VQA scores. We perform this experiment on the DesignBench image-caption dataset.
The specific implementation is as follows:
1. Use the 30 ground truth (GT) prompts of DesignBench and generate 10 images from 10 different random seeds with Imagen.
2. Take average DINO (I2I) score for all images against GT image, this was found to be 0.7637.
3. Take the first 5 Q-A pairs from Ag2 turn each into a binary yes or no question for which the correct answer yes.
27

---

Proactive Agents for Multi-Turn Texte Generation Under Uncertainty
ImageIn Words T2T DSG
Coco Captions T2T DSG
1.00
1.00
T21
T21
0.95
Ag3
0.95
Ag3
Ag2
0.90
Ag2
Text Similarity (0
0.90
Ag1
Ag1
0.85
Text Similarity
0.80
0.80
0.75
0.70
0.75
0.65
0.70
0.60
0.65
0.55
2
4
5
6
7
8
9
10
11
12
13
14
15
4
5
6
7
8
9
10
11
12
13
14
15
Turns
Turns
DesignBench T2T DSG
1.00
T2I
0.95
Ag3
0.90
Ag1
Text Similarity (0
0.85
0.75
0.70
0.65
0.60
10
12
Turns
Figure/ 11. DSG score comparison between ground truth prompt and agent generated prompt reported at each turn. The performance of all agents increase with increase in number of turns.
4. Run the VQA scorer over all 10 images per caption.
5. Choose best image of the ten by selecting the image with highest score.
6. Take average DINO (I2I) score for best image against GT image: this was found to be 0.7838.
7. Calculate the between before and after filtering out images via agent QA pairs: = +.02
In conclusion, this experiment demonstrates that by using the QA pairs from the agent in combination with the VQA score, we can improve image fidelity by filtering out images that do not follow the prompt. T21 models do not always follow a prompt exactly. They can make small errors or ignore a single detail while retaining all others. This is an inherent bottleneck of our agents; however, our results show that using the QA pairs from the agent-user interaction history allows us overcome this limitation.
G. Details on the Agent Interface
Below is a showcase of how users could interact with the belief graph and clarifications in a hypothesised interface, to better iterate their inputs, to reach a higher quality and satisfaction of outputs. This a crudely hypothesised, intentionally simple interface for the sake research, but could be iterated and improved upon in many ways depending on application and users.
28

---

Proactive Agents for Multi-Turn Text-to-Imape Generation Under Uncertainty
Algorithm 2 User-Agent Self-Play Algorithm
1: Input: Initial prompt Po, User u, Agent (with Po), moz turns
2: Output: Refined prompt P
3: Pf <
4: for turn id = 0 to max Jurns - do
5: action - a S.electAccion)))
6: question +- VerbalizeActionn (action)
7: answer < ww..AnswerQuestionnquesss
8: a. I.Transition answer)
9: Pf + a.prompt
10: end for
11: return Pf
1. Default state On load the app, there would be a text prompt input and space for output images, as is common across typical T2I interfaces. There would also be space for the user to view either clarifications from the model, or graph interface, part of the overall "input" section these would act further input for future model output iterations. See Figure 12 below reference.
NS.1
CUIPUTS
prompt
Couts
CLARFICATIONS
GRAPH
Reluel
Figure 12. Default state of a possible interface.
Output images, with Clarifications Once the user has submitted the prompt and the model has responded, there would be set of images, initial outputs from the users prompt. Below the input prompt would be set of "Clarifications" in its populated state. These clarifications would ask the user specific questions that would be necessary to increase the specificity of the prompt, for the model to get more accurate results aligned to the users intention, or to help the user realise their intention. Options would be given of the highest probability options for each Clarification, but the user could also fill in totally new option via free text field. Once answered by selection or text input, the clarifications would be added to the above, primary prompt for regeneration when the user selects. See Figure below as reference.
29

---

Proactive Agents for Multi-Turn Text-to-Imape Image Generation Under Uncertainty
! App
CUTPUTS
A dominating image of the Eiffel Tower with the Olympic rings prominently displayed the structure.
CLARFICATIONS
GRAPH
spran 23 castz
exectly
Olympic ringe
Midde
Borsom
Side
Typee here
Q: What style the Biffel Tower depicted
Protorellitic
Sightll painting
Skatch
Rendar
Type
Q: What the angle from which the Eiffel Tower is depicted? ?
angle
Q: What the overall lighting condition the image?
bighht
Epre
Inage
Ymage pere
Figure 13. Interface once prompt has been input with clarifications.
3. Graph Entities & Attributes Instead of the clarifications, the user could select to instead view a Graph by clicking Tab above the clarifications themselves. This graph would be populated with all Entities from the prompt explicit and implicit visually defined differently (in this diagram by the dotted line surrounds implicit entities, but is filled line when surrounding explicit entities). The graph layout will be structured, depicting relationships concentrically i.e. "on", "in" "under" for example, will become child entities, and be displayed within the parent entities' boundary. For example 'Mug' that has the relationship of "on's 'Table' entity, will sit within the boundary of 'Table', as also would 'Plate' if that had the same child-pane parent relationship.
Below the Graph would also be a list of 'cards' (i.e. boxed groups of information), one for each "explicit" or "implicit" entity. Within each card user could see the status of implicit explicit, and change this status to confirm or deny its presence. The user could also see list of "attributes" associated that entity, which the model has assumed. Each of these attributes could be changed by interacting with list of alternatives via drop down. These lists determined in terms of which items and order of items, based the probability by which the model sees them, ordered with highest first. This probability would be made clear to the user to define the order by seeing the percentage next to the label. See Figure 14 below as reference.
4. Graph Relationships The user would also be able to change the state of the Graph and Cards, to instead focus on the relationships between entities, by toggling to "Relations". In this state the user would be able to focus on two specific entities (e.g. "mug" and "table'), ), see the description the relationship (e.g. "the mug is sitting on the table") and desired change the relationship an alternative (e.g. 'on', changed to 'under') via drop down of options which the model determined as alternative options ordered by probability, per attributes. See Figure Figure 15 below as reference.
Once any of these changes made the user could initiate regeneration via the updated prompt to create new set of output images, which can then be further refined via the same method.
H. Details on User Studies for the Agent Interface
Below we describe the exact guideline definitions we shared with the user for user study.
30

---

Proactive Agents for Multi-Turn Text-to- Image Generation Under Uncertainty
1 App
CUTPUTS
dominating image Tower Olympic rings prominently displayed on the structure.
Craata
CLARFICATIONS
GRAPH
Ehtycrn
Daytinal
Theas
View
Erttie stuutu
<pex of tei:
Mgo
Ymage pere
Olecal Mage cusbur Ae1
uearca
Parks Ofyongge "oouch
Figure Interface with Graph displaying Entities, with cards below enabling user to change altributes associated each entity.
H.1. Hypothesized Frustrations
We presented participants with the following hypothesized frustrations related to T2I model usage:
1. Prompt Misinterpretation:
The model misunderstands complex relationships between entities in the input prompt.
2. Many Prompt Iterations: The model does not immediately generate what the user intends, requiring numerous iterative changes to the input prompt.
3. Inconsistent Generations:
The model reinterprets the input prompt differently between iterations, causing unwanted changes in the generated images.
4. Incorrect Assumptions:
The model makes incorrect assumptions assumptions when encountering gaps in the details provided in the input prompt, leading to undesired outputs.
Explanations of terms given users of:
1. "Entities" are single items that are intended to in the image e.g. "Cat" and "Ball", from "make sketch Cat playing with Ball"
2. "Prompt" means the text written to communicate the intended output image e.g. the sentence "make a sketch of Cat playing with Ball" is the "Prompt", also known as "Input"
3. "Iterations" are each set of different image outputs by the model, taken from a different input, or even the same input just regenerated
The question asked for each Frustration were: "Please score the below frustrations (or issues) that could be related to Text to Image Generation"." Rank in terms of how much they relate to your current usage, with your most commonly used model or app."
31

---

Proactive Agents for Multi-Turn Text-to-Inee Image Generation Under Uncertainty
1 App
CUTPUTS
dominating image prominently displayed on the structure.
Olympic rings
Craata
CLARFICATIONS
GRAPH
Eitycrnn
Daytina
Lighteed
Theas
Solta
Views
Ertiiis attittuues
242
Deuptive Greale senterr dusiced
Cooplnt Thu chou"
Ymage here
aeciean rece men ncr
Figure Interface Graph displaying relations between Entities, cards below enabling user change relationships between entities.
H.2. Hypothesized Features
We proposed the following features as potential solutions to address the identified frustrations:
1. Clarifications: The model would ask specific clarifying questions about uncertainties in the prompt. These details would then be incorporated into subsequent iterations. For example: "Is the cat playing with: 1. a ball of wool, or 2. tennis ball?"
2. Graph of Prompt Entities: A visual representation of all entities in the prompt graph, allowing to and edit attributes of each entity. E.g., seeing that the model has assigned "round," "small," and "wooden" as attributes to "table" and allowing the user to change them to "square" and "metal."
3. Graph of Prompt Relationships: A visual representation relationships between entities in the prompt, allowing users and edit these relationships. E.g., seeing that "donut" is "next to" "coffee" and allowing the user change the relationship to "on top of."
The questions asked for each feature were:
1. "How likely this feature is to help your current workflow if you had it now?". With response options of: "Very unlikely to help", "Unlikely to help", "Could help", "Likely to help", "Very likely to help"..
2. "How soon would this feature deliver value to your work?" with response options of: "Very soon / immediately", "Sometime, "Not very soon".
Image references were given for each Feature as listed out below:
32

---

Proactive Agents for Multi-Turn Text-to-Iman Image Generation Under Uncertainty
1. Clarifications:
App
dominating imaga
with Clympic rings
prominently displayed on structure.
muacty on the Tower the Olympkk iccated?
Whot wither Teaer de repanadier
londer
o Whol likging serkagee
Figure 16. Stimulus image in the survey to test the Model clarifications feature.
2. Graph of Prompt Entities:
App
OUTINT
chocolate glazed donut topped with chopped nuls sits plate with bite taxen of it, next a full coffee
graph
rmgg ponr
Yage porr
Dowe
Caffee
Atribuies
ertmee enuy ridee
sago
Figure 17. Stimulus image the survey test the Model Graph of Entities and Attributes feature.
33

---

Proactive Agents for Multi-Turn Text-to-Inen Image Generation Under Uncertainty
3. Graph of Prompt Relationships:
Apo
OUTNNT
chocolate glazed donut with chapped nuls sits plate with bite taxen of it, next a full coffee
Create
graph
regge poerre
rage poer
Reiat onctlp confidance contrit
onton
Figure 18. Stimulus image the survey to test Model Graph Entity Relations feature.
H.3. Human Study Results
Table 3 details the T2I usage frequency of the human subjects. Table shows the percentage of human subjects that reported different kinds of frustrations in their experience of using T2I. Table summarizes the results on expected speed of value delivered from different features of our agent prototypes. These results highlight the impact our contributions.
Table Breakdown of T2I usage frequency of 143 participants recorded
Usage Frequency
No. of participants
(%)
Many times a day Many times a week At least once week At least once month
13
9.1
30.8
25.2
50
35,0
Table 4. Reported User Frustrations with existing T2I processes (% of participants)
Frustration
V. Freq. (%)
Freq. (%)
Occas. (%)
V. Occas. (%)
No Issue (%)
Prompt Misinterpret.
19.6
43.4
23.1
Many Iterations
10.5
44.8
28
11.9
4.9
Inconsistent Gen.
11.2
20.3
39.9
21
7.7
Incorrect Assumptions
23.1
39.2
20.3
10.5
I. Details on Recruitment and Participant Consent
In this human user study participants were recruited with the following requirements:
1. Situated in North America and fluent English.
34

---

Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty
Table 5. Expected speed of value delivered from features (% of users)
Feature
Very soon / immediately (%)
Sometime(%o
Not very soon. (%)
Clarifications
57.7
37.2
Entity Graph
49.6
34.8
15.6
Relation Graph
41.8
44
14.2
2. An age of 18 above.
3. Have experience (i.e. at least once in the past month) using Text Image Al Generation tools (e.g. Mid Journey, Stable Diffusion, ImageFX, DALL-E, Adobe Firefly etc.), for any purpose (work, non-work, just for fun).
4. The survey must be completed on desktop computer not mobile phone.
5. Participants were recruited in North America and were compensated based on industry standards.
An internal review of the human human-study experiment was first created to determine potential risks and harm to participants and found be low risk and we were not collecting any personally identifiable information. The review found that there were two potential risks. The first was the risk of study with experimental technology: T21 AI models still under development and may produce inaccurate, biased, potentially offensive content. Participants must caution when relying on the model's output and always exercise critical thinking. The second risk regarded potential biases. The model may reflect the biases present in the data it was trained on. These biases can manifest in various ways, such as favoring certain groups or stereotypes. By using this model, participants acknowledge that they have read and understand this disclaimer. They also agree to use the model responsibly and ethically.
In response the findings of the internal review, risk mitigation was performed by serving generated images rather than having participants generate images themselves using the model. All image-prompt pairs and all generated images shown to users were pre-approved disclaimer and self consent form were placed the beginning of the human-study survey to let participants know the type of questions they would be answering.
way to reduce risk and remove any overtly harmful or offensive content. Additionally
35

---

Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty
Template of Human Rater Task 1: Evaluation of Issues in Individual Questions
Example Dialogue Analysis:
Instructions: Pretend you are in the following scenario you are asking an Al model to create an image you have in mind, this image is displayed on the left. You first prompt the model with short non-detailed description of the image, this description is called "original prompt" and is below the goal image on the left.
The model proceeds to ask you various questions to understand the specificities of the image you are trying to create. The dialogue between you and the Al model is shown in the middle column. You will go through the dialog turn by turn and answer the same rater questions about each turn so just focus on the highlighted turn rather than the entire dialog.
Your job in this task is to rate the clarity, soundness and efficiency of the highlighted question in the dialogue. This question was asked by the Al model in order to generate an image similar to the one you had in mind. Provide your rating by answering the questions in the rightmost column.
Goal Image
Human Al dialogue
Rater Questions
Q1. Which way is the cat facing?
Select one or more issues for this particular turn:
A1. To the left.
[] The question completely unclear and ambiguous
Q2. What type of butterfly is on the cat?
The question irrelevant to the goal image and prompt
A2. Not sure what type but it is a blue butterfly with black around the edges of the wings.
[] The question is not helpful; doesn't gain any new information from what was previously asked
Original Prompt
Q3. Where is the butterfly positioned on the cat's face?
Figure 19. An example of the template presented to human raters. Human raters are asked to mark any issues question contains that could pose a disturbance to user. Approximately 8k questions per Agent are rated. The results are shown Figure 4.
36

---

Proactive Agents for Multi-Turn Texte Generation Under Uncertainty
Template of Human Rater Task 2: Evaluation of Similarity between the generated image and the Human-Al dialog and the original prompt.
Example Image Analysis:
Instructions: Pretend you in the following scenario you are asking Al model to create an image you have in mind, this image is displayed the left. You first prompt the model with a short non-detailed description of the image, this description is called "original prompt" and is below the goal image on the left.
The model proceeds to ask various questions to understand the specificities of the image you are trying to create. The dialogue between you and the Al model is shown in the middle column.
Based on the original prompt and the human and Al dialogue the Al model produces final image shown in the 3rd column.
Your job in this task to rate the produced image based on how well it fits the goal image and separately how well it fits the prompt and the dialogue. Do this by answering the questions in the rightmost column.
Goal Image
Human Al dialogue
Produced Image
Rater Questions
Key 1: Very Close,
Q1. Which way is the cat facing?
2: Fairly close some fair differences, 3:
Equally close and different 4: Fairly different some similarities, 5: Very different no similarities
A1. To the left.
Q2. What type of butterfly is the cat?
Original Prompt
A2. Not sure what type but is blue butterfly with black around the edges of the wings.
Rank how well the produced image corresponds to the original prompt and the dialogue? (1 Scale)
A fluffy gray and white cas with a butterfly on its face.
Q3. Where is the butterfly positioned on the cat's face?
A3. On the cat's nose.
Figure 20. An example of the template presented to human raters. Human raters asked to rank the correspondence of each image to the agent-user dialog and original prompt. Approximately 1.5k image-dialog pairs are rated using human raters, Results Figure
37

---

Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty
Template of Human Rater Task 3: Evaluation of the generated image similarity to the goal image.
Example Image Comparison:
Instructions: Pretend you are in the following scenario - you are asking an model to create an image you have in mind, this image is displayed on the You first prompt the model with short non-detailed description the image, description called "original prompt"
below goal image the left.
Your in this task is to compare the two model's produced images against each other given the goal image and original grompt. Do this by answering the questions in the rightmost column,
Goal Image
Model 1:
Model 2:
Model 3:
Questions
Rank which produced image matches best the goal image in terms content?
Rank which image matches best to the goal Image in terms of style and aesthetics?
Original Prompt flusty grey white sat with butterfly on its face.
Figure 21. A template of the task presented human raters. Human raters asked the images produced by the three proposed multi turn agents and single-turn T21 model against a ground truth image for which the original prompt was derived and the answers to the agents questions
derived. Approximately 550 image-dialog pairs (250 from Imageln Words, 250 from COCO-Captions and 21 from DesignBench) per agent are rated using 3 human raters. The generated images were presented a random order and were unlabeled and the human rater tasked with ranking the images from best to worst. The results from the study shown in Table 1.
38


Reviews:
[{"review_id": "1zG0vWoskG", "reviewer": "Reviewer_F62f", "summary": "This paper addresses the issue of suboptimal image generation caused by vague or incomplete prompts provided by users to text-to-image (T2I) generators. It introduces proactive T2I agents designed to improve image generation by actively asking users clarification questions. These agents maintain their understanding in the form of a belief state, which users can view and modify directly. Additionally, the paper presents a scalable, automated evaluation benchmark for assessing T2I systems. Experimental results indicate that 90% of human participants found the proactive agents and editable belief states beneficial, with the proposed approach significantly enhancing the quality of generated images, as evidenced by higher VQAScores.\n\n## update after rebuttal\nThe reviewer still recommends weak acceptance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: “Is the capability of the whole pipeline limited by the prompt-following capabilities of the text-to-image model?” Also in Experimental Designs Or Analyses: “misalignment between the textual exploration and the image generation.”\n\nThe current agent prototypes call off-the-self T2I models directly, i.e., treating T2I API as tools. The benefit is that it is seamless to switch to better T2I models when they become available. \n\nWe have now included a new module under our agent framework to further improve text-image alignment with vision feedback. The key idea is to use the existing <agent question, user answer> history and ask a VLM those agent questions on the generated image. If the VLM’s answers do not match the corresponding user answers, the agent can refine the prompt (such as adding “ensure the color of the rabbit is grey”; this refinement component can be replaced with better prompt optimization methods) and call T2I models to re-generate the image. We are testing this approach and will include more results in the updated paper.\n\n\n> Q2: Related work\n\nThank you for sharing the related work. This paper proposes to use clarification questions to resolve ambiguities (a prompt has multiple meanings), but we aim to use clarification questions to resolve underspecification (the prompt is not ambiguous, but lacks information to fully describe the image). We will include this paper in the related work and note the difference.\n\n> Q3: Clarity of agent descriptions.\n\nThank you for pointing this out. The main paper has page constraints and we had to make certain tradeoffs. The core of this paper is the new framework and overall design of proactive agents and automated evaluation. All 3 agents share the implementation of belief parsing (Algorithm 1), belief transition (Section 4.1) and principles of asking questions to collect information (Section 4.2). Those are detailed in the main paper. However, we recognize that readers may also be curious about the specific implementations. We will consolidate the relevant sections in appendix and add the important details back to the paper, including (1) Ag1 selects actions by maximizing the approximated information gain of questions, so that the questions can reduce the entropy of belief graphs; (2) Ag2 uses in-context learning and generates questions conditioned on user prompt, belief graph and conversation history; (3) Ag3 uses the LLM to generate questions conditioned on the principles of asking questions and conversation history.\n\nWe will fix the formatting of tables.", "claims_and_evidence": "* Claim: Proactive T2I agents improve user interaction\n* Claim: Belief state interpretability and editability\n\nThese two claims are supported by the experimental result indicating that 90% of human participants found proactive agents and the belief state useful, providing clear and convincing evidence.\n\n* Claim: Improvement in image alignment\n\nThe work shows improved quality of generated images, supported convincingly by reported significant increases in metrics such as VQAScores.", "methods_and_evaluation_criteria": "The evaluation protocol that guides the proposed DesignBench, as discussed in Sec 5.1, is sound and well-defined. The proposed DesignBench offers a good example of how to evaluate the performance of T2I systems with proactive agents.", "theoretical_claims": "No theoretical claims are made in the paper.", "experimental_designs_or_analyses": "The overall experiment design is sound and well-defined. However, there are some concerns in the design of the agents in the prposed work:\n\nAg2 and Ag3 use LLMs without vision input, which indicates that the exploration is only conducted in the textual space. This might cause misalignment between the textual exploration and the image generation.", "relation_to_broader_scientific_literature": "The proposed work improves the text-to-image workflow by introducing proactive agents that can ask questions to the user to improve the quality of the generated images. The work builds on prior text-to-image generation works and focuses on a novel aspect of it.", "essential_references_not_discussed": "Asking for clarifications for text-to-image generation is not a new idea. The work missed this related work [1] that also investigated into the ambiguity in user's prompt and asked the user for clarifications.\n\n[1] Is the Elephant Flying? Resolving Ambiguities in Text-to-Image Generative Models. https://arxiv.org/abs/2211.12503", "other_strengths_and_weaknesses": "The writing is a bit unclear in the explanation of each agent. The reader needs to look at the appendix to understand the design of each agent.", "other_comments_or_suggestions": "One minor detail: the bottom lines of Table 1 and Table 2 are missing.", "questions_for_authors": "Is the capability of the whole pipeline limited by the prompt-following capabilities of the text-to-image model (i.e., if given a prompt that has a lot of details from clarifications, does the model always generate an image that is aligned with the prompt)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UXm4c3PeCH", "reviewer": "Reviewer_CQhi", "summary": "This paper proposed a proactive text-to-image agent designed to mitigate the issue of uncertain prompts by allowing users to do multi-turn interactions. The key contributions of the paper include:\n\n1. Belief Graphs – A structured representation of model uncertainty, allowing users to visualize and edit entities, attributes, and relationships.\n2. Proactive Questioning – The agent actively seeks clarifications from users to refine image generation.\n3. New Benchmark (DesignBench) – A dataset created to test agent performance in complex scenes with both long and short descriptions.\n\nThe study demonstrates that multi-turn agents significantly outperform standard T2I models in both automated and human evaluations, achieving at least twice the VQA score within five interaction turns.\n\n## Update after rebuttal\n\nThe authors have addressed all of my concerns. I vote to accept the paper and appreciate their detailed and thoughtful responses.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> baseline\n\nWe used only one single-turn T2I baseline because we want to keep the T2I backbone consistent between the baseline and all agents. To ensure the consistency, we will run all agent experiments with a different model (such as the suggested ones) and we can definitely add this comparison to the paper. \n\n\n> the clarity of the descriptions of the three agents\n\nPlease see Q3 in the rebuttal for Reviewer F62f.\n\n> Why are Agent 1 and Agent 2 designed in the current setting?\n\nWe designed the two agents to compare rule-based approach and LLM-based question-asking without explicit rules. Ag1’s action selection is purely rule-based: it maximizes the approximated information gain of questions (weighted by importance scores in the belief graph) and the model can only ask about the existence of an entity, the attribute of any entity or the relation between any pairs of entities in the belief graph. Ag2 also relies on the belief graph but it does not compute any heuristic scores and instead puts the belief graph in the context to prompt an LLM to ask questions. \n\n> why should Agent 1 ask whether the entity \"cake\" is present in the image, as shown in Fig. 5 of the supplement? Does this occur before or after generating the first image?\n\nBecause Ag1 is rule-based and checks both the probabilities and importance scores, it may try to confirm the existence of an entity the user mentioned, because the importance score for that entity is very high and the probability is less than 1. The reason that the probability is less than 1 is because sometimes, when an entity is mentioned, the image doesn’t necessarily have to include the entire entity. Moreover, the belief graph parsing is not perfect, so errors can occur since Ag1’s question purely depends on the belief graph. In our current implementation, this occurs before generating the images.\n\n> Does the performance of direct LLM prompting—where an image is given, the LLM describes it as a prompt, and a T2I model generates an image in a single turn—outperform multi-turn iteration?\n\nIf the LLM describes the image in detail and the T2I model has good text-image alignment, it should outperform the multi-turn approach. This is because the multi-turn approach’s goal is to eventually collect all information about the image, and the detailed description is the ground truth of what information should be collected.\n\n> How does the belief graph scale to complex scenes with dozens of entities? Are there practical limitations on the number of entities and relationships that can be handled effectively?\n\nThe belief graph can scale to dozens of entities as long as the belief graph parsing prompts and every entity together with their attributes in the belief graph fits the context length and generation length limits of the LLM. Practical limitations: as shown in Algorithm 1, the time complexity of belief parsing is linear in the number of entities and the number of relations, but in practice, we make parallel calls to the LLM to generate attributes of entities and relations in parallel. So the time complexity is limited by the requests per second the LLM can afford and how many parallel threads / processes the python environment can afford.\n\n> To what extent does the T2I model adhere to the belief graph representation?\n\nThe belief graph describes the state of the agent, not the T2I model. The belief graph entails a distribution over possible prompts with different combinations of entities, attributes and relations. The distribution over the images the agent generates will adhere to the belief graph relatively faithfully if the T2I model’s T2I alignment is good. We also describe an approach to further enhance the alignment between agent behaviors and user intents in Q1 rebuttal to Reviewer F62f.\n\n> If the generated image does not follow the belief graph and the clarification question fails to identify the issue, how can the user provide feedback to help the agent improve the generated image?\n\nThe user can modify the current prompt the agent has and inspect/modify the belief graph to ensure it aligns with the user intent. Admittedly, the T2I models may still fail to generate images that align with the text descriptions. We described an approach to further enhance the alignment in Q1 rebuttal to Reviewer F62f. It is still possible that the VLM might not be able to tell whether the answers to questions conditioned on the generated image match with the user answers. These are intrinsic limitations of the current T2I models or VLMs, and improving those models are out of the scope of this paper. However, if there exist improved T2I models and VLMs, we can seamlessly integrate them into our agents.", "claims_and_evidence": "1. Claim: Multi-turn interaction improves T2I alignment\n    \n    Evidence: Table 1, Fig. 2 and 3 demonstrate that multi-turn agents outperform single-turn T2I models across various datasets, with at least a 2x improvement in VQAScore.\n    \n2. Claim: Belief graphs effectively communicate and refine uncertainty\n    \n    Evidence: Over 85% of human participants found belief graphs helpful.\n    \n3. Claim: Proactive questioning enhances user experience\n    \n    Evidence: Human evaluations indicate that 90% of participants expect interactive clarifications to be beneficial.\n\nIssue: The main contribution of this work is the proposal of the belief graph and multi-turn interaction with users. However, the comparison includes only one baseline focused on single-turn generation. I suggest expanding the comparison to include more baselines, such as Recraft V3 or Midjourney v6, which have been reported to outperform Imagen 3 in numerical reasoning cases in the Imagen 3 paper. This would provide a more comprehensive evaluation of general performance improvements and adaptability compared to current state-of-the-art methods.", "methods_and_evaluation_criteria": "Yes, the paper employs 2 evaluation methods for evaluating how the generated image align with user’s intent (prompt)\n\n1. Automated Evaluation – Uses a simulated user to converse with a T2I agent\n2. Human Studies – Participants rate the efficacy of the proposed framework with human subjects", "theoretical_claims": "The paper does not propose new theoretical results but extend concepts from belief state representations.", "experimental_designs_or_analyses": "- The experiments are well-structured, covering both automated and human evaluations.\n- The use of DINOv2 embeddings for image similarity and VQAScore for text-to-image alignment are reasonable choices.", "relation_to_broader_scientific_literature": "This work contributes to clarify the prompt before generation which would enhance the efficiency for trial and error on prompt engineering", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "- Strengths\n    1. The idea to proactively clarify user input is novel and helpful for efficiently generate a target image.\n    2. Leveraging LLM as a user to provide clarification on a pre-given image is an efficient strategy.\n- Weaknesses\n    1. The description of the three agents is difficult to understand in terms of their differences, and the information is scattered across too many sections, including Section 4.3, Supp. C, and E. I suggest improving the clarity of the descriptions of the three agents in the main paper.", "other_comments_or_suggestions": "None", "questions_for_authors": "- The motivation for including Agents 1 and 2 in the comparison is unclear. \n   - Why are Agent 1 and Agent 2 designed in the current setting? \n   - Additionally, why should Agent 1 ask whether the entity \"cake\" is present in the image, as shown in Fig. 5 of the supplement? Does this occur before or after generating the first image?\n- Does the performance of direct LLM prompting—where an image is given, the LLM describes it as a prompt, and a T2I model generates an image in a single turn—outperform multi-turn iteration?\n- How does the belief graph scale to complex scenes with dozens of entities? \n   - Are there practical limitations on the number of entities and relationships that can be handled effectively?\n- To what extent does the T2I model adhere to the belief graph representation?\n- If the generated image does not follow the belief graph and the clarification question fails to identify the issue, how can the user provide feedback to help the agent improve the generated image?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fw48Dn479U", "reviewer": "Reviewer_tR3z", "summary": "This paper addresses the challenge of underspecified user prompts in text-to-image (T2I) generation by introducing proactive agents capable of multi-turn interactions. These agents actively seek clarification through targeted questions and utilize a \"belief graph\" to represent and refine their understanding of user intent. The proposed approach aims to bridge the gap between user expectations and model outputs. Empirical evaluations show that the method achieves a VQAScore twice as high and is rated as helpful by 90% of human participants.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The universal prompt design: It seems like the design of the prompt lacks consideration of the alignment and prompt-following capabilities of the text-to-image model. From my personal user experience with T2I models, the correct wording and structure of the prompt may also greatly influence the quality of the generated image.\n\nWe have added a new method to take into account the T2I alignment of T2I models. Please see more descriptions in Q1 rebuttal to Reviewer F62f\n\n> writing \n\nThank you for pointing these out. We will rewrite the long sentences and make the writing more accessible.", "claims_and_evidence": "The claims made in the submission supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed method focuses on utilizing an editable belief graph for human-AI interaction. The idea is simple yet seems reasonable and effective.", "theoretical_claims": "The paper have no theoretical claims.", "experimental_designs_or_analyses": "The analysis consists of two parts: the VQAScore and human opinions. It can be considered sound.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\n- Important problem and practical value: The problem of underspecified prompts is common and significant in user-AI interactions. A common user without prompt-engineering training always finds it hard to convey their intentions efficiently to the models, causing frustration. This method effectively addresses the common issue of vague or ambiguous prompts, leading to more accurate and user-aligned image generation.\n\n- Effectiveness: The authors employ both human studies and automated evaluations to assess the effectiveness of their approach. Notably, over 90% of human subjects found the proactive agents and belief graphs beneficial to their T2I workflow. Additionally, the agents achieved at least twice the VQAScore compared to standard single-turn T2I generation.\n\nWeaknesses:\n- The universal prompt design: It seems like the design of the prompt lacks consideration of the alignment and prompt-following capabilities of the text-to-image model. From my personal user experience with T2I models, the correct wording and structure of the prompt may also greatly influence the quality of the generated image.", "other_comments_or_suggestions": "The writing of the paper needs improvement. Please try to avoid redundant wording and excessively long sentences. This issue appears even in the abstract, e.g., \"As a result, users **commonly** have to **painstakingly** and **repeatedly** refine their prompts.\" This writing style makes the paper difficult to read.", "questions_for_authors": "See weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "01rV225DbW", "reviewer": "Reviewer_dRg9", "summary": "This paper focuses on the information-missing issue of existing AI applications, specifically image generation in the paper. Instead of passively waiting for humans to revise the prompt, this paper proposes a proactive design such that the agent system could proactively interact with the users to get the missing information. This paper has five main contributions:  (1) the introduction of belief graphs for uncertainty modeling; (2) a proactive T2I agent prototype; (3) an evaluation pipeline; (4) a benchmark; and (5) extensive experiments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The current brief graph requires heavy human design, which will limit the generalization of the proactive agent system to other domains other than the image generation task. How do you prevent that?\n\nGenerating the belief graph only requires some few-shot examples, which are not difficult to write. Writing those examples offers humans the opportunity to insert their expert knowledge, so that the LLM’s behavior can be controlled. Moreover, for new domains, it is sometimes not necessary to re-design the belief graph parsing approach. We tried the same belief parsing approach without modifying anything for creative writing tasks, and the belief graph we obtained from the models is reasonably good, e.g., it is able to identify the characters, their attributes like clothing, super power etc and relations between characters. One future direction to further lower the effort of human design is to perform meta learning, where we give the model examples of different domains, and ask it to generalize to new domains.\n\n> In real application, how do you guarantee that the users will clearly answer the questions raised by the system?\n\nWe cannot “guarantee” the user behavior but the agents can guide users. The questions the agents ask often contain several options for the answer. For example, the agent may ask “What is the color of the rabbit? (a) white, (b) grey, (c) brown, (d) mixed colors.” The user can choose an option (this guarantees that the answer is clear), or answer with words directly. \n\n> This proactive design will introduce an extra efficiency issue. The efficiency might be a limitation for the proactive agent system to be useful in real applications. How do you prevent that?\n\nWe can use parallelism and selective generation to prevent efficiency issues. The agent is a system that operates in multi-threads/multi-processes. In the agent prototypes we developed, the efficiency of belief parsing was significantly improved by generating attributes and relations for different entities in parallel. The T2I models can be called while the next belief state and action selection are in progress. Our framework also supports the development of more sophisticated agents in real applications, which can incorporate strategies to selectively generate partial belief states that are important for showing to the users. This can be especially useful if the belief state is very large.\n\n\nWe will fix the formatting of tables.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A, no theoretical claim", "experimental_designs_or_analyses": "Yes, I checked the dataset, experiment results, and case studies.", "relation_to_broader_scientific_literature": "The proactive agent design has been used in the general agent design but not in the image generation scenario, to the best of my knowledge.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strength:\n1. A proactive agent is a sound solution for solving the information-missing problem and might be an essential step of future agent design.\n2. The paper is clearly written and easy to follow.\n3. The experiments and analysis is comprehensive\n\nLimitations:\n1. Some detailed design choices require more justifications; for details, please refer to the questions for the authors section.\n2. The efficiency might be a limitation for the proactive agent system to be useful in real applications.", "other_comments_or_suggestions": "The bottom lines of all tables are missing. Better to add them.", "questions_for_authors": "1. The current brief graph requires heavy human design, which will limit the generalization of the proactive agent system to other domains other than the image generation task. How do you prevent that?\n2. In real application, how do you guarantee that the users will clearly answer the questions raised by the system?\n3. This proactive design will introduce an extra efficiency issue. How do you prevent that?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received 4 reviews and 2 ethics reviews. The reviewers found the problem tackled in the paper relevant (tR3z), the idea presented sound (dRg9), the claims well supported (F62f, CQhi, tR3z, dRg9), and the evaluation well designed and overall convincing (F62f, CQhi, dRg9). The reviews raised concerns about: 
1) the paper's presentation, in particular, the clarity of the agents description and motivation (F62f,CQhi, tR3z); 
2) the experiments, which included only one T2I model (CQhi); 
3) some design choices, which did not appear adequately justified (F62f, dRg9) - e.g. the exploration being conducted in the text space; 
4) the design, which disregarded the alignment and prompt-following capabilities of the T2I model (tR3z, F62f). 

The reviews also highlighted some missing related work (F62f), and questioned the efficiency of the approach (dRg9). 
The ethics reviews raised concerns about the lack of details on participants, their concern, and the overall recruiting process (DJcV, sCzK). 

The authors' response addressed most of the reviewers' questions, comments and concerns. In particular, the response clarified the concerns related to presentation by answering the reviewers' questions in detail, adequately discussed the design choices questioned by the reviewers, and explained how to increase the efficiency of the approach. The response also positioned the paper w.r.t. the missing reference. The authors also summarized the review/response process for the ACs. 

The AC takes the authors' message into consideration when discussing with the reviewers and making the recommendation. After discussion, there is consensus that the paper addresses an important problem with high practical value, and that the proposed approach is sound and well validated. Therefore, the AC recommends to accept and encourages the authors to add the clarifications requested by the reviewers as well as the information requested in the ethics reviews (participants recruitment and consent) to the camera-ready version of the paper.
Decision: Accept (poster)