Preference Adaptive and Sequential Text-to-Image Generation
Authors: Ofir Nabati
Guy Tennenholtz
ChihWei Hsu
Moonkyung Ryu
Deepak Ramachandran
Yinlam Chow
Xiang Li
Craig Boutilier
Venue: ICML 2025 poster (2025)
Keywords: text-to-image generation
reinforcement learning
diffusion models
multi-modal large language models
human-in-the-loop
sequential decision making
Primary area: deep_learning->large_language_models
TL;DR: We introduce PASTA, a multimodal LLM based RL agent, for sequential, adaptive text-to-image generation. PASTA improves user intent via iterative prompt refinements. We also release a new dataset for sequential text-to-image generation.

Abstract: We address the problem of interactive text-to-image (T2I) generation, designing a reinforcement learning (RL) agent which iteratively improves a set of generated images for a user through a sequence of prompt expansions. Using human raters, we create a novel dataset of sequential preferences, which we leverage, together with large-scale open-source (non-sequential) datasets. We construct user-preference and user-choice models using an EM strategy and identify varying user preference types. We then leverage a large multimodal language model (LMM) and a value-based RL approach to suggest an adaptive and diverse slate of prompt expansions to the user. Our Preference Adaptive and Sequential Text-to-image Agent (PASTA) extends T2I models with adaptive multi-turn capabilities, fostering collaborative co-creation and addressing uncertainty or underspecification in a user's intent. We evaluate PASTA using human raters, showing significant improvement compared to baseline methods. We also open-source our sequential rater dataset and simulated user-rater interactions to support future research in user-centric multi-turn T2I systems.

Preference Adaptive and Sequential Text-to-Image Generation
Ofir Nabati' |
Guy Tennenholtz'
ChihWei Hsu
Moonkyung Ryu' Deepak Ramachandran Yinlam Chow" Craig Boutilier!
Xiang Li
Abstract
such, single-turn T2I generation may fail to encapsulate a user's nuanced and evolving image conception. This is especially true for complex or abstract concepts, where a user's initial prompt may not suffice achieving a desired visual representation.
We address the problem of interactive text-to- image (T2I) generation, designing a reinforce- ment learning (RL) agent which iteratively im- proves set of generated images for user through a sequence of prompt expansions. Using human raters, create novel dataset of sequential pref- erences, which we leverage, together with large- scale open-source (non-sequential) datasets. We construct user-preference and user-choice mod- els using an EM strategy and identify varying user preference types. We then leverage large multimodal language model (LMM) and a value- based RL approach to suggest an adaptive and diverse slate of prompt expansions to the user. Our Preference Adaptive and Sequential Text-to- image Agent (PASTA) extends T2I models with adaptive multi-turn capabilities, fostering collabo- rative co-creation and addressing uncertainty or underspecification user's intent. We evaluate PASTA using human raters, showing significant improvement compared to baseline methods. We also open-source our sequential rater dataset and simulated user-rater interactions to support future research in user-centric multi-turn T2I systems.
Addressing this challenge requires moving beyond the paradigm of one-shot T2I generation towards a more it- erative and interactive process (von Rütte et al., 2023; Wang, 2024; Liu et al., 2024), e.g., a collaborative setting where an assistive agent interacts with a user, guiding them through series of refinements of their initial prompt. This inter- active setup could allow for a more nuanced elicitation of the user's intent, gradually shaping the generated images towards desired outcome.
To this end, we introduce a Preference Adaptive and Sequential Text-to-image Agent (PASTA), which learns from user preference feedback to guide the user through sequence of prompt expansions, iteratively refining the generated image. This sequential approach (see Figure allows users to articulate their vision more completely by gradually reducing uncertainty underspeciffiation in their original prompt. Our framework leverages the power of large multimodal language models (LMMs) (Gemini-Teamm 2024; Achiam et al., 2023) and reinforcement learning (RL) (Sutton, 2018; Fan al., 2023) facilitate user-adaptive co-creation.
1. Introduction
Our approach for PASTA involves multi-stage data collec- tion and training process. We first collect multi-turn interac- tion data from human raters with a baseline LMM. Using this sequential data, as well as large-scale, open source (single-turn) preference data, we train a user simulator. Par- ticularly, we employ an EM-strategy to train user prefer- ence and choice models, which capture implicit user prefer- ence types in the data. We then construct a new large-scale dataset, which consists of interactions between simulated user and the LMM. Finally, we leverage this augmented data, encompassing both human and simulated interactions, to train PASTA - our value-based RL agent, which presents a sequence of diverse slates of images to a user. PASTA
Advances in text-to-image (T2I) generation, fueled by pow- erful diffusion models (Croitoru et al., 2023; Gu et al., 2023; Rombach et al., 2022: Saharia et al., 2022; Yang et al., 2023; Yu et al., 2022: Zhang et al., 2023; Liang et al., 2024), have unlocked unprecedented possibilities for image generation, as users can readily translate textual descriptions into stun- ning visuals. That said, capturing precise user intent remains major challenge (Wu et al., 2023; Liang et al., 2024). As
'Google Research *Google DeepMind. Correspondence to: Ofrr Nabati <ofirabbai gomi, Guy Tennenholtz <guytenn comi.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
'Both human and simulation data is open-sourced to sup- port research on multi-turn T2I generation. Link to the dataset: https//wwwwkaggle com//ttaset//googlaa/astttt

---

Preference Adaptive and Sequential Text-to-Image Image Generation
A white cat perched on a fallen log the forest
A white cat looking out from behind tree in the forest
A white staring at a wateriall cascading through colorful forest
A white cat looking playful and mischievous In a vibrant forest
A white cat eating fish
cartoon white cat
A white cat
A white with a butterfly on its nose forest
while cat forest with glowing, magical trees
while cal forest with rainbow mushrooms
white cat colorful and vibrant forest
A white cat in
cute white cat
3
Figure 1:
An illustration of an agent-user user interaction with L 4 prompt expansions at each step, and images per prompt expansion. The user selection is outlined in blue. The agent presents prompt expansions based on the user's previous responses to maximizes the expected cumulative user satisfaction (i.e., value). See Appendix G for additional examples using M 4 images for T. prompt expansions.
interacts with the user and sequentially refines its generated images to better suit their underlying preferences.
Large Multimodal Models (LMMs) (Gemini-Team, 2024; Achiam et al., 2023) use Transformer models (Vaswani, 2017), and are trained to predict the next token in a se- quence, where tokens can represent textual or visual infor- mation. These models learn joint representations of text and images, making them useful for prompt engineering in in- teractive T2I systems, as they allow to modify generated outputs based on previously generated responses.
Our contributions are follows. (1) We formulate the prob- lem of iterative prompt expansion using LMMs and T21 models. (2) We collect new, large-scale user-agent inter- action dataset for the multi-turn prompt expansion setup. Our dataset consists of almost 40,000 interaction rollouts. (3) We train user utility and user-choice models with this dataset, creating a simulator that used to generate addi- tional simulated data. (4) Finally, we utilize our datasets to train PASTA, our vasue-based RL agent, demonstrating its effectiveness through comprehensive human evaluations, with significant improvements in user satisfaction compared to baseline LMM.
Reinforcement Learning (RL) (Sutton, 2018; Fan et al., 2023) frameworks train agents to make a sequence of de- cisions in an environment maximize cumulative reward. Value-based RL methods learn value function that esti- mates the expected cumulative reward for taking a particular action in given state. In our work, an RL agent learns to select actions (prompt expansions) that lead to higher user satisfaction.
2. Background and Problem Setup
2.2. Problem Formulation
We begin with a short background on diffusion models, LMMs, and RL. We then formulate our problem setup.
We consider an interactive, sequential T21 decision problem in which a user engages with an agent, visualized in Figure 1. At time = 0, the user issues an initial prompt (e.g.. "A white cat") intended to capture target image. At each turn 2 the agent proposes L prompt expansions, which are fed to T2I model. The T21 model then generates M images for each prompt. The user sees the slate of M x L images and selects the set of M images (corresponding to one of the prompt expansions) that best reflects their preferences. The process repeats for up to turns.
2.1. Background
Diffusion models (Ho et al., 2020; Ho & Salimans, 2022:
Croitoru et al., 2023; Gu al., 2023; Rombach al., 2022: Saharia et al., 2022; Yang al., 2023; eu al., 2022; Zhang et al., 2023; Liang et al., 2024) have become a prominent approach for T2I generation. These models add Gaussian noise to an image until it becomes pure noise, and are then trained to reverse this process, iteratively removing noise to reconstruct the image. This denoising process is guided by prompt, enabling the generation of images that semantically align with the textual description.
Formally, the problem (U, G, R, vo). where U set of user types, reflecting different preferences; P is of feasible
given by tuple
2

---

Preference Adaptive and Sequential Text-to-Image Generation
text prompts; I is set of feasible images; G: P + Az is T21 generative modef?; C : u x P x IML [u is user choice function"; R: U x P I + [0.1] is user utility function; H is the horizon: and Yo H Auxp is a distribution of users and initial prompts. At time t = 0, a user-prompt pair u, Po E U x P is sam- pled from the initial distribution vo- At each time I S H the agent selects slate of L prompt ex- pansions Pa = {pe,t E P) and feeds them to the T21 model G. which generates M images for each prompt,
3. PASTA: Preference Adaptive and Sequential Text-to-image Agent
We solve the sequential prompt expansion problem using RL. Our Personalized And Sequential text-to-image Agent (PASTA) engages with a user to adapt the prompt to their preferences, and maximize (latent) user utility. The user's type u E l is unknown to the agent and must be inferred during the interaction. This framework is related meta- RL (Wang al., 2016; Duan et al., 2016; Finn et al., 2017; Zintgraf et al., 2020), where each episode (or meta-episode) samples a latent MDP from predefined problem distribu- tion. In our setting, the agent must adapt within H steps to the unknown MDP, and optimize the reward accordingly. This requires balancing exploration and exploitation: the agent must take actions that, on the one hand, provide im- ages that reflect (its estimate of) the user's preferences, and the other, improving its estimate of those preferences by exploring other types of expansions/images This can be viewed as an implicit form of preference elicitation (Keeney, 1993; Salo & Hamalainen, 2001; Chajewska et al., 2000; Boutilier, 2002; Meshi et al., 2023).
{ (Tmme. ~ ((pr:)) )-
t=1 The user observes the slate of M L images, and selects set of M images corresponding to one prompt, and according the choice We
function C, i.e., Ce C(u, po, (I.IA...,,,MM... assume a (latent) reward ~ R(u. Pu, I,,,,, ................................ IM,)).)) reflecting user u's satisfaction with the selected images.
2.3. RL Formulation
Our problem can be formulated as a latent contextual MDP (Hallak et al., 2015; Kwon et al., 2021), where U/ is the latent context space. The state space consists of the user-agent interaction history, with the state (history) hy at time t given by:
3.1. Candidate Action Generator And Selector
The state space of user interaction histories serves as suffi- cient statistic (i.e., belief over users' types (Aberdeen et al., 2007)). To solve our problem effectively, each interaction with a user should provide the agent sufficient information about the user (e.g., through value of information (Boutilier et al., 2003)). This requires the action space be rich and diverse enough to enable information gain.
time step
he-
Po initial prompt agent action A1
L.M Umm.e. 1.m.t.Jeee..msst Im- CD
transition
time step
A straightforward approach to action space design uses an LMM to construct action candidates (in our case, prompt expansion candidates). Specifically, we use an LMM to process the current interaction history and generate broad set of Le > L candidate prompts from which a slate of L prompts can be chosen. Generating large set of can- didates has been shown to introduce diversity and induce exploration (Tennenholtz et al., 2024). Instead of relying on given LMM to directly output L prompts, we encourage diversity by generating Le prompts, and then selecting the L-subset our agent deems optimal. This, in turn, expands the effective action space our agent can leverage during training.
(pra)e I.
L.M Umeeseeatmma
agent action At
tramsition
the action space is any selection of L prompts, transitions are induced by the user choice model C and the T21 model G, and reward is given by a user utility function R. See Ap- pendix A for full description of this latent MDP.
A stochastic policy : H teraction histories to distribution over slates of L prompts. The value of policy π is its expected cu- mulative sum of rewards over users and initial prompts, i... V = Euwwo Σ/ u.,]] An optimal pol- icy n² E arg m x. maximizes the value. The state- action value function for any h E H and P E pL is T. (h, P) = En-ry E#=re Tor u, h4 = h Pt P,T
Apt. maps in-
Formally, we structure our policy class using a candidate generator policy TE H + Aptc and a candidate selector policy as phke + Apz. We first sample a set of Lc can- didates [e Le ~ (h,))) and then select prompts from
those candidates, i.e., P.- = {prc ~ Tg ((. We assume TO to be fixed, capable LMM, and focus here on training the candidate selector TS.
2 MMoee generally, G can be a function of user u or interaction history. Here we assume G is a non-adaptive T2I model.
'More generally user choice function can also depend on prompt expansions. In our work we assume users only view the generated images and the initial prompt.
3

---

Preference Adaptive and Sequential Texaelo-Image Generation
without assessing state-actions outside the dataset distri- bution, and has proven effective in offline RL with LLMs (Snell et al.; Zhou et al., 2024). The IQL loss is:
{
Candiciós Cemmrntor
Cancllia Sebecter
Ps
<C'
[(), = P.rahee (go(tee f Pt) ne
Candidato Sclector
hr
P1c
-no(h++1)) + 12 (ggglh, Pt) upp us.(
fo
Ps1
do ah Pt)
where Lg(x) (a x² E [0.5, 1] and Up is the o-expectile value estimate. Most notably, the IQL loss does not require searching for the best slate (which is oftentimes out of the distribution of the training data), making it very efficient for training (see Appendix E.1 for further details).
14 Pi,
fo
(A)(A) e ary max wihh
Figure 2: Top. PASTA policy framework: The LMM is used as candidate generator of candidate from which the candidate selector policy used select slate. Bottom. Each prompt in the slate is evaluated individually using a prompt-value model, and the overall slate value is calculated as the average of the individual prompt values.
4. PASTA Dataset
Training PASTA relies on the availability of sequential user-agent interaction data. While single-turn T21 prefer- ence data is readily available (Wu et al., 2023; Kirstain et al., 2023; Pressman et al., 2022: Liang al., 2024), sequential datasets not. Hence, rater data for our sequential setup, and further enrich it with simulated data. Below, we describe our data cre- ation process. All of our datasets are open-sourced here: http://wwwwwwllleooo////////////
3.2. Value-Based Candidate Selector
collect human-
We use a state-action value function to define our selector policy, TT. Given candidates { and a state-action value model qo(h, P) parameterized by d, we define the
selector policy by
(h) E arg
max PEP: (1p))te)
20(h, P),
(1)
4.1. Human Rater Sequential Data
We use human raters to gather sequential user preferences data for preference-adaptive T2I generation. 4 Participants are tasked with interacting with an LMM agent for five turns. Throughout our rater study we use Gemini 1.5 Flash Model (Gemini-Teane -eamm 2024) as our base LMM, which acts as agent. At each turn, the system presents 16 images, ar- ranged in four columns (L = 4, M 4), each representing different prompt expansion derived from the user's initial prompt and prior interactions. Raters are shown only the generated images, not the prompt expansions themselves.
{P : |P|= L} is the set of all pos- where =
sible slates of size L. Enumerating PL HE)) is, of course, computationally expensive when To is large (a detri- ment to both training time and inference efficiency). In- spired by le et al. (2019), we decompose the value function into a (weighted) average of prompt-values Jo HxP + IR. Specifically, we compute the value each prompt, and esti- mate the value of a slate by:
1 Σ fo(h.)). (h, P) - I, PEP
At session start, raters are instructed provide an initial prompt of most 12 words, encapsulating specific vi- sual concept. They are encouraged to provide descriptive prompts that avoid generic terms (e.g., "an ancient Egyptian temple with hieroglyphs"insteee 'instead of "a temple"). At each turn, raters then select the column of images preferred most (see the UI used in Appendix B); they are instructed to se- lect a column based on the quality of the hest image in that column w.r.t. their original intent. Raters may optionally provide free-text critique (up 12 words) to guide sub- sequent prompt expansions, though most raters did not use this facility.
This approximation reduces the exponential complexity of finding the best slate to OL log Lc) (i.e., by sorting the prompt values over the candidate set). See Figure 2 for a schematic of the PASTA policy and value function. PASTA first prompts an LMM me (i.e., candidate generator) to generate Le candidates fr (h)... then uses its candidate selector ih select slate P of prompts according to Equation (1).
We train PASTA using implicit Q-Learning (IQL) (Kostrikov et al., 2021), which estimates the TD error with the Bellman optimality operator. IQL employs a soft estimate with a value function trained to approximate the high expectile
'Raters were paid contractors. They received their standard contracted wage, which is above the living wage in their country of employment.
4

---

Preference Adaptive and Sequential Texae Generation
After five turns, raters enter evaluation phase where they answer questions about each turn, including: (1) re- confirmation of their preferred columns; (2) quantifying whether the image slate in the current turn is better than the previous; and (3) a free-text explanation for their selection. This process yields a dataset comprising sequential interac- tion trajectories with user preference feedback and provides valuable insight into the dynamics of multi-turn, preference- adaptive T2I generation. Our human rater dataset includes over 7,000 five-step sequences (more than 500,000 images total), annotated by about 100 human raters using a random candidate selector. See Appendix B for comprehensive description of the rater study.
tion system. Specifically, this model takes as input user type E U, an initial prompt p P, and slate of im- LLM ages P. = mm 144,m and estimates the probability that prefers image column { { over the others J.M {mm.,e 1 .m=1. I.c.
L.M P((mme a. > (Tm.Lt) (m (2)
In the typical single-turn setup (Wallace et al., 2024), by setting M = 2, L= 1, and not assuming any user types, Equation (2) reduces to a probabilistic model that estimates the "global" (non-user specific) pair-wise preference be- tween two images, i.e.,
4.2. Simulated Sequential Data
Buwwo! P(1 > I2) [u,p].
Simulated data can drastically improve performance of RL agents (Kaiser et al., 2019; Yu et al., 2021; Liu et al., 2023; Hafner et al., 2020; Micheli et al., 2022). For this reason, we enrich the rater data above with additional simulated agent-user interaction data. To do so, we develop user model that encodes distinct user types that reflect range of preferences. This model has two components: (1) A user choice model which predicts the image column a user selects; and (2) a user utility model which predicts a user's ss satisfaction with an image slate. We outline details of the user model in Section
By contrast, our model is able to capture user-specific pref- erences over generic sets of multiple images.
Our user choice model C' exploits the preference model in Equation (2), making the usual assumption that users select images based on their inherent preferences; i.e..
Assumption 5.1 (Choice-Preferenn Equivalence). Assume C == P. where P is the preference model in Equation (2).
While, in general, user choice may also be be biased by other factors (e.g., previous interactions, position bias, etc.), this idcalized model proves very useful in experiments.
We begin by sampling a user and initial prompt from joint prior. We use randomized exploration policy for the agent, encouraging diverse interactions to distinguish preferences across user types. At each turn in a simulated trajectory, the agent proposes a slate of prompt expansions, and the user (via the choice model) selects their preferred prompt. The user utility model is then used to assign satisfaction score for the images generated using the selected prompt. This process is repeated for five turns (see Appendix C for further details and analysis of our data generation process). The simulated user data includes over 30,000 rollouts (exceeding 2.5 million images). Augmenting our human-rater data with this simulated data allows for more robust training of PASTA, as we show later in Section 6.
To further model user utility, motivated by the axioms of ex- pected utility theory (Von Neumann & Morgenstern, 2007), we model user utility R U x P x I. + R. leverag- ing the preference model in Equation (2) to ensure user's utility consistent with their preferences. Specifically, we assume that a user prefers one set of images another if the utility of the former images is higher, as formalized by the following assumption.
Assumption 5.2 (Utility Consistency). For any u,p, Equa- tion (2) is satisfied for l* E {1,...,L] iff for all l + E
M R (eeeer.mi ) lu, > (l ) w.
5. User Model
Assumptions 5.1 and 5.2 ensure a tight coupling of our user utility and choice models, which we leverage in model training.
The data generation process described in Section 4.2 re- quires a user simulator consisting of both user choice and utility models. We leverage our sequential human-rater data (Section 4.),, together with large-scale open-source single- turn T2I preference data (Wu et al., 2023; Kirstain et al., 2023; Pressman et al., 2022), to build these models.
5.1. A Parametric User Model
To train user utility and choice models, employ parameterized score model 8p : 14 x I x P - [0, 1] to serve as the backbone of our user simulator: We assume discrete set of K (unknown) user types, i.c., U =
We first propose user preference model based on funda- mental choice theory (Pennock et al., 2000) that mimics user's preference behaviors in an interactive recommenda-
K}.
Exploiting Assumptions 5.1 and 5.2, we define the utility of user type ic over an arbitrary image slate (T,,,,t )
M m=1 that
5

---

Preference Adaptive and Sequential Texa-lo-Image Generation
erences. This model leverages the score function 89 that assesses the compatibility between image-prompt pairs and different user types. The EM algorithm iteratively refines the model parameters 0 and user type prior distribution to maximize the likelihood of observed user feedback. Training proceeds in two phases: a main training phase on large-scale datasets with frozen CLIP parameters, followed by fine-tuning phase on human-rated data where all param- eters trained. We detail the training procedure, specific loss functions tailored to different data types (single-turn preferences, relevance scores, and sequential multi-turn), the model architecture, and hyperparameter settings in Ap- pendix D.I.
kk, P,
c/u
Dot Product
Image
K Prompt Embeeidnng (fer wach user)
Embaidinng (br eoch userr)
FFN
Image CLIP Encoder
CLIP Encoder
6. Experiments
To assess the effectiveness of our approach, we conduct two empirical evaluations. First, we evaluate the quality of our user model and the simulated data it generated. This involves analyzing the model's ability to accurately capture user preferences and generate realistic interaction data. Sec- ond, we study the performance of PASTA, our multi-turn T2I agent, trained using our rater and simulated datasets. We perform comprehensive analysis at each stage to assess the quality of the user model, the accuracy of the simulated data, and the overall performance of the T2I agent.
Image I
Prompt
Figure 3: Score function architecture. The image and prompt are fed into CLIP encoders followed by user encoders to generate K user embeddings. The score for each user type is the inner product between the corresponding user image embedding and user text emhedding.
associates to a prompt using the following model:
k. Rig = Ro (Ro (. I k.
6.1. Setup
Agg((s L1,e)). .., so IM,,,))
The slate size, number of images per prompt, and problem horizon are fixed L 4, M = 4 and H 5 in all experi- ments. We use Stable Diffusion XL (Podell et al., 2023) as our T2I model and multimodal Gemini 1.5 Flash (Gemini- Team, 2024) as our (prompt) candidate generator LMM, with Le 25. Our learned value function is fine-tuned from Gemma 2B (Gemma-Team, 2024). Our experiments use sparse reward, with rewards provided only in the final round, as our primary interest lies in the end result of inter- active refinement process. We use softmax sampling for the user model's 's aggregation function in Equation (3), as raters were instructed to select a column based on the best image from that column.
where Agg is an aggregator (e.g., average, max, Softmax sampler) operator. User choice probabilities are given by
Colks,, p, {mm JM { (mm,L,l
=Softmax (T To.k
(4)
where Oи he [ 1 R., is parameterized tem- perature constant which depends on the scores of different image columns. Such a temperature parameterization en- sures our user choice model satisfies Assumption 5.2, while allowing for greater flexibility in modeling.
To balance model capacity and computational efficiency, our score function adopts pre-trained CLIP (Radford et al., 2021) text and image encoders, followed by user encoders (a head for each user type). The user encoder transforms CLIP embeddings into a ssse-fipe-speccfiic representation that cap- tures individual preferences for images and prompts. The final score of an mmage-prompt pair is the inner product of the (user-type-speet -peectceieseeeec image embedding and correspond- ing text embedding, multiplied by a learned temperature parameter (see a schematic of our architecture in Figure 3).
To further encourage diversity in the generation of prompt candidates, we require the LMM prompt generator to parti- tion the candidate set into five categories, with each category focusing on a different aspect of the prompt expansion. Cat- egories include both generic forms (e.g., rephrasing, adding more details) as well as prompt-specific approaches (e.g.. different types of dogs, cloud shapes, etc.). Moreover, we restrict the candidate selector to select most one prompt from each category. This simple technique induces signifi- cant diversity (hence implicit exploration), and is used for data generation, policy training, and at inference time.
We employ an Expectation-Maxiine to learn a user model capable of capturing diverse pref-
(EM) framework
6

---

Preference Adaptive and Sequential Text-to-Image Generation
RickApic accuracy
Rank correlation
Anime abstract
0.25
0.60
0.20
0.58
0.15
Scenic views
2
16
64
16
32
64
Choice model accuracy
Cross turn accuracy
Animals
0.69
0.68
0.66
Bears
0.64
0.62
0.66
8 16 2
4 8 16
Figure 4: The graphs present the performance of trained user model as a function the number user types considered. The top row displays the model's accuracy on the Pick-a-Pic test set (left) and its Spearman's 's rank correlation (right). The bottom shows the model's choice accuracy (left) and cross-turn preference accuracy (right), both evaluated on our human-rated test set.
Food
Figure 5: Emergence of user-specific preferences: Each row displays the top five images scored by the user model, prior to fine-tuning from the HPS test set for one of five specific user types. We highlight user types where differences across the top five images were especially salient. The category labels (Animals, Food, etc.) are simply meant to be evocative on the style or content of the most preferred images.
the HPS test set
6.2. User Model Evaluation
We evaluate the quality of our user model by assessing its ability to capture user preferences and generate realis- tic interaction data. We train the model using large-scale single-turn datasets (HPS V2 (Wu et al., 2023), Pick-a-Pic (Kirstain et al., 2023), and Simulacra Aesthetic Captions human-
for our human-rater data using the user model, and train our state-action value function in three different settings: using only human rater data, using only simulated data, and using both data sources. The T2I agents trained by these algo- rithms are evaluated with human raters, for which the raters judge whether the current turn's 's chosen image was better, worse, or equally preferred to that in the previous turn. To make PASTA more efficient for numan-rater studies, we dis- till the T2I agent into a single, fine-tuned Gemini 1.5 Flash (Gemini-Teame 2024) LMM, and serve that in real-time generate the proposed image slates directly (without explicit prompt expansions). We also conduct an experiment with simulated users, using our user model, whose results are described in Appendix together with other experiments for both PASTA and the user model.
(Pressman et al., 2022)) and fine-tune it with rater sequential data. During training, CLIP model weights are initially frozen, focusing on learning user image and text encoder parameters. See Appendix D for full training details, including loss functions and architectures.
We analyze model performance in two stages. First, we assess prediction accuracy of the and Pick-a-Pic testset (Kirstain et al., 2023) and ranking using Spearman's 's rank correlation (Spearman, 1961) on the HPS dataset (Wu et al., 2023). Second, we evaluate prompt choice prediction accu- racy and cross-turn preference accuracy on our human-rated data. Results in Figure indicate that increasing the number of user types significantly improves performance, plateauing around 16 types. This highlights the importance of model- ing diverse user preferences. Notably, even with a moderate number of user types, model achieves approximately 70% accuracy on the human-rated dataset metrics. A further notable observation is the emergence of visually distinct domain preferences for different user types; this tends to be- come more prominent as the number of user types increases. An illustration for the 32-type model is shown in Figure
We compare our agents (each using a different state-action value function) with untrained multi-modal Gemini 1.5 Flash model. The results of human-rater evaluations are shown in Figure 6. These results demonstrate that training on real human data enhances our agents' performance over the pre-trained LMM baseline (i.e., over the off-the shelf Gemini 1.5 Flash), whereas training with synthetic data alone leads to a slight decline in performance. Unsurpris- ingly, training on both real and synthetic datasets offers the best performance. This suggests that, while user simula- tion may not fully reflect the real-world distribution of user behaviors (partially due to its simplified assumptions), the additional data generated by our user model does capture some key dynamics of real user interactions, yielding more accurate user choice predictions and sequentially generated
6.3. PASTA Results
Using our user model, we generate more than 30,000 simu- lated user-agent trajectories. We also generate reward labels

---

Preference Adaptive and Sequential Text-to-Image Generation
Is Turn 2 Better Than Turn 1?
Is Turn 3 Better Than Turn 2?
Is Turn Better Than Turn 3?
Is Turn 5 Better Than Turn 4?
Base
Only Syrthetic
Only Rater
Synthetic Rater
0.2
0.0
Better
Worze
Same
Better
Worse
Somre
Better
Worse
Some
Better
Worze Some
Figure 6: PASTA Evaluation. We ask the previous turn. The raters can choose one of three options: Better, Worse, or Same. Results show percentage of raters who chose each option. Experiments compare the hase Gemini Flash model (without fine-tuning) to PASTA (also using Gemini Flash) varied by training on different datasets: (I) only simulated user data, (2) only real rater data, and (3) the combination of both datasets. Plots show 99% confidence intervals.
determine, for each interaction turn, whether the shown images are improvement over
User Score
6.4. Generalization: Flux T2I model
Base
We use Flux.1 5 together with our PASTA agent and com- pare it to the baseline Gemini 1.5 Flash. Particularly, we do not retrain PASTA over new Flux. 1 data, but rather test PASTA's capability to generalize from the SDXL T2I model to the Flux.1 T2I model. We present the raters the final images constructed by PASTA against the final images gen- erated with the Gemini baseline. Indeed, our results show that PASTA achieves 20% relative improvement over the baseline without any further training over Flux data. This highlights PASTA's ability to adapt to different T2I models.
5
Only Synthetic
Only Rate"
Synthetic
2
0
2
3
4
Rounds
Figure 7: PASTA Evaluation with the Score Model. We use trained model to evaluate the rollouts of real users with different baselines.
6.5. Abstract Prompts with Simulated Users
To visualize the differences in user preferences and PASTA's ability to adapt to them, we use PASTA with various user types using broad, abstract prompts such as "an image of happiness". Figure shows an example of different rollouts of three distinct user types. We observe a clear preference emerging, favoring specific styles or content. All users starts with the same prompt and initial images. We present only the images and their corresponding prompt-expansion at the last 5-th step. Each color represent a different user type. For more examples, see Appendix F.2.
images that are tailored to user preferences. Moreover, a high rank correlation between real user preferences and the predictions from our learned model over a vast number of user types further supports this conclusion. We also present the user model score for each step averaged over all rollouts in Figure 7. The user type was chosen as the one that maxi- mizes the posterior over the user's data. The results show that PASTA monotonically improves the user score along the rollouts and are consistent with the real users' results in Figure 6. This evaluation further validates the correlation between our user model and real users.
7. Related Work
Our work is related to recent advances in interactive and preference--daptive image generation. Existing methods ex- plore iterative refinement using visual feedback (von Rütte et al., 2023; Wang, 2024) or leverage LLMSs for prompt engineering (Liu al., 2024; Brade et al., 2023; Hao et al., 2024), often focusing on optimization of perceptual qual- ity user satisfaction in single interaction (Fan al., 2023; Chidambaram et al., 2024; Wallace et al., 2024; Wu et al., 2023). Our PASTA framework extends this by framing multi-turn image generation as a sequential decision decision-making
Finally, we conduct new human rater evaluations over the final turn of PASTA. Particularly, we conduct a rater study where we explicitly show raters the final turn images (for a given prompt) of PASTA compared to the baseline Gemini 1.5 Flash model and ask them compare the generated last turn images. We find that PASTA receives 85% relative improvement over the baseline model when directly comparing the last turn. We believe that directly comparing the last turn emphasizes the strength of our approach and the significance our results in terms of the outcome of using our model.
hhttps://flxl..r Lai
8

---

Preference Adaptive and Sequential Text-to-Image Generation
"An kmage of lose"
User Type A
Rollout
Rollout
Rollout
light tonos. intimate moment"
ligting enhancing
elewny a
detailed
"As kmage of love"
User Type B
Rollout
Rollout
Rollout 2
Sepealty magh
stanghh suisestilisee withi portoeying
skyle,
Inlenceed
Figure 8: An illustration of three different rollouts of PASTA with two different user types for the abstract initial prompt: "An image of love". Rollouts show the final images in the rollout (i.c., step f 5).
problem, where an RL agent learns to extend and person alize prompt through a series of expansions guided by user feedback. This enables iterative refinement towards desired visual outcome, capturing nuances in pref- types. Notably, while many
8. Conclusion
This work introduced PASTA, novel RL agent for preference-adaptive and sequential T2I generation. We for- mulated the problem of iterative prompt expansions with LMMs T2I models as a sequential decision-making problem that drives a collaborative, multi-turn image gener- ation with the user. Critically, we have produced new dataset that captures sequential interactions between an agent and user, which we release to support further inves- tigation of new T2I techniques in the research community. We developed user utility and choice models, learned from this dataset, which we used to create user simulator that enabled generation of additional synthetic data (which we release well). Finally, used our sequential data train PASTA, and demonstrated its effectiveness through comprehensive human evaluations, showing significant im- provements in user satisfaction.
erences across diverse existing methods focus on modifying internal aspects of text-to-image (T2I) models (e.g., diffusion noise (Tang et al., 2023), attention maps (von Rütte et al., 2023) or prompt embeddings (Salchi et al., 2025)), PASTA operates by di- rectly adapting the input prompt, offering a model-agnostic approach adaptive preference learning in T2I generation.
PASTA draws upon rich history of preference elicita- tion (PE) research (Boutilier, 2002; Chajewska et al., 2000; Keeney, 1993; Salo & Hamalainen, 2001), especially that in content-based recommender systems (Boutilier et al., 2003;
Spearman, 1961; Rashid et al., 2002; Meshi et al., 2023), and addresses the challenge of optimizing PE for diverse downstream tasks. Existing research highlights the impor- tance of diversity in recommendations (Sidana et al., 2018) and explores diversity in elicitation strategies (Parapar & Radlinski, 2021). PASTA tackles this by training agent that considers both immediate user feedback and long-term goals to generate preference-aligee alligned images. Our novel data collection and simulation methodology further support the training of robust user simulators, which show greatly improve policy learning.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There many potential societal consequences of our work, none which we feel must be specifically highlighted here.

---

Preference Adaptive and Sequential Texa-to-Image Generation
References
Finn, C., Abbeel, P., and Levine, S. Model-agnostie meta- learning for fast adaptation of deep networks. In Interna- tional conference machine learning, pp. 1126-1135. PMLR, 2017.
Aberdeen, D., Buffet, O., and Thomas, O. Policy-gradient for psrs and pomdps. In Artificial intelligence and statis- tics, pp. 3-10. PMLR, 2007.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.,, Akkaya, I., Aleman, F. I.., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Semini-Team Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint arXiv:2403.055300 2024.
Gemma- Team. Gemma: Open models based on gemini re- search and technology. arXiv preprint arXiv:2403. 28220 2024.
Boutilier, C. A pomdp formulation of preference elicitation problems. In AAAI/IAAI, pp. 239-246. Edmonton, AB, 2002.
Gu, J., Zhai, S., Zhang, Y., Susskind, J. M., and Jaitly, N. Matryoshka diffusion models. In The Twelfth Interna- tional Conference on Learning Representations, 2023.
Boutilier, C., Zemel, R. S., and Marlin, B. Active collabo- rative filtering. In Proceedings of the Nineteenth Annual Conference on Uncertainty in Artificial Intelligence (UAI- 03), pp. 98-106, 2003.
Hafner, D., Lillicrap, T., Norouzi, M., and Ba, J. Mas- tering atari with discrete world models. arXiv preprint arXiv:2010.02193 2020.
Brade, S., Wang, B., Sousa, M.. Oore, S., and Grossman, T. Promptify: Text-to-image generation through interactive prompt exploration with large language models. Pro- ceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pp. 1-14, 2023.
Hallak, A., Di Castro, D., and Mannor, S. Con- textual markov decision processes. arXiv preprint arXiv: (Eii::550200254 2015.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-38) 345, 1952.
Hao, Y., Chi, Z., Dong, L., and Wei, F. Optimizing prompts for text-to-image generation. Advances in Neural Infor- mation Processing Systems, 36, 2024.
Chajewska, U., Koller, D., and Parr, R. Making rational decisions using adaptive utility elicitation. In Aaai/laai, pp. 363-369, 2000.
Ho, and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207. 12598 2022.
Chidambaram, K., Seetharaman, K. V., and Syrgkanis, V. Direct preference optimization with unobserved prefer- ence heterogeneity. arXiv preprint 4XXiv22455150065 2024.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840 6851, 2020.
Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The curious case of neural text degeneration. arXiv preprint 2019.
Croitoru, F.-A., Hondru, V., Ionescu, R. T., and Shah, M. Diffusion models in vision: A survey. IEEE Transac- fions Pattern Analysis and Machine Intelligence, 45 (9):10850-10869, 2023.
Ic, E., Jain, V., Wang. J., Narvekar, S., Agarwal, R., Wu, R., Cheng, H.-T., Chandra, T., and Boutilier, SlateQ: tractable decomposition for reinforcement learning with recommendation sets. In International Joint Conference Artifical Intelligence (IJCAI), pp. 2592-2599, Macau, 2019.
Dorfman, R., Shenfeld, I., and Tamar, A. Offline meta reinforcement ceaming-iientifiabbe challenges and ef- fective data collection strategies. Advances in Neural Information Processing Systems, 34:4607-4618, 2021.
Duan, Y., Schulman, J., Chen, X., Bartlett, P. L., Sutskever, I., and Abbeel, P. RL": Fast reinforcement learn- ing via slow reinforcement learning. arXiv preprint arXiv: 161 I. 02779, 2016.
Kaiser, L.. Babaeizadeh. M., Milos. P., Osinski, B., Camp- bell, R. H., Czechowski, K., Erhan, D., Finn, C., Koza- kowski, P., Levine, S., et al. Model-based reinforce- ment learning for atari. arXiv preprint arXiv:1903.0037. 2019.
Fan, Y., Watkins, O., Du, Y., Liu, H., Ryu, M., Boutilier, C., Abbeel, P., Ghavamzadeh, M., Lee, K., and Lee, K. Rein- forcement learning for fine-tuning text-to-image diffusion models. Advances in Neural Information Processing Systems, volume 36, 2023.
Keeney, R. L. Decisions with multiple objectives: Prefer- ences and value tradeoffs. Cambridge university press, 1993.
10

---

Preference Adaptive and Sequential Texaelo-Image Generation
Kirstain, Y., Polyak, A., Singer, U., Matiana, S., Penna, J., and Levy, O. Pick-a-pie: An open dataset of user prefer- ences for text-to-image generation. Advances in Neural Information Processing Systems, 36:36652-36663 2023.
Pennock, D. M., Horvitz, E., Giles, C. L., et al. Social choice theory and recommender systems: Analysis of the iomatic foundations of collaborative filtering. AAAI/AAA,, 30:729-7344 -34, 00777422000 2000.
Kostrikov, Nair, A., and Levine, S. Offline reinforce- ment learning with implicit q-learning. arXiv preprint 10.00619, 2021.
Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffusion models for high resoutteno image synthesis. arXiv preprint arXiv:2307.01990 2023.
Kwon, J., Efroni, Y., Caramanis, C., and Mannor, S. for latent mdps: Regret guarantees and lower bound. Advances in Neural Information Processing Systems, 34: 24523-24534 2021.
Pressman, J. D., Crowson, K., and Contributors, S. C. Simulacra aesthetic captions. Technical Report Ver- sion 1.0, Stability AL, 2022. url http:://iihub.com/DD
P Pisimulcne-aesthiicc-aaiiiss
Liang, Y., He, J.. Li. G., Li, P., Klimovskiy, A., Carolan, N., Sun, J., Pont-Tuset, J., Young, S., Yang, F., et al. Rich human feedback for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19401-19441 2024.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Liu, X.-Y., Zhou, X.-H., Gui, M.-.,, Xie, X.-L., Liu, S.-Q.. Wang, S.-Y., Li, H., Xiang, T.-Y., Huang, D.-X., and Hou, Z.-G. Domain: Mildly conservative model-based offline reinforcement learning. arXiv preprint arXiv:2309.0891 2023.
Rashid, A. M., Albert, Cosley, D., Lam, S. K., McNee, S. M., Konstan, J. A., and Riedl, Getting know you: learning new user preferences in recommender systems. In Proceedings of the 7th international conference on Intelligent user interfaces, pp. 127-134, 2002.
Liu, Y., He, M., Yao, F., Ji, Y., Tao, S., Du, J., Li, D., Gao, J., Zhang, L., Yang, H., al. What do you want? user- centric prompt generation for text-to-image synthesis via multi-turn guidance. arXiv preprint arXiv:2408. 12910, 2024.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Loshchilov, and Hutter, Sgdr: Stochastic gra- dient descent with warm restarts. arXiv preprint (xXiv16608.0998. 2016.
Saharia, C., Chan, W., Saxena, S., Li, L.,, Whang. J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to sox-to-imaae dif- fusion models with deep language understanding. Ad- vances in neural information processing systems, 35: 36479-36494, 2022.
Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA. USA, May 6-9, 2019, 2019.
Salehi, S., Shafiei, M., Yeo, T., Bachmann, R., and Zamir, A. Viper: Visual personalization of generative models via individual preference learning. In European Conference on Computer Vision, pp. 391 406.. Springer, 2025.
Meshi, O., Feldman, J., Yang, L., Scheetz, B., Cai, Y., Bateni, M., Salisbury, C., Aggarwal, V., and Boutilier, C. Preference elicitation for music recommendations. In ICML 2023 Workshop The Many Facets of Preference- Based Learning, 2023.
Salo, A. A. and Hamalainen, R. P. Preference ratios in multiattribute evaluation -liiite-elicitation and decision procedures under incomplete information. IEEE Transac- tions on Systems, Man, and Cybernetics A: Systems and Humans, 31(6):533-545 2001.
Micheli, V., Alonso, E., and Fleuret, Transform- ers are sample-efficient world models. arXiv preprint 008888 2022.
Parapar, J. and Radlinski, F. Diverse user preference elic- itation with multi-armed bandits. Proceedings of the 14th ACM international conference on web search and data mining, pp. 130-138, 2021.
Shazeer, N. and Stern, M. Adafactor: Adaptive learning rates with sublinear memory cost. International Con- ference on Machine Learning. pp. 1596-4604, PMLR, 2018.

---

Preference Adaptive and Sequential Texaelo-Image Generation
Sidana, S., Laclau, C., and Amini, M.-R. Learning rec- ommend diverse items over implicit feedback pandor.
Yang, L.., Zhang, Z., Song, Y., Hong, S., Xu, R., Zhao, Y., Zhang, W., Cui, B., and Yang, M.-H. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1-39, 2023.
In Proceedings of the 12th ACM Conference on Recom- mender Systems, Pp. 427-431, 2018.
Snell, C., Kostrikov, I., Su, Y., Yang, M., and Levine, S. Offline rl for natural language generation with im- plicit language q learning, 2022. URL https://axiv. org/abs/220..11877
Yu, J., Xu, Y., Koh, J. Y., Luong, T., Baid, G., Wang, Z., Vasudevan, V., Ku, A., Yang. Y., Ayan, B. K., Hutchinson, B., Han, W., Parekh, Z., Li, X., Zhang, H., Baldridge. J., and Wu, Y. Scaling autoregressive models for content- rich text-to-image generation. Transactions on Machine Learning Research, 2022.
Spearman, C.
The proof and measurement of association between two things. 1961.
Yu, T., Kumar, A., Rafailov, R., Rajeswaran, A., Levine, S., and Finn, C. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems, 14:28954-28966 2021.
Sutton, R. S. Reinforcement learning: An introduction. A Bradford Book, 2018.
Tang, Z., Rybin, D., and Chang, T.-H. Zeroth-order opti- mization meets human feedback: Provable learning via ranking oracles. arXiv preprint arXiv:2303.0355. 2023.
Zhang, C., Zhang, C., Zhang, M., and Kweon, I. S. Text-to- image diffusion models in generative Al: A survey. arXiv 2023.
Tennenholtz, G., Chow, Y., Hsu, C.-W., Shani, L., Liang, E., and Boutilier, C. Embedding- language models. In Advances in Neural Information Processing Systems, volume 37, 2024.
preprint arXiv:2303.07999
Zhou, Y., Zanette, A., Pan, J., Levine, S., and Kumar, A. Archer: Training language model agents via hierarchical multi-turn rl. arXiv preprini arXiv:2402.
2024.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Zintgraf. L., Shiarlis, K., Igl, M., Schulze, S., Gal, Y., Hof-
mann, K., and Whiteson, S. VariBAD: very good method for bayes-adaptive deep RL via meta-learning. In 8th International Conference on Learning Representa- tions, ICLR, 2020.
Von Neumann, J. and Morgenstem, O. Theory of games and economic behavior: 60th anniversary commemorative edition. In Theory of games and economic behavior. Princcton university press, 2007.
von Rütte, D., Fedele, E., Thomm, J., and Wolf, L. Fabric: Personalizing diffusion models with iterative feedback. arXiv preprint arXiv:2307.10155 2023.
Wallace, B., Dang, M.. Rafailov, R., Zhou, L., Lou, A., Purushwalkam, S., Ermon, S., Xiong, C., Joty, S., and Naik, N. Diffusion model alignment using direct pref- erence optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 8228 12388823,2222 2024.
Wang, C. diffusion-based method for multi-turn composi- tional image generation. In Proceedings of the IEEF/CVF Winter Conference Applications of Computer Vision, pp. 381-391, 2024.
Wang, J. X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, J. Z., Munos, R., Blundell, C., Kumaran, D., and Botvinick, M. Learning to reinforcement learn. arXiv preprint arXiv: 1,05763, 2016.
Wu, X., Sun, K., Zhu, F., Zhao, R., and Li, H. Human preference score: Better aligning text-to-image models with human preference. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 2096- 2105, 2023.
12

---

Preference Adaptive and Sequential Texae Generation
A. Latent Markov Decision Process (LMDP)
begin with defining Latent Markov Decision Process introduced in (Hallak al., 2015; Kwon et al., 2021).
Definition A.1. Suppose that a set of MDPs M with a joint state space S and joint action space A in finite horizon setting with horizon Н. let K = [ S = [S] and A = |A]. Each MDP Mr E .M it tuple (S, Tk. Rg, vk) where Tk: S
As is a transition probability that maps state-action pair into a distribution over the next state, Rg S A 1 a probability measure for rewards that maps a state-action pair into a distribution over rewards and initial state distribution Ve. Let 7/. ................................ '] be the mixing weights of LMDPs such that at the start of every episode, one MDP M M is randomly chosen with probability T)k.
The goal of the problem is to find policy within policy class II that maximizes the expected return:
UMM
max WEl
Th
ET
Σ
rt
where [-] is expectation taken over the k-th MDP with policy æ. The policy क H. S - 44 maps the current state and history into distribution actions. Generally, the history is all experience model parameters are known, history is sufficient statisties and can be summarized into belief states:
far H = (8, [0, 1])". When the
(81)
by (kkk) MaPK ($1)"
by (k))T(((54+4 (8041/88, or (r at)
bk41 = Σe ( 0z) ( (reesss,
We formulate our reinforcement learning problem as latent MDP, where each user from a discrete set induces distinct transition and reward kernels. Specifically, in our sequential text to-aage decision problem the state space is a prompt and image slate S = x IM and the action space is the prompt slate = PL. The transition kernel for the k-th user type composed of the T2I model and user choice model, i.e. the next state the tuple composed of slate of images generated from the chosen slate (the action) and chosen prompt:
81+1 (f11metd
m= t=1 P..). Pm.)
where I,,e G(pe,) and a C(k, Po. I I11, I.IL,)). The reward kernel corresponds to the user's utility function:
Tt R(k, po, I.sa...... I,.....,,MM.... IMAL
Similar standard LMDPs, the posterior update during user model training in ?? mirrors the belief state computation, driven by observed samples collected from diverse human raters.
13

---

Preference Adaptive and Sequential Texae Generation
B. Human Rater Dataset
To gather data for training and evaluating PASTA, we conducted human rater study. We recruited paid contractors as raters, and utilized a web-based interface where raters interacted with Gemini 1.5 Flash Model acting as the agent in our multi-turn image generation process.
An ancient Egyptian temple with hieroglyphs. Turn
Column selection
(Optional) Enter critique for the images above
An optional critique (c g., add more color)
submit button (move to next tum)
Figure 9: A screenshot of the interface raters used in the interaction phase. Raters were not shown the prompt expansions. They were instructed choose a preferred column (based on the best image of each column). Raters could optionally also add an up to 12 word critique.
B.1. Rater Instructions: Interaction Phase
Each rater participated in five-turn interaction session. At the beginning of the session, raters were presented with the task's instructions. They were then asked to provide an initial text prompt, with maximum length of 12 words, representing specific visual concept they wanted to see realized an image. The guidelines stressed that prompts should encapsulate meaningful intent (thinking about specific image they wished to see) and avoid generic prompts (c.g., "a temple"). Instead, they were encouraged to use descriptive prompts within the 12-word limit (e.g., "An ancient Egyptian temple with hieroglyphs"). They were assured that they need not worry about fitting all of their thoughts into the w2-dord limit, as the interaction process with the agent would help them refine their concept.
14

---

Preference Adaptive and Sequential Text-to-Image Generation
Column
Column
Column
Column
Image 1A
Image 2A
Image 3A
Image 4A
Image 1B
Image 2B
Image 3B
Image 4B
Image 1C
Image 2C
Image 3C
Image 4C
Image 1D
Image 2D
Image 3D
Image 4D
Best Image in Column
Best Image in Column
Best Image in Column
Best Image in Column
Image 1B
Image 2A
Image 3A
Image 4D
Best image
Image 2A
Figure 10: Illustration shown raters on how to select a column. Columns should be compared based on the best image of each column.
In each of the subsequent five turns, the LMM agent presented the rater with 16 images, arranged in four columns (L=4, M=4). Each column represented different prompt expansion derived from the rater's 's initial prompt and their choices in previous turns. Figure 9 illustrates this setup. Critically, the raters were only shown the generated images, not the expanded prompts themselves. This ensured their feedback was based purely on visual perception and alignment with their initial intent, rather than on their interpretation
textual modifications made by the agent.
The core task for the rater in each turn was to select the column of images they preferred most. The instructions, visually depicted in Figure 10, emphasized that this selection should be based on the best image within each column, judged in terms of general quality and its correspondence with the rater's 'eers original prompt intent. For example, if Image 2A was the best overall image of the 16, then Column should be selected. This selection strategy was used to help reduce burden on raters. It also benefited our choice model training procedure, which used this as inductive bias for training our model.
Optionally, raters could provide a free-text critique of the presented images, limited to 12 words (Figure 9). This critique could address any aspect of the images, such as style, composition, color, or content, but raters were explicitly instructed not
15

---

Preference Adaptive and Sequential Text-to-Image Generation
DataCompute
Click on "End task" after answering questions for Turn 5
074244
100B
tack
Tumns
Tumn
Taok
Insivctions
Questions for Turn
Tam2
imagess prefer
comparing columns, compare the quality of the best
original
Tume
Golume 2
Ignore Turn
Golume
Based your image preferenoes give absollute descriing sattsfaction
images
Columnn sélection during interactions
score of the best image colume your original prompl's Intent.
Screenshot of the evaluation phase for raters. This second phase (after the interaction phase) included questions about every turn. See more details Appendix B.2.
Figure 11:
to use the critique field to change their original intent. For example, "Add more color" is valid critique. The purpose of the critique was to provide additional signals to the agent for refinement within the scope of the original visual concept, not to redirect the generation process towards a different image altogether. Overall, approximately 20% the collected data have one or more critiques.
B.2. Rater Instructions: Evaluation Phase
After completing five interaction turns, raters were prompted to click the "Enter rater mode" button. The system then transitioned the evaluation phase. For each of the five turns, raters answered set of questions designed capture detailed feedback on their experience and the agent's performance. Figure 11 demonstrates the interface used during this stage. For each turn, raters were asked the following questions, with their chosen column highlighted for reference:
1. Column Preference: Raters were asked to re-confirm their preferred column for each turn, ensuring consistency in their choices (See Figure 11).
2. Absolute Column Satisfaction: For each of the four columns in each turn, raters provided an absolute satisfaction score 5-point scale (I: very dissatisfied, 5: very satisfied). This score reflected their satisfaction with the best image in the column, relative to their original prompt intent. This absolute scoring allowed for finer-grainet assessment of the individual prompt expansions, capturing nuances beyond the simple relative ranking implied by column selection.
3. Selection Rationale: For their selected column each turn, raters were asked to provide a free-text explanation of their choice.
4. Turn-over-Tie -oroe--urrurm Improvement: For turns 2-5, raters assessed whether the of images in the current turn was better, worse, about the same the set of images in the previous turn. This comparison, again focused the quality of the selected columns in each turn relative to the original prompt intent, provided measure of the agent's 's progress and its ability to refine the generated images over time.
5. Overall Turn Satisfaction: Raters provided an overall absolute satisfaction score (1-5 scale) for all of the images presented in each turn. This average score across columns gave holistic impression of the agent's 's performance that turn, complementing the column-specific feedback.
16

---

Preference Adaptive and Sequential Text-lo-Image Generation
Raters were finally instructed press the "End Task" button (as shown in Figure 11) complete their session, which submitted their responses for all five turns.
B.3. Limitations of Rater Study
recognized limitation of the rater study conducted in this work is the potential bias between groups of raters. Figure compare PASTA trained different datasets with the pre-trained Gemini Flash model. We could not concurrently run different experiment arms because of lack of A/B testing infrastructure which shows different agents to raters random. Consequently, the timing of experiment arms and rater availability could have introduced bias. Furthermore, the rater pool might not be fully representative of all potential user demographics for PASTA, and the initial prompts provided to raters might not have sufficient diversity.
Human errors and system issues, such as connection timeouts, were observed during the study. We did not implement any formal mechanisms for raters report errors during specific sessions although raters can somehow "correct" their mistakes during the evaluation phase. Raters could prematurely abandon sessions without recording data from problematic sessions. Raters could provide critique of the presented images but subsequent generations might not be aligned with these critiques. Although raters were explicitly instructed not to evaluate based on critique alignment, it might still cause worse rater experience for those who provided critiques.
B.4. Generating Rewards for Human Rater Data with User Model
To leverage human rater data for value model training, we utilize our learned user utility function to generate rewards for each interaction step. For each trajectory, we first compute the posterior distribution over user types using the model described in Appendix D.3. We then employ the utility function of the most likely user type to generate rewards for each timestep in the trajectory.
17

---

Preference Adaptive and Sequential Texae Generation
C. Offline Simulated Sequential Data
This section describes the generation of an offline dataset for training PASTA. We detail the process of creating interaction trajectories with simulated users, based on our trained user model and random prompt selection policy. We also address the challenge of MDP ambiguity inherent in offline learning, specifically in the context of diverse user types, and present conditions for ensuring identifiability of the offline dataset.
C.1. Offfine Dataset Creation
Leveraging our trained user model and user type prior, we generated a dataset comprising over 30,000 simulated user interaction trajectories. We assume independence between the initial prompt distribution and the user type distribution, i.e., v(po, u) (po))oo(u) Therefore, at the beginning of each trajectory, prior from user model training (70 = 17) and sample imitial prompts uniformly from the available datasets, while limiting the initial prompts to be with no more than 10 words in order to avoid very large prompt at the end of the rollout.
approximate the true user prior with the learned
To ensure diverse and representative data for training, the data generation policy needs to produce coherent prompts that encourage exploration and sufficiently cover the state-action space, thereby mitigating potential MDP ambiguity (discussed further in the next subsection). We achieve this by using the candidate generator detailed in Appendix E.2 coupled with random selection policy, denoted Tand This policy is constrained select maximum of one prompt per category, promoting exploration across different prompt categories.
At each interaction step, the candidate generator produces a categorized set of Lc prompts. The random policy Taad then samples a slate of L prompts, for which the text-to- tomi model generates M images each. A prompt c is selected according to the user choice model, and the corresponding user satisfaction level (reward) is recorded based on the utility function defined in Section 2.2.
C.2. Identifiable Offline Dataset
After constructing the user choice model and utility function, we use a user simulator to generate an offline dataset for PASTA training. The offline reinforcement learning framework is chosen due its computational efficiency during training. However, offline LMDP introduces the challenge of Markov Decision Process (MDP) ambiguity (Dorfman et al., 2021), and in our context, user type ambiguity:
Definition C.1. User type ambiguity (Dorfman et al., 2021). Consider data generated from set of K user types U = [1,2,.... .., K]. each corresponding to a user choice model and utility function {Ci, R, ) The data is collected by sampling a subset of N user types { C U4 and applying their respective behavior policies {= { Ji=1 This results in
N
N data distributions {Pu.. N. over image slates, initial prompt and rewards, with p P x IMLL x [0, 1] H [0, We define the collected data as ambiguous if there exists a user type u² E l and two policies प, x' such that Pu.. Pur.. and P m Pat,m' for some i + j. Otherwise, the data is considered identifiable.
In essence, the data is ambiguous if a user type under different policies results in same data distributions of two different user types and their corresponding behavioral policies, regardless of how much data is collected. By introducing the concepts of identifying state-action pairs and overlapping state-actions, we can define sufficient condition for identifiability.
Definition C.2. Identifying Slate of Images.For a pair of user types i,j U, we define (po, {4m, m= as M an identifying slate of images if R(i,po, (Im, ( M. # R(j M )=) C(i, (mm,)) ) # M,L.
M,L.
Clj.Po, {mm,e] { M.....)))
Definition C.3. Overlapping Slate of Images. Given data collected described in Definition C.I. slate of images (po, (m,,e) } m= is said to overlap for pair of user types i, E l if it has positive slate-probabilie under both user types, i.c., P (Po, { M > O and Paag (po, {mm.e) M.L M m=),e=)) > O.
M.L
A sufficient condition for identifiability is as follows:
Propusition C.4. (Dorfman et al., 2021). Consider a data collected according to the settings in Definition C.I. The data is identifiable if, for every pair of distinct user types + j, there exists an identifying slate of images that overlaps.
Given that in our scenario user types vary not only in their utility functions but also in their choice models, the candidate
18

---

Preference Adaptive and Sequential Texa-lo-Image Generation
generator combined with the random sampler Frand likely to satisfy the conditions in Proposition C.4. This approach introduces sufficient randomness explore wide range of states, allowing reach identifying slates of images for all user types, while still maintaining a coherent policy capable of generating high-quality prompts. In the rare case where two user types generate identical data distributions under Trand we consider them the same user type and merge them.
D. User Model
We employ an Expectation Expectatio-Maxxiizaaiii Maximization (EM) framework to learn a user model capable of capturing diverse preferences. This model leverages score function 8g that assesses the compatibility between image-prompt pairs and different user types. The EM algorithm iteratively refines the model parameters 0 and likelihood of observed user feedback. Training proceeds in two phases: main training phase large-scale datasets with frozen CLIP parameters, followed by a fine-tuning phase on human-rated data where all parameters are trained. We detail the training procedure, specific loss functions tailored to different data types (single-tury preferences, relevance scores, and novel sequential multi-turn data), the model architecture, and hyperparameter settings in the following subsections.
type prior distribution
maximize the
D.1. Training
The user model, which always incorporates the score function sp, adapted to suit different datasets and label types. Specific model formulations and loss functions are detailed subsequent subsections. D and estimate the fog-likelihood loss as follows:
1. Mini-Batch Sampling: A mini-batch B is sampled from the dataset, and the posterior is estimated solely for samples within B using target network to prevent overestimation.
2. Prior Update: An exponential moving average of the mini-batch posteriors used to update the prior.
3. Gradient Descent: Instead of fully optimizing the model parameters, single gradient descent step is taken with
respect to the loss function:
[()) = - -.E.B, (o0g Vi. k:)]
The user model always composed of the score function 8y and may vary for different types of datasets and labels, and need be adjusted accordingly. We provide detailed models and loss function at the following sub-sections. Algorithm I provides a complete description of the mini-batee training procedure. Training proceeds in two phases:
1. Main Training: The model is trained on a combination of large-scale datasets: HPS V2 (Wu et al., 2023), Pick-a-Pic (Kirstain et al., 2023), and Simulacra Aesthetic Captions (SAC) (Pressman et al., 2022). The CLIP model parameters are frozen during this phase.
2. Fine-tuning with Sequential Data: A shorter fine-tuning phase follows, utilizing human-rated data, Here, the entire model, including the CLIP model, is trained.
19

---

Preference Adaptive and Sequential Texae Generation
Algorithm I Mini-Batch Exppetaatoon-Maxiiizaiiin User Model Optimization
1: Dataset D, number of user types K, parameter Oprior, learning rate B.
2: Initialize 7) = (1/K, . target network 0 = 8g
3: for in {0,..., 7 do
4: Sample mini-batch B ~ D.
5: E-step. Calculate posterior " for each example (æ4, yk) E B and user class & [K]
7k I ( ki,s,
(k)
1= me 11 (2 Bi.s
6: M-step. Update parameters 0, m
1 f Oprior 17 (1 Aprior Binbhi
Os+1 = 0: (VL(())
7: Every steps: Update target network 0 + 0r
8: end for
9: Return: model weights 0r.
D.2. Losses and Models Single Step
The EM framework is a general method applicable to various dataset and model types. In this work, we utilize two common single turn dataset types:
Preference Model (HPS, Pick-a-Pic): This model utilizes data samples consisting of two images generated by the same prompt, 4; (Li, p'), where the annotator indicates their preference, yi (vi .²). To connect the model to the score function, we employ the Bradley-Terry (BT) model (Bradley & Terry, 1952), adapted from the RLHF reward learning framework to accommodate different user types. The user-specific BT model defines the probability that a user type prefers the first image over the other as:
exp(so ('f,p',))) + exp(so(4),p',)) exxp(s((1,.''.))
of gree (zi, Bc. kk)
(5)
In this case, the user loss function simplifies to:
Curef (0) B, ( (T) k) so( k))
where a represents the Sigmoid function.
Relevance model (SAC): This model provides direct rating for an image- pair, i.e., x; (I',p'), with the label being the integer rating level provided by the annotator, yi = s; E [S]. We employ a simple user-conditional Gaussian model, where the score function acts as the mean estimator:
ret (zi, yi = s;, k)) expx (so (T'', ,p',k) 8,)
0
(6)
2v2
where v is a predefined standard deviation. The user loss for the relevance model is:
Cra (0) 2n² B. (s0 (''p''''')) {,
When using relevance datasets, the rating levels typically normalized to the range [o, 1].
To maximize the number of training samples for the user model, a mixture of data types can be used. In the case of a batch from each type, B = (Beref, Brel), mixture loss function can be employed to balance between the two terms:
(())
2 Bi-Brat bny (I', p', k) s
B Awyy (s. k) so so ((..p.)))] {k))]
where K1 controls the balance between the two terms.
20

---

Preference Adaptive and Sequential Tmxae Generation
D.3. Losses and Models Sequential
In this work, we introduce a novel multi-turn dataset. This dataset, generated by querying users across H steps following the setting described in Section 2.2, consists of trajectories of user-agent interactions. Each sample 2; { [t= has Fi,t {po, ) 1,8-1
m= } at each timestep t, with corresponding labels:
1. The chosen prompt index each turn: Vi,t" Ci,t E [L].
2. In-turn preferences between the best images for each prompt: Vi.
in-pref ((vt' = ye Je,res where Q represents
the of all unordered pairs of size Q (2)
3. Cross-turn preferences between chosen images consecutive turns: Vi,t cross-turn preferences per sample.
cross pross-pref
(wi² itt = ve "i""), resulting in H
Preferences within a sample are modeled by applying the BT model (Equation 5) to both in-turn and cross-turn preferences. User choices are modeled using our user choice model (Equation 4):
08 cm (Fi Mi,t Ci,t, k) Softmax(Tas).)
This leads to the following cross-entropy loss for the score function and choice model parameters:
L (0) Ei,t-B, [- log (Softtmax(7ss))))..
In addition, we add regularization term to stabilize the CLIP model finetuning by adding & norm between the the original CLIP embeddings and the finetuned CLIP marked as (
The total multi-turn loss is then given by:
[(0) =
L (0) + K2 por--peffpe (0) + rross pross- ())- + (0),,
where 62, Kig and 14 are balancing coefficients.
D.4. Architecture
Our score function utilizes pre-trained CLIP- (Radford et al., 2021) text and image encoders, augmented with user-specific encoders (one per user type). These user encoders transform CLIP embeddings into user-type-speciffc representations, capturing individual preferences for images and prompts. The final score for an calculated the inner product of the user-specific image and text embeddings, scaled by learned temperature parameter (see schematic of our arch. in Figure 3).
pair is
We modified the CLIP encoders by removing the output 12-normaizzatinn User preferences are modeled by a four-layer fully connected network with ReLU activation, structured as dCLIP dCLIP x 2 dELIP x 4 - dCLIP x K, where dELIP = 768 denotes the output dimension of the CLIP encoders. The final layer is partitioned into vectors, each of dimension dCL.P, representing the different user types. These vectors act as residual offsets to the original CLIP output, individually added it before final lg normalization ensures each user score falls within the range [0, 1]. Each user score multiplied by leamed temperature parameter. This architecture is illustrated in Figure 12. As for the temperature model, a two-layer fully connected network with ReLU activation, followed by scaled Sigmoid function to bound the output in the range [0, ]. The network receives the utility score for each prompt candidate and produces the choice temperature scalar at the output.
21

---

Preference Adaptive and Sequential Text-to-Image Generation
Per User Type Addition
User Encoder
CLIP Encoder
Normalizaion
CLIP Embeddings
K User Residual Embeddings
User Embeddings
Figure 12: A diagram of our user encoder. The same framework is applied to both the text and image encoding process.
D.5. Hyperparameters
Table 1: Main training hyperparameters
Learning rate
Cosine annealing scheduler (Ir=3c-4, T=10c3) (Loshchilov & Hutter, 2016)
Training steps
50e3
Batch size
2048
Update target network phase
256
Optimizer
AdamW (Loshchilov & Hutter, 2019) (weight decay le-4)
K1
Oprior
0.999
Table 2: Fine-tuning hyperparameters
Learning rate
Cosine annealing scheduler (Ir=3e-7, T=10e3) (Loshchilov & Hutter, 2016)
Training steps
50e3
Batch size
8
Gradient norm clipping
0.5
Update target network phase
256
Optimizer
AdamW (Loshchilov & Hutter, 2019) (weight decay = le-2)
Rig
0.01
K3
K44
0.1
Aprior
0.999
3
22

---

Preference Adaptive and Sequential Texteto-Image Generation
E. PASTA Details
We employ PASTA, a novel framework designed for prompt selection interactive image generation scenarios. PASTA utilizes Implicit Q-Learnng (IQL) train the candidate selector policy that selects optimal prompt expansions based on user interaction history. This section details the training process, including the IQL objective and adaptations for token-level training; the architecture and functionality of the candidate generator and selector components, which employ large language models (LLMs); and the hyperparameters used for training.
E.1. Training
Generally, would like to optimize the Q-Learning objective:
( (malhh Pr) Pt) - (
2) )
[()) = E Bb Parehee Ptt
as(4, P) (PPL(
max
where Ø the target network and Pi {pt] be is the candidate set at time t. Decomposing the slate value function into an average of prompt value function we get
(8 P) ) PEPa(PEtt)
((Ø) = Prn.
T.
Σ So(hu, PEP
max
Although the value function is trained on the value of entire utterances, we enhance its robustness by also optimizing on sub-prompts, focusing tokens within the prompts that share the same target:
(+ PER
(
Tp
)
21
(()) = E. Parthee UND
folhe,ptit') T
max PEPL(P(t1) 92(h, P)
where Tp the (token) length of prompt p. Offline Reinforcement Learning struggles with overestimation of values for unseen cate-action pairs when using the Bellman optimality operator for TD targets. implicit Q Q-Learning (IQL) (Kostrikov et al., 2021) addresses this by learning an a-expectile value, Ug. which avoids the explicit maximization over the Q-function. The IQL objective is as follows:
L(ø, 1) En,, Parihe (us(he, Pt) - re (x4(4)1)) Lg (1g(he, Pt) - us (1x))
where (x) |a [eco)} x², a E [0.5, 1). Plugging in the decomposition of the value function and token-level training we get:
L(6. (/) = P
T
Σ
fo(he,p2:t - -ug(hat))
+
L
Σ PEP.
I
1
fz(he.p)
2 (he)
PEP: t=l
E.2. Candidate Generator
We utilize a multimodal Large Language Model (L.M) based on Gemini 1.5 Flash [Team, 2024] as our candidate generator policy. This LLM takes the user's current interaction history as input and generates of Le prompt expansion candidates, categorized into distinct groups. To avoid overly long prompts, especially in later interaction steps, impose length constraint of N words on the prompt at each step t 1..... H. This constraint is dynamically adjusted according the formula:
Nt-1 m N
NO.
10
H
23

---

Preference Adaptive and Sequential Texa-to-Image Generation
where Naar represents the maximum permissible word count at the final step (N < Ww and N is the initial prompt's word count. Candidate generation employs nucleus sampling (Holtzman al., 2019) with 0.8. The structure of the candidate generator prompts is illustrated in Figure 13.
Candidate Generator Prompt Structure (example for t=3)
You an agent interacting with user. The wants generate image using off-the-hhelf diftusion model. The user starts by providing prompt for the diffusion model.
At every interaction step you (the agent) must decide on twenty-five (25) possible prompts that would satisty the based on your understanding of their preferences given your interaction history with them.
Your prompts should be diverse enough to help you quickly capture the user preferences. new prompt could be change the prompt itself, or an expansion of the prompt.
prompt should remain consistent with the original prompt.
At every interaction slate four (4) of the prompt candidates will be used to generate slates of four (4) images each, every
for total of sixteen (16) images presented
the
The user will then select exactly one of the four (4) slates images (l.e., corresponding one of the new prompts). After five (5) interaction Your objective is to find an optimal slate of images for the user in the final interaction, based on their selections.
with the user, the will choose of the final image slates.
Output your slate in the following format. You should decide on five (5) categories, each listing exactly five (5) possible prompt candidates. The format should look like so:
<category__t 1>
new_prowe htt___ppopppd__ddeee candidate_]
new new_prompt_comm candiit_e
new htt._poomp___caddddeee
<category_55
hww__rrmptt_annddee candidate
new new_prompt__omm candidate_2
new cammittcandidat__5
Below is the current interaction history with the user:
pnitiat_prompt: "(K initial__pone promptt }
initial_image_oute ootpptt << image_samples >>
AGENT: Agent shows four prompts (with corresponding slates images) the
1. Prompt: "{{
exppaniinn 1y,", Samples: << image samples >>
Prompt: "[( prompt_essee _expansion 2))", Samples: image samples
Prompt: "{{ prompt_ee expansion 3y>*, Samples: << image samples
4. Prompt: "({ prompt_ expansion 4)),, Samples: << Image samples >>
USER: selects prompt number {{ image_number }} corresponding
prompt: "({ prompt_expansion }
AGENT: Agent shows four prompts (with corresponding slates of images) to the user:
1. Prompt: "CC eeppansix ,))", Samples: << image samples >>
Prompt: "[{ promp__xxansiog 2) Samples: image samples
3. Prompt: "(( prompt_exee expansion 3y>,, Samples: << image samples >>
4. Prompt: "{{ prompt_expanse expansion 43y,, Samples: << image samples >>
USER: User selects prompt number {( image_number }} corresponding to the prompt: "[{ prompt_eee eeprppsteeeniioonn >>"
Based on the interaction history above, please provide twenty-five (25) prompt candidates to the final selected prompt.
Output should divided five (5) categories, each listing five (5) possible candidates,
described above.
The prompt candidates should be prompts that can be used as input in diffusion model. The category titles won't be used as input. One categories should Any prompt candidate should be at most {( num_ wordsssss// }} words long.
prompts that very different than the previous chosen prompt (for example rephrasing).
Figure 13: Template for the multimodal candidate generator prompt at step = 3, with slate size L = 4, Lc = -25 candidates across five categories (colors for visual distinction).
E.3. Candidate Selector
The candidate selector policy leverages the prompt-value function:
T (h) E arg max go P)
arg max Sohh))).
PEP (())))))
PEPL (Ir) PEP
This policy is initialized with pre-trained Gemma 2B model (Gemma-Teanm 2024) for prompt-value estimation. In order to encourage diversity and exploration, we constraint the candidate selector to pick only up to one prompt from every category. Given an input (h. p). represented as a token sequence of length Th + T", the transformer produces an output of the same length, with vocabulary- logits for each token. We designate a specific logit, 4gp as the "q logit." The value for a history-prompe pair is then defined as the value of this q logit at the final position in the output sequence, corresponding to
24

---

Preference Adaptive and Sequential Texae Generation
the last input token:
fo(h.p) (LLM(
where [] denotes concatenation, We also employ the same model for the a-expectile value estimator Up (setting 4/ = d):
(h) (LLM(h;
where f, represents the a-expectile value logit. Empirically. we observed optimal performance when fy and lg are identical. We prioritize efficiency and a small footprint for our prompt value function model, making Gemma 2B a suitable choice. Since Gemma 2B a text-only model, we represent the history as text, encompassing the initial prompt, selected slates, and user-chosen prompts. When evaluating a tistory-promp pair, the candidate prompt is also included in the input. The prompt template detailed in Figure 14.
History-Promp Prompt Value Function Input Structure (example for t=3)
INITTAL PROMPT>><< {{ prompt }} <SELECTE SLATE>> SLATE>>>(<Promp ((prompt (prompt, 2 (prompt__3))
((prompt_4) 4)) <USEE LE---UUSSCCOOSSSSEE CHOSEN_ PROMPT>> > {{ chosen chosen_prompt }> <<LEECCEED SLATE> > <<ATE=> (t)ppp__))iilll ())]((/promo_2>>)))((00ooo ()))
((prompt_4) <
chosen_prompt } <*PROMPT_ <<FROMPT_CANDDDTTSS CAADDDAAANDD (( candditaa }}
History Function Input
(example (=3)
<<INITIAL PROMPT>><( (f initial prompt <SLEECED SLATE> > ((prompt (prompp 2) 3)>1
((prompt 4) <CSER WHOSEN_PROMPTEE (( chosen_ prompt } <<SELECTED (ut_etttttoooooLtLLiiiiottooooen ((prompt_44)
{( chosen_prompt }>
Figure 14: Text-only input templates for the value functions at step f 3 with a slate size of The top template corresponds to the history-prompt value function, while the bottom template corresponds to the prompt value function.
E.4. Hyperparameters
Table 3: PASTA hyperparameters
Learning rate
1e-5
Training steps
le4
Batch size
128
Optimizer
Adafactor (Shazeer & Stern, 2018) (weight decay le-2)
Gradient norm clipping
Update target network phase
256
Expectile parameter
0.7
&g
651
L
651
L
4
M
4
Le
25
Number of categories
5
H
5
N Amas
62
25

---

Preference Adaptive and Sequential Text-to-Image Generation
F. Additional Experiments
This section presents additional experiments conducted to further validate the effectiveness of our proposed framework. first evaluate the performance of PASTA with simulated users under different training data regimes and reward settings. Subsequently, showcase PASTA's ability to adapt diverse user preferences by visualizing interactions with various user types abstract image generation prompts.
F.I. Simulated Users Experiment
We performed an experiment with simulated users, leveraging our trained user model. Specifically, we evaluated our trained value models across three configurations: (1) trained using only human rater data, (2) trained using only simulated data, and (3) trained with mixture of both. These models were compared to a random sampling policy isolate the effect of the trained value models. All approaches employed the same candidate generator, which is based on the multimodal Gemini 1.5 Flash (Gemini-Teann 2024). The experiment was conducted two reward settings: (1) dense reward, where the utility function provided rewards at every step, and (2) sparse reward, where rewards were given only at the final step, focusing on scenarios where only the final outcome matters. We evaluated all methods over 1000 trajectories, each generated as described in Section 2.2. The results are summarized in the table below:
Random
Human
Simulated
Mixed
Sparse
7.38 = 1.13
8.7 0.11
9.26 0.07
8.81 = 0.06
Dense 38.57 5.81
43.49 0.54
46.35 0.38
44.03 0.32
The results demonstrate that all value- methods outperform the random sampling policy. Notably, the value model trained exclusively on simulated data achieves the best performance. This outcome aligns with expectations, as the simulated data matches the dynamics of the simulated user model precisely. While human data offers some improvement in isolation, its divergence from the simulated dynamics reduces the effectiveness when combined with simulated training data.
26

---

Preference Adaptive and Sequential Text-to-Image Generation
F.2. Abstract Prompts with Simulated Users
To visualize the differences in user preferences and PASTA's ability adapt them, we ran PASTA with various user types using broad, abstract prompts such "an image of happiness." These prompts imposed minimal constraints on the user model, allowing each user type express its unique preferences freely. For preference emerging, favoring specific styles or content. All users starts with the same prompt and initial images. We present only the images and their corresponding prompt-expansion at the last 5-th step. Each color represent a different user type. The results are as follow:
types, observed distinct
"An image of happiness":
"An
User Type
Rollout
Wostract happy faces Art Deco inspieed promento shopes, mused jowal loned backgrourd"
Dynueric Hherpay anrple alspes caloes, rellecing eeubbrant
"ocstrac garden soene with birds, composes of geomentc shapes and vibrant colore"
image hopolness
User Type
Rollout
Rollout 3
Rollout
vilirett, eaprisossn ve plecaw of wil repreeeninn The nwwsthddming foeling of happiness*
hoporica bac poining ecstate ahioron playing a vioran, sun crenched park, conteti talling, dynamic motion bur"
Vietaentioo painting of downa heppy mantie playing with DOERE delictte brushstrokes, sorend mood"
Image hopoiness"
User Type C
Rollout
Rollout
Rollout 2
"A teid of luminnus fowenss indesoent insects, and tentariial creatures buthed the soft light of double ralnloow"
"A Feldd suntowers bethed in den ou nurlight, with vitrent colors and intricte detals, photoreal stoe
"Goden hour meadow: wishooer bathed in worm surligt atmosphiere, photoroalise"
27

---

Preference Adaptive and Sequential Text-to-Image Generation
"An image of love":
Ap of Itewe
User Type
Rollout
Rollout
Rollout
goidon light,
lluniates
ulderfy moment light,
enhancing
Win mmge of lovee
User Type B
Rollout
Rollout
Rollout 2
"Avisual exploration
inlensly, forms, dynamoo
"Alacesty tostures, rough interooerr portraying abenglts
poomeett represerting love, stylo, paste
family
"An image of Inve"
User Type C
Rollout
Rollout
Inoressioniste portaaal family birthday emphasidingg light,
"Famly drnner dapictud unizual enggle unexpected perspective etiguee
geomatric represertin ding style,
28

---

Preference Adaptive and Sequential Text-to-Image Generation
"nn image of hope":
"An imsgg Inrw"
User Type A
Rollout
Rollout 3
Rollout
AArl Dwcn style uubiation pastel colors, cheryy blossoms on lase*
AA Ausreet depiction fresh blssaaming fiower represontin new love"
Appwwrl reprosentation reproserted with genti colors and forms"
User Type B
Rollout
Rollout
"Aphomnic riing from aumouaned elnent and dynamic energy"
Playfu, colirili light pariccee Forning celestal pattems, symbo ng hope and littloos possibiities
"An alsened
brihht reclant, oppiiridiio
colortll owiss
Image hope"
User Type C
Rollout
Rollout
Rollout
shom.
"The koft beacch sesshell,, the cocan's gende woves loop at shore"
"Clossup phonore shot of fimmy's feet walking beach duting sunset, warm cand"
peace and hog"
29

---

Preference Adaptive and Sequential Text-to-Image Generation
G. PASTA Examples
illustrate complete 5-turn interaction between our user model and the PASTA agent with examples. Each example begins with the initial prompt and images. Each turn shows the agent's 25 generated candidate prompts (organized into categories) on the left. Four of these prompts (highlighted red) are offered to the user, selected by a sampling policy based on value function. The four generated images corresponding to each proposed prompt are displayed below. The prompt ultimately chosen by the user is highlighted in blue and shown above the image set.
Example #1:
Initial prompt: going
Turn
Agent
User prampt number journay,
Catesonc Vaniation in Liphiis and Time Omy
3 tráin under stormyy dky.
4 tain at right glowiing lights
Variations
Compostiion:
0 long teight tain on tmale.
7. modewmn parsenger
books.
colorful
douulic docier suin on Vablo
Satgagry: Vadiliion 3t. Enstrammnn and Sotlios
miuunia Pack.
14. Train paasing thro-gh Ioreet
anowy lendecape.
Campletalyy
(Banteasingl:
Hailroad
Tr. Medal beset on sdeel rall.
Locomolive speeding onward
wheels
Category: Adding Artintic Styles and Effectt::
Train trackis impre salonist style.
hypersal
Minne
24. Tran dota ponell skatoh.
25. Train on sodis, viorant colers.
Turn
Agent
User
Gategarx, canding Scane,
selocts prompt number 15:
grean
dreamscape setting"
Train
dramads
3 A vintage dwam toin crossnng righ lrigge over river.
4 Ralmnod mode stresding horzon sunset over Bolds.
passing
countryside village,
Focusing the
6. Majecto steem boommiiiee pulling lucurous parsenger cars
Moders high-epeed speading thouugh tedecase
git canying
13, woathered vitagge
boouthul setting
Catereex Actinia Stiiss
lindsopp,
style Van
tran
Photoseslees passing
14. wh maiai cariomndlyee tran in herney bandscape.
Train vibrant, surnea dreamacape setting.
nest
ansking thouuh winding valley.
12. View from rece tnan car, bockng out scanery
long aopesure
reght, aggt
dramaticall.
from the trainss cab, looking
tmike.
Categery: Repitrasing and Variatione::
Sown
journyy froo duems bestec
Reiraat achention,
24. Journey careeyla apcotacular counthysick.
25. Train rals, bosutfll and loylic nolare.
30

---

Preference Adaptive and Sequential Text-to-Image Generation
Turn
Agent
User
Category: Surmal Landacapen Traino:
prompt number &
Astasm tann thiggng thougn x held ol gantt intressent flowos under a swdiing galeoy.
"A surraal train station nestled within a giant, glowing mushroom foresi"
2. Aviiiistc train jouncy trroug o candy colores forest lo lplp recs, and may marshmmllinn low clouds.
3. fituaai twnn giding over nentow teiqge w city made of cystad and flosting labents glowing
anitauu station nestled within
tellight forost.
Cateocry, Emphania Vitransy Color.
5. vibannt i. Iypareukiii ben spanding ng through a landscape of impoastlyy bhy bright ookara.
7. A peychoceli tran nie through a kalerdoscppe of owiting pattene and neon ligts.
made
glase traventing lincecape equalt
staledd glass.
though features though feed glowing sunfoower Eategery: Focus on Dreamitte Armosphace:
majeallc fran golden
11, A lonc vain disepco oa iing into hazy creamiss lic log, loaving trail of of ahimoingg sorrust train flasting affortienaly above wee cloude, bathed sott, otherwal light glicigg slendly a lantecapp
dreamilii stadon, where
platorms tathed moonlight
bas compated of stiling marphing ahraps,
trough
changing landhacape.
Catenacx, Diffecent Intecoesiation the Paamtt
Irin jurney trough impossiie bie aendccaee detying grawiy and perspective loeling rg riding sun in trant cream, expresing ig frsedom des
othenworiily
fill vieeer color 20 train's pansage through vitrant dreamacape, amphasizing the contrasd between reality and fentway.
sumwal cinamliie depictin a tnin
Call Moda92Ex_.dll Sacolfflss..lss.... Strias
bas -drard meml Sheamacee atyle 1950 contonn
foouring rondered styln
Dall.
Aiphonso Mucha.
hua tnnwling Brough vionent surmel dresmmccpp lacue Nousea style.
25. A vibrrtt sumeal eaarrsaappeee with twin saing bard, aogressivel bnalnarrkes or Imgressioteiisggg style.
Turn
Agent
User
Category: Vibrant Mushroam Foreats & Trains
User selecs number 13:
T.Aiiinzac toan station nestess amongst delunneeeenn mushrooms with a stsampurk zesthotl. vitenl, mixid men sceeding trough a fentest foweel of glant gloweng matrrame
"A train composed of metting clocks and surredl objects a Daliesque landscape"
syyld, glowing mushroom S. A nen pulinng sleam, ceesing through durse lorad atterwalily masemoom.. Cateeace, Surmel Landacansa. Iraimn
thouugh emergiing detailed Illustration
magical atmosphere.
5. Ta n giddng across a rainbow brlgge i-- summa creamiike aantacape
Aran Frvwning landscope foclling islands
archtcctur
joumsy tontootical andeeccc Bled with Ne form 10 insin dhocpporing im a meiting serwe of rslors in summel dheent kas landecape.
vibnart
9.
Call Gote Dallssoue Trainsseess
melling badk tran De beseep bendboup,, with datored perspection and mailing forma. elongsted
orain composed metting diocks
Doll landscape.
distrted 15. surval trea acane with destrted senpaadive and sell Blwing lines the styl of Dall
baa
caeert landecape maaling
Dell'
Cate Ethereal & Dreamilke Trains:
train floating through storyy night sky, with othereal lights and glowing constellations. thouuhh
dieamy
protell coles, doudss pesseruil trenyu stmoophweee
18 taan gierty glating thoough a son mady a with senene and prscetal htmoophane.
20. train traveling through a sea of clouds, bothed soft go Won light poacoful and serone.
Category: Completely Difherent Prompte (Raphrasingl:
thruugh mountaln cheesin black and white photto trin paasing trosugh bualiig ciyy. 26. An mage a thight train tneraporinng goads aking long ne hway lin.
realistti shollgpeph
hightspeed
steam train
25. simple draving of trann on bracks a strighttnwwors realisti styl.
Turn 5
Agent
User
Category: Dall encuu Variations:
User seleets prompt number 22: "An athereal undersater scena depicting a queen, her crown formed by Holuminettant crastures, radiating an enchanting glow."
T.A surreal train speeding through landscape of metting clocks and distorted figures, the style of Balvador Dalk.
2. menn compowed arriwlly mekting clacko, sumeel otjects. and imposs wechibetion, Dadleeque dheameceige. elongeled dream landscape, perseettvve Estvador Dal., 5. A surraa toin diatlon with distonis fed andhitccuure and metling dooks, in the style of Salvador Dall hysomcal ctc. Catepocy. Enhaaced Surrallam:
Dall-eeuu
through
lims
traiis made liquid modal flowing through a landscape shifting colors and impossible geometries. glddng through vibrant sureal dreamscape shopes, cookk muuuded distorted paychecel 10 insin jounsy though a hygermalitii, surreal tendecape with unsopected elements and juteppontoon.
fortaadcal creatures impossibe structures.
n
9. emerging from glant,
imporsdble
Cata pate Eacua.ssa on the Train:
Datededd drowwu somel bwin engine with me ting chncks and intricate dwoits
perspective of
insido Dall
whncssing surcal landscope.
Highty
nemtering
amrpheddiing wechares malwisch.
15. sumeel trai hanunng on in cnuuaa design and mecherrs..
Cato pory. Diffrcent Actistle Styles:
1:ann surres ahodsoppe
style René Magrite, with focus on uncxpected Iotapootions. uneopected combirations.
sumwal train through
the styl Max Emrdl fest collage lice
18. A sumnel train dep cled a vbbantt pop wt styin with hold colors and graphic elemeris.
20. A surreal train a cubist landscape, emphesszing fr fragmented forms and multiple perspectives.
Category: Rephrasing Simpliicctioon:
meling clock nwn Dhll world.
26. Sumeel lman in divioted bendbccppe.
25. drsomiikk toin jumey trouuhh a strange land.
31

---

Preference Adaptive and Sequential Text-to-Image Generation
Example #2:
Initial prompt:
"A princess wearing an animal skull mask with ribcage skeleton and cybernetic jellyfish fnatures'
Turn 1
Agent
User
Cate Eshansed Detail & Lighting
unimall and mask, oyberedic jalyfah, intricate details, dramatic liitiing 2. Prineass skull mask, jubyfiah meteett strere goo tspervalatte, closs-up up
1. C. pertiat,
Use: selects promot number 8: "Close up, princess, animal shull headdress, jelyfish, lighting"
Marm portel prinosos bone makk akmminweert jalfish stay tocus, rich color
Detalld portist noess, omate
glowing cypomotic hhitsh backg cund.
lighting
Category: Abemativve Maak & Cybametica:
6. Peimraee portost, exgart ted acall moes, subil cytee wil antrenomentss soft lightng
7. Cless up princess futuritle nncc mask intograeed cybometie jolyfish vibcrtt zolors
Mocro
prineess omato cragon mask glowing sytometio
blumed baskground
background,
minimallet sytemetic Category: Varfations in Background& & Setting:
Portssk princcss,
sharp focus.
11, Princess, shull mask, cybemetic jullyftsh, underwater scens, shimmering bubbles. Portsalt prineess, bowe mask, oybametc enhanoements, mysical bockground prnowss,
diyecape backdrop Provoss, dort meok telurrnweee wiyfish oherworisty netsin
15. Maoro portall, prinosss, emato mask cybomitto doments onchanted gardon sotting. Category: Different Artanii Styles:
Prisoess, abuil maok cytemneic jellish resover style wlepart and boaeng frwws princoss, bone
orbomoti orhancements, dark
st/c.
photorolletic Prisoess, slul mark tellurraseerr yiyfish arl style dhamalic lifting.
Close
20 Maruu portnal, priszees umata makk opperetts elemants dagial parring style Categery: Completely Diffecent Concept (Raphrasing::
rugal weman, adommed with iniiate sone joustry and gloiing bnduminssonn accerts. mysttrious
subtie hore struttowe.
powerfd with rytweeeto entrnnameete binding seartisdyy
24. caprusttnng portoil of somes, combaning elegans with futeee tinmechantzz alements.
25. An ofhereal being, Monning organis l and syrthedd cements in hamoorccus compositonn
Turn
Agent
User
Gaberacy Refined Dataila & Listting:
User onlects prompt number S
Clos--pp it portall pronosss Inticale sbudi healthess, bit mmbreeeent jeiydon, drametic lighting at seeois tion lighing shopp bacigrourd Macro portalt princess, skall headdress with jollfish, charrosouro lighting ethercall stmospheree hyperealistic
"Close-up: up: pr princess, shall headfress, bio bio-mechanical jollyfshh noon lighting, underwater effeer"
jeayain
ese .p priccese shull mask, tio mechan col jolyfsh, cinemati lghting rich tonuree. Gatorsary Variation on ielldiih.
je
lightng
Croe-uap porbad,
skull hachhress indeeceet
mosely arroaenca, dhtalls.
Princess socletal mosk sybemetic jellfish owam, deep ocean setting. cinematic lighing.
Close-up up- prinoess, skull headsress. blo-mcchanccll jellyfioh, noon lightigg underwater effect. Phemoass mask pubsuiin jalyfah, background. drumati lgiting reacuatio. Category: Emphassis
inticately carved skut mark jelyfish wooerts, diaamitc lightin 12. Prinooss. detalled and stull mask, bi mechanical jolyssh, dark fartaty d'amade ligting.
13. Close up: princess, omale skull hoaddress. bio median on (olylsh, dark farlaay art sharp foous.
14. Pherosss
skull maok syternils je bylkh, dark bhekky cound, cremal tyring decaying
beature
pelyfish,
lighting
othereal
Calleooy: Compldally Different Skyle:
18. Etepert prrcesos kour als crown, sutt techanooogical dasells bs eofi lighting, cootal oolars
17. Portss a regal woman, othensal crown, flwing gwin cott liht, serene exprossion.
powerfil queen, majestic pose. omate jowo
lighting
20. Seautiui waman, ammle seabband ratural loft, lecus creamy atmesphere
Calagory: Exploring Backgrounds& & Settings:
21. Glose-up, prinoeso. shull headdress, bio alp-mechanical jollffshh Toggy forest background.
22 Penooss,
folyyfsh crown, onnmmat lonting, anciont background.
Phensese skael meok jalyfah, cisematic gattic aechilction backgoound
25. Close up: princess, skull headdress, bio mechanical jolyfoh, story night sky bookground.
32

---

Preference Adaptive and Sequential Text-to-Image Generation
Tur 3
Agent
User
Galleeey, Noon DreamseiaREs
selects promet number detailed "Prineess, Jellfishhh todures, ncon lightig undorwater cave''
roct, wbbart underwaler Undorwater princess glowing jollyfish heoddress, neon lights, deco style.
Cyberpurk princees skul mosk with neon jsky)ah, fulriiti underwater dit.
caue, heattivess aesumiteecer jl yluh.
Categary: Subtie Shitts je
cosen, sysitross Backgronnd:
Chosevip portoel precesss shull headtmmee, bolummeceer jakyku, stff neon glow, dart background
Pinnoss soull jo byftch crown, undenwater soene suibi neon occents carr atmosshere
Underwater prficess
teattresss bi boo peoeoniccall jelyfichh postal noon background
scall
deop
arttiance
10 Close-up princess, skull readtress jelyfon dir neon igriing andlyy underwater cave.
Gataezey, Virationa on the Jntlyiaas
Princess, akul hosderess, glowing jellyfish tentacles, neon lights, ocean.
18. Undemedter prineoss, ahut hea daross ticlum rescom fflh swarn dieon glow.
14. Clcenop prineess abarl mase, intecale july headSrnen neon cocel, deep nowar
Prinoeex, abi headerees homeertorio pelyiih, neen keep toreel frowing fetris States
Categary: Complately
16:
intricale
teckire, dramali: lyghing
17. Elegant woman, mystorioss headdress, flawing gown, detailed portralt. studlo lighting.
18. Nomon with a copthariigg pone elatorone heasdress. rich colors, paintorly sty
up portolt strong fourre originlli expression, totures, stucls cotting. Fastion
Cateeary, Eostaniann Taature and Datail:
21. Pypareel als portaat priceess dataded stu headdrras, tro-mader kasl paly naon lighting underwater scame.
Princess, skull jellyfish crown, decailed teaturee. lighting, underwater
24 Macro pornet prioones instudd skul mase, Ni do--mechnntall jekyloh, neon glow, underwater mades.
25 Denll ed portalt princees, auill heattowss mnlledic jelytoh, neon sccentss coean floor
Turn 4
Agent
User User selctis pronct number 12 "Clase-up, princess with jelyiteh crown. highly tootumineacant patlerss, underwater resolation"
Gatbogry, Noon Und Prinecssss
princess adomed with Molunineseent jollyfish crown, swimming in vibrant noon coral reef, intricate details, resolution.
glowing
undensater An wtherol of resse with looiig ng hei a lylih heedhhess em ling swn liftt anvirming gracddlly am the glowing jelyishh
Portnit i prinzsso her
cbacuied
wearing pottapee palylie crown, vibeatt rwon kyrtig underonte cacw sedtngg deese ad tellume. Cattogery: Dark Fantasy Princesses:
Clore-ip
underwater
weoring
of glowing joltytsi and sketetal elements, mustorlous ofhereal
7 Dark furtasy portool a princess with jollyfish crown mace sone and neon lynts, sumounced deep tea oraaures.
A gottic princess wearing healdreess crafee decoying jolyfsh and skulis, bathed in noon underwarer light diaaalic 2006
mystertuu dus uncereater prrneess
vollec by jekylian hesadrrsse glowing neon accents, dars and moody atmosgmare.
gowing
Caleoary, Detelled auliiiah Headeesar.
11. Exremicy detalle porsut prnooss wouring an intriatt jullyishh hoaddress, neon lgtting, underwater setting real si sodures
12. Cinse-ip prineess with jalyrish crown, Seaturing highty detsiled tentacies and tooluminesennt pattern,, underwater scane, Bk resolation.
dedt lighting undensoee baskgmandd dinwreekd lgghing
glowing 15. Highly delacled image ol princess useeing plytah crewn, showcasing the inticall wertowe and rach ginn underivteer wilttommet
14. Princesx porfont: waborste
Gallogey, Alternaties Intomniiaiionss
regal figure, parsally obscured by mask of loateyy woven light and shadow, with
17, coptivaing woman, hor fooo partilly iiddon a fantastical heacpicce romin scent of deep-
see lhs, balted in wtheread light.
mysherizza and slegant wommn with ceddeeeee combening ny natural and tederologior elements
TO, An eragmate lqure underwater asting astoned with crown of athareal kight and anoicate, tectunenescannt partarrs.
20, A Aidlon of einganin and mystery a women with unique readplece repired by mathe life, emitiin gerdo, glowing light
Category: Styliaiic Varlations:
portolt:
wearing a geomettic jolytsh
undewwaler ontinng eiegent compostion
22. Surraiktt portuit prinoss with a metinng nn jolyf crown, noon light creamt be undoreater soens, oxperded elemerta.
23. improsserrste ritle undomester portaal: a prineess with jolldiin clow,, soft loous, passel neon oa ors, floving lines and shapss.
princess a jetlyish headdress style Alahonse Nucha, diegant lines, soft noon lighting, underwater background. painted
Turn 5
Agent
User
Cathgory:: portaal prossss
Lighting
User oelects rumber 22:
titueeeeee wlyfish crewn, imioarie dets testadee, zottt undersanee righting reeokaionn hypermrliiic
"An otharnal underwster scene depictin a queen, her crown formed by bisluminescatf sea creatures, rediating an enchanting glow."
Pri Privoues with a jelylish omann detxled wetachee with indewoent kradee, soft bis and sinc lighingg undenaster woetn perene eqgrenalen.
Lidderare portal pricesss je byich reen with glrning accente, sutrie shimmiing partidee ci lerting aaply candied teokures
prnsess, Miiic crows, viscett coral badiground realisle rondering, shars focus, dccllled textures lggting.
anatomy, submerged crystal
otmosshhrre
Galterey Striisoo Vadallonss
Nourcou poro with plyfich orewn. Towing licss colors underwore sedingg elogant composition on
jalytthh crown, Deca style, geamstric patterns, neon lighting underwater Art Baco architecture.
perspective
9. impresstinal
ady crown. sofi tradiaattiees undenseder trumed backgronnd paotal huse.
1o, Prancess, jilyfih croen, the style of Gurra Kim, gold leal accents, undansster soer at smmannn afrect Calngory: Rnstronmental Changes:
background
vbrant snaliw sunkg flaiiig 13. Prisssss jellytoh omeen anciett sunion city, nine, mydtca undersater atmosphere dets led mecuees cho lighting
Princess, jelyfish
glowing cond reef cherse
14, Princess, jsllylsh crown, mysterious undenwater cave, glowing cryslats, othercal light, sense of wonder BK resolation. Undenwater prineess joty crwwn sumou
schools lumrnnoe fal, viorant crysia dlaa water enemaai lighting
Internstations
Choeet portat princese with reettiee made inticasely
jelybun, underwsiv afheree lightig
17. Prinsexx umste hestomes repored by ye witsh strooe dets bn Bowing behrr underaster conerst: iighing it neeciaior. Portmet prrcoes creer woapled plwd from
inwrieiiiggginnggg underwster toolggound mysteraus amoophew
lightng
headdress orated trandluce lifihh glow undorwatr Callogery Comdestlly Differant Promeits Rogheiiings
Close
bro laking undenwiler portait rogal woman. odomed with lumitous, organic hoodpeess, evo a sense whorwordyy beauty,
An underwater osene depioding a queon,
formed by bsoluminessint see
rodlating an enchanting
imaga
Squar submeerged
mesamen underwaler gortost featurig aereme with crown made glowing jellyluu, shooccsiin intraate detatl and visuut 25. Ar slegert women, her head crowned with a daccing doglly of beclumineaeeet jellylshh creating a sturning underwater apectacl..
33


Reviews:
[{"review_id": "F6p703qWBH", "reviewer": "Reviewer_ym4T", "summary": "This paper introduces PASTA, a novel reinforcement learning framework for interactive text-to-image generation. It addresses the challenge of capturing precise user intent through iterative prompt expansion. The core ideas involve using a large multimodal language model (LMM) for prompt candidate generation, a value-based RL agent for prompt selection, and a user model trained on both human and simulated data to guide the agent. The paper also contributes a new dataset of sequential user preferences.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive feedback on the novelty of our approach and our dataset contribution for the community. Bellow we address the concerns raised by the reviewer:\n\n**Comment 1: \"...the performance gains from the user simulation require further scrutiny.\"**\n\n**Response:** The user simulator is used for two reasons: (1) generating synthetic data that is user-coherent in order to extend the real human rater dataset; and (2) labeling the human rater dataset. Our experiment with real users shows that adding additional synthetic data improves the model’s performance, indicating that our generated dataset is beneficial for training in real-world scenarios. In addition, we examine the ability of our user model to predict real users choices with 70% accuracy (Fig 3), which further indicates that our user simulator does capture some key dynamics of real user interactions. We will add a plot showing the user model score over time in order to add an additional performance evaluation metric. We will comment here with these new results during the discussion period.\n\n**Comment 2: \"The dependency on particular pre-trained models(Gemini, Gemma, SDXL) might limit the generality of the results.\"**\n\n**Response:** Thank you for raising this point. In order to address it we are running additional experiments with different models during test-time (i.e., evaluating using models that our agent was not trained on). Specifically, we are running an additional rater study using our method with the Flux.1 T2I model. We will also compare our method to the use of the GPT4.5 model as an agent. We will comment here with these new results during the discussion period.", "claims_and_evidence": "The claims are generally well-supported by experimental evidence. The human evaluation demonstrates a significant improvement in user satisfaction compared to baseline methods, particularly when PASTA is trained on a combination of real and synthetic data. However, the performance gains from the user simulation require further scrutiny.", "methods_and_evaluation_criteria": "The proposed methods are well-suited for the problem of interactive text-to-image generation. The decomposition of the problem into candidate prompt generation (using a large multimodal language model) and candidate selection (using value-based reinforcement learning) is a sensible approach. The LMM allows for exploration of a diverse set of prompt expansions, while the RL agent learns to choose the most promising prompts based on user feedback. The implicit Q-learning (IQL) algorithm is a reasonable choice for offline RL in this context, given its ability to handle overestimation issues.\n\nThe evaluation criteria are also appropriate. The human evaluation, where raters judge the improvement of images across turns, directly measures the effectiveness of the interactive refinement process.  The use of both absolute satisfaction scores and turn-over-turn comparisons provides a comprehensive assessment of user experience.  The simulated user experiments offer a valuable means to analyze the impact of different training data regimes and reward settings.\n\nHowever, the reliance on specific, potentially proprietary, pre-trained models might raise concerns about reproducibility and accessibility.  It would be beneficial to discuss the potential impact of model choice on the overall performance.", "theoretical_claims": "No Obvious Problem.", "experimental_designs_or_analyses": "The experimental design appears sound, with appropriate baselines and evaluation metrics. The analysis of the impact of different training data regimes is insightful. More discussion is needed on limitations on rater bias and diversity.", "relation_to_broader_scientific_literature": "This paper builds upon recent advances in interactive and preference-adaptive image generation, particularly those leveraging LLMs and RL. It extends existing methods by framing multi-turn image generation as a sequential decision-making problem, allowing for iterative refinement towards a desired visual outcome.", "essential_references_not_discussed": "NAN", "other_strengths_and_weaknesses": "Strengths: The paper tackles an important problem with a novel and well-designed framework. The dataset contribution is valuable. The use of both human and simulated data is clever.\nWeaknesses: The gains from the user simulation require further justification. The dependency on particular pre-trained models(Gemini, Gemma, SDXL) might limit the generality of the results.", "other_comments_or_suggestions": "Consider adding a section discussing the limitations of the user simulation and potential biases it may introduce.", "questions_for_authors": "Could you provide a more detailed analysis of the specific dynamics captured by the user simulator that contribute to the performance gains observed in the simulated user experiments? (Understanding this better would strengthen the justification for using simulated data).\nWhat are the computational costs for implementing and training PASTA?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IXPgdzLwHp", "reviewer": "Reviewer_udgs", "summary": "This paper introduces PASTA, a RL framework for interactive text-to-image (T2I) generation.\nThis method enables multi-turn collaboration between models and users to refine prompts/images iteratively to align the preferences of users. \nA novel dataset of sequential user interactions is proposed and a user simulator trained via EM strategies to model diverse preference types. Evaluations show PASTA outperforms baseline models in human ratings\nThis paper compared the situations of training on combined real and simulated data or training on sole dataset.\n\n## update after rebuttal\nThe provided additional evaluation addressed most of the reviewer's concerns thus the reviewer choose to enhance the rating to 3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer’s constructive feedback and for pointing out our dataset contribution as the first T2I dataset for multi-turn interaction setting. Bellow we address the concerns raised by the reviewer:\n\n**Comment 1: \"This paper does not mention the scale of user annotations.\"**\n\n**Response:** Thank you for highlighting this oversight. Our human rater data consists of over 7,000 five-step rollouts (totaling more than 500,000 images), annotated by approximately 100 human raters using a random candidate selector. The simulated user data includes over 30,000 rollouts (exceeding 2.5 million images). We will add this information to the paper. \n\n**Comment 2: \"The reviewers are concerned that the annotation scale might be too large, costly and not reusable.\"**\n\n**Response:** To encourage research in this novel setting, we are making all datasets used during training publicly available—we hope this will allow researchers who lack the resources to conduct such studies to benefit. Furthermore, we have proposed that exploiting trained user simulators, independent of large language models and easily trainable with the provided dataset, is an effective solution to mitigate data scarcity. Finally, we are sharing the synthetic dataset created by our user simulator to further assist researchers in addressing these challenges with greater ease.\n\n**Comment 3: “Generalizability of the method.”**\n\n**Response:** Thank you for raising this point. To address this we are now running additional experiments with different models during test-time (i.e., evaluating using models that our agent was not trained on). Specifically, we are conducting an additional rater study using our method with the Flux.1 T2I model. We are also running a study to compare our method to a GPT4.5-based model as an agent. We will comment here with these new results during the discussion period.\n\n**Comment 4: Analyze the synthetic data.**\n\n**Response:** Our approach to analyzing the synthetic data involves assessing the user model, which represents the sole distinction between the real human data and the data produced by the simulated user model. Visually, we can observe a clear distinction between user types by either scoring the top five highest-rated images in the HPS test set or by reviewing the rollouts of PASTA against the user model, both of which highlight distinct preferences across different user types. For a numerical perspective, we refer to Section 6.2, particularly Figure 3, where our user model achieves a prediction accuracy of 70% compared to real human raters, suggesting that the synthetic dataset effectively captures essential aspects of real user interactions. However, we agree with the reviewer’s perspective that further analysis is necessary. As such, we will add additional evaluation metrics to our experiments, including: user model score, FID, and LPIPS. We will comment here with these new results during the discussion period.\n\n**Comment 5: “...domain gap between single-turn and multi-turn preferences..”.**\n\n**Response:** Indeed, there is a distributional shift between single-turn and multi-turn preferences. However, our user model parameterization, encompassing both the user choice model and utility, relies on a single-turn score function $s_\\theta$. Consequently, we can utilize single-turn datasets like Pick-a-Pic to augment the data available for training the user model. That said, due to the distributional disparity and our emphasis on the multi-turn context, these single-turn datasets are employed solely in the initial phase of the user model training to provide a stronger starting point for the subsequent phase training with the human rater multi-turn dataset, which is our primary focus. This point may not be entirely transparent in our exposition: we will elaborate further in our discussion of the user model training process in the paper.\n\n**Comment 6: “Why was the number of interaction turns set to five?”**\n\n**Response:** We settled on five steps to balance between providing a challenging enough interaction length and meeting the real-time expectation of users for relatively swift engagement with the agent. Exploring the (variable) optimal rollout lengths is something we have deferred to future research, given that the PASTA framework includes numerous components that warrant deeper study.", "claims_and_evidence": "Refer to the questions in \"Experimental Designs Or Analyses\".", "methods_and_evaluation_criteria": "Strength:\nPrompt rewriting is essential for text-to-image generation. This paper introduces the first RL-based multi-turn interactive text-to-image generation framework PASTA, mainly for generating a better prompt for T2I that can progressively refines the generated results through dynamic prompt expansion. \n\nThis paper constructed the first dataset that encompasses multi-turn interactive behaviors of human evaluators. \n\nWeakness:\nThis method drawback lies in the costly data collection process. This paper does not mention the scale of user annotations. If a different generative model is used, would the same annotations be required? The reviewers are concerned that the annotation scale might be too large, costly and not reusable. \n\nThis paper does not analyze whether the synthetic data contains noise, the extent of such noise, or the overall quality of the data. Whether there exists a domain gap between single-turn and multi-turn preferences for models trained on single-turn preference datasets requires thorough analysis. \n\nGeneralizability of the method. It remains to be verified whether this approach would still be effective if applied to other LMMs (Large Multimodal Models) and visual generators. Additionally, the generalizability of the trained value model (in this case, Gemma) needs to be assessed.", "theoretical_claims": "The theoretical claims are sound.", "experimental_designs_or_analyses": "The article only conducted experiments on Gemini + SDXL, failing to demonstrate the generalizability of the method. \n\nWhy was the number of interaction turns set to five? According to Figure 5, the results continue to improve even in the fifth turn, necessitating a convergence analysis. \n\nThis paper should evaluate more metrics, in addition to Pick-a-Pic and HPS accuracy, other metrics such as FID (for assessing quality) and LPIPS (for measuring diversity) could also be evaluated.", "relation_to_broader_scientific_literature": "This paper holds significant implications for the application of reinforcement learning in the realm of text-to-image synthesis.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "No.", "other_comments_or_suggestions": "Conduct experiments on a wider array of generative models.", "questions_for_authors": "Could the authors perform a manual or automated analysis of the synthetic data's quality?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CzzCX0XC3y", "reviewer": "Reviewer_LRxo", "summary": "This paper introduces PASTA, a reinforcement learning (RL) agent designed for preference-aligned, sequential text-to-image (T2I) prompt expansion. The framework aims to enhance T2I generation by formulating multi-turn interactions as a sequential decision-making problem, leveraging LMMs (Gemma) and image diffusion models (Stable Diffusion XL). The paper evaluates PASTA through human and simulated experiments, claiming improved user satisfaction and model-agnostic adaptability.\nMain Contributions:\n1. Formulate the challenge of multi-turn T2I generation as a sequential decision-making problem.\n\n2. The use of IQL for offline RL and user category-based prompt generation is innovative, enhancing diversity and personalization.\n\n3. Use LMM(Gemma) and the Image Diffusion Model (SD-XL) to construct the generation framework.\n\n4. Collect sequential sample rater data and simulated data for multi-turn T2I generation and facilitating further research.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed review and hope that the following response addresses the reviewer’s concerns. \n\n**A: \"The lack of experiments with other SOTA T2I generators..\"**\n\n**Response:** To address this, we are running additional test-time experiments evaluating models our agent wasn't trained on. This includes a rater study using our method with Flux.1 and comparing our method to a GPT4.5-based agent. Results will be shared during the discussion period.\n\n**B: \"Could you compare your with other preference-alignment reward models?\", \"lacking quantitative and objective metrics\"**\n\n**Response:** We agree further analysis is helpful.  We will add additional evaluation metrics to our experiments during the author-reviewer discussion phase (by next week) including: user model score, FID, and LPIPS.\n\n**C: “Strict word limits..”**\n\n**Response:** The constraint arises from the pretrained CLIP text encoders used in the user-score model, which are limited to 77 tokens. To stay below this threshold after five turns, we set the initial prompt to 12 words. Additionally, the text-to-image (T2I) model has a limited context capacity, restricting the level of detail that can be included. Despite this, the model still accommodates a variety of starting prompts that can evolve into detailed and rich expansions. Moreover, keeping initial prompts short shifts the focus toward feedback derived from user choices rather than critiques, which aligns more naturally with generalizing the standard single-turn preference setup. We intentionally minimize the influence of critiques to maintain this balance. That said, using improved encoders like LLM2CLIP is left for future exploration.\n\n**D: “What is the intention of not allowing users to see intermediate prompts?”**\n\n**Response:** PASTA refines prompts preserving user intent. Users choose the prompt-image pair best matching their initial prompt, preserving core elements. This method keeps the sequence of generated images closely tied to the content of the starting prompt, reducing the likelihood of producing images that stray too far from the user's starting point (e.g., beginning with a prompt about a train and ending up with an image of a dog). We experimented with displaying intermediate prompts to raters but found that presenting these prompts especially given their often complex nature placed too great a cognitive load on raters. We also believe this could compromise the quality of feedback provided by raters. Prompt refinement and expansion are natural (and commonly used) interaction modes for users engaged in iterative image generation. Surfacing the prompts as suggestions along with the images is something to be explored, but might require more \"controlled\" prompt expansion to elicit proper responses. We will elaborate further in our paper.\n\n**E: “What is the essence of recognizing user’s category or type in a multi-turn generation process instead of explicitly using textual interactions?”**\n\n**Response:**  PASTA doesn't identify user types explicitly. Types only simulate consistent preferences in synthetic data creation. The agent adjusts to the user by leveraging the interaction history, adapting to their preferences. Our method offers advantages over textual instructions because users are often unaware of the prompt possibilities provided by the agent, and selecting images is significantly more user-friendly than drafting instructions.\nThat said, PASTA can incorporate textual instructions by integrating them into the candidate generator’s prompt, allowing it to focus on specific categories or directions specified by the user. Our dataset includes rollouts (20%) with text critiques for potential model training. We kept the framework simple, leaving text integration for future work.\n\n**F: “Better / Worse / Same selection results between different turns could be influenced a lot by psychological factors”**\n\n**Response:** User feedback inconsistency due to psychological factors is expected. However, average human evaluations remain a valuable performance measure. Our results show the trained PASTA agent significantly outperforms a standard LLM.\n\n**G: “Is the “sample rater data” provided in the supplementary material the full version of your curated human rater data?”**\n\n**Response:** Our human rater data, collected with a random policy selector, consists of over 7k 5-step rollouts (>500k images), hundreds of prompts, approx. 100 raters. Simulated data: over 30k rollouts (>2.5M images). The supplementary sample is small due to size limits. A link to the full dataset will be provided in the final version of the paper.\n\n**H: “How do these prompts align with real-world usage..”**\n\n**Response:** Abstract prompts help discern user type preferences during interactions better than typical prompts (e.g., 'dog running'), which often have universally preferred styles and biases. Supplementary materials include full rollouts for typical T2I prompts.", "claims_and_evidence": "1.In section 7, “PASTA operates by directly adapting the input prompt, offering a model-agnostic approach to adaptive preference learning in T2I generation”. Could you provide more evidence for this? As different T2I models may perform different given the same prompt settings, also LMM’s capability could affect this prompt expansion process. The lack of experiments with other SOTA T2I generators (e.g., DALL-E 3) or LMMs weakens this claim. Additional validation across diverse models is necessary to substantiate model-agnostic adaptability.\n\n2.Presented User Model improves user satisfaction compared to baseline(Gemini 1.5 Flash). Could you compare your with other preference-alignment reward models (e.g. ImageReward), T2I refiner(G-Refine, Idea2Img) or more advanced LMMs (e.g. GPT-4o)?", "methods_and_evaluation_criteria": "1. Strict word limits (12 for human, 10 for simulated) may not reflect real-world usage, limiting applicability.\n\n2 . What is the intention of not allowing users to see intermediate prompts?", "theoretical_claims": "1. What is the essence of recognizing user’s category or type in a multi-turn generation process instead of explicitly using textual interactions? For example, if a user wants to have realistic and scenic images about a Railroad Journey (Example #1 in Appendix G), why couldn’t we lay out several generation styles for choosing and then generate more style-aligned images efficiently?", "experimental_designs_or_analyses": "1. Human evaluations rely on subjective turn-over-turn judgments (Better/Worse/Same), lacking quantitative and objective metrics like image quality scores (e.g., FID, IS). This limits the assessment of PASTA’s performance in terms of visual fidelity and semantic accuracy, which are critical for T2I tasks. We may need more experiments in image generation benchmarks evaluating both preference and objective metrics like DreamBench++ to show that your framework could generate images having significantly better performance than baseline in more objective metrics.\n\n2.Better / Worse / Same selection results between different turns could be influenced a lot by psychological factors, only by sampling different images, users may also select “Better” at a probability, even if there is no significant prompt changes between different turns. The paper does not provide “blank” experiments results to measure this bias.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "I am willing to raise my score if the authors can address my concerns.", "questions_for_authors": "Illustrations (e.g., \"An image of happiness/love,\" Figure 6) focus on abstract prompts, which may not be representative enough of typical T2I applications. How do these prompts align with real-world usage, and what is PASTA’s performance on different categories of prompts?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0R07vxrmpF", "reviewer": "Reviewer_z2d3", "summary": "The paper proposes a novel method for text-to-image (T2I) generation that adapts to user preferences over the course of multiple turns. The problem is framed as a Markov decision process (MDP) where the initial state is some prompt $p_0$ provided by the user, each subsequent state is given by the history of interactions up to the current turn (we assume that there are at most $H$ turns), the action is to propose a selection of $L$ text prompts for the next batch of images ($M$ images are generated for each prompt), and the transition is induced by the user's choice of his/her preferred prompt among the proposed selection. The reward is given by a _user utility function_, which quantifies the user's satisfaction with a given prompt based on the images that were generated with it.\n\nThe task now becomes to maximize the cumulative reward, i.e. the user utility, over the course of $H$ turns by proposing an appropriate set of prompts in each turn. To this end, an agent is trained with offline reinforcement learning (implicit Q-learning) to select the best $L$ prompts from a set of $L_C$ candidate prompts generated using a fixed multi-modal language model (Gemini 1.5 Flash). Training data is obtained using a baseline prompt generator that interacts with human annotators who, for each turn, label their preferred prompt among the presented slate. In order to quantify the _utility function_ encoded in these preference labels, a _user model_ is trained on the gathered data as well as preference datasets from the literature. The user model's objective is to predict which prompt was chosen as the preferred one by the human annotator. To bulk up the dataset, a simulated dataset of user interaction is generated by using the trained user model as a proxy for human annotators. Finally, the prompt selection agent is trained using offline RL where the reward signal is obtained from the user model.\n\nEvaluation consists of three components: The user model's accuracy on human preference prediction, the user model's accuracy on user choice prediction, and the selection agent's ability to adapt the selected prompts to the user's preference. Positive results are reported in each case.\n\n### Update after rebuttal\n\nAfter reading the other reviews and the authors' rebuttals, and taking into account the additional experimental results, I believe that the paper proposes a good idea and executes it well. Of course there is still room for improvement: ablating the importance of the LLM and/or prompt as well as improving the clarity of the writing comes to mind. However, if the promised clarifications can be implemented, and given the significance of the investigated problem and the potential impact of releasing the full training data, I believe that the paper meets the acceptance threshold. To reflect this, I will increase my score from 3 (weak accept) to 4 (accept).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s helpful feedback, as well as their recognition of the significance and originality of our work and the value of sharing our distinctive dataset. Below, we respond to the reviewer’s comments.\n\n**A: Model weights and training code release**\n\nWe are currently reviewing options for releasing our user model in addition to the already open-sourced complete datasets.\n\n**B: “...proposed agent is trained with offline RL (as opposed to online) is worth highlighting more clearly..”**\n\nThis is a good point. Offline RL was chosen because: 1) It allows (also) training directly on the rich human rater dataset, capturing authentic complexities. Online RL with real people is impractical, forcing reliance on simplified user models. 2) Offline RL is faster, avoiding slow environment interactions and inference with large user models/LLMs required by online RL. We will highlight this. \n\n**C: “...is the user choice model a different model?”, “the main text uses $R$ to denote user utility, but the same quantity seems to be called $s$ in the appendix”**\n\nIn the paper we assume equivalence assumption between choice and preference and consistency in utility. A trained score function $s_\\theta$ builds the utility function (Eq 3), scoring image slates (per prompt/user type). Applying softmax over utility scores (Eq 4) yields a user choice probability consistent with utility and the preference model.  A learned temperature adds flexibility while keeping the assumptions (scaling the entire vector of utility scores by a constant $\\tau_\\theta$ does not alter the ranking consistency). Thus, both models share $s_\\theta$ with the choice model derived from utility output. $R$ refers to the overall utility concept, while $s$ is the specific learned score function implementing it.\n\n**D: “The reported 99% confidence intervals in Figure 5 may be overly pessimistic”**\n\nConfidence intervals don't overlap vs. the base model, but some overlap exists when comparing our method with different data types. We'll run another rater study assessing full trajectory improvement (final vs. first turn), expecting clearer results as cumulative improvement is what we ultimately care about. Updates during discussion.\n\n**E: “How can accuracy on Pick-a-Pic and Spearman correlation on HPS be computed if the user type is considered”**\n\nThe test sets for Pick-a-Pic and HPS are designed such that each sample contains a specific annotator's samples. To assess the posterior of the annotator over user types, we sampled a subset (of size 3) from the full set of annotated samples, then calculated the accuracy or rank of the remaining held-out samples and averaged the results over the posterior. We found this comparable to using the full set for the posterior computation. This detail was omitted due to space and it will be added in revision. \n\n**F. Questions For Authors:**\n1. See comment A. \n2. For human data creation, we used Gemini 1.5 Flash as the candidate generator. Instead of a trained selector, a random selector proposed the slate, picking up to one prompt per category from the generator. This combination provides quality expansions and broad exploration, crucial for building an unambiguous offline latent MDP dataset.\n3. Our user model consists of the utility component as well as the user choice model that translates this reward into user choices, as is standard in utility theory, choice modeling, econometrics, etc. Indeed, for a fixed choice model the user model is a generalized (user specific, set-based) preference model.  \n4. Our evaluation consists of 60 human raters, each experiment was evaluated over 200 trajectories. Exact metrics will be in the appendix.\n5. Agent trained with sparse rewards (only at final 5th round), determined by the learned user utility function. Human rater data was also labeled using this same learned utility function post-collection.\n6. Eq 2 is the generalized preference probability of preferring image set $\\ell^*$ over the other sets. This is not an equality but a description of the preference probability. \n7. Since the user model's goal is mimicking users for data generation, our metrics focus on performance across user types (averaged over posterior). However, we will add original baseline scores from the Pick-a-Pic/HPS papers for comparison.\n8. The y-axis represents the accuracy outlined in the subtitles, while the x-axis indicates the number of user types used in the user model. The shaded area is 95% CI. We will clarify this in the paper. Appendix D.3 details cross-turn accuracy: preference accuracy between selected images at consecutive test set turns.\n9. Softmax sampling: The Eq 3 aggregation function can be max, average, softmax sampling, etc. Softmax sampling means selecting an element randomly based on the softmax distribution over scores. We chose it over max to add randomness reflecting user selection variability observed during data collection and sequential T2I.\n10. see comment C.", "claims_and_evidence": "The paper’s claims are generally well-supported, with a couple opportunities for improvement.\n1. The fact that the proposed agent is trained with offline RL (as opposed to online) is worth highlighting more clearly, as this is a big distinction.\n2. The paper claims that there are two separate models, one user _utility_ and one user _choice_ model (L239 ff.), but from what I can tell, there is only a single model being trained (the preference model). Besides the added temperature parameter, is the user choice model a different model?\n3. The reported 99% confidence intervals in Figure 5 may be overly pessimistic. As presented, all confidence intervals overlap, making the results not statistically significant, which, if it’s the case, should be discussed. A different statistical test may offer a higher significance level.\n4. Releasing the code and trained models in addition to the dataset would go a long way to bolster the paper’s claims, as it resolves any concerns about reproducibility stemming from somewhat limited details/clarity on the model architecture and training procedure.", "methods_and_evaluation_criteria": "The conducted experiments make sense for evaluating both the user model as well as the T2I agent, and the use of human raters for evaluation is a big strength.\n1. Since we have a model-based way to measure user preference, it would have been nice to see an improvement over multiple turns in that regard as well. E.g. what is the average satisfaction/improvement according to the user model in each round, and does it saturate or keep increasing?\n2. Given that there already exist methods for multi-turn text-to-image generation (e.g. von Rütte et al., 2023; Liu et al., 2024), some sort of comparison (qualitative or quantitative) would feel appropriate. What does the proposed method bring to the table that previous approaches didn’t?", "theoretical_claims": "As far as I can tell, there are no major theoretical claims.", "experimental_designs_or_analyses": "The conducted experiments seem sound besides what is already mentioned above.", "relation_to_broader_scientific_literature": "The paper introduces an adaptive prompt-refinement agent that is trained using offline RL, which to the best of my knowledge has not been done before and is a valuable addition to the literature of both prompt-refinement and personalization through iterative feedback. In terms of performance, it is not clear how the proposed method stacks up to existing approaches, and by extension, whether it is worth the fairly considerable increase in complexity.\n\nBesides the proposed model, the collection and release of iterative preference data is a valuable contribution to the field that may lay the foundation for future work.", "essential_references_not_discussed": "The background of preference-adaptive image generation is covered fairly well. Additionally, there exist methods for iterative prompt improvement based on human feedback [1] or based on automated scoring [2] that may be worth mentioning.\n- [1] Martins et al., 2023. https://cdv.dei.uc.pt/wp-content/uploads/publications-cdv/martins2023metaprompter.pdf\n- [2] Mañas et al., 2024. https://arxiv.org/abs/2403.17804", "other_strengths_and_weaknesses": "Strengths:\n- The paper tackles an important problem in text-to-image generation in trying to reduce the burden of prompt engineering, continuing a line of work that consists of incorporating user feedback to adapt the generation process.\n- The proposed approach to do this is novel and makes sense, leveraging value-based RL to train a user-adaptive prompting agent.\n- Releasing the training data will allow future work to build and improve on the results of this work.\n- Evaluating the trained model using human raters directly measures the downstream performance.\n\nWeaknesses:\n- Clarity of writing is the paper’s biggest weakness. This includes multiple separate aspects:\n  - Flow: It may be helpful to switch the order of sections to more accurately reflect the logical flow of the project, which is dataset creation -> user model training -> synthetic data generation -> offline RL training -> evaluation (consisting of user model evaluation -> agent evaluation). As is, the paper (to me) felt somewhat out-of-order, and it took me a couple of passes to wrap my head around it.\n  - The formalism introduced in Section 2.2 feels a bit contrived and unnecessarily sophisticated. This may obfuscate the actual problem formulations and confuse some readers, a clear and simple textual introduction may be more appropriate. For an MDP, we just need to know the state space, the action space, and the reward function. For the sake of Section 2.2, it may suffice to state this in natural language.\n  - Important details are deferred to the appendix (presumably due to lacking space caused by, IMO, unnecessary formalism), leading to obfuscation. The model architecture and training is a core contribution of the paper and should be covered properly in the main text. In fact, I would argue that all of the formalism introduced at the beginning of Sections 2 and 5 is fluff that should be replaced with the actual meat that is the design and training of the user model and prompting agent. This is what the paper is about, not whether the user actually does or does not choose the image that they prefer most (Section 5).\n  - The lack of clarity somewhat extends to the evaluation setup: How can accuracy on Pick-a-Pic and Spearman correlation on HPS be computed if the user type is considered given in the user model? How is the user type determined at inference time? Or, when evaluating PASTA, is the preferred prompt chosen by the human rater or by the user model?\n  - Minor questions regarding clarity are deferred to the “Questions For Authors” section.\n- Not releasing the code and trained models is closely behind as the paper’s second biggest weakness. As-is, I would not feel confident in being able to reproduce the results presented in the paper, due to lacking details and/or vague/confusing writing (just one example: the main text uses $R$ to denote user utility, but the same quantity seems to be called $s$ in the appendix).\n- The prompt generation/expansion module of the proposed agent is a significant factor that potentially has major impacts on performance and warrants dedicated evaluation/ablation, or at the very least discussion. For example, does the proposed way of sampling 5 prompts from 5 different categories actually help? And if so, by how much? Other things like the “system prompt” (which is chosen arbitrarily), model, output format, etc. can also plausibly affect the performance of the final agent in a significant way.\n- Impact statement: The field of T2I modeling has a potentially big impact on the economics of visual content creation. Agentic content recommendation also has potential risks associated with it. Providing some perspective on this feels appropriate.\n\nGiven the relevance of the tackled problem and the strength of the contributions, I am inclined to recommend an accepting decision. However, considering the clarity of the writeup and the mentioned concerns regarding reproducibility, I cannot give a strong recommendation.", "other_comments_or_suggestions": "Nits:\n- Line 31, Column 2, also other occurrences: Gemini report is referenced as “Team et al.”, should be “Gemini Team”.\n- Line 45, Column 2: missing period after “LMM”\n- Section 2.1, RL: The value function, strictly speaking, measures the expected cumulative reward of a given state, not state and action (this would be the state-action value function, or Q function).\n- Footnote 4 is of little value without further details about the contractors (e.g. country/method of employment country, compensation)\n- Eq. 3: Why introduce a generic aggregator if we are anyways only going to use softmax?", "questions_for_authors": "1. Will the model weights and training code be released?\n2. How is the human-rated sequential data generated? Specifically, how are the prompt slates generated and, precisely, how is the slate selected from candidate prompts? It feels like this would already require an agent, which, presumably, we don’t have yet.\n3. In what sense is the user model _not_ simply a reward/preference model? This term is much more common in the literature and very easy to understand, so if appropriate, it should be preferred.\n4. What is the setup and sample size for human evaluation of the final agent?\n5. L302, Col. 2: “Rewards provided after the final round” – what are the rewards? Based on a model or based on human raters?\n6. Assumption 5.2: What does it mean for Equation 2 to be satisfied? Equation 2 seems to be an expression, not an equality.\n7. What are baseline performances on Pick-a-Pic and HPS? Without even the scores from the original papers, the presented numbers are rather meaningless.\n8. Figure 3: What is the shaded area? X- and y-labels are missing. Unclear what “cross-turn accuracy” is.\n9. L304, Col. 2: What does “softmax sampling” mean? Do we sample a random index based on the softmax distribution? Or do we compute a weighted average over the scores, with the weights given by the softmax distribution?\n10. L306, Col. 1: How does the temperature parameterization “ensure” that Assumption 5.2 is satisfied? Did we not make this assumption beforehand in order for this parameterization to be valid?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel reinforcement learning framework named PASTA for interactive text-to-image generation. The final recommendations are 1 "Weak reject", 2 "Weak accept",  and 1 "Accept". The reviewers generally agree that
- the problem is important, 
- the formulation is novel,
- claims are well-supported by experiments,
- evaluation criteria are appropriate, and
- dataset contribution is valuable.

Their major concerns include 
- clarity of writing, and
- generalizability.

The authors povided additional results after the rebuttal to address the concern on generalizability. 

I recommend "Accept" based on the importance of the problem and the novelty of the formulation.
Decision: Accept (poster)