SPRO: Improving Image Generation via Self-Play
Authors: Ritika Jha
Aanisha Bhattacharyya
Yaman Kumar Singla
Rajiv Ratn Shah
Changyou Chen
Balaji Krishnamurthy
Venue: NeurIPS 2025 poster ()
Keywords: diffusion model
self play
llm
large language models
prompt improvement
Primary area: deep_learning
TL;DR: We introduce SPRO (Self-Play Reward Optimization), an annotation-free framework that aligns images with human preferences by using vision-language models and reward signals to optimize prompts and images via self-play.

Abstract: Recent advances in diffusion models have dramatically improved image fidelity and diversity. However, aligning these models with nuanced human preferences -such as aesthetics, engagement, and subjective appeal remains a key challenge due to the scarcity of large-scale human annotations. Collecting such data is both expensive and limited in diversity. To address this, we leverage the reasoning capabilities of vision-language models (VLMs) and propose Self-Play Reward Optimization (SPRO), a scalable, annotation-free training framework based on multimodal self-play. SPRO learns to jointly align prompt and image generation with human preferences by iteratively generating, evaluating, and learning to refine outputs using synthetic reward signals such as aesthetics and human engagement. This self-improving feedback loop eliminates the need for external supervision. SPRO comprises three stages: (1) SPRO-Prompt, which trains a Guider-VLM via self-play to generate diverse, high-reward prompts targeting objectives such as PickScore (user preference), LAION-Aesthetics, and EngageNet (engagement); (2) SPRO-Image, which fine-tunes the diffusion model on high-reward images derived from these prompts; and (3) SPRO-Multimodal (SPRO-MM), which integrates both components for full end-to-end alignment. Without relying on human-labeled data, SPRO achieves an average 30\% improvement across preference objectives. Moreover, its generated prompts generalize across both open- and closed-source diffusion models. Through iterative self-play, SPRO discovers prompting strategies rarely authored by humans such as emphasizing visual harmony for aesthetics or leveraging shadow-based cues for engagement. SPRO offers a scalable path toward aligning generative models with complex subjective human values.

SPRO: Improving Image Generation via Self-Play
A
BII
T III
M
Ritika Jha"
Aanisha Bhattacharyya
Yaman K Singla"
I
Tafa
Rajiy Ratn Shah
Changyou Chen
Balaji Krishnamurthy
Adobe Media and Data Science Research,
HITDD
SUNY at Buffalo
Abstract
Recent advances in diffusion models have dramatically improved image fidelity and diversity. However, aligning these models with nuanced human preferences -such as aesthetics, engagement, and subjective appeal remains a key challenge due to the scarcity of large-scale human annotations. Collecting such data is both expensive and limited in diversity. To address this, we leverage the reasoning capabilities of laiiuuee models (VLMs) and propose Self-Play Reward Optimization (SPRO), scalable, annotation-free training framework based on multimodal self-play. SPRO learns to jointly align prompt and image generation with human preferences by iteratively generating. evaluating, and learning refine outputs using synthetic reward signals such as aesthetics and human engagement. This self-improving feedback loop eliminates the need for external supervision. SPRO comprises three stages: (1) SPRO-Prompt, which trains Guider-VLM via self-play to generate diverse, high-reward prompts targeting objectives such as PickScore (user preference).
and EngageNet (engagement); (2) SPRO-Image, which fine-tunes the diffusion model on nigh-reward images derived from these prompts; and (3) SPRO-Multimoda (SPRO-MM), which integrates both components for full end-to-end alignment. Without relying on numan-labelle data, SPRO achieves average 30% improvement across preference objectives. Moreover, its generated prompts generalize across both open- and closed-source diffusion models. Through iterative self-play, SPRO discovers prompting strategies rarely authored by humans such as emphasizing visual harmony for aesthetics or leveraging shadow-based cues for engagement. SPRO offers scalable path toward aligning generative models with complex subjective human values.
Introduction
Recent advances in diffusion models have transformed image generation, enabling the creation of highly realistic and diverse visuals 7 State-of-the-ar systems such as DALL-E [20), Stable Diffusion (21) and SDXL [17 have demonstrated remarkable capabilities across tasks including text-to-image synthesis, style transfer, and image inpainting. Despite these advances, diffusion models remain fundamentally limited in their ability to align with nuanced human preferences- as aesthetic appeal, engagement, and subjective taste. These dimensions of alignment are inherently abstract, context-dependent, and difficult to quantify, making them challenging to capture through conventional supervised training paradigms.
number of approaches have been explored to enhance alignment between generated images and human preferences, broadly falling into two categories: prompt optimization and model optimization.
Equal Contribution. Contact recharoriinhheeeiiddggooogggoooo
for questions and suggestions.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Objective: Aesthetics
Objective: User Preference
Objective: Eigage ment
sothal stedim, gren
Deogle dog
king on
eagle
rands sky.
LAKON Scarc 6.71
LA Scare
Pul-Sorr::
Pra-saan:: 23.6
Suntes
Huge wide open plae house rany couches
Three domnbaandees
Max ca drving towrs
heined look fart
urunged in
LAION Scom:
LAJON Boans 3.30
Picksconc 20.2
Picedcore:
EDIG Scon: EE
woraa with
blond pemplat
Versie
Raid jurpla
riding while
rebetique. Irstecarris
horsa insate fiece with a horned cons
metalliqun on color
ortired
LAION Scora: 6:66
LAION Scone 7.50
Ploddcore:
P1aSScON:
E010 83.91
2010 90.30
Figure 1: Visual comparison between images generated from original captions (left) and those optimized using our SPRO (Self-Play Reward Optimization) framework (right). SPRO generates images that are more aligned with diverse human preferences aesthetic appeal, user preference, and engagement, without relying on any human-annotated data. The effectiveness of SPRO is measured using specialized scorers for each objective LAION Score [22] for aesthetics, PickScore [ for user-preference, and EngageNet (EOIGScore)
[9] for engagement.
Prompt optimization methods steer generation by refining textual inputs while keeping the diffusion model frozen 15] 29). In contrast, model optimization approaches directly fine-tune diffusion weights using preference-based feedback while holding other components constant I 27). Both directions, however, depend heavily on manually curated datasets containing prompts and their preferred or human-improved variants. This reliance on human-aute authored data imposes fundamental limitations: designing prompts that consistently yield high-quality outputs is difficult due to the nonlinear and often unintuitive relationship between linguistic phrasing and visual realization. For instance, subtle textual changes, such as adding terms, like "highly detailed," "ultra realistic," or invok- ing specific artists' styles, can dramatically alter visual quality in unpredictable ways. Consequently, models trained on curated datasets tend to replicate only human-discoverable stralegies, constraining their creative potential. Morcover, large-scale human curation is costly and yields datasets with limited scope and diversity, further restricting progress toward robust preference alignment.
promising direction for reducing reliance on high-quality synthetic data that capture diverse dimensions of human preference. One mechanisn sm that enables this is self-play, a reinforcement learning paradigm which model iteratively generates, evaluates, and learns from its own outputs to improve over time. This paradigm removes the need for explicit human supervision by establishing structured feedback loops through which model refines its behavior. Originally introduced in TD-Gammon (26). where neural network learned to play backgammon by competing against itself, self-play later achieved widespread prominence through AlphaGo and AlphaZero 24 25), which reached expert-leved performance games like Go and Chess without relying on human gameplay. More recently, the concept has been extended to large language models (LLMs) for complex reasoning tasks such mathematics and programming. For example, DeepSeek-Re RI 4 employs a reinforcement learning framework in which the model synthesizes its own examples across multiple iterations, boosting pass @1 accuracy the AIME 2024 benchmark from 15.6% to 71.0%. Similarly, SPIN [ demonstrates that self-play can enhance instruction followwig by generating tasks, solving them, scoring outputs, and fine- munittunng on high-quality completions, yielding over a 10% improvement on benchmarks such as GSM8k and TruthfulQA.
data is the autonomous generation of
In the image domain, self-play has also been explored through SPIN Diffusion diffusion models by having them compete against carlier versions of themselves in an iterative general-sum minimax game. This setup allows the model to progressively improve without requiring human- aumaan-annotttd preference labels. SPIN Diffusion is evaluated on two human preference objectives. For user preference, it achieves score of 22.00 and for aesthetic appeal, it reaches score of 6. 6.24, after three iterations, showing clear improvement over the base SDXL model which scores 20.99 and 5.67 respectively.
32 which finetunes
common theme across recent advances both text and image domains is the use of self-play to iteratively refine models through synthetic feedback, thereby reducing reliance on human-annotated data. Despite its success, most existing approaches remain confined to single modality and lack mechanisms for discovering strategies that generalize across architectures preference objectives.

---

In language models, self-play has enabled scalable exploration of reasoning strategies and shown strong gains on tasks such as mathematical problem solving and code generation. However, these applications are largely restricted to well-defined, iimulator-friendfy tasks with unambiguous reward structures, limiting their broader applicability to open-ended generative domains. In the visual domain, approaches such as SPIN Diffusion [32] demonstrate that diffusion models can learn from their own checkpoints, but remain constrained to image space, exhibit poor cross-mode generalization, and require large sample budgets (over 500k images) to achieve modest aesthetic gains. These observations highlight the need for a unified framework that combines the strengths of prompt-space exploration and image-space optimization within multimodal self-play paradigm that is capable of aligning both language and vision models toward complex, human-centedd objectives.
We introduce Self-Play Reward Optimization (SPRO), a unified framework that aligns diffusion models with human preferences through multimodal self-play. At its core, SPRO employs vi- sion-language model (VLM), referred to as the Guider- VI.M, to autonomously explore and generate optimized prompts via iterative self-play. These optimized prompts are then used to fine-tune a diffusion model, producing high-quality prompt-image pairs aligned with diverse human preference objectives-all without human- auman-annotated data. By coupling language- reasoning with visual generation, SPRO enables coordinated improvement across modalities and promotes the discovery of strategies that extend beyond human- -authred prompting patterns.
SPRO operates in three complementary stages. prompt-space, image-space, and joint-space opti- mization. In SPRO-Prompt, the Guider-VLM, takes an image- pair as input and generates diverse, prasoning-augmente prompt diversity [T. Each prompt is passed frozen diffusion model, which generates corre- sponding images that are scored by reward models such as PickScore [1]. LAION-Aesthetio [22]. or EngageNet [9]. High- and low-reward samples form contrastive datasets used to fine-tune the Guider-VLM via Direct Preference Optimization (DPO) [1)] enabling it to iteratively learn more effective reasoning and prompting strategies a vast search space.
prompts. Including explicit reasoning chains improves exploration and
In SPRO-Image, the diffusion model is tnnett using synthetic. high- reward images generated through SPRO-Prompt and their corresponding base captions. This stage improves preference alignment directly in image space without relying on additional prompt tuning.
Finally, SPRO-Multimodal (SPRO-MM) integrates both components: the Guider-VI.. continues optimizing prompts while the fine-tuned diffusion model generates images. Unlike SPRO-Prompt (which uses a frozen generator) SPRO-Image (which fixes prompts), SPRO-MM leverages the full capacity of both models, achieving scalable, annotation-free alignment across modalities.
We evaluate SPRO across three key human preference objectives: aesthetic appeal, engagement, and user preference. On SDXL, SPRO achieves win rates of 99.42% for aesthetic appeal (LAION- Aestheties), 71.60% for engagement (EngageNet). and 65.70% for user preference (PickScore). Unlike prior methods such as SPIN Diffusion 32). which optimize only the diffusion model. SPRO in- troduces gassonng-auugmente self-play via the Guider- VLM, yiclding a 0.42-point gain in PickScore and a 1. 1.97-point improvement in aesthetic appeal while using less data. This demonstrates that agent-driven self-play enables more efficient exploration of prompt and image space.
On average, SPRO outperforms both model- and prompt-based baselines, including CAPO, DDPO, and Promptist (Tables 2 and 3 The trained Guider- VLM also produces diffusion-agnostic prompts that generalize effectively across both open- and closed-source models, achieving win rates of 78.99 9% on SDXL [17. 80.9% on Flux (2), and 67.67% 67% on DALL-E [16), all without additional tuning. In human evaluations, images generated using SPRO prompts were preferred over those from GPT-4o in 66.7% of cases and over base captions in 78.9% (Table 22). confirming stronger perceptual alignment with humaan preferences.
We make the following key contributions:
Multimodal, Multi-Model Self-Play: To the best of our knowledge, this is the first work to apply a self-play paradigm across multiple models and modalities for preference alignment. SPRO enables coordinated optimization between a vision-language model and a diffusion model, demonstrating the potential of self-play beyond single-domain applications. State-of-the-Art Alignment Across Three Human Preferences: Through extensive experiments, we show that SPRO achieves state-of-the-art performance on three human preference objectives:
3

---

aesthetics, engagement, and user preference. SPRO surpasses all baselines by an average of 30% across these objectives, without relying on any human- annotated data.
Intelligent Exploration via the Guider- VLM: The Guider-VI.M drives intelligent exploration in prompt space, automatically discovering novel and often counterintuitive prompting strategies that improve alignment. This results in broader and more efficient optimization compared to traditional human curated data-driven approaches.
Reasoning Improves Generation Quality: We demonstrate that incorporating reasoning chains during prompt generation yields more optimal outputs than scaling compute alone achieving an average 2% improvement across all preference objectives (Table
Diflusion-Model Agnostic Framework: SPRO operates independently of the underlying diffusion architecture, enabling seamless integration with any pretrained model. It achieves 73.59% average win rate when applied to other open- and closed-source generators.
Large-Scale Synthetic Preference Dataset: We release synthetic dataset of image-prompl pairs aligned with human preferences, generated entirely through self-play using SPRO. This dataset provides scalable, annotation-free resource for future research in preference alignment. The dataset is available here
million
2 Method
The goal of Multi-Model, Multi-Modal Self-Play Reward Optimization (SPRO) framework is to optimize images for various human preference objectives, such as aesthetic appeal, engagement, or user preference. Given user-provided base image, caption and target objective, the framework produces more refined output images while preserving the original intent. Our method operates through three progressive stages, each targeting a distinct axis of optimization: prompt-space opti- mization, image-space optimization, and joint optimization. Next, we cover the formal definition of the optimization problem and describe SPRO framework.
2.1 Problem Formulation
To operationalize our framework, we formalize SPRO as an optimization problem over prompt and image generation spaces. Each stage corresponds distinct optimization objective with respect to the model components and the reward function.
Let D3 denote diffusion model with parameters 0, p a prompt generated by Guider VLM. and R. a reward function encoding human preferences as aesthetic appeal, engagement or user preferences. We define the optimization objectives as follows:
SPRO-Prompt:
We optimize prompts while keeping the diffusion model parameters 0 fixed, allowing the system to discover effective conditioning strategies.
max ((Ro(Do(p))]
SPRO-Image: We fine-tune the diffusion model parameters 0 using high-quality synthetic images generated from optimized prompts p' obtained in the previous stage. During this process, the prompts p are kept fixed (to the original prompts), allowing the model to better align its generations with the target objective O in the image space using self-play data.
max ((Ro(D)(P))]
We jointly leverage both the optimized prompts p' and the diffusion model weights allowing for co-evolution and deeper alignment:
SPRO-MM:
max E(Ro(D((')))
p'.0
2.2 SPRO-Prompt
Self-play refers to the process progressively improving a model by learning from the outputs of its own previous iterations. In the prompt-space optimization stage, our Self-Play Reward Optimization (SPRO-Prompt) framework leverages this paradigm by training a Guider- VLM through self-play to generate optimal prompts for given human preference objective. The Guider-Vii VLM receives

---

SPRO-prompt
SPRO-image
Maran Phalesrce Oipective
Phongt
Image
Visio amggang Mode <
Reeeoting chaino Promgra
Dillction Modil A
Images
Roward indro
Reward Beolss
tentice u
6 P9
E
Big
29
Fittring
(0,9 )
D
Ro
(0,p', [O.p), 1... 2...
D
(2
buts poncrutes previcus
10.
DPO Fioetriing
[+++]
Figure Our SPRO framework consists of three stages: SPRO-prompt (Left): It iteratively finetunes VLM L' using DPO to generate reasoning chains rj and prompts conditiion on human preferences O, base prompts p", and images 1 for 1st iteration and in further iterations it uses prompts and images generated in previous iteration. These prompts guide a diffusion model D' to reward function Ro, resulting in scores s². High-reward
produce images rj. which are scored samples are filtered to form synthetic data. The SPRO-image (right) fine-tunes the diffusion model D' for objective O, using base prompts p", and corresponding winner image I to directly align image generation with human preferences.
image, its base caption, and a target objective as input, and produces both reasoning chains and optimized prompts as output.
SPRO-Prompt operates in two phases. (1) Synthetic data generation: the Guider- VLM produces diverse reasoning chains and prompt candidates, sampled at varying temperatures to encourage exploration. Each prompt passed a frozen diffusion model generate images, which are then evaluated by trained reward model. Samples with rewards below the mean of the distribution are discarded, and CLIPP similarity [18] used sanity check to ensure that generated images remain semantically consistent with the base image (see Table 9). (2) Direct Preference Optimization (DPO) [9]: : the Guider- VLM is fine-tuned using contrastive feedback derived from the synthetic data, enabling it to iteratively learn more effective reasoning patterns and prompt strategies tailored to each objective. The overall SPRO-Prompt pipeline is illustrated in Figure 2 and is applied across multiple human preference objectives, including aesthetic appeal, engagement, and user preference.
2.2.1 Synthetic Data Generation
For given human preference objective O, we begin with a image- base caption pairs. In the first iteration (i.e., i = 1). let these base caption pairs be denoted by pe and I belonging to the dataset X
1. A VLM L' takes the image-captions pair (p",1") as input, along with the details of the target human preference O as a prompt, and generates set of n candidate reasoning chains and corresponding improvised prompts for the diffusion model, denoted as (r'. PJ) for
j E (O..... n), sampled at different temperatures to ensure diversity.
The diffusion model D now generates images using the candidate prompts:
I -Dw)) Vj (0,
(4)
3. Each image is evaluated using reward function R, for that objective:
= Ro(T)) Vj E n0.
(5)
2.2.2 Iterative DPO Refinement
For given objective O, iteration i, we generate a set of prompts PI for j E (0...., the Guider VLM L', each of which is used to generate an image via a frozen diffusion model. These images are then scored using trained reward model, and the corresponding reward scores are used to rank the prompts.
n). from
In each iteration i, we select the highest- and lowest-reward prompts, (pi.: Pf). based on their associated image scores, and use them as winner loser pairs for training. Samples are discarded if the reward improvement is below the mean of the generated distribution. To ensure semantic consistency, we also filter out prompt-image pairs with CLIP similarity below 0.75, ensuring comparisons focus on alignment with the objective while preserving original content. This filtered dataset, after applying the threshold condition, becomes the new dataset X'.
5

---

Crasy Trog, on the wheel mococcyck, dead
Persefi 522 Tu tlazss donr Die TIREE repturing spaeti preceion
Ayyuing gittin
toxands them oneii beach
Teature Apply
Temure: Apply sharpe first, kauping the gr and testrr intact. Colors, adjus palette inte inte prescrving orighnalily hase
tonur smooth ng and nolse reduction for clean riace. Colors increase cortrass and brigheness for inmediate visual mosel
tone seumeet and gaidee sesthetios. racing beamingsmlen
Scene Eir phosaae ciurity detal.
Tonme Fraire soem
Tones: Add her ween and cool tones.
and for
5010 Soore: 5220
realisicc
Emosion: Highight foor iosoness and playtuness.
-0.40
0.44
+30.00
ctmesaherc.
Inmertae
dramatic feel.
Picticore
DO G Score $1.70
Score
Score:
LAION
Figure 3: For each base caption and image (top), -PRO-Prompt generates multiple reasoning chains and refines the prompt for the diffusion model. Red arrows indicate lower reward generations; blue arrows denote higher reward generations. SPRO discovers novel reasoning strategies targeting aspects like texture, color, etc- enabling reward-driven image refinement through self-play. More samples of strategies in Table 21
3.
We fine-tune the Guider VLM L' using Direct Preference Optimization on the preference pairs in X obtain the updated model L
LLi Lier) = - BR) Ro) llgg - [Rw)]
(6)
L' ((),
where Ri = log [rer((ri, pt) C,
R., = log Tnee((ru w. Pw: T,C,0)
(7)
Lyer is the reference model pure-finettunig), and / is hyperparameter.
We optimize the Guider VLM iteratively, using the fine-tuned model from iteration i- to generate preference data for training iteration This process, forms positive feedback loop that gradually shifts the model's 's output distribution toward higher rewards in the preference space. Across different objectives, we observe consistent improvements in reward metries over successive iterations. with performance typically saturating after 3-4 rounds. This iterative refinement enables the Guider VLM to discover novel prompting strategies aligned with the target objective that are often beyond human intuition and are not constrained by any fixed dataset, producing preferred prompts that yield improved images while preserving user intent.
2.3 SPRO-Image
For image-space optimization, we extend the exploration conducted in prompt-space optimization via SPRO- SPRO-prompt. Given a target objective O, we fine-tune the diffusion model using only high-reward (winner) synthetic images generated through the iterative self-play process. This approach directly aligns the diffusion model with human preference objectives, allowing to produce optimized images even from simple base prompts. As illustrated in Figure 2 our pipeline leverages SPRO- SPRO-prompts obtained through self-play to drive image-level fine-tuning.
leverage the SPRO-Prompt to get best image 7 for entire training data in each iteration. 2. These high-reward images are paired with their corresponding base captions {p", T to form synthetic training dataset Si uain for each iteration
3. We fully finetune the diffusion model 0 to minimize the reconstruction loss between images generated by base captions and the neward-optiminn targets for every iteration i:
(0) Egp" .1. Stvain [1]P>(p") [=]
(8)
2.4 SPRO-MM
In the joint optimization stage. SPRO-MM, we combine the strengths of optimized prompts and a finetuned diffusion model. Unlike SPRO-Prompt, where the diffusion model remains frozen and only the Guider VLM is trained to produce optimized prompts, or SPRO-Image, where the diffusion model finetuned using reward-aligned synthetic high quality images generated while keeping prompts fixed, SPRO-MM uses both the Guider VL.M to generate optimized prompts for given objective O, and these prompts are then input into the diffusion model that has already been finetuned for the same objective, using self- self-play data.

---

3 Experiments
We conducted experiments to evaluate the effectiveness of our self-play framework, SPRO, optimize images for various human preference objectives. aata-generation and training details are provided in Appendix A.4
Reward Metrics We focus on three human preference objectives: aesthetic appeal, user preference, and engagement, each representing a distinct dimension of evaluation. These objectives are quantified using learned reward models. For user preference, we use PickScore [. a CLIP-based model trained on large-scale preference data. For engagement, trained on Twitter data to predict the social media engagement an image is likely to receive. For aesthetic appeal, we employ the LAION aesthetic scorer [22], trained on numan-annoter
adopt EngageNet [9] foundation model
aesthetic ratings. To assess generalization, we also evaluate on unseen reward models, including VILA [8] and the Improved Aesthetic Scorer
23 for aesthetics, well as ImageReward
(28 for user preference.
Evaluation We evaluate our proposed framework on distinct test sets tailored to each human pref- erence objective. For aesthetic appeal, we use held-out set of 514 images from the Flickr30k dataset. We evaluate on the PartiPrompts [31] and Pick-a-pic test split. For engagement, we adopt the same test set from ingaginggmmageNe dataset to ensure consistency with prior benchmarks. For user preference, we evaluate on the Pick-a-Pis test set, as well as the PartiPrompts dataset to capture diverse user preferences.
4 Results
We evaluate SPRO against state-of-the-ee methods in two categories: prompt-only approaches, which optimize text prompts without altering the diffusion model, and diffusion-only approaches, which directly tune the image generator using preference rewards annotated data. Prompt-only baselines include Promptist [6] and PAE [I5]. both reliant on real datasets and supervision. Diffusion-on -nnly baselines include DDPO M. SPIN Diffusion [32] and EngageNet [9, which is trained on Twitter engagement-llbelled images. We also compare with general-purpose LLMs as prompters like GPT-4o and ILAMA-1 IB using COT.
We benchmark all three stages of our framework, SPRO-Prompt, SPRO- and SPRO-MM. on aesthetic appeal, user preference, and engagement. For aesthetics (Table 2). SPRO-Prompl achieves LAION score of 8.60, outperforming Promptist (7.23), PAE (6.27), GPT-o3 (7.67) and GPT-4o (7.57) and showcases a 34% gain over base captions. For user preference (Table 1. SPRO-Prompt attains a PickScore of 22.42. surpassing both prompt-only GPT-4o (22.31). GPT-o3 (19.47) and diffusion-optimizze models (SPIN: 22.00, DDPO: 21.59), demonstrating that synthetic slay in prompt-space can rival data-intensive methods. On engagement (Table 3). SPRO-Prompt scores 83.48, 88.36, and 90.24 across low, medium, and high engagement buckets more than doubling EOIG's score in the low bucket (38.76 76 to 83.48), showing SPRO's ability to amplify weak content. Further ablations and experiments are provided in Appendix A.2 Qualitative samples are shown in A.6
Synthetic data alleviates the need for large-scale annotation. Unlike prior optimization methods such as DDPO [. which relies on explicit pairwise preference annotations and achieves a PickScore of 21.59, our method reaches higher mean score of 22.42. Similarly, Promptist, which depends on manually curated prompts, attains a LAION aesthetic score of 7.23, while our method achieves 8.60. We also find that diffusion models finetuned on self-play data outperform those trained on real data. For example, on the engagement objective, SPRO-Image delivers an average improvement of 42% over EOIG-PFT which was trained on images sourced from real tweets. This underscores SPRO's scalability and strength in aligning with human preferences using only synthetic data without costly manual annotation.
Guider VLM leads to more efficient exploration and broader optimization. Promp,, which uses self-play in prompt-space, achieves higher aesthetic (8.02 vs. 6.05) and user preference scores (22.84 vs. 22.31) than SPIN Diffusion [32], while using only 30k training examples compared to SPIN Diffusion's 's 500k. This underscores that self- in prompt space is substantially more sample- efficient than self-play image space. The guider VLM enables richer and targeted exploration in strategies shown in Figure 3 suggesting that intelligent reasoning, not scale alone, is key to optimizing human preference objectives. Moreover, when this synthetic data used to finetune a diffusion model directly, it also outperforms self- pelf-play in diffusion space.
7

---

Prompt Model
Approach
PickScore
Base captions
20.99
Prompt Model
Approach
LAION
LLAMA 11B
Prompt
21.96
GPT4o
Prompt
22.31
Base captions
6.40
GPT-o3
Prompt
19.47
Prompt Auto Edi[[]
Prompt
Qwen-3
Prompt
22.10
Promptist[4]
Prompt
7.23
DDPO[2]
Image
21.59
LLAMA 1IB
Prompt
7.38
SPIN [3f
Image
22.00
GPT4o
Prompt
7.57
Prompt
22.42
GPT-o3
Prompt
7.67
SPRO SPRO-image(Ours)
Image
22.09
Qwen-3
Prompt
7.55
(PRO-MM(OUr)
Both
22.42
SPRO-prompt((Ors)
Prompt
8.60
SPRO-image( (Our))
Image
7.48
Base captions
21.89
SPRO-MM(Ouns)
Both
8.42
DDPO[2]
Image
22.27
SPIN [if
Image
22.31
Base captions
5.67
CAPO[I]
Image
22.83
DDPO[2]
Image
5.77
SPRO-prompt(Ouus)
Prompt
22.84
Spin [if
Image
6.05
SPRO-image(Ours)
Image
22.55
prompt(
Prompt
8.02
SPRO-MM(OunS)
Both
22.77
SPRO-image (Ours)
Image
7.01
SPRO-MM(OUn)
Both
8.02
Table I: The top block of the table reports re- sults on the Pick-A-Pic (PAP) dataset[[]], while the bottom block presents results the Par- tiPrompts [31]. The SPRO-prompt achieves a higher PickScore
Table 2: The top block of the table reports results
on the Flickr dataset [30 while the bottom block presents results on the PartiPrompts [31] The 110--PromttSPRROOO outperform all prompt-space baselines, Promp- tist (6) and PAE [55 on aesthetic alignment mea- sured using LAION score [2 both datasets.
and SPRO-MM
the test set of the Pick-A- Pic dataset as well as PartiPrompts than existing diffusion tuning baselines such as CAPO [12), DDPO [ and SPIN Diffusion [3), indicating stronger alignment with user preference.
Prompt optimization excels by discovering effective strategies. comparison across stages of our framework shows that SPRO-Prompt, which performs self-play in prompt space, consistently uncovers novel strategies that improve outcomes. As shown in Figure the Guider-VLMM learns these strategies iteratively through reasoning chains. Table highlights how strategies vary by objective: user-preference prompts benefit from technical and systematic phrasing, while aesthetic prompts perform better with artistic phrasing. For engagement, prompts emphasizing motion blur outperform those using motion lines. These insights underscore the value of strategy exploration in prompt space. SPRO-Prompt outperforms SPRO-Image across all objectives, showing that such strategies are harder to teach through diffusion model finetuning alone. SPRO-MM, which combines both, yields further gains, demonstrating the complementary benefits of prompt and image-space optimization.
SPRO-Prompt generalizes across diffusion models. Prompts optimized by the guider VLM through self-play are diffusion-agnostic.. Though trained using SDXL-base the frozen diffusion model, these prompts transfer effectively to both open and closed-source models in zero-shot. Applying the same test-set prompts Flux [2] and DALL-E 3 [16], we observe consistent improvements over base prompts across all objectives. For aesthetic appeal (LAION score (22)). Flux improves from 6.77 7.86 and DALL-E 3 from 6.79 7.98. On engagement, using Flux, we observe 21.5% average gain across all buckets. While DALL-E already performs well in low-engagement scenarios, SPRO-Prompt boosts its quality medium and high buckets. For user preference (PickScore [T). SPRO- SPRO-Prompt improves Flux from 21.99 to 22.20. Results in Tonfirr that SPRO- SPRO-Prompt produces broadly transferable, high-quality prompts.
Zero-shot generalization across unseen reward models. SPRO demonstrates robust zero-shot transferability across reward signals, showing that learns general prompting strategies tied preference objective rather than overfitting to the specifics of any particular reward model. For aesthetic appeal, although trained with the LAION scorer [22], it also improves performance on unseen metrics, raising scores from 7.38 to 8.60 on LAION, 60.74 to 66.16 on VILA [8), and 6.30 7.47 on the Improved Aesthetic Scorer. For user preference, trained with PickScore (I]) SPRO-Promm not only improves PickScore itself (20.99 to 22.42) but also achieves higher scores on ImageR (28 (0.65 to 1.02) and CLIP similarity [88] (87.76 89.32). Results in Tables [3] and [4 confirm that SPRO-Promm Prompt produces generalizable, high-quality prompts aligned with human preferences. Beyond trained reward models, we also evaluate SPRO in a setting guided only by human aesthetic ratings from the AVA dataset, where SPRO-Promp improves average scores from 5.90 to 6.80. This experiment, detailed in Appendix A.2.8. highlights SPRO's adaptability even when no external reward function is available.
8

---

Generalization across multiple datasets. SPRO demonstrates strong robustness when evaluated beyond the training distribution, consistently improving performance across diverse datasets and reward models. For the aesthetic appeal objective, as shown in Table T8 SPRO-Promp significantly improves scores on the PartiPrompts dataset across three independent evaluators: LAIÓN Score (6.16 to 8.01), VILA (55.15 15 to 64.81), and the Improved Aesthetic Scorer (5.60 to 6.78). Similarly, for the user preference objective (Table SPRO-Prompt achieves a notable gain in PickScore (21.89
to 22.84), reinforcing its ability to generalize preference-aligned strategies to new distributions of prompts and images. We further validate this finding on the LexicaDB dataset [33, following the RATTPO [10] setup. Since the original test set is unavailable, optimize them with our user peeeeerreefeeeeeueeeeeeee tundeeddeeeeeed SPRO-Pres Prompt.. As shown in Table 20) SPRO- Prommt achieves ImageReward score of 1.439, substantially higher than both the base prompt (0.1588) and the RATTPO baseline (1.132). These results demonstrate that SPRO generalizes effectively across datasets and reward models, reinforcing its robustness and applicability to diverse prompt-image distributions.
sample 160 random instances and
SPRO supports multi-objective optimization. Beyond single-objective training, SPRO can ac- commodate multiple reward signals, enabling alignment with complementary, orthogonal, or even conflicting preferences. In the first setup, we optimize for both aesthetics (LAION Score) and seman- tie fidelity (CLIP similarity). As shown in Table 15| PPRO-Prompt improves aesthetics from 6.40 to 7.90 while maintaining stable CLIP similarity (83.19 to 83,28), showing that it can enhance aesthetic quality without compromising semantic consistency. We then consider a more challenging case with orthogonal objectives: aesthetics (LAION Score) and user preference (PickScore), these two metrics represent distinct and potentially unaligned preferences. Table 16 shows that SPRO-Prompt improves both, raising LAION from 6.40 to 7.66 and PickScore from 22.69 to 22.96, demonstrating its ability to balance distinct and potentially unaligned rewards. To further test flexibility, we evaluate SPRO under conflicting objectives by designing composite pseudo-reward function defined as 2 x LAION Score - 0.5 x PickScore, which favors higher aesthetics while penalizing user preference. This setup simulates real-world scenarios where artistic choices may enhance visual appeal but reduce broader audience preference. Using this objective, SPRO-PrORm behaves as intended (Table [7). increasing LAION from 6.40 to 7.62 while reducing PickScore from 22.69 21.70. Our findings demonstrate that the SPRO framework effectively supports multi-reward optimization, including both orthogonal and conflicting objective settings.
SPRO is model-agnostic. SPRO consistently enhances performance across diverse VLM backbones, demonstrating its robustness independent of model choice. As shown in Table 12 reasoning-bard based backbones such as Qwen-3-7B achieve the largest improvements, rising from 7.55 to 8.46 in a single iteration. Non-reasoning models also benefit, with LLaVA-7B improving from 6.76 7.22 and LLaMA-3.2-1 IB improving from 7.38 to 8.30. These results demonstrate that SPRO generalizes across reasoning -agmmented and standard VLMs, with stronger backbones converging more efficiently but all models consistently gaining from self-play.
Human Study To assess alignment with human preferences beyond automatic rewards, we conducted a human evaluation comparing images generated from three prompts: base caption, GPT-4o-optimized prompt, and our SPRO-prompt. For each input, three images were generated using the same diffusion model (SDXL or Flux), and annotators selected their preferred image in a randomized, blind setting. As shown in Table 22, SPRO-prompt was preferred over GPT-4o in 66.7% of cases and over base captions in 78.99 78.9% The human study protocol is detailed in Appendix A.5
Related Works
Recent research on aligning text-to-image diffusion models with human preferences falls broadly into two complementary streams: model-space optimization and prompt-space optimization.
Diffusion Model Tuning, Early efforts directly fine-tune diffusion parameters using preference feedback. For instance, DPO-Diff [27) introduces text-guided gradients for efficient optimization within restricted prompt domain, while Diffusion-Di DPO (27 removes the explicit reward model and instead optimizes diffusion parameters via implicit rewards. DDPO [ and DPOK (5) extend this line of work by training reward models on human preference datasets and fine-tuning diffusion models through reinforcement learning. CaPO [2] further calibrates multi-objective rewards to balance competing alignment goals. Although these methods achieve improved alignment, each

---

Prompt Model
Approach
EOIG Score
Law
Mid
High
Base captions
38.76
51.69
56.14
Prompt Model
Diffusion Model
Pick
LAION
EOIG Score
LLAMA 11B
Prompt
83.19
87.72
89.32
Score
Score
Low
Mid
High
GPT40
Prompt
77.70
87.83 89,95
GITT-03
Prompt
75.91
75.97 79.51
Base captions
SDXI.
20.99
6.40
64.34
71.75
75.10
Qwen-3
Prompt
74.12
83.12 84.23
SPRO-promptioure
SDXL
22.42
8.60
83.48
88.36
90.24
ROIG-PFTTT,
Imuge
43.12
56.10 62,30
Base captions
Flux
21.99
6.77
64.53
71,80
70.80
EOIG-RLHF-ES101
Image
40.20
53.10
58.27
SPRO prompitours)
Flux
22.20
7.86
77.25
85.50
89.01
SOIG-RLHE-DSSE
Imuge
39.49
52.97 57.91
SPRO P0e00-rompii(uue
SPRO-image(Ours,
Prompt
Image
83.48
70.70
88.36 90.24
77.94 77.43
SPRO prompttsps)))))
Base captions
DALLE-3
DALLE
22.07
22.07
7.98
6.79
61.68
66.13
74.69
72.36
60.01
77.98
SPRO-MM(OUns)
Joint
83.20
87.39 89.38
Table 4: Prompts optimized using SPRO-prompl gen-
Table 3: SPRO SPRO-prompt has the highest FolG
eralize well other diffusion models as Flux [2] and DALLE-3 [16], in zero-shot, across all three objectives, highlighting the diffusion agnostic nature of our frame- work
Scores p which is human engagement on a scale of 00100) 100. We show results for the three buckets of data samples low, medium and high made based on the EoIG Scores of base images.
tuned model remains tied a specific backbone and objective, and their dependence on large-scale human feedback datasets limits scalability.
Prompt-Space Optimization. An alternative direction focuses modifying text prompts while keeping diffusion weights frozen. PromptCoT [29) introduces a Chain-of-Thought (Co'T) approach that aligns prompts with high-quality image descriptions and fine-tunes large language model (LLM) on curated text data. PROMPTIST [ adapts user-written prompts into model-preferred versions through supervised fine-tuning on engineered examples, followed by reinforcement learning to enhance visual appeal while preserving intent. Dynamic Prompt Optimization [55 employs online reinforcement learning dynamically re-weight insert tokens during generation. Despite their effectiveness, these systems rely on human-authored prompt-image pairs and therefore explore only strategies.
Across these two lines of research, prompt-basee methods overlook image-space fine-tuning, while diffusion-basel methods neglect textual conditioning. Our proposed Self-Play Reward Optimization (SPRO) framework unifies both directions: a Guider-VL. VL. autonomously discovers novel-ame often counterintuitive- promppppiiiieeepppiiiinggg strategies that yield high-reward images, which are then used to further fine-tune diffusion models. In doing so, SPRO bridges the gap between prompt-only and diffusion-only approaches, introducing a scalable, multimodal framework for preference alignment.
6 Conclusion
We introduce Self-Play Reward Optimization (SPRO), unified framework for objective-driven prompt optimization for image generation. SPRO leverages a Guider- VLM to engage in iterative self-play, generating and evaluating diverse reasoning chains and prompts conditioned on base image- caption pairs. By combining prompt and image space optimization in a multi-stage process, SPRO significantly improves alignment with multiple human preference objectives including aestheties, user preference, and engagements. Unlike prior approaches that rely on human-annotated datasets or handcrafted prompts, SPRO autonomously explores and discovers effective strategies without additional human supervision. Our experiments demonstrate consistent gains in each reward metric, validating SPRO's capability to generalize across objectives.
Limitations While promising, SPRO has several limitations. SPRO's effectiveness is bounded by the reliability of external reward models, which act as proxies for human preference but may not fully capture nuanced human preference. Moreover, these human preferences are addressed by SPRO independently. it does not yet support dynamic trade-off optimization in scenarios involving multiple goals (e.g., maximizing aesthetic quality while maintaining high engagement). Future work could explore reward balancing and preference-awar optimization to address these challenges.
References
[1] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. 2023.
10

---

[2] Black Forest Labs. Plux.I-deyy A 12b parameter rectified flow transformer for text-to-image generation. httpa://huggingtee
faw.aac--ooret--aaaaaaaaaaaa 1-dev 2024. Accessed: 2025-05-15.
[3] Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self Self-play fine-tuning con- vertsweak language models to strong language models. In Proceedings of the 41st International Conference Machine Learning, ICML'24 24. JMLR.org, 2024.
[4] DeepSeek-Al, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuashu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng. Chengda Lu. Chenggang Zhao, Chengqi Deng. Chenyu Zhang, Chong Ruan, Damai Dai. Deli Chen, Dongjie Ji, Ethang Li. Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen. Guowei Li. H. Zhang. Han Bao, Hanwei Xu, Haocheng Wang. Honghui Ding. Huajian Xin, Huazuo Cao, Hui Qu. Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingehang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang. Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang. Minghua Zhang, Minghui Tang. Meng Li, Miaojun Wang. Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang. R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen. Shengfeng Ye, Shiyu Wang. Shuiping Yu. Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye. Tao Yun, Tian Pei, Tianyu Sun, Wang. Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang. Wenjun Gao. Wenqin Yu, Wentao Zhang. W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang. Xiaokang Chen, Xiaotao Nie Xin Cheng. Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song. Xinyi Zhou, Xianzu Wang. Xinxia Shan, Y. K. Li, Y. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li. Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu. Yichao Zhang. Yifan Shi, Yiliang Xiong. Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He. Yunfan Xiong. Yuxiang Luo. Yuxiang You. Yuxuan Liu, Yuyang Zhou, Y X. Zhu, Yanhong Xu, Yanping Huang. Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang. Yukun Zha, Yuting Yan, Z. Z. Ren, Zchui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL httpa://arxix. .rgg 250b./2201... 12948
[5] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mo- hammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image image diffusion models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 01858-79885, Curran As- sociates, Inc., 2023. URL Ihttpe: / httpop:://pooceedingg. wew.eeeeeeddinnenneeeeeeeeeeeeee nwwriee.rrcooo papppa/enpneerpppppppppppperrrr httper//iiieeeepeee//////////// /223//file/ Paper Confeeenoooeeeeeceeeee pdf
[6] Yaru Hao, Zewen Chi, Li Dong, and Furu Wei. Optimizing prompts for text-to-image genera- tion. A. Oh. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Ad- vances in Neural Information Processing Systems, volume 36, pages 66923-66939 66939, Curran Asso- ciates, Inc., 2023. URL https: stttps:/pprocedding neur ips. httpsscccpppprrpepppppppp wwtpappr.files/ppeeeeeeeeeeeeeee wtt.e///ppppr//22/////t Paper- Paper-Conerrrreeeeeeeee Conference. pdf
[7] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6840-6851. Curran Asso-
ciates, Inc.. 2020. URL lttpps://proceedings. nwwrinerrccooo ctturpps.ccpaapaee cc/paper_filesse wwt.:ffwwwsepeeeeeeeeeeeeaeeeeee https//aappp/222/////// 22000/2120/ (ebbcceecbeaar0007711111110
[8] Junjic Ke, Keren Yc, Jiahui Yu, Yonghui Wu, Peyman Milanfar, and Feng Yang. Vila: Learning image aesthetics from user comments with vision-language pretraining, 2023. httpps/////arx www.arwwww...rraaaa. org/bbs/ 2303. 14302
[9] Varun Khurana, Yaman Kumar Singla, Jayakumar Subramanian, Changyou Chen, Rajiv Ratn Shah, zhiqiang xu, and Balaji Krishnamurthy. Measuring and improving engagement of text-to-image generation models. In The Thirteenth International Conference on Learning Representations, 2025. URL https //openreview.co pepperreview.ne/forruee www.neeet/oonmmeeomm www.ooorwwiiiittttooooooa
[10] Semin Kim, Yeonwoo Cha, Jachoon Yoo, and Seunghoon Hong. Neward-agnosiee prompt optimization for ext-to-image diffusion models, 2025. URL httpss://araii wwweexivv..orr//e /200/ 16853
11

---

[11] Yuval Kirstain, Adam Polyak, Uriel Singer. Shahbuland Matiana. Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. In Thirty-seventh Conference on Neural Information Procexsing Systems, 2023. URL B wwt.
[12] Kyungmin Lee, Xiaohang I.,, Qifei Wang. Junfeng He, Junjie Ke, Mang-Huunn Yang, Irfan Essa, Jinwoo Shin, Feng Yang, and Yinxiao Li. Calibrated multi- prete-preeeeence optimization for aligning diffusion models, 2025. URL lattps:///axaati htt.pp/s/aaaxiv.orgaa/////////// wtg//os////////ooooooe 02588
[13] Lexica. Lexica.
n.d. Accessed: 2025-10-24.
[14] Meta AL Llama 3.2 11b vision instruct.
Vision 2024. Accessed: 2025-05-15.
[15] Weayi Mo, Tianyu Zhang, Yalong Bai, Bing Su. Ji-Rong Wen, and Qing Yang. Dynamic prompt optimizing for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26627-26636, June 2024.
[16] OpenAL. Dall-e 3 system card. cessed: 15.
http://pna./aoo/////ddd////aee-- system-card/////// 2023, Ac-
[17] Dustin Podell, Zion English, Kyle Laccy, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, 2024, URI.ttps: httpps://opeureviev
[18] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763. PMLR, 18-24 Jul 2021. URL htt.htps///// / //popeeed/// mlr. press//139//adee/ ord21a. html
[19] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In seirth- Conference on Neural Information Processing Systems, 2023. btttps:/:////eeeeeee
net/forum?
id=HPuSIX.Jaa9
[20] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint wxxiv:2/02.1209) 2021.
[21] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffuusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patiern Recognition (CVPR). pages 10684-10695. IEEE, 2022.
[22] Christian Schuhmann, Romain Vencu, Ryan Beaumont, Georgios Kordopatis-Zibe Zilos, Ramakrishna Vedan- tam, Tomasz Mazur, David Cox, Arthur Mensch, Hunter Gao, Ludwig Zhu, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. In NeurlPS Datasets and Benchmarks Track, 2022.
[23] Christoph Schuhmann. improved aeaprove--acshetticc https://gi thub. wtw. http://hiisttopssshhaaaaamnnnnee http:rrooooddddtetteeeeeeeee weesshetic-prddeeee GitHub reposi- tory, Apa license.
Clip+mlp aesthetic score predictor, 2023. URL
[24] David Silver, Thomas Hubert, Julian Schritwiesser, loannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel. et al. Mastering chess and shogi by self-play with a general reinforcement lcarning algorithm. arXiv preprint arXiv: 011811, 2017a
[25] David Silver, Julian Schrittwieser, Karen Simonyan, Joannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 55077666333334-33355 2017b.
[26] Gerald Tesauro. Td- ga-mon, a self- teaching backgammon program, achieves master-leve play. Neural Computation, 6(2):215-219 03 1994. ISSN 0899-7667. doi: 10.1 162/ncco. 1994.6.2.215. URLphttpe
7/doi. //àoi.org//0. 10. www2//ecoo..o.. 1994............... 1994.6.99... 6. 2.12
[27] Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong. Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8228 8238, June 2024.
12

---

[28] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong. Qinkai Li, Ming Ding. Jie Tang, and Yuxiao Dong. Imagereward: learning and evaluating human preferences for text-to-image generation. In Proceedings the 37th International Conference on Neural Information Prucessing Systems, pages 15903-15935, 2023.
[29] Junyi Yao, Yijiang Liu, Zhen Dong, Mingfei Guo, Helan Hu, Kurt Keutzer. Li Du, Daquan Zhou, and Shanghang 7hang. Promptcot: Align prompt distribution via adapted thahn-of- In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7027-7037, June 2024.
[30] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metries for semantic inference event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014. doi: (0.1162/tac_aa000 URL https
//aclanthology. org. orgg/14--- Q14 1006/
[31] Jiahui Yu, Yuanzhong Xu, Jing Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfci Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parckh, Xin Li, Han Zhang, Jason Baldridge. and Yonghui Wu, Scaling autoregressive models for content-rich texl-to-image generation, 2022. URL |Rttttps::/
htttt:://aaaaa http:////axxv.oooo org abw/ 10789
[32] Huizhuo Yuan, Zixiang Chen, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning of diffusion models text-to-image generation. The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. btttp:://oppnreeeeee wwt.e//wwww..eeeoooonndddaaaaaae
A Technical Appendices and Supplementary Material
A.1 Acknowledgments and Disclosure of Funding
Rajiv Ratn Shah is partly supported by the Infosys Center for AI, the Center of Design and New Media, and the Center of Excellence in Healthcare at III Delhi. This work is partially supported by NSF AI Institute-222966 Any opinions, findings and conclusions recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation, the Institute of Education Sciences, or the U.S. Department of Education.
NSF RI-2223292, an Amazon research award, and an Adobe gift fund.
A.2 Ablation
A.2.1
Effect of Reasoning-Baan Based Prompting
We conduct an ablation study to evaluate the impact of reasoning- based prompts versus standard prompts across two VLMs. The results (T show consistent improvements in aesthetics, human preference, and engagement scores when reasoning is incorporated.
Model
Reasoning
Aesthetics Laion Score
Human Preference
Engagement (EOIG Score)
PickScore
Low
Mid
High
LLaMA IIB
X
8.19
22.31
81.27
86.88
89.64
LLaMA 11B
/
8.37
22.37
82.53
87.96
90.64
LLaVA 7B
X 6.82
21.91
82.10
86.94
89.03
LLaVA 7B
/
7.22
21.98
81.28
86.99
89.53
Table 5: Ablation study comparing SPRO-prompt method with and without reasoning for two VLMs across three evaluation metrics. Reasoning improves aesthetics (Laion Score), user preference (PickScore), and engagement (EOIG Score) for all buckets.
A.2.2 Effect of Training Data Size Across Iterations
We analyze how the size of the training dataset affects the performance of our model across iterations. In particular, we train our model on three different dataset sizes and evaluate the resulting PAP scores. The results demonstrate that using 6K pairs yields the best performance shown in Tablç
13

---

Training Data size
PAP Score
3k
22.33
6k
22.39
60k
22.15
Table 6: Performance of models trained at different dataset sizes. Best result obtained with 6K pairs.
These findings suggest that moderate data size with high-quality pairs is more effective than using significantly larger datasets.
A.2.3 Iterative Refinement Analysis
To explore the potential of prompt-image refinement without additional training, conducted an iterative feedback loop experiment. The model was prompted with its own generated image and caption produce refined outputs. This repeated for two hops to observe runtime improvements.
This experiment focused on difficult samples in the bottom third quartile of initial caption scores. Table summarizes score changes across hops.
Objective
Metric
Initial Score
Hop 1
Hop 2
Human Preference
PickScore 20.31
+0.73
+0.84
Aesthetics
LAION Score
8.18
+0.10
+0.13
Engagement
EOIG Score
77.50
-0.43
+1.75
Table 7:
Runtime improvements on low-scoring samples using iterative refinement.
These results indicate that even without retraining, iterative prompting enables the model to refine suboptimal generations over multiple steps. This has practical potential for human numan--in-hhe -eee-in--tthooge toop systems or adaptive image enhancement pipelines.
A.2.4 Training and Testing data score distributions
The figure Iillustrates the model's improvement over training iterations. With each iteration of data generation using the training set, the score distribution progressively shifts toward higher values, indicating performance gains. similar trend is observed on the test set, demonstrating consistent improvement across all three methods: SPRO-Prompe SPRO-Image, and SPRO-MM.
SPNC Nol
ara
Dapaors
264
10000
LSC0
LE
Figure 4: Evolution of self-play scores over training iterations and testing iterations. (i) Displays the LAION score distribution of four iterations of SPRO-text for training data (ii) Displays the LAION score distribution three iterations of SPRO-text for testing data and Base captions (iii) Displays the LAION score distribution of three iterations of SPRO-image and SPRO-MM. The average scores of generated samples improve rapidly in early iterations but gradually saturate, indicating convergence. The score difference between high- and low low-quality samples also narrows, suggesting increasing generation consistency.
14

---

A.2.5
Comparison with CAPO on Aesthetic Quality (VILA Score)
The CAPO paper [12 evaluates aesthetics using the VILA score
8 g
To enable direct comparison,
we use the Parti Prompt Dataset [31] and compute VILA scores approach.
for images generated by our
Model
VILA Score
CAPO SPRO-Prompt
6.14
6.48
SPRO-Image SPRO-MM
5.95
6.39
Table 8: VILA aesthetic scores on Parti prompts using the SPRO methods and CAPO. SPRO-Prompt surpasses CAPO baseline, with other variants showing competitive results.
Our SPRO- PrRO-Prompt variant outperforms the CAPO baseline, indicating that our method enhances aesthetic quality even standard prompt benchmarks.
A.2.6 CLIP Score Sanity Check for SPRO Methods
To ensure the similarity of the generated images relative to the original base images, we computed the CLIP Score [8] similarity between each base image and its corresponding generated image across all evaluation metrics. This sanity check validates that the optimized images remain semantically aligned with the input images while improving
user preference, aesthetics, and engagement by adding details to prompt that do not reflect much semantic change in new image. Table summarizes the mean CLIP Score values for each SPRO method across the three human preference objectives.
Method
User Preference
Aesthetics
Engagement
SPRO-Prompt 77.31
86.61
75.82
SPRO-Image 79.88
88.02
78.10
SPRO-MM
79.12
87.44
77.34
Table 9: Mean CLIP Score similarity between base images and generated images for different SPRO methods, across user preference, aesthetics, and engagement objectives.
A.2.7 Winrates of prompt-space methods
We evaluate the comparative winrate of prompt space method using different VLM across multiple diffusion models and metrics.
Winnoee
Marri Aesthetics
Contucion
Confaciin Marrirr prefevece
30h0 500L
20-4
20u4
90.0
SPRD S0RL-
585
562
55.5
17
SP10 S0XL-
2011
4B
7LP
42.7
Ceyioos K00
13.0
21.
25.2
27.1
467
RDCL
24.3
C2.
261
2144
BIN
Nechor
Figure 5: Winrate confusion matrices comparing performance of prompt engineering methods (SPRO- prompt, GPT, Base Caption) paired with diffusion models (SDXL, FLUX) across aesthetic quality (left), user engagement (center), and preference (right) metrics. Values represent percentage frequency with which row methods outperform column methods direct comparisons.
15

---

A.2.8 Evaluation without Dependence on External Rewards
To examine SPRO's reliance on learned reward models, we conducted an additional experiment using direct human ratings from the AVA dataset as supervision. The goal was to test whether SPRO can align with human aesthetic preferences without any pretrained reward model.
An untrained LLaMA-3.2-11B was used as the Guider VLM. Starting from a base prompt, the model generated three candidate prompts (Pt,P2,P3) Ps) aimed improving aesthetics. Instead of scoring generated images with a reward model, each prompt retrieved its top-5 most relevant AVA images using CLIP-based text-image similarity. From these retrieved images, we selected those with the largest difference in human aesthetic ratings to form preference pairs for Direct Preference Optimization fine-tuning.
During evaluation, each test prompt was assigned the highest human aesthetic rating among its retrieved AVA matches. Despite the weak supervision, as shown in Table [00 SPRO achieved clear improvement 6,80 versus 5.90 for base captions indicating that it can effectively learn aesthetic alignment directly from human annotations.
Model
Training Method
Human Annotated Score
Base-Captions
5.90
GPT-4o
Few-shot
6.20
-lama-3.2211B
Base
6,50
Llama- 3.2-11BBB-11B
SPRO iteration-1
6.80
Table 10: Evaluation of aesthetics using human-annotated scores from AVA dataset without reliance
on external reward models.
A.2.9 Comparison with Strong Reasoning Models on all three Human Preference Objectives
To further evaluate SPRO's generalizability, we compared our framework against advanced reasoning- augmented models, including Qwen-3, GPT-4o, and GPT-o3, across three human preference ob- jectives: aesthetics (LAION Score), user preference (PickScore), and engagement (EOIG Score). Engagement was measured across three difficulty buckets low, medium, and high following established evaluation protocols from prior work.
As shown in Table I1] SPRO-Prompt consistently outperforms all baselines across all metrics. On the aesthetic objective, it achieves the highest score of 8.60, surpassing Qwen-3 (7.55), GPT-4o (7.57), and GPT-o3 (7.67). For user preference, SPRO-Prompt reaches 22.42, outperforming GPT-o3 (19.47), GPT-4o (22.31), and Qwen-3 (22.10) 10). Most notably, across all EOIG engagement buckets, PPRO-Prompt attains the best performance: 83.48 (low), 88.36 (medium), and 90.24 (high).
These results demonstrate that SPRO-Prompt surpasses both general purpose and advanced reasoning models in aligning generated outputs with human preference objectives.
Model
Aesthetics
User Preference
EOIG Score Mid
Low
High
Qwen-3
7.55
22.10
74.12
83.12
84.23
GPT-4o
7.57
22.31
77.70
87.83
89.95
GPT-o3
7.67
19.47
75.91
75.97
79.51
LLaMA-3.2-11B
7.38
21.96
83.19
87.72
89.32
SPRO-Prompt (Ours)
8.60
22.42
83.48
88.36
90.24
IpRO-Image
7.48
22.09
70.70
77.94
77.43
SPRO-MM
8.42
22.42
83.20
87.39
89.38
Table 11: Performance comparison of SPRO and other aessoing--augmentn models across
aesthetics(LAION Score), user preference(PiieSScore)) and engagement metrics(EOIG Score).
16

---

A.2.10 Effect of SPRO Iteration on Reasoning-Enabee Models for objective of Aesthetics
Motivated by the strong performance of SPRO-Prompt, further investigated whether self-play training could enhance the ability of reasoning-base based models to generate prompts optimized for aesthetics. Specifically,
employed Qwen-3-7B as the Guider VLM and conducted single iteration of self-play using the human preference objective of aesthetics. The model generated reasoning chains and corresponding prompt candidates across varying temperature settings. These prompts were used to generate images with a frozen SIXL. model, which were then scored using the LAION aesthetic scorer. Based on these scores, preference pairs constructed and used for fine-tuning via Direct Preference Optimization (DPO), jointly conditioning on reasoning and prompts.
We evaluated the fine-tuned models the test set using the LAION aesthetic score. As shown in Table T2 SPRO training led to consistent improvements across both reasoning and non-reasoning VLMs. Notably: Qwen-3-7B improved from 7.55 to 8.46 after one iteration, while Llava-7B improved from 6.76 to 7.22 after three iterations and Alama-332-11 1BB improved from 7.38 to 8.30.
Model
Training Method
LAION Score (Aesthetics)
Base- Case-Captions
6.40
Qwen-3-7B
7.55
Qwen-3-7B
SPRO-it-1
8.46
Llava-7B
6.76
Alava-7B
SPRO-it-1
7.22
Llama-3.2-111
7.38
-lama-3.2-111
SPRO-it-1 SPRO-it-3
8.30
Llama-3.2-11B
8.60
Table 12: Comparison of LAION aesthetic scores before and after SPRO iterations of sesoning-capablee models.
A.2.11 Zero-Shot Evaluation on multiple metrics for same objective
To verify that SPRO-Prompt does not exploit any specific reward function and instead learns gener- alizable strategies, we conducted zero-shot generalization experiments across multiple metrics of each objective. While SPRO-Prompe was trained using single reward signal per objective, it was evaluated on unseen reward models representing the same underlying human preference dimension.
For the aesthetic objective, SPRO-Prome was trained using preference pairs derived from the LAION Aesthetic Score, but evaluated on unseen scorers VILA, the Improved Aesthetic Scorer, and CLIP similarity, using the Flickr dataset. As shown in Table 13 SPRO-Prome base prompt across all metries, achieving substantial gains on both VILA and the improved aesthetic scorer. These results confirm that SPRO-Promp generalizes beyond the LAION reward model, capturing a broader notion of aesthetic quality while maintaining semantic alignment as indicated by stable CLIP similarity.
consistently surpasses the
Model
LAION Score
VILA
Improved Aesthetic
CLIP
Base Prompt
7.38
60.74
6.30
83.21
SPRO-Prompt
8.60
65.16
7.47
79.50
Table 13: Generalization of SPRO-Prompt to multiple aesthetic scorers and CLIPP alignment on the Flickr dataset.
For the user preference objective, the model was trained with PickScore as the reward signal and evaluated using IewaeReward as an unseen metric on the Pick-a-Pic dataset. As presented in Table 14 SPRO-Prompt outperforms the base prompt across all evaluation metries-Pickeee 20.99), ImageReward (1.02 vs. 0.65), and CLIP similarity (89.32 vs. 87.76). This demonstrates that SPRO-Prompt is not overfitted to PickScore but instead learns prompting strategies that generalize across different models of user preference assessment.
(22.42 vs.
17

---

Model
PickScore
ImageReward
CLIP
Base Prompt
20.99
0.65
87.76
SPRO-Prompl 22.42
1.02
89.32
Table 14: Generalization of SPRO-Prompt to multiple user preference scorers and CLIP alignment on the Pick-a-Pic dataset.
Together, these findings show that SPRO-Prompt generalizes across multiple reward models within the same objective class, reinforcing that optimizes genuine human-aligned quality rather than engaging in reward hacking.
A.2.12 Multi-Objective Optimization: Aesthetics and CLIP Similarity
To assess SPRO's capability for multi-objective optimization, we extended training to jointly optimize two reward signals: (a) LAION score for aesthetics as the primary objective, and (b) CLIP similarity as a measure of semantic alignment. During self- self-play, the Guider VLM was conditioned balance both objectives simultaneously.
As shown Table [5] SPRO achieves substantial improvement in aesthetics while maintaining nearly identical CLIP similarity compared to the base prompt. This indicates that SPRO effectively supports multi mujei-obeective optimization, enhancing aesthetic quality without compromising text-image semantic consistency.
Model
LAION Score
CLIP Score
Base Prompt
6.40
83.19
SPRO-Prompt
7.90
83.28
Table 15: SPRO scores for for Multi-objective optimization using LAION aesthetic score and CLIP similarity. SPRO improves aesthetics without compromising image-text alignment.
A.2.13 Dual Optimization: Aesthetics and User Preference
To evaluate SPRO's ability to optimize multiple human preference objectives simultaneously, we conducted experiment using two orthogonal signals: aesthetics (LAION Score) and user preference (PickScore). Unlike CLJP similarity, which primarily ensures semantic consistency, these two metrics represent distinct and potentially unaligned preferences. Preference pairs were generated conditioned on both LAION and PickScore, retaining only those where improvements occurred in both metrics. These filtered pairs were used train the Guider VLM for one iteration, enabling the model to generate reasoning chains and prompts that jointly optimize both objectives.
As shown in Table I6, SPRO-Prompt improves performance across both metrics: LAION Score rises from 6.40 to 7.66, and PickScore increases from 22.69 to 22.96. These results highlight SPRO's capacity learn prompting strategies that align with multiple non-conflicting human preferences, demonstrating the framework's flexibility in multi-reward scenarios.
Model
LAION Score
PickScore
Base Prompt
6.40
22.69
-PRO-Prompt
7.66
22.96
Table 16: SPRO scores for for Dual objective optimization of aesthetic quality and user preference. SPRO improves both simultaneously.
18

---

A.2.14 Optimization Under Conflicting Objectives
To investigate SPRO's ability to handle conflicting objectives, we designed a composite pseudo-reward function:
Conflicting Score 2 LAION Score 0.5 x PickScore,
which favors images with higher aesthetic appeal while penalizing user preference. This simulates real-world scenarios where maximizing one objective may necessitate compromising another, such as artistic or stylistic choices that enhance aesthetics but reduce broader appeal.
generated based improvements in this conflicting score and used train the Guider VLM for one iteration via SPRO-Prompt. As shown in Table 17 the model successfully increases the LAION aesthetic score from 6.40 7.62 while reducing PickScore from 22.69 to 21.70, reflecting the intended trade-off. These results demonstrate that SPRO can effectively optimize for non-aligned conflicting objectives, highlighting its applicability complex human preference modeling scenarios.
Preference pairs
Model
LAION Score
PickScore
Base Prompt
6.40
22.69
SPRO-Prompt
7.62
21.70
Table 17: SPRO scores for Conflicinn-oojeetive optimization: SPRO increases aesthetics while decreasing user preference intended by the design of Conflicting reward function.
A.2.15 Generalization of SPRO across multiple datasets
To further evaluate the robustness and adaptability of SPRO, we tested the trained models on additional datasets containing diverse base prompts and image distributions. This experiment examines whether the prompting strategies learned through self-play generalize effectively across datasets that differ from those used in training, while maintaining strong performance human preference objectives,
For both the aesthetic and user preference objectives, we conducted evaluations on the PartiPrompts dataset. As shown in Table [8] SPRO-Prompi demonstrates consistent improvements over base prompts across three independent aesthetic reward models: LAION, VILA, and the Improved Aesthetic Scorer. Specifically, SPRO-Prompt improves LAION Score from 6.16 to 8.01, VIIAA Score from 55.15 to 64.81, and the Improved Aesthetic Scorer from 5.60 to 6.78.
Method
LAION Score
VILAA
Improved Aesthetic
Base Prompt
6.16
55.15
5.60
SPRO-Prompt
8.01
64.81
6.78
Table 18: Evaluation of SPRO-Prompt on the PartiPrompts dataset for the aesthetic objective using multiple reward models.
For the user preference objective, report results in Table 19 where SPRO-Prompt again achieves a substantial improvement over the base prompt, increasing PickScore from 21.89 to 22.84.
Method
PickScore
Base Prompt
21.89
SPRO-Prompt
22.84
Table 19: Evaluation of SPRO-Prompp on the PartiPrompts dataset for the user preference objective.
Further, we assess its performance on the LexicaDB dataset- benchmark used Reward- Agnostic Prompt Optimization for Text-to-Image Diffusion Models (RATTPO) [10]. This setting allows us to test whether our model, trained on distinct objectives, can effectively generalize to unseen reward functions and datascts.
19

---

The RATTPO baseline is explicitly trained to optimize prompts for the ImageReward function, which aligns text-to-image generations with human preference. In contrast, SPRO-Prompt has never been exposed to this reward model during training, providing a true zero-shot generalization test.
Following the RATTPO setup, randomly sample 160 image-caption pairs from LexicaDB (as the original test set is unavailable), For each instance, we use the base image and prompt as inputs our user-preference optimized SPRO-Prompt model, generate an improved prompt, and evaluate the resulting generations using the ImageReward metric.
[3
Model
Approach
Initial ImageReward score
Optimised ImageReward score
RATTPO
Prompt
0.049 + 0.143
1.132 0.049
SPRO- (PRO-Prompt(ours)
Prompt
0.1588
1.439
Table 20: Evaluation of SPRO-Prompt the LexicaDB dataset for the user preference objective.
These results confirm that SPRO generalizes effectively across datasets and reward models, reinforcing its robustness and applicability to diverse prompt-image distributions. The model's 's ability to maintain improvements unseen datasets highlights that it learns transferable strategies rather than overfitting to the training distribution.
A.3 Analysis of reasoning chains generated via Self-play
In this section, analyze several strategies discovered by our Self-Play framework. These strategies, emerged through iterative optimization via Self-play of intelligent VLM guided by reward score. Using the complete set of data generated through self-play, we categorize the strategies into two groups- those that consistently improve scores and the ones that tend to degrade performance for specific human preference. This analysis reveals insightful patterns about what types of prompt modifications or stylistic shifts align with specific human preferences as shown in Table 21
A.4
Experimental Setup
A.4.1 Setup
Our Prom-Prompt framework employs a Guider VLM generate reasoning chains and optimized image prompts. We use LLaMA-3 3.2-1 2-111B B-B- Vision-Instruce [14] as the backbone for the VLM. All experiments are conducted on NVIDIA A 100 GPUs DPO fine-tuning is performed on same with per-device batch size of 2, using gradient accumulation to achieve larger effective batch size. Training is run for epochs using 6 precision and updates all model parameters.
For image generation, we use frozen Stable Diffusion XL (SDXL) [7] as the base diffusion model. In the SPRO-Image method, we finetune the same SDXL model using high-reward synthetic images paired with their corresponding base captions. We use resolution of 512x512, a learning rate of 1e-6, the 8-bit Adam optimizer, and train for 50 epochs with gradient accumulation factor of 4.
In the SPRO-MM setup, we combine both components: prompts are generated using the Guider VLM trained via SPRO-Prompt, and images generated using the diffusion model trained via SPRO-Image.
To assess generalizability of prompts, we also validate our results using two alternative diffusion models: FLUX. 1-dev 22 and DALL-E [16].
Seaf-Play Data Generation We generate training data through an iterative self-play procedure involving the Guider VLM and frozen diffusion model. The process begins with a seed dataset
of image-caption pairs tailored to each human preference objective: (1) Aesthetic appeal: 30,000 image-caption pairs from the Flickr30k dataset [0). (2) Engagement: 35,000 images sampled from the EngaginglmageNNt train set (9. with equal representation from the top and bottom 20th percentiles of like counts to capture both highly and poorly engaging content. User preference: 17,400 images from the validation split of the Pick-a-pic dataset [11.. Using these seed datasets, the Guider VI.M generates prompts via self-play, which are used with the diffusion model to create new image-caption pairs scored by the appropriate reward model. Only samples that exceed the reward
20

---

Human Preference
Focus Area
Winning Strategy
Losing Strategy
Language Style
More technical systematic
More artistic and conceptual
Atmospheric effects grouped with vignette
Atmospheric effects grouped
User Preference
Mood Lighting
with lighting
Noise reduction and texture details separated in different strategy
Noise reduction and combined strategy
Noise and Texture
Combines cnlor grading + contrast
Color and Exposure
Separates contrast and color
Sharpening clarity unified function
Detail and texture enhancement separately
Detail Work
Artistic and interpretive (e.g., "artistic flourishes," "visual harmony")
Technical and grounded (e.g- "posture," "composition,"" "depth"")
Language Style
Aesthetics
Focus more style and cinematic scenes
Focus on realism and depth in scenes
Scene details
Guiding viewers attention through placement
Combine depth of fields and spatial balance
Composition
Includes broader scope colours
Focus vibrancy keeping natural hues
Color Palette
Digital Artists and Concept Artists
Photographers Visual Designers
Persona Point
Technical enhancements like lighting, motion, and composition
Clear emphasis on visual appeal and connection with audience
Strategy focus
Engagement
Straightforward, action- cetion-oriented focusing on engagement tactics
Tone
More detailed and descriptive
Lighting and Shadows
Focus on hackground and shadow effects
Focus dimension and depth
Focus on motion blur and dynamic poses
Mentions motion lines or animation
Dynamic elements
Does not directly address emotional connection; focuses more visual elements and energy through motion.
Directly calls for enhancing emotional connection through facial expressions, hody language, and context.
Emotional Connection
Table 21: Analysis of Winning and Losing Strategies Across Focus Areas for each Human Preference. These advance stratgies are highly technical, human poreference specific and counter intuitive and hence difficult for humans explore.
distribution's 's mean score for base-captions and pass a CLIP-based image similarity threshold are retained.
A.4.2 Prompts used to generate data via iterative self-play
Prompt for Aesthetics
Think stop-by-step analyze image to iteratively
60l-word model prempt produce
of the image with improved acsthetics. Ensure all original details are preserved, and the enhancements match the style and essence the original image while improving its visual appeal. The base caption for image is: caption Output format:
Stepi_Plan: [Specific plan step-1]
Stap2_Plan: [Spacific plan for stap-2, building on stop-1]
Step3_Plan: [Specific plan for step-3, building on step-1 step-2]
Caption: [Generated proupt]
Generate up to six steps and the final prompt. The prompt is extremely important and must not exceed 60 words.
21

---

Prompt for User Preference
Think step-by by-step and analyze the image to sterutively enhance it. Generaaee 60-word diffusion model prompl to produce version image aligus more closely general hurnan preferences. Ensure original preserved, the enhancements remain consistent with the original image's is style and essence.
The base caption the image caption
Output format:
Stop1_Plann [Spacific plan for stap-1]
=lam2_Plam: [Epecific plan for step-2, building on step-1]
Step3_Plan:
[Specific plan for step-3, building step-1 stop-2]
Caption: [Generated proupt]
Generate up sleps and the linal prompt. The prompt is extremely important and must exceed 60 words.
Prompt for Engagement
Think step-by-step analyze image to iteratively enhance Generate 60-word diffusion model prompt to produce version the inage that would receive engagement in the form of likes social media platforms (e.g. Twitter). Ensure all original details are preserved, and the enhancements match the style and essence the original image. The enhanced image should align with the accompanying
and brand image of the account posting You may accompanying the image ist caption
the content
approved. The tweet
(weel is posted by: company Make the prompt always present. needed, shortening prompt allowed. caption acts prompt generate the new image and should he suitable for input into an image generation application. Output format:
Stop1_Plam: (Spacific plan for stap-1]
Step2__Pass [Spacific plan stap-2, building on step-1]
Step3_Plem: (Specific plan for step-3, building step-1 step-2]
Caption: [Generated prompt]
Generate to six steps and the final prompt. The prompe is extremely important.
A.5
Human Study
A.5.1
Human Evaluation Instructions
We conducted human evaluation study to assess the quality of synthetic images generated by our pipeline. Below are the exact instructions provided to the participants during the evaluation:
Instructions:
You will be shown 20 prompts.
Each prompt will be presented with images generated using the same prompt.
For each prompt, please select the one image you prefer the most.
You must make selection to proceed to the next prompt.
A.5.2 Results from Human Study
To assess alignment with actual human preferences beyond automatic reward signals, we conducted human evaluation comparing image generations from three prompt types: the original base caption, GPT-4o-optimized prompt, and our SPRO-prompt. For each input, three images were generated using the same diffusion model (SDXL. or Flux), each corresponding to one prompt type. Human annotators (N=20) selected their preferred image in a randomized, blind setting. As shown in Table 22 SPRO- prompt is preferred over GPT-4o in 66.7% of cases and over base captions in 78.9%. In contrast, base captions are chosen over GPT-4o in 52.38% cases, but only 21.1% when compared to SPRO. These results confirm that prompts optimized via self-play lead to significantly higher human preference.
A.6
Qualitative analysis
A.6.1 Caption and Images generated by SPRO-Prompt method
In this section, we present representative examples from the test set across various human preference categories. For each example, we show: the base caption, the image generated by SDXL. [7 using the base caption, the SPRO-Prompt i.e our method's refined caption and the image generated by SDXL using the SPRO- SPRO-Prompt. We observe consistent improvement in reward scores, significantly
22

---

Comparison
Preferred Candidate
Win Rate (%)
Base Caption vs. GPT-4o
Base Caption
52.38
Base Caption vs. SPRO-Promp
Base Caption
21.10
SPRO-Prompt vs. Base Caption
SPRO-Prompt
78.94
SPRO-Prompt vs. GPT-40
SPRO-Prompt
66.67
Table 22: Pairwise human preference win rates between prompts: base captions, GPT-44--ptimizze
prompts, and SPRO- prompts. Each row shows the preferred candidate and the percentage of times images generated using the candidate prompt was selected over the alternative.
improvised prompt with more strategic details and notable visual differences between the base and improved images generated. These qualitative improvements further validate the quantitative gains achieved by method. The samples for user preference is in Table 23 for aesthetics refer Table 24
and for engagement refer Table 25
A.6.2 Qualitative results of SPRO- and SPRO-MM method
This section presents qualitative analysis of outputs of SPRO-Image and SPRO-MM method by. Specifically, we compare images generated using original, unmodified captions, images generated by aPRO-Image method i.e fine-tuned diffusion model that receives the base captions input and SPRO-MM where the fine-tuned diffusion model is guided by SPRO-Prompt enhanced captions. For the objective of user preference check Table26 for engagement refer Table 27 and for aesthetics refer Table 28
A.6.3 Diffusion Model agnostic prompts
We show qualitative samples of images generated using both SDXL. and Flux with SPRO-Prompt generated captions. Good reward scores using both models show diffusion agnostic nature of prompts generated. For aesthetics refer Table 29), similarly for objective of user preference refer Table 30 and for objective of enagagement refer Table 31
A.7 Broader Impact
Our work contributes the growing effort of aligning synthetic image generation with human preferences through an explicit alignment framework. This has several positive societal implications. By reducing reliance on large-scale real-worke datasets, our approach can alleviate data privacy concerns and promote more ethical and inclusive data practices. Moreover, the ability to generate preference aligned synthetic images could support in multiple applications such in education to generate more memorable images. However, this capability also introduces ethical challenges. If the alignment process inadvertently captures biased or skewed preferences, it may amplify societal biases or aesthetic norms, particularly in sensitive contexts involving identity, culture, or representation. Care must therefore be taken audit and evaluate alignment criteria and ensure transparency in the deployment of such models. We encourage future work to further investigate mechanisms for robust preference modeling, fairness-aware alignment, and safeguards against misuse.
A.8 Dataset Licenses
In this work, we use three available datasets: Flickr30k 30 and Pick-a-Pic [I]] and EngagingIma- geNet (9] We ensure that all datasets used in accordance with their respective licenses.
Flickr30k Dataset: The images in the Flickr30k dataset are provided under the Creative Commons Auribution-NnnCCnmmraal----
3.0 License (CC BY-NC-SA 3.0).
PickAPic Datawset: The Pick-a-Pic dataset is released under the Creative Commons 1.0 Universal Public Domain Dedication.
EigagingImaggeNe Dataset: The
dataset is released under the Creative
Commons Attribution 4.0 International License.
23

---

Caption
Cap
SPRO-Prompt
SPRO-Prompt + SDXL
generate a cute purple cat with purple lavender fur and
A cute very purple cat with purple lavender fur and cute black hat-hat. very cyan blue background
black top-hat standing on
very cyan blue background,
with enhanced lighting, color gradients, and textures,
maintaining the original expression and details, while sharpening facial features and adjusting saturation for vibrant, dreamy atmosphere.
PickScore: 25.94
PickScore: 26.26
Generate an enhanced image of an evil villain holding mini earth, with refined facial features, atmospheric lighting, and detailed textures, set in dark, ominous environment, maintaining the original details and style, with focus on contrast and color grading to create more dramatic and sinister mood.
evil villain holding mini earth
PickScore: 24.42
PickScore: 26.90
Generate a high-quality image of a freshly baked loaf of bread, showcasing its warm golden crust and soft interior. the image should be well-lit, with subtle shadows and highlights. preserve the natural texture and details of the bread. style realistic, food photography. ensure the image is visually appealing and inviting, making the viewer want to
bread
PickScore: 21.24
PickScore: 22.89
take a bite.
Generate a high-quality image of pirate in a pirateship, preserving original details, with enhanced color tone, contrast, and texture, showcasing the pirate's 's features, clothing, and the ship's 's wooden hull, with warm and inviting atmosphere, as if the pirate is about to set sail, with subtle noise reduction and local adjustments for a refined look.
A Pirate in a Pirateship
PickScore: 22.15
PickScore: 23.20
Table 23: Comparison of base captions and SPRO-Promm improvised captions, along with the corresponding images generated by SDXL for higher user prefernce.
24

---

Caption
Caption+
SPRO-Prompt
SPRO-Prompt + SDXL
Vibrant tapestry of life unfolds contemplative
figure, amidst a kaleidoscope of warm and cool hues, with golden glow illuminating his introspective face, surrounded by richly detailed fabrics, inviting the viewer into world of cozy introspection.
brown-haired man is sitting in front of sewing machine looking down.
LAION Score: 7.59
LAION Score: 8.36
caucasian woman in a blue top and black slacks looks off the side of the
A vibrant, sunlit scene unfolds with a woman in radiant blue top, her smile radiant amidst a warm, inviting atmosphere, surrounded by detailed, rustic furniture and lush, detailed brush strokes in the background, where figures mingle, their faces aglow in the soft, golden light, inviting the viewer into world of joy and connection.
camera with a drink in one hand and amused look
upon her face, while man sits next to her, looking unamused, drink in one hand, straw in the other.
LAION Score: 7.07
LAION Score: 8.25
Generate the aesthetic the runners navigating through stream, preserving their vibrant colors and detailed expressions while subtly adjusting hues, contrasts, and depth to draw the viewer into the lively scene. create an image where the runners take center stage amidst harmonious, detailed, and vibrant background.
group of people wearing marathon signs run through a stream.A
LAION Score: 6.73
LAION Score: 8.16
a majestic, golden retriever with vibrant fur and playful eyes glides effortlessly through the serene waters, surrounded by tapestry of blue sky, wispy clouds, and colorful aquatic blooms, all bathed in warm, golden glow amidst a soft, dreamy palette, inviting the viewer into its whimsical world.
A black and white dog is swimming through some water.
LAION Score: 5.62
LAION Score: 7.23
Table 24: Comparison of base captions and SPRO- improvised captions, along with the corresponding images generated by SDXL for better aesthetics.
25

---

Caption
Caption+SDXI
SPRO-Prompt
SPRO-Prompt + SDXL
A person working on sleek, modern laptop at wooden desk, with their hands poised over the keyboard. The screen displays vibrant desktop wallpaper featuring a stunning mountain landscape reflected in a calm lake during sunset, suggesting a peaceful and
A person is typing on a laptop keyboard.
EOIG Score: 33.78
EOIG Score: 67.38
focused working environment.
Warm whimsical celebration of pets, with pastel colors, soft textures, surrounded by hearts, flowers, and a warm glow, all set against clean and uncluttered background, showcasing grateful, expressive face, with a slight texture overlay add depth and tactility.
A french buildog walkboard beach
EOIG Score: 16
EOIG Score: 92.42
Compact gaming laptop. big-time performance. ROG Strix GL 12: Upgrade, customize, dominate. Sleek, high-contrast design meets bold, Lournament-gradd power. Get the edge you need play like a pro.
A computer monitor with game of Overwatch on the screen
EOIG Score: 71.25
EOIG Score: 74.71
Revamped Boeing-buil built Saturn V first stage, astronaut trio: stand heroically in front of a vibrant, and rocket plume, all set against a warm, nostaggic glow, emphasizing their monumental Apollo 10 mission..
Astronauts posing in front of American flags
EOIG Score: 14.32
EOIG Score: 54.02
Table 25; Comparison base prompts and SPRO-Prompt improvised captions, along with the corresponding images generated by SDXL. for better engagement.
26

---

Caption
SDXL
SPRO-Image
SPRO-MM
A curious exploring a haunted mansion
PickScore: 21.47
PickScore: 22.20
PickScore: 22.87
Woodeut print giant octopus attacks pirate ship, ultra detailed
PickScore: 21.40
PickScore: 23.62
PickScore: 23.53
Flower frame symmetrical.on white background pink, purple, peach,cream, vibrant, loose watercolour
gouache textured paper naive folk art, whimsical, Liila Rogers agency illustrator storybook style, subtle patternsstripes
PickScore: 19.92
PickScore: 20.13
PickScore: 20.48
Shiho www.irastttttttommmmm cann
BELE
Store with a sign that says Stable Diffusion.
PickScore: 20.76
PickScore: 22.60
PickScore: 21.47
Table 26: Comparison of base captions and corresponding images generated by: (i) SDXL using the base caption, (ii) SPRO-Image. fine-tuned diffusion model prompted with the base caption, and (iii) SPRO-MM, which uses the fine-tuned model with SPRO-Pros Promptr enhanced prompts. The comparison highlights improvements in image quality and alignment with user preferences.
27

---

Caption
SDXL
SPRO-Image
SPRO-MM
Graceful fishes swimming among blue, blooming water.
EOIG Score: 92.37
EOIG Score: 98.59
EOIG Score: 96.64
Man wandering through vast desert landscapes, guided by words.
EOIG Score: 38.43
EOIG Score: 51.17
EOIG Score: 52.22
Vibrant meals served on a table.
EOIG Score: 28.49
EOIG Score: 42.73
EOIG Score: 60.35
A yellow car is driving down road.
EOIG Score: 89.3
EOIG Score: 98.59
EOIG Score: 96.64
Table 27: Comparison of base captions and corresponding images generated by: (i) SDXL using the base caption, (ii) SPRO-Image, fine-tuned diffusion model prompted with the base caption, and (iii) SPRO-MM, which uses the fine-tuned model with SPRO-Promm enhanced prompts. The comparison highlights improvements in image quality and higher EOIG score which reflects enagagement.
28

---

Caption
SDXL
SPRO-Image
SPRO-MM
A white dog is splashing through the water.
LAION Score: 6.91
LAION Score: 7.95
LAION Score: 8.42
Female fire dancer performing in the middle of a city street with a crowd watching.
LAION Score: 6.51
LAION Score: 8.25
LAION Score: 8.78
black and white dog is swimming through some water.
LAION Score: 5.62
LAION Score: 7.97
LAION Score: 7.30
Old guitarist in white clothes performing to audience.
LAION Score: 6.57
LAION Score: 8.57
LAION Score: 8.08
Table 28: Comparison of base captions and corresponding images generated by: (i) SDXL using the base caption, (ii) SPRO-Image, fine-tuned diffusion model prompted with the base caption, and (iii) SPRO-MM, which uses the fine-tuned model with SPRO-Prompp enhanced prompts. The comparison highlights improvements in image quality and higher LAION score which reflects aesthetics.
29

---

Caption
Base Image
SDXL
FLUX
A man business suit holds the hand of young boy as they walk across lawn.
LAION Score: 8.75
LAION Score: 8.10
LAION Score: 6.43
A gray bird stands majestically on beach while waves roll in.
LAION Score: 6.43
LAION Score: 7.42
LAION Score: 7.57
Table 29: Comparison of original image with images generated using SDXL and FLUX with prompt input generated by EPRO-promph for better aesthetics.
30

---

Caption
Base Image
SDXL
FLUX
chanel letterss in the minddle, chanel style, black and white color, decorate design, advertising disign, with water color style red and black poppy flowers around the letters, hand painting, hand water color style, colorful water color background, some blurry poppy flower shapes in background color
CHANEL
CHANEL
PickScore: 16.65
PickScore: 20.67
PickScore: 21.40
COW
PickScore: 18.35
PickScore: 22.79
PickScore: 22.91
Table 30: Comparison of original image with images generated using SDXL and FLUX with prompt input generated by SPRO-prompy for higher user preference.
Caption
Base Image
SDXL
FLUX
A white airplane flying over the clouds
AVAR
EOIG Score: 76.95
EOIG Score: 98.20
EOIG Score: 95.40
group people holding the Indian flag
EOIG Score: 53.87
EOIG Score: 72.50
EOIG Score: 92.50
Table 31: Comparison of original image with images generated using SDXL and FLUX with prompt input generated by SPRO-prompt for better engagement.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims outlined in the abstract and introduction about method SPRO are supported by empirical results presented Section and A.2
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In the dection&@cnccueeee section, we discuss the limitations of our approach.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
32

---

Justification: All results presented in the paper are accompanied by clearly stated method, pseudo code, pipeline and assumptions. and A.4
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes using the setup, data and metric details provided in Section 3 and A.4 in appendix, one can reproduce the the entire results table.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
33

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Yes using the setup, data and metric details provided in Section B one can reproduce the main results. The code with all details will be added in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ce)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
//nips btsips.cc/public/guudde
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In the Section and we provide details of data splits, metrics, hyperpa- rameters and model used.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Our experiments are conducted on large-scale datasets, and all reported scores represent the average over three independent runs to account for variability. While we do not explicitly plot error bars, this averaging provides stable estimate of performance trends across methods and datasets. The use of large datasets further helps mitigate variance due to sampling noise, supporting the reliability of the reported results.
Guidelines:
The answer NA means that the paper does not include experiments.
34

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes provide details of GPU used and computation time in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http::////aarrpp..cccaaaaa//ahaa
Answer: [Yes]
Justification: The research strictly adheres to the NeurIPS Code of Ethics. We ensure responsible use of datasets, transparency in methods, and avoid any potential harm or misuse.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
35

---

Justification: Our work focuses on generating synthetic images that are aligned with human preferences using an alignment framework. The positive societal impact includes less dependency real data, reducing data privacy issues. However, the ability to model human preferences and generate realistic synthetic images could be misused for deceptive content. We discuss this in Section A.7
Guidelines:
The answer NA means that there is no socictal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: The models used in work built upon publicly available backbones such as LLaMA 3.2 Vision instruct and SDXL., both of which follow established safety protocols and have documented safeguards in place. Our fine-tuned models inherit these safety features and have been further trained to align with human preferences, avoiding the generation of harmful, offensive, otherwise risky content.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
36

---

Justification; All external assets used in this work, including models, datasets, and codebases, are properly cited. Additionally, licenses and terms of use for these assets are respected and documented in the appendix Section A.8 for transparency and compliance.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cww.coo//aao com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release new assets in the form of models and generated datasets, all of which are well documented. Documentation includes details about training settings. pre trained model used, dataset used and licensing terms. This information is provided alongside the assets through structured templates and supplementary materials. As our assets are synthetic and do not involve real individuals, there are no consent-related related concerns.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [Yes]
Justification: We conduct human evaluation of the synthetic data generated by our pipeline. Humans were asked choose among images, whichever they like the most. The full instructions provided to participants, along with details of the evaluation setup, are included in the Appendix Section A.5.I
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
37

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [Yes]
Justification: We took ERB approval.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLM was used in limited capacity, only for editing and formatting purpose.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.M for what should or should not be described.
38


Reviews:
[{"review_id": "3RCzefK6Zp", "reviewer": "Reviewer_TZhB", "summary": "This paper introduces Self-Play Reward Optimization (SPRO), a novel framework designed to align image generation models with nuanced human preferences without relying on extensive human annotations. The core idea leverages the reasoning capabilities of Vision-Language Models (VLMs) in a self-play paradigm to iteratively refine prompt generation and image synthesis. SPRO significantly outperforms prior methods and commercial baselines, demonstrating a scalable path toward aligning generative models with diverse human preferences.", "questions": "1. Did the authors evaluate SPRO with latest (and more powerful) reasoning LLMs, e.g., Qwen 3, DeekSeek r1, and/or o3?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. SPRO is presented as the first framework to explore self-play strategies in a multi-model setup for improving image generation. It integrates both prompt-space and image-space optimization within a unified self-play framework.\n2. SPRO achieves SOTA performance across different human preferences and is agnostic to different diffusion models.\n3. The authors plan to release a large-scale synthetic dataset of over 1 million human preference-aligned image-prompt pairs generated by SPRO, which will be a useful resource for future research.\n\nWeaknesses\n1. The paper acknowledges that SPRO's effectiveness is bounded by the reliability of external reward models, which are proxies for human preference and may not fully capture nuanced aspects. This is a fundamental limitation that could impact the true \"human preference alignment.\"", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful and encouraging review. We truly appreciate your recognition of SPRO’s novel multi-model self-play setup, its state-of-the-art performance across diverse human preferences, and its ability to remain agnostic to different diffusion models, both literature and commercial. \n\n**Dependence of External Rewards:** In the Self-Play Reward Optimization (SPRO) framework, we demonstrated strong performance across three human preference objectives: aesthetic appeal, engagement, and user preference. However, based on your suggestion, we wanted to investigate whether SPRO can function without reliance on such learned reward models. Therefore, in the limited rebuttal time, we conducted an interesting experiment using direct human ratings as the supervision signal. We cover the exact methodology and the results below:\n\nTo optimize for the aesthetic objective, we employed an untrained **LLaMA3.211B** as the **Guider VLM**. Starting from a base prompt b (sampled from the Flickr training set), the guider generated three reasoning chains and corresponding candidate prompts (p₁, p₂, p₃) designed to enhance image aesthetics. \n\nUsing these candidate prompts, we then generated images with a frozen SDXL model and evaluated them with the **LAION aesthetic scorer**, treating the scores as conditional signals for constructing preference pairs. To reduce reliance on this external reward model, we introduce an alternative strategy: instead of directly scoring the generated images, we use the candidate prompts (p₁, p₂, p₃) to retrieve the top5 most relevant images for each prompt from the **AVA dataset**, which provides human annotated aesthetic ratings. Retrieval is performed via a **CLIP based text–image similarity search**. \n\nFrom the 15 retrieved images (5 per prompt), we identify the two with the largest difference in aesthetic score and use their associated prompts to form a preference pair: \n\n$$\n(p1,p2,max⁡Δ(i1,i2))\n$$\n\nwhere p₁ and p₂ originate from the same base prompt b, and i₁, i₂ are the corresponding retrieved images. To ensure meaningful comparisons, we discard all pairs where the aesthetic difference is less than 2.29, the mean difference observed across the dataset. Then, the Guider VLM was fine-tuned using Direct Preference Optimization on the resulting reasoning–prompt pairs. Once the model was trained for one iteration, we evaluate the model to see if it is aligned with human preferences pertaining to aesthetic quality. \n\nWe evaluate the baselines and the SPRO-trained model on our test set using human ratings from the AVA dataset. For each prompt, we retrieve the top matching images in the test set, based on CLIP image-text similarity and assign the prompt the highest aesthetic score among the retrieved images. This serves to evaluate the model with actual human annotation in the absence of a trained reward model. \n\nDespite inherent noise in the retrieval process, this evaluation setup yields a clear improvement over the base model. The SPRO-trained model achieves an average score of 6.80, compared to 5.90 for the base captions. For context, the mean aesthetic rating across the AVA dataset is approximately 5.30, indicating that both baselines and our model operate in the higher-quality regime. These results suggest that SPRO can effectively leverage real human ratings as weak supervision, even in the absence of direct reward signals, highlighting the framework’s adaptability in settings where pretrained reward models are unavailable or misaligned with the target objective. \n\n| Model               | Training Method         | Human Aesthetic Rating |\n|--------------------|-------------------------|:----------------------:|\n| Base-Captions      | -                       |          5.90          |\n| GPT-4o             | -                       |          6.20          |\n| Llama-3.2-11B      | -                       |          6.50          |\n| **Llama-3.2-11B**      | **SPRO - Iteration 1**      |          **6.80**          |\n\n**Evaluation using other models:** Previously, we compared our framework with GPT-4o. Following your suggestion, we expanded our evaluation to include advanced reasoning-augmented models, including Qwen-3 and GPT-O3, and benchmarked performance across three key human preference objectives: aesthetic appeal (LAION Score), user preference (PickScore), and engagement. Engagement is evaluated using the EOIG metric, split across three difficulty buckets representing low-, medium-, and high-performing assets, as established in prior work. \n\nAs shown in the table below, SPRO-Prompt consistently outperforms all baselines across all metrics. On the aesthetics objective, SPRO-Prompt achieves 8.60, compared to 7.55 (Qwen-3), 7.57 (GPT-4o), and 7.67 (GPT-O3). For user preference, SPRO-Prompt reaches 22.42, outperforming GPT-O3 (19.47), GPT-4o (22.31), and Qwen-3 (22.10). Most notably, across all three EOIG engagement buckets, SPRO-Prompt attains scores of 83.48 (low), 88.36 (medium), and 90.24 (high), outperforming Qwen-3 (74.12, 83.12, 84.23), GPT-4o (77.70, 87.83, 89.95), and GPT-O3 (75.91, 75.97, 79.51). \n\n| Model               | Aesthetics | User Preference | Engagement                                  |\n|---------------------|:----------:|:---------------:|:-------------------------------------------:|\n| Qwen3               | 7.55       | 22.10           | Low - 74.12, Medium - 83.12, High - 84.23   |\n| GPT-4o              | 7.57       | 22.31           | Low - 77.70, Medium - 87.83, High - 89.95   |\n| GPT-O3              | 7.67       | 19.47           | Low - 75.91, Medium - 75.97, High - 79.51   |\n| LLAMA-3.2-11B       | 7.38       | 21.96           | Low - 83.19, Medium - 87.72, High - 89.32   |\n| **SPRO-Prompt (Ours)** | **8.60**   | **22.42**        | **Low - 83.48, Medium - 88.36, High - 90.24** |\n\nThese results clearly demonstrate that SPRO-Prompt surpasses both general-purpose and advanced reasoning models, including GPT-4o and O3, in aligning generated outputs with human preference objectives. The consistent superiority across diverse and difficult evaluation settings highlights the effectiveness of SPRO’s self-play fine-tuning in learning targeted, preference-aligned prompt strategies that general reasoning alone fails to capture.\n\nMotivated by this insight, we further investigated whether **training a reasoning-based model** via self-play could enhance its ability to generate prompts optimized for aesthetics. Specifically, we employed Qwen-3-7B as the Guider VLM and conducted a single iteration of self-play for the human preference objective of aesthetics. We generated reasoning chains and corresponding prompt candidates across varying temperature settings for improving aesthetics. These prompts were used to generate images using frozen SDXL model, which were then scored using the LAION aesthetic scorer. Based on the resulting scores, we constructed preference pairs and applied Direct Preference Optimization (DPO), jointly conditioning on both the reasoning and prompts. This enabled the model to learn prompt strategies that yield more aesthetically aligned generations. \n\nWe infer the trained model on our test-set: \n\n| Model            | Model Type        | Training Method        | LAION Score (Aesthetics) |\n|------------------|------------------|-------------------------|:-------------------------:|\n| Base-Captions    | -    | -                       | 6.40                      |\n| Qwen-3-7B         | Reasoning        | -                       | 7.55                      |\n| **Qwen-3-7B**     | **Reasoning**    | **SPRO - Iteration 1**  | **8.46**                  |\n| Llava-7B         | Non-Reasoning    | -                       | 6.76                      |\n| **Llava-7B**      | **Non-Reasoning**| **SPRO - Iteration 1**  | **7.22**                  |\n| Llama-3.2-11B    | Non-Reasoning    | -                       | 7.38                      |\n| **Llama-3.2-11B** | **Non-Reasoning**| **SPRO - Iteration 1**  | **8.30**                  |\n| **Llama-3.2-11B** | **Non-Reasoning**| **SPRO - Iteration 3**  | **8.60**                  |\n\nOur experiments reveal that using a reasoning VLM based backbone, such as Qwen-3-7B, in the self-play setup leads to substantially greater improvements in prompt quality compared to both baselines and the corresponding base version. We attribute this to the model’s enhanced capacity for discovering novel and non-trivial prompting strategies that better align with the aesthetic objective.  \n\nAs shown in the table above, we compare performance gains across three models: Llava-7B, LLaMA-3.2-11B, and Qwen-3-7B. While all models benefit from training with SPRO, the magnitude of improvement correlates with model strength. Llava-7B improves from 6.76 to 7.22, LLaMA-3.2-11B improves from 7.38 to 8.30, and Qwen-3-7B improves from 7.55 to 8.46 in a single iteration of training. These results suggest that SPRO is broadly applicable across both reasoning-augmented and standard VLMs, with stronger models showing higher gains and converging more efficiently.  \n\nWe thank the reviewer for the thoughtful feedback and constructive suggestions. These three additional experiments were conducted directly in response to your questions, and we believe the resulting insights significantly strengthened the paper. So, thank you for these insights. \n\nThe added results highlight the flexibility, generalizability, and robustness of the SPRO framework across models, objectives, and evaluation settings. We hope these results help comprehensively address the reviewer’s concerns and reinforce the contributions of our work."}, {"review_id": "UL467samip", "reviewer": "Reviewer_WacU", "summary": "This paper proposes SPRO, a framework that utilizes self-play to enhance text-to-image generation performance. Specifically, compared with previous approaches that utilize self-play for text-generation or image generation only, this paper proposes a joint framework that contains three stages: SPRO-Prompt trains a Guider-VLM through self-play to generate diverse prompts, optimized with reward from image generation score; SPRO-Image fine-tunes diffusion model with self-play data, SPRO-Multimodal combines both components with optimized prompts and a fine-tuned generator for full-capacity alignment. Empirically, the framework SPRO achieves significant improvement in both PickScore and LAION score.", "questions": "N/A", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The primary contribution of this paper is the combination of prompt-space optimization and image-space fine-tuning into the same framework to improve the human preference together. The full framework doesn't rely on any human-annotated data and can be trained all with synthetic data, which can be generally adapted to different human preferences. \n2. Detailed empirical analysis on PickScore and LAION score demonstrates the effectiveness of the proposed approaches. \n3. The paper is well written and easy to follow. \n\nWeaknesses:\n1. The idea of using self-play to improve prompt generation/image generation is not novel enough, as there has been previous work that utilizes self-play for improving LLM, and utilizes self-play for improving the diffusion model. The main contribution seems to lie in combining the two objectives in a single framework. However, it's also not clear in Sec. 2.4 that how SPRO-MM combines the two together (i.e., what's the training objective, how the loss/gradients flow across the two parts, or is it just a combination of the two models during inference).\n2. In experiment results, it seems that most of the improvement comes from the SPRO-Prompt, which is the prompt optimizer, and the SPRO-image and SPRO-MM only help marginally. This observation harms the contribution of the paper, as the main novelty lies in the joint framework. Besides, based on this observation, more reasonable baselines to be compared with are approaches that work on prompt optimization for diffusion models (e.g., Optimizing Prompts for Text-to-Image Generation, Dynamic Prompt Optimizing for Text-to-Image Generation, The Devil is in the Prompts: Retrieval-Augmented Prompt Optimization for Text-to-Video Generation, Reward-Agnostic Prompt Optimization for Text-to-Image Diffusion Models, RePrompt: Reasoning-Augmented Reprompting for Text-to-Image Generation via Reinforcement Learning).", "was_revised": true, "final_justification": "Thank the author for providing additional experiments and comparison with more approaches. These experiments address my concerns, and I'm raising my score to borderline accept.", "rebuttal": "Thank you for your thoughtful review and for highlighting the key strengths of our work. We are glad that you found our integration of prompt-space and image-space optimization within the SPRO framework to be a meaningful contribution. We also appreciate your recognition of SPRO’s ability to train entirely with synthetic data without requiring human annotations, and your positive remarks on the clarity and empirical rigor of the paper.  \n\n**Main Contribution:** We agree and acknowledge in the paper as well, that self-play has been explored in diffusion space, with approaches like SPIN Diffusion [Yuan et al., 2024] demonstrating that models can learn from their own checkpoints. However SPIN-Diffusion and similar methods are limited along three key dimensions. First, it operates exclusively in the image space, relying on self-generated image samples from earlier checkpoints to improve future generations. This approach is computationally expensive and sample-inefficient, requiring large-scale training (e.g., 500k samples) to achieve meaningful improvements. On the other hand, we achieve higher improvements with only 30k samples, this underscores that self-play in prompt space is substantially more sample-efficient than self-play in image space. Second, because it lacks prompt-level reasoning, it cannot perform strategic or semantic exploration, making it less effective at discovering diverse or counterintuitive prompting strategies. We show that because of our novel exploration strategies, we achieve higher scores on all metrics compared to them. Third, the self-play loop is tightly coupled to a specific diffusion model, resulting in poor generalization across diffusion backbones. In contrast, SPRO performs self-play in the prompt space, guided by a reasoning-capable VLM, enabling more efficient, zero-shot transferable, and semantically rich optimization across both models and objectives. \n\n**User Preference (PickScore) on Pick-a-pic dataset**\n\n| **Method**              | **PickScore** |\n|:------------------------|:-------------:|\n| Base Caption            |     20.99     |\n| SPIN-Diffusion          |     22.00     |\n| **SPRO-Prompt (Ours)**  |   **22.42**   |\n| **SPRO-Image (Ours)**   |   **22.09**   |\n| **SPRO-MM (Ours)**      |   **22.42**   |\n\n**Aesthetic Appeal (LAION score) and User Preference (PickScore) on PartiPrompts dataset**\n\n| **Method**              | **Laion Score** | **PickScore** |\n|:------------------------|:---------------:|:-------------:|\n| Base Caption            |      5.67       |     21.89     |\n| SPIN-Diffusion          |      6.05       |     22.31     |\n| **SPRO-Prompt (Ours)**  |   **8.02**      |   **22.84**   |\n| **SPRO-Image (Ours)**   |   **7.01**      |   **22.55**   |\n| **SPRO-MM (Ours)**      |   **8.02**      |   **22.77**   |\n\n\nIn the table above, we compare our SPRO framework against SPIN-Diffusion [Yuan et al., 2024], a prior self-play method operating in diffusion (image) space. SPRO-Prompt, which performs self-play in prompt space using a reasoning-guided Guider VLM, achieves significantly higher scores in both aesthetic appeal (8.02 vs. 6.05) and user preference (22.84 vs. 22.31), while using only 30k training samples compared to SPIN’s 500k. This highlights the superior sample efficiency of prompt-space optimization and the advantages of intelligent prompt exploration over scale.\n\nFurther, SPRO-Image, which fine-tunes the diffusion model using synthetic data generated via SPRO-Prompt, also outperforms SPIN-Diffusion (7.01 vs. 6.05 in LAION score), demonstrating that even tuning on self-play–discovered images leads to better alignment with human preferences. Thus, we show that we are using self-play in a significantly different way compared to how it has been explored and experimented in literature solutions and that we are empirically better in performance.\n\nWhile SPRO-MM, which combines textual and image-space optimization, is an important component of our framework, it is not the central contribution of this work. The core novelty lies in the Guider-VLM, an intelligent vision-language agent trained via self-play to iteratively explore diverse prompt variations. Unlike prior approaches that rely on curated prompt datasets or handcrafted heuristics, the Guider-VLM autonomously discovers novel and often non-intuitive prompting strategies that improve image quality across specific human preference objectives, enabling targeted and generalizable alignment without human-in-the-loop supervision.\n\n**Clarification on SPRO-MM:** SPRO-MM combines prompt-space and image-space optimization using a decoupled training strategy, where the Guider-VLM and the diffusion model are trained independently. This separation allows each component to specialize in its respective objective, prompt generation and image synthesis, before being integrated at inference time. \n\n At this final stage, we employ a late fusion approach, combining the outputs of both models without joint fine-tuning. Specifically, we use the final iteration of the trained Guider-VLM to generate optimized prompts. These prompts are then fed into the diffusion model that was fine-tuned using synthetic image-prompt pairs generated during earlier SPRO-Prompt iterations (up to iteration n-1). The final generated images are evaluated using the appropriate reward model (e.g., LAION Score, PickScore, or EOIG Score). \n\nEmpirically, SPRO-MM achieves an average improvement of 28.64% across all three human preference objectives compared to the base diffusion model and outperforms prior preference optimization baselines that operate solely in image space. This demonstrates that combining the outputs of independently optimized prompt and image modules at inference time yields better alignment with human preferences than optimizing either component alone like in works as Promptist and PAE (for prompt space) and SPIN, DDPO (for image-space). \n\n**Comparison with other prompt-space baselines:** Thanks for pointing out the baselines to compare our method against. In the paper we had already compared two of these baselines: Optimizing Prompts for Text-to-Image Generation (Promptist) and Dynamic Prompt Optimizing for Text-to-Image Generation (Prompt Auto Edit or PAE) for the objective of improving aesthetic appeal of image.\n \n| **Model**             | **Approach** | **LAION Score** |\n|:---------------------:|:------------:|:---------------:|\n| **Prompt Auto Edit**  |   Prompt     |      6.27       |\n| **Promptist**         |   Prompt     |      7.23       |\n| **SPRO-Prompt (Ours)**|   Prompt     |      8.60       |\n| **SPRO-Image (Ours)** |   Image      |      7.48       |\n| **SPRO-MM (Ours)**    |   Both       |      8.42       |\n\nAs shown in above table, all our methods outperform prompt-based baselines such as Prompt Auto Edit (6.27) and Promptist (7.23), with SPRO-Prompt achieving a significantly higher LAION score of 8.60. These baselines typically rely on curated datasets of human refined prompt pairs, often constructed from high-quality image-caption corpora or handcrafted web-scale data. While such approaches can improve performance through reinforcement learning on aesthetic or semantic reward models, they remain limited by their dependence on human-authored strategies. The nonlinear and unintuitive mapping between prompt phrasing and image quality makes it difficult for such methods to generalize beyond human-discoverable patterns. In contrast, our reasoning-driven self-play framework allows unconstrained and diverse exploration of the prompt space, uncovering novel strategies that better align with aesthetic objectives. Notably, even SPRO-Image (7.48) and SPRO-MM (8.42) outperform these baselines without relying on any form of human annotation. \n\nWe appreciate the reviewer’s suggestion to compare with recent baselines. However, we would like to clarify that these works were released around or after the paper submission deadline. Nevertheless, we have made an effort to examine them in the context of our work.  \n\nWe evaluated our model in a reward-agnostic setting using the benchmark from Reward-Agnostic Prompt Optimization for Text-to-Image Diffusion Models (RATTPO). This baseline is trained to optimize prompts for a different reward function, ImageReward, aimed at aligning with user preference. Although our model is not explicitly trained on this reward, we assess whether it generalizes in a zero-shot manner to new reward models and datasets. \n\nThe evaluation was conducted on the LexicaDB dataset, following the RATTPO setup. Since the original test set is not publicly released and the paper reports performance on 160 randomly sampled image–caption pairs, we sampled 160 random instances from LexicaDB as well. For each example, we used the base image and prompt as input to our user-preference-optimized SPRO-Prompt model to generate an improved prompt. The final generations were scored using the ImageReward metric. \n\n| **Model**        | **Approach** | **Initial ImageReward score**       | **Optimised ImageReward score**     | \n|:----------------:|:------------:|:-----------------:|:-------------------:|\n| RATTPO             |   Prompt     |  0.049±0.143      |     1.132 ±0.049    |  \n| **SPRO-Prompt(ours)**     |   **Prompt**     |     **0.1588**        |      **1.439**               |  \n\nAs shown in the table, SPRO-Prompt achieves a significantly higher ImageReward score than both the base prompt (1.439 vs. 0.1588) and the RATTPO baseline (1.439 vs. 1.132). These results demonstrate that our approach generalizes effectively to novel reward functions, highlighting the robustness of reasoning-guided self-play in optimizing user preference even under unseen evaluation criteria.\n\nWe thank the reviewer for their constructive feedback. We have incorporated additional comparisons with strong baselines, which we believe meaningfully strengthen the paper and help address the concerns raised."}, {"review_id": "r4RHX7Oq1k", "reviewer": "Reviewer_MSs3", "summary": "The paper introduces SPRO, an annotation-free self-play framework to improve text-to-image generation. The author tried three approaches:\n1. SRPO-prompt, which trains a VLM to generate better prompts, with DPO.\n2. SPRO-Image, which only finetunes the diffusion model.\n3. SRPO-MM, which combines the two.\n\nThe reward signals are PickScore, LAION-aesthetics, and EngageNet. Experiments show that all three objectives improve a lot after the training. Also, SRPO-prompt generalizes to different diffusion models.", "questions": "1. From the experiments, it seems that SRPO-image has worse performance compared to SRPO-prompt. I look closely and find the authors is finetuning the diffusion model with the base prompt and image pairs. I am wondering what if we freeze the VLM and finetune the diffusion model with the SRPO-prompt generated prompts? Will that be better?\n\n2. It would be great to add scores on more benchmarks using different metrics, to show that this approach is not reward-hacking.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The motivation is clear, and the experiments are comprehensive.\n2. The paper is well-written and easy to read.\n3. The gain of the prompt optimization is huge, showing the effectiveness of the approach.\n\nWeaknesses:\n1. The model is trained and tested with the same set of rewards. It would be interesting to do more experiments on the rewards to make sure that the model is not reward hacking. For example, evaluate with different metrics. (For example, VQA-based approaches for text-image alignment). Human preference experiments partly address this issue.\n2. The SRPO-image is worse than SRPO-prompt, and the SRPO-MM model is mostly on par with the SRPO-prompt phase. It seems only the first SRPO-prompt stage is effective. More analysis is needed on this issue.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the encouraging feedback and recognition of our motivation, the clarity of the paper, and the comprehensive evaluations. We appreciate the reviewer’s acknowledgment of the significant improvements achieved through the SPRO-prompt approach and the generalization capabilities across different diffusion models.  \n\n**Concern about Reward Hacking and Evaluation with Diverse Metrics:** In our main experiments, the same reward model is used for both training supervision and test-time evaluation. While the Guider VLM in SPRO-Prompt is not directly exposed to the reward model itself, it receives preference objectives in the form of ranked preference pairs generated using the reward signal. This indirect conditioning could, in principle, lead to overfitting or reward hacking, where the model learns to exploit patterns specific to a particular reward function rather than capturing the broader notion of human preference. \n\nTo assess whether our model exhibits such behavior, we conducted a set of generalization experiments aimed at evaluating robustness beyond the training reward model. Specifically, we tested the user-preference-optimized SPRO-Prompt model across four settings that involved unseen reward models (i.e., reward functions not used during training) and different evaluation datasets. These experiments were designed to probe whether the model's improvements persist when evaluated under alternative preference signals and data distributions, thereby validating that the learned prompting strategies align with human preference objectives more broadly rather than exploiting nuances of a single reward model. \n\n**Zero-shot Generalization across Reward Models:** To evaluate the robustness of our approach and mitigate concerns of reward hacking, we assess SPRO-Prompt across a broader set of reward metrics that were not used during training. These evaluations span multiple objectives, reward models, and datasets, and are designed to test whether the model generalizes to the underlying preference objective, rather than overfitting to a specific reward function. \n\nFor the aesthetics objective, we report performance using three distinct metrics: the LAION Aesthetic Score (used to construct preference pairs during training), VILA, and improved aesthetic scorer from prior work. While all three metrics aim to capture aesthetic appeal, VILA (VILA: Learning Image Aesthetics from User Comments with Vision-Language Pretraining) and improved aesthetic scorer (Improved-aesthetic-predictor by Christoph Schuhmann) were not used during training and are included exclusively for zero-shot evaluation.  \n\n**Table:** Aesthetic Appeal on **Flikr Dataset**\n\n| **Method**      | **LAION Score** | **VILA** | **Improved Aesthetic Scorer** | **CLIP** |\n|:---------------:|:---------------:|:--------:|:------------------------------:|:--------:|\n| Base Prompt     |      7.38       |  60.74   |             6.30              |  83.21   |\n| **SPRO-Prompt (Ours)** |   **8.60**      | **65.16**|           **7.47**            |  79.50   |\n\n\nWe find that SPRO-prompt consistently outperforms the base prompt across all metrics, demonstrating that SPRO-Prompt does not rely on memorization or overfitting to the LAION scorer but rather learns prompt strategies that generalize to the broader aesthetic preference objective. Similarly, being consistent in CLIP similarity between base images and generated images demonstrates that text-image alignment is maintained in the process. As shown in the table below, SPRO-Prompt consistently outperforms the base prompt across all aesthetic evaluation metrics on the Flickr dataset, including LAION (8.60 vs. 7.38), VILA (65.16 vs. 60.74), and the improved aesthetic scorer (7.47 vs. 6.30). This indicates that SPRO-Prompt does not overfit to the LAION reward model used during training but instead learns prompt strategies that generalize to the broader aesthetic preference objective. \n\nFor the user preference objective, our model was originally trained using PickScore as the reward signal to construct preference pairs. To evaluate its generalization beyond the training signal, we tested the trained SPRO-Prompt model using ImageReward, an alternative reward model that also aims to assess user preference alignment in generated images. \n\n**Table:** User Preference on **Pick-a-pic Dataset**\n\n | **Method**       | **PickScore** | **ImageReward** | **CLIP Score** |\n|:----------------:|:-------------:|:---------------:|:--------------:|\n| Base prompt      |     20.99     |      0.65       |     87.76      |\n| **SPRO-prompt (Ours)**     |     **22.42**     |      **1.02**       |     **89.32**      |\n\nWe conducted this evaluation on the Pick-a-Pic dataset, comparing base prompts and SPRO-optimized prompts across three metrics. As shown in the table below, SPRO-Prompt achieves higher scores than the base prompt across all metrics: PickScore (22.42 vs. 20.99), ImageReward (1.02 vs. 0.65), and CLIP similarity (89.32 vs. 87.76). Notably, the model exhibits strong performance on ImageReward despite never being trained with it, indicating robust zero-shot generalization across reward functions that share the same underlying preference objective. \n \nThese results demonstrate that SPRO-Prompt is not tightly coupled to the specifics of PickScore, but instead learns generalized prompting strategies aligned with human notions of preference quality. \n\n**Generalization across multiple datasets:** To further assess the generalizability of our approach, we evaluate the trained models on additional datasets containing diverse base prompts and images. This allows us to examine whether the learned prompting strategies can adapt to varying input distributions while consistently optimizing for the target preference objectives. For aesthetics and user preference objective we also tried our proposed method on a different dataset that is - Parti prompts dataset and show consistent improvements over the base prompt across all metrics LAION, VILA, PickScore, Improved aesthetic scorer, PickScore and ImageReward. \n\n**Table:** Aesthetic Appeal on **PartiPrompts Dataset**\n\n| **Method**       | **LAION Score** | **VILA Score** | **Improved Aesthetic Scorer** |\n|:----------------:|:---------------:|:--------------:|:------------------------------:|\n| Base prompt      |      6.16       |     55.15      |             5.60              |\n| **SPRO-prompt (Ours)**     |      **8.01**       |     **64.81**      |             **6.78**              |\n\nAs shown in above table for the aesthetic appeal objective, we evaluate SPRO-Prompt on Parti Prompts dataset using three independent reward models: LAION Score, VILA, and an improved aesthetic scorer. As shown in the table below, SPRO-Prompt consistently outperforms the base prompt across all metrics, improving from 6.16 to 8.01 on LAION Score, from 55.15 to 64.81 on VILA, and from 5.60 to 6.78 on the improved scorer. \n\n**Table:** User Preference on **PartiPrompts dataset**\n\n| **Method**       | **PickScore** |\n|:----------------:|:-------------:|\n| Base prompt      |     21.89     |\n| **SPRO-prompt (Ours)**      |     **22.84**     |\n\nFor the user preference objective, we report PickScore on the same dataset. SPRO-Prompt again yields a significant gain, improving from 21.89 to 22.84. \n\nThese results demonstrate that SPRO generalizes effectively across datasets and reward models, reinforcing its robustness and applicability to diverse prompt-image distributions.\n\n**Finetuning VLM with SPRO generated prompts:** Following your suggestion, we fine-tuned the diffusion model using prompt–image pairs generated by SPRO-Prompt. Within the limited time available, we trained the model using only the top 30k synthetic samples collected across SPRO-Prompt iterations. The resulting model outperforms the original SPRO-Image variant, indicating that the diffusion model benefits significantly from training with detailed, self-play–discovered prompts compared to base captions, which often lack alignment with the generated image content. While the original goal of SPRO-Image was to explore how well a diffusion model could be optimized using basic prompts alone, your suggestion opened up a promising new direction. We are grateful for the idea and plan to investigate it more thoroughly.\n\n| **Method**                         | **LAION Score** |\n|:----------------------------------:|:---------------:|\n| Base caption                       |      6.40       |\n| SPRO-Prompt                        |      8.60       |\n| SPRO-Image                         |      7.42       |\n| **SPRO-Image with Generated prompts**  |      **8.29**       |\n\n**Comparison between SPRO Prompt, SPRO Image and MM:** Empirically, we observe that SPRO-Image, when fine-tuned on base captions and synthetic images generated through SPRO-Prompt, shows a moderate drop in performance compared to SPRO-Prompt alone. This suggests that relying only on image-space finetuning limits the ability to discover preference-aligned strategies. In contrast, SPRO-Prompt consistently achieves higher scores across all objectives, highlighting the effectiveness of prompt-space exploration in uncovering strategies that are difficult to encode directly into diffusion models. SPRO-MM, which combines the outputs of independently optimized prompt and image modules at inference time, outperforms prior methods such as Promptist and PAE in prompt-space, and SPIN and DDPO in image-space. These findings indicate that integrating both modules yields stronger alignment with human preferences than optimizing either component individually.\n\nWe are truly thankful for your insightful reviews. We hope the additional experiments adequately address your queries."}, {"review_id": "AJKHdBvRUd", "reviewer": "Reviewer_uVAj", "summary": "The given work proposes the Self Play Reward Optimization technique, for objective driven prompt optimization. The authors decompose it into stages. The first stage prompt optimization that iteratively fine tunes a vision-language model with direct preference optimization (DPO for better reasoning chains and prompts. In the next stage, (SPRO-image), that curated “best-of” bundle (human preferences, seed prompt, and winning pics) finetunes the diffusion model itself, teaching it to hit the target style without extra coaching. Finally, the SR-MM Stage, combines these two (unfreezing operations).  They present their result on various backbones on the Pick-A-Pic, PartiPrompts and Flickr dataset.", "questions": "* Is there an intuition on susceptibility to reward hacking with this synthetic data generation?\n* How will this work with multiple (potentially conflicting) objectives?\n* Could you share more details about the human eval experiment ? Specifically, to have a better intuition on the scale/statistical significance of it.", "limitations": "Yes. While not directly in scope, I would encourage authors to consider implications of these self optimizing loops directly on (hackable) reward models without human oversight.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n* Paper is clearly written and easy to follow.\n* Self play mitigates need for human in the loop\n* Approach can be generalized across diffusion models.\n\nWeaknesses:\n* Limited niche evaluation. The generalization/success metric is measured on a set of 514 Flickr30k images and some other splits. The human eval study information is also limited.\n* Single objective optimization", "was_revised": true, "final_justification": "Updating my score to \"Accept\", the authors have adequately answered my queries on generalization and multi objective optimization.", "rebuttal": "Thank you for the thoughtful and constructive review. We appreciate your recognition of the clarity of the paper, the value of self-play in removing human-in-the-loop supervision, and the generalizability of our approach across diffusion models. \n\n**Generalization Across Datasets:** In the main paper, we evaluate SPRO across three human preference objectives using distinct test sets: a held-out set of 514 images from the Flickr30k dataset for aesthetic appeal, the EngagingImageNet test set for engagement, and the Pick-a-Pic test set for user preference. To further assess generalizability, we evaluate SPRO on the PartiPrompts dataset, which features diverse base prompts and image distributions. The results are included below:\n\n**Table**: Aesthetic appeal tested on *Parti Prompts* dataset\n\n| Prompt Type           | LAION Score |\n|------------------------|:-----------:|\n| Base Prompt            | 6.16        |\n| **SPRO-Prompt (Ours)** | **8.01**    |\n\n**Table**: User preference (PickScore) evaluated on *Parti Prompts* dataset\n\n| Prompt Type           | PickScore |\n|------------------------|:---------:|\n| Base Prompt            | 21.89     |\n| **SPRO-Prompt (Ours)** | **22.84** |\n\nSPRO consistently outperforms base prompts across both aesthetic appeal and user preference objectives, improving LAION score from 6.16 to 8.01 and PickScore from 21.89 to 22.84. These results demonstrate the robustness and adaptability of the learned prompting strategies to unseen datasets.\n\n**Multi-objective Optimization**: In the main paper, we focus on optimizing for a single human preference objective. However, based on your suggestion, we conducted some additional experiments to see whether SPRO can support multi-objective optimization. \n\n(1) In this preliminary setup, we considered two reward signals: (a) LAION Aesthetic Score, which serves as the primary training signal, and (b) CLIP image–image similarity, which we previously used only as a filtering constraint. During self-play training, we conditioned the Guider-VLM to optimize for both objectives simultaneously. \n\n**Table**: CLIP and LAION score optimization\n\n| Prompt Type           | LAION Score | CLIP Score |\n|------------------------|:-----------:|:----------:|\n| Base Prompt            | 6.40        | 83.19      |\n| **SPRO-Prompt (Ours)** | **7.90**    | **83.28**  |\n\nAfter one iteration of self-play, we observed that the LAION score improved substantially, indicating successful aesthetic optimization, while the CLIP similarity remained relatively stable compared to the base captions. This suggests that SPRO can accommodate multiple objectives in training, balancing optimization of a primary human preference metric while preserving semantic consistency via CLIP similarity score. \n\n(2) To further evaluate SPRO’s capacity for multi-objective optimization, we designed an experiment using two orthogonal human preference objectives: aesthetic appeal (measured by LAION Score) and user preference (measured by PickScore). Unlike CLIP similarity, which serves as a sanity metric, these two reward functions represent distinct and potentially unaligned preferences, making the setup more challenging. We generate preference pairs conditioned jointly on LAION and PickScore, retaining only those where both improve. These filtered pairs are used to train the Guider-VLM for one iteration of DPO, conditioning it to generate reasoning chains and prompts that optimize both objectives.\n\n**Table**: Multi-reward optimization \n\n| Prompt Type           | LAION Score | PickScore |\n|------------------------|:-----------:|:---------:|\n| Base Prompt            | 6.40        | 22.69     |\n| **SPRO-Prompt (Ours)** | **7.66**    | **22.96** |\n\nAs shown in the table above, SPRO-Prompt improved performance across both metrics compared to the base prompt: LAION Score increased from 6.40 to 7.66, and PickScore improved from 22.69 to 22.96. These results demonstrate that SPRO can effectively learn prompting strategies that align with multiple human preference objectives, reinforcing the flexibility of our framework in multi-reward settings. \n\n(3) We further investigated whether SPRO can handle scenarios involving conflicting reward objectives, where improving alignment with one objective may require compromising on another. Although no existing reward models explicitly capture such inverse relationships, we constructed a composite pseudo-reward function to simulate this setting: \n\n$$\n\\text{Conflicting Score} = 2 \\times \\text{Aesthetic Score (LAION)} - 0.5 \\times \\text{User Preference (PickScore)}\n$$\n\nThis formulation encodes a preference for images with higher aesthetic appeal and lower user preference scores, reflecting a controlled trade-off that is occasionally observed in real-world settings where certain stylistic or artistic choices may be visually appealing but less preferred by a broader audience. We curated preference pairs based on improvements in this conflicting score and trained the Guider-VLM for one iteration using SPRO-Prompt. \n\n**Objective**: Optimize for conflicting rewards  \n\n| Prompt Type              | LAION Score | PickScore | Conflicting Score |\n|--------------------------|:-----------:|:---------:|:-----------------:|\n| Base Prompt              | 6.40        | 22.69     | 1.46              |\n| **SPRO Prompt (Ours)**   | **7.62**    | **21.70** | **4.39**          |\n\nAs shown in the table above, the trained model achieves the intended effect: the LAION aesthetic score increases from 6.40 to 7.62, while the PickScore decreases from 22.69 to 21.70. This demonstrates that SPRO can be effectively extended to optimize for non-aligned or conflicting objectives, making it applicable in complex human preference modeling scenarios. \n\nAcross the three experiments, our findings demonstrate that the SPRO framework effectively supports multi-reward optimization, including both orthogonal and conflicting objective settings.\n\n**Zero-shot Generalization across Reward Models:**  In our main experiments, the same reward model is used for both training supervision and test-time evaluation. While the Guider VLM in SPRO-Prompt is not directly exposed to the reward model, it receives preference supervision via ranked preference pairs generated from the reward signal. This indirect conditioning could potentially lead to overfitting or reward hacking, where the model exploits artifacts of a specific reward function rather than learning the broader notion of human preference. \n\nBased on your suggestions, to evaluate the robustness of our approach and address concerns of reward hacking, we assess SPRO-Prompt across a broader set of reward metrics not used during training. These evaluations span multiple objectives, reward models, and datasets to test whether the model generalizes to the true preference objective rather than a specific scoring function.\n\nFor the aesthetics objective, we report performance using three distinct metrics: the LAION Aesthetic Score (used to construct preference pairs during training), VILA, and improved aesthetic scorer from prior work. While all three metrics aim to capture aesthetic appeal, VILA (VILA: Learning Image Aesthetics from User Comments with Vision-Language Pretraining) and improved aesthetic scorer (Improved-aesthetic-predictor by Christoph Schuhmann) were not used during training and are included exclusively for zero-shot evaluation.  \n\n**Objective**: Aesthetic Appeal (Flikr test set) \n\n| Prompt Type              | LAION Score | VILA (Improved Aesthetic) | CLIP Score |\n|--------------------------|:-----------:|:--------------------------:|:----------:|\n| Base Prompt              | 7.38        | 60.74                      | 83.21      |\n| **SPRO-Prompt (Ours)**   | **8.60**    | **65.16**                  | **79.50**  |\n\nAs shown in the table below, SPRO-Prompt consistently outperforms the base prompt across all aesthetic evaluation metrics on the Flickr dataset, including LAION (8.60 vs. 7.38), VILA (65.16 vs. 60.74), and the improved aesthetic scorer (7.47 vs. 6.30). This indicates that SPRO-Prompt does not overfit to the LAION reward model used during training but instead learns prompt strategies that generalize to the broader aesthetic preference objective. \n\nFor the user preference objective, SPRO-Prompt was trained using PickScore to construct preference pairs. To evaluate generalization beyond this signal, we tested the model on the Pick-a-Pic dataset using ImageReward, another reward model that targets user preference. \n\n**Objective**: User Preference  on Pick-a-pic dataset\n\n| Prompt Type              | PickScore | Image Reward | CLIP Score |\n|--------------------------|:---------:|:------------:|:----------:|\n| Base Prompt              | 20.99     | 0.65         | 87.76      |\n| **SPRO-Prompt (Ours)**   | **22.42** | **1.02**     | **89.32**  |\n\nAs shown in the table, SPRO-Prompt achieves higher scores than the base prompt across all metrics: PickScore (22.42 compared to 20.99), ImageReward (1.02 compared to 0.65), and CLIP similarity (89.32 compared to 87.76). These results indicate strong zero-shot generalization to alternative reward functions aligned with the same objective.\n\n**Human Study Details:** We conducted a human study with 35 participants from diverse backgrounds; each saw 20 random prompts, with 3 images (SPRO, Base, GPT-4o) per prompt for comparison. They picked one image that they preferred the best among the three.\n\n| Comparison                     | Preferred Candidate     | Win Rate (%) |\n|--------------------------------|--------------------------|:------------:|\n| Base caption vs GPT-4o         | Base caption             | 53.33        |\n| **SPRO-Prompt** vs Base caption| **SPRO-Prompt (Ours)**   | **65.38**    |\n| **SPRO-Prompt** vs GPT-4o      | **SPRO-Prompt (Ours)**   | **68.90**    |\n\nThank you for your constructive questions. We believe the additional results address your concerns and significantly strengthen the paper’s contributions."}]

Author rebuttal:

Meta-review: The submission introduces a framework for aligning image generation models with diverse human preferences without extensive human annotations. The idea is to leverage the reasoning capabilities of VLMs in a self-play formulation to iteratively improve prompt generation and the final synthesized images. The method, Self-Play Reward Optimization (SPRO), outperforms commercial baselines and other techniques and provides a sensible and efficient way for aligning models. 

All reviewers acknowledged the importance of the problem and the effectiveness of the method. The rebuttal was considered and the reviewers led to two borderline and two accept ratings. The AC also agrees that the paper has merit and should be accepted. The questions the reviewers raised clarified several aspects where the submission can benefit from improvements. The authors are strongly recommended to incorporate them into the camera-ready version.
Decision: Accept (poster)