Policy Optimized Text-to-Image Pipeline Design
Authors: Uri Gadot
Rinon Gal
Yftah Ziser
Gal Chechik
Shie Mannor
Venue: NeurIPS 2025 poster ()
Keywords: ComfyUI; Text-to-Image workflow design; Text-to-Image generation
Primary area: applications

Abstract: Text-to-image generation has evolved beyond single monolithic models to complex multi-component pipelines that combine various enhancement tools. While these pipelines significantly improve image quality, their effective design requires substantial expertise. Recent approaches automating this process through large language models (LLMs) have shown promise but suffer from two critical limitations: extensive computational requirements from generating images with hundreds of predefined pipelines, and poor generalization beyond memorized training examples.
    We introduce a novel reinforcement learning-based framework that addresses these inefficiencies. Our approach first trains an ensemble of reward models capable of predicting image quality scores directly from prompt-workflow combinations, eliminating the need for costly image generation during training. We then implement a two-phase training strategy: initial workflow prediction training followed by GRPO-based optimization that guides the model toward higher-performing regions of the workflow space. Additionally, we incorporate a classifier-free guidance based enhancement technique that extrapolates along the path between the initial and GRPO-tuned models, further improving output quality.
    We validate our approach through a set of comparisons, showing that it can successfully create new flows with greater diversity and lead to superior image quality compared to existing baselines.

Policy Optimized Text-to-Image Pipeline Design
Uri Gadot
Rinon Gal 2
Yftah Ziser 2,3
Gal Chechik
Shie Mannor 1.2
Techhioo NNVIDIA Research
University of Groningen
Abstract
Text-to-image generation has evolved beyond single monolithic models to complex multi-component pipelines. These combine fine-tuned generators, adapters, upscal- ing blocks and even editing steps, leading to significant improvements in image quality. However, their effective design requires substantial expertise. Recent approaches have shown promise in automating this process through large language models (LLMs), but they suffer from two critical limitations: extensive computa- tional requirements from generating images with hundreds of predefined pipelines, and poor generalization beyond memorized training examples. We introduce a novel reinforcement learning-basel framework that addresses these inefficiencies. Our approach first trains an ensemble of reward models capable of predicting image quality scores directly from prompt-workfion combinations, eliminating the need for costly image generation during training. We then implement a two-phase training strategy: initial workflow vocabulary training followed by GRPO-based op- timization that guides the model toward higher-performing regions of the workflow space, Additionally, we incorporate classifier-fre free guidance based enhancement technique that extrapolates along the path between the initial and GRPO-tuned models, further improving output quality. We validate our approach through set of comparisons, showing that it can successfully create new flows with greater diversity and lead to superior image quality compared existing baselines.
Introduction
Recent advancements in generative AI have significantly improved the quality and diversity of text-to-image generation. Early models relied monolithic architectures, where single neural network directly translated textual prompts into visual outputs. However, as the field matured, became clear that combining multiple specialized components- models, super-resolution modules, or specialized embeddings, into more sophisticated workflows leads superior image quality and greater creative control [5, 40, 63). This shift from monolithic models to modular workflows has been supported by user-friendly platforms such as ComfyUl', popular open-source tool that allows users visually construct complex generative pipelines through interconnected nodes represented in JSON format. ComfyUl has rapidly gained popularity due to its intuitive node-based interface, enabling users assemble diverse generative models (e.g., Stable Diffusion, ControlNet, LoRAs) into flexible workflows tailored to specific Despite accessibility, designing effective workflows remains challenging due to the vast space of possible component combinations and their Noompt-dependen effectiveness. Consequently, crafting high-quality workflows typically requires considerable expertise and manual experimentation.
fine-tuned diffusion
tasks.
To address this challenge, recent work introduced ComfyGen [16], which uses large language models (LLMs) automate the construction of prompt-adaptive workflows within Co However, key limitation of Comfy was its inability to generate genuinely novel workflow structures. At its core, their approach required synthesizing images using an extensive collection of pre-defined workflows
39th Conference on Neural Information Processing Systems (NeurIPS 2025).
'https://www. .rg/

---

and prompts, an expensive process limiting their training set's size. Constrained by this small set, their upproach essentially learned a classifier over existing flows rather than synthesizing original graph topologies or selecting novel model combinations. This limitation significantly constrains the potential creativity and adaptability of automated workflow generation systems and, as we later show -may also limit their downstream performance.
In parallel, reinforcement learning (RL) has emerged as a powerful paradigm for fine-tuning large language models (LLMs), enabling them to optimize their outputs directly based on reward signals derived from human preferences or other evaluative metrics. Techniques such as Reinforcement Learning from Human Feedback (RLHF) have demonstrated remarkable success in aligning model behaviors with human expectations by iteratively refining model parameters based on explicit reward feedback. Furthermore, recent developments like Group Relative Policy Optimization (GRPO) introduced memory-efficiet RI. algorithms capable of optimizing policies without separate value functions, making them particularly suitable for complex sequential decision- tasks. Building on these advancements, we propose FlowRL., a novel extension that integrates reinforcement learning into the workflow prediction framework to overcome its originality limitations. Specifically, we formulate workflow generation as an RL problem where an LLM-based policy sequentially constructs workflow graphs by selecting nodes and connections conditioned on textual prompts. To efficiently guide this process without incurring prohibitive computational costs associated with direct image generation for each candidate workflow during training. we introduce surrogate reward model trained to predict image quality scores directly from prompts and workflow structures.
Finally, we adopt GRPO combined with per-token reward attribution mechanisms to provide granular feedback during policy updates. This affords our RI. agent greater precision identifying decisions within a generated workflow that contribute positively or negatively toward overall image quality.
In summary, our contributions are as follows
We introduce ComfyGen-RL, the first RL-based approach for generating genuinely novel ComfyUl workflows tailored align with human preference feedback.
We propose surrogate numan-preferene reward model enabling efficient RL training without computationally expensive image generations.
We integrate GRPO with per-token reward attribution for stable and memory-efficient policy optimization.
Through these innovations, FlowRL significantly advances automated workflow generation capabili-
ties, enabling richer creativity and greater adaptability in eox-to-imago synthesis pipelines.
2
Related Work
Workflow Generation
recent line of research explores the use of compound systems, where multiple models or modules are chained together, often yielding superior performance compared to isolated models. These multi- component systems have been applied across fields ranging from programming challenges [1] and olympiad-level mathematics [53] to medical diagnostics [38] and video generation [64]. However, building compound systems presents significant challenges. Models must be chosen not only for their individual strengths, but also for their ability to complement each other. Moreover, the parameters of the different components should selected with the entire system mind. To address these difficulties, recent work has explored meta-optimization frameworks, where the structure and parameters of entire pipelines are automatically tuned for downstream performance [28]. Others have adopted graph-based architectures allowing dynamic reconfiguration of component interactions [68].
In the realm of text-to-image generation, recent work explores the use of pipelines using agentic systems [67, 61, 23], genetic algorithms [51] or by fine-tuning LLMs using large flow datasets tagged with human preference scores [16]. Although the human preference-basee
framework has shown promising results, it relies on creating and ranking images using large sets of flows. This, in turn, leads to challenges in effectively scaling the dataset and to a lack of ability to synthesize unseen flows at inference time. Our work aims to address this challenge by leveraging a policy-optimization approach for more effective exploration of the flow parameter space, coupled with a surrogate reward function which avoids the need to generate and rank a large set of images.

---

Fine-Tuning LLMs with RL: Reinforcement learning (RL) has become increasingly central to the development large language models (LLMs), playing key role aligning model outputs with user preferences and enhancing task-specific capabilities. A prominent example is Reinforcement Learning from Human Feedback (RLHF) [39], which fine-tunes models using reward signals derived from human preferences to better align with communicative goals and social norms [8, 24]. Beyond alignment, RL has shown promise in improving LLMs' performance on domains requiring precise reasoning. such as mathematics [54, 56, 34] and code generation [30, 33]. Recently, [48] proposed Group Relative Policy Optimization (GRPO) as scalable alternative to Proximal Policy Optimization (PPO). GRPO removes the need for critic model by optimizing contrastive objective based intra-group ranking. yielding better sample efficiency, improved stability, and reduced computational complexity [36, 46]. TRPPO-traince LLMs demonstrated state-of-the performance in mathematical problem solving and code generation, highlighting its effectiveness on tasks requiring structured reasoning and adherence correctness [48].
Improving Text-to-Image Generation Quality The rapid adoption of text-to-image models [45, 37, 44, 13, 41] has led to many research efforts focused on improving their image quality and better matching human preferences. Some works focus on inference-time modifications, either optimizing noise seeds towards better behaving regions of the diffusion space [14, 43] applying self-guidance and frequency-based modulations [21, 49, 35] to the generated features.
More commonly, models tuned to provide better quality outputs. This is often done through carefully selected high-quality datasets or better captioning methods [9, 3, 47]. Another approach uses reward models [29, 59, 60, 31] to guide the generation process. These reward models can be used with reinforcement learning [4, 11, 15, 66], or through direct optimization [6, 42, 55].
Finally, recent methods explore the use of LLMs to improve rext-to-imag" generation [62]. commonly by using them construct workflows featuring multiple models or chained editing tools [67. 51, 16). Our work similarly uses LLMs to construct workflows, but better aligns them to human preferences through the use of reward models coupled with reinforcement-lemmingg feedback mechanism.
Stage 1: SFT
Prompt
x500k
Flow
pairs
Prompt
Flow
Data
Stage 2.1: Surrogate Reward Training
Prompt
Score
Prompt
Modern BERT
Flow
x100k
Score
triplets
Flow
Data
Stage 2.2: Policy-Optimizztion
Flow
Score
Modern BERT
Prompt
LLM
Prompt
x5000
Flow
Score
Data
Figure 1: Pipeline overview. Step 1: Finetune LLM for general flow generation (SFT, 500K prompt- flow pairs). Step 2.1: Train reward model (100K prompt-flow-scov triplets). Step 2.2: Optimize for quality using GRPO. O learning, & frozen.
3

---

3 Methodology
Our goal is to enable efficient training of a human- human-preeereen preference based, prompt-to-workion prediction system. Ideally, this system should be able to innovate and produce novel, unseen flows. Prior work struggled with this aspect, primarily due to their reliance on scoring images generated with large set of fixed flows, whose parameters were sampled uniformly from predefined set of options. To overcome this hurdle, we propose a two-phase training strategy. the first, we pre-train on large set of un-scored flows. This avoids the need to generate and score images, allowing us to use a much larger set to teach the LLM the structure of flows and the available components. Then, we perform predictor models jointly with recent
second tuning stage, where leverage numan-preferenn seffforeemen-lcarrnin ideas (GRPO [48]) to drive the model towards petter-performen subsets of the flow space. As training progresses, more samples are drawn from these regions, and hence, less computation is wasted on inefficient exploration.
However, generating and scoring images during LLM training is itself costly process, which requires an order of a minute for every training step. Hence, we draw on ideas from the autonomous driving literature, where costly simulations are often replaced by faster predictors trained to replicate simulation outputs [2, 25, 26). Here, we apply this idea by learning surrogate reward models that predict the final image score directly from the prompt and workflow pair. Notably, prior work has observed that such surrogates are susceptible to findings that ensembles can mitigate reward hacking [7, 65]. we train an ensemble of such models and use their variance as a measure of uncertainty, allowing us to filter out samples that optimize for any individual surrogate reward model. Below present these core components in greater detail and provide an overview of additional design choices or components that allow us to increase efficiency further refine our results. An overview of our training pipeline is shown in Figure 1.
solutions [17, 50, 58). Motivated by
3.1 Training Data
To train our model we use the flow and prompt dataset of ComfyGen [16]. This set contains 33 human-created flows that define an overall graph structure, further augmented by randomly sampling novel parameter choices for existing blocks such as different base models, differnet LoRAs, diffusion samplers even the number of steps and guidance scale. Since we do not need to score images for our first stage, we can apply more extensive augmentations and create 2, 000 variants from each baseline flow structure (compared with ComfyGen's 's 100). The set also contains 10000 prompts taken from the generation sharing website CivitAl.com. We keep the 500 prompts used to test ComfyGen as holdout, and train using the rest.
3.2 Stage 1: Supervised Fine-Tuning on Flow Dataset
The first stage involves supervised fine-tuning (SFT) an LLM on a dataset of prompt-flow pairs without explicit score labels. At this stage, our goal is to teach the LLM the appropriate vocabulary and flow structure while maintaining output diversity. Our flow dataset DSFT consists of pairs (p,, fa) where p; represents randomly sampled prompt and fs represents randomly sampled flow. We tune the model to take the sampled prompt p: and return its matching flow fs. The full LLM query is shown in the supplementary. After fine-tuning, we evaluate the model's perplexity DSrT. achieving a score of 1.9, which reflects strong alignment with the encoded workflows structural patterns.
Efficient Flow Representation Scheme While prior work [16] directly predicts Comfy JSON representations, we note that these JSONs typically contain thousands of tokens, leading long generation times and increasing memory requirements. An inspection of the tokenized ISONs shows that many tokens are wasted on maintaining the ISON format (e.g., brackets quotation marks) or on breaking down model or component names. Hence, to improve training efficiency and reduce token usage, we propose modify the encoding scheme, using a novel structured representation that captures essential components while reducing token count. Additionally, we introduce specialized tokens to represent key elements of the flow. (e.g. tokens for ComfyUl node names or for model choices). An example of the difference between the two tokenization methods is outlined in Figure 2.
This new encoding scheme yields significant practical advantages resulting in substantial improve- ments in both computational efficiency and memory utilization. Quantitatively, the 86.7% 7% reduction

---

Expt
NiS
Beal
Switter
pellan
MONOTS
600
class
Type
Checkoctitt
Loaden
Strple
Froquency
Lood
Check
polint
(a)
L0K8
2006
J000 4830
Flow length (# tokens)
Our tolennization
Checkppui www.ppiiin..oeeiiii..... EApt mumm 50K. Li0ttctte
vitbel feterrsrr
Original tokenization method
(c)
Figure 2: An example of single ComfyUl node tokenized. (a) displays the original JSON input as tokenized by the standard Llama tokenizer. (b) shows our custom encoding, with introducing additional tokens explicitly represent relevant components within workflow. Colored segment corresponds to a different token. (c) histogram of flows length (in token) of all training-set
in the average token length (1500 - 200 tokens per workflow) enabled 16x batch size increase (2 32 samples/batch) during the first-stage stage training. Uliimtely reaching 3x time improvement over the original tokenization. These enhancements make it feasible to train complex models and apply memory-intensive algorithms, such as GRPO.
3.3 Stage 2:
Reward-based policy-optimization
In the second stage, our goal tune the workflow-prediction L.I. to better align it with flows that produce high quality outputs for given prompt. To do so, propose to leverage the recently introduced Group Relative Policy Optimization (GRPO) approach, which estimates advantages by comparing responses within groups of similar prompts, rather than relying on a separate value function. Using GRPO has two main benefits: (I) eliminates the need to learn a separate value function, enabling better memory utilization during training and (2) its group-based reward normalization encourages greater exploration and diversity in generated workflows. However, the use of this approach requires to score and rank the different candidate flows generated for each input prompt at training time. Naively, we could simply generate images with each such flow and score them using the human- preferenee predictors used by ComfyGen [16]. However, for complex flows, creating the images might take an order of a minute, greatly limiting the speed of training. Hence, we propose to avoid this lengthy generation step and instead train surrogate reward model that will directly estimate the final reward from pair of prompt and flow inputs.
Surrogate Reward Model Training We implement the surrogate reward model R, top of ModernBert [57] backbone, with novel output head trained to map the CLS token into a score. To tune the model, feed with strings containing prompt and flow pair, and task human-preference score for the image produced by this pair. For data, we use the ComfyGen dataset DR. which contains triplets of prompt Pt. flow and score sy. The surrogate's 's loss then:
predict the
In(
Σ (D..OS))EDR
Although the construction of the original Comfy dataset still required generating images and scoring them, we find that the surrogate reward is much more sample efficient, performing well with just the 330 post-augmentation flows of ComfyGen (compared with our own 80k unscored flows).
3.3.1
Component-Aware Hybrid Reward Formulation
Since downstream flow performance can be heavily influenced by relatively few tokens (model choices, existence of specific blocks), we propose to further refine our surrogate model with a prefix- prediction score that is better able to assign credit to specific components. Specifically, we tune an additional reward model R
pirefie to predict the generated image score even when presented only with

---

randomly sampled prefixes of the flow:
Leare (0)
E
M Jil1 ;1);))))))) 81).
(2)
Our final reward design combines these two complementary signals to assign different reward to each token 4, depending on both the expected performance of the full flow, as well as a prefix ending with its component:
R(t) fp + 1 LE,
R (p, fiz).
(3)
j=1
where T are the tokens comprising the same flow component L, and the contribution of the entire component.
3.3.2
Uncertainty-Awan Reinforcement Learning
Finally, prior work [17, 50, 58] observed that the use of surrogate reward models can lead reward hacking. To avoid this pitfall, we train an ensemble of N surrogate models {R Røgi... Røx h each using different split of training data. The ensemble provides us with both more robust mean prediction, as well as with an uncertainty estimate:
u(p,
1 N
Σ
Ro (p.
Σ (R, 1)).
N
We can then define an ancertainty-awar reward function:
R(p, f) = (00(0))) (((p.))
(p, ST fp. 0
where a threshold parameter. This pessimistic approach assigns zero reward prompt-flow pairs with high uncertainty, preventing the model from optimizing specific subsets of the reward ensemble, or from drifting to regions where the surrogate's predictions are unreliable.
3.4 Dual model guidance
As an additional step, we propose that results may be further improved through the use of novel inference mechanism inspired by classifier-rree guidance (CFG, [20]). Specifically, we draw on recent work on image generation [27] which demonstrate that diffusion models can be guided by extrapolating the predicted scores along the direction from an under-trained version of the model, and the fully trained one. We propose to apply similar idea here, where we consider both our policy-optimized model (Morro. stage 2) and "undertrained" SFT version (MSrT, stage 1). At inference time, generations are sampled by interpolating the logits of of both models:
log PogPoFGJJil PoFo(( logpsFT( (5 P + 7( log PGRPO (/ - logpssT (JHls<s)))
where PCFG represent the sampling distribution, PSFT is the next-token distribution of stage I model and PGRPO is the next-token distribution of stage 2 model. Finally, 7 2 controls the guidance strength. Unless otherwise noted, we use = 1.5.
4 Experiments
4.1
Comparisons
We follow [16] and compare our approach to set of baselines across two main metrics: (1) The GenEval [18] benchmark which measures prompt-adherence by using object detection and classification modules to evaluate correct object generation, placement, and attribute binding. (2) Human preference, using the CivitAl prompt-set ComfyGen [16]. For the latter, we evaluate our approach using both automated preference metric (HPS v2, [59]) as well as user study.

---

We compare our approach against the following types of baselines: (1) Fixed, monolithic mod- els including: SDXL, popular fine-tuned versions thereof, and SDXL-DPO, which was directly fine-tuned with human preference data. (2) Fixed, popular workflows, where we use the same workflow to generate all images regardless of the prompt. (3) Prior pipeline construction ap- proaches, including agentic workflows that select and use off-the-shelf editing tools to cor- rect generated content (GenArtist, [67]) and reward-based fine-tuned LLMs (ComfyGen [16]).
Prompt adherence: As summarized in Table I, FlowRL demonstrates strong performance on the GenEval benchmark despite not being explicitly trained for prompt adherence. It achieves an overall score of 0.61, matching the bestpperforming baseline, ComfyGen. Notably, our approach outperforms other methods in the "two objects" (0.85 vs. 0.82) and "binding" (0.38 vs. 0.29) categories, indicating improved capability in handling complex com- positional prompts. A representative qualitative example illustrating prompt adherence is provided in Figure 4.
Visual Quality: To automatically evaluate the visual quality of FlowRL's 's outputs, we follow [55, 43, 16] and use a pair-wise comparison of HPS v2 [59] score between FlowRL. and each baseline and report the average win rate. These comparisons use the full CivitAI test set of [16]. The win-rate of each baseline over FlowRL is reported in Table Additionally, we conducted a user study were we show users 35 randomly sampled prompts and the images generated for each, using FlowRL and one of the baselines. Here, we focus on the best performing baseline from each category, as well as ComfyGen [16]. We then ask them to select the image that they prefer, taking both prompt adherence and visual quality into account. We report the aggregated win percentage in figure 5, and add more details in the supplementary. This experiment demonstrates FlowRL's 's capability to create more performant ComfyUI workflows for the given input prompts. Representative qualitative comparisons highlighting these improvements are provided in Figure 4, where our outputs consistently exhibit better prompt alignment and structural coherence compared to baseline generations.
Figure 3: Example of generations with FlowRL
Model
Single Two object object
Counting Colors Position
Attribute binding
Overall
HPSv2 winzate FlowRL
SDXL
0.98
0.74
0.39
0.85
0.15
0.23
0.55
2% + 0.6%
JuggernautXI.
1.00
0.73
0.48
0.89
0.11
0.19
0.57
51%.1: 1%
DreanShuperXL
0.99
0.78
0.45
0.81
0.17
0.24
0.57
3% 0.6%
DPO-SDXI.
1.00
0.44
0.59
5 1S:
Most Popular Flow
0.95
0.38
0.26
0.77
0.06
0.12
0.42
13% 1%
2nd Most Popular Flow
1.00
0.65
0.56
0.86
0.13
0.34
0.59
14%:: 1%
GenArtist Dast
0.94
0.41
0.40
0.72
0.24
0.07
0.47
5% :: 1%
1.00
0.64
0.21
0.89
0.20
0.35
0.55
3 0. 0.8%
ComfyGen
0.99
0.82
0.50
0.90
0.13
0.29
0.61
40% 2%
FlowRL (Ours)
1.00
0.85
0.44
0.86
0.11
0.38
0.61
Table 1: GenEval and HPS v2 comparisons. FlowRL is on on-par with ComfyGen on GenEval and outperforms all other baseline approaches in overall score. On human preference metrics, FlowRL significantly outperforms prior methods. Cls are calculated as one standard deviation from the mean.
Novelty of generated flows: A key advantage of our approach lies in its capacity to generate workflows that are not merely copies of those seen during training. To quantify this novelty, we generate 500 flows using the CivitAl test set, and calculate the normalized Levenshtein distance (NLD) [52, 32] between each generated workflow and its nearest training sample. We further normalize these values by the NLD between training samples, giving us measure of what fraction of the variance in training data we manage to preserve. Additionally, we report how many generated flows exist "as-is" in the training data, and how many unique flows were created in the 500 output set.
The results are reported in Table 2. Our experiments confirm the findings of [16) which report that their approach learned to copy flows from the training data. FlowRL meanwhile achieves significantly
7

---

Fixed Flow
GenArt
Juggernaut ComfyGen
FlowRL
CivitAI
victorion genteman octopuss enjoyny tea time elgant partor
déliche of groeza with hanging simving feces.."
GenEval
"a photo of a the basebal bat and pink book
hair drer left tolkz!"
Figure 4: Qualitative results CivitAI and GenEval prompts.
60%
40%
70%
30%
23%
75%
25%
0%
50%
100%
0%
50%
100%
Flow (Ours)))
ComfyGen
juggernautXL
GenArtist
Fixed flow
Figure 5: Human study win rate of FlowRL vs other relevant baselines
higher novelty, demonstrating the ability to generalize to new parameter combinations. These results highlight the effectiveness of reinforcement learning framework in encouraging the LLM to explore and produce a broader range of complex workflows.
Effects of Dual model guidance: Next, investigate the impact of our dual-model guidance approach. Specifically, prior work [12] highlighted the ability of guidance-based methods to trade diversity for performance (or recall for precision). We show that similar behavior be observed here. As shown Table 2, while increasing guidance strength (r) improves HPS v2 scores win-rate ComfyGen, significantly impacts the structural diversity of the generated workflows. At 1.5, our method maintains the uniqueness of generated flows. However, as increases, we observe dramatic reduction in the uniqueness ratio just 8%.
Notably, our FlowRL variants maintain near-zero overlap with training data (0-1% "exists in data" ComfyGen's 94%), and the NLD ratio actually improves with guidance (from 0.6 without CFG to 0.75 at = 2). This pattern suggests that stronger guidance pushes the model to consistently generate smaller subset of high-performing workflows, effectively concentrating probability mass on patterns that maximize reward but reducing exploration of the solution space. Conceptually, this mirrors

---

observations in image generation with CFG, where higher guidance strengths produce higher-qualij but less diverse output.
unique ratho (%)
exists (%)
NLD ratio
HPSv2 win-rute Vs ComfyGen
Method
ConfyGEN
7%
94%
0
FlowRL (wlo CFG)
41%
0.6
59%
F C 1.5)
41%
0%
0.74
60%
FlowRL CFG
8%
0%
0.75
63%
Table 2: Comparison of originality of flow generation models
4.2 Ablation study
To quantify the impact of individual components in FlowRL, we conducted an ablation study com- paring variants with and without our key improvements. We evaluated the following modifications: reward model, (2) removing the uncertainty ensemble cutoff, (3) varying number of BERT models in our reward ensemble, (4) dropping the SFT step (stage 1), and (5) dropping the GRPO-tuning step. For (5). we instead use the stage-1 model to sample five flows per prompt, and use our reward ensemble to score them in relation to the prompt. Then, we generate an image with the highest scoring flow. Finally, to ensure that our benefits are not grounded in the novel encoding scheme, we also evaluate a baseline ComfyGien [16] model trained on this new representation. We compare all scenarios against both the original ComfyGEN and against our full model, using HPSv2 scores on the CivitAl prompt set. The errors reported are the 1 Wald interval.
(I) removing the component-awan
w/o prefix reward
w/o reward cutoff
Ensemble of 3
wlo SFT stage
win ratio
$
ComfyGen (+ encoded)
SFT only
Berts
55
57
58
56
56
29
vS ComfyGen(Se)
+2.22
42.22
12.22
12.21
12.22
42.16
+2.02
VS ours (%)
42
45
33
34
36
26
19
22.2
42.21
42.1
#2.12
12.15
1199
+ 1.75
Table 3: The win ratio on the HPSv2 score for each component of our method compared to (1) the ComfyGen baseline and (2) the full ComfyGenRL model, using head-to-head comparisons.
The results are presented in table 3. These demonstrate the vital contribution of each component to overall performance. The full model consistently outperforms all ablations, with particularly significant drops observed when removing the SFT stage entirely (0% win rate against ComfyGen and our full model). This emphasizes the critical nature of proper initialization before applying reinforcement learning methods. Looking at specific components, "prefix reward" proves the most beneficial, showing the importance of assigning more granular rewards. The "ComfyGen (+encoded)" variant, which uses our encoding scheme but lacks reinforcement learning, achieves only 37% win rate against the original ComfyGen, highlighting that our encoding improvements work synergistically with the GRPO training approach.
5 Discussion
This paper presents a novel approach for fine-tuning LLMs using a combination of supervised learning on flow data, surrogate reward modeling, and uncertainty- reinforcement learning. Our method addresses several key challenges in LLM fine-tuning, including reward hacking, distribution shifts, and training efficiency. The results demonstrate that our approach outperforms existing baselines across multiple metrics. Importantly, compared to prior workflow generation work, our approach demonstrates greater output diversity and successfully generalizes novel flows that did not exist in the training data.
Although it improves on the current state-of-the-ard in multiple aspects, our approach still maintains many of their limitations. First, it remains focused on text-to-image workflows, with no support for editing tasks or video modules. Second, introducing new workflow components to the LLM would require retraining our entire stack. In the future, we hope to explore more efficient ways of adapting novel models or blocks.

---

By enabling reliable and diverse automated workflow generation, our work advances generative Al systems that adapt to human preferences. We hope will help foster more collaborative innovation by streamlining the integration of independently trained, specialized modules.
References
[1] Google DeepMind AlphaCode Team. Alphacode technical report. https://storag:: googleapis. com/ http://maaaaaiiaaaaaaaaaaaaaaaaa RaaaaaaooddeTTeee_e_epprrrrr Recoth_epppttttt pdf, 2024.
Techh
[2] Halil Beglerovic, Michael Stolz, and Martin Horn. Testing of autonomous vehicles using surrogate models and stochastic optimization. 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), pages 1-6. IEEE, 2017.
[3] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. htlps://cdn. openai. confpapers/dall-e-3 pdf. 2(3):8, 2023.
[4] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024.
[5] Yujie Cao, Azhan Abdul Aziz, Wan Nur Rukiah Mohd Arshard, Chang Xu, Muhamad Abdul Aziz Bin Ab Gani, and Issarezal Bin Ismail. Ai 2d-3d generation in architectural design: A divine hand pandora's 's box. In 2024 IEEE 22nd Student Conference Research and Development (SCOReD), pages 103-208. IEEE, 2024.
[6] Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards, In The Twelfth International Conference on Learning Representations, 2024.
[7] Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. arXiv preprint arXiv:2310.0220 30002223 2023.
[8] Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu. Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310. 12773, 2023.
[9] Xiaoliang Dai, Ji Hou, Chih-Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang. Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, et al. Emu: Enhancing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309.15807 2023.
[10] Daniel, Michael, and the Unsloth Community. Unsloth: Fast, memory--fficient I1m fine- tuning library. https:// /ttps:///ithbb com/ unssoothaiunsslth. 2024. Version 2.0. Accessed: 2025-05-20.
[11] Fei Deng, Qifei Wang, Wei Wei, Tingbo Hou, and Matthias Grundmann. Prdp: Proximal reward difference prediction for large-scale reward finetuning of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7423-7433,
2024.
[12] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
[13] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow trans- formers for high-resolution image synthesis. In Forty-first international conference on machine learning.
[14] Luca Eyring. Shyamgopal Karthik, Karsten Roth, Alexey Dosovitskiy, and Zeynep Akata. Reno: Enhaneing one-step text-to-image models through reward-based noise optimization. arXiv preprint a4x.v:240604433 2024.
10

---

[15] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine-tuning text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[16] Rinon Gal. Adi Haviv, Yuval Alaluf, Amit H Bermano, Daniel Cohen-Or, and Gal Chechik. Comfygen: Prompt-adaptive workflows for text-to-image generation. arXiv preprint arXiv:2410.01331, 2024.
[17] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835-10866. PMLR, 2023.
[18] Dhruba Ghosh, Hannanch Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36, 2024.
[19] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
[20] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[21] Susung Hong. Gyuseong Lee, Wooseok Jang, and Seungryong Kim. Improving sample quality of diffusion models using self-attention guidance. In Proceedings the IEEF/CVF International Conference on Computer Vision. pages 7462-7471, 2023.
[22] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. ArXiv, abs/2106 10220000066,, 2021.
[23] Oucheng Huang, Yuhang Ma, Zeng Zhao, Mingrui Wu, Jiayi Ji, Rongsheng Zhang, Zhipeng Hu, Xiaoshuai Sun, and Rongrong Ji. Comfygpt: A self-optimizing multi-agent system for comprehensive comfyui workflow generation, 2025.
[24] Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. Ai alignment: A comprehensive survey, arXiv preprint arXiv:2310. 19852, 2023.
[25] Keyur Joshi, Chiao Hsieh, Sayan Mitra, and Sasa Misailovic. Gas: Generating fast and accurate surrogate models for autonomous vehicle systems. arXiv preprint arXiv:2208.022:: 02232, 2022.
[26] Maria Kalweit, Gabriel Kalweit, Moritz. Werling, and Joschka Boedecker. Deep surrogate q- learning for autonomous driving. In 2022 International Conference on Robotics and Automation (ICRA), pages 1578-1584. IEEE, 2022.
[27] Tero Karras, Miika Aittala. Tuomas Kynkäänniemi, Jaakko Lehtinen, Timo Aila, and Samuli Laine. Guiding diffusion model with bad version of itself. Advances in Neural Information Processing Systems, 37:52996-53021 2024.
[28) Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, al. Dspy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint arXiv:2310.03714, 2023.
[29] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[30] Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 55:21314-21322 2022.
11

---

[31] Kimin Lee. Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302. 2023.
[32] Vladimir Levenshtein. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady, 10(8):707-710, 1966.
[33] Zeyuan Li, Yangfan He, Lewei He, Jianhui Wang, Tianyu Shi, Bin Lei, Yuchen Li, and Qiuwu Chen. Falcon: Feedback-driven adaptive long/short-term memory reinforced coding optimization system. arXiv preprini arXiv:2410.2133 2024.
[34] Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical rea- soning for large language models via reinforced evol-instruct. arXiv preprint 0rXiv:2308.09583, 2023.
[35] Yang Luo, Yiheng Zhang. Zhaofan Qiu, Ting Yao, Zhineng Chen, Yu-Gang Jiang, and Tao Mei. Freeenhance: Tuning-frre image enhancement via content-consistent noising-ande sussing-and-deoisinn process. arXiv preprint NWW:2409.07451,, 2024.
[36] Youssef Mrouch. Reinforcement learning with verifiable rewards: Grpo's s effective loss, dynam- ics, and success amplification. arXiv preprint arXiv:2503.0663 06663, 2025.
[37] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.1074: 2021.
[38] Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, et al. Can generalist foundation models outcompete 2023.
purpose tuning? case study in medicine. arXiv preprint 1r
[39] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[40] Iván J Pérez-Colado, Manuel Freire-Morán, Antonio Calvo-Morata, Victor M Pérez-Colado, and Baltasar Fernández-Manjóh Ai asyet another tool in undergraduate student projects: Preliminary results. In 2024 IEEE Global Engineering Education Conference (EDUCON), pages 1-7. IEEE, 2024.
[41] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, 2024.
[42] Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text- to-image diffusion models with reward backpropagation. arXiv preprint arXiv:2310.03739, 2023.
[43] Zipeng Qi, Lichen Bai, Haoyi Xiong, et al. Not all noises are created equally: Diffusion noise selection and optimization. arXiv preprint arXiv:2407. 14041, 2024.
[44] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 2022.
[45] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, IA, USA, June 18-24, 2022, pages 10674-10685. IEEE, 2022.
[46] S Sane. Hybrid group relative policy optimization: A sulti-sample approach to enhancing policy optimization. arxiv, 2025.
12

---

[47] Eyal Segalis, Dani Valevski, Danny Lumen, Yossi Matias, and Yaniv Leviathan. A picture is worth thousand words: Principled recaptioning improves image generation. arXiv preprint arXiv:2310. 16656, 2023.
[48] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03332 2024.
[49] Chenyang Si, Ziqi Huang. Yuming Jiang, and Ziwei Liu. Freeu: Free lunch in diffusion u-net. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4733 4733-4773,, 2024.
[50] Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett, A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716, 2023.
[51] Dominik Sobania, Martin Briesch, and Franz Rothlauf. Comfygi: Automatic improvement of image generation workflows, 2024.
[52] Keiichiro Tashima, Hirohisa Aman, Sousuke Amasaki, Tomoyuki Yokogawa, and Minoru Kawa- hara. Fault-prone java method analysis focusing on pair of local variables with confusing names. In 2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pages 154-158. IEEE, 2018.
[53] Trieu H Trinh, Yuhuai Wu, Quoe V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 2024.
[54] Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprini arXiv:2211 (.fi::22/1.142777. 2022.
[55] Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8228-8238, 8238, 2024.
[56] Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations. arXiv preprint arXiv:23/2.08933, 2023.
[57] Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, et Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. arXiv preprint arXiv:2412.1366. 2024.
[58] Jiaxin Wen, Ruiqi Zhong, Akbir Khan, Ethan Perez, Jacob Steinhardt, Minlie Huang, Samuel R Bowman, He He, and Shi Feng. Language models learn to mislead humans via rlhf. arXiv preprint a4Xiv:2409. 12822, 2024.
[59] Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.0334/, 2023.
[60] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding. Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
[61] Xiangyuan Xue, Zeyu Lu, Di Huang, Zidong Wang, Wanli Ouyang, and Lei Bai. Comfy- bench: Benchmarking l1m- agents comfyui for autonomously designing collaborative ai systems, 2024.
[62] Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Stefano Ermon, and Bin Cui. Mastering text-to-image diffusion: Recaptioning, planning, and generating with multimodal 1ms.. In International Conference on Machine Learning, 2024.
13

---

[63] Honglin Yao. Improving performance of face generation for specific individuals based on lora. In 2024 International Conference on Electronics and Devices, Computational Science (ICEDCS), pages 77-80. IEFE, 2024.
[64] Zhengqing Yuan, Ruoxi Chen, Zhaoxu Li, Haolong Jia, Lifang He, Chi Wang, and Lichao Sun. Mora: Enabling generalist video generation via a multi-agent framework. arXiv preprint a7X.v:2403.13248, 2024.
[65] Yuanzhao Zhai, Han Zhang, Yu Lei, Yue Yu, Kele Xu, Dawei Feng, Bo Ding, and Huaimin Wang. Uncertainty-penalizi reinforcement learning from human feedback with diverse reward lora ensembles. arXiv preprini arXiv:2401.00243 2023.
[66] Yinan Zhang, Eric Tzeng, Yilun Du, and Dmitry Kislyuk. Large-scale reinforcement learning for diffusion models. arXiv preprint arXiv:2401. 122244 2024.
[67] Wang Zhenyu, Li Aoxue, Li Zhenguo, and Liu Xihui. Genartist: Multimodal Ilm as an agent for unified image generation and editing. arXiv preprint arXiv:2407.05660 2024.
[68] Mingehen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. Gptswarm: Language agents optimizable graphs. In Forty-first International Conference on Machine Learning, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Each claim made in the introduction is elaborated in the main text.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
See section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
15

---

Justification:
The paper has no theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide an explanation about the reproducibility of our experiments the supplementary.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [No]
Justification: The paper relies on data from a prior paper. The prior data is not public but can be requested from the authors.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
Subliie/giides/CdeSabbbiss
ionillooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https: ggtgg/uies////// wwweeeuummssssonPee..
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide an explanation about the reproducibility of our experiments in the supplementary.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We detail how confidence intervals were computed for the reported results in the main text.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
17

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification We provide an explanation about the reproducibility of our experiments in the supplementary.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https://neurip:: httppuulic/ttiicaaaaiiiieeeeeeaa
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impact in the supplementary.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
We do not release data or models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All relevant work is cited in the references.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
We provide an example of user-study questions in the supplementary.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
There are no risks involved our user study.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: LLM is part of our method, and we provied details explanation how it is combined within it in the main text.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: https///eeuup...eeeee cc cc//oonferencee//22///11 for what should or should be described.
21


Reviews:
[{"review_id": "mtaoa7Gn3p", "reviewer": "Reviewer_mZDq", "summary": "This paper introduces a new framework that uses reinforcement learning to automate the design of complex text-to-image generation pipelines, including a surrogate reward model, two-stage training strategy and dual model guidance. The experimental results demonstrate that the proposed approach not only generates superior visual quality compared to existing methods but also exhibits better diversity.", "questions": "- This paper demonstrates that the dual-model guidance technique improves performance by trading off diversity. Have the authors tried dynamically adjusted guidance strength, just like the recent developmenet in CFG[1] to achieve a better balance between quality and diversity ?\n- Given the weaknesses discussed above, have the authors considered about how the framework can be extended to add new components ? For instance, could new workflow nodes be added by fine-tuning only a small part of the reward models ?\n\nReference:\n\n[1]  Analysis of classifier-free guidance weight schedulers", "limitations": "Yes, the authors discuss the limitations in Section 5.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- This paper combines multiple advanced techniques when building the system, including a two-stage training process, first SFT then GRPO, and dual model guidance to further improve generation quality.\n- The introduction of surrogate reward model predicts image quality directly from a prompt-workflow pair, which avoids the expensive step of generating an image for every candidate workflow and enables efficient RL training.\n- Extensive experimental results are provided, including automated metrics in GenEval and HPSv2, user study, and thorough ablation study. \n\nWeaknesses:\n- This paper is actually building an agentic system. However, every time a new model or component is added, a new round of training is needed. This limitation will make the agentic system lose its inherent advantages\n- The performance of the entire system, especially the surrogate reward model, is heavily depending on the quality and diversity of the score dataset from ComfyGen. Any biases or gaps in this dataset will accumulate to the final system.", "was_revised": true, "final_justification": "Thank authors for the rebuttal. My concerns have been addressed and I will keep the score.", "rebuttal": "We appreciate your review and the points raised regarding our framework's strengths, including the two-stage training, dual model guidance, surrogate reward model, and experimental validation.\n\nWe address your weaknesses and questions as follows:\n\n**Q:**\n> \"This paper is actually building an agentic system. However, every time a new model or component is added, a new round of training is needed. This limitation will make the agentic system lose its inherent advantages\" & \"Given the weaknesses discussed above, have the authors considered about how the framework can be extended to add new components ? For instance, could new workflow nodes be added by fine-tuning only a small part of the reward models ?\"\n\n**A:** We acknowledge that our current framework will require some retraining as new types of components are introduced and believe this topic to be an excellent direction for future work. We did discuss some pathways for adding more components as they become available, but did not conduct sufficient experiments down this path to offer more than a hypothesis. In general we believe the difficulty of adding new components highly depends on their nature (a new version of an existing model vs. an entirely new type of graph block).\n\nAdding a new version of an existing model (any scenario where the new model is just a parameter change in an existing block, e.g., a new fine-tuned version of SDXL) can likely be done in two stages: (a) It has to be introduced to the surrogate reward model, which can be done by adding a new token (and word embedding) to represent the novel model, and then optimizing the embedding so that the surrogate model correctly predicts the scores of prompt+flow pairs containing the new model. This is similar to existing soft-prompting or personalization techniques. (b) The LLM can be tuned to recognize this component (e.g., with a new embedding + LoRA) by continuing the GRPO training process, with predicted flows occasionally having their models swapped for the new model (and then scored as usual with the surrogate model).\nHowever, we believe this replacement strategy is unlikely to work for entirely new blocks since dropping them into existing LLM-predicted flows is non-trivial.\n\n\n**Q:**\n> \"The performance of the entire system, especially the surrogate reward model, is heavily depending on the quality and diversity of the score dataset from ComfyGen. Any biases or gaps in this dataset will accumulate to the final system.\"\n\n**A:** We agree that indeed biases or gaps in the dataset could impact the system. However, note that the current dataset already leads to improved performance compared to existing alternatives, and more data could likely lead to additional gains. On the prompt-understanding side, the model uses an LLM backbone which can generalize well to new categories.\n\nReferencing our detailed response to Reviewer 2NbJ03, we also evaluated the reward model's generalization to unseen data by re-training it on a subset of our dataset. We found that while the model can generalize to novel graph structures, performance degrades when they contain entirely unseen blocks or models (e.g., new unseen tokens). However, the model does fine with new graph structures containing the known blocks. Future work could explore active learning or synthetic data generation to further enhance coverage.\n\n**Q:**\n>  \"This paper demonstrates that the dual-model guidance technique improves performance by trading off diversity. Have the authors tried dynamically adjusted guidance strength, just like the recent developmenet in CFG[1] to achieve a better balance between quality and diversity\"\n\n**A:** In our initial work, we primarily focused on demonstrating the benefits of dual-model guidance using fixed strengths. We did not experiment with dynamically adjusted guidance strength, but are happy to add such experiments to the revision if the reviewer believes them important."}, {"review_id": "3rjrWYRM7u", "reviewer": "Reviewer_2NbJ", "summary": "The paper introduces FlowRL, a reinforcement learning based framework designed to automate the text-to-image generation workflow within the ComfyUI ecosystem. FlowRL is consisted of 2 stages: a supervised fine-tuning on dataset of unscored prompt-flow pairs to learn structural priors, and a RL optimization using group relative policy optimization (GRPO), which predict image quality directly from prompt-flow pairs instead of generated images. In addition, the model employs a classifier-free guidance mechanism at inference to balance quality and diversity. Experimental evaluations show that FlowRL outperforms baseline methods.", "questions": "- Can the authors elaborate on how transferable FlowRL is to other domains or platforms?\n- How does the performance of FlowRL scale as the diversity of prompts and flow complexity increases? Is there evidence of reward model generalization in unseen domains or under distribution shifts?", "limitations": "N/A", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Pros:\n- The paper has strong empirical validation. The proposed method is comprehensively benchmarked against baselines with GenEval and HPSv2 metrics. The experiments show better performance than previous methods.\n- The paper shows good technical innovation. The proposed RL framework overcomes the limitation of prior model (ComfyGen) and improves efficiency and scalability. By introducing a surrogate reward model ensemble, the authors effectively eliminate the need for image generation during training.\n- The paper has good concept of framing of workflow generation as a sequential RL task using LLMs is a fresh and significant departure from prior classifier-based or static methods.\n\nCons:\n- The method could have limited domain generalization since it is focused on ComfyUI-based text-to-image pipelines. The generalization to other domains is not tested and discussed in depth. Additionally, since the model builds heavily on the ComfyGen dataset, the reproducibility is limited.", "was_revised": true, "final_justification": "The authors addressed my questions and I would love to keep my score.", "rebuttal": "We thank the reviewer for the valuable feedback. We're pleased that the reviewer found our empirical validation strong and the technical innovation of our proposed RL framework to be novel. We address the concerns below:\n\n\n**Q:**\n>  \"Can the authors elaborate on how transferable FlowRL is to other domains or platforms?\" & \"The method could have limited domain generalization since it is focused on ComfyUI-based text-to-image pipelines. The generalization to other domains is not tested and discussed in depth. Additionally, since the model builds heavily on the ComfyGen dataset, the reproducibility is limited.\"\n\n**A:** The trained FlowRL model is not directly transferable to other domains or platforms because it has learned the specific \"language\" (API, tokens) and structure of ComfyUI workflows, and was trained exclusively on ComfyUI flow data. However, the core framework – using reinforcement learning for pipeline design with a surrogate reward model – is conceptually transferable to other pipeline design domains. In the context of Text-to-Image (T2I) workflows specifically, if an adaptation layer or mechanism exists to convert ComfyUI workflows into equivalent workflows for another T2I framework, then the knowledge gained might be transferable due to the shared underlying task. \n\n\n**Q:**\n> \"How does the performance of FlowRL scale as the diversity of prompts and flow complexity increases? Is there evidence of reward model generalization in unseen domains or under distribution shifts?\"\n\n**A:** Following the reviewer question, we conducted an experiment where we re-trained the reward model using half of our original training set, and evaluated the generalization capabilities of our reward model on the remaining half. We further ensured that some graph structures appear only in the hold-out set, so we can evaluate performance on entirely unseen flow graphs.\n\nThe results show good overall performance on the hold-out set, with an $R^2$ score of $0.643$ and a pearson correlation of $0.816$, indicating a strong relationship between predicted and actual values. A more in-depth look shows that: (1) The model generalizes very well to scenarios which contain only parameter or prompt changes compared to what it saw during training (Pearson 0.928). (2) Performance remains good for flows with novel graph structures, but using only seen components (models / blocks) (Pearson 0.667). (3) Performance drops significantly for flows with entirely unseen components, which contain tokens that the reward model has never seen (Pearson 0.152). \n\nAfter running this experiment, we also evaluated the GRPO stage using the new reward model. The model trained on the full dataset outperforms the model trained on the partial data (71% HPSv2 win-rate), showing the benefit of additional data. Due to time constraints, we were unable to repeat this experiment with additional data amounts, but we would be happy to conduct more such experiments for the revision and present a graph illustrating the scaling of performance as a function of data."}, {"review_id": "l4lKoMfjuA", "reviewer": "Reviewer_p5tr", "summary": "This paper presents a policy optimization strategy for workflow prediction for text-to-image tasks. The idea drawing from previous work is to prompt/finetune an LLM to directly predict the ComfyUI workflow for the task of text-to-image generation. Similar to several LLM fine-tuning pipelines these days, this involves a supervised fine-tuning stage on a curated dataset of workflows, followed by training a surrogate reward model, and then a policy optimization stage (using GRPO) to maximize these rewards. Results on prompt following and other reward model evaluations indicates that the proposed method outperforms prior works exploring similar concepts for the task.", "questions": "There are 2 major aspects where I am curious to see if there are improvements possible:\n\na) Choice of Evaluation: While GenEval provides a clean benchmark for prompt following on relatively short and basic prompts, the ideal usecases of FlowRL/ComfyGen etc. are probably quite different. Would it be possible to have more meaningful evaluations of the proposed methods beyond the standard evaluations provided in the paper to the kind of tasks that an end user of such a framework might actually care about?\n\nb) Novelty of Generated Flows/Source of Improvements: An interesting aspect of end-to-end optimization is that we might observe novel sources of improvements. While the analysis in Tab. 2 is a first step in this direction (observing the novel generated flows), it might be worthwhile to have a deeper dive into this to see the exact nature of these novel flows to see what causes the underlying improvements?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nThe paper makes the natural and principled step towards end-to-end optimization of the workflow generation concept (as opposed to just prompting/in-context learning) with a reward ensemble which provides a more robust optimization strategy.\n\nFurther, the paper also has a robust ablation analysis covering important components of the proposed pipeline providing convincing results. \n\nThe paper also does a good job of presenting the paper in a clear and easy to understand manner. \n\nWeaknesses:\n\nThe only major weakness I can see in the paper is that it is a \"minimal\" improvement over ComyGen in many aspects (the overall formulation, training data for SFT etc.). In terms of the overall improvements, while ComfyGen and the proposed FlowRL have the same GenEval results, the HPSv2 and the human study seem to indicate improvements for FlowRL over ComyGen. However, it might be a good idea to clarify the aspects in which FlowRL improves over ComfyGen and how these might not be captured by GenEval.   \n\n\nOverall, I do think that the paper broadly is a step in the right direction and therefore would merit acceptance to me, however I do think it would be a good idea to clarify both the comparisons, evaluations, and have better analysis on the source of improvements/novelty of generated workflows.", "was_revised": true, "final_justification": "I would like to thank the authors for providing a clear clarification in the rebuttal. Having looked over the paper, the reviews, as well as the rebuttal, I would like to stick with my original rating (Borderline Accept). I do think that the paper approaches an important problem and provides valuable insights, and therefore merits acceptance to me, despite the fundamental method being a relatively direct application of GRPO for agentic text-to-image generation.", "rebuttal": "We thank the reviewer for the comprehensive review and valuable feedback on our paper. We are encouraged by the recognition of our principled step towards end-to-end optimization of workflow generation for T2I tasks, and the robust ablation analysis which underscores the effectiveness of our approach. We address the the reviewer concerns below:\n\n**Q:**\n>  \"Choice of Evaluation: While GenEval provides a clean benchmark for prompt following on relatively short and basic prompts, the ideal usecases of FlowRL/ComfyGen etc. are probably quite different. Would it be possible to have more meaningful evaluations of the proposed methods beyond the standard evaluations provided in the paper to the kind of tasks that an end user of such a framework might actually care about?\" & \"The only major weakness I can see in the paper is that it is a \"minimal\" improvement over ComyGen in many aspects (the overall formulation, training data for SFT etc.). In terms of the overall improvements, while ComfyGen and the proposed FlowRL have the same GenEval results, the HPSv2 and the human study seem to indicate improvements for FlowRL over ComyGen. However, it might be a good idea to clarify the aspects in which FlowRL improves over ComfyGen and how these might not be captured by GenEval.\"\n\n\n**A:** We acknowledge your observation regarding the comparison with ComfyGen. While GenEval results might appear similar, FlowRL's improvements are primarily evident in HPSv2 and the user study (70% win-rate in the user study, and 60% win-rate in the HPSv2 score). Those metrics better reflect overall image quality and aesthetic preference. GenEval, being a prompt-following metric, may not fully capture these qualitative improvements in image generation. In addition, while GenEval serves as a valuable academic benchmark with its simplified prompts, the CivitAI prompts are taken from images created and shared by real users, and thus offer a more representative evaluation of the model's performance in real-world user scenarios. \nThe core improvement of FlowRL lies in its end-to-end policy optimization with a reward model, which allows it to discover more optimal and nuanced workflow configurations than ComfyGen's supervised learning approach. The \"novelty of generated workflows\" (as discussed in your second question) is a direct outcome of this optimization, leading to better-performing flows not present in the initial SFT dataset.\n\n**Q:**\n>  \"Novelty of Generated Flows/Source of Improvements: An interesting aspect of end-to-end optimization is that we might observe novel sources of improvements. While the analysis in Tab. 2 is a first step in this direction (observing the novel generated flows), it might be worthwhile to have a deeper dive into this to see the exact nature of these novel flows to see what causes the underlying improvements?\"\n  \n**A:** We concur that a deeper dive into the exact nature of novel flows and their contribution to improvements is valuable. The GRPO stage inherently encourages exploration beyond the initial SFT dataset, allowing the policy to discover workflow configurations that yield higher rewards but were not explicitly present in the training data.\nTo further explore the nature of these novel flows, we followed ComfyGen and conducted a TF-IDF analysis focusing on the specialized usage of different models across different categories of prompts. For example, we observed that *JibMixRealisticXL* dominates in Photo-realistic and People categories, *TurboVisionXL* was mostly used in darker, more technical themes such as Gothic and Horror, *CrystalClearXL* is stronger in artistic, creative, and abstract categories, *DreamShaperXL* is mostly used in natural and cosmin themes. In addition, *AlbedoBaseXL* is used in very specific and niche categories (such as Steampunk and Underwater). This analysis reveals distinct specializations and category clusters among the models used in the optimized flows, indicating how the policy leverages specific models for particular content generation needs. We are happy to add a supplementary table outlining the most prominent model choices for each prompt category."}, {"review_id": "UG1MEAo0Z1", "reviewer": "Reviewer_9pZC", "summary": "The paper introduces FlowRL, a reinforcement learning based framework for generating text-to-image workflows in ComfyUI. It uses RL-based approach for generating novel ComfyUI workflows aligned with human preferences and predicts image quality scores directly from prompt-workflow pairs, avoiding costly image generation during training. It also introduces dual-model guidance between SFT and GRPO-tuned models during inference.", "questions": "1. See weakness  \n\n2. I think the writing quality needs improvement. The article actually uses a lot of content to describe methods for solving engineering problems, but I believe its contribution in terms of methodological innovation is limited.  \n\n3. Some parameters in the article may require more detailed descriptions to make the paper easier to understand.", "limitations": "Yes.", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n1. No need to generate images for scoring and obtaining rewards.\n2. Tested across multiple metrics.\n\n### Weaknesses\n1. On line 156 of the paper, perplexity reaches 1.9, but the significance of this value is not explained—e.g., no comparison with other methods.\n2. The third contribution of the paper: \"We integrate GRPO with per-token reward attribution for stable and memory-efficient policy optimization.\" seems relatively weak. The paper achieves higher batch sizes and faster training speeds by compressing JSON format, which appears to be an optimization of dataset processing rather than a methodological contribution.\n3. The paper mentions \"flow,\" but could provide more examples—such as its format and components—for clarity.\n4. One issue is that BERT learns a reward model for prompt-flow pairs, but the scores are assigned to generated images. A key question is whether higher scores for prompt-flow pairs (without image generation) equate to improved image quality. This point is not well demonstrated in the paper. Particularly in the Geneval evaluation, FlowRL performs worse than ComfyGen in counting, colors, and position.\n5. The terms p_SFT and p_GPRO in Equation (5) are not defined in the paper.\n6. The methodology primarily applies GRPO for reinforcement learning training of LLMs, introducing some training tricks (e.g., optimizing dataset format, per-token reward). However, these are largely engineering solutions for GRPO training. The main contribution—applying GRPO to T2I workflows—seems somewhat limited. To elevate the paper’s quality, more innovative elements should be incorporated.", "was_revised": true, "final_justification": "I appreciate the author’s efforts in addressing my concerns, but regarding the reinforcement learning optimization on workflows, I still have doubts about the contribution in terms of the algorithm.", "rebuttal": "We thank the reviewer for the feedback on our paper. We address the reviewer concerns:\n\n**Q:**\n> \"On line 156 of the paper, perplexity reaches 1.9, but the significance of this value is not explained—e.g., no comparison with other methods.\"\n\n**A:** The perplexity of 1.9 on line 156 indicates that our model is highly confident and accurate in predicting the next token in the workflow sequence, reflecting a strong understanding of the workflow structure and syntax. Which is an indication that the SFT stage is finalized and the optimization can be switched to the GRPO stage. We explored the option of optimizing the RL agent without the initial SFT stage in our ablation(section 4.2). We found this stage to be crucial due to the highly structured nature of the generated patterns, which necessitates its use as a prior.\n\n**Q:**\n> \"The third contribution of the paper: \"We integrate GRPO with per-token reward attribution for stable and memory-efficient policy optimization.\" seems relatively weak. The paper achieves higher batch sizes and faster training speeds by compressing JSON format, which appears to be an optimization of dataset processing rather than a methodological contribution.\"\n\n**A:** We respectfully clarify that while compressing the JSON format to our \"encoded\" format is an engineering optimization for efficiency, the \"integration of GRPO with per-token reward attribution for stable and memory-efficient policy optimization\" is indeed a methodological contribution. This approach was crucial for effectively applying GRPO to a sequential generation task like workflow design, where rewards need to be attributed to specific actions (tokens) within a long sequence. It addresses the challenges of credit assignment in complex, structured output generation, which goes beyond simple dataset processing.\n\n**Q:**\n>\"The paper mentions \"flow,\" but could provide more examples—such as its format and components—for clarity.\"\n\n**A:** We acknowledge that providing more detailed examples of flow formats and components could enhance clarity. We will add examples of workflows in the supplementary material of the final version.\n\n**Q:**\n>\"One issue is that BERT learns a reward model for prompt-flow pairs, but the scores are assigned to generated images. A key question is whether higher scores for prompt-flow pairs (without image generation) equate to improved image quality. This point is not well demonstrated in the paper. Particularly in the Geneval evaluation, FlowRL performs worse than ComfyGen in counting, colors, and position.\"\n\nA: The surrogate reward model (BERT-based) is trained to predict generated image quality scores directly from prompt-workflow pairs without generating an image. Specifically, these scores are derived by generating images and scoring them with off-the-shelf human preference predictors (e.g., Pickscore[1], HPS[2], ImageReward[3]) which have been shown (in their own published papers) to correlate strongly with human preference. Our experiments also use such models to validate the quality of our final outputs, and this is further verified with a user study. These experiments demonstrate that optimizing with the learned reward model indeed leads to improved image quality. The GenEval metric, which focuses on prompt-following, does not measure image quality and may not fully capture these qualitative improvements.\n\n**Q:**\n>\"The terms p_SFT and p_GPRO in Equation (5) are not defined in the paper.\"\n\n**A:** We apologize for this oversight. In Equation (5), $p_{SFT}$ refers to the policy learned during the Supervised Fine-Tuning (SFT) stage, and $p_{GRPO}$ refers to the policy optimized using Group Relative Policy Optimization (GRPO). We will ensure these terms are clearly defined in the main text of the final version.\n\n**Q:**\n>\"The methodology primarily applies GRPO for reinforcement learning training of LLMs, introducing some training tricks (e.g., optimizing dataset format, per-token reward). However, these are largely engineering solutions for GRPO training. The main contribution—applying GRPO to T2I workflows—seems somewhat limited. To elevate the paper’s quality, more innovative elements should be incorporated.\"\n\n**A:** While GRPO is an existing algorithm, its application to the sequential generation of structured ComfyUI workflows is non-trivial. As our experiments demonstrated, our approach relies on components such as improved reward attribution and a surrogate reward ensamble design, which together represent a methodological advancement for a new, challenging problem space. Moreover, the use of GRPO allows us to bypass fundamental limitations (diversity) of the best performing methods in this problem space, representing not only a quantitative improvement, but a push for a better training approach.\n\nFinally, we note that studies on non-trivial adaptations of RL methods for image generation tasks are regularly published in top tier conferences (e.g., Diffusion Model Alignment Using Direct Preference Optimization, CVPR 2024[4]).\n\n**Q:**\n> \"I think the writing quality needs improvement. The article actually uses a lot of content to describe methods for solving engineering problems, but I believe its contribution in terms of methodological innovation is limited.\" & \"Some parameters in the article may require more detailed descriptions to make the paper easier to understand.\"\n\n**A:** We appreciate the reviewer feedback on writing quality and will review the paper for areas of improvement to enhance clarity and emphasize our methodological contributions more explicitly. We agree that more detailed descriptions of certain parameters could benefit reader understanding. We will ensure that additional details on key parameters are provided in the supplementary material of the final version to improve clarity.\n\n**References:**\n\n[1] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy.\nPick-a-pic: An open dataset of user preferences for text-to-image generation. In Thirty-seventh\nConference on Neural Information Processing Systems, 2023.\n\n[2]Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng\nLi. Human preference score v2: A solid benchmark for evaluating human preferences of\ntext-to-image synthesis. arXiv preprint arXiv:2306.09341, 2023.\n\n[3]Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao\nDong. Imagereward: Learning and evaluating human preferences for text-to-image generation.Advances in Neural Information Processing Systems, 36, 2024.\n\n[4]Wallace, B., Dang, M., Rafailov, R., Zhou, L., Lou, A., Purushwalkam, S., Ermon, S., Xiong, C., Joty, S., & Naik, N. (2024). Diffusion Model Alignment Using Direct Preference Optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 8228-8238)."}]

Author rebuttal:

Meta-review: In this the paper, a reinforcement learning based method FlowRL is proposed to build an agent to automate the workflow within the ComfyUI ecosystem for text-to-image generation. The FlowRL consists of two steps, a SFT step to finetune LLM to predict workflow from a given prompt with supervised pairs, and a RL step with GRPO to finetune LLM to predict workflow with a surrogate reward (more specifically, the ensemble of multiple surrogate rewards to mitigate reward hacking).

Overall, I feel this approach of building an agent to automate the T2I generation workflow via RL with  surrogate reward is quite interesting, and has some novelty.  Even though this submission has some limitations like the proposed approach heavily depends on ComfyGen ecosystem, challenges to add new components/models, lack of methodology novelty (applying existing SFT/GRPO methods to T2I generation workflow agent), however, I still believe this paper is setting a new and important problem of T2I generation workflow agent, and proposed one of the first solutions and valuable insights. 

So I would recommend "Accept (poster)".
Decision: Accept (poster)