Title: Policy Optimized Text-to-Image Pipeline Design
Venue: NeurIPS 2025
Accepted: yes
Keywords: ComfyUI; Text-to-Image workflow design; Text-to-Image generation
Abstract: Text-to-image generation has evolved beyond single monolithic models to complex multi-component pipelines that combine various enhancement tools. While these pipelines significantly improve image quality, their effective design requires substantial expertise. Recent approaches automating this process through large language models (LLMs) have shown promise but suffer from two critical limitations: extensive computational requirements from generating images with hundreds of predefined pipelines, and poor generalization beyond memorized training examples.
    We introduce a novel reinforcement learning-based framework that addresses these inefficiencies. Our approach first trains an ensemble of reward models capable of predicting image quality scores directly from prompt-workflow combinations, eliminating the need for costly image generation during training. We then implement a two-phase training strategy: initial workflow prediction training followed by GRPO-based optimization that guides the model toward higher-performing regions of the workflow space. Additionally, we incorporate a classifier-free guidance based enhancement technique that extrapolates along the path between the initial and GRPO-tuned models, further improving output quality.
    We validate our approach through a set of comparisons, showing that it can successfully create new flows with greater diversity and lead to superior image quality compared to existing baselines.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces FlowRL, a reinforcement learning based framework for generating text-to-image workflows in ComfyUI. It uses RL-based approach for generating novel ComfyUI workflows aligned with human preferences and predicts image quality scores directly from prompt-workflow pairs, avoiding costly image generation during training. It also introduces dual-model guidance between SFT and GRPO-tuned models during inference.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. No need to generate images for scoring and obtaining rewards.\n2. Tested across multiple metrics.\n\n### Weaknesses\n1. On line 156 of the paper, perplexity reaches 1.9, but the significance of this value is not explained—e.g., no comparison with other methods.\n2. The third contribution of the paper: \"We integrate GRPO with per-token reward attribution for stable and memory-efficient policy optimization.\" seems relatively weak. The paper achieves higher batch sizes and faster training speeds by compressing JSON format, which appears to be an optimization of dataset processing rather than a methodological contribution.\n3. The paper mentions \"flow,\" but could provide more examples—such as its format and components—for clarity.\n4. One issue is that BERT learns a reward model for prompt-flow pairs, but the scores are assigned to generated images. A key question is whether higher scores for prompt-flow pairs (without image generation) equate to improved image quality. This point is not well demonstrated in the paper. Particularly in the Geneval evaluation, FlowRL performs worse than ComfyGen in counting, colors, and position.\n5. The terms p_SFT and p_GPRO in Equation (5) are not defined in the paper.\n6. The methodology primarily applies GRPO for reinforcement learning training of LLMs, introducing some training tricks (e.g., optimizing dataset format, per-token reward). However, these are largely engineering solutions for GRPO training. The main contribution—applying GRPO to T2I workflows—seems somewhat limited. To elevate the paper’s quality, more innovative elements should be incorporated.\n\nQUESTIONS:\n1. See weakness  \n\n2. I think the writing quality needs improvement. The article actually uses a lot of content to describe methods for solving engineering problems, but I believe its contribution in terms of methodological innovation is limited.  \n\n3. Some parameters in the article may require more detailed descriptions to make the paper easier to understand.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a policy optimization strategy for workflow prediction for text-to-image tasks. The idea drawing from previous work is to prompt/finetune an LLM to directly predict the ComfyUI workflow for the task of text-to-image generation. Similar to several LLM fine-tuning pipelines these days, this involves a supervised fine-tuning stage on a curated dataset of workflows, followed by training a surrogate reward model, and then a policy optimization stage (using GRPO) to maximize these rewards. Results on prompt following and other reward model evaluations indicates that the proposed method outperforms prior works exploring similar concepts for the task.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe paper makes the natural and principled step towards end-to-end optimization of the workflow generation concept (as opposed to just prompting/in-context learning) with a reward ensemble which provides a more robust optimization strategy.\n\nFurther, the paper also has a robust ablation analysis covering important components of the proposed pipeline providing convincing results. \n\nThe paper also does a good job of presenting the paper in a clear and easy to understand manner. \n\nWeaknesses:\n\nThe only major weakness I can see in the paper is that it is a \"minimal\" improvement over ComyGen in many aspects (the overall formulation, training data for SFT etc.). In terms of the overall improvements, while ComfyGen and the proposed FlowRL have the same GenEval results, the HPSv2 and the human study seem to indicate improvements for FlowRL over ComyGen. However, it might be a good idea to clarify the aspects in which FlowRL improves over ComfyGen and how these might not be captured by GenEval.   \n\n\nOverall, I do think that the paper broadly is a step in the right direction and therefore would merit acceptance to me, however I do think it would be a good idea to clarify both the comparisons, evaluations, and have better analysis on the source of improvements/novelty of generated workflows.\n\nQUESTIONS:\nThere are 2 major aspects where I am curious to see if there are improvements possible:\n\na) Choice of Evaluation: While GenEval provides a clean benchmark for prompt following on relatively short and basic prompts, the ideal usecases of FlowRL/ComfyGen etc. are probably quite different. Would it be possible to have more meaningful evaluations of the proposed methods beyond the standard evaluations provided in the paper to the kind of tasks that an end user of such a framework might actually care about?\n\nb) Novelty of Generated Flows/Source of Improvements: An interesting aspect of end-to-end optimization is that we might observe novel sources of improvements. While the analysis in Tab. 2 is a first step in this direction (observing the novel generated flows), it might be worthwhile to have a deeper dive into this to see the exact nature of these novel flows to see what causes the underlying improvements?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces FlowRL, a reinforcement learning based framework designed to automate the text-to-image generation workflow within the ComfyUI ecosystem. FlowRL is consisted of 2 stages: a supervised fine-tuning on dataset of unscored prompt-flow pairs to learn structural priors, and a RL optimization using group relative policy optimization (GRPO), which predict image quality directly from prompt-flow pairs instead of generated images. In addition, the model employs a classifier-free guidance mechanism at inference to balance quality and diversity. Experimental evaluations show that FlowRL outperforms baseline methods.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n- The paper has strong empirical validation. The proposed method is comprehensively benchmarked against baselines with GenEval and HPSv2 metrics. The experiments show better performance than previous methods.\n- The paper shows good technical innovation. The proposed RL framework overcomes the limitation of prior model (ComfyGen) and improves efficiency and scalability. By introducing a surrogate reward model ensemble, the authors effectively eliminate the need for image generation during training.\n- The paper has good concept of framing of workflow generation as a sequential RL task using LLMs is a fresh and significant departure from prior classifier-based or static methods.\n\nCons:\n- The method could have limited domain generalization since it is focused on ComfyUI-based text-to-image pipelines. The generalization to other domains is not tested and discussed in depth. Additionally, since the model builds heavily on the ComfyGen dataset, the reproducibility is limited.\n\nQUESTIONS:\n- Can the authors elaborate on how transferable FlowRL is to other domains or platforms?\n- How does the performance of FlowRL scale as the diversity of prompts and flow complexity increases? Is there evidence of reward model generalization in unseen domains or under distribution shifts?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new framework that uses reinforcement learning to automate the design of complex text-to-image generation pipelines, including a surrogate reward model, two-stage training strategy and dual model guidance. The experimental results demonstrate that the proposed approach not only generates superior visual quality compared to existing methods but also exhibits better diversity.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- This paper combines multiple advanced techniques when building the system, including a two-stage training process, first SFT then GRPO, and dual model guidance to further improve generation quality.\n- The introduction of surrogate reward model predicts image quality directly from a prompt-workflow pair, which avoids the expensive step of generating an image for every candidate workflow and enables efficient RL training.\n- Extensive experimental results are provided, including automated metrics in GenEval and HPSv2, user study, and thorough ablation study. \n\nWeaknesses:\n- This paper is actually building an agentic system. However, every time a new model or component is added, a new round of training is needed. This limitation will make the agentic system lose its inherent advantages\n- The performance of the entire system, especially the surrogate reward model, is heavily depending on the quality and diversity of the score dataset from ComfyGen. Any biases or gaps in this dataset will accumulate to the final system.\n\nQUESTIONS:\n- This paper demonstrates that the dual-model guidance technique improves performance by trading off diversity. Have the authors tried dynamically adjusted guidance strength, just like the recent developmenet in CFG[1] to achieve a better balance between quality and diversity ?\n- Given the weaknesses discussed above, have the authors considered about how the framework can be extended to add new components ? For instance, could new workflow nodes be added by fine-tuning only a small part of the reward models ?\n\nReference:\n\n[1]  Analysis of classifier-free guidance weight schedulers\n\nLIMITATIONS:\nYes, the authors discuss the limitations in Section 5.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-06"}