ComfyGen: Prompt-Adaptive Workflows for Text-to-Image Generation

## Abstract

Abstract The practical use of text-to-image generation has evolved from simple, monolithic models to complex workflows that combine multiple specialized components. While workflow-based approaches can lead to improved image quality, crafting effective workflows requires significant expertise, owing to the large number of available components, their complex inter-dependence, and their dependence on the generation prompt.
Here, we introduce the novel task of prompt-adaptive workflow generation , where the goal is to automatically tailor a workflow to each user prompt.
We propose two LLM-based approaches to tackle this task: a tuning-based method that learns from user-preference data, and a training-free method that uses the LLM to select existing flows. Both approaches lead to improved image quality when compared to monolithic models or generic, prompt-independent workflows. Our work shows that prompt-dependent flow prediction offers a new pathway to improving text-to-image generation quality, complementing existing research directions in the field.

## 1 Introduction

As the field of text-to-image generation (Rombach et al., ; Ramesh et al., ) matures, researchers and practitioners shift from simple, monolithic workflows to more complex ones. Instead of relying on a single model to produce an image, those advanced workflows combine a variety of components, or blocks, designed to enhance the quality of the generated image (AUTOMATIC1111, ; Zhang, ; comfyanonymous, ).
These components may include fine-tuned versions of the generative model, large language models (LLMs) for refining the input prompt, LoRAs (Luo et al., ; Ryu, ) trained to correct poorly generated hands or to introduce specific artistic styles, improved latent decoders for creating finer details, super resolution blocks, and more.

Importantly, effective workflows are prompt-dependent. The choice of blocks often depending on the text prompt and the content of the image being created.
For example, a workflow aimed at emulating nature photographs may elect to use a model fine-tuned for photorealism, while workflows focused on generating human images often contain the term “bad anatomy” as a negative prompt or leverage specific super-resolution models that also correct distorted facial features, such as the eyes.
Due to the richness of available blocks and complexity of workflows, building a well-designed workflow often requires considerable expertise.

In this work, we propose to learn how to build text-to-image generation workflows, conditioned on a user prompt. Specifically, we propose to leverage an LLM to take as input a prompt describing an image, and output a workflow that is specifically tailored to that prompt. Below, we outline two approaches to achieving this goal.
The prompt-specific workflow can then be used to synthesize images for that prompt, resulting in improved quality compared to using fixed base models or popular human-crafted workflows. Importantly, using an LLM enables the model to leverage its extensive prior knowledge to parse the prompt and match its content to the most appropriate blocks.

To represent flows, we build on ComfyUI (comfyanonymous, ), a widely used tool that stores workflows as JSON files, which can be easily parsed by recent LLMs. The popularity of ComfyUI also provides access to multiple human-created workflows, which we then augment to create a more diverse training set.
To teach the LLM the link between flow components and image quality, we collect $500$ diverse prompts from human users.(^1^11Sampled from [https://civitai.com/](https://civitai.com/) after filtering out NSFW content.) These prompts are used for generating images using each workflow in our training set, and the results are scored by an ensemble of aesthetic predictors and human preference estimators (Kirstain et al., ; Xu et al., ; Wu et al., ). This process effectively creates a training set composed of triplets of (prompt, flow, score).

We then consider two approaches for matching flows to novel prompts. In the first, we leverage a closed-source LLM, and provide it with a table of flows and their scores across a closed-set of categories automatically derived from our training prompts. This table serves as a context for a followup request, where we ask the LLM to select the flow that is most suitable for a novel prompt. In the second approach, we fine-tune an open LLM (Dubey et al., ) so that, given a prompt and an ensemble score, it predicts the flow that achieved that score. During inference, we provide the LLM with an unseen prompt and a target score and ask it to provide us with an appropriate workflow. We name these approaches ComfyGen-IC and ComfyGen-FT respectively. The design choices behind each approach and their motivations are discussed below.

We compare our prompt-adaptive approach against several baselines, including: (1) single-model approaches (the baseline SDXL model (Podell et al., ), popular fine-tunes, and a DPO-optimized version (Rafailov et al., ; Wallace et al., )), and (2) prompt-independent, popular workflows.
ComfyGen outperforms all baselines on both human-preference and prompt-alignment benchmarks, highlighting the benefit of prompt-dependent flows.

Finally, we analyze the workflows selected by our method, demonstrate their relation to the domains represented in the input prompts, and investigate the scaling behaviors of our model.

## 2 Related work

#### Improving Text-to-image generation quality.

With the growing popularity of text-to-image diffusion models (Rombach et al., ; Nichol et al., ; Ramesh et al., ), a range of works sought to improve the visual quality of their outputs, and their alignment to human preferences.

One approach is to fine-tune pretrained models using curated, high quality datasets or improved captioning techniques (Dai et al., ; Betker et al., ; Segalis et al., ). Instead of collecting data, a range of works use reward models (Kirstain et al., ; Wu et al., ; Xu et al., ; Lee et al., ) to guide text-to-image generation. This can be done using reinforcement-learning (Black et al., ; Deng et al., ; Fan et al., ; Zhang et al., ). However, these methods can be computationally expensive and struggle to generalize effectively. As an alternative, the model can be fine-tuned using differentiable rewards (Clark et al., ; Prabhudesai et al., ; Wallace et al., ). Instead of tuning the model directly, one can also use reward models to explore the diffusion input-noise space (Eyring et al., ; Qi et al., ), finding seeds for which the output is of higher quality. Finally, some approaches leverage self-guidance (Hong et al., ) or frequency-based feature manipulations (Si et al., ; Luo et al., ) to drive the model towards more detailed and sharper outputs.

Our work proposes a new, orthogonal path to improving image quality. Instead of modifying the diffusion model or intervening in its sampling process, we use reward models to better match workflow components to a given prompt, aligning the entire pipeline towards human preferences.

#### LLM-based tool selection and Agents

Recent advancements in large language models have demonstrated significant improvements in reasoning abilities and their capacity to adapt to novel content and tasks.
This adaptability can be achieved through efficient fine-tuning methods, but more commonly simply through zero-shot prompting or in-context learning.

Building on these capabilities, a range of works proposed to leverage LLMs for tasks beyond text generation. A common line of work aims to equip the LLM with external tools (Schick et al., ), either through appropriate API tags within the generated text (Schick et al., ), by providing in-context API documentations (Wang et al., ; Surís et al., ), model descriptions (Shen et al., ) and code samples (Gupta & Kembhavi, ), or by retrieving models from a pre-defined collection. (Wu et al., ). Such tools are often referred to as LLM agents, and their latest variants are often equipped with components such as memory mechanisms, retrieval modules or self-reflection and reasoning steps, all aimed at improving their overall performance.

Our work can similarly be viewed as an agent, as it employs an LLM to directly select and connect external tools. Here, we focus on the novel task of prompt-adaptive pipeline creation, and on tapping this under-explored path to improving the quality of downstream generations.

#### Worfklow generation

An emerging trend in machine learning research is the use of compound systems, where multiple models are used in collaboration to achieve state-of-the-art results. These systems have been successfully used across various domains, ranging from coding competitions (AlphaCode Team, ) to olympiad-level problem solving (Trinh et al., ), medical reasoning (Nori et al., ) and video generation (Yuan et al., ).
Crafting such compound systems can be a daunting task, as the components must be carefully selected and their parameters tuned to perform well in tandem,
rather than optimized on each individual step of the task.
To address this, recent approaches have proposed optimization-based frameworks that tune pipeline parameters for improved end-to-end performance (Khattab et al., ), or even optimize the connections within a graph representing the components of a complex system (Zhuge et al., ).

Our work similarly tackles the task of pipeline generation. Here, we focus on text-to-image models, and demonstrate that their performance can be enhanced by designing compound pipelines that depend on the user’s prompt.

## 3 Method

Given an input prompt describing an image, our goal is to match it with an appropriate text-to-image workflow, leading to improved visual quality and prompt alignment. We hypothesize that effective workflows will depend on the specific topics present in the prompt. Therefore, we propose to tackle this task by leveraging an LLM that can concurrently reason over the prompt and identify these topics, while also serving as a means to directly select or synthesize the new flow.

In the following section, we provide a detailed description of ComfyUI and our method, focusing on our training data, as well as our retrieval-based and score-based tuning approaches.

Figure: Figure 2: (a) A simple ComfyUI pipeline using a base model and a face restoration block, as well as both a positive and a negative prompt. (b) Distribution of scores for the prompt, flow pairs in our training set. (c) Example images produced for the same prompt by flows with different scores. A higher score typically correlates with more detailed and vibrant results, and fewer artifacts.
Refer to caption: extracted/5894234/figures/flow_example/pipeline_image_thin.jpg

### 3.1 ComfyUI

Our work focuses on ComfyUI, an open-source software for designing and executing generative pipelines.
In ComfyUI, users create pipelines by connecting a series of blocks that represent specific models or their parameter choices. In fig. 2a, we show a simple example ComfyUI pipeline.
This pipeline includes blocks for loading a model, specifying prompts and latent dimensions, a sampler, a VAE decoder, and a face restoration upscaling model. More complex pipelines may involve additional components like LoRAs (Ryu, ) or embeddings (Gal et al., ), ControlNets (Zhang et al., ), IP-Adapters (Ye et al., ), blocks that re-write and enhance the input prompt, and more. In many cases, complex blocks are introduced into ComfyUI through user-created extensions, which are then shared across the community.

Importantly, each ComfyUI pipeline can be exported to a JSON file which outlines both the graph nodes and their connectivity. ComfyUI’s standard JSON format also contains UI information, such as the position and color of the blocks. We use the simpler API version which excludes this UI-specific information. Moreover, the API format of the flow can be used to trigger novel generations without using the UI, allowing us to automate much of our process.

We note that the concurrent work of Xue et al. () also leverages ComfyUI pipelines. However, their work focuses on using ComfyUI as a test bed for exploring the stability of collaborative workflow generation approaches. Hence, their evaluation focuses on examining the rate at which ComfyUI-compliant workflows are created. In contrast, we focus on learning to tailor specific workflows to a user’s prompt, with the aim of improving downstream generation quality.

### 3.2 Training Data

As a starting point, we collect a set of approximately $500$ human-generated ComfyUI workflows from popular generative-resource-sharing websites such as Civitai.com. We limit ourselves to text-to-image workflows, flitering out video generation flows, and flows that take a control image as an input. We further discard highly complex flows, whose JSON representations often span tens of thousands of lines. Finally, we discard flows that use community-written blocks appearing in fewer than three flows. This leaves us with a small set of $33$ flows, which we augment by randomly switching diffusion models (see supplementary for list), LoRAs and samplers, or changing parameters like the guidance scale and number of steps. In total, this process resulted in $310$ distinct workflows.

Recall that our goal is to predict effective flows for a given prompt, which will enhance the quality of the generated output. To achieve this, we need a way to assess workflow performance. To do so, we begin by collecting a set of $500$ popular prompts from Civitai.com and using them to synthesize images with each flow in our dataset. These images are then scored using an ensemble of quality prediction models (LAION Aesthetic Score (Schuhmann et al., ), ImageReward (Xu et al., ), HPS v2.1 (Wu et al., ), and Pickscore (Kirstain et al., )). We standardize the outputs of these models so that they are of approximately the same scale, and sum them up, assigning higher weights to models that better align with human preferences. This process yields a single scalar score for each prompt and flow pair, where higher scores typically correlate with better image quality. Figure 2b,c shows the distribution of scores in our data set, along with visual examples of images created with the same prompt, across a range of scores.

Our final dataset consists of triplets of prompt, workflow, and score. We use these to implement both the in-context and the fine-tuning based approaches detailed below.

### 3.3 ComfyGen-IC

As a first approach to providing prompt-dependent flows, we look to in-context based solutions that leverage a powerful, closed-source LLM. To do so, we first need to provide the LLM with some knowledge about the quality of results produced by each flow. We thus start by asking the LLM to come up with a list of $20$ labels which will best fit our $500$ training prompts. These include object-categories (“People”, “Wildlife”), scene categories (“Urban”, “Nature”) and styles (“Anime”, “Photo-realistic”). A complete list of the labels is provided in the supplementary.
With these labels in hand, we can now calculate the average quality score of images produced by each flow across all prompts belonging to a specific label. Repeating this for all flows and all labels gives us a table of flows and a measure of their performance across all $20$ categories.

Ideally, we would have liked to provide the LLM with the full JSON description of the flows, allowing it to learn the relationships between flow components and downstream performance on specific categories. Unfortunately, the flows are too lengthy to fit more than a handful into the context window of most LLMs. Hence, our table contains only flow identities, and we simply ask the LLM to choose the flow that it believes will perform best on a given, unseen prompt.

All in all, this approach provides us with a classifier capable of parsing new prompts, breaking them down into relevant categories, and selecting the flow that best matches these categories. We name this variation ComfyGen-IC.

### 3.4 ComfyGen-FT

As an alternative approach, we can fine-tune an LLM to predict high-quality workflows for given prompts. One way to approach this problem could be to simply fine-tune the LLM so that, given an input prompt provided in-context, it would need to predict the flow that achieved the highest score for that prompt. However, this approach has several drawbacks: it significantly reduces the number of training tokens, using only one flow per prompt instead of all $310$; it’s more sensitive to randomness in our data creation process, such as the seed chosen for each generated image; and it doesn’t allow learning from negative examples, which could help the model identify ineffective flow components or combinations.

Instead, we propose an alternative fine-tuning formulation where the LLM takes as its context both the prompt and a score. The model is then tasked with predicting the specific flow that achieved the given score for the corresponding prompt.
This approach tackles the drawbacks of the best-scoring-flow method. First, it greatly increases the number of tokens available for training by utilizing all available data points, not just the highest-scoring ones. Second, it reduces the impact of random fluctuations by considering a wider range of scores and their associated flows. Third, it allows the model to learn from negative examples, i.e., flows that achieved low scores for a given prompt.
At inference time, we can simply provide the LLM with a prompt and a high score and have it predict an effective flow for our prompt. We name this variation ComfyGen-FT.

### 3.5 Implementation details

We implement ComfyGen-IC using Claude Sonnet 3.5, and ComfyGen-FT on top of pre-trained Meta Llama3.1 8B and 70B checkpoints (Dubey et al., ). Unless otherwise noted, all ComfyGen-FT results in the paper use the 70B model with a target score of $0.725$. In all cases, we finetune for a single epoch using a LoRA of rank $16$ and a learning rate of $2e-4$. Additional details are in the supplementary.

## 4 Experiments

Figure: Figure 3: Our method can generate higher quality images across diverse domains and styles. Prompts are available in the supplementary.
Refer to caption: extracted/5894234/figures/ours_qualitative/claude/the_entire_observable_universe_in_a_single_bottle,_Dreamlike,_Surreal_landscapes,_Mystical_creatures.jpg

We begin by showcasing images generated with our approach across a range of prompts, including subject-focused, photo-realistic imagery, as well as artistic or abstract creations. These are shown in fig. 3, with additional large-scale figures in the supplementary.

Next, we compare the images produced by our approach with those generated by a series of baselines. We consider two types of alternative approaches: (1) Fixed, monolithic models, where we simply use the prompts to directly condition a pre-trained diffusion model. (2) Generic workflows, where we use the same workflow to generate all images, regardless of the specific prompt.

For (1), we consider the original SDXL model (Podell et al., ) and its two most popular (most downloaded on CivitAI) fine-tuned variations: JuggernautXL and DreamshaperXL. We further consider a version of SDXL fine-tuned with DPO (DPO-SDXL, (Wallace et al., )). For (2), we selected the two most popular flows (based on download counts) from our training corpus. These flows use SSD-1B (Gupta et al., ) and Pixart-$\Sigma$ (Chen et al., ) respectively.

We evaluate our models and the baselines on two fronts. First, we use the GenEval benchmark (Ghosh et al., ), which uses object detection to assess generative models across prompt-alignment tasks like single-object generation, counting, and attribute binding. Quantitative results are reported in table 1 with qualitative samples shown in fig. 4. Our tuning-based model outperforms all baselines, despite only using human preference scores during training. The in-context approach underperforms. We hypothesize that it suffers due to GenEval’s short, simplistic prompts, which typically only describe a few objects in one or two words each. This challenges the LLM’s ability to match the prompts with labels, and performance degrades.

**Table 1: GenEval comparisons. ComfyGen-FT outperforms all baseline approaches, despite being tuned with human preference scores, and not strictly for prompt alignment.**
|  | Single | Two |  |  |  | Attribute |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Model | object | object | Counting | Colors | Position | binding | Overall |
| Single Model - SDXL | 0.98 | 0.74 | 0.39 | 0.85 | 0.15 | 0.23 | 0.55 |
| Single Model - JuggernautXL | 1.00 | 0.73 | 0.48 | 0.89 | 0.11 | 0.19 | 0.57 |
| Single Model - DreamShaperXL | 0.99 | 0.78 | 0.45 | 0.81 | 0.17 | 0.24 | 0.57 |
| Single Model - DPO-SDXL | 1.00 | 0.81 | 0.44 | 0.90 | 0.15 | 0.23 | 0.59 |
| Fixed Flow - Most Popular | 0.95 | 0.38 | 0.26 | 0.77 | 0.06 | 0.12 | 0.42 |
| Fixed Flow - 2nd Most Popular | 1.00 | 0.65 | 0.56 | 0.86 | 0.13 | 0.34 | 0.59 |
| ComfyGen-IC (ours) | 0.99 | 0.78 | 0.38 | 0.84 | 0.13 | 0.25 | 0.56 |
| ComfyGen-FT (ours) | 0.99 | 0.82 | 0.50 | 0.90 | 0.13 | 0.29 | 0.61 |

Figure: Figure 4: Qualitative results on GenEval prompts. ComfyGen shows better performance on multi-subject prompts, colorization and attribute binding, but may struggle with positioning.
Refer to caption: extracted/5894234/figures/geneval/sdxl/a_photo_of_a_cake_and_a_stop_sign_00001_.jpg

To better evaluate the visual quality of our images, we turn back to CivitAI, and sample $500$ prompts from the $10,000$ highest-ranked images on the website, after filtering out prompts used for training our model and prompts that contain nudity or excessive violence. We assess the results both automatically — using HPS V2.0, a model not included in the weighted score used during training — and through a human preference study.
For HPS, we follow Wallace et al. (); Qi et al. () and perform a pair-wise comparison, each time pitting our method against a baseline using the same prompt. We report the fraction of prompts for which our approach received a higher score.

For the user study, we pit each version of our model against each baseline in a two-alternative forced-choice setup and report the fraction of times our model was preferred over each baseline. We sample roughly $20$ prompts for each baseline and ComfyGen version pair, for a total of $231$ questions. We collected a total of $682$ responses from $35$ users. More details are provided in the supplementary.

Figure: Figure 5: HPS V2.0 and User Study win rates. We compare each baseline against both ComfyGen-FT (green) and ComfyGen-IC (teal). ComfyGen variants are favored over all baselines.
Refer to caption: x3.png

The results on the CivitAI prompts are shown in fig. 5, with visual samples for our approach and the $4$ best baselines provided in fig. 6.
Both of our approaches outperform all baselines, with notable improvement over simply using the baseline SDXL model. Curiously, we observe
that some fine-tuned versions of SDXL are competitive with fixed flows, further emphasizing the importance of tailoring flows to specific use cases.

Figure: Figure 6: Qualitative results on CivitAI prompts. Please zoom in to better appreciate the differences.
Refer to caption: extracted/5894234/figures/civit/sdxl/8k_digital_nature_photography,_an_idyllic_landscape_with_mountains_in_the_distance,_shiny_ruby_s.jpg

## 5 Analysis

Having shown that our approach outperforms existing baselines, we next turn to analyzing its behavior.
We examine three aspects of ComfyGen’s performance: (1) the originality and diversity of the generated flows, (2) whether they show human-interpretable patterns, and (3) the effect of using the target score in the ComfyGen-FT prompts. The findings for these aspects are reported below.

### 5.1 Originality and diversity

We begin by assessing ComfyGen-FT’s ability to generate novel flows by comparing its predictions for the $500$ CivitAI prompts to the nearest neighbors in our training corpus. While ComfyGen-IC is retrieval-based and has an expected similarity score of 1.0, ComfyGen-FT achieves 0.9995 similarity, indicating that at the scale of our model, there is little to no generation of unseen flows. This indicates that our fine-tuning approach has also learned to classify flows. However, we note that in contrast to ComfyGen-IC, it has learned so directly from the data with limited ad-hoc choices in the process, and indeed it outperforms ComfyGen-IC in most of our evaluations. Looking ahead, we hope that future methods will also be able to synthesize unseen flows with novel graph structures.

In terms of diversity, we observe that over $500$ prompts, ComfyGen-IC makes use of $41$ unique flows, while ComfyGen-FT uses $79$, indicating a higher diversity. Recall also that our base set contained only $33$ human created templates, which were then augmented through random parameter changes. Hence, both variations identified useful flows which differ from the initial human-created set. This suggests that more data or a more involved search over the input parameter space could yield more diverse outputs and possibly improved performance.

### 5.2 Analyzing the chosen flows

Next, we want to see whether we can identify any patterns in the chosen flows that would provide useful information about the strengths of existing models. Towards this goal, we want to see which models are popular across different categories, and which ones are especially prevalent for prompts within a particular category.

To identify models most strongly associated with specific labels, we parse all flows selected for our $500$-prompt test set. We scan each flow for base models, LoRAs, and upscaling models, appending their names to a to a document corresponding to each label associated with the prompt that generated the flow.
Then, we use TF-IDF (Sparck Jones, ) to rank the models across these label-documents. In table 2, we report the top-scoring model for each of four distinct labels, as well as the most common models across the entire flow set (“General”).

We observe that, in many cases, the choices make intuitive sense. For example, the GFPGAN face restoration model is closely tied to the “People” category. Similarly, “Anime” prompts make more frequent use of models that better preserve human anatomy, or a LoRA tuned for an anime aesthetic. However, while such patterns exist in the data, the choices are not always intuitively clear. In the future, it may be beneficial to have the LLM explain the reasoning behind its component selections.

**Table 2: Top workflow components by TF-IDF scores for selected categories. In many cases, selections align with human intuition (e.g., a face upscaling model is favored for the “People” category).**
| Category | “People” | “Nature” | “Anime” | “Abstract” | General |
| --- | --- | --- | --- | --- | --- |
| Top Base Model | Proteus v3 | Stable Cascade | JibMixXL v9 “Better Bodies” | SDVN7-NijiStyleXL | crystalClearXL |
| Top LoRA | SDXL FaeTastic v24 | Add-Detail XL | AnimeTarot | LogoRedmond | MidJourney52 v1.2 |
| Top Upscaler | GFPGAN v1.4 | Real-ESRGAN | UltraSharp x4 | None | UltraSharp x4 |

### 5.3 The effect of target scores

Figure: Figure 7: (left) Average HPS V2 score on CivitAI prompts as a function of the inference target score. The 8B and 70B variations of our model perform equally well, and significantly outperform the variation trained to predict the highest scoring flow (green). (right) Model outputs for the same prompt at different target scores.
Refer to caption: x4.png

Recall that ComfyGen-FT was fine-tuned to predict a flow based on a given prompt and a target score. Here, we examine the performance of the model according to the target score provided at inference time.
To do so, we repeat the CivitAI prompt experiments of section 4, while adjusting the target score used in our prompts.
We evaluate both a model tuned from the Llama3.1 8B version and one from Llama3.1 70B.
The quality of the generated images is assessed using HPS v2.0, and we report the average outcomes. The results are presented in fig. 7.
For reference, we provide the scores of the baseline SDXL model, as well as ComfyGEN-IC. We additionally examine a scenario where instead of tuning the model to predict a flow given a prompt and a score, we simply tune it to predict the highest scoring flow (”Predict best”).

We observe that the ComfyGen-FT model has indeed learned to associate the target scores with flows of varying quality. With an appropriate choice of score (near the top of the training score distribution), ComfyGen-FT achieves comparable performance to ComfyGen-IC. Notably, attempting to predict the best model instead of the score-based tuning leads to greatly diminished performance, highlighting the importance of our approach. We further note that both model sizes achieve comparable performance, hinting that we are far from saturating the capabilities of the models.

Finally, it is important to note that our target scores were calculated using an ensemble of human-preference predictors, excluding HPS v2.0. Therefore, there should be no expectation for alignment between the values on the y-axis and the x-axis.

## 6 Limitations

While ComfyGen’s prompt-dependent workflow approach demonstrates improvements over monolithic models and constant flows, it is not free of limitations.
Our current model is limited to text-to-image workflows, and cannot address more complex editing or control-based tasks. However, this could potentially be resolved in the future through the use of vision-language models (VLMs).

Similarly, both of our approaches require us to generate images using a large number of flows. With typical generations taking an order of $15$ seconds, even a modest set of $500$ prompts and $300$ flows requires a month of GPU time to create. Therefore, scaling up the approach would likely require significant computational resources or more efficient ways (e.g., Reinforcement Learning) to explore the flow parameter space.

Finally, each of our two methods has its own unique drawbacks.
The fine-tuning approach cannot easily generalize to new blocks as they become available, requiring retraining with new flows that include these blocks.
On the other hand, the in-context approach can be easily expanded by including the new flows in the score table provided to the LLM. However, this increases the number of input tokens used, making it more expensive to run and eventually saturating the maximum context length.
We hope that these limitations can be addressed through more advanced retrieval-based approaches or through the use of collaborative agents.

## 7 Conclusions

We introduced the task of prompt-adaptive workflow generation, and presented ComfyGen - a set of two approaches that tackle this task. Our experiments demonstrate that such prompt-dependent flows can outperform monolithic models or fixed, user created flows, in a sense providing us with a new path to improving downstream image quality.

In the future, we hope to further explore prompt-dependent workflow creation methods, increasing their originality and expanding their scope to image-to-image or even video-related tasks. Perhaps in the future we could collaborate with the language model on the creation of such flows, providing it feedback through additional instructions or examples of outputs, thereby enabling non-expert users to further push the boundary of content creation.

## 8 Acknowledgements

We would like to thank Yotam Nitzan, Ron Mokady, Yael Vinker and Linoy Tsaban for providing feedback on an early version of this manuscript or its figures. We thank Shahar Sarfaty for his assistance in collecting user prompts, Assaf Shocher, Yoad Tewel and Tomer Wolfson for useful discussions, and Yaki Tebeka for his assistance with compute infrastructure.

A very special thanks to Or Patashnik for always lending a willing ear, and for providing moral support throughout the project.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 1.	It is not entirely clear how the author selected 500 workflows and what rules they followed.
2.	Does extending only 33 streams to 310 streams using a data augmentation approach reduce diversity?
3.	What is the definition of generation quality in the article? Should the definition be clearly described, and is there a comprehensive consideration of image quality rating and aesthetic rating?
4.	It is still uncertain which of the two ship images in Figure 2c has better quality.
5.	What basis do the selected 20 tag categories depend on?
6.	COMFYGEN-IC seems to allow LLM to directly infer statistical distributions, so would the lack of input on the specific content of JSON harm LLM's understanding?
7.	Can you develop a new structured representation that falls between text prompts and workflow JSON descriptions? This would pique my interest and potentially solve the issue of excessively long tokens. Of course, COMFYGEN-FT also solves the token issue.
soundness: 2
strengths: The article defines an innovative text2image workflow that approaches text-to-image generation from a new perspective. Automated generation of JSON files utilizing the powerful structured data understanding capabilities of large language models. The results of comparative experiments and user testing show that the method proposed in this article has achieved better performance. From the current scale of data collection, this method will have a certain potential when more data pairs are screened out in subsequent work. The writing logic of the article is relatively clear.
summary: This paper introduces a novel approach to text-to-image generation by developing adaptive workflows that cater to specific user prompts, enhancing the quality of generated images. The authors propose two Large Language Model (LLM)-based methods: ComfyGen-IC, which uses an in-context learning approach, and ComfyGen-FT, which involves fine-tuning an LLM. These methods are designed to automatically synthesize workflows that better match user prompts, improving the standard approach of using a single, monolithic model.
weaknesses: 1. The article as a whole tends to emphasize engineering design. If more theoretical analysis and qualitative discussions can be introduced (such as analyzing different data selection methods), it will enhance the value of the article.
2. The visualization of images in the article is insufficient. Providing more results for text2 images and the workflow proposed in the article can alleviate readers' concerns about the diversity and overall quality of the workflow, although the author has already discussed the issue of diversity.
3. The article contains a large amount of data filtering, and it is recommended to provide charts to more clearly describe the filtering rules and process.
4. The lack of a detailed description of evaluation indicators in the main text affects the understanding of experimental results.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Line 140 has a missing citation.

Footnote #1 – please define NSFW

Would it be possible to see user study comparing ComfyGen-FT to ComfyGen-IC?

What are the metrics reported in Table 1? Could the authors clarify?

Adding more metrics would strengthen the validation too. How is the introduced method performing in terms of MSCOCO validation set FID? How about CLIP score?
soundness: 2
strengths: The paper is well written and easy to follow.

The idea of automatizing flows with an LLM is interesting.

Qualitative results look nice.
summary: The paper considers image generation with workflow systems such as ComfyUI. The main issue when using the workflow systems is the challenge of developing a valid flow that would lead to high quality image generation. The suggested approach leverages a LLM to generate a prompt conditioned flow. In particular, the paper is exploring two approaches: finetuning and in-context learning. Automatic and human evals show that the proposed approaches are outperforming fixed flows as well as off-the-shelf models.
weaknesses: The way the paper is presented it assumes that the reader is familiar with ComfyUI framework. However, the paper would be more complete when presenting more details about this framework, e.g., hos difficult it is to design a new flow in ComfyUI? As the reviewer never sued this environment, it is not easy to assess fully the value and complexity of the contribution.

The validation of the proposed method only includes a handful of models. Would it be possible to include more recent models such as FLUX and SD3? How strong the fixed flow baselines are? Overall, it is not easy to assess the significance of the reported results. It would also be nice to see more details about the validation – see some of the questions below.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. It would be valuable to explore how LLMs can be leveraged to generate a greater variety of new workflows, which would enhance the significance of this paper's contributions. This is my primary concern.
2. What are the results if ComfyGen-FT utilized higher scores during inference?
3. Why do the qualitative results for ComfyGen-FT show discrepancies in the number of generated objects, and what explanations can be provided for the identified issues related to "positioning"?
soundness: 3
strengths: 1. This paper is the first to identify the challenges associated with using workflows for creative tasks, addressing a valuable issue. And it  leverage LLMs to automate this process, which will significantly assist the community, particularly beginners, in enhancing their creative output.
2. The descriptions of ComfyGen-IC, ComfyGen-FT, and the dataset setup are clear and well-designed.
3. ComfyGen-FT demonstrates superior performance compared to other methods, including fixed models and commonly used workflows. Furthermore, ComfyGen-FT is trained on diverse scores while simultaneously learning from both positive and negative examples, which contributes positively to the model training process.
4. The results of the user study underscore the effectiveness of the proposed method, and the analysis section provides meaningful insights that enhance the understanding of this approach.
summary: This paper proposes a prompt-adaptive workflow generation approach that utilizes LLMs to automate the workflows, thereby assisting users in their creative processes. The authors implement both retrieval-based and fine-tuning methods, building a comprehensive dataset consisting of <prompt, workflows, score>. The experiments demonstrate that the proposed method outperforms existing approaches. Additionally, a detailed user study is conducted, and the authors analyze the originality, diversity, and human-interpretable patterns of the method.
weaknesses: 1. ComfyGen-IC is merely a retrieval-based method utilizing LLMs, and the diversity of workflows generated by both methods is somewhat limited. 
2. The user study involved only 35 participants, which may limit the scalability of the findings.
3. There is a lack of discussion regarding the changes and corresponding explanations that would arise from using a higher score (rather than 0.725) in ComfyGen-FT during inference, as a higher score should correlate with better quality outputs.
4. In the qualitative results for ComfyGen-FT, the number of generated objects often appears incorrect, and there is insufficient explanation regarding the noted issues with "positioning" mentioned in the text.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 4
questions: 1.	Could the authors clarify if any workflow validity checks are in place to ensure that generated flows will execute correctly in the ComfyUI environment? This would improve the method’s reliability.
2.	Given the high computational demands of ComfyGen-FT, would it be feasible to use a simpler classification model (8B model seems to be on par with 70B version) for selecting workflows? Since ComfyGen-IC functions similarly to retrieval, could lightweight models be considered?
3.	Does the limited diversity of workflows impact the model's generalizability for different prompt domains? It would be helpful to consider additional ways to expand flow diversity beyond parameter tuning, possibly by creating new flow templates or expanding the set of base workflows.
4.	The model’s component choices are not always intuitive, especially when associating specific LoRAs and upscalers with certain prompt types. Explainability mechanisms, such as scoring explanations, could help users understand why specific components are chosen for particular prompts, potentially increasing trust in the model's outputs. The authors can consider using more illustrations to show which model/LoRA/hyperparameters works better in which scenarios.
soundness: 3
strengths: 1.	Originality: The approach is novel in treating the text-to-image generation as a prompt-specific workflow generation task rather than using single, static models or workflows. By utilizing LLMs to dynamically create workflows based on prompts, this research introduces a new way to improve image quality.
2.	Quality: The quality of the work is strong, with well-designed experiments, clear distinctions between the ComfyGen-IC and ComfyGen-FT methods, and comparative baselines. Evaluation on both quantitative and qualitative benchmarks strengthens the work’s validity.
3.	Clarity: Overall, the paper is well-written, explaining both the advantages and challenges of prompt-adaptive workflows. Diagrams and tables are well-used to illustrate performance differences, and the methodology is well-detailed.
4.	Significance: The significance lies in expanding the text-to-image generation field by proposing a workflow-based approach that could potentially improve downstream applications. The work suggests a pathway for adaptive pipeline design in generative AI, a promising direction for improved customization in user-generated content.
summary: This paper introduces a new approach to text-to-image generation by dynamically adapting workflows to input prompts, rather than using monolithic models. The authors present two methods for workflow generation: ComfyGen-IC (an in-context prompt classifier) and ComfyGen-FT (a fine-tuned model for predicting workflows based on prompt and desired image quality score). By leveraging ComfyUI, which stores workflows as JSON files, the model maps prompts to workflows. The evaluations based on preference estimation models and user study are performed to validate the superiority of the proposed pipeline over fixed workflows and single-model baselines.
weaknesses: 1.	High Resource Requirements: ComfyGen-FT, in particular, requires extensive resources due to the need for numerous training data generated with different prompts and flows, with each generation taking around 15 seconds. This high computational overhead could be a limitation in scaling and adoption, given the user prompts and the needs are highly variant in real applications. In addition, during inference, the LLM part may take even longer to generate a JSON workflow (as discussed in the paper, there may be thousands of lines) than generating the images with diffusion models. The memory and computation overhead may make this pipeline not as applicable as expected.
2.	Workflow Validity and Robustness: ComfyGen-FT lacks explicit guarantees of workflow validity, meaning that some generated workflows may fail when applied. This issue could affect robustness and reliability, particularly when new workflows or blocks are introduced that require retraining or revalidation. Including a validity check could address this.
3.	Marginal Improvements: The performance improvement over some baselines (such as the 0.02 increase from 0.59 to 0.61 in GenEval scores) is modest. Considering the computational cost, it may not be sufficient to justify this approach compared to simpler models or retrieval-based systems without significant gains in other metrics like diversity or robustness.
4.	Diversity of Generated Flows: The diversity of generated workflows is limited, as noted by the authors themselves, with ComfyGen-IC appearing more like a retrieval-based method than a generation method. This limits the model's ability to create genuinely novel workflows beyond existing templates.

---

confidence: 5
contribution: 3
initial_rating: 3
presentation: 3
questions: 1. Does the proposed method have the capability to design new, previously unseen blocks based on the prompt, or is it limited to the blocks present in the training data?

2. Is there a mechanism in place to ensure that the generated workflows are correct and functional? Given the complexity of the connections between workflow blocks, any logical errors in the generated workflows could significantly impact the user experience.

3. Can the method integrate real-time feedback during workflow generation? For example, if the user wants to modify certain aspects of the generated workflow on the fly, is there a mechanism to incorporate this input dynamically?

4. How does the method handle complex workflows that involve many interdependent blocks? Given that the training data excluded complex workflows, how well does the model perform when tasked with generating workflows that require multiple custom or community blocks with intricate interdependencies?
soundness: 3
strengths: 1. This paper addresses an unexplored and intriguing problem: the automatic generation of ComfyUI workflows based on user-provided prompts, which holds significant potential for creative industries.

2. The prompt-adaptive workflow generation approach is novel, providing flexible solutions through both tuning-based and training-free methods.

3. The paper is clearly written and easy to follow, with a well-structured explanation of the problem and proposed solutions.
summary: This paper introduces the task of **prompt-adaptive workflow generation** for text-to-image models, addressing the complexity of creating effective workflows that combine multiple specialized components. The authors propose two LLM-based approaches: a tuning-based method that learns from user preferences, and a training-free method that selects existing workflows based on the prompt. Both methods improve image quality compared to monolithic models or prompt-independent workflows, highlighting the potential of prompt-dependent flow prediction as a new pathway for enhancing text-to-image generation.
weaknesses: **1. Practicality Concerns**

The authors only trained the proposed model on 310 different text-to-image workflows and excluded workflows with uncommon blocks and complex workflows. This limited training data seems overly simplistic. The model may struggle to learn how to create complex workflows that incorporate a variety of custom blocks. As a result, the model might merely combine simple existing blocks and adjust parameters accordingly, which could significantly limit its practical effectiveness.

**2. Insufficient Validation** 

While generating workflows from prompts is an interesting idea, it may not be the most optimal approach for improving text-to-image generation quality. Since this method indirectly improves image quality by refining the workflow rather than addressing the base model's inherent issues (e.g., attribute misbinding, counting errors), it may have limited effectiveness. 
For instance, in Fig. 4 (third row), the proposed ComfyGen-FT fails to interpret the prompt "a green apple" and instead generates two apples. When compared to other methods that directly leverage LLMs to improve text-to-image generation quality, such as Lavi-Bridge [1], ELLA [2], and PixArt-Alpha [3], the proposed approach does not demonstrate a clear advantage. A more thorough analysis and comparison with these methods are recommended.

**3. Generalization Issue** 

The proposed method struggles to generalize to new community blocks. Incorporation of these new blocks requires retraining the entire model, which poses limitations on its adaptability.

*_[1] Bridging Different Language Models and Generative Vision Models for Text-to-Image Generation. ECCV 2024._*

*_[2] ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment. Arxiv 2024._*

*_[3] PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis. ICLR 2024._*