Title: ComfyMind: Toward General-Purpose Generation via Tree-Based Planning and Reactive Feedback
Venue: NeurIPS 2025
Accepted: yes
Keywords: General-Purpose Generation

Collaborative AI system

ComfyUI
Abstract: With the rapid advancement of generative models, general-purpose generation has gained increasing attention as a promising approach to unify diverse tasks across modalities within a single system. Despite this progress, existing open-source frameworks often remain fragile and struggle to support complex real-world applications due to the lack of structured workflow planning and execution-level feedback. To address these limitations, we present ComfyMind, a collaborative AI system designed to enable robust and scalable general-purpose generation, built on the ComfyUI platform. ComfyMind introduces two core innovations: Semantic Workflow Interface (SWI) that abstracts low-level node graphs into callable functional modules described in natural language, enabling high-level composition and reducing structural errors; Search Tree Planning mechanism with localized feedback execution, which models generation as a hierarchical decision process and allows adaptive correction at each stage. Together, these components improve the stability and flexibility of complex generative workflows. We evaluate ComfyMind on three public benchmarks: ComfyBench, GenEval, and Reason-Edit, which span generation, editing, and reasoning tasks. Results show that ComfyMind consistently outperforms existing open-source baselines and achieves performance comparable to GPT-Image-1. ComfyMind paves a promising path for the development of open-source general-purpose generative AI systems.


Peer reviews:
comments: SUMMARY:
This work proposes ComfyMind, which is a planning system built on top of ComfyUI, a node-based execution system for image/video generation. ComfyMind contains two main components: 1) a semantic interface that allows natural language inputs and 2) a planning algorithm with localized feedback that searches for stable and high-quality workflows. ComfyMind has been evaluated on open-source benchmarks, and results showed that it consistently outperforms existing open-source baselines.

STRENGTHS AND WEAKNESSES:
Strengths:
Clarity: The work is presented clearly, with well-defined motivations and a straightforward, easy-to-follow pipeline. Detailed comparisons with prior works further enhance understanding.
Significance: The proposed algorithms have been proven to be useful in three sets of experiments.

Weaknesses:
While the search tree planning algorithm is described at a high level, and its efficacy is justified through experimental results and ablation studies, a more detailed quantitative analysis would strengthen the work. For instance, the claim that "This allows the system to..., ...lead to strategy oscillations and failure to converge" could benefit from direct quantitative comparisons (e.g., efficiency, stability) between the proposed method and baselines like ReAct. Presenting experiments with more detailed ananysis, rather than the final metric, would be helpful.

QUESTIONS:
n/a

LIMITATIONS:
n/a
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ComfyMind, a collaborative AI system built on the ComfyUI platform, designed for robust and scalable general-purpose generation. Addressing the fragility of existing open-source frameworks in complex real-world applications, ComfyMind introduces two core innovations: a Semantic Workflow Interface (SWI), which abstracts low-level node graphs into callable functional modules described in natural language, enabling high-level composition and reducing structural errors; and a Search Tree Planning mechanism with localized feedback execution, which models generation as a hierarchical decision process, allowing adaptive correction at each stage. Evaluated on three public benchmarks (ComfyBench, GenEval, and Reason-Edit) spanning generation, editing, and reasoning, ComfyMind consistently outperforms existing open-source baselines and achieves performance comparable to GPT-Image-1, paving a promising path for open-source general-purpose generative AI systems.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Innovative High-Level Semantic Abstraction: The Semantic Workflow Interface (SWI) is a key highlight, abstracting ComfyUI's low-level node operations into natural-language function modules. This significantly simplifies workflow construction, reduces syntax errors, and enhances the language model's ability to understand and compose complex tasks.\n\n2. Robust Localized Feedback Mechanism: The Search Tree Planning with Local Feedback Execution is highly effective. By modeling workflow construction as a hierarchical decision process and triggering local corrections upon execution failures, it avoids global rollbacks and redundant computations, greatly enhancing system robustness in complex tasks.\n\n3. Broad Generality and Excellent Performance: ComfyMind demonstrates strong performance across three diverse benchmarks (ComfyBench, GenEval, Reason-Edit) for generation, editing, and reasoning. It not only significantly outperforms open-source baselines like ComfyAgent but also matches or even surpasses the closed-source GPT-Image-1 in some aspects, proving its powerful general-purpose generation capabilities.\n\nWeakness\n\nLocalized feedback is crucial for ComfyMind's robustness, but the paper does not quantify how this feedback is implemented. For instance, the granularity of VLM parsing and annotation, the specific format of failure signals, and how parent nodes revise their planning based on this feedback are not detailed.\n\nQUESTIONS:\n1. Within the search tree planning, is MCTS (Monte Carlo Tree Search) or another specific search algorithm used to guide node selection and path exploration? If not, how does the agent effectively avoid local optima or inefficient exploration?\n\n2. The localized feedback relies on a VLM for parsing and annotating generated content. The VLM's own performance and its ability to understand complex visual details could become a bottleneck for feedback quality and overall system robustness.\n\nLIMITATIONS:\nSee weakness and questions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ComfyMind, a system that optimizes prompts through reactive feedback and leverages downstream models to support a wide range of visual generation tasks. Specifically, the proposed Semantic Workflow Interface (SWI) abstracts low-level nodes into callable modules, facilitating high-level composition across diverse tasks. Extensive experiments demonstrate the advantages and effectiveness of ComfyMind in visual generation, editing, and reasoning.\n\nSTRENGTHS AND WEAKNESSES:\n**Main Strengths**\n+ The paper is well-written and easy to follow.\n+ It provides numerous qualitative examples and system prompts, which support the reproducibility of the proposed system.\n+ The proposed pipeline is model-agnostic, allowing newer downstream visual generation models to benefit further.\n\n**Main Weaknesses**\n+ Although the performance is promising, ComfyMind appears to be a combination of LLM agents [1, 2] and prompt rewriting techniques [4, 5, 6], which are already well-known to enhance visual generation.\n\n[1] DiffAgent: Fast and Accurate Text-to-Image API Selection with Large Language Model\n\n[2] GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editing\n\n[3] Improving Image Generation with Better Captions\n\n[4] Tailored Visions: Enhancing Text-to-Image Generation with Personalized Prompt Rewriting\n\n[5] Dynamic Prompt Optimizing for Text-to-Image Generation\n\n[6] Prompt Adaptation as a Dynamic Complement in Generative AI Systems\n\nQUESTIONS:\n+ Apart from the final results, an analysis of the LLM planning process should be conducted. How robust is this process? If the LLM makes incorrect decisions or generates flawed prompts, will the errors propagate to the final performance? How can this potential issue be mitigated? What is the bottleneck of the proposed pipeline — the LLM or the downstream visual generation models?\n+ In the appendix, instead of only showing the final visualizations, the entire LLM reasoning process should be presented. Failure cases should also be discussed to inform future research improvements.\n+ It is already known that prompt rewriting can significantly boost GenEval performance. In Table 2, do the baseline models also use rewritten prompts? If not, is this a fair comparison?\n+ Since ComfyMind requires LLM planning and iterative feedback, how does its efficiency compare to each of the target baselines? Does the performance gain justify the additional time overhead?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents ComfyMind, a system for image generation and editing from complex text prompts by coordinating various foundation models as tools. The key idea is to utilize VLM to construct a workflow tree and call APIs from a tool library with feedback. Experiments on GenEval show promising results.\n\nSTRENGTHS AND WEAKNESSES:\n## Strenth\n- ComfyMind is an agent-based system that is training-free and easy to use.\n- The paper is clearly written, and the results look promising.\n\n## Weakness\n- **Major concern:** My major concern lies in the technical novelty. A closely related work, [GenArtist](https://openreview.net/pdf?id=Ur00BNk1v2) [1], proposes a similar approach and is not cited or discussed. To be specific, GenArtist also proposes to use VLM to decompose tasks into a tree-based tool execution flow that is executed with feedback control. GenArtist also supports various tasks like image generation and editing from language descriptions. While some details like the structure of every tree node are different, and ComfyMind supports more tasks by using more agents and more advanced foundation models, the core idea of ComfyMind is basically the same as GenArtist. Please clarify it and do a thorough comparison.\n- Typically, such agent-based image edition and generation systems are prone to the quality of the first generated image or video. The quality might not be sufficiently good, while the evaluation might still be processed as good, since the overall quality is not bad. One way to address this might be executing image generation path several times and pick the best one, but this might be time-consuming.\n- Besides the benchmarks used in this paper, please try to conduct experiments on T2I-CompBench [2] and DreamBench++ [3], and make a comparison with CompfyMind. These are good benchmarks for demonstrating the generation capability and will enrich the solidity. Note that you should try to use the same agents to make the comparison fair.\n\n\n[1] Wang, Zhenyu, Aoxue Li, Zhenguo Li, and Xihui Liu. \"Genartist: Multimodal llm as an agent for unified image generation and editing.\" Advances in Neural Information Processing Systems 37 (2024): 128374-128395. \\\n[2] Huang, Kaiyi, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. \"T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation.\" Advances in Neural Information Processing Systems 36 (2023): 78723-78747.\\\n[3] Peng, Yuang, Yuxin Cui, Haomiao Tang, Zekun Qi, Runpei Dong, Jing Bai, Chunrui Han, Zheng Ge, Xiangyu Zhang, and Shu-Tao Xia. \"Dreambench++: A human-aligned benchmark for personalized image generation.\" The Thirteenth International Conference on Learning Representations (2025).\n\nQUESTIONS:\nWhat is the inference latency of the proposed method? Can authors make the comparison to the compared methods?\n\nLIMITATIONS:\nThe paper has discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-06"}