Title: JudgeFlow: Agentic Workflow Optimization via Block Judge
Primary area: applications to robotics, autonomy, planning
Keywords: LLM-agent, Agentic Workflows, Multi-agent System

Abstract: Optimizing LLM-based agentic workflows is challenging for scaling AI capabilities, some current methods rely on coarse, end-to-end evaluation signals and lack fine-grained signals on where to refine, often resulting in inefficient or low-impact modifications. To address these limitations, we propose JudgeFlow, an Evaluation-Judge-Optimization-Update pipeline. We incorporate reusable and configurable logic blocks into agentic workflows, capturing fundamental forms of logic. On top of this abstraction, we design a dedicated Judge module that inspects execution traces, particularly failed runs, and assigns rank-based responsibility scores to problematic blocks. These fine-grained diagnostic signals are then leveraged by an LLM-based optimizer, which focuses modifications on the most problematic block in the workflow. Our approach improves sample efficiency, enhances interpretability through block-level diagnostics, and provides a scalable foundation for automating increasingly complex agentic workflows. We evaluate JudgeFlow on mathematical reasoning and code generation benchmarks, and the results demonstrate that JudgeFlow achieves superior performance and optimization efficiency compared to existing methods.

TL;DR: We propose JudgeFlow which incorporates a judge module to identify problematic block, guiding the LLM optimizer.