Weak-for-Strong:  Training Weak Meta-Agent to Harness Strong Executors
Authors: Fan Nie
Lan Feng
Haotian Ye
Weixin Liang
Pan Lu
Huaxiu Yao
Alexandre Alahi
James Zou
Venue: COLM 2025 (2025)
Keywords: large language models
reinforcement learning
workflow generation
TL;DR: We train a weak meta-agent to better leverage strong models according to environment feedback through reinforcement learning.

Abstract: Efficiently leveraging of the capabilities of contemporary large language models (LLMs) is increasingly challenging, particularly when direct fine-tuning is expensive and often impractical. Existing training-free methods, including manually or automated designed workflows, typically demand substantial human effort or yield suboptimal results. This paper proposes Weak-for-Strong Harnessing (W4S), a novel framework that customizes smaller, cost-efficient language models to design and optimize workflows for harnessing stronger models. W4S formulates workflow design as a multi-turn markov decision process and introduces reinforcement learning for agentic workflow optimization (RLAO) to train a weak meta-agent. Through iterative interaction with the environment, the meta-agent learns to design increasingly effective workflows without manual intervention. Empirical results demonstrate the superiority of W4S that our 7B meta-agent, trained with just one GPU hour, outperforms the strongest baseline by 2.9% ～ 24.6% across eleven benchmarks, successfully elevating the performance of state-of-the-art models such as GPT-3.5-Turbo and GPT-4o. Notably, W4S exhibits strong generalization capabilities across both seen and unseen tasks, offering an efficient, high-performing alternative to directly fine-tuning strong models.

Published as a conference paper at COLM 2025
Weak-for-Strong:
Training Weak Meta-Agent to Harness Strong Executors
Fan Nie
Lan Feng
Haotian Ye
Stanford University, USA niefan@stanford.c
EPFL, Switzerland lan. feng@epfl.ch
Stanford University, USA hootiayyesstaffrddeed
Weixin Liang Stanford University, USA wwLiang@stanford.eed
Pan Lu
Huaxiu Yao UNC-Chapel Hill, USA huaxiuuca unc. edu
Stanford University, USA canluestanford.c
Alexandre Alahi
James Zou
EPFL., Switzerland alexandre, alahi@epfl.ch
Stanford University, USA
jamesz@stanford
Abstract
Efficiently leveraging of the capabilities of contemporary large language models (LLMs) is increasingly challenging, particularly when direct fine- tuning expensive and often impractical. Existing training-free methods, including manually or automated designed workflows, typically demand substantial human effort or yield suboptimal results. This paper proposes for--nor Harnessing (W4S), a novel framework that customizes smaller, cost-efficient language models to design and optimize workflows for harnessing stronger models. W4S formulates workflow design multi-turn markov decision process and introduces reinforcement learning for agentic workflow optimization (RLAO) to train weak meta-agent. Through iterative interaction with the environment, the meta-agent learns to design increasingly effective workflows without manual intervention. Empirical results demonstrate the superiority of W4S that our 7B meta- agent, trained with just one GPU hour, outperforms the strongest baseline by 2.9% ~ 24.6% across eleven benchmarks, successfully elevating the performance of state-of-the-art models such as GPT-3.5-Turke and GPT- 4o. Notably, W4S exhibits strong generalization capabilities across both seen and unseen tasks, offering an efficient, rerghperforming alternative directly fine-tuning strong models. Code is available here.
1 Introduction
Despite the rapid advancement of large language models (LLMs) such as GPTT-4o (OpenAI, 2024), Claude (Anthropic, 2024), Deepseek-R1 (DeepSeek-Al et al., 2025) and Llama (Dubey et al., 2024), how to effectively harness their capabilities in workflows remains a significant challenge. Directly querying these powerful models often yields inadequate results on complex domain-specific tasks. Meanwhile, fine-tuning strong models achieve desired behaviors can be prohibitively expensive and even infeasible, especially with closed-source, commercial models. This raises a critical research question: how can we unleash the potential of powerful LLMs without directly finetuning them?
To this end, training-free methods have emerged as potential solutions, ranging from simple heuristics like Few-shot Prompting (Brown, 2020), Chain-of-Thought (COT) (Wei et al., 2022), In-context Vectors (Liu et al., 2024a) to more intricate hand-designed agentic work- flows (Yao et al., 2023; Zhou et al., 2023; Zhong et al., 2024b; Lu et al., 2025b). While heuristic approaches enhance performance, they struggle with complex tasks requiring multi-step reasoning (Prasad et al., 2024). Sophisticated hand hand-designed workflows mitigate some
1

---

Published as a conference paper at COLM 2025
Weak-to-Stromt Generalization
Weak-for-Strong Harnessing
Suparvise
Workflow
Blevng
Answer
Weak Supervisur
Strong Student
Muth Raport
Critical Thinker
Accuracy:
Weak-as-Router Dispatching
Weak Mataragent
Workflow
Environment
8 Weak Router
Answar
Programmn
Pythun Raweutor
Query
Accuracy: 0.72
will updated via troining
Fawways
Figure 1: Comparison of paradigms: Weak-to-Strong Generalization uses weak models to supervise strong models, akin to superalignment; routing-based methods train weak models to dispatch queries across strong models; in contrast, Weak-for-Strone Harnessing (W4S) trains a weak model to optimize a strong model's performance on specific task.
limitations but require labor-intensive trial-and-error and domain-specific manual tuning, resulting in high labor costs. Moreover, these manual strategies lack adaptability across tasks or models and fail to fully exploit LLM potential (Cemri et al., 2025), aligning with the "bitter lesson" (Sutton, 2019) that -and-ennieeered solutions are outpaced by adaptive, data-driven systems. Recent efforts have explored representing workflows as executable code, enabling powerful models like GPT-4o Claude to automate workflow generation and optimization (Hu et al., 2024; Zhang et al., 2024a). However, these training-free ap- proaches underutilize historical data and environmental feedback, sometimes performing no better than random workflow sampling (App. E.1), highlighting the inadequacy of such approaches in practice.
The challenge becomes even more pronounced with superintelligent models whose be- haviors might not be fully predictable or comprehensible human users (Burns et al., 2024), raising critical questions about the optimal strategies for their utilization. Given the limitations of existing training-free methods and the intractability LLMS directly, this paper turns into the idea of training weaker model that can understand the behaviors strong models as well as the downstream task, to harness the strong models based on its understanding in the place of human.
fine-tuning strong
Our Contributions. We propose new paradigm: Weak-for-Strong Harnessing (W4S), which trains weak model to leverage the strengths of strong models. W4S casts the problem of harnessing strong models as a workflow optimization problem, and employs weak model as meta-agent trained specifically for the problem. Unlike previous meth- ods (Zhuge et al., 2024; Zhang et al., 2024a) that predefine agentic modules, we maximize the degree of freedom of the meta-agent by constraining only the workflow interfaces. This allows the meta-agent to design every internal component in freedom, including prompts, hyperparameters, and building blocks, enabling more expressive and tailored solutions. We formulate this as multi-turn Markov decision process (MDP), and introduce reinforcement learning for agentic workflow optimization (RLAO) to teach the meta-agent to design and refine workflows. Through iterative interaction with both the task environment and the behavior of strong models, the weak meta-agent learns to design and improve workflows for strong models based on history and feedback.
Our approach introduces a novel perspective on the potential ways of interaction between weak and strong models, distinct from existing paradigms such as weak-to-strong gen- eralization (Burns et al., 2024) and weak-dispatch-strong routing framework (Frick et al., 2025), illustrated in Figure 1. This new paradigm emphasizes the weak meta-agent's role in unlocking latent capabilities of existing models without modifying them directly. Our paradigm is significantly more efficient and less expensive than finetuning strong
2

---

Published as a conference paper at COLM 2025
models directly, while outperforming both finetuning weak models targeted tasks and training-free methods.
We conduct comprehensive evaluations across eleven widely adopted benchmarks, in- cluding question answering, mathematics, and code generation tasks. Empirical results demonstrate that 7B meta-agent, trained with only one GPU hour five tasks, can design workflows that effectively leverage strong models, significantly outperforming all the base- lines. W4S surpasses manually designed methods by 3.3% 27.1% % and outperforms the strongest automated design baseline by 2.9% ~ 24.6%. Notably, the workflows generated by our method exhibit strong generalization and transferability across tasks and strong models, demonstrating the robustness and adaptability of the learned weak meta-agent in orchestrating
workflows.
2 Method: Weak-for-Strong Harnessing
This section presents the Weak-for-Strong Harnessing (W4S) framework that trains weak models to optimize agentic workflows for stronger models. The key insight is that workflow optimization can be formulated as sequential decision-making problem where a weak meta-agent iteratively improves workflows through interactions with an environment, guided by performance feedback.
Specifically, we define an agentic workflow W as a structured and executable Python function that internally invokes strong model to perform specific downstream tasks. The W4S framework operates as an iterative process of workflow generation, execution, and refinement, as depicted in Figure 2(a), and is unfolded as follows:
Workflow Generation. The weak meta-agent analyzes the task, historical workflows, and prior feedback to design a new workflow to leverage the given strong model, represented as executable Python code. A self-correction mechanism addresses coding errors.
Execution and Feedback. The generated workflow is executed by strong model on validation samples, producing performance feedback (e.g,, Accuracy, Error Cases).
Refinement. The meta-agent uses feedback to iteratively improve the workflow, adapting to the task and the strong model's behavior over multiple turns.
This process enables the meta-agent to learn task-specific strategies and harness the strong model's capabilities efficiently, without requiring direct fine-tuning of the strong model. To rigorously analyze this optimization problem, below we formalize it multi-turn Markov Decision Process (MDP), and present our Reinforcement Learning for Agentic Workflow Optimization (RLAO) algorithm for training the weak meta-agent.
2.1 Workflow Optimization as Multi-Turn MDP
An MDP is denoted by tuple M (S,A, R), where S and A are the state space and the action space, respectively. In our case, represents the current knowledge about the task, the model and workflow history, A consists of possible workflow designs, P S x A S [0, 1] is the transition probability function, and R. S x A x S + is the reward function.
For each iteration the agent takes action a; at state s; according to a learnable policy Ny (a]s) S x A - jo, 11, where 9 is the parameters of the meta-agent. The environment executes the workflow and provides feedback f, and feedback-based reward r, transiting to the next state S/+1 = [ fil. This process continues for a fixed number of iterations or until predefined convergence criterion is met, allowing the agent to refine workflows based on feedback.
Initial State Setup. The initial state s consists of Instructions I, Task description T, Example workflow wh and its feedback fo (if available). Details about T and wy are shown in Appendix A.
$1 [I;T; Wo: fo)-
Action Design. Each action includes two steps: analysis and workflow generation.
3

---

Published as a conference paper at COLM 2025
Data Collection
RL Training
"Your horness the model tongeted Tasks."
[nsetnattion]] / Taakk decrrptioon
/ [Jiaampla Workfoow and Yeedbacki
Eniiaal
Anslyiia
Previoci Pluts
Woek
Connuttion
+
4
4
Evaluction
X
#2
/21
Je d
x2
Weak LIM
Validattin Assureey 0.25
(Canerete Samplas Paclure Canne)
Pleuse impreve the workffow to incrause performence
+
4
(a) W4S Optimization
(b) Overview of RLAO
Figure 2: (a) The weak meta-agent harness strong models by optimizing the workflows iteratively based on task and environment feedback. (b) To collect effective data for offline RL training. the meta-agent will sample times in each iteration, and using the best samples to form the next state. The data form multi-turn trajectories for offline RL training.
1. Analysis:
The meta-agent is required to first conduct analysis include interpreting the task, history workflows, and feedbacks, and plans for improvements. Adding the analysis into the action space can bridge the gap between the pretrained language priors of LLMs and the environment, providing context for what adjustments should be made next.
2. Workflow Generation: Based on the analysis, the meta-agent produces function- represented workflow W,. Unlike previous work such Zhang et al. (2024a) that specifies predefined agentic modules (e.g., ensemble module, revision module), which constrains the creativity of LLMs, our approach only specifies the interface of the workflow function and provides helper functions like LLM calls and code execution. More details about the helper functions can be seen in Appendix. A.2. This gives the meta-agent complete freedom to design the prompts, hyperparameters and internal logic of the workflow, fostering greater innovation and adaptability.
Error Handling via Self-Correction. To address potential coding errors in the generated workflows, we implement a self-correction mechanism by executing the workflow W; on single validation sample. If execution fails due to bugs, the meta-agent will be prompted to perform self-correction to fix the identified bugs. This process can iterate up to 3 times, with the error message provided to the meta-agent at each step:
w (i+1) SelfCorrect(Instrutiionn w.()) Error,).
where w/(h is the workflow at the j-th correction attempt and Error; is the corresponding error message. After self-correction, the complete action is then denoted as:
a; = AAnalysss; Wi].
where W; is now the workflow of the last correction attempt. If the workflow continues to produce errors after correction attempts, the current iteration is skipped, and the erroneous workflow is not recorded.
Evaluation Feedback. Upon successful execution, the workflow is evaluated on both private and public validation sets to generate feedback:
1. Validation performance Uy: Accuracy measured on the private validation set.
2. Case studies: Examples of incorrect predictions from the public validation set, including input prompts, model answers, and correct answers.
The feedback is formally represented as:
fi [Dy; CaseStudies,
4

---

Published as a conference paper at COLM 2025
2.2 RLAO: Reinforcement Learning for Agentic Workflow Optimization
To train the weak meta-agent, we propose Reinforcement Learning for Agentic Workflow Optimization (RLAO), an offline RL algorithm tailored for this MDP, as shown in Figure 2(b). Online RL is less efficient due to the high cost of real-time workflow execution, so we collect trajectories offline and optimize the policy accordingly.
Reward Mechanism. Based on the feedback fü, we define reward as follows:
1,
v. > max-k
0.5, if o; > 8=-1 otherwise
This reward function encourages both absolute improvement (surpassing all previous iterations) and relative improvement (surpassing the most recent iteration).
Data Collection. We collect dataset of optimization trajectories for training the weak meta-agent. At each iteration i, we sample и candidate actions. Subsequently, we select the best action based on validation performance to serve as the current action respectively to form the new state and execute the next action. Our dataset consists both selected actions and unselected alternatives. At each iteration we generate candidate actions:
{ [ a a!")
Then we select the best action based on validation performance:
arg max of. kc (cmm)
a
where of represents the validation performance of the workflow produced by action af. This selection mechanism serves a dual purpose: it ensures that only the most effective workflow proceeds to the next iteration while simultaneously enriching our training dataset with both successful and unsuccessful attempts. This best-of-m approach helps to create high-quality trajectories for training while maintaining diversity.
Policy Optimization. We train the meta-agent using the offline variant of reward regression (RWR) (Lee al., 2023; Qu et al., 2024) that optimizes the policy πtg.
T
exp("))
max E p
Σ
log no (a st)
(1)
where p = ( ST, ST, (T) is a trajectory from dataset 2, i'is the trajectory length, and T is a temperature hyperparameter controlling reward scaling.
3 Experiments
3.1 Experimental Setup
Baselines. We compare workflows discovered by W4S against manually designed methods
for LLMs, including 5-shot prompting, COT (Wei et al., 2022), Self Consistency CoT (5 answers) (Wang et al., 2022), Self-Refine (max 3 iteration rounds) (Madaan et al., 2023), LLM Debate (Du et al., 2023), Quality Diversity (Lu al., 2025a) and Dynamic Assignmen (Xu al., 2023a). We also compare against workflow designed by automated workflow optimiza- tion method ADAS (Hu al., 2024) and AFlow (Zhang et al., 2024a). Besides, we compare against training- based baseline where GPT-4o-mini is fine-tuned the validation dataset for fair comparison. More details are provided in Appendix D.2.
Datasets. We utilize eleven public benchmarks for experiments: (1) math reasoning, use MGSM (Shi et al., 2023), GSMSK (Cobbe et al., 2021), GSM Plus (Li et al., 2024a), GSM Hard (Gao et al., 2023), SVAMP (Patel et al., 2021) and MATH (Hendrycks et al., 2021). For

---

Published as a conference paper at COLM 2025
the MATH dataset, we follow (Hong et al., 2024a) in selecting 617 problems from four typical problem types (Combinatorics & Probability, Number Theory, Pre-algebra, Pre-calculus) at difficulty level 5; (2) question-answering,
use DROP (Dua et al., 2019) for evaluating reading comprehension, MMLU Pro (Wang et al., 2024) for evaluating multi-task problem solving and GPQA (Rein et al., 2023) for evaluating the capability of solving graduate- level Science questions; (3) code generation, we use HumanBval (Chen et al., 2021), and MBPP (Austin et al., 2021). For ADAS and AFlow, we conduct the searching for workflows on a validation set. For W4S, we further randomly split the validation set into a private validation set and a public validation set. All the evaluation results are conducted on the same held-out testing set. We follow the data splits used in established practices (Hu et al., 2024; Zhang et al., 2024a). More details about datasets can be found in Appendíx. D.1.
Metrics. For HumanEval and MBPP, we report the pass@1 metric as presented in (Chen et al., 2021) to assess code accuracy. For multiple-choice datasets MMLU Pro and GPQA and mathematical datasets, we use Accuracy. For DROP, we report the F1 Score.
Data Collection Details. To manage computational constraints during training, we impose trajectory truncation strategy in RLAO. Trajectories are limited to a horizon of = 2 turns, with states reset every two iterations as follows:
$1
if i O,
$2
0, $2: (22 1,
42
fzi 11)-
[s); W2s
fail,
if
where $1 is the initial state, W2; is the workflow from the previous selected action, and fzi is its feedback. This results in a dataset D comprising single-turn trajectories (from unselected actions) and two-turn trajectories (from selected actions), formally:
D D= (==((sh) de Pt. +)
[D]
|=
T'e [1,2].
For the following experiments results, we set m = candidate actions per iteration to collect offline data, yielding 212 trajectories for Table 1 and 145 trajectories for Table 2.
Implementation Details. For ADAS and AFlow, we use GPT-40 as the meta-agent. For W4S, we employ and train Qwen2. -B-sstrr------sttrrct as the weak meta-agent. We also report the performance of directly utilizing GPT-4o without RLAO as meta-agent or training meta-agent with SFT on our framework in ablation studies. For execution, we employ GPT-3. 111-3.5-Turbe 5-Turbo, GPT-4o-mini in main text. More experiments using GPT-4o and Claude-Sonnet as executors are shown in Appendix We set iteration rounds to 20 for AFlow, and 30 for ADAS, following their original settings. We set iteration rounds to 10 for W4S. Training is conducted on 2 Nvidia H100 GPUs with a learning rate of le-5. The temperature т for weighting the reward is set to 0.4. At inference time, W4S only samples one action in each iteration. More implementation details can be seen in Appendix D.
3.2 Experimental Results
W4S significantly outperforms baseline methods across seen and unseen tasks. As illustrated in Table 1, W4S, employing 7B model as a weak meta-agent trained with RLAO, markedly surpasses few-shot learning, manually designed workflows, and automated workflow baselines with only 10 iterations. In this experiment, the meta-agent is trained on five tasks (DROP, MMLU Pro, MBPP, GSM Hard, Math) and generalize to two unseen tasks, The execution LLM is GPT-40-mini. 'Finetuned GPT-4o-mini represents using surpervised learning to train GPT-4o-mini on validation dataset, which yields unsatisfactory results, highlighting that leveraging weak model trained via RLAO effectively outper/ direct aine-tuning on strong models under limited data conditions. Besides, 'W4S w SFT' represents training the weak model using the same data of RLAO with SFT. Notably, W4S with RLAO outperforms its untrained and SFT trained counterpart, further demonstrating the effectiveness of RLAO.
outperforms
W4S demonstrates generalization capabilities across different mathematical tasks. Table evaluates the generalization of W4S mathematical reasoning tasks. Despite being

---

Published as a conference paper at COLM 2025
Seen Task
Unseen Task
Method
DROP
MMLU Pro
MBPP
GSM Hard
Math
GPQA
HumanEivl
3-shot
80.9
60.8
69.5
43.0
57.1
37.4
87.8
Finetumed GPT-4o-mini
73.9
61.1
76.2
41.2
41.8
Hand-designed Workflows
CoT
78.5
56.6
72.4
39.5
56.9
36.7
88.8
COT SC
84.2
58.0
74.2
45.0
58.1
39.4
90.3
Self Refine
79.1
57.5
70.4
47.5
53.0
38.4
85.0
73.9
89.1
Quality Diversity
80.0
39.1
71.8
46.5
35.3
40.1
86.0
Dynamic Assignment
80.2
57.4
71.8
41.5
56.9
36.0
90.1
Training- free Automated-designed Workflows
ADAS (30iter)
82.0
58.4
74.0
52.5
51.4
39.6
90.8
AFlow (20iter)
80.6
59.2
83,9
52.0
58.4
42.0
92.1
W4S w/o RLAO (10iter)
85.3
61.0
86.0
60.6
58.6
39.8
92.7
W4S w/ (10iter)
63.0
72.4
61.9
39.6
94.3
W45 (10iter)
87.5
64.8
86.8
76.6
63.0
45.9
95.4
Table 1:
Comparison of performance (%) between W4S and baselines. All methods are executed using GPT-4o-mini, with each tested three times, and average results reported.
Seen Task
Unseen Task
Method
GSM Plus
MGSM
GSM8k
GSM Hard
SVAMP
Hand-designed
Workflows
CoT
24.5
28.0
38.5
14.0
77.8
CoT SC
27.1
28.2
43.0
15.0
78.2
Self Refine
25.8
27.5
40.5
14.5
78.5
LLM Debate
39.0
49,0
18.0
76.0
Quality Diversity
21.1
31.1
29.0
14.0
69.8
Dynamic Assignment
27.1
30.1
34.0
19.5
73.0
Training-frer free
Automated-designet Workflows
ADAS (GPT-4o 15iter)
52.0
47.5
54.5
31.5
80.8
ADAS (GPT-4o 30iter)
57.4
53.4
61.1
34.5
82.8
AFlow (GPT-4o 20iter)
62.8
54.8
76,8
40.6
81.3
In-distribution Domains
Generalize to Other Math Domains
W4S (10iter)
68.2
66.2
86.5
61.8
84.2
Table 2: Comparison of performance (%) between W4S and baselines. All methods are executed using 109T-3.5-Trrte with each tested three times, and average results reported.
trained solely on GSM Plus and MGSM, W4S achieves substantial improvements over all baselines when tested on unseen tasks such GSM8K, GSM Hard, and SVAMP. Particularly, W4S exceeds the strongest baseline methods by 10% on GSM8K and 20% on GSM Hard, highlighting W4S as scalable and effective method for harnessing powerful executors.
MGSM
GSMAK
60
80
Arcuracy (%6
60
40
Accurscy
40
20
WAS (Uwen 731
20-
WAS (Ulven 713)
10-
WSB win RLAD (Owen 7R)
WHS wio RLAD (0wen 79)
W4S BLAG (CPT:4o)
WAS RLAO (CPT-4u)
(CP/T-4)
(CITT-u)
2
10
2
10
Itrration
Iteratio
(a) Seen Task
(b) Unseen Task
Figure 3: Ablation Studies on MGSM and GSM8K. The purple line represents the perfor- mance of W4S using 7B model trained on MGSM and GSM Plus with RLAO.
7

---

Published as a conference paper at COLM 2025
DJOP
E
Queriie-- Task
002
Durs
Molilizgual
Mall Yuck
Pyrhon Progaammr
64
022
ADXS
Delate
Soweecch
S Sueey
Seee
Basson Repert
Sceetaat
Grlinal Thinke
.007
Salf Buding
Cade
Coa
Cod
10
Pyller
MBPP
Fine
Vina
Pige
Belentisn
Bellciic
Po0002 (N)
00
Tikle
Majoelty Vocig
ADAS
OCT
LLM Dubate Seling
Midertz Veting
10
Fizal Answer
Ploal Answer
Anawer
Analyni DROP
Studs
MGBM
MMLU
Figure 4: Cost Analysis (a) and Case Studies (b, c) of W4S on different benchmarks.
Method
Optimization Wall-clock Time (min) Mela-Agent (os
Exacution (5)
Wall-clock Time (min)
Execution Testing
Inferenze Cost (5)
Total (5)
Passe1
ADAS
131
11.3
Aflow
61
0.6
0.4
10.9
1.3
92.1
was
33
0
0.4
2.7
0.5
0.9
95.4
Table 3: Efficiency comparison between W4S and state-of-the-ar baselines on HumanEval, using GPT-4o-mini as the executor. Testing set execution metrics are averaged over three runs, with costs reported for all runs.
Ablation Study. Figure 3 illustrates iteration curves for MGSM (seen task) and GSM8K (unseen task). W4S, leveraging weak meta-agent trained via RLAO, demonstrates stable and consistent improvements over iterations on both seen and unseen tasks. Conversely, ADAS, employing GPT-4o directly as the meta-agent, has very random performance and often output workflow with performance of O. Besides, W4S trained with RLAO out- performs directly using the 7B model without training, demonstrating the efficacy of our training method. Notably, utilizing trained weak meta-agent also outperforms directly using strong model like GPT-4o to optimize the workflow, validating the necessity and effectiveness of our weas-for- paradigm facilitated by RLAO training.
Cost Analysis. In Figure 4(a), we demonstrate the comparison of performance and API calls between the baselines and the workflows found by ADAS, AFlow (using GPT-4o as meta- agent) and W4S (using trained 7B model as meta-agent) GPT-4o-mini as execution LLM. Results demonstrate that W4S can design workflows that harness strong models to have a better performance with less test-time compute compared with hand-designed workflows. Besides, by automating the design of effective agentic workflows, W4S eliminates the human labor costs previously required. Although W4S adds more cost of training, this training cost is negligible compared to finetuning strong model on targeted task. Training a 7B model on five tasks in Table 1 requires only one GPU hour, which can actually be amortized over repeated use across different benchmarks. Table provides a detailed efficiency comparison on an unseen benchmark, including API cost and wall-clock time and testing performance. Compared to ADAS and Allow, W4S achieves a Pass@1 score of 95.4 with a significantly reduced optimization time (33 minutes) and zero meta-agent API cost. Test-time execution remains comparable to baselines, with wall-clock time of 2.7 minutes and an inference cost of $0.5, underscoring W4S's ability to balance efficacy and efficiency.
DROP and MBPP, and using
Case Study. Figure 4(b) and (c) visualizes the workflows designed by W4S on MGSM and MMLU Pro. For MGSM, the workflow employs a Translator LLM that converts multilingual problems to English, followed by a Python Programmer generating multiple code imple- mentations. Successful code executions are aggregated via Majority Voting, with Math
8

---

Published as a conference paper at COLM 2025
Expert as fallback for challenging problems. This adaptive approach dynamically adjusts strategies based on execution results. For MMLU Pro, W4S creates a parallel multi-agent workflow with specialized experts that each develop multiple reasoning paths. After Reflection phase where agents review their answers, a Majority Voting mechanism pro- duces the final answer. Both workflows demonstrate how W4S automatically discovers task-specific decomposition strategies and effective coordination mechanisms that combine specialized expertise with critical evaluation.
4 Related Works
Agentic Workflows. Agentic workflows and autonomous agents represent distinct LLM application paradigms: the former follows structured, multi-step processes, while the latter dynamically solves problems. Unlike agents requiring custom decision patterns, agentic workflows leverage human expertise for automated construction. They have been applied to problem-solving (Wei al., 2022; Wang al., 2022; Madaan et al., 2023; Wang et al., 2023; Han et al., 2025; Zhou et al.), code generation (Hong et al., 2024b; Ridnik et al., 2024; Zhong et al., 2024a), data analysis (Xie al., 2024; Ye al., 2024; Zhong al., 2024a; Zhou al., 2023), and mathematics (Zhong et al., 2024b; Xu et al., 2023b).
Recent research automates workflow design via prompt tuning (Fernando al., 2024; Yüksekgönül et al., 2024; Yang et al., 2024; Khattab et al., 2024; Liu et al., 2024b), hyperpa- rameter optimization (Saad-Falcon et al., 2024), and end-to-end workflow optimization (Li et 2024b; Zhou et al., 2024a; Zhuge et al., 2024; Hu et al., 2024; Yin et al., 2024). Methods like GPTSwarm (Zhuge al., 2024), ADAS (Hu et al., 2024) and AFlow (Zhang et al., 2024a)
explore structured representations, yet efficient workflow discovery remains challenge. Unlike previous methods relying on human-defined logic, our approach employs reinforce- ment learning (RL) to autonomously optimize workflows, achieving superior scalability and performance. Besides, unlike previous methods that treat workflows as graphs with predefined agentic modules as nodes, we maximize the creativity of the meta-agent by constraining only the workflow interfaces.
Weak-to-Strooe Generalization. Weak-to-strong generalization refers to stronger models outperforming weaker supervisors after fine-tuning. While Burns et al. (2024) empirically demonstrated this effect, its limitations remain. Theoretical analyses (Charikar et al., 2024; Lang et al., 2024) and practical approaches-incclnne sasy-to-strong generalization (Sun et al., 2024), small model search (Zhou et al., 2024c), hierarchical mixture of experts (Liu & Alahi, 2024), reliability-aware alignment (Guo & Yang, 2024), alignment with weak LLM feedback (Tao & Li, have been explored. Unlike prior work focused on supervised improvements, we introduce a learning-basee agentic optimization approach to harness strong models via weak models.
LLM debates (Kenton et al., 2024),
Concurrent Work. MaAS, ScoreFlow, and MAS-GPT (Zhang et al., 2025; Wang et al., 2025; Ye et al., 2025) also explore automatic workflow generation for LLM-based systems. MaAS (Zhang et al., 2025) optimizes distribution over multi-agent architectures. Score- Flow (Wang et al., 2025) conducts gvaluation-based preference optimization, yet lacks interaction-driven refinement. MAS-GPT (Ye et al., 2025) conducts supervised learning and lacks feedback adaptation. In contrast, W4S trains a weak agent via RL to iteratively opti- mize workflows with environment feedback, achieving adaptive strong model harnessing.
5 Discussion
Safety Considerations. Although is highly unlikely that the meta-agent employed in our setting generate malicious behaviors, they might inadvertently produce unsafe outputs due to limitations in model alignment (Rokon et al., 2020; Chen et al., 2021). We mitigate this risk through containerized execution of all generated code within secure, isolated environments, automated detection of potentially unsafe code patterns and manual safety inspections.
In fact, our training methodology offers an advantage from a safety perspective compared with training-free methods that rely directly on potentially less-aligned strong models to

---

Published as a conference paper at COLM 2025
design workflows. The weak meta-agent could be specifically trained to avoid generating workflows that might misuse the strong model's capabilities or produce harmful outputs. While we didn'tt explicitly optimize for safety in this paper, future work could integrate safety-oriented objectives by penalizing harmful patterns and rewarding safe workflows.
Limitations. The strong models utilize certainly powerful, but they do not rep- resent the frontier of srosed-souree models, such as OpenAI ol (OpenAI et al., 2024) and Deepseek R1 (DeepSeek-Al et al., 2025). As models continue to advance in capability, the gap between weak models and strong executors may widen, introducing new challenges. Additionally, our experiments focuses primarily on question-answering and reasoning datasets, representing only slice of potential applications. Complex tasks like long-horizon planning and real-world agentic tasks may require further methodological refinements. Nevertheless, despite these limitations, our current results remain highly encouraging. They demonstrate the viability and effectiveness of training weak models better understand the behaviors and leverage the potential of stronger models, suggesting promising direction for future research as Al systems continue to advance in capability. Our work represents an important proof of concept that will become increasingly valuable as the capability gap between accessible and cutting-edge models continues to widen.
6 Conclusion
We propose Weak-for-Strong Harnessing (W4S), novel framework that trains a weak meta- agent design and optimize agentic workflows, effectively harnessing the capabilities of stronger language models. By formulating workflow optimization as a multi-turn MDP and leveraging Reinforcement Learning for Agentic Workflow Optimization (RLAO), our approach enables 7B model to harness sfate-of-the-art models, achieving significant performance gains across diverse benchmarks. key benefit of Weak feak-for-Strong that the meta-agent is smaller model that's easier and cheaper to train with RL and also easier to control because it's open source. As LLMs continue to advance, W4S establishes promising paradigm for efficiently unlocking their potential, paving the way for future exploration into adaptive, learning-driven agentic systems.
Acknowledgments
We would like to thank Leitian Tao, Siwei Han, Shirley Wu, Yuxiao Qu, Zhenting Qi and the members of Zou group for helpful advice and discussions.
10

---

Published as a conference paper at COLM 2025
References
Anthropic. Introducing claude 3.5 sonnet. claude-3-5- sonnet, 2024.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Program synthesis with large language models. CoRR, abtpabbs...///990 2021.
Tom Brown. Language models are few-shot learners. arviv preprint ar xrXi::2005.14166 2020.
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschen- brenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong generalization: eliciting strong capabilities with weak supervision. In Proceedings of the 41st International Conference on Machine Learning, PP. 4971-5012, 2024.
Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E. Gonzalez, and lon Stoica. Why do multi-agent Ilm systems fail?, 2025. 13657.
URL
Moses Charikar, Chirag Pabbaraju, and Kirankumar Shiragur. Quantifying the gain in weak-to-strong generalization. Advances in neural information processing systems, 37:126474- 126499, 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mo- hammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374 2021.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichir Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. arXiv preprint ar 0rXiv:2110.14166 2021.
DeepSeek-AT, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning, 2025. URL 12948.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improv- ing factuality and reasoning in language models through multiagent debate, 2023. URL https://arxivv wrtp://///////....... 14325.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In NAACL-HLT (1), pp. 1368-2375 Association for Computational Linguistics, 2019.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd models. Xiv preprint arXiv:2407.21783, 2024.
Lan Feng, Fan Nie, Yuejiang Liu, and Alexandre Alahi, Tarot: Targeted data selection via optimal transport. arxiv preprint ar XrXt:22412.00420, 2024.
11

---

Published as a conference paper at COLM 2025
Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Promptbreeder: Self-referential self-improvement via prompt evolution. In ICML. OpenReview.net, 2024.
Evan Frick, Connor Chen, Joseph Tennyson, Tianle Li, Wei-Lin Chiang, Anastasios N Angelopoulos, and Ion Stoica. Promptto-leaderboarr. arXiv preprint ar SrXiv:2502.18855, 2025.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, pp. 10764-10799. PMLR, 2023.
Yue Guo and Yi Yang. Improving weak-to-strong generalization with reliability-aware alignment. arviv preprint ar arXio:2406.19032, 2024.
Siwei Han, Peng Xia, Ruiyi Zhang, Tong Sun, Yun Li, Hongtu Zhu, and Huaxiu Yao. Mdocagent: A multi-modal multi-agent framework for document understanding. ar Xiv preprint Xit::2503.1364 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasels and Benchmarks Track (Round 2), 2021.
Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Danyang Li, Jiaqi Chen, Jiayi Zhang, Jinlin Wang, Li Zhang, Lingyao Zhang, Min Yang, Mingchen Zhuge, Taicheng Guo, Tuo Zhou, Wei Tao, Wenyi Wang, Xiangru Tang, Xiangtao Lu, Xiawu Zheng, Xinbing Liang, Yaying Fei, Yuheng Cheng. Zongze Xu, and Chenglin Wu. Data interpreter: An LLM agent for data science. CoRR, abs/ 2402218679 2024a.
Sirui Hong, Mingchen Jonahhnn Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. Metagpt: Meta programming for A multi-agent collaborative framework. In ICLR. OpenReview.net, 2024b.
Jonathan
Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. ar preprint arXiv:2408. 0.445,,, 2024.
Zachary Kenton, Noah Siegel, János Kramár, Jonah Brown-Cohen, Samuel Albanie, Jannis Bulian, Rishabh Agarwal, David Lindner, Yunhao Tang, Noah Goodman, et al. On scalable oversight with weak 1ims judging strong 1lms. Advances in Neural Information Processing Systems, 37:75229-75276, 2024.
Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. Dspy: Compiling declarative language model calls into state-of-the-art pipelines. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. ReenRevie 2024.
Hunter Lang, David Sontag, and Aravindan Vijayaraghavan. Theoretical analysis of weak- to-strong generalization. Advances in neural information processing systems, 37-46837-46880, 2024.
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter
Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback, 2023. URL https:///axiiii org. 12192
Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. Gsm-plus: A com- prehensive benchmark for evaluating the robustness of llms as mathematical problem solvers, 2024a. URL
wttp://ww..rg///////////////// 19255.
12

---

Published as a conference paper at COLM 2025
Zelong Li, Shuyuan Xu, Kai Mei, Wenyue Hua, Balaji Rama, Om Raheja, Hao Wang, He Zhu, and Yongfeng Zhang. Autoflow: Automated workflow generation for large language model agents. Xiv preprint arXiv:2407.1288 2024b.
Sheng Liu, Haotian Ye, Lei Xing, and James Zou. In-context vectors: Making in context learning more effective and controllable through latent space steering, 2024a. URL
http:::/arxxv.orrr/b//2333 06666.
Yuejiang Liu and Alexandre Alahi. Co-supervised learning: Improving weak-to-strong generalization with hierarchical mixture of experts. arXiv preprint ar Xir:2400.11555, 2024.
Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. A dynamic llm-powered agent network for task-oriented agent collaboration, 2024b. URL. 02170.
Cong Lu, Shengran Hu, and Jeff Clune. Intelligent go-explore: Standing on the shoul- ders of giant foundation models. In The Thirteenth International Conference on Learning Representations, 2025a. URL https: / /tpp///veeeeeeeeeeeeeeeeeee tttt/forummidaaprrGGerr
Pan Lu, Bowen Chen, Sheng Liu, Rahul Thapa, Joseph Boen, and James Zou. Octotools: An agentic framework with extensible tools for complex reasoning, 2025b. URL https: //arxii..org/bbs/550000 11271.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xian- gru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. Crosslingual generalization through multitask finetuning, 2023. URL https: htttss://axxi..org/ab/////
01786.
OpenAL. Hello gpt-4o.
2024.
OpenAL, ;, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card, 2024. URL httpp:://arxvv.or//aaaatee
16720.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Associalion for Computational Linguistics: Human Language Techmologies, pp. 2080-2094, Online, June 2021, Association for Computational Linguistics. doi: 10.18653/ w1/2021.naacl-min.
Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, and Tushar Khot. Adapt: As-needed decomposition and planning with language models, 2024. URL https:://rxivv wwwpiiiwwog/////e///eeee abrgg//ss/31111111 05772.
Zhenting Qi, Xiaoyu Tan, Shaojie Shi, Chao Qu, Yinghui Xu, and Yuan Qi. Pillow: Enhancing efficient instruction fine-tuning via prompt matching, 2024. URL httttt://arxiv...o abs/2312. ...6221...
Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how to self-improve, 2024. URL htttt:t/arxivv.oot abs/2407. 18219.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2024. URL htt.httss//rxxi..rr///aaa
18290.
13

---

Published as a conference paper at COLM 2025
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google- proof q&a benchmark, 2023.
Tal Ridnik, Dedy Kredo, and Itamar Friedman. Code generation with alphacodium: From prompt engineering flow engineering. arXiv preprint Xii::4401.08500, 2024.
Md Omar Faruk Rokon, Risul Islam, Ahmad Darki, Evangelos E. Papalexakis, and Michalis Faloutsos. SourceFinder: Finding malware Source-Code from publicly available reposito- ries in GitHub. In 23rd International Symposium Research in Attacks, Intrusions and De- fenses (RAID 2020), pp. 149-163, San Sebastian, October 2020. USENIX Association. ISBN 978-1-939133-18-2 URL https: http:///wwwwsn.eee omar.
Jon Saad-Falcon, Adrian Gamarra Lafuente, Shlok Natarajan, Nahum Maru, Hristo Todorov, Etash Guha, E Kelly Buchanan, Mayee Chen, Neel Guha, Christopher Ré, et al. Ar- chon: An architecture search framework for inference-time techniques. arXiv preprint arXio:2409.15254, 2024.
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Language models are multilingual chain-of-thouge reasoners. In The Eleventh International Conference Learning Representations, 2023.
Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, and Chuang Gan. Easy-to-hard generalization: Scalable alignment beyond human supervision. arxiv preprint ar (rXim:2403.09472 2024.
Richard Sutton. The bitter lesson. Incomplete Ideas (blog), 13(1):38, 2019.
Leitian Tao and Yixuan Li. Your weak Ilm is secretly strong teacher for alignment, 2024. URL https:://arxii wwwpriiii.ooraaaao wrgp 2001s/2229.088.3... 08813.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. https://gi thub. com/ wwt.cwgg.chhceccceee wengfacc/rrrr teaeeerrrrrr 2020.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Repre- sentations, 2022.
Yinjie Wang, Ling Yang, Guohao Li, Mengdi Wang, and Bryon Aragam. Scoreflow: Mas- tering IIm agent workflows via score-based preference optimization. arXiv preprint orXio:2502.04306, 2025.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: more ro- bust and challenging multi-task language understanding benchmark. arv preprint arXiv:2406.01574, 2024.
Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Un- leashing the emergent cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration. arXiv preprint arXi::2307.0530 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-2488 2022.
Yupeng Xie, Yuyu Luo, Guoliang Li, and Nan Tang. Haichart: Human and ai paired visualization system. arxio preprint ar Xtv::2406..1100 11033, 2024.
14

---

Published as a conference paper at COLM 2025
Benfeng Xu, An Yang, Junyang Lin, Quan Wang, Chang Zhou, Yongdong Zhang, and Zhendong Mao. Expertprompting: Instructing large language models to be distinguished experts, 2023a.
Yiheng Xu, Hongjin Su, Chen Xing, Boyu Mi, Qian Liu, Weijia Shi, Binyuan Hui, Fan Zhou, Yitao Liu, Tianbao Xie, et al. Lemur: Harmonizing natural language and code for language agents. Xiv preprint ar X1::23100006880 2023b.
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In ICLR. OpenReview.net, 2024.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023. URL http:///arxiv.oog/abbsess http//bb//21...... 03629.
Rui Ye, Shuo Tang, Rui Ge, Yaxin Du, Zhenfei Yin, Siheng Chen, and Jing Shao. Mas-gpt: Training ilms to build Ilm-based multi-agent systems. arXiv preprint arXi::2503.0366, 2025.
Yilin Ye, Jianing Hao, Yihan Hou, Zhan Wang, Shishi Xiao, Yuyu Luo, and Wei Zeng. Generative ai for visualization: State of the art and future directions. Visual Informatics, 2024.
Xunjian Yin, Xinyi Wang, Liangming Pan, Xiaojun Wan, and William Yang Wang. Gödel agent: A self-referential agent framework for recursive self-improvement, 2024. URL https://arxiv orggabs//////////ooo abs/2410 04444.
Mert Yüksekgönül, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. Textgrad: Automatic "differentiation" via text. CoRR, abs /bb//200,07796, 2024.
Guibin Zhang, Luyang Niu, Junfeng Fang, Kun Wang, Lei Bai, and Xiang Wang. Multi-agent architecture search via agentic supernet. arXiv preprint arXiv:2502.04188, 2025.
Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. Aflow: Automating agentic workflow generation, 2024a. URL https://arxiie hwwpxii.org//bb//411111 2410 10762.
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, and Guoyin Wang. Instruction tuning for large language models: A survey, 2024b. URL
orgg 10792.
Li Zhong, Zilong Wang, and Jingbo Shang. Debug like a human: A large language model debugger via verifying runtime execution step-by-step. arXiv preprint ar Xie::2402.16966, 2024a.
Qihuang Zhong, Kang Wang, Ziyang Xu, Juhua Liu, Liang Ding, and Bo Du. Achieving 97% on gsmak: Deeply understanding the problems makes llms better solvers for math word problems. arXiv preprint ar arXio:2404.14966 14963, 2024b.
Wangchunshu Zhou, Yixin Ou, Shengwei Ding, Long Li, Jialong Wu, Tiannan Wang, Jiamin Chen, Shuai Wang, Xiaohua Xu, Ningyu Zhang, et al. Symbolic learning enables self- evolving agents. Xiv preprint ar Xiv:22406.11 18532, 2024a.
Xuanhe Zhou, Guoliang Li, and Zhiyuan Liu. Llm as dba. arXiv preprint ar 2r0::22308.05481, 2023.
Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-turn rl, 2024b.
Yiyang Zhou, Zhaoyang Wang, Tianle Wang, Shangyu Xing, Peng Xia, Bo Li, Kaiyuan Zheng, Zijian Zhang, Zhaorun Chen, Wenhao Zheng, et al. Anyprefer: An automatic framework for preference data synthesis. In The Thirteenth International Conference on Learning Representations.
15

---

Published as a conference paper at COLM 2025
Zhanhui Zhou, Zhixuan Liu, Jie Liu, Zhichen Dong, Chao Yang, and Yu Qiao. Weak-to- strong search: Align large language models via searching over small language models. arXiv preprint ar drXiv:2405.1926,, 2024c.
Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. Gptswarm: Language agents as optimizable graphs. In Forty-first International Conference on Machine Learning, 2024.
16

---

workflow function in Python block, Published as a conference paper at COLM 2025
A Technical Details
A.1 Prompt
We use the following prompts for the meta agent in W4S.
System Prompt for the Meta Agent
You AI system inprovement and ttate-of-the-are LLM agent architectures. Your nission is to evolve and optinize agentic systens through innovative prompts, strategies, and architectural patterns. Your core focus is on continuously enhancing systen performance through
specializing in LLM prompting techniques
1. analysis historical agentic systems
performance
2. Creative exploration of novel architectures and techniques
3. Systenatic improvement by optinizing the agentic system code based on empirical results
carefully study evaluation feedback identify pronising directions for improvenent. Think critically about what worked, what didn 't, and why. while maintaining system stability.
actionable insights
understanding design targeted enhancements
inprovements should boundaries through principled innovation each iteration building upon proven successes while thoughtfully exploring new approaches. fields.
broadly agent research
Main Prompt for the Meta Agent
#4$ e*Agentic Systen Interfacess:
Function you should optinize: workflowwsgent, task: str) diet Description: Solve the target task using agent.
-Input: task (str) The question/prablen to be solved. the solution; value answer" should be converted to string.
nandatory
Available [AP1s]
#** Task Rescription
The task your designed agentic systen should solve is:
[TASK]
#*# History Agentic Systems
Here is the archive of the history agentic systens and their evaluation feedback. 'system code is the code of the solver function feedb_feedback includes perfornance netrics and randonly selected validation samples:
[HISTORY]
### Output Fornat
You NUST respond with:
analysis
complete implementation
formatted EXACTLY as follows:
ppython def workflow(agent, task: str):
Fill your function.
Any helper functions import should included this
****
return return_dict
17

---

numeric LaTeX from Published as a conference paper at COLM 2025
Prompt for the Self Correction when runtime error occurs.
Errar during evaluation:
[ERROR]
WARNING: DO NOT USE ANY TRY-EXCEPT BLOCKS IN YOUR SCLUTION. the root cause the error, not to catch it.
Your task
Requirements:
Analyze
message detail
2. Explain the specific changes needed to fix the core issue Provide
the problen directly 4. Do nat include any error handling or try-except blocks
Emplementation
Please strictly Follow the following output format:
[Your analysis here]
Code:
python
def (aree,ow(agent, task: str):
\"**
Fill in your code here.
\"**
return return_dict dict
A.2 Helper Function
We implement the following APIs for meta-agent to within the workflow. The helper function description will be added into the main prompt for the meta agent.
Available APIs.
"agent. cawl__json_arrrreeeeeaeaaaeeeeee 1ormat__1mmnesssaaa temperature. nun_of_response agent_role, return_dict_keys. instructions)" call OpenAI APIS and return list of dictionary fornat responses containing the keys specified "retur__dit__kess"
"agent.call call_alllllaa cinsssn,,, temperature, nun_of mun_of_response agent_role, instructions)". Call OpenAI APIs and return list of text fornat responses.
"agent. execute_code (code)": Execute the code return the output. code MUST contain "solution" function. The output of (coeue)) will be the return value of the 'solution" function the code is executed successfully or exception.
"agennt extract_answer._e (respposee) LLM response (str).
"agent.extract. cereeettcood_bblccceeeeeeee
entry_point=' "slutionn""" Extract the code that contains def <entry_point>" from the LLM response (str).
"agent, best_on_public_tere (takk, solution_code, entry_point, test_ Execute
solution public test set, return "results" (dict), 'results(" (reuults"resutt)) result "True" or 'False" "resulcs('solution" solution '3" is the updated solution code, "results[" feedback ')" is the feedback:
A.3 Task Description
A task description T briefly describes the task and the input and output format.
18

---

Published as a conference paper at COLM 2025
A.4 Example Workflow
Example workflows anchor the format of the workflow, showing the correct function signature and how to utilize LLM call helper function. This helps avoid syntactic errors LLM call errors. Besides, the example workflow will be executed on validation set before the first iteration provide feedback.
Example workflow Wo is chosen by domain: (1) code generation, we choose CoT; (2) question-answering we choose CoT and Self Consistency CoT; (3) math reasoning, we choose CoT, Self Consistency CoT, and a CoT-with-code variant that executes the generated code Python.
B
Case Study
B.1
Case Studies for W4S
The workflow generated for MBPP
def wareflow(agent, task: entry_point: str):
instructions "Requirements:'al Please explain your solution step by step. answer
valid Python function. Aná.. clear add comments for clarity."
prompt fouour Task: \n(task] the complete function below with the function equal to (entry_point):
ressages [f"role": "user", "content": prompt]]
response = agent. jon.icloooosssnfroattt... nessagesemessages,
temperature=0.3.
nun_of_response-3,
Peent_role="Python Programmer",
(ouunnddet.keyy=("reeeoo "answer"),
instructions inntructiooossssstttttiioos.o strip().
return_dicts response correct. None
for return_dict in return_dicts:
solution_code geturnn.itt.ettt"nnseeeeee. "*) results = agent.
solution_code, eniry_point,
test_1oop=33
if results( "rssuts(rresutt"] correct_solutton results['oouttion
break
correct_solution
If no correct solution is found, take the first one correct.solution return_dicts[e]f answer'
return_dict {
"answer": str (ccttreett.ollution)) "reasoning": return_dicts[$), ("rrassonin""" ""),
return return_dict
The workflow generated for DROP
def workflow(agent task: str):
Solve the target task using current agent. Use 'agent. call_ formaaoratt_llmm to call OpenAI APIs.
Fill in your code here. helper functions inport shauld be included this function.
19

---

Published as a conference paper at COLM 2025
instructions """Requirenents:
1. Please explain step by step.
2. Please answer
question directly.
3. The answer NUST be a concise string.
4. If the problem asks for a nunber, provide it in precise float forn (e.g., use 3 instead of "three',
93.89 instead of 93).
5. Ensure a deep understanding of the context provided in the passage.
nessages [("role"; "user", "content"; Your Task:\n(task)""]
Generate multiple solutions with different temperatures responses agent. carpettccll..ssnfoortttt... ressages=messgges
temperature=0.7.
nun_of_responsess, a Generate 5 different solutions agent_role="read comprehension expert", wetptitesess"eeaaeeeeeeee mnstructions--isttuutiiin strip))
"answer"],
for response in responses:
try:
str (response. get( ("answer", ""))
answers. append (answer)
except:
continue
Ensemble prompt to select the most consistent answer ansemble_promp f"civen the task as follows: Jnsteeeal solutions have been
to address the given question. They
evaluate these solutions and identify the crucial ansemble_nessages [("role": "user", "content": ensenble__romptt]] risemble_responss agent. agent. call_json_fornat_lll ressegesnenseeblle.emsssee temperature=0.3.
fallaws: :
frequently. This consistency for determining the most reliable solution."
that appears
nun_of_responser
agent_role="read comprehension expert",
"answer"),
instructionns:nstruueeeee strip()
[00]
return_dict
"answer";
"nnswerr))
return return_dict
The workflow generated for GSMHard
workflow(agent, task: str):
You should generate valid Python code to solve the nath problen. Requirements: The code nust define solution() function and return only the final numerical answer.
Use only basic arithmetic operation.
3. Do not introduce a dead loop.
edge
messages [["role": "user", "content": f"Write Python code to solve this math problen. The code should fellow the requirements. Problen: (task)")]
responses waw...json.orrratt... nessages-messages. temperature=0.3. nun_of_response=s, Generate 5 different solutions
agent_role=" programmer".
return_dict_keysat "reasonnnn"""""""""" "cade"], instructions=programae itt.mmmeiiistuutttissssiiiiiii
20

---

Published as a conference paper at COLM 2025
answers []
respanse responses:
try:
result response.get("ceee get("code",
code response.
execute_code(codt)
ssinstance(resull (int, fleat)):
answers. append(result) except Exception 55 continue
not answers:
Fallback reasoning if volid code is generated mathhexpert_instructie
Requirements:
1. Please explain step by step.
2. answer MUST a float.
nessages [("role": "user", "content": f"A Your Task:\n(task)"]
respanse seent.call.jsonffoomtt_llll messages-nessages temperature=9... nur_of_response=l)
agent_ expert",
"etun__iit..kes=""rreaon "answer"],
return response(0]
return_dicti "aun_dditt"ansser") str(return_ dicturrndditttgtteeeee setaaddt"e..et"seeee"" (0.0")) return return_dict
Use self-consistency
the
from collections import Caunter
answer_counts Counter(answers) nost_common_ answer answer_counts msseer_conts.mott_commmm most__commonn (1) (8]
return_dict "answer"; str comoto__emoo)))))rr)
return_dict
The workflow generated MATH
def workflowwagent, task: str):
# Define the instructions for the LLM
instructions v.a
Requirements:
Please explain step by step.
2. The answer HUST be formatted correctly.
3. the task requires numerical answer, provide it as precise number or LaTex expression.
* Call the LLH with the task and instructions messages [("role": "user", "content"; f"* Your Task: Ask::\nttskk))) cgl nessageszmessages, tamperaturesa.7. run.of_response=5, agent_role=" expert". mnstructions=inssrruttiisss strip(),
Extract answers from the responses
[agent. extract_answer.ct (respanse) for response response]
Count frequency
from collections import Counter
answer_cound (ansters))swers)
Get the majority answer
answwr. answer_counts most_common (1)
21

---

Published as a conference paper at COLM 2025
# Format the answer correctly
try:
a Try to convert the answer float Float_answer [loat(majrityy_mm if Float_answer- is_integer
majority_anseer int(fleat_answer)
majority.answer float_answer except ValueError:
If not nunber, keep it is pass
Create the return dictionary return_dict t'answer"; anjoitty_nnswer return return_dict
The workflow generated for MMLU Pro
def workflow task: str): collections import
import randon
def get_initisl_ cea.tiaaarresoosesseeeee agent_role):
nessages [("role": "user", "content"; f" Your Tnsk:In(task)
responses sgent.call.soonfoontt.
messoges=messages temperature=0.7.
nur_of_response=5
sgent_role=agent.rrlle.
return_dict.keys=""rreaaee Ant. Please explain step by step. An2. The F
"answer"],
return responses
def refinn_ressonse initial_response. agent_role):
("role": "user", "content": f"a Your Task An(kskn((ss)"),
("role": "assistant", "content"; f"Your initial solution AnReasoning: (initial_ reasoning mAnssee::::
rinpoial answer "1""
refined_response agent. fell__jsoooorrst______.
messoges=iessaage,
temperature=0.35
mur_of_respons==l,
agent_roleeaget_rrols
"revised_answer"",
instructtioss="RRqureeeeent Consider other experts solutions
carefully.An2. Provide inproved reasoning needed. The
revised_answer MUST J."
ar E F or
[[8]
return refined_response
def get_Final_answer (refinnd_reepoosse]]
answers = [response[ revisseeeeviiieneeeeee for respanse in refiine_.
Counter (answers)
nost_common_ansee answer_counts sesser_counts.msst.commm. (1) most_comnon_ansee
Dynamic role assignment based on task complexity agent_roles ["Knowledge and Reasoning Expert", "Scientist", "Critical Thinker"] len(task.split(() 28:
agent_roles agant_roles[:2]] Simplified task, fewer roles
Initial responses
initial_ = for role agent_roles:
role))
Refine responses
22

---

Published as a conference paper at COLM 2025
Domain
Dataset
#Validation
#Test
MGSM
128
800
GSM Plus
128
800
GSM Hard
128
800
Math Reasoning
GSMSK
128
SVAMP
128
800
MATH
119
486
MBPP
341
Code Generation
HumanEval
33
131
DROP
128
800
Question Answering
MMLU Pro
128
GPQA
60
138
Table 4: Dataset Statistics.
defined_responee = []
respanse in initial_ initial_responser
responesessponses. append (esponneeessopssseeeeeeeeeee (task, response,
randon. (ho.ice(eggetttroee)))
$ Get final answer
final_answer aee_finnl_anseerrree (rweeaeeeeeeeeeeeeeeeeeeeee
return_dict
"answer": final_
return return_dict
C More Related Work
LLM Post-Training Modern LLMs undergo various post-training processes to enhance task-specific capabilities and align outputs with human preferences, including instruction tuning (Zhang et al., 2024b; Muennighoff et al., 2023; Feng et al., 2024; Qi et al., 2024), preference learning (Rafailov et al., 2024), and reinforcement learning (DeepSeek-A) 2025; Zhou et al., 2024b). Our W4S framework is most closely related to multi-turn RL algorithms for LLMs. Qu et al. (2024) employed multi-turn RL to train language models in self-correction and self-improvement, while Zhou et al. (2024b) developed hierarchical multi-turn RL for training LLMs on complex interactive tasks. Unlike these approaches that directly enhance model capabilities, W4S trains weak meta-agent to harness stronger models without modifying their parameters.
al.,
D More Implementation Details
D.1 Datasets
We evaluate W4S on eleven datasets, including mathematical reasoning, question answering and code generation. For MATH, MBPP and HumanEval, we follow the data splits in Zhang et al. (2024a). For the other datasets, we follow Hu et al. (2024) and randomly split the dataset into validation and test splits. The dataset statistics are included in Table
D.2 Baselines
We evaluate W4S against several established methods, organized into three categories.
First, we include standard LLM approaches: Vanilla (direct LLM invocation) and 5-shot prompting. Second, we compare against six hand-designed agentic workflows: (1) Chain- of-Thought (COT) (Wei et al., 2022), (2) Self-Consistency with Chain-of-Though Thought (COT-SC)
(Wang et al., 2022), (3) Self-Refine (Madaan et al., 2023), (4) LLM Debate (Du et al., 2023), (5) Quality Diversity (Luu et al., 2025a), and (6) Dynamic Assignment (Xu al., 2023a). Finally,
23

---

Published as a conference paper at COLM 2025
Hyperparameters
Value
Learning Rate
le-5
Training Epochs
Number GPUs
LR Scheduler
cosine
Per Device Batch Size
Gradient Accumulation Steps
Table 5: Hyperparameters for Training with W4S.
benchmark against two recent automated workflow design methods: ADAS (Hu al., 2024) and AFlow (Zhang et al., 2024a).
In COT, we prompt the LLM to think step by step before answering the question. In COT-SC, we sample n = answers and then perform an ensemble by either a LLM query (QA, Code task) or majority voting (Math task). For Self-Refine, we allow up five refinement iterations, with an early stop if the critic deems the answer correct. In LLM-Debate, each debate module is assigned unique role, such as Math Expert or Physics Expert, and the debate lasts for two rounds. In Quality-Diversity, we conduct three iterations to collect diverse answers based on previously proposed ones, In Role Assignment, we use one LLM query to first choose a role from predefined set, and then use another LLM query to answer the question by acting within the chosen role.
For the hand-designed workflow implementations, from the ADAS framework to ensure fair comparison. For AFlow, we reproduce the results using their official codebase and im implementation.
adopt the standardized versions
D.3 Details for Data Collection
In our experiments, we set the number of candidate samples mI and select the best- performing action to determine the next state. We filter out actions yielding workflows with extremely poor performance to ensure quality. Trajectories of 10 iterations per task in Table 1 and 15 iterations per task in Table 2. To manage com- putational efficiency, we apply trajectory truncation with horizon of T = resetting the state every two iterations and correspondingly resetting the maximum historical validation performance.
collected over maximum
D.4 Implementation Details
Hyperparameters for Fine-Tuning with W4S. For finetuning, we utilize the TRL (von Werra et al., 2020) codebase, but we customize the loss function and the dataset preprocessing, The base models are directly loaded from Hugging Face: Owenn2.-CCoder7B--nntttu The hyperparameters used for finetuning are specified in Table 5.
Hyperparameters for Inference. For inference, we employ the meta-agent with temper- ature of 0.5 to sample once for each iteration, different from best-of-m sampling during training. In order to keep consistent with the training data, also apply trajectory trunca- tion during inference, with a horizon T = 2.
E More Experimental Results
E.1 Limitation of Previous Work
Figure 5 illustrates a key limitation of ADAS, The 'Sequential' condition reflects its standard setup, where the history archive is updated each iteration, while 'Random' involves gen- erating 30 independent workflow samples in the initial iteration. The results reveal that ADAS's sequential performance is comparable to random sampling, with its peak accuracy failing to surpass the best outcome from the 30 random samples. This suggests that ADAS struggles to leverage historical information effectively for iterative improvement.
24

---

Published as a conference paper at COLM 2025
Sequential Iteration vs Random Sampling
Random (wlo History Update)
Sequential
50
Test Accuracy (%)
30
5
10
15
20
25
30
Iteration
Figure 5: The Test Accuracy (%) of ADAS on MGSM dataset. 'Sequential" denotes the default configuration, updating the history archive iteratively; 'Random' indicates 30 independent workflow samples generated in the first iteration. Results show that ADAS's sequential performance closely mirrors random sampling, with its maximum accuracy not exceeding the best random sample.
Execution LLM
GPT-4o
Claude-3-5-sonnet
Dataset
MBPP
Vanilla
75.9
77.7
+W4S
90.9 (+15.0%)
89.8 12)1)%
Dataset
GSM Hard
Vanilla +W4S
55.0
77.6 (+22.6%)
78.2 (+24.4%)
Table Cross-model transferability of W4s. The meta-agent is trained for harnessing GPT-4o-mini. We report the performances before and after equipping the Execution LLM with the designed workflow.
E.2 Cross-Model Transferability
Table demonstrates the cross-model transferability of W4S. We train the meta-agent to optimize workflows for GPT-4o-mini, and directly transfer the workflow designed for GPT-4o-mini to other models.
E.3 Cross-Dataset Transferability
Table demonstrates the cross-dataset transferability of W4S. We train the meta-agent for GPT-4o-mini on one dataset, and directly transfer the optimal workflow to other datasets.
Dataset
MBPP - H-Eval
GSM-Hard MCSM
MMLU GPQA
GPQA MMLU
Vanilla
87.7
82.9
39.1 44.4 (+5.3%)
56.1
W4S
(+8.7%)
87.4 (+4.5%)
64.1 (48%)
Table 7: Cross-dataset transferability of W4S. The Execusion LLM GPT-4o-mini. "MBPP-+H-Eve means we train our meta-agent on MBPP, and evaluate on Eva. We report the performances before and after equipping the Execution LLM with the de- signed workflow.
25

---

Published as a conference paper at COLM 2025
E.4 Comparison between Direct Task Training and W4S
We compare directly training a similarly sized open-weight model with GRPO on the validation dataset versus using W4S. On GSM Hard, W4S paired with strong executor GPT-4o-mini remains substantially better under limited compute.
Table 8: Performance comparison between W4S with untrained baselines and directly training weak models on validation dataset.
Model / Setting
Train?
Acc.
(CoT) Owen2.5-7B-Instruuc (GRPO)
No
32.2
Yes
52.8
GPTT-o-mini (Col)
No
39.5
GPTT-4-mini (W4S)
Yes
76.6
E.5
Sensitivity to Initial Workflow Wo
We vary the initial example workflows Wo and compare the performance after training. Table 10, 9 and 11 demonstrate the sensitivity results on three different datasets with different task types. 'CoT w/ Test' means using CoT strategy followed by testing the generated code on the public code test set. 'CoT w/ Code' means instead of instructing the models generate text-based answer, instructing the models generate step-by-step reasoning with a code as final output and executes the code to get the final answer.
Table 9: MBPP: sensitivity to Wn.
Wo
Acc.
CoT
86.8
CoT TTTest
87.6
CoT CoT-SC x 5 CoT w/ Test
87.9
Table 10: DROP: sensitivity to Wo.
Wo
Acc.
CoT
87.9
CoT-SC x 5
87.3
CoT + CoT-SC
87.5
E.6
Helper Function Usage Statistics
We quantify the prevalence of tool tool/helper usage inside learned workflows.
E.7 Training Cost Analysis
Training the weak meta-agent five datasets (DROP, MMLU Pro, MBPP, GSM Hard, and Math) requires approximately H100 GPU (30 minutes on 2 GPUs). Training on single dataset requires only about 0.2 GPU hour. The APT cost for collecting training trajectories varies by dataset, about 10S 20$ USD per dataset, with GPT-4o-mini as executor L.MSs. These computational and API cost could be further amortized when applying the trained meta-agent to multiple unseen datasets without additional training. We anticipate even stronger generalization capabilities when the meta-agent is trained across a more diverse range of domains.
26

---

Published as a conference paper at COLM 2025
Table 11: MGSM: sensitivity to Wi.
Wo
Acc.
CoT
65.5
CoT-SC
66.8
CoT w/ Code
61.5
CoT CoT-SC CoT w/ Code
68.2
Table 12: Number of helper functions used per task (typical best workflows).
MATH
MBPP
MMLU-Pro
GSM Hard
# helper functions
2
2
6
3
27


Reviews:
[{"review_id": "ycnp2fm21v", "reviewer": "Reviewer_iWtW", "summary": "This paper proposes the novel idea of using a weak model to generate workflows for a strong model to execute and using offline RL to train this weak model for specific tasks. The authors present this framework as an alternative to training the strong model. They evaluate their framework on eleven benchmarks, covering math-reasoning, question-answering, and code generation. Their framework achieves superior performance across the benchmarks compared with the strongest workflow optimization baselines while being more efficient in time and cost in both workflow generation and execution.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The task formulation of using a weak model to generate workflows so that training can be done just on the weak model is novel. \n\n2. The superior efficiency in both workflow optimization and execution compared with other workflow optimization baselines show the promise of the overall approach in real-life applications.", "reasons_to_reject": "1. Lacks baselines that directly train the strong model. The only such baseline, finetuned GPT4o-mini, just uses supervised finetuning on the validation set, so it’s not surprising that the results are unsatisfactory. The authors should also try other training methods for the strong model, such as more data-efficient approaches and common algorithms of online and offline RL. Other open-sourced strong models should easily support different training algorithms. \n\n2. Using a meta-agent alone without RLAO training already improves performance on several benchmarks, and the improvements from RLAO seem marginal on DROP, MMLU pro, MBPP, and HumanEval. \n\n3. In Table 2, the “unseen” math tasks seem similar to the seen tasks, so it’s unclear whether the experiments can serve as evidence of generalizability. \n\n4. Lacks comparison between the RLAO algorithm and other algorithms used for improving LLM coding and tool use. It would be helpful to see discussions on which parts of RLAO are novel.\n\n5. The experiments only use one model as the weak meta-agent, so it’s unclear if the method generalizes to other weak models. Why did you choose Qwen2.5-Coder-7B-Instruct? Are there anything specific about this model that makes the framework work well?", "questions_to_authors": "1. Have you tried putting different caps on the inference-time compute allowed for the executor? Maybe it’s not happening in your experiments but is it possible that some meta-agent would learn to just let the executor scale up inference-time compute?\n\n2. Do the selected tasks adequately reflect the real-life tasks that require the strong model to solve? Can some of them be solved by the weak coding model directly?", "extra_scores": {}}, {"review_id": "DhXi50LxvW", "reviewer": "Reviewer_4Sc7", "summary": "This work proposes a new framework (W4S) for optimizing LLM workflows that use weaker meta-agents to harness stronger models for execution. The system is built on two key concepts: 1) instead of a node-based workflow, W4S uses code generation with workflow interfaces (enabling more expressiveness) 2) instead of a training-free approach, or tuning a large model, W4S uses RLAO to tune a weaker model. W4S is evaluated on a range of reasoning and QA LLM benchmarks, comparing against hand-crafted workflows and training-free related work. W4S demonstrates superior performance to both across seen and unseen tasks. A cost analysis is given showing that W4S is more cost effective than training-free approaches ADAS and AFlow.\n\n----\n**update**\n\nI have updated my score based on the response that provides further quantitative and qualitative evaluation that makes it easier to understand the types of workflows generated by W4S and the improvement with respect to related work.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Interesting and well-motivate problem domain.\n- Novel implementation that improves over the selected baselines and related work.\n- Clear presentation and writing.", "reasons_to_reject": "The paper offers good results however it could be improved through more detailed analysis and explanation. Giving a stronger presentation for how the demonstrated improvements are achieved, and what the system is learning, would lift this paper and warrant a higher score.\n\n*Unclear why the system is better:*\n\nThe W4S system describes two notable differences to related work (AFlow): a training approach using RLAO, and a code-based approach rather than graph-based. The presentation does not make it clear how code-based workflows improve the system. Are the resulting workflows produced by W4S inexpressible by AFlow, or are the resulting workflows simply more effectively instantiations of the available techniques/nodes?\n\nWhat does the model learn through training? The generated workflow for MATH appears like a standard combination of CoT and Majority answer. How does it compare to the AFlow solution (why is it better by nearly 5% compared to AFlow and COT SC). Why does W4S beat AFlow by 24% on GSM Hard, but only  8% on MGSM)?\n\nBroadly, it would be interesting to understand what contributes to the improvements. Is it optimization of hyperparameters, prompt wording, novel combinations of approaches, or stacking multiple approaches?\n\n*Limited to well-known reasoning and QA tasks:*\n\nThe authors acknowledge this limitation and the results are compelling even within this domain. I have two main concerns:\n\nAs mentioned below, these are well studied datasets, and while solutions may not leak into training sets, approaches and workflow code, could. Is there a risk of this?\n\nThe authors present generalization results from the seen GSM Plus and MGSM to the unseen GSM8k, GSM Hard, and SVAMP. However, many of these datasets are similar to, or derivations of, GSM8k. Do these datasets offer significant differences and challenges with respect to the workflow code required to solve them, seeing as they are all math problems? A more detailed comparison would be helpful here. For instance, how does performance differ if the GSM Plus workflow is applied to GSM Hard, or vice versa? If there is a significant difference that would be a stronger indicator of differences between the datasets.\n\n*Only one model is evaluated as trained weak meta-agent:*\n\nOnly one model is used (Qwen2.5-Coder-7B-Instruct) as a trained weaker-agent. To what extent does code or instruction tuning matter? Given that many of the datasets are publicly discussed in research papers, and LLM workflows for these datasets exist on GitHub (not the test sets themselves), is it possible that a model could be biased towards generating better workflows for these datasets via exposure during pre-training? Discussion of model choice or comparisons using other weak models would improve the results. As would discussion of whether exposure to existing workflow code or approaches could be an issue.\n\n*‘Weak for strong’ narrative is unconvincing:*\n\nIn this work, the best results are achieved through a training approach not generally accessible on strong models (due to cost or API restrictions). However, if such a training framework was available, would we expect the resulting system to be better than one trained using a ‘weak’ or ‘weaker’ meta-agent?\n\nFurther, from a cost-argument, given that workflows employ strong models that can be executed many times once built, is it reasonable to expect that the overall cost of a system will be dominated by the workflow execution, with costs of workflow optimization amortized across usage?\n\nI applaud the authors for demonstrating SOTA results using a weak meta-agent, and my concern here is not the lack of using strong meta-agents. My concern here is using terminology that was motivated by fundamental differences in a context that is driven by practical differences. For example, the labels (reward) used in W4S are the same throughout the system (the answer to the qa or reasoning tasks), rather than weak labels from humans for superhuman tasks. Is there a scientific reason that motivates the weak/strong distinction?\nMy concerns could addressed in multiple ways:\n- Stronger cost analysis that demonstrates the gains of using a weaker meta-agent over a realistic usage scenario (not a benchmark evaluation).\n- Reducing or clarifying the use of ‘Weak for strong’ which intentionally draws comparison to (Burns et al. 2024) but lacks the same fundamental motivation.", "questions_to_authors": "(115) How is w_0 chosen. How sensitive is the system to the initial workflow example?\n\n(Table1) Is the SFT on 4o-mini using the same dataset as W4S w/ SFT, or a different dataset?\n- If it is different, why not FT 4o-mini using the same dataset. Is it due to cost?\n- If it is the same, could you elaborate on why W4S is better? \n\n(Table2) Why does Table 1 use 4o-mini and Table 2 use 3.5 Turbo?\n\n(Table3) AFlow has cheaper inference cost (0.3 vs 0.5) but much higher wall-clock time (10.9 vs 2.7). Could you explain why W4S is much faster? Is it due to parallelization in the workflow code?\n\n(Workflow for GSMHard) The generated workflow uses the prompt: “The code must define a solution() function and return only the final numerical answer.” which matches the dataset description, but does not seem a requirement for the validation process (that just compares returned answer). How did the system learn to do this? Was it from the task description, or the pre-training or fine-tuning?", "extra_scores": {}}, {"review_id": "k5b9zPZNF3", "reviewer": "Reviewer_vbtG", "summary": "This paper presents W4S, a meta-agent framework that aims to train a small model capable of designing agentic workflows that involve stronger models. The weaker model (a 7B Qwen instruct model) is trained to maximize the performance of a workflow on a validation set, optimized via offline RL with best-of-m rollouts. Using this meta-agent as a \"controller\" for stronger prompted models (GPT-3.5 and GPT-4),  authors show better workflows than hand-designed methods / existing optimization approaches across multiple benchmarks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The motivation and overall pipeline of W4S framework seems very intuitive and promising. The framework reminds me of prompt-tuning methods such as textgrad, but this one focuses more on improving agent-style workflow which is timely.\n- The authors provide results across multiple benchmarks and demonstrate consistent improvements in these tasks; notably the improvement does not require much compute despite involving additional training of the meta-agent.", "reasons_to_reject": "- While I like the framing, \"Weak-for-Strong\" is misleading. In fact, off-the-shelf Qwen2.5-7B-Instruct (the weak model here) achieves much stronger performance in many tasks than what is reported as performance of W4S. For example, the weak model achieves 75.5 on MATH (https://qwenlm.github.io/blog/qwen2.5-llm/) while W4S with GPT-4o-mini (along with fine-tuned Qwen2.5-7B-Instruct as meta-agent) achieves 63.0 in Table 1.\n- **Issues with clarity and (seemingly) arbitrary design choices**: A lot of components in the method but are not well motivated (at least does not explain why we need those components). For example, why do we need “example workflow” as input to the agent (line 115)? What exactly is the example workflow? How granular is the task T (i.e., does it require multiple calls to agents or tools, or is it just a one-shot API call to a stronger model)? How do you define \"public and private validation sets\" (line 139)? What happens when the current iteration is skipped due to error (line 143)? Do we have some fallback option? Some of these questions are not answered until the end of the paper, some gets only clearer after checking the examples in the appendix. Perhaps a better way of presentation is to include a concrete example in the method, and follow along that example for illustration.\n- Adding to above, what is exactly done at test-time? The output of training phase would be the trained specialized meta-agent (based on Qwen2.5) capable of generating workflows. Now, at test-time, do we just sample a trajectory from the trained meta-agent? What is the input to the model during test time? Is it just the task description T (along with instruction, example workflow and feedback) for the benchmark of interest?\n- **Need for Qualitative Analysis**: An important aspect of “workflow engineering” is that practitioners understand how qualitatively the generated workflows are different from canonical methods, as the proposed method introduces additional overheads such as model training and validation. The authors fail to deliver such qualitative analyses, such as how different the automated workflows are from typically used prompting methods, or what failure cases they bring in, or the impact of pre-defined helper functions (e.g., does the model learn to effectively use helper functions? What helped in what tasks and what did not?). In fact, the example workflow for GSM (line 595) doesn’t look any different from a typical Program-of-Thought prompting.\n- As far as the reviewer knows, RWR is not an offline algorithm as authors state (https://arxiv.org/pdf/2107.09088, https://pure.mpg.de/rest/items/item_1790444_3/component/file_3075658/content). What the authors are doing is just a single iteration (in an EM framework) of RWR where the trajectories are sampled from current policy and then the policy is trained with return-weighted log likelihood.\n- Overall, the tasks evaluated in the paper are relatively traditional problem-solving benchmarks such as GSM and DROP. While these benchmarks are easier to evaluate and iterate, they may not be the best place for sophisticated agentic workflows -- as noted in the first bullet point, off-the-shelf LMs are amazingly good at these tasks in zero-shot setting. I suggest the authors to consider more complicated benchmarks like SWEBench as the testbed for W4S, where simple CoT won't just work.\n\nOverall, I think the suggested framework has a lot of potential, but the paper would benefit from another round of editing.", "questions_to_authors": "- Typo: Figure 2 caption: using ⇒ use\n- Equation below line 158 seems wrong. argmax is over the index in 1 ~ m, but the LHS assigns that to $a_i^*$.\n- How does the best-of-m selection of action enriches training dataset with both successful and unsuccessful attempts?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents W4S, a framework where a small meta-agent generates workflows to leverage stronger untrainable API-based LLMs. By combining code-based workflow generation with offline reinforcement learning and best-of-m action selection, W4S enables efficient orchestration of powerful models. The method is evaluated across 11 benchmarks—including math, QA, coding, and the GAIA agentic benchmark—consistently outperforming state-of-the-art baselines like ADAS and AFlow. Key strengths include a strong empirical results, efficiency and broad generalization. It excels at automating and optimizing workflow composition rather than inventing entirely new prompting strategies. Overall, this is a well-executed contribution and I recommend acceptance.
Decision: Accept