Helmsman: Autonomous Synthesis of Federated Learning Systems via Collaborative LLM Agents
Authors: Haoyuan Li
Mathias Funk
Aaqib Saeed
Venue: ICLR 2026 Poster (2026)
Keywords: Multi-agent System
Federated Learning
LLM-based Agent
Primary area: foundation or frontier models, including LLMs

Abstract: Federated Learning (FL) offers a powerful paradigm for training models on decentralized data, but its promise is often undermined by the immense complexity of designing and deploying robust systems. The need to select, combine, and tune strategies for multifaceted challenges like data heterogeneity and system constraints has become a critical bottleneck, resulting in brittle, bespoke solutions. To address this, we introduce Helmsman, a novel LLM-based multi-agent framework that automates the end-to-end synthesis of federated learning systems from high-level user specifications. It emulates a principled research and development workflow through three collaborative phases: (1) interactive human-in-the-loop planning to formulate a sound research plan, (2) modular code generation by supervised generative agent teams, and (3) a closed-loop of autonomous evaluation and refinement in a sandboxed simulation environment. To facilitate rigorous evaluation, we also introduce AgentFL-Bench, a new benchmark comprising 16 diverse tasks designed to assess the system-level generation capabilities of LLM-driven agentic systems in FL. Extensive experiments demonstrate that our approach generates solutions competitive with, and often superior to, established hand-crafted baselines. Our work represents a significant step towards the automated engineering of complex decentralized AI systems. Code is available at: https://github.com/haoyuan-l/Helmsman.

Published as a conference paper at ICLR 2026
HELMSMAN: AUTONOMOUS SYNTHESIS OF FEDER- ATED LEARNING SYSTEMS VIA COLLABORATIVE LLM AGENTS
Haoyuan Li, Mathias Funk & Aaqib Saeed Eindhoven University of Technology, The Netherlands fh.y.li, m. funk, a. a.saeed]etue...
ABSTRACT
Federated Learning (FL) offers a powerful paradigm for training models on decen- tralized data, but its promise is often undermined by the immense complexity of designing and deploying robust systems. The need to select, combine, and tune strategies for multifaceted challenges like data heterogeneity and system constraints has become critical bottleneck, resulting in brittle, bespoke solutions. To address this, we introduce Helmsman, novel multi-agent system that automates the end- to-end synthesis of federated learning systems from high-level user specifications. It emulates a principled research and development workflow through three collab- orative phases: (1) interactive human-in thnnt-hoooppp planning to formulate a sound research plan, (2) modular code generation by supervised agent teams, and (3) a closed-loop of autonomous evaluation and refinement in a sandboxed simulation environment. To facilitate rigorous evaluation, we also introduce AgentFL--Bench new benchmark comprising 16 diverse tasks designed to assess the system-level generation capabilities of agentic systems in FL. Extensive experiments demon- strate that approach generates solutions competitive with, and often superior to, established hand-crafted baselines. Our work represents a significant step towards the automated engineering of complex decentralized Al systems. Code is available at: https::...............tt.....h / g thub. com/ hwwyuan- 1 H/He msman.
INTRODUCTION
Federated Learning (FL) holds immense promise for privacy-centric collaborative AI, yet its practical deployment is complex (Luping et al., 2019; Wu et al., 2020; Liu et al., 202 Designing effective FL system requires navigating a range of challenges, including statistical heterogeneity, systems constraints, and shifting task objectives (Marfoq et al., 2021; Jhunjhunwala et al., 2022; Lu et al., 2024). To date, this design process has been a manual, labor-intensiv effort led by domain experts, resulting in static, bespoke solutions that brittle in the face of real-world dynamies. This paper argues that this manual design paradigm is the critical bottleneck hindering the widespread adoption FL (Li et al., 2020b; Linardos et al., 2022; Tang al., 2023). We propose to address this bottleneck by introducing Helmsman, new frontier focused on the automated synthesis of FL systems using autonomous AI agents. While recent breakthroughs have produced LLM-based agents capable of remarkable feats in general-purpose purpose code generation (Zhang et al., 2024; Tao et al., 2024a; Islam al., 2024; Nunez al., 2024; Tao et al., 2024b; Novikov al., 2025), their ability to reason about and systematically construct the complex, multi-component systems required for robust federated learning remains fundamental open challenge.
The difficulty of FL design is rooted in its vast and combinatorial nature. Real-world deployments rarely present a single, isolated challenge. Instead, they involve a confluence of issues: clients may have non-IID data (Li et al., 2020a; Reddi et al., 2020), heterogeneous computational capabilities (Diao et al., 2020; Kim et al., 2023), and unreliable network connections (Chen et al., 2020; Liu et al., 2024a), all while pursuing diverse task objectives (Marfoq et al., 2021; Lu et al., 2024). Current FL research often operates in silos, developing point solutions for individual problems. While effective in isolation, these solutions are difficult to compose, and their interactions unpredictable. This leaves

---

Published as a conference paper at ICLR 2026
Planning
Coding
Evaluation
/aeed to depicy on abi
Planning
Supervisor
Modelar Coding Groups
Debugger
Code
rops 12 rebill alonco
cansection claee
Human Approval
Clenk
Reshect- 18
Mooistarie
Stranogy
Server
Orchestrators
User
Feedbock
Plan
Updit
4. Setup
2. Summery
Challanges
Strotogy
Guide
Task
MocKa Plae
Server
Sieblttio 1 Cupley
B13
Research Plan
Strenag
Server Coda
RL 9
Tools
Self-reflection
Implementation
Codebase
Evaluator
The automated FL development workflow of Helmsman. (a) Planning: A user query is refined into an actionable research plan via human-in- in-the--ooooop dialogue. (b) Coding: Specialized agent teams, managed by Supervisor, collaboratively build a modular codebase. (c) Evaluation: The final code is autonomously tested and refined in a closed simulation loop until correct.
Figure 1:
practitioners facing an intractable design space, underscoring the need for an automated approach that reason holistically about multifaceted constraints and synthesize integrated, deployment-ready solutions.
While the need for automation is clear, existing agent-based code generation frameworks are ill- equipped for the unique demands of FL.. Systems like those proposed by Muennighoff al. (2023) and Dong et al. (2024), often built single-agent architectures with prompting techniques like Chain-of-Thought (Wei et al., 2022) or ReAct (Yao et al., 2023), excel at solving self-contained programming tasks. However, they struggle when confronted with the system-level complexity of FL. The challenge is not merely to generate correct algorithm, but to orchestrate an entire distributed system comprising interdependent modules for data handling, client-side training, server- side aggregation, and overall strategy. This requires reasoning about algorithmic trade-offs, resource constraints, and communication protocols simultaneously- holistic design task that exceeds the capacity of monolithic, single-agent approaches and is amplified by the need for a realistic simulation environment for validation.
To bridge this gap, we introduce Helmsman, multi-agent system designed to automate the end- to-end research and development of task-oriented FL systems. navigates the complex FL design space by structuring the process into three collaborative phases: (1) Interactive Planning, where human-in-thee -he--oooooopppp process refines a high-level user query into a verifiable research plan; (2) Modular Coding, where specialized agent teams collaboratively implement the plan across distinct framework components; and (3) Autonomous Evaluation, where the integrated codebase is executed, debugged, and refined closed-loop simulation environment. By automating the time- and resource-intensii cycle of design, implementation, and testing, Helmsman dramatically lowers the barrier to entry for creating robust and sophisticated FL. solutions for both experts and non-experts FI. alike. Our key contributions are:
We develop Helmsman, an end-to-end agentic system that translates high-level specifica- tions into deployable FL frameworks through interactive planning, modular coding, and autonomous evaluation.
We introduce AgentFL-Bench, a benchmark with 16 tasks across 5 research areas, designed for the rigorous and reproducible evaluation of automated FL system generation.
We conduct extensive experiments our benchmark, demonstrating that solutions generated by Helmsman achieve performance competitive with and often exceeding that of established, hand-crafted FL baselines.
2 BACKGROUND
2.1 RELATED WORK
key insight from recent agentic Al research that complex problem-solving is often best addressed not by single monolithic agent, but by collaborative ensemble of specialized agents. This multi-
2

---

Published as a conference paper at ICLR 2026
Tasks
Challenges
Strategies
7220000712020
Fed-il
Semmnnaret ooney
KuRa (2023)
One-shoc FL
(2305)
Derconaiond Tt.
Hctoront, (2020)
SedErm 12000)
Fed-Q.
Data litermgenatty
TICL (204)2004)
DeothEL:2000
COMCRA (2023)
CR5L120011)
Coloso ophic Fongcting
(2024)
Freameen eity
Testiwett (2301)
Pafe te Aga maa Anadks
7242ET:22233)
Ligttassgg (2322)
Secure 429 ip20ion
NuthRound Seclog (2033)
Syitlam hetersgeneity
Futh5 (2524)
hasy
Robustness
FedLC (2002)
Figure 2: The intractable design space of FL, created by the combinatorial task of matching diverse challenges with specialized strategies.
agent paradigm, rooted in the principle of "division of labor," has proven highly effective in automated code generation. For instance, systems like AgentCoder (Huang et al., 2023) separate the roles of code implementation and test case generation, while CodeSim (Islam et al., 2025) introduces simulation-driie process to verify plans and debug code internally. This approach of decomposing a complex
emulating the human programming cycle of retrieval, planning, coding, and debugging (Islam et al., 2024) consistently enhances the robustness and quality of the generated solution.
The power of this collaborative approach extends beyond self-contained coding tasks to system-level engineering and scientific discovery. Multi-agent teams have demonstrated superior performance in resolving real-world GitHub issues in benchmarks like SWE-bench (Jimenez et al., 2024; Chen et al., 2024; Liu et al., 2024b), navigating complex engineering design spaces (Ni & Buehler, 2024; Ocker et al., 2025), and even automating scientific discovery by generating and testing novel hypotheses (Gottweis al., 2025; Naumov al., 2025). These successes highlight clear trend: as the complexity of task grows, so too do the benefits of structured, multi-agen agent approach. However, despite its proven potential, this paradigm has not yet been systematically applied to the unique, multifaceted challenges of designing and deploying Federated Learning systems. This is the critical gap our work addresses.
2.2 MOTIVATION
Current FL research practices are struggling to keep pace with the field's growing complexity. The manual design of FL systems is bottlenecked by what term the intractable design space, which arises from three core issues:
Combinatorial Complexity of Strategies. Real-world FL problems rarely one-dimensional. As illustrated in Figure 2, typical deployment requires addressing a combination of challenges spanning data heterogeneity, system constraints, robustness, and more. For each challenge, plethora of specialized algorithms exist-FedProx for stragglers (Li et al., 2020a), SCAFFOLD for client drift (Karimireddy et al., 2020), FedNova for system heterogeneity (Wang al., 2020), etc. Manually selecting, combining, and tuning these strategies for specific problem is a combinatorial task that quickly becomes intractable for a human expert, highlighting the need for a system that can automate this exploration.
Brittleness in Dynamic Environments. Hand-crafted solutions are often static and optimized for specific set of assumptions about client data, network conditions, and model architectures. Consequently, they are brittle and behave unpredictably when deployed into dynamic, real-world environments where these conditions may fluctuate (Wu et al., 2022; Li et al., 2021). For instance, a strategy may be sensitive to the number of participating clients (Jhunjhunwala et al., 2022; Xu et al., 2021) the degree of resouree heterogeneity (Diao et al., 2020; Kim al., 2023). An automated system must be able to design solutions that are not only performant but also robust to such environmental variations.

---

Published as a conference paper at ICLR 2026
Dichotomy in FL Frameworks. The FL ecosystem is split between two poles. On one hand, research frameworks like Flower (Beutel et al., 2020) and PySyft (Ziller et al., 2021) offer the modularity and flcxibility essential for rapid prototyping and innovation. On the other, industrial platforms like FATE (Liu et al., 2021b) and NVIDIA FLARE (Roth et al., 2022) prioritize the scalability, robustness, and operational efficiency required for production. This dichotomy creates a gap between experimental research and deployable solutions. An ideal automated system must bridge this gap, combining the innovative flexibility of research frameworks with process that ensures production reliability.
AGENTIC FL SYSTEM
To systematically navigate the intractable design space of federated learning, introduce Helmsman, multi-agent framework designed to emulate the principled, iterative workflow of human-led research and development. Our system transforms a high-level user objective (or query) into fully functional and validated FI. codebase by decomposing the complex end-to-end process into three distinct, orchestrated stages, as depicted in Figure
3.1 INTERACTIVE AND VERIFIABLE PLANNING
The initial stage of the Helmsman pipeline is dedicated to transforming high-level user objective into robust, executable research plan. This process comprises as a two-step verification loop, combining autonomous self-correction with human-in-the-loop oversight to ensure the plan is both sound and aligned with user intent.
Agentic Plan Generation and Self-Reflection. Upon receiving user query (formatted as per Section 4.1), a specialized Planning Agent is tasked with drafting an initial research plan. This agent instrumented with tools for both external web search and internal knowledge retrieval from a curated FL literature database (see Section 3.4). This ensures the proposed strategies grounded in established best practices and recent research.
However, to counter the risk of agent fallibility (e.g.. hallucination or flawed reasoning), we introduce a crucial cogaicco step: self-reflection. Before the plan is presented to the user, a Reflection Agent performs an automated critique. This agent systematically evaluates the draft against predefined set criteria, including logical coherence, completeness of experimental setup, and feasibility (detailed in Appendix A. 2.2. Then generates structured assessment, categorizing the plan as either COMPLETE or TNCOMP LETE with actionable feedback. This internal verification loop allows Helmsman to autonomously correct and refine its own output, significantly improving the quality of the plan before human intervention is required.
Hlmman-in-hhe-Love (HITL) Verification. The self-corrected plan undergoes a final HITL valida- tion. While our system is capable of full automation, we argue that for complex scientific discovery and system design, intentional HITL is critical component for ensuring reliability and control. This interactive step not merely "sanity check"; it serves three purposes: a) ensuring guaranteed alignment and safety by acting final safeguard against flawed or misaligned research trajectories, thereby mitigating costly downstream errors; b) enabling resource optimization, as user feedback on constraints and design choices effectively prunes the search space, reducing both LLM context costs and subsequent simulation expenses; and c) providing fine- fie-grained experimental control, which allows for the precise customization of parameters essential for scientific reproducibility and tailoring the solution to specific deployment constraints. Only after receiving explicit user approval does the system transition to the next stage, ensuring that the research plan is rigorously vetted, both autonomously and manually.
3.2 MODULAR CODE GENERATION VIA SUPERVISED AGENT TEAMS
With user-verified plan in hand, Helmsman translates the high-level strategy into executable code. This stage is managed by a central Supervisor Agent that orchestrates the development process based on the software engineering principle of separation of concerns.
4

---

Published as a conference paper at ICLR 2026
Table The structured natural language template for specifying tasks to Helmsman. This template guides the user to provide comprehensive and unambiguous problem definition, ensuring the Planning Agent receives the necessary context regarding the application domain, data characteristics, and desired FL objectives. The provided query example shows complete instantiation of this template.
Component
Description
Pattern Template
Problem Statement
Defines the high-level deployment context, including the application domain, system scale, and target in- frastructure.
"I need deploy fapplication type] on/across [number] [device rypes]..."
Task Description
Specifies the data characteristics, heterogeneity patterns, and any domain-specifie challenges.
"Each client holds [dataset characteris- tics] with [specific challenge].."
Framework Requirement
Outlines core FL objectives. model architecture to be trained, and the metrics for evaluation.
"Help me build a federated learning frame- work that [objectives], training a [model architecture). evaluating performance by [metrics]."
Query Example:
"I need to deploy a personalized handwriting recognition app across 15 mobile devices. Each client holds FEMNIST data from individual users with unique learning framework that balances global knowledge with local user adaptation for CNN model, evaluating performance average client test accuracy."
writing styles. Help me build personalized federated federaa
Blueprint Decomposition. To promote modularity and align with the canonical architecture of federated systems, the Supervisor first decomposes the plan into detailed blueprint. This blueprint creates a clear separation of duties, dividing the system into logical modules. This approach isolates the user-specific task itself from the mechanies of the federated process, allowing components be modified or replaced independently, tailored for the dynamic deployment environment. The resulting modules are:
Task Module: Contains the core data loaders, model architecture, and training utilities.
Client Module: Manages all client-side operations, including local training and evaluation.
Strategy Module: Implements the specific federated aggregation algorithm (e.g., FedAvg).
Server Module: Orchestrates the FL process, handling global model updates and evaluation.
Collaborative Implementation. The Supervisor then initiates a dependency-aweer aware workflow. spawns four dedicated teams, one for each module. Each team consists of Coder Agent for implementation and a Tester Agent for real-time verification and debugging. This inner loop of coding and testing ensures modular correctness. The Supervisor enforces dependency graph- for instance, work on the Server module only begins after the Strategy and Task modules are stable. Once all components are individually verified, the Supervisor integrates them into single, cohesive script, preparing it for the subsequent evaluation phase.
3.3 AUTONOMOUS EVALUATION AND REFINEMENT
While the modular coding stage ensures syntactic correctness, does not guarantee system-level robustness due potential integration failures. To certify the final codebase. Helmsman employs closed-loop process of autonomous evaluation, diagnosis, and refinement. Let the integrated codebase iteration be denoted as C, The refinement cycle proceeds as follows:
Sandboxed Federated Simulation. The codebase C; is first executed in sandboxed simulation environment for a small number of federated rounds (N = 5). This produces simulation log L, Simulate(C,, N). We use short run deliberate design choice; it is computationally inexpensive yet typically sufficient to expose critical runtime and integration errors, providing a strong signal for the verification phase.
5

---

Published as a conference paper at ICLR 2026
Hierarchical Diagnosis. Next, Evaluator Agent, fevol, analyzes the log L; using predefined set of heuristics H. To distinguish between catastrophic crashes and more subtle algorithmic failures. This verification is performed hierarchically to distinguish between critical crashes and subtle logical flaws. The agent first conducts a Runtime Integrity Veriftcation (I.J), scanning logs for explicit error signatures like Python exceptions stack traces. If the simulation completes without crashing, it then proceeds to deeper Semantic Correctness Verification (L2), analyzing structured outputs to identify algorithmic bugs such as stagnant training metries, zero client participation, divergent model behavior. This two-layer diagnostic process yields a status S; E {SUCCESS, FAIL) and detailed, context-aware error report E, that specifies the nature of the failure: (S,, E.) feear H).
Automated Code Correction. A FAIL status triggers the final step of the loop. A specialized Debugger Agent, fdebug. is invoked, taking the faulty codebase C, and the rich error report E, as input. The report E; is crucial, as provides the necessary context for targeted fix, whether it's correcting a simple API misuse (L1 failure) or adjusting the aggregation logic (1.2 failure). The agent then produces a patched codebase, C.41 fachng (Ci, E.).
Termination and Failure Handling. This cycle of simulation, diagnosis, and correction continues until a codebase, Cfinal, successfully passes both LI and L.2 verification layers. The result is a system that has been autonomously certified as not only executable but also semantically correct.
While this iterative repair process highly effective, is not guaranteed to converge for all problems. In particularly complex ill-posed tasks, the Debugger Agen! may fail to find a valid patch, leading to cycle of unsuccessful attempts. To ensure termination, we impose predefined threshold, Taarr on the number of correction attempts. If the iteration count exceeds this budget (i Tmox) without achieving a SUCCESS status, the process halts. This fail-safe mechanism flags the problem as requiring higher-level strategic intervention, such as rebooting the system for initial plan refinement or leveraging human expertise to resolve the specific impasse.
3.4 AGENT INSTRUMENTATION AND TOOLING
The advanced capabilities of the agents within Helmsman are not derived solely from the base LLM, but are significantly augmented by carefully curated suite of tools. These tools provide essential functionalities for knowledge retrieval and code validation, grounding the agents' reasoning and actions verifiable, external contexts.
Knowledge Access for Informed Planning. To ensure research plans are both state-of-the-art and technically sound, the Planning Agent leverages a dual-source knowledge system. A Web Search Tool (via Tavily API) provides access to the most current, real-world information, such as up-to-date library documentation and best practices, grounding the plan in practical realities. Complementing this, Retrieval-Augmented Generation (RAG) pipeline queries vector database of seminal FL literature from arXiv to retrieve state-of-the-arr techniques. This RAG process employs a multi-stage approach--nitiatinn with hybrid BM25 (Robertson al., 2009) and vector search for broad recall, followed by a Cohere rerank-v3.5 model (Cohere, 2024) with Voyage-3- embeddings (VoyageAI, 2025) to ensure high precisin--allowing allowing the agent to identify the most relevant algorithms for the user's specific problem.
Sandboxed Execution for Refinement. The autonomous evaluation and refinement loop (Section 3.3) is critically dependent reliable execution environment. For this, Helmsman utilizes the Flower framework (Beutel et al., 2020) as a sandboxed Simulation Tool. This environment is the and-reai cycle. provides the essential ground-truth feedback by executing
core of the the generated code and capturing the resulting logs, enabling the Evaluator and Debugger Agents to analyze system behavior and perform targeted corrections.
EXPERIMENTAL SETUP
4.1 BENCHMARK DESIGN
Evaluating generative agents like Helmsman demands a paradigm shift from traditional software and ML benchmarks. Code generation benchmarks such HumanEval (Chen et al., 2021) assess an agent's 's ability to solve self-contained, algorithmic problems. Standard FL benchmarks, conversely, evaluate pre-defined model's performance static dataset. Neither sufficient for measuring an

---

Published as a conference paper at ICLR 2026
Table Performance evaluation on heterogeneous federated learning benchmarks (a), We compare our agent-synthesized strategy against task-specific baselines. Results are averaged over independent runs with standard deviation. The best performing method marked in bold, while the second-best underlined. Results for specialized methods are denoted by symbols: FedNova" (Wang et al., 2020), FedNSI (Li al., 2024), and HeteroFL (Diao et al., 2020).
ID
Task
Dataset
Problem
FedAvg
FedProx
Specialized
Ours
Data Heterogeneity
Q1
Object Recognition
CIFAR-IO-LT
Quantity Skew
+0.08
69.65+0.42
78.961..41
76.71 20.43
Q2 Object Recognition CIFAR-I00-C
Feature Skew
33.96 1.668
35.13
42.79
39.76 10.30
Q3 Object Recognition
CIFAR-ION
Label Noise
+3.95
78. 78.7840.40
50.55
81.6220..6
Distribution Shift
Q4
Object Recognition
Office-Hlmme
Domain Shift
53. 53.0211.03
50.6710.39
57.261...0
54.49 10.97
Q5 Human Activity
НAR
User Heterugeneity
44.84
95. 95.22+0.32
95. 95.192...
96.28
Q6 Speech Recognition Speech Commands
Speaker Variation
84.44 10.35
84.1940.29
83.4840.49
86.58
Q7 Medical Diagnosis
Fed-ISIC2019
Heterogeneity
57.00 +1.44
61.11
62.88
63.75
Q8
Object Recognition
Caltech101
Class Imbalance
47 40.67
47.20
63.771027
50.88 1.:22
System Heterogeneity
Q9 Object Recognition CIFAR-100
Resource Constraint 59 59.96+0.93
59.43+0.68
62.62
62.94 00.63
agent's 's capacity for end-to-end research automation the ability to take a high-level scientific goal and autonomously synthesize complete, functional, and performant system.
To address this gap. we introduce AgentFL- Bench, a benchmark designed to rigorously eval- uate the capabilities of agentic systems in fed- erated learning. To ensure thorough test of agent versatility, it comprises 16 unique tasks curated span five pivotal FL research domains: data heterogeneity, communication efficiency, personalization, active learning, and continual learning (Figure 3). Furthermore, these tasks are designed for realism, reflecting the multifaceted nature of real-world FI. challenges where issues like non-IID data and system constraints often co-occur (see Figure 2). This requires the agent to reason about and integrate appropriate strate- gies, rather than solving isolated toy problems. Finally, to enable consistent and reproducible evaluation, every task is defined by a standard- ized natural language query (Table 1). This provides an unambiguous problem specification that minimizes prompt-engineeriig variability and allows for fair, apples-to-apples comparisons between different agentic systems. The complete set of task queries, along with their specific experimental configurations and established baseline implementations, are detailed in Appendix A.4 and A.5.
Federated Continual Learning
Active Learning (2)
12.5%
12.5%
Personalized FL
56.2%
Meterogeneous FL
12.5%
Communicaioonefficdennt FL
Figure 3: Distribution of the 16 tasks in our AgentFL-Bench benchmark across five key FL research domains.
4.2 IMPLEMENTATION DETAILS
Our agentic system, Helmsman, is constructed upon the LangGraph (LangChainAI, 2025) frame- work, leveraging LangChain (LangChainAA, 2024) for tool integration. The system's 's L.MM backbone consists of Google's Gel (Comanici et al., 2025) for the planning stage and Claude- Sonnet-4.0 (Anthropic, 2025) for the coding and evaluation stages. We also conduct additional experiments with different LLM choices in the Appendix (Claude-Sonnet-4; and GPT-5.1). The maximum debugging attempts during the evaluation stage is set to 10 times. We evaluate Helms- on AgentFL-Bench benchmark, which covers cross-silo (i.e., 5 clients) and cross- (i.e., 10 clients) scenarios. We report the extensive details of datasets, models, and taec-spec configurations for each research query in Tables 16 and 17.
7

---

Published as a conference paper at ICLR 2026
Table Performance evaluation on heterogeneous federated learning benchmarks (b). We compare our agent-synthesized strategy against task-specifie baselines. Results are averaged over independent runs with standard deviation. Results for specialized methods are denoted by symbols: FedNova" (Wang et al., 2020), FedPer (Arivazhagan et al., 2019), and FedWelt (Yoon et al., 2021).
ID Task
Dataset
Challenge
FedAvg
FedProx
Specialized
Ours
Communication-ESiicei Eificiency
QIO
Object Recognition
CIFAR-I00
Bandwidth Limits
41.77
25.21
45.77
48.78
Q11
Handwriting Recog.
FEMNIST
Connectivity Limits
87. 87.4610.43
87.95
89.111..20
89.73
Personalization
Q12
Handwriting Recog.
FEMNIST
Local Adaptation
74.12 74.12
75.47
77.43
76.41.0.0
Q13
Object Recognition
CIFAR-10
Distribution Skew
72.5922.1
79.76g0.27
81.56
29.22
To ensure consistency, all queries run for 100 communication rounds, with each client performing local updates per round. We adopt the 80/20 data split across all FL tasks, where the training data is partitioned and distributed to each client based on the specific research task requirement. To evaluate the efficacy of our system's 's strategy module, we perform comparative analysis against several FL. baselines. This achieved by substituting our generated strategy with baseline method while maintaining identical task configurations. Our comparison includes foundational baselines, such as FedAvg (McMahan et al., 2017) and FedProx (Li et al., 2020a), alongside specialized methods designed for specific challenges: FedNova (Wang et al., 2020) for distribution shifts, FedNS (Li et al., 2024) for noisy data, HeteroFL (Diao et al., 2020) for heterogeneous models, FedPer (Arivazhagan et al., 2019) for personalization, FAST Li et al. (2025) for active learning, and FedWelT (Yoon et al., 2021) for continual learning.
Table 4: Performance evaluation on heterogeneous federated learning benchmarks (c). We compare agent-synthesized strategy against task-specific baselines. Results are averaged over independent runs with standard deviation. Results for specific federated learning methods are denoted by symbols: FedWelt (Yoon et al., 2021), FAST (Li et al., 2025).
ID
Task
Dataset
Challenge
FedAvg
FedProx
Specialized
Ours
Active Learning
Q14
Medical Diagnosis
DermaMNIST
Sample Selection
71.13
71. 71.431022
74.71 #
$0.41
72.87 72.87
QIS
Object Recognition
CIFAR-10
Distribution Skew
63. 13..5211.00 oo0
66. 66.0220.25
77.95 10.65
68 28.21
Continual Learning
Q16
Object Recognition SplitCCIFARI00
Incremental Tasks 15.38
15.86
29.45 72
50.95
RESULT
This section presents a comprehensive evaluation of Helmsman on the AgentFL-Bench benchmark, designed to systematically assess the generative system's capability devise effective solutions for diverse challenges in FL.
Our initial investigation addresses the multifaceted issue of heterogeneity. As categorized in Table we evaluate performance on tasks related to data heterogeneity (Q1-Q3), distribution shifts (Q4Q88, and system heterogeneity. The initial tasks (Q1-Q3) explore challenges data quality, spanning quantity imbalances and feature/label noise. In the subsequent tasks addressing distribution shifts (Q4-Q8), solutions generated by Helmsman consistently outperform a majority of the baseline methods. Analysis of the generated code reveals that Helmsman often synthesizes hybrid strate- gies, combining multiple techniques to achieve robust performance against the specified challenge. We then escalate task complexity by introducing compound challenges (Q10-Q13). For instance, tasks Q10-Q11 require optimizing for communication efficiency under non-IID data distributions. Likewise, tasks Q12-Q11 address federated personalization, demanding solution that facilitates both personalized model tuning and effective global knowledge aggregation. The results in Table demonstrate that the solutions from Helmsman highly competitive with existing methods.
8

---

Published as a conference paper at ICLR 2026
Finally, evaluation extends to interdisciplinary research domains by incorporating tasks from Federated Active Learning (FAL) and Federated Continual Learning (FCL) (Q14-Q16). These scenarios probe the system's capacity for addressing nuanced, crossddiccpplnarry problems, such as decentralized data selection in FAL or mitigating catastrophic forgetting in FCL. As shown in Table 4, Helmsman continues to generate effective solutions despite the inherent diffieulty. For Q14 and Q15, the generated strategies surpass standard baselines, positioning them as viable starting points for novel research. Notably, for task Q16, the solution synthesized by Helmsman substantially outperforms even highly specialized method, a finding further analyzed in Section 6.
DISUCCUSION
Discovering Novel Algorithmic Combinations. Helmsman advances complex problem-solving by strategically integrating human intelligence if available. While Helmsman can achieve 100% full automation (see Appendix A.I), complex queries such as Q16 can benefit from this human expertise via collaborative plan refinement to resolve ambiguity. As illustrated in Appendix A.3, our agile automated simulations can confront malicious or unaligned input queries.
In the extensive evaluation on Q16 (Ta- ble 5), the solution synthesized by Helms- man surpassed all competing continual learning methods and exhibited markedly reduced catastrophic forgetting. This im- provement stems from a novel combinato- rial strategy that integrates client-side ex- perience replay with global model distil- lation. By enhancing knowledge transfer across sequential tasks, this hybrid mecha- nism enables Helmsman to produce robust, deployment-ready solutions that overcome the limitations of conventional research- oriented approaches.
Table 5: Performance on Split-CIFAR100 (a = 0.5). We evaluate methods under 5-task and 10-task scenarios, reporting average accuracy (Acc f) and forgetting (F 4).
Best results are in bold.
Task 5
Task = 10
Method
Acc (5% f J. J Acc (%)
F J
FodAvg (McMahan et al., 2017)
16.38
0.75
7.83
0.76
FedProx (Li et al., 2020a)
16.71
0.76
8.19
0.75
FedEWC (Loe
2017)
16.06
0.68
10.07
0.62
FedWeIT (Yoon et al., 2021)
28.56
0.49
20.48
0.45
FodLwF (Li & Hoiem, 2017)
30,94
0.42
21.74
0.46
TARGET (Zhang et al,, 2023)
34.89
0.24
25.65
0.27
Helmsman (Ours)
51.04
0.07
47.53
0.18
Evaluation on Helmsman System Components. To evaluate the contribution of each system component in Helmsman, we conduct ablation study. As discussed in Section 3, Helmsman consists of three critical components. For a comprehensive comparison, we consider six configurations.
We select seven representative query tasks from each research domain in EgenthL--Bench. As shown in Table 6, the single ReAct agent fails on all tasks. Although Configurations 2 and 3 solve a subset of queries, they fail most, indicating that all system components required for Helmsman to be fully functional. In contrast, the full Helmsman system achieves a 100% success rate. Notably, while removing the luye- verification reduces average API cost, it causes failures on all tasks, underscoring the essential role of verification in ensuring system robustness and stability.
Table 6: Ablation study of Helmsman system component contributions on AgentFL-bened tasks (Saude-Sonnet-455) across six settings. Components denote: ① Planning Group with Supervision; 2 Collaborative Modular Coding Group; 3 Sandboxed Simulation with Dual-Layer Verification. The fifth setting denotes the full Helmsman System without having numan-in-the-loow (HITL).
X
Q1
Q4
Q12
Q14
Q16
Success Rate
Avg.
Single ReAct Agent
x
Fail
fuil
Fail
full
fail
0%
1.75
Single ReAct (wl Dual Verisccation
x
Gill
fail
5000655
ful
fail
full
fail
14.29%
1.28
Helmsmas (wlo Collab. Coding)
x
fail
fall
fail
SU0c05S
SUCCeSs
fail
fail
28.57%
2.11
Helmsman (wlo Dual Verification)
/
fail
Helmsman (Full System wio HITL)
V
100%
1.14
Helmsman (Full System)
/
success
success
SUccess
SUCCESS
SUCCess
success
sucuess
100%
0.98
The stability of Helmsman with different types of input schema. In Table 1, we introduce the structured research query pattern used in AgentFL-Benen We identify three prerequisite components for constructing a complete FL query schema: the problem statement, task description, and framework requirements. This consistency in input structure ensures fairness in our experiments and supports a reliable evaluation of Helmsman's performance in FL system generation. However, in real-world

---

Published as a conference paper at ICLR 2026
setting, the user input query schema can be misaligned with the tasks in AgentFL-Bench. Therefore, introduce the human- human-in-the- in-the-loop (HITP) in Helmsman to improve the system' 's robustness.
To justify this, conducted an additional ablation study evaluating Helmsman's behavior under different structured imput schemas (c.g., paraphrased, incomplete, and out-of-schema queries), During the planning stage, the planning team, which comprises the planner and self-reflection agents, collaborates to mitigate the risk of erroneous or unstable research plans. As show in the below Table 7 for incomplete input schemas, where essential information is missing for downstream synthesis, the planning team provides actionable feedback to the user to request the necessary details, as shown in the tables below. For paraphrased queries, Helmsman remains capable of producing valid research plans as long as the core information (as present in Table 1) is preserved. For out -ftte---shema inputs, Helmsman routes the query back the user for a "sanity check," thereby safeguarding the system from flawed assumptions or misaligned research trajectories. In Appendix A.3, we further show the complete interaction of Helmsman- -uman when confronted with different types of misaligned input queries during the planning workflow, using research query the example.
Table 7: Helmsman's 's Response to Different Input Schemas for Query (CIFAR-IO-LT FL Task)
Input Type
User Query
Helmsman Workflow
Original
need to deploy photo-sorting app 15 smartphones. Fach phone stores highly imbalanced slice of long-tail distributed data (CIFAR-IO-LI) Help me build federated learning framework train MobileNet-VI model, evaluating perfor- mance by top-1 test accuracy.
Plan Construction Haman Verification
Transition to Coding Group
Paraphrased
I want train MobileNet model 15 mobile devices that have unbalanced CIFAR-IO-LE data. Can you help up federated system and measure the curacy?
Plan Construction
Human Verification
Transition to Coding Group
Incomplete
need federated learning for 15 phones with imbalanced data. Model Mo- bileNet.
Detect Missing Infe Request User Input Receive Input +
Plan Construction on
Human Verification
Transition to Coding Group
Out-of- Schema
Hello w
Detect Anoma ly
Manual Sanity Check +
User Correct/on
Plan Construction
Human Veritication
Transition to Coding Group
7 CONCLUSION
This work introduces Helmsman, agentic system that marks significant step towards automat- ing the end-to-end design and implementation of FL systems. Our evaluations on the AgentFL- Bench benchmark demonstrate that coordinated, multi-agent collaboration can successfully navigate the complex FL design space, yielding robust and high-performance solutions for decentral- ized environments. Future work will aim to endow Helmsman with self-evolutionary capabilities, creating a meta-optimization loop where the system learns from experimental feedback to refine both the generated code and its internal strategies. This path will advance the development of more autonomous, AI-driven tools for engineering complex FL systems.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the NGF AiNed Fellowship Grant of A.S. We also acknowledge the use of the Dutch National Supercomputer Snellius for essential computational tasks.
REFERENCES
Anthropic. Claude opus 4 & claude sonnet 4: System card. Technical report, Anthropic, May 2025. URL https: /wwww ant anthropic. com/ cwwcde-aeeeeeeeee saassemmdcrd... System card; updated July 16, 2025; revised Sep 2, 2025.
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Feder- ated learning with personalization layers. arXiv preprint
2019.
Daniel J Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Javier Fernandez-Marques, Yan Gao, Lorenzo Sani, Kwing Hei Li, Titouan Parcollet, Pedro Porto Buarque de Gusmão, et al. Flower: A friendly federated learning research framework. arXiv preprint arXiv:2007.14390, 2020.
Dong Chen, Shaoxin Lin, Muhan Zeng, Daoguang Zan, Jian-Gang Wang, Anton Cheshkov, Jun Sun, Hao Yu, Guoliang Dong, Artem Aliev, et Coder: Issue resolving with multi-agent and task graphs. arXiv preprint arXiv:2406.011 304, 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint wrXiv22107003374 2021.
Yujing Chen, Yue Ning, Martin Slawski, and Huzefa Rangwala. Asynchronous online federated learning for edge devices with non-iid data. In 2020 IEEE International Conference on Big Data (Big Data), pp. 15-24. IEEE, 2020.
Cohere. Introducing rerank 3.5: Precise ai search, December 2024. URL https://tcoheeeeeeeeeeeeeeeeeeee com/blog/ree htagg/rerank-3330 3pt
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.066 .06261, 2025.
Enmao Diao, Jie Ding. and Vahid Tarokh. Heterofl: Computation and communication efficient federated learning for heterogeneous clients. arXiv preprint arXiv:2010.01264, 2020.
Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. Self-collaboration code generation via chatgpt. ACM Transactions on Software Engincering and Methodology, 33(7):1-38, 2024.
Juraj Gottweis, Wei- Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, al. Towards an co-scientist. arXiv preprint arXiv:2502. 8864, 2025.
Dong Huang, Jie Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming Cui. Agent- coder: Multi-agent-basee code generation with iterative testing and optimisation. arXiv preprint arXiv:2312. 13010, 2023.
Md Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. Mapcoder: Multi-agent code generation for competitive problem solving. arXiv preprint arXiv:2405. 11403, 2024.
Md Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. Codesim: Multi-agent code generation and problem solving through simulation-driven planning and debugging. arXiv preprint arXiv:2502.05664 2025.
Divyansh Jhunjhunwala. Pranay Sharma, Aushim Nagarkatti, and Gauri Joshi. Fedvarp: Tackling the variance due to partial client participation in federated learning. In Uncertainty in Artificial Intelligence, pp. 906-916. PMLR, 2022.
11

---

Published as a conference paper at ICLR 2026
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? The Twelfth International Conference Learning Representations, 2024. URL https:/ net net/forumms d=VTF ByNQM66.
Sai Praneeth Karimireddy, Saryen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning, pp. 5132-5143. PMLR, 2020.
Minjae Kim, Sangyoon Yu, Suhyun Kim, and Soo-Mook Moon. Depthfl: Depthwise feder-
ated learning for heterogeneous clients. In The Eleventh International Conference on Learning Representations, 2023.
LangChainAL. The platform for reliable agents, September 2024.
URL https: /www.
langchain. com/.
LangChainAL. Balance agent control with agency, August 2025. langchain.com com.magggraph..
URL https: //wpp::/www... /www.
Sang-Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung- Woo Ha, and Byoung-Tak Zhang. Overcom- ing catastrophic forgetting by incremental moment matching. Advances in neural information processing systems, 30, 2017.
Haoyuan Li, Mathias Funk, Nezihe Merve Gürel, and Aagib Saeed. Collaboratively learning federated models from noisy decentralized data. In 2024 IEEE International Conference on Big Data (BigData), pp. 7879-7888. IEEE, 2024.
Haoyuan Li, Mathias Funk, Jindong Wang, and Aaqib Saeed. Fast: Federated active learning with foundation models for communication-efficenn sampling and training. IEEE Internet of Things Journal, 2025.
Qinbin Li, Bingsheng He, and Dawn Song. Monel-contrastive federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10713-10722, 2021.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization heterogeneous networks. Proceedings of Machine learning and systems, 2:429-450. 2020a.
Xiaoxiao Li, Yufeng Gu, Nicha Dvorek, Lawrence H Staib, Pamela Ventola, and James S Duncan. Multi-site fmri analysis using privacy- preservin federated learning and domain adaptation: Abide results. Medical image analysis, 65:101765, 2020b.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions pattern analysis and machine intelligence, 40(12):2935-2947 2017.
Akis Linardos, Kaisar Kushibar, Sean Walsh, Polyxeni Gkontra, and Karim Lekadir. Federated learn ing for multi-center imaging diagnosties: simulation study in cardiovascular disease. Scientific Reports, 12(1):3551, 2022.
Ji Liu, Juncheng Jia, Tianshi Che, Chao Huo, Jiaxiang Ren, Yang Zhou, Huaiyu Dai, and Dejing Dou. Fedasmu: Efficient asynchronous federated learning with dynamic staleness-aware model update. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 13900-13908, 2024a.
Su Liu, Jiong Yu, Xiaoheng Deng, and Shaohua Wan. Fedepf: An efficient-commuiictioon federated learning approach for vehicular edge computing 6g communication networks. IEEE Transactions on Intelligent Transportation Systems, 2362):116-1169, 2021a.
Yang Liu, Tao Fan, Tianjian Chen, Qian Xu, and Qiang Yang. Fate: An industrial grade platform for collaborative learning with data protection. Journal of Machine Learning Research, 22(226):1-66 2021b.
Yizhou Liu, Pengfei Gao, Xinchen Wang, Jie Liu, Yexuan Shi, Zhao Zhang, and Chao Peng. Marscode agent: Ai-native automated bug fixing. arXiv preprint arXiv:2409.003: 008999 2024b.
12

---

Published as a conference paper at ICLR 2026
Yuxiang Lu, Suizhi Huang, Yuwen Yang, Shalayiding Sirejiding, Yue Ding, and Hongtao Lu. Fedhca2: Towards hetero-client federated multi-task learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5599-5609, 2024.
WANG Luping, WANG Wei, and LI Bo. Cmfl: Mitigating communication overhead for federated learning. In 2019 IEEE 39th international conference on distributed computing systems (ICDCS), pp. 954-964. IEEE, 2019.
Othmane Marfoq, Giovanni Neglia, Aurélien Bellet, Laetitia Kameni, and Richard Vidal. Federated multi-task learning under a mixture of distributions. Advances in Neural Information Processing Systems, 34: 14:15434-1 2021.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Ar- cas. Communication--fficienn learning of deep networks from decentralized data. Artificial intelligence and statistics, pp. 1273-1282. PMLR, 2017.
Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023.
Vladimir Naumov, Diana Zagirova, Sha Lin, Yupeng Xie, Wenhao Gou, Anatoly Urban, Nina Tikhonova, Khadija Alawi, Mike Durymanov, Fedor Galkin, et al. Dora scientist: Multi-agent virtual research team for scientific exploration discovery and automated report generation. bioRxiv, 2025.
Bo Ni and Markus Buehler. Mechagents: Large language model multi- collaborations can solve mechanics problems, generate new data, and integrate knowledge. Extreme Mechanics Letters, 67:10213, 2024.
Alexander Novikov, Ngån Vū, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang. Adam Zsolt Wag- ner, Sergey Shirobokov, Borislav Kozlovskii, Francisco JR Ruiz, Abbas Mehrabian, et al. Alphae- volve: A coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506. 13131, 2025.
Ana Nunez, Nafis Tanveer Islam, Sumit Kumar Jha, and Peyman Najafirad. Autosafecoder: A multi-agent framework for securing IIm code generation through static analysis and fuzz testing. arXiv preprint #tXiv:2409.10737 2024.
Felix Ocker, Stefan Menzel, Ahmed Sadik, and Thiago Rios. From idea to cad: language model- driven multi-agent system for collaborative design. arXiv preprint arXiv:2503.0441 2025.
Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumnr, and H Brendan McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003.00295, 2020.
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends@ in Information Retrieval, 3(4):333-33) 389, 2009.
Holger R Roth, Yan Cheng, Yuhong Wen, Isaac Yang, Ziyue Xu, Yuan-Ting Hsieh, Kristopher Kersten, Ahmed Harouni, Can Zhao, Kevin Lu, et al. Nvidia flare: Federated learning from simulation to realdworld. arXiv preprint arXiv:2210.13291 2022.
Zhiri Tang, Hau-San Wong, and Zekuan Yu. Privacy-preserving federated learning with domain adaptation for multi disease ocular disease recognition. IEEE Journal of Biomedical and Health 2023.
Informatics, 1
Leitian Tao, Xiang Chen. Tong Yu, Tung Mai, Ryan Rossi, Yixuan Li, and Saayan Mitra. Codelutra: Boosting Ilm code generation via preference-guie -uided refinement. arXiv preprint arXiv:2411.05199 2024a.
Wei Tao, Yucheng Zhou, Yanlin Wang, Wenqiang Zhang. Hongyu Zhang, and Yu Cheng. Magis: Lim-based multi-agent framework for github issue resolution. Advances in Neural Information Processing Systems, 37:51963-51993, 2024b.
13

---

Published as a conference paper at ICLR 2026
VoyageAI. 3aygge the new state-of-the-art general-purpose embedding model, January 2025.
URL pestps://blog. voyageai, com// com/ 201//02225/// wttp:///////////////////////////
3-large/.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:761 1-7623, 2020.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thouggh prompting elicits reasoning large language models. Advances in neural information processing systems, 35:24824-248377 2022.
Chuhan Wu, Fangzhao Wu, Lingjuan Lyu, Yongfeng Huang, and Xing Xie. Communication--fficinnn federated learning via knowledge distillation. Nature communications, 13(1):2032, 2022.
Wentai Wu, Ligang He, Weiwei Lin, Rui Mao, Carsten Maple, and Stephen Jarvis. Safa: A semi- asynchronous protocol for fast federated learning with low overhead. IEEE Transactions on Computers, 70(5):655-668 2020.
Jing Xu, Sen Wang, Liwei Wang, and Andrew Chi-Chih Yao. Fedem: Federated learning with client-level momentum. arXiv preprint arXiv:2106. 10874, 2021.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
React: Synergizing reasoning and acting in language models. In International Conference Learning Representations (ICLR), 2023.
Jaehong Yoon, Wonyong Jeong, Giwoong Lee, Eunho Yang, and Sung Ju Hwang. Federated continual learning with weighted inter-client transfer. International conference on machine learning, pp- 22076-12086. PMLR, 2021.
Jie Zhang, Chen Chen, Weiming Zhuang, and Lingjuan Lyu. Target: Federated class-continual learning via exemplar-free distillation. Proceedings the IEEE/CVF International Conference Computer Vision, pp. 4782 2023.
Yaolun Zhang, Yinxu Pan, Yudong Wang, and Jie Cai. Pybench: Evaluating Ilm agent various real-world coding tasks. arXiv preprint arXiv:2407.16732 2024.
Alexander Ziller, Andrew Trask, Antonio Lopardo, Benjamin Szymkow, Bobby Wagner, Emma Bluemke, Jean-Mickael Nounahon, Jonathan Passerat- Palmbact, Kritika Prakash, Nick Rose, et al. Pysyft: A library for easy federated learning. Federated learning systems: Towards next gexeraener AI, pp. 111-139, 2021.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 EVALUATION AGAINST STATE-OF-THE-ART CODE SYNTHESIS METHODS
To further validate Helmsman's effectiveness, we compare it against two state-of-the-art code- synthesis pipelines: Codex (GPT--51-codee and Claude Code (lauue--Somee-4.5) We conduct additional experiments across all tasks in AgentFL-Bench using these baselines. As shown in Tables achieves a 100% success rate across all
8-12, Helmsman (with either GPT-5.1 or research queries, substantially outperforming Codex (37.5%) and Claude Code (43.75%).
Furthermore, Helmsman exhibits significantly lower API costs, benefiting from efficient agentic coor- dination throughout the research pipeline and thus surpassing existing antemauuomation approaches in cost efficiency. An additional observation is that GPT-5.1 serves as the most economical LLM backend for Helmsman, owing to its lower token consumption.
Table 8: Comprehensive performance comparison across federated learning tasks (Part a: Q1-Q3). For each task, we report Cost ($, lower the better), Total Tokens (in thousands, lower the better), Walltime (seconds, lower the better), and Outcome, Best results per metric are in bold.
Q1
Q2
Q3
Method
Cost ($)
Token
Walltime (s)
Outeome
(S)
Token
Walltime (6)
Ouscome
Cost (S)
Token
Walhime (s)
Outcome
Codex 5.I Codes)
2,639K
s0ccess
4,338
2.683
success
Chadd Code (Clande Sonnet 4.5)
1,16
657k
634
fail
1.08
1,329k
16617
1,02
969k
1,254
Helmsman (C Clande Sonnet 4)))
1.64
287k
841
M0000ss
2.00
372k
1.057
suacess
1.88
343
773
Helmsman (C Clande Sonnet 5)
23.K
1,274
13%
1,154
164k
Blelmsmun 5.I)
213k
success
11lk
SUacess
190k
success
Comprehensive performance comparison across federated learning tasks (Part b: Q4-Q6). For each task, we report Cost ($, lower the better), Total Tokens (in thousands, lower the better), Walltime (seconds, lower the better), and Outcome. Best results per metric are in bold.
Table 9:
Q5
Method
Cost
Token
Walltime
Outeome
Cost
Token
Walltime
Ouscome
(S)
Token
Walhime
Outcome
Cndes (GP Codes)
1.02
2,777k
743
fail
0.73
1,429k
618
soccess
1.43
3,939
1,322
fail
Chando Code (Clande Sonnet 4.5)
1.69
2.245k
1,179
fuil
2:38
3.165k
928
s00cess
1.31
1,634k
1,436
(Clunde Sonnet
253k
41.3
865
Hetmsman (Cluude Sonnet
0.68
134k
0.39
146k
469
1.65
200k
948
Jelmsman (GPT-5.1)
0.72
199k
851
success
0.56
187%
592
s0ocess
0.80
295k
754
success
Comprehensive performance comparison across federated learning tasks (Part e: Q7-Q9). For each task, we report Cost ($, lower the better), Total Tokens (in thousands, lower the better), Walltime (seconds, lower the better), and Outcome. Best results per metric are in bold.
Table 10:
Q7
Q8
Q9
Method
Cost ($)
Token Walltime (s)
Outcome
Cost
Token
Walltime (s)
Ouscome
Cost (S)
Token
Waltiim (s)
Ouicome
Codes (.I....1 Codes)
1,74kk
3,227%
1,405
success
Chado Code (Clande Sonnet 4.5)
1.85
1.844k
1,341
fail
1.37
3,273k
1.722
Giil
2.61
3,002k
1,435
Helmsman (Claude Sonnet 4.0)
1.52
278k
923
success
1.13
207k
605
50ccess
2.75
47%k
781
success
Helmsman (Claude Sonnet 4.5)
0.94
362k
823
supcess
1.99
256k
774
soccess
1.20
20%
1,084
success
Belmsman (.11
324k
s00005s
138k
s0ccess
150k
success
A.2 EVALUATION ON REPRODUCIBILITY OF HELMSMAN FRAMEWORK
We further conduct an ablation study to assess the stability and reproducibility of Helmsman across three independent runs. We compare the final FL system structures generated by Helmsman with those produced by state-of-the-an art code onde-generation baseline (Claude Code). As shown in Figure 4,
'The configuration "Helmsman (Claude-Somnet-440) refers to earlier experiments conducted prior to the release of Sonnet 4.5.
15

---

Published as a conference paper at ICLR 2026
Table 11: Comprehensive performance comparison across federated learning tasks (Part d: Q10-Q13). For each task, we report Cost ($, lower the better), Total Tokens (in thousands, lower the better), Walltime (seconds, lower the better), and Outcome, Best results per metric are in bold.
QL3
Melaal
CorrS)
Tokos
Willinne bi
Ostonne (Ooc(S)
Tikcn
Willinn by
Oruceed
Coot
Tokes
Willinac of
Outorecc
Tike
Welline of Orkcenc
Ootes (OPP)5 (-Colc))
6.7
1.30%.
726
0.99
24534
942
0.62
1,38%
994
0.70
1.4%
663
(Cladd
1.57
1.71
254
2314k
1.43
638
244
Betrozza (Claad: S.0nd.400 4.0
24/6
319
660000
1.94
3576
1154
1006
578
1.666
156
Hiloozza (Caakk: 4.5)
0.90
128A
792
0.86
198k
627
PANCc
1.06
1966
978
M6000
127
2041
714
Beteozza (PPSS.D
063
1608
313
0.48
120k
988
0.57
166k
742
9.48
1426
5M4
Table 12: Comprehensive performance comparison across federated learning tasks (Part e: Q14-Q16 and Overall Summary). For each task, we report Cost (8, lower the better), Total Tokens (in thousands, lower the better), Walltime (seconds, lower the better), and Outcome. The Average column shows mean performance across all 16 tasks with Success Rate. Best results per metric are in bold.
Mitholl
Trkm
Dokoees
Tikea
Wothire to
Oktioeee
Tikre
Wallow
Garoaee
Tiben
Williwed
Sac Bran
Codox (CFFS Codes)
8.5
1.5k
712
E41
044
1.7%
948
Est
1,00
48374
1.207
0,33
24008
909
37.50
Ckaslo CakerClaakk 4.))
|
1G5
1/226
1229
1.00
2.44
1,445
Fel
3114
23234
1.555
1.30
22344
1.218
42:55
Beltorainn (Chande Spooer
10)
1,79
361
3.72
677c
1.388
1866
637
01:35
Beborman (Chak Sienci483
138k
166
083
600
14%
1086
154
39)
T0
1761
Helmsman consistently produces identical FL system structures across all runs, whereas Claude Code generates divergent system designs. This demonstrates the stability of Helmsman's 's modularized architecture. Moreover, the modular design enables the Helmsman-genenn generated FL systems to be easily adapted to alternative FL strategies by simply replacing the strategy module.
During the coding stage, a supervisor agent functions as the research lead, decomposing high-level research plans into implementable modules for downstream coding groups As illustrated in Fig. 1, these coding groups operate in an interdependent manner, reflecting the intrinsic coupling between components in FL systems (e.g.. client and server modules must coordinate to satisfy protocol requirements). The orchestration stage then ensures global logical consistency across the entire system. After verification by the orchestrator, the synthesized FL system proceeds to the evaluation stage, where a dual-layer sandboxed simulation is executed to validate system functionality and readiness for deployment.
project
project
project
project
README.md
configs
README.md
results
requirements.txt
data
USAGE GUIDE.md
task.py
config.py
models
requirements.uk
strategy-py
data_utils.py
strategies
data Joader,pp
client client_app-py app-py
fl cliient.py
utils
evaluation.py
pever_app.py
fl_ server.py
README.md
config.py
run.py
requirements.te
model.py
model.py
federated trainer.py
utils.py
visualize.py
utils.py
train_ federateddy
main.py
deploy.py
(a) Claude Code (Run 1) (b) Claude Code (Run
(c) Claude Code (Run 3) (d) Helmsman (All Runs)
Figure 4: Code generation stability comparison on Q1 task across independent runs. Claude Code Claude-Sonnet-44. produces distinct folder structures and implementations in each run (a-c), demonstrating inconsistent code generation. In contrast, Helmsman maintains an identical system structure across all runs (d), ensuring reproducibility and enabling plug-and-play modularity.
16

---

Published as a conference paper at ICLR 2026
A.3 ABLATION STUDY PLANNING STABILITY IN HELMSMAN
In the following tables, we show the complete interaction of Helmsman-Huuma when confronted with different types of misaligned input queries during the planning workflow, using research query 1 as the example. Specifically, Figure for original input query. Figure 6 for query type of out-of-schema, Figure for query type incomplete schema, and Figure for query type paraphrased schema.
A.3.1 HUMAN-HELMSMAN CONVERSATION WITH ORIGINAL INPUT
HELMSMAN
System
Starting Planning Workflow
HELMSMAN
Input Required
Enter your federated learning research question:
USER
need to deploy a photo-sorting app on 15 smartphones. Each phone stores a highly imbalanced slice of long-tail distributed data (CIFAR-IO-LT) Help me build federated learning framework to train MobileNet-V model, evaluating performance by top-1 test accuracy.
HELMSMAN
Agent: Planning
Planning Agent Team: Thinking and researching...
Calling tools: web_search, docs_search
Web Searen Results: (' ququr'' "..', "answer': "...', "images'
[ [...], 'reuulss'
search Results: {...}
HELMSMAN
Agent: Reflection
Initial Plan:
Self-reflection:
The plan comprehensive actionable for research project. clearly defines objective, identifies the specific challenges (Non-IID, long-tail data), proposes a concrete methodology. does not require further information from the user to proceed.
Plan is complete and sound, routing to human for final verification...
HELMSMAN
Approval Required
Approve this plan? (yes/ /no)
USER
Yes
HELMSMAN
Complete
Workflow Complete!
Figure 5: The complete interaction of Helmsman Helmsman-Human during planning workflow research
query task 1, The input query type follows the original schema in AgentFL-Bench.
17

---

Published as a conference paper at ICLR 2026
A.3.2 HUMAN-HELMSMAN CONVERSATION WITH -C-HEMAS INPUT
HELMSMAN
System
Starting Planning Workflow
HELMSMAN
Input Required
Enter your federated learning research question:
USER
Hello @ #$#$%
HELMSMAN
Agent: Planning
Planning Agent:
Thinking and researching...
Self-reflction::
INCOMPLETE: The agent is explicilly asking for more information from the user and has not yet generated any part of the research plan (summary, challenges, tasks, or technical setup).
Plan incomplete, routing to human feedback...
HELMSMAN
Refinement Required
Agent Analysis:
PLAN INCOMPLETE. Missing Information:
The agent is explicitly asking for information from the and has yet generated any part of the research plan (sunmmary, challenges, tasks, or technical setup).
CURRENT AVAILABLE INFORMATION:
No information retrieved yet
Please provide additional information help complete FT. research plan.
Awaiting user input:
USER
need to
HELMSMAN
Agent: Reflection
Initial Plan:
Self-reflection:
Plan is complete, routing human decision...
HELMSMAN
Approval Required
Approve this plan? (yes/no)
USER
Yes
HELMSMAN
Complete
Workflow Complete!
Figure 6: The complete interaction Helmsman- during planning workflow on research
query task 1, The input query type follows the Out-of-Schema setting in AgentFL-Bench.
18

---

Published as a conference paper at ICLR 2026
A.3.3 HUMAN-HELMSMAN CONVERSATION WITH INCOMPLETE INPUT
HELMSMAN
System
Starting Planning Workflow
HELMSMAN
Input Required
Enter your federated learning research question:
USER
need federated learning for 15 phones with imbalanced data. Model is MobileNet.
HELMSMAN
Agent: Planning
Planning Agent: Thinking and researching...
Calling tools: web_search, socs_search
Web Search Results: ('query': ...', 'answer': "...', 'images'
[...], 'results': '...')
Docs Search Results: {...}
HELMSMAN
Agent: Reflection
Initial Plan:
Self-refletioo::
INCOMPLETE: The plan missing specific details for the dataset, the precise data partition strategy for imbalance, and the primary evaluation criteria, explicitly requested by the agent its final message.
PLAN INCOMPLETE Missing Information:
The plan missing specific details for the dataset, the precise data partition strategy for imbalance, and the primary evaluation criteria, as explicitly requested by the agent in its final message.
Plan incomplete, routing to human feedback...
HELMSMAN
Refinement Required
Provide additional information requirements:
USER The dataset is
HELMSMAN
Agent: Reflection
Initial Plan:
Self-reflection:
Plan is complete, routing to human decision...
HELMSMAN
Approval Required
Approve this plan? (yes/Ue /no)
USER
Yes
HELMSMAN
Complete
Workflow Complete!
Figure 7: The complete interaction Helmsman- during planning workflow on research query task 1. The input query type follows the Incomplete schema in AgentFL-Bench.
19

---

Published as a conference paper at ICLR 2026
A.3.4 HUMAN-HELMSMAN CONVERSATION WITH PARAPHRASED INPUT
HELMSMAN
System
Starting Planning Workflow
HELMSMAN
Input Required
Enter your federated learning research question:
USER
want to train MohileNet model on 15 mobile devices that have unbalanced CIFAR-10-TA data. Can you help me up a federated system and measure the accuracy?
HELMSMAN
Agent: Planning
Planning Agent Team: Thinking and researching...
Calling tools: web_search, docs_search
Web Search Results: ('query': ...', 'answer': ....', 'images'
[ [...], 'resulss' '...!) ...
Does Search Results: (...}
HELMSMAN
Agent: Reflection
Initial Plan:
Self-reflection:
The plan comprehensive and actionable for research project. clearly defines the objective, identifies the specific chatlenges (Non- IID, long-tail data). proposes a concrete methodology. does not require further information from the user to proceed.
Plan is complete and sound, routing to human for final verification...
HELMSMAN
Approval Required
Approve this plan? (yes/ /n)
USER
Yes
HELMSMAN
Complete
Workflow Complete!
Figure 8: The complete interaction of Helmsman-Human during planning workflow research
query task 1. The input query type follows the paraphrased schema in AgentFL-Benchh
20

---

Published as a conference paper at ICLR 2026
A.4 AGENTFL- BENCHMARK FOR AGENTIC SYSTEM
This section introduces details of AgentFLLBeence a comprehensive benchmark for assessing the capabilities of agentic systems in the domain of federated learning research. The benchmark consists of 16 unique tasks, detailed in Tables 13, 14, and 15. These tasks are structured according to the query template Table and are organized into five key research domains, each characterized by distinct set of challenges. Task queries are further categorized by their specific problem space using symbols, and essential information within each query is highlighted in green.
Table 13: Benchmark Dataset for User Query Evaluation Agentic Federated Learning Systems
(I). We report the problem space of each user query with the following symbols: Quantity Skew, Quality Skew (Feature), A Quality Skew (Label), Distribution Skew. For consistency, each user query follows the defined template pattern illustrated in the user query template. Q.n represents the identifier of the user query.
Research Area
Challenge
User Query
Q1 "I need deploy photo-sorting app on 15 smart- phones. Each phone stores highly imbalanced slice of long- tail distributed data (CIFAR-10-LE) Help me build a federated learning framework to train a MobileNet-V VI model, evaluating performance by top-1 test accuracy."
Heterogeneous FL
Data Heterogeneity
Q2 "I need to deploy an object classification program on 15 sensing devices. Each client holds slice of CIFAR- 100 data whose images are corrupted by high level Gaussian noise, Help me build a federated learning frumework to truin ShuffleNet model, mitigating the effect of imput data noise and targeting high globul test accuracy."
Q3 y "I need to deploy an image classification app on 15 smartphones. Each phone stores a local slice of CIFAR-ION ION data with varying rates of label noise (class flipping). Help me build a noise-robus robust federated learning framework to train a ResNet-8 model, evaluating performance by top-1 test accu- racy."
Q4 y need to deploy an object classification app across 4 digital camera devices. Each client holds a domain-specific slice of Office-Home dataset (Art, Clipart, Product, Real- World). Help me build federated learning frumework that copes with this domain shift across the 4 clients and trains ResNet-18 model, judging success by global top-1 accuracy."
Q5 y: need to deploy a human action recognition sys- tem across 15 wearable devices. Each client stores local ac- celerometer and gyroscope data from the UCI-HAR dataset with different user movement patterns and sensor placement variations. Help build federated learning framework train an LSST model that handles the sensor data heterogene- iry, evaluating performance by global test accuracy across six activity classes."
"I need to deploy a voice command recognition app across 15 mobile devices. Each client holds local Speech Com- mands data from users with different accents and speaking patterns. Help me build federated learning framework to train a CNN model that handles this speaker heterogeneity, evaluating performance by classification accuracy."
Q6 D:
21

---

Published as a conference paper at ICLR 2026
Table 14: Benchmark Dataset for User Query Evaluation in Agentic Federated Learning Systems (2). We report the problem space of each user query with the following symbols: * Communication Overhead, V Distribution Skew, + Catastrophic Forgetting, Resource Constraint. For consistency, each user query follows the defined template pattern as illustrated in the user query template. Q.n represents the identifier of the user query.
Research Area
Challenge
User Query
Q7 y: "I need to deploy skin lesion diagnosis system across 5 hospitals. Each hospital holds slice of Fed add-SSC2019 dermo- scopic images with different patient demographics and lesion type distributions. Help me build a federated learning frame- work to train a ResNet-8 model that handles data heterogeneity, evaluating performance by top-1 test accuracy.
Data Heterogeneity
Heterogeneous FL
Q8 ): "I need to deploy an object classification system across 15 mobile devices. Each client holds local slice of the Caltech101 dataset, containing images from different object calegories with varying visual styles, Help me build federated learning frumework to train a CNN model that can cope with this data heterogeneity, and evaluate performance by global test accuracy." "
Model Heterogene-
Q9 ( y: "I need to deploy an image classification app across 15 mobile devices with varying computational capacities. Each client will train a different-size 1/4 capacity) on local CIFAR-10 10 data slices. Help me build a fedevated learning framework that handles these heterogeneous model architectures and trains effective global model, evalu- ating performance by top-1 test accuracy.
ity
ResNet-18 model (full, 1/2,
Communication- efficient FL
Bandwidth Limits
Q10 (+); "I need to deploy object detection system across 15 mobile devices with limited device connection rale. Each client trains a ResNet-18 model on local CIFAR- 100 data, but the training process is construined by low server bandwidth. forcing low client participation rate (30%). Help me build federated learning framework that implements an adaptive client selection strategy to optimize training under these con- straints, evaluating performance by global test accuracy"
Connectivity Limits
Qu1 y "I need to deploy a handwritten character recognition system across 15 mobile devices with limited connectiviry. Each client trains a CNN model on local FEMNIST data, but network outoges limit communication to only 100 rounds. Help me build federated learning framework that achieves high accuracy with minimal communicatiin rounds."
Local Adaptation
"I need to deploy a personalized handwriting recog- nition app across 15 mobile devices. Each client holds FEM- NIST data from individual users with unique writing styles. Help me build a personalized federated learning framework that balances global knowledge with local user adaptation for a CNN model, evaluating performance by average client test accuracy."
Q12 (
Personalized FL
Data Heterogeneity
Q13 y: "I need deploy a personalized image classification app across 15 mobile devices. Each client holds local slice of CIFAR-I0 data with distinct image class preferences. Help me build personalized federated learning framework that adapts each client's unique data distribution while leveraging global knowledge, training a MobileNet-V VI model, and evaluating performance by average client test accuracy."
22

---

Published as a conference paper at ICLR 2026
Table 15: Benchmark Dataset for User Query Evaluation in Agentic Federated Learning Systems (2). We report the problem space of each user query with the following symbols: Communication Overhead, v Distribution Skew, & Catastrophic Forgetting. For consistency, each user query follows the defined template pattern as illustrated in the user query template. Q.n represents the identifier of the user query.
Research Area
Challenge
User Query
Federated
Sample Selection
Q14 (+): "I need to deploy medical image classification system across 5 hospitals. Each client holds a local pool oftnlabeled dermoscopic skin-lesion images from the DermaM- NIST dataset, can only afford label 20% subset due expensive expert annotation costs. Help me build a federated active-learingg framework that efficiently selects the most infor- mative samples for labeling while minimizing communication rounds, trains MobileNet-V2 model, and evaluates perfor- mance by top-1 accurucy."
Active Learning
Data Ileterogeneity
Q15 ()) "I need deploy image classification system across 15 mobile devices. Each client has unlabeled CIFAR-10 data with non-lID class distributions. Help me build a federated active learning framework that selects informative samples for labeling across heterogeneous clients, trains a CNN model, and evaluates performance by top-1 accuracy." "
Fed-CL
Incremental Tasks
Q16 (+): "I need deploy image classification system across 15 mobile devices that learn new object categories over time. Each client sequentially learns tasks from Split- Aplit-CIFARION (5 non-overlapped tasks, 20 classes each) but suffers from catas- trophic forgetting when learning new tasks. Help build federated continual learning framework that preserves knowl- edge of previous tasks while learning new ones, training ResNet-18 model, and evaluating performance by average for- getting and accuracy across tasks."
23

---

Published as a conference paper at ICLR 2026
A.5 EXPERIMENTAL SETUP FOR AGENTIC FEDERATED LEARNING BENCHMARK EVALUATION
This section details the experimental configurations for each task in the DpentFl-Benchhenchie with setups for cross-silo and cross-device scenarios presented Table 16 and Table 17, respectively. Unless specified otherwise, all federated training processes rounds. The primary evaluation metric is global test accuracy, with exceptions for personalization tasks (Q12, Q13), which use average client accuracy, and the continual learning task (Q16), which uses average task accuracy. The tables provide further task-specific settings, including datasets, models, and the unique client data distributions or constraints applied to each query.
conducted for 100 communication
Table 16: Experimental Setup for Agentic Federated Learning Benchmark Evaluation (Part A: Cross- Silo Scenarios). All experiments use 100 communication rounds and evaluated using accuracy as the primary metric unless otherwise noted. We use global test accuracy as the evaluation metric for all the tasks.
Query
Task
Dataset
Model
Task-Specific Setting
Cross-Silo Scenarios (4/5 Clients)
Q4
Object Recognition
Office-Home
ResNet-18
Distribute data across do- mains (Art, Clipart, Product, Real-World)
Q7
Medical Diagnosis
Fed-ISIC2019
ResNet-8
Distribute data based ITD patient demographics vari- ation
Q14
Medical Diagnosis
DermaMNIST
CNN
Configure 20% labeling budget for active sample selection per hospital
24

---

Published as a conference paper at ICLR 2026
Table 17: Experimental Setup for Agentic Federated Learning Benchmark Evaluation (Part B: Cross- Device Scenarios). All experiments use 100 communication rounds and are evaluated using accuracy as the primary metric unless otherwise noted. We use global test accuracy as the evaluation metric for all the tasks. Evaluated using average client accuracy. Evvlluted using average task accuracy.
Query
Task
Dataset
Model
Task-Specific Setting
Cross-Device Scenarios (15 Clients)
Q1
Object Recognition
CIFAR-IO-LT
MobileNet-
Distribute data with long-tail class imbalance and quantity skew across clients
V1
Q2
Handwriting Recogni-
FEMNIST
ShuffleNet
Apply high-level Gaussian noise corruption to input image features
tion
Q3
Object Recognition
CIFAR-ION
ResNet-8
Introduce varying rates of label noise through class flipping cor- ruption
Q5
Human Activity Recog-
HAR
LSTM
Distribute sensor data with dif- ferent human movement pat- terss
nition
Q6
Speech Recognition
Speech mands
Com-
CNN
Distribute with speaker heterogeneity from different ac- cents and patterns
Q8
Object Recognition
10
CNN
Distribute data with category imbalance and visual-style het- erogeneity across clients
Q9
Object Recognition
CIFAR-10
ResNet-18
Deploy heterogeneous model architectures (full, 1/2, 1/4 ca- pacity) across clients
Q10
Object Recognition
CIFAR-100
ResNet-18
Device connection rate is con- strained to 30% due the low server bandwidth
Q11
Handwriting
Recogni-
FEMNIST
CNN
Limiting the training 100 rounds to reduce the commu- nication overhead
tion
Q12
Handwriting
Recogni-
FEMNIST
CNN
Balance global knowledge shar- ing with local user-specific adaptation
tion
Q13'
Object Recognition
CIFAR-10
MobileNet-
Handle severe class imbal- ance with personalized cate- gory preferences per user
V1
Q15
Object Recognition
CIFAR-10
CNN
Select 20% subset informa- tive samples for labeling across non-llD client distributions
Q16
Object Recognition
Split- COIi-CIFARR10
ResNet-18
Learn 5 sequential and non- overlapped tasks while mitigat- ing catastrophic forgetting
25

---

Published as a conference paper at ICLR 2026
A.6
AGENT CONFIGURATION FOR PLANNING STAGE
A.6.1 PROMPT FOR PLANNING AGENT
Prompt template for the planning agent responsible for analyzing user requirements and generating comprehensive federated learning research plans. The agent follows a prree-phase approach: initial analysis, iterative information gathering, and final plan generation with tool integration.
Planning Agent Prompt Template
Agent Role: You are a Planning Agent specialized in federated learning (FL) systems. Your role is to:
1. Analyze user requirements and create detailed, actionable plans
2. ACTIVELY USE AVAILABLE TOOLS to gather information for informed plan- ning:
Use web.search to find current information and best practices
Use search_docs to find relevant internal documentation
3. Iterate on plans based user feedback
4. Remember and build upon previous planning iterations analyze user queries about fed- erated learning problems, engage in iterative dialogue to gather complete requirements, and produce comprehensive, actionable execution plans for downstream agents.
Task Output Format: When writing the research plan for a given user query. you must output with the following format: PLAN:
Summary: concise restatement of the user's federated learning objectives and key requirements.
2. Challenges:
Explanation of the key technical and operational challenges implicit in the query.
3. Tasks: A prioritized, ordered list of steps needed to tackle those challenges.
Number each task sequentially
Include brief descriptions of objectives
4. Technical Setup: For each major component, specify detailed configurations:
Model Architecture: {model}
Datasets: {data}
Client Configuration: (rum.clients}
Data Partition Strategy: {split it (split.methoo .eehhod]
Local Training Epochs: 5 (default is 5 training epochs per client)
Evaluation Criteria: fcriterial (metrics for optimization goals)
Privacy Mechanisms: (privacy) (None if no privacy mechanisms is applied)
Guidelines for Planning:
Phase 1: Initial Analysis
Parse the user's query to understand the core federated learning objective
Identify what information is provided and what is missing
Determine you have sufficient information create a complete plan
Phase 2: Information Gathering (lterative) If critical information missing:
1. Ask Specific Questions: Request missing technical details, constraints, require- ments
2. Wait for User Response: Allow user to provide additional information
26

---

Published as a conference paper at ICLR 2026
3. Validate Understanding: Confirm your interpretation of their responses
4. Repeat if Needed: Continue until all essential information is captured
Phase 3: Plan Generation Once you have sufficient information:
1. Create a comprehensive execution plan following the specified format
2. EXPLICITLY REFERENCE techniques and methods found tool results
3. Request final approval or modifications from users
4. Finalize the plan only after user confirmation
Tool Invocation Template:
Thought: Do need to use tool? [Yes/No with reasoning]
Action: [tool_name]
Action Input: ("query": "[specific search terms or request]", "context"; "[relevant background information]""
Observation: [tool response will appear here]
Available Tools: (docs.tool, search.tool}
Important Notes:
Never proceed with incomplete information, concisely asking for clarification
The word "PLAN:" must appear on its own line, followed by the plan.
IMPORTANT: You MUST use the available tools to research and gather information before creating your plan. Do not just rely your general knowledge actively search for relevant information.
WHEN TOOLS RETURN INFORMATION, YOU MUST INCORPORATE
Include specific techniques, algorithms, and best practices from the search results in your plan
IMPORTANT: Use only ASCII characters your code. Do NOT use Unicode characters like Greek letters, instead use their English names.
27

---

Published as a conference paper at ICLR 2026
A.6.2 PROMPT FOR REFLECTION AGENT
Prompt template used by the reflection agent to evaluate plan completeness. The agent performs two-step verification process to determine whether the generated research plan contains all necessary components for federated learning task execution.
Reflection Agent Prompt Template
Instructions: You MUST perform the following two-step self-reflection determine if the plan is complete or not:
First, you need to check the message history sto ee the agent is asking for more information. If so, then the plan is incomplete.
2. Analyze this federated learning research plan for completeness. Be reasonable this is research plan, not an implementation specification.
Response Format: YOU MUST respond starting with EXACTLY of these two options:
COMPLETE: [brief justification]
.
INCOMPLETE: [what major components are mi missing]
Input Variables:
USER QUERY: {user query)
GENERATED PLAN: caatrent.pparn
Message history: {message}
Completeness Criteria:
A COMPLETE research plan should have:
1. Clear objectives/summary of what to achieve
2. Identified challenges or considerations
3. High-level approach or methodology (specific algorithms/tecnnques))
4. Key tasks steps to follow
5. Basic technical setup (dataset, model type, evaluation metrics)
A plan is INCOMPLETE only if it's missing major components like:
No clear objective
No methodology or approach
No tasks or steps
Too vague be actionable
Note: Implementation details like exact hyperparameters, layer configurations, or specific parameter values can be determined during the research phase and are NOT required for complete plan.
28

---

Published as a conference paper at ICLR 2026
A.7 AGENT CONFIGURATION FOR MODULAR CODING STAGE
A.7.1 PROMPT FOR SUPERVISION AGENT
Prompt template for the supervision agent responsible for converting high-level FL research plans into detailed implementation specifications. The agent analyzes research objectives and generates module-specific technical requirements, experimental configurations, and interdependency mappings the FLOWER FL framework.
Supervision Agent Prompt Template
You are a Supervision Agent for Federated Learning (FL) research project. Your task is to analyze high-level FL research plan, create detailed implementation plan broken down by code modules, including the comprehensive experiment setup and FL challenge-specifte techniques for each module. Your expertise in FL, distributed systems, and machine learning will be crucial for this task.
Role:
Task: You will be provided with a high-level FL research plan. Analyze this plan carefully and for each module in order to solve the FL challenges in
identify specific algorihhns/requrrenet the plan. You need to actively use tools to find specific technique requirements for each model.
Required Modules:
Task Module: Define classes/functtin for the model and data, including training and testing method.
Client Module: Define the stateful clients based on FLOWER FL framework for this FL. research.
Server Module: Define the server application based on FLOWER FL framework.
Strategy Module: Define custom FL strategy to solve the FL challenges in the research.
Guidelines:
For each module (data, client, server, strategy), list numbered steps describing:
Usage of the module
IMPORTANT: Based on the research plan key challenges, you MUST specify additional technique for each module to solve the challenges
2. Experimental configurations based on the provided research plan
3. Module interdependency analysis: To Identify how modules interact and depend on each other
4. IMPORTANT: Use only ASCII characters in your plan. Do NOT use Unicode characters like Greek letters, Instead use their English names.
Input Format:
<fl efi_research_plan>
[rescarch_plan) </fl_reseeach_planns
Output Format:
Implementation Plan Structure:
1. Task Module Implementation
IMPLEMENTATION PLAN: Summary Technical requirements (1, 2, 3...)
CONFIGURATIONS Dataset, Batch Size, Model Architecture, Data Partition Strategy, Number of Clients
2. Client Module Implementation
IMPLEMENTATION PLAN: Summary + Technical requirements (1, 2, 3...)
CONFIGURATION: Local Training Epochs, Evaluation Criteria, Number of Clients
29

---

Published as a conference paper at ICLR 2026
3. Server Module Implementation
IMPLEMENTATION PLAN: Summary Technical requirements (1, 2, 3...)
CONFIGURATION: Communication Rounds, Evaluation Criteria, Number of Clients, Client Participation
4. Strategy Module Implementation
IMPLEMENTATION PLAN: Summary Technical requirements (1, 2, 3...)
5. Module Interdependency: List interactions and dependencies (1, 3...)
Tool Invocation Template:
Thought: Do I need to use tool? [Yes/No with reasoning]
Action: [tool name]
Action Input: ("query"; "[specific search terms or request]", "context": "[relevant background information]")
Observation: [tool response will appear here]
Available Tools: (docs.tool, search.tool]
30

---

Published as a conference paper at ICLR 2026
A.7.2 TASK MODULE CODING GROUP
Prompt template for the task module coder responsible for implementing or debugging FL task modules. The agent operates in two modes: debugging existing implementations or creating new implementations from scratch using the FLOWER FL framework with FederatedDataset integration.
Task Module Coder Prompt Template
Role: You Task Module Coder for a Federated Learning (FL) research project.
Conditional Execution:
Condition 1: Debugging Mode
Task: Debug the Task Module implementation based on provided codebase and testing feedback
Input Variables:
Task Description: {task)
Codebase: (state. get ("codebase task", ""}}
Test Feedback: (state. get "ttakk.eettfeeebbaeee ""}
Output: Fixed Python code implementation in 'python blocks only
Condition 2: Implementation Mode
Task: Implement Task Module from scratch using FLOWERR framework with FederatedDataset
Input: Task Description: {task}
Implementation Requirements (Condition 2):
STEP 1: Model and Data Analysis (CRITICAL.)
Dataset ID Mapping:
CIFAR-10: "zoft-cs/cifaz100
CIFAR-100: "uoft-ccs/cifar100"
FEMNIST: "ELwslabss/eemistt
OfficeHome: "ELwrlabs//officcetoome
Speech Commands: hrg.mmddd..googgesspppeecooommm
Fed-ISIC2019: 0bao0//0ee1111bbb/eeee-ss1210199
Caltech101: "Elwraaba/caatechh1111
UCI-HAR: "Beothukk/uci---er--eeeeeee
2. Partition Strategies:
IID: LidPartitioner
Nonn-IDDDirrihhltt DirichletPartitioner with alpha=0.5
-Lnggttil: ExponentialPartitioner
STEP 2: Model Implementation
Implement EXACT model architecture from task description
Implement get_model () function
Implement train () and test () functions
STEP 3:
Data Loading with FederatedDataset (CRITICAL)
MUST USE: flwr hwwr_ddtasets.Feddeee FeteratedDDttase
MUST USE: Appropriate Partitioner from Step
Cache FederatedDataset using global variable
Split partition for train/test using train test split
STEP 4: Additional Techniques
31

---

Published as a conference paper at ICLR 2026
Implement additional techniques to solve FL challenges if necessary
Code Structure Requirements:
Concise coding with necessary docstrings only
Include imports from flwr.datasets
partition id provided by Context in client fm
Use global fds variable for caching
80/20 train/test split with seed=42
Required Function Structure:
class TaskSpeccffcMModeennnn... Modull)le))
def train(net, trainloader, epochs, Ir, device)
def test(net, testloader, device)
def apply www.ttrnaaannsttt transforms(batch))
def apply_eval cra..trranfoommscaach)))
def get detadaatppatiioonn id, num uur-tiaiiionn, batch_size)
def get_model()
UCI-HAR Transform Example:
spply seply_train www.ttrrinntttnnteeeeeettte (ostch)
ched
(neechh/"taggeee (argget"")
1: io_hatched:
batch_ size len (batch| (aattchi"target"]]
Fescures__ls
sargete_list
for idx in range sizel:
semple_feturree Features = sample.
(5611
sample aggerd ibscch str(i:]
list.
torgete_list ltppeeen_ettpppeeddd append itach"tittth [idx] - Convert 1-6 to 3-5
feacures torch. corchhteneer ifestures 1st, atypeetarch.flee:535
sargets torch.tensod (tarpetst,,,,, tozc..)ooc..onng)
return |"teatures"; teatures, "label"; targets:
Output Format: Complete Python code implementation wrapped "'python" blocks only.
32

---

Published as a conference paper at ICLR 2026
A.7.3 STRATEGY MODULE CODING GROUP
Prompt template for the strategy module coder responsible for implementing or debugging custom FL strategies. The agent operates in two modes: debugging existing strategy implementations or creating new strategies from scratch using the FLOWER FL framework with precise method signatures for v1.19.0 compatibility.
Strategy Module Coder Prompt Template
Role: You are a Strategy Module Coder for a Federated Learning (FL) research project.
Conditional Execution:
Condition 1: Debugging Mode
Task: Debug the Strategy Module implementation based provided codebase and testing feedback
Input Variables:
Task Description: {task)
Codehase: (state. get ""codebase.strategyy" "")}
Test Feedback: fstate. get www.rattey..est.feedbee.. "")
Output: Fixed Python code implementation in 'pythor .. blocks only
Condition 2: Implementation Mode
Task: Implement Strategy Module from scratch using FLOWER FL framework
Input: Task Description: [task}
Objective: Implement additional techniques/methods to solve specific FL. challenges
Implementation Requirements (Condition 2):
1. Implement all functionality of the custom strategy including additional technical requirements
2. Implement additional techniques/metthods to solve FL challenges in the task
3. Include all necessary imports
4. Be concise at coding and only add necessary docstrings
5. CRITICAL: ALL values in configuration dictionaries and metrics dictionaries MUST be scalar types only
CRITICAL Method Signature Requirements (Flower Framework v1.19.0):
evaluate(self,, server round: int, parameters: Parameters)
configure fit(self,, server round: int, parameters: Parameters, client manager: Client- Manager)
gegregate_fit(sele server pruuderrooddn:-ooooo int, results: Lis("Tuple(ChiemPPooyy FitRes]], failures: ListUnnion/TupteeCCeenoooo FitRes], BaseExceptionl)
menfigure_evaluateee server:round: int, parameters: Parameters, manager: ClientManager)
aggregate evaluate(self,, server round: int, results: Liss[Tuple(ChiemPPoxyy Evalu- ateRes]]] failures:
FitRes], BaseExceptionn])
Required Imports and Class Structure:
fron impoct ABC, abatractnathod
From typing import hist, list, Options', ly "up' Uniin, Gs Gallable
from Elat.comon ingort Dazameskss, Scalar, paranetera_toordae cllen:Maaagge
Clar.sesves.ot weleeeclleeetnnaeeeeee
From Flar.server.dd fent fiet_ prexy import Prony
from Clar.oumon ingort Fillna, Fitles, EvalusteIos, aablles Elxt. wwr.sseeeesetttttn import Strategy
class YourCus
(Blealngy):
#xeouston FL Strategy implemenaction"
33

---

Published as a conference paper at ICLR 2026
def init (self,
inttioi__erommcees: Optianal Faranstors] eraluate tn: [etioonl[Cal_aa_.. None,
on fit cit__oonig___t fri Options [Calleble] Noney
oipezt) inittt_
sel selfiinitia) peraneters initial parameters
colf.evallato_.n evaluacs_tn
seit,on_tit._c cittcootiig_tr.. mnuuii__cnttigii
strateoy parametors 12
Required Method Implementations:
initialize parameters(self, client manager: ClientManager) Initialize global model parameters
configure_fit(seff, fit((slff, server_round, parameters, mlient_ round
configure
Configure training
aggregate
aggregate_fit(self fit(sllf, server,round, results, failures) Aggregate training results
configure evaluate(self,, server round, parameters, client manager)- Configure eval- uation round
engregate_evaluattee server_round results, failures) Aggregate evaluation results
evaluate(selff, server_round parameters) EEvalute current model parameters
Critical evaluate() Method Template:
eveluate isclt, sorver_round: int, paromocees: Faramoters; (Zuplc (ict
111:
" "Evaluate the current medel parametors.
cRITICAL: This nethed eveisate/sele, server_round: int, paremeters: Faramotors: Optional [Tuplc [flost, pict(ste, Beslor 111
exactly:
Arça:
secver_ The caszent SHIVME round paranacers: The nodel parametars to evaluate
Call the avaluation fanction (fzon server nodule;
The eveluate inn parametera to may (pazaneters) paramstessnndesrae loss, metrics wol...valuata_fee (n(sovverr_eund paramaoes_nnarrraya awture Lows, netrica
(serowr round, paaaetera_dddrreeeeeeee config)
Output Format: Complete Python code implementation wrapped in " "'python blocks only.
34

---

Published as a conference paper at ICLR 2026
A.7.4 CLIENT MODULE CODING GROUP
Prompt template for the client module coder responsible for implementing or debugging FL client components, The agent operates in two modes: debugging existing client implementations or creating new client applications from scratch using the FLOWER FL framework with proper data partitioning and stateful client management.
Client Module Coder Prompt Template
Role: You are a Client Module Coder for a Federated Learning (FL) research project.
Conditional Execution:
Condition 1: Debugging Mode
Task:
Debug the Client Module implementation based on provided codebase and testing feedback
Input Variables:
Task Description: [task}
Codebase: (state. get ("codebase.clleent") "")]
Test Feedback: (state get ("alient.test.ffeedaaee ""))
Output: Fixed Python code implementation in "python '* blocks only
Condition 2: Implementation Mode
Task: Implement Client Module from scratch using FLOWER FL framework
Input: Task Description: {task}
Objective: Implement additional techniques/methodds solve specific FL challenges
Implementation Requirements (Condition 2):
Implement all functionality of clients including additional technical requirements
2. Extract part t ioniic and num_part, rt i ons from context. nade_config
3. Use get_data () from task module with extracted partition_id and num pprttition
4. Use train () and test () functions from task module for local training and evaluation
5. Implement additional techniques/methodss solve FL. challenges in the task if necessary
6. Be concise at coding and only add necessary docstrings
Include all necessary imports
8. CRITICAL: client_fn must return FlowerClient wtw.:eliiiintttt.eetrteee.e.e to.elient (), not just Flower-
Client instance
9. ALL values configuration dictionaries and metrics dictionaries MUST be scalar types only
Required Imports and Class Structure:
fiwx.siient import ClientAps, NumPycliant
from Clar.oumon inguet Acsay, Acraylicucd, Concext, Becordbiut Ezon task inport gat_data, get_nodal, train, test Snport
class PlovarClient (MunPyCliont]:
24 PrrDefine the stateful Flover alientrr
Linit (aalf, partition_idi int, trainioador, coatloades, device) sel self.gastit partitionnid idd partiition_d self, Lrainloads: = Leainloade: testinadt testloader delfidevice device
self.model gel__ndeel (Lofdevio)
def
paraneters, configl:
vrrasain nodel using paransleza from servez on clienz'a dacaset. Return updated parameters and netriegtr
Set nodel parameters from server
Train sodel
Return updated parameters and metrics
pass
paraneters, configl:
Vrauaalale notel sent surowe un cliwnt's idation wet.
Return performance netrics,"
35

---

Published as a conference paper at ICLR 2026
Set model parameters zrom Server Pualu the model
pas0
CRITICAL client Implementation:
dzf salcnt__fftcsccaaaa Context]:
11 FloverClient
partition.
CRITICAL;
Ex-ract partitiongid and cunppaccitione Exom context. code_ Fleserclient (...) (...)..t__cleennt
NinPyClient Client
Gez pastition configuration from context
partation_id aontaxt..ode_cooo contial "aarili""srtttte "sattitoonii"" nampprrs itlion
pu (fnun pertitiion")
Load the partition data using the function fxon task module trainlwwder, Lest loader gel_dabaa
parsition idsgastition num aun_particiona wow.rrrriiooooooaaaaaataoooo
batek
Oz gel. from punfig
ioe
device torch. Mouda:: if tozch, cuda, labevalabbll else *apu"!
Flowe:llient instance
flower_client FlosarCllant pars idde idegart ion, scaircoadeeee ee www.eetaaillooeeo ealloade,, loaderttest ceedeviee
IMPORTANT: Convect lent unding souel fent coture: Soover_cllndt Lounl iint
the ell lent generatiion Fanesian clieni_app ClientAps Seieet_tteeel
Constrrust the eI ClientApg
CRITICAL Implementation Notes:
MUST extract partition.is from context.node confleloo"lll" paanigglppatttiooniie
2. MUST extract partitions from contextnode configl"num ''nmm num-partitions"]
3. get.data() takes partition id, num.partitions, and batch_size
4. client.fn fn MUST return FlowerClient(....ooocoo
Required Methods:
init (self, partition id, trainLoadez, testloader, device) Ini- tialize client with data partition
fit (self, parameters, config) Local training with server parameters
evaluate (self, parameters, config) Local evaluation with server parameters
client En (context: Context) Factory function returning client instance
Output Format: Complete Python code implementation wrapped in "python blocks only.
36

---

Published as a conference paper at ICLR 2026
A.7.5 SERVER MODULE CODING GROUP
Prompt template for the server module coder responsible for implementing debugging FL server components. The agent operates in two modes: debugging existing server implementations or creating new server applications from scratch using the FLOWER FL framework with proper strategy integration and centralized evaluation setup.
Server Module Coder Prompt Template
Role: You are a Server Module Coder for Federated Learning (FL) research project.
Conditional Execution:
Condition 1: Debugging Mode
Task: Debug the Server Module implementation based provided codebase and testing feedback
Input Variables:
Task Description: {task}
Codebase: {state. get ("codebase.servee", "")}
-est Feedback: (state. ""server.test.feebbeee "") }
Output: Fixed Python code implementation in 'python blocks only
Condition 2: Implementation Mode
Task: Implement Server Module from seratch using FLOWER FL framework
Input:
Task Description Available Codebases (task.py, strategy-py)
Objective: Import necessary classes/functions and implement additional FL tech- niques
Implementation Requirements (Condition 2):
STEP 1: Server Configuration Analysis (CRITICAL)
1. Custom Strategy: Extract class name from strategy.py
2. Evaluation Requirements: Identify evaluation function needs for centralized evalua- tion
3. Server Configuration: Determine server parameters (num evaluation fre- quency)
4. Additional FL. Techniques: Identify server-side FL. techniques to implement
STEP 2: Evaluation Function Implementation (CRITICAL)
1. CRITICAL: Implement gen.evaluate.fn () with EXACT signature:
evaluate sserverrroun parameters.ndarran config) NO client manager parameter
2. Parameter Handling: Convert parameters_ndarranl model state_dict
3. Test Data Loading: Use get_data () from task module for server evaluation
4. Evaluation Logic: Use test () function from task module
STEP 3: Server Configuration (CRITICAL)
Strategy Integration: Import and instantiate custom strategy from strategy-py
2. Initial Parameters: Extract model parameters using ndarrays ttt.sttpppprar parameters
3. ServerConfig: Configure with appropriate num rounds (3 for testing)
4. ServerAppCComponetts:: Properly construct with strategy and config
STEP 4: Additional Server Techniques
Implement additional server-side techniques to solve FL challenges necessary
Code Structure Requirements:
37

---

Published as a conference paper at ICLR 2026
Concise coding with necessary doestrings only
Include imports from flwr.server and flwr.common
ALL values in dictionaries MUST be scalar types only
CRITICAL: Evaluation function signature: evaluate (server.round,
parameters andarrays,, config)
CRITICAL Function Signature Requirements:
(tn.evalluate..fhtessttt (testllodet device) -> Callable
evaluate (sereer...orr axrameters config) ->
Tuple (fleat, Dict (str, Scalar]]
mn.fitt.oofiggssevver.oon..
int) -> Dict (str, Scalar]
server (context: Context) -> ServerApsCompnents
Required Imports and Structure:
odarsays
parawaers_to_ndarrayy
fzon fixt.servez import Sorvesλpe, ServeshesComeeennss Servertonfig rzon tass inport gct_dsta, get get_modal, test fzen atzategy inpozt YourCustenStrateeyy
Inport cuaton aczacegy
ger_evalicate, per_oornisses_tttttooooooee devical:
Genarate the fanction for cantralized evaluatioa.'' def evalcate (sarver__
parameters_ndzrraye config):
"VEvalluee glubal model
ised
CRITICAL:
fanction aignature be exactly: evaicato (sarver_ teonver_roddd, aasmeters_nddrree ceafig) Do NOT add clieni_ paramelez," Convest pass
and evaluate
coLurt: evaluate
Fst_configgleee roands
Cons.suct config that clienta seceive when zanming firl) pass
del server_fi Context]:
Bead paranetors context canfia, inscantiste modal, convert Parematers, prepare datasel for evaluation configuee strategy, and reburn Gecooeckgpinnggmessss...
Instialise
get parareters
model mel_m mode! padanteas ndderayg_to_prraeees (Ival. caurpy (1
for vel in model, satate ot val
Secup evaluation use pastition server evaluation. mm parsitions = context.ran run config.get (fhun- itotn"" 15)
Lest get spactit idn_ wom.peetiliommennn.o part 2e eveluate gen_eva
loades, device="opu")
Configure accelegy
strazegy DourGsstenStratenn
disiallppeeee parantt erettrranddelll.. _empdeel_aarantteet
avalcale_ fomeeva Eneevallt ustae_
or Fit _it__ittoostttt config freon, fit fit__cotiig
Configure server
for ServecConf tna. rounds-3)
returr Serverleptompoentt letrategg-stzeteeey contigrconzig!
Const rucl Serowrkpp server_
ServerApg wew.wer_tneserveeee tnwservvrr.tt)
Output Format: Complete Python code implementation wrapped in " "'python' blocks only.
38

---

Published as a conference paper at ICLR 2026
A.7.6 ORCHESTRATOR CODING GROUP
Prompt template for the orchestrator agent responsible for implementing or debugging the run.py script that coordinates all FL modules. The agent operates in two modes: debugging existing orchestration implementations or creating new run.py scripts from scratch using FLOWER simulation framework to execute the complete federated learning experiment.
Orchestrator Agent Prompt Template
Role: You Orchestrator Federated Learning rescarch project.
Conditional Execution:
Condition 1: Debugging Mode
Task: Debug the run.py code to run HL simulation experiment by coordinating implementation of cach module
Objective: Fix issucs in run.py code based on test feedback
Input Variables:
Imported modules: py, client app-py, server apperrppppp strategy py; my
Codehase each module: (codebase se task). peodebbaae.lleeng (codebaae.server)
Ccodebaaeeattraleee (codebase.run)
"")} Oulpal: Fixed rum.y implementation "python" in blocks only
Feedback: (state get
Condition 2: Implementation Mode
Task: Write run.py script for running simulation that orchestrates all modules
Objective: Import delined clusses/lunctions from available modules to run FL simulation
Input Variables:
Implementation Overview: (Impl ementat on overvies)
Available Modules: task.py. client app-py, server app-py. strutegy-py
Codebase for each module: (codebase.task). (codebase.client). (codebese.sezvez).
stodebase.strateggy
Available Modules and Purposes:
task.py: Contains model, data preprocessing and loading
client.app-py: Contains FlowerClient and client.fo
server.app-py: Contains server configuration and ServerApp
py: Contains the custom strategy
Requirements for run.py (Condition 2):
Import nccessary clascesffunctios from each module
simulation parameters based the implementation overview
3. Run the simulation and collect resalts
4. Saveldisplay experiment results and metrics
Be concise at coding and only add necessory docstrings
Conligure FL rounds to for testing
Simulation Structure Example:
backend_config (MoLient rescurcee." ("mun_opuaa: 2, "cun_gpua": 0.011 run simiiationnn
SeroerApp
ell eettett wappepppellleoco aeieee_pppp
AlientApp to execated each SuparNode nan_supeznoder NUN CLIENTS, i Nunber of hedes that run Clien:App
backend Loon www.bckeedddd......o.oo.
Besourcw al lon for simclation
Critical Implementation Notes:
Import client client_app from client.app-ps
Import server app server app-py
Use flwz. simulat sun.s
Lon for orchestration
Conligure backend.config ig with uppropriate
allocation
Set num_supe rnodes hased on number of clients in experiment
Configure simulation for rounds for testing purposes
Output Format: Complete Python code implemeniation wrapped 'python blocks only.
39

---

Published as a conference paper at ICLR 2026
A.8 AGENT CONFIGURATION FOR EVALUATION STAGE
A.8.1 PROMPT FOR EVALUATOR AGENT
Prompt template for the evaluator agent responsible for analyzing FL simulation outputs to determine experiment success failure. The agent examines return codes, stdout/stderr logs, and applies comprehensive criteria to assess whether the federated learning simulation executed properly with meaningful results.
Evaluator Agent Prompt Template
Role: Analyze FL simulation outpur and determine if it successfully.
Input Variables:
Return Code: (retumcode)
STDOUT: [stdour[-5000:] if len(stdout) 5000 else stdout)
STDERR: {stdemf -3000:] if lentstderr) 3000 else stdem)
Success Criteria SUCCESSFUL FL simulation must:
1. Complete multiple Fl. rounds without errors
Show changing loss/aceuracy valuos (oot stuck)
Have no Python errors, exceptions, or "Esror" messages
Show successful clicat participation. NOT show "O results" in aggregate or aggregate.evaluaee
lines containing
loading data", "Eror:" "Exception"" similar
Failure Conditions- IMPORTANT: Even simulation completes and shows summary, consider FAILED
Any lines with "Ereer" "Exception. "Tracebcck"
Client ClientApp/Seen App exceptions
"received results" in aggregation functions
Data londing errors or path issues
Lossfaccuracy values 't change between rounds
Any error messages
Analysis Approach:
1. Check Return Code: Verify process completed successfully
Sean Error Keywords: Search for "Error" "Exception", "Traceback", "Failed"
Verify Progress:
multiple completed changing metrics
received results Check Data Loading: Fasure data loading path issues
Validate Client Participation:
aggregation
Assess Metric Evolution: Verify loss/accuracy values change across rounds
Common Failure Patlerms:
ClientAppExeeption: slice indices must integerss
aggregate evaluate received results
loading
FileNooFFounEzroor or pail-relltted
Metrics remain constant rounds
Process termination with non-zero return code
Output Format:
SUCCESS:
[Yes/No]
REASON: (Brief explanation of
failure] ERROR: [If failed, the key error message]
Example Outputs:
Success: SUCCESS: Yes, REASON: Simulation completed 3 rounds with decreasing loss values, ERROR: None
Failure: SUCCESS: No, REASON: Client participation failed, ERROR: aggregate fit received 0 results
40

---

Published as a conference paper at ICLR 2026
A.8.2 PROMPT FOR DEBUGGER AGENT
Prompt template for the simulation debugger responsible for analyzing runtime errors and fixing problematic code in FL simulation systems. The agent examines error feedback and applies targeted fixes while preserving the correct function signatures.
Simulation Debugger Prompt Template
Role: You are Simulation Dehugger for Federated I Leaming project using FLOWER framework, The FL. simulation failed with runtime crrors. Your job analyze the fix the problematic code.
Input Variables:
Feedback: (exror_feedback)
Current Code Files:
py: (code.Ciles [un.
py: (cede_files Lask.py"1)
plpcat.app.py: (code_files client..pppppp 1}
server_up--p: (cade_ft lea ("server_app-pp)
stratgypy: (code f1 ei "strrttgy-pp"
Debugging Process:
Analyze Error Feedback: Identify which file(s) contain the bug
Fix Runtime Errors: Correct issues the relevant code lile(s)
Common Issues to Check:
Import eττоrs (missing imports, circular imports)
Incorrect function signatures
Missing required parameters
Tyne mismatches between
Data leading or partitioning erroes
Validation: Ensure fixes address specific error without breaking other functionality
Common Categories:
Import Issues: Missing circular imports between modules
Function Signatures: Mismatched parameters between function calls and definitions
Type Mismatches: Incorrect data passed between modules
Data Loading: Purtitioning erress, missing datasets, or puth issues
Framework Compatibility: FT FLOWER version-specifee specificcii method signatures
Parameter Handling: Missing or incoreetly formarted configuration parameters
Output Format: MUST ONLY output the corrected code for ONLY the files changes.
Single File Format:
FILE: filename.
" 'python
[complete corrected code]
multiple need changes, output using Important: Only include files that actually need modifications. Fix ONLY what's braken. Don't refactor or optimize unrelated cude.
41


Reviews:
[{"review_id": "OhH6qhZoK5", "reviewer": "Reviewer_Rzfo", "summary": "Helmsman is a multi-agent system that designs federated learning frameworks. It uses planning, coding, and evaluation agents that work together to generate and refine working FL systems. Authors also design tasks to evaluate their method. Experiments show it produces results as good as or better than hand-crafted methods.", "questions": "How stable are Helmsman’s synthesized FL systems across multiple independent runs? What mechanisms could ensure reproducibility?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper explores multi-agent system design in federated learning, a space that has been largely underexplored\n\nThe authors introduce AgentFL-Bench to test and compare autonomous FL systems\n\nHelmsman is a meta agent, a system that builds other AI systems, automating the design and testing process. This approach offers a promising path toward scalable and efficient FL pipeline development.", "weaknesses": "The system depends heavily on large computational resources and simulation time which limiting real-world scalability.\n\n62.5% of tasks ran fully automatically, while the remaining tasks needed some human-in-the-loop input, which suggests that true autonomy is not yet achieved.\n\n\nThe paper doesn’t compare Helmsman with other meta-agent systems that could also design federated learning setups, missing a fair baseline for comparison.\n\nHelmsman’s variability arises from LLM randomness, agent interactions, stochastic simulations, and iterative debugging, all of which make outcomes slightly different each time it runs. However, the paper does not report standard deviations for these results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mLumX8Slut", "reviewer": "Reviewer_ygHn", "summary": "The paper presents HELMSMAN, a multi-agent framework designed to automate the end-to-end synthesis of robust Federated Learning (FL) systems based on high-level user specifications. The introduction details how FL deployment is challenged by data heterogeneity, system constraints, and the manual, brittle nature of current solutions. HELMSMAN systematically addresses these through a three-phase workflow: (1) interactive human-in-the-loop planning, (2) modular code generation by coordinated agent teams, and (3) autonomous iterative evaluation and refinement in sandboxed environments. The system integrates advanced tools for knowledge retrieval and real-world simulation, ensuring plans are grounded and code is validated. The authors introduce AgentFL-Bench, a new benchmark with 16 tasks reflecting diverse FL challenges. Experimental results show that HELMSMAN outperforms or matches handcrafted and specialized baselines across various domains, demonstrating its ability to synthesize innovative algorithmic solutions, especially in complex settings like continual learning. Discussions highlight the benefits of multi-agent collaboration and human oversight, as well as current limitations regarding computational costs and task complexity. The work concludes by emphasizing HELMSMAN’s progress toward autonomous FL engineering and outlining ambitions for self-evolutionary capabilities in future iterations.", "questions": "- While HITL phases are discussed, the conditions, guidelines, and extent of required human intervention (especially for more ambiguous tasks) are somewhat vague. Could the authors provide explicit criteria or real-world examples delineating when and to what degree HITL is invoked, and how its frequency impacts system autonomy and scalability? Understanding the precise role and burden of human intervention is important for assessing Helmsman’s autonomy and practicality.\n\n- The choice of agent specialization (planning, coding, debugging, etc.) is motivated by division of labor, but could the authors elaborate whether alternative agent decompositions were explored? For instance, would a two-agent system (planner+debugger) suffice, or are the current modular teams truly necessary for observed performance? This helps validate the necessity and impact of the multi-agent structure versus simpler alternatives.\n\n- Could you provide more explicit detail on the communication and coordination protocols among agents (Supervisor, Planning, Reflection, Evaluator, Debugger, etc.), especially in cases of conflicting suggestions or persistent failures? Understanding the orchestration mechanics and how conflicts or deadlocks are resolved is crucial for reproducibility and for assessing the practical reliability and scalability of the system.\n\n- What is the impact of key hyperparameters (e.g., T_max, agent model choice, number of local updates, communication rounds) on system performance, and is there an automated or principled tuning mechanism in Helmsman? The methodology relies on several non-trivial, possibly task-sensitive settings; transparency around their selection, sensitivity, and potential for automation is necessary for adoption and rigorous evaluation.\n\n- Could you clarify the computational resources (runtime, memory, LLM API usage) required for typical benchmark runs, and to what extent does Helmsman facilitate fully reproducible experiments on AgentFL-Bench? Resource efficiency and reproducibility are essential aspects for practitioners and for fair comparison to baselines; disparities here might hinder wider use or interpretation of results.\n\n- Could you clarify whether the reported results (in Tables 2–5) are averaged over multiple runs, and if so, how many runs and what the variance or confidence intervals were? Assessing consistency and statistical significance is critical for experimental rigor in FL, where stochasticity can meaningfully affect outcomes.\n\n- Are the baseline methods (both standard and specialized) reimplemented under identical data splits, model architectures, and hyperparameter configurations as Helmsman, or are external numbers reported? Ensuring a fair and reproducible comparison is essential for drawing valid conclusions about Helmsman's relative performance.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Pioneers a principled, modular, multi-agent approach to automating FL system synthesis, beyond monolithic or single-agent methods prevalent in prior works.\n- Empirically demonstrates that solutions are competitive with, or exceed, established baselines on a diverse benchmark.\n- The AgentFL-Bench is broad (16 tasks, 5 domains); task specifications and comparisons are thorough.\n- The system workflow, agent interactions, and evaluation setup are mostly well articulated.\n- The division between planning/coding/evaluation stages is clear and aids reproducibility.\n- Provides a tangible tool and benchmark that can catalyze further research on automated ML/FL system synthesis.", "weaknesses": "- Results lack statistical robustness details: variances or confidence intervals on reported metrics are not provided, which undermines confidence in the reliability of improvements.\n- The number of runs, error bars, or statistical significance testing are absent from main tables.\n- Details on whether all baselines were re-implemented and evaluated under identical splits, models, hyperparameters, etc. are scattered (mainly deferred to the appendix), making it hard to assess fairness of comparisons from the main text.\n- Key configuration details (data splits, client counts, etc.) are not summarized in the main paper.\n- Computational costs (runtime, LLM API usage, memory) for Helmsman, particularly relative to hand-crafted alternatives, are not sufficiently quantified in the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WT0zQZ9HMY", "reviewer": "Reviewer_8n78", "summary": "This paper introduces Helmsman, a multi-agent system for synthesizing a federated learning system through user specifications. It (1) involves human-in-the-loop planning (2), multi-agent code generation (3), and improvement from a simulation environment. The paper also introduces AgentFL-Bench to evaluate the system generation in FL. Experiments show improved performance from heuristic baselines.", "questions": "(1) Does human-in-the-loop indicate the system will fail without such a component?\n(2) Did you ever try single-agent baselines without such a complex design?\n(3) If you were to design a minimalist system, what are the key components that you would keep, and what are disposable?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1) This paper is presented clearly with its methodology and benchmark\n(2) According to my knowledge, applying multi-agent systems to federated learning is novel, and the problem of automated research experiments is interesting.\n\nDisclaimer: I don't have a federated learning background; I am from a more classic multi-agent RL community. This paper is a multi-agent application paper from our side. I invite other reviewers from the federated learning community to comment on the novelty and significance of federated learning.", "weaknesses": "(1) The takeaway of the paper is \"multi-agent helps federated system more compared to heuristics when carefully designed\". It is closer to a system design report.  In research papers, more ablation studies should demonstrate each design component for rigor.\n(2) The \"human-in-the-loop\" part of the method makes the experiments potentially highly manipulable and may cause unfairness when comparing the method with other fully automated methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J8NqVkZ4cl", "reviewer": "Reviewer_VyT8", "summary": "The authors introduce Helmsman, a multi-agent framework that aims to automate the design, implementation, and evaluation of federated learning systems. The architecture is made up of a planning agent that creates research plans from user-defined text prompts, coder–tester agent pairs that produce and verify modular code components (which include task, client, strategy, server), and an evaluation agent that runs and debugs the code in a sandboxed Flower simulation. The authors claim that this approach achieves autonomous system synthesis across 16 benchmark tasks (AgentFL-Bench, which the authors have created), reaching full automation on around 60% of them and outperforming standard FL baselines.", "questions": "The questions are all about the weaknesses, and so each weakness should be seen as a question about the paper.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 4, "contribution": 1, "strengths": "The system itself is well articulated and makes sense for the problem at hand. The pipeline itself demonstrates seemingly good software engineering. The benchmark itself is a useful addition as AgentFL-Bench gives a controlled setting for evaluating end-to-end FL code synthesis and could be useful for reproducibility studies. The paper is well written and documents the workflow, examples, and failure handling pretty clearly.", "weaknesses": "The architecture which is developed doesn't seem to have anything intrinsically to do with federated learning. The architecture, multi-agent planning, code generation, and testing is generic and could apply to any modular ML task . The FL focus seems to come about only from the chosen templates and evaluation tasks, not from algorithmic design. This raises the question of why the problem is framed as FL rather than general code-synthesis automation.\n\nBecause of this, it should be compared with other code-synthesis automation pipelines, which have been left out. The system is pretty similar in philosophy, and at times details to previous frameworks such as SWE-Agent, AutoGPT, Voyager, and CodeAct, all of which perform multi-agent planning, code writing, and iterative debugging. Helmsman introduces no new learning mechanism or reasoning algorithm beyond applying this template to FL code. The paper doesn't compare against or even reference these lines of work, making the claimed novelty really difficult to justify.\n\nThere's also a real circularity in the evaluation and fragility in the prompt design. All tasks in AgentFL-Bench follow the same structured Problem–Task–Framework schema that Helmsman is designed specifically to take. Because of this, the benchmark measures success within a self-aligned template rather than general robustness to unstructured or underspecified prompts. There is no test of whether the system works if prompts are paraphrased, incomplete, or out of schema, which severely limits claims of autonomy and generalization.\n\nThe results given of 62.5 % automation rate and 5–10 % improvements over FL baselines lack any statistical grounding. No confidence intervals, standard deviations, or number of runs are reported. It has to be presumed that this number is from a single run, with presumably hand-tuned prompts, so without checking for robustness from prompt design, these results are really not meaningful. In addition, because of the inherently stochastic nature of LLM generation, there must be some considerable variance in results which are never specified. Again, this makes the claimed improvement of a few percent impossible to judge.\nIn addition, the baselines (FedAvg, FedProx) measure learning accuracy, not system-synthesis performance, so these comparisons are not really comparable at all.\nThe empirical results should then be viewed as illustrative demonstrations, not evidence of reliable performance gains.\n\nIn addition to all of this, for around 40% of tasks, human verification was required. Given this, it is unclear what advantage Helmsman offers over ready-made tools such as GitHub Copilot or Cursor, which already provide interactive code completion, conversational debugging, and execution feedback. Without quantitative comparison to really simple baselines such as these, the added complexity of the multi-agent architecture is not justified.\n\nFinally, the paper has no ablation experiments testing sensitivity to prompt wording, number of agents, or LLM choice, nor anything focusing on the value of planning vs. coding agents. Again, because of this it is unclear which components, if any, are responsible for the reported successes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)