ClimateAgent: Multi-Agent Orchestration for Complex Climate Data Science Workflows
Authors: Chenyue Li
Hyeonjae Kim
Wen Deng
Mengxi Jin
HUANG Wen
Mengqian Lu
Binhang Yuan
Venue: Accepted by TMLR ()

Abstract: Climate science demands automated workflows to transform comprehensive questions into data-driven statements across massive, heterogeneous datasets. However, generic LLM agents and static scripting pipelines lack climate-specific context and flexibility, thus, perform poorly in practice. We present ClimateAgent, an autonomous multi-agent framework that orchestrates end-to-end climate data analytic workflows. ClimateAgent decomposes user questions into executable sub-tasks coordinated by an Orchestrate-Agent and a Plan-Agent; acquires data via specialized Data-Agents that dynamically introspect APIs to synthesize robust download scripts; and completes analysis and reporting with a Coding-Agent that generates Python code, visualizations, and a final report with a built-in self-correction loop. To enable systematic evaluation, we introduce Climate-Agent-Bench-85, a benchmark of 85 real-world tasks spanning atmospheric rivers, drought, extreme precipitation, heat waves, sea surface temperature, and tropical cyclones. On Climate-Agent-Bench-85, ClimateAgent achieves $100\%$ task completion and a report quality score of $8.32$, outperforming GitHub-Copilot ($6.27$) and a GPT-5 baseline ($3.26$). These results demonstrate that our multi-agent orchestration with dynamic API awareness and self-correcting execution substantially advances reliable, end-to-end automation for climate science analytic tasks. The source code of ClimateAgent is available
at https://github.com/Relaxed-System-Lab/ClimateAgent.

Published in Transactions on Machine Learning Research (04/2026)
ClimateAgent: Multi-Agent Orchestration for Complex Cli- mate Data Science Workflows
Chenyue Li
ckich@@ormect..st...
The Hong Kong University of Science and Technology
Hyeonjac Kim
hikimar@ccorneet.uu...
The Hong Kong Universiby of Science and Technology
Wen Deng
wadengon&@comneet.us..kk
The Hong Kong University of Science and Technology
Mengxi Jin
mjjnaaf@conmcct.st..hkk
The Hong Kong University of Science and Technology
Wen Huang
whhuangbp@commeet..uuk
The Hong Kong University of Science and Technology
Mengqian Lu
menggian..u@ust..k
The Hong Kong University of Science and Technology
Binhang Yuan
biyuan
The Hong Kong University of Science and Technology
Reviewed on OpenReview: https: opperevveeeerevveeee. net/ formff id= XLWvXNumGa
Abstract
Climate science demands automated workflows to transform comprehensive questions into data-driven statements across massive, heterogeneous datasets. However, generic L.M agents and static scripting pipelines lack climate-specific context and flexibility, thus, per- autonomous multi-agent frame- work that rachestrates end-to-end climate data analytic workflows. CLIMATEAGENT decom- poses user questions into executable sub-tasks coordinated by an ORCHESTRATE--GEEN and a PLAN-AGENT; acquires data via specialized DATA-AGENTS that dynamically in- trospect APIs to synthesize robust download scripts; and completes analysis and report- ing with a CODING-AGENT that generates Python code, visualizations, and final re- port with built-in self-correction loop. To enable systematic evaluation, we introduce GLIMATE-AGENT--ENCCH
form poorly in practice. We present CLIMATEAGENT,
benchmark of 85 real-world tasks spanning atmospheric rivers, drought, extreme precipitation, heat waves, surface temperature, and tropical cy- clones. On MLINAATE-AEENT-BENNCC and a report quality score of 8.32, outperforming GITHUD-COPILOT (6.27) and a GPT-5 baseline (3.26). These results demonstrate that our multi-agent orchestration with dynamic APT awareness and self-correcting exccution substantially advances reliable, end-to-end au- tomation for climate science analytic tasks. The source code of CLIMATEAGENT is available http:::///ggthuuub.oo/eeeeee www.eeeaweeedyttteeee-e-eaeee Laaestee-Lab/limmaataaaa
CLIMATEAGENT achieves 100% task completion
"Equal contribution.
i "Corresponding author.

---

Published in Transactions on Machine Learning Research (04/2026)
1 Introduction
The rapid pace of climate change and the growing severity of its impacts have created an urgent need to advance data-centric climate science, which could deliver timely insights for adaptation and policy (Deelman et al., 2009; Overpeck et al,, 2011 King et al., 2009). Automating the workflows in climate science can translate comprehensive analytical questions into executable pipelines, enabling rapid, reproducible analysis of complex environmental phenomena and supporting extreme-event forecasting, impact assessment, and adaptation planning. On the other hand, such analytic workflows need to process special climate datascts (e.g. datasets from Copernicus climate data store (CDS) and the European centre for medium--aange weather forecasts (ECMWF)) with high volume, heterogeneity, and complexity, where intelligent automation has become essential for processing and integrating these datascts at scale (Hersbach et al., 2020 Benestad et al. 2017 Buizza et al. 2018). In this paper, aim to explore how Al agents can perform compler dlmate-sccene analyticol tasks through carefully designed agentic orchestration.
Building an Al-driven climate agent is a compelling and crucial effort since the stakes of climate change demand faster and more intelligent ways to derive insights from the complex data. The accelerating pace of global change and the severity of its impacts suggest that climate scientists and policymakers urgently need timely, data-driven information for mitigation and adaptation decisions (Overpeck et al., 2009). At the same time, climate science workflows need to process massive and diverse datasets from multi-model simulations to satellite and in-situ observations and turning this data deluge into useful information has become a bottleneck for discovery and decision-making. A large language model (LLM) based Al agent can aulomate complex analytical workflows to address this bottleneck, dramatical accelerating analysis that would otherwise take a team of experts days or weeks. By translating high-level scientific questions into executable pipelines, such an agentic paradigm could enable rapid, reproducible analysis of complicated climate phenomena like extreme events, climate impacts, and future scenarios (Deelman et al., 2009 Overpeck et. al., 2011). In essence, an AT climate agent can serve integrates data from various sources on-the-fly and explores many potential hypotheses iteratively, which would augment human scientists' abilities, i.c., allowing them to focus on interpretation and strategy, and accelerate the cycle of the analytic workflow (King el al.; 2009 Reichstein et. al. 2019). By empowering more interactive and comprehensive data exploration, such an agent could usher in a new paradigm for climate science, where insights emerge at the pace of computational power rather than human labor.
al.) 2011 King
productive research assistant that
On the other hand, developing a robust climate agent is challenging due to several inherent complexities of this domain, where data volume and heterogeneity are primary obstacles: modern climate datasets are enormous in size and varied in format. For example, single state-of-the-art reanalysis (ERA5) encompasses petabytes of multidimensional data (Hersbach et al. 2020), and observational records, climate model outputs, and remote sensing products each come with different resolutions, units, and conventions (Benestad et al., 2017]. A simple, hand-crafted pipeline or off- off-the-shelf use of LLMs (c.g., standard zero-shot LLM approaches) will quickly break down when faced with such diversity and complexity. Note that many climate workflows require on-the-fly decision making and expert knowledge for instance, selecting appropriate data sources, applying bias corrections, crafted solution cannot easily accommodate these muanced choices - attempts to force flexibility into static pipelines often lead to brittle, error-prone processes, highlighting the need for more intelligent, adaptive automation (Buizza et al. 2018).
choosing relevant statistical tests to determine significance. A rigid, hand-
In this paper, we view automating complex climate scientific workflows as a specialized form of planning and data-processing code generation. Recent L.LM-basod agentic systems show promising performance, but those applied to scientific domains still face critical shortcomings (Austin et al. 2021). Most current agents use generic, -anniii LLMs that miss the specialized requirements of climate science (Wang et all, 2024 Yao et al.) 2022). Concretely, general LLM agent could be unaware of climate data access APIs, data formats, and valid parameter choices, while static scripts and librarics lack the fiexibility to handle new scientific questions or evolving data sources. These limitations result in high error rales for LLM-generated code and a steep learning curve for non-expert users. More importantly, generic approaches cannot support the iterative, hypothesis-driven nature of climate study they fail to autonomously handle multi-stop inquiries where initial results could spur new sub-questions leaving a substantial gap in achieving truly end-to-end, autonomous climate analysis without requiring human intervention.
2

---

Published in Transactions on Machine Learning Research (04/2026)
To address these gaps, we introduce CLIMATEAGENT, autonomous multi-agent framework specially de- signed for climate science workflows. Our approach employs a multi-agent orchestration strategy: a com- prehensive climate query is first decomposed into structured sequence of sub-tasks, where each sub-task is executed by a specialized agent. This modular design injects domain-spocific knowledge at every step and dynamically adapls as the workflow progresses. Additionally, CLIMATEAGENT also equipped with robust error handling and self-correction, enabling to detect failures, recover, and adjust plans without human intervention in the loop. Concretely, we make the following contributions:
Contribution 1. We propose multi-agent orchestration paradigm to support complex climate data science workflows, which comprises the following key agents:
Agents for planning and orchestration: We introduce an ORCHESTRATE--GENT for workflow man- agement and & PLAN-AGENT for task decomposition. Concretely, the ORCHESTRAATE-AGGN manages experiment directories and persistent context, while the PLAN-AGENT interprets the user request, for- mulates a detailed execution plan, breaks down the high-level goal into discrete subtasks, and delegates each to the appropriate specialist agent. Together, these two agents provide top-level oversight, adjusting the plan as needed and ensuring the overall workflow stays track, including handling runtime issues or re-planning when necessary.
Agents for climate data acquisition: We implement a set of DATA-AGENTS each tailored to a specific data source and handles data retrieval by dynamically introspecting its target API e.g., fetching the latest valid parameters dataset metadata at runtime and then generating robust download scripts. This capability allows users to adapt to evolving datasets and API utilization, preventing errors such as invalid parameter use or format mismatches during data acquisition.
Agents for programming and visualization: We include a set CODING-AGENT that generate Python code for data analysis and generate final reports, complete with text summarics and visualizations. The MODING-AGENTY implement self-correction loop to debug code based on execution feedback.
Together, these agents form a cohesive system that can autonomously manage the entire climate data science life-cycle from data acquisition to final analysis, making sophisticated climate science accessible and efficient.
Contribution 2. To systematically evaluate the performance over the climate data science tasks, we in- troduce AAEAATE-AGETT--E AGENT-BENCH--8 BENCH-85, a benchmark of 85 real-world climate workflow tasks spanning six domains, including atmospheric rivers (AR), drought (DR), extreme precipitation (EP), heat waves (HW), sea surface temperature (SST), and tropical cyclones (TC). Each task is specified in natural language with explicit scientific objective, required datasets, mandatory external tools (e.g., TempestExtremes) and strict output contracts (filenames and formats), driving multi-step pipelines from data acquisition through processing, analysis, and visualization. For reproducibility, every task includes a curated reference solution with a validated Python code base and a human-readable report with figures, and lasks are stratified by difficulty - easy (single-source), medium (mmllti-source), and hard (external tool integration with dynamic parameters) support controlled comparisons across complexity levels.
Contribution 3. We comprehensively evaluate the proposed multi-agent framework, CLIMATEAGENT, on where the experimental results indicate that CLIMATEAGENT demonstrates
CLIMATE ACENT--ENNCE the ability to autonomously plan and execute the entire workflow. To be specifie, our system achieves a 100% success rate in generating the report across the benchmark, with an overall report quality score of and 3.26 for the GPT-5 baseline. We believe CLIMATEAGENT
8.32, compared to 6.27 for SUTHUB-COPILAN significantly reduces the manual effort and specialized expertise required, demonstrating a powerful new paradigm for Al-driven climate data science discovery and advancing the tfeee-of-the-art in automated climate research.
2 Related Work
LLM-Basod Agentic Systems and Scientific Automation. Large language models such as GPT-3 [Brwnn et al.) 2020), Llama 2 (Tourron et al.) 2023), and PaLM (Chowdhery et al.] (023) have become the backbone for agentic systems that interpret user instructions, plan tasks, and interact with external APIs
3

---

Published in Transactions on Machine Learning Research (04/2026)
Yao et al.
2022 Shinn et al. 2023 Frameworks like AutoGPT (Richards) 2023
LangChain Chase 2023
and CrewAT [Crew AT 2023) enable multi-step workflows and collaborative agents, while recent advances have extended these systems with memory, tool usage, and inter-agent collaboration capabilities (Chen et al. 2024 Park ct al. 2023 In scientific domains, systems such as ChemCrow (Bran et al. 2024) and SciFact (Wadden et al. 2020 remonstrate automated protocol generation ion and fact checking. However, these approaches are generally evaluated in synthetic domains or require extensive human oversight. What remains missing is system that combines domain-specifie knowledge integration, persistent workflow state management, and robust error recovery for end-to-end scientific analysis with live, evolving data sources.
Climate Science Workflow Automation. Workflow automation in climate science has evolved from generic orchestrators like Kepler (Altintas et al. 2004) and Apache Airllow to domain-adapted engines such as Cylc (Oliver et al., 2018) and ESMValTool (Righi et al.) 2020). Programmatic access to climate datasets is enabled by librarics like odsapi, comwf-api-client (Copernicus Climate Change Service (C3S) 2019 European Centre for Medium- Medium-Range Weather Forecasts (ECMWF) 2022), and processing interfaces
like CDO and xarray (Hoyer & Harmman| 2017). Recent specialized systems have begun targeting aulonomous climate analysis: ClimSim-Onling (Srichar et al. 2023) explored goal-driven automation for climate impact modeling, while TorchClim (Fuchs et al. 2024) and EarthML (Project) 2020) focus on doep learning-powerod powered workflows. However, existing systems either require manual intervention at each workflow stage or lack the robustness to handle the dynamic nature of climate APTs and heterogeneous data sources, preventing truly autonomous sud-to-end analysis.
Autonomous Research and Multi-Agent Planning. Advanced LLM architectures incorporating meta- prompting (Zhang et al. 2025c), chain-of-thought reasoning (Wei et al. 2022), , and self-correction feedback loops Kamoi et al. 2024] have improved task decomposition and reliability in autonomous research sy8- tems. Prompt-based approaches like PromptCast (Xue & Salim 2022) and LLMTime (Gruver et al. 2023 demonstrate zero-shot capabilities in time series analysis, while automated debugging agents (Gao et al. 2024) enhance code reliability. However, the gap between high-lovel planning capabilities and exccution-level robustness remains unaddressed: few systems successfully translate broad user goals into complex mulli-agent plans with rrcor-eecoeeaabee execution for scientific APIs, particularly in climate science.
To address these gaps, i.c., domain-agnostic agents, brittle workflow exccution, and lack of comprehensive evaluation benchmarks, we introduce CLIMATEAGENT, a specialized multi-agent framework that embeds climate expertise at every workflow stage while maintaining the flexibility to recover from inevitable failures.
3 ClimateAgent
Climate science workflows present a fundamental challenge for automation: rescarchers must coordinate mul- tiple interdependent tasks across heterogeneous data sources, each requiring specialized domain knowledge. A typical workflow begins with identifying and acquiring data through diverse APIs reanalysis products, forecast models, observational datasets then proceeds through multi-dimensional data processing, statis- tical analysis, and finally report generation. Each phase demands expertise in domain-specific conventions: climate data APTs impose unique constraints on spatial and temporal queries, scientific computing libraries require precise parameter configurations, and analysis must respect the statistical properties of geophysical data. Existing LLM-based approaches cannot handle this complexity: single-agent systems lack the special- ized knowledge required at each stage, while monolithic code generation cannot recover from API violations, parameter errors, or dependency failures.
We observe that the structure of climate workflows naturally suggests a solution. Rather than attempting monolithic generation,
decompose complex analyses into specialized subtasks - mirroring how climate rescarchers organize their own work into data acquisition, processing, analysis, and reporting phases. By assigning each subtask to an expert agent with targeted domain knowledge, and coordinating execution through shared context that accumulates artifacts and enables iterative refinement, we leverage LLMs' code generation capabilities while building in the error recovery mechanisms that long scientific workflows require.
We present CLIMATEAGETT, an autonomous multi-agent framework that executes end-to-end climate analy- ses without human intervention. CLIMAteAGENT decomposes workflows into specialized agents that collabo-
4

---

Published in Transactions on Machine Learning Research (04/2026)
Final Report
(1) Manages context and assigns subtasks to agents
"Analyze and visualize a tropical cyclone 2023138N05151."
USER
ORCHESTRATE -AGENT
OCC
(3) Fetches and validates climate data (ERA5/CDS)
DATA-AGENT
PLAN-AGENT
(4) Generates and debugs analysis code
CODING-AGENT
(5) Creates scientific plots and figures
(2) Requests are decomposed into subtasks
Figure 1: Overview of the CLIMATEAGENT system archilecture. The workflow illustrates how user queries are decomposed and processed by specialized agents, with robust error recovery and context management, to produce comprehensive climate science reports.
rate through shared context and adaptive feodback loops. The ORCHESTRATE--AGEN coordinates exccution while the PLAN-AGENT decomposes queries into executable subtasks. DATA-AGENTS introspect climate APIs to generate validated download scripts, and AGENTS produce analysis code with built-in debugging and self-correction. To achieve truly autonomous scientific workflows, CLIMATEAGENT realizes three core capabilities that address the fundamental requirements of climate analysis automation:
Coordinated Task Planning. The PLAN-AGENT decomposes comprehensive climate queries into struc- tured subtasks and delegates each to specialized agents with targeted domain expertise, forming coop- erative division of labor that improves the completeness and quality of the resulting analysis and reports.
Contextual Coordination. Agents operate on shared, persistent context that enables cross-step communication and propagation of intermodiate results, dynamically adapting as the workflow progresses and maintaining methodological consistency across multi-stage analyses.
Adaptive Self-Correction. CLIMAtEAGENT incorporates built-in error detection and recovery mecha- nisms that allow agents to diagnose failures, revise plans, and adapt without human intervention, ensuring robustness under evolving data and API conditions.
In this section, we first formalize climate report generation sequential state transformation ($3.1) then describe how our lhyee-layer archilecture achieves Coordinated Task Planning ($3.2), how persistent con- text enables Contextual Coordination ($3.3), and how complementary error recovery strategies provide Adaptive Self- Correctiooinnn 3.4). Finally, we synthesize these mochanisms into a unified orchestration al-
gorithm 3.5
We also provided detailed CLAMATEAGET implementation reproducibility in Appendix
3.1 Problem Formulation
We formalize climate workflow execution as a sequential state transformation. Given a task T, the system produces a scientific report R as the final output of a multi-stage analytical workflow through coordinated multi-agent operations that maintain a persistent workflow context C, accumulating all generated artifacts:
C (task: T, plan: P, code: {c data {d results: {ry})=1, logs {4)/=t
The PLAN-AGENT decomposes T into an ordered sequence of subtasks P [s1, $2........ sn), each specifying the action, required data, and target agent. Specialized agents A execute subtasks via the transition:
5

---

Published in Transactions on Machine Learning Research (04/2026)
Exccuto(C--1,;;; 81; A&)
This design achioves workflow coherence through contert accumulation: each agent operates on the complete history of prior decisions and artifacts. The context serves as (1) an inter-agent communication protocol, (2) a checkpoint for workflow resumption, and (3) a provenance record for reproducibility.
3.2 Multi-Agents for Coordinated Task Planning
To realize the coordinated task planning capability, CLIMATEAGENT employs three-layer hierarchical archi- tecture (see Figure D that mirrors how climate scientists organize collaborative investigations. Specialized agents at each layer encode distinct domain expertise: the PLAN-AGENT and ORCHeSTRAN TRATE-AENT coor- dinate workflow execution, DATA-AGENTA handle data acquisition, and CODING- AGENNT perform analysis and synthesis.
Agents for Planning and Orchestration. ORCHESTTRATE--GRNT manages workflow execution (Step in Figure D by creating timestamped experiment directories, persisting context, and coordinating agent invocation. PLAN-AGENT decomposes queries into executable subtasks using clinate-domain reasoning: it recognizes standard analysis patterns (climatology + anomalies + extremes + report), dataset dependencies (reanalysis vs. forecasts), and temporal constraints (initialization dates, lead times, aggregation windows).
Agents for Climate Data Acquisition. DATA-AGENTS interface with heterogeneous climaaee data sources (Step 2) through specialized implementatious. The CDS variant uses cdsapi access the Copernicus Climate Data Store, validating variables, time ranges, and spatial domains. The ECMWF variant uses ecmwf-api-client to retrieve S2S forecasts, encoding knowledge of model origins (ECMWF, NCEP, JMA), parameter types (pressure-level, surface, daily-averaged), and forecast conventions.
Agents for Programming and Visualization. AGENNSSAGENTS generate Python code (Steps 3-5) for data processing and report visualization. They incorporate expertise in climate libraries (xarray, cartopy, cf-python), copair-appropriate statistical methods (anomaly calculations, rumning means, extreme value statistics), and scientific computing practices (vectorized operations, memory management, unit handling).
This three-layer architecture separates concerns that require distinct expertise: data acquisition demands API knowledge, processing requires computational skills, and reporting requires synthesis ability. Encoding domain expertise at each layer achieves modularity and reliability.
3.3 Contextual Coordination
Building upon the hierarchical architecture $3.2 CLIMATRAGENT achieves the second core capability, i.e. contextual coordination, by maintaining persistent and interpretable workflow memory shared across all agents. This shared context records plans, data artifacts, and exccution outputs, cnabling continuity, communication, and reproducibility throughout the workflow. Formally, the persistent workflow context C, serves as the central mechanism that connects agents, supports error recovery, and preserves the system state. Context functions as both communication protocol and state management system.
Context Evolution. In this procedure, each agent receives C,-1 and produces C, by appending new artifacts (scripts, outputs). This monotonic accumulation preserves all information across agent transitions. For example, CODING-AGENT accesses original data files, processed results, and prior code when generating visualizations.
The agents should then leverage context to generate coherent code. Concretely, CODING-AGENT examines C,-1 to discover available files, understand data structure, and main- tain consistent naming conventions. This prevents common errors: hard-coded paths, incorrect variable assumptions, and redundant computations.
Context-Driven Code Generation.
Serialization and Persistence. We serialize context to JSON after each subtask, enabling: (1) workflow resumption from the last completed subtask after interruption; (2) reproducibility by preserving complete

---

Published in Transactions on Machine Learning Research (04/2026)
decision history; (3) debugging without re-executing carlier stages. This persistent state transforms indepen- dent agents into coherent, traceable workflow engine.
3.4 Adaptive Ceff-Correctio
Building on the contextual coordination mechanisms in $3.3 CLIMATEAGENT realizes the third core capa- bility, ie., adaptive self-correction, which provides robustness and adaptability under real-world scientific constraints. Note that climate workflows could face unexpected exccution failures: APIs impose inconsis- tent parameter rules, data availability fluctuates, and generated code contains subtle errors in coordinate transformations, unit conversions, array indexing. Single mistakes cascade through entire workflows in baseline LLM systems.
CLIMATEAGENT employs three complementary error recovery strategies. Mulli-candidale generation ad- dresses API variability: DATA-AGENT generates m = 8 candidate download scripts with varying interpre- tations (spatial bounds, temporal aggregations, variable selections) and executes them sequentially until one succeeds. Iterative refinement handles runtime errors: CODING-AGENT retries up to Rmax times, incorporating diagnosties from previous failures, with up to 5 debug iterations per candidate. LLM-bassd semantic validation catches subtle correctness issues (wrong statistical tests, incorrect climatological periodss that produce scientifically invalid results without runtime crrors.
These strategies complement each other: multi-candidatee generation explores hard-to-predict parameter spaces, iterative refinement fixes implementation bugs using error feedback, and semantic validation ensures scientific correctness. Together, they enable reliable execution of complex workflows involving unpredictable APIs and intricate computations.
3.5 Orchestration Algorithm
Bringing together the capabilities of Coordinated Task Planning, Contextual Coordination, and Adaptive Self-Correction, A Algorithm synthesizes the three core capabilities into a unified orchestration process (Steps 1-5 in Figure D. This orchestration approach ensures that complex climate science workflows are executed reliably through: (1) content-driven coordination enabling agents to build on prior work, (2) multi-strategy error recovery handling diverse failure modes, (3) complete slate persislence supporting reproducibility and debugging, and (4) graceful degradation with clear failure reporting when tasks cannot be completed. These mechanisms collectively enable researchers focus on scientific questions rather than technical implemen- tation details.
4 Climate Agentic Workflow Benchmark
Another critical question for applying LLM agents for climate workflows is: how one systematically evaluale whether these design choices trunslate inlo reliable, high-quality scientific oulpuls? This requires a benchmark that captures the authentic complexity of climate research diverse phenomena, heterogeneous data sources, multi-step reasoning, and domain-specific correctness criteria.
Evaluating L.M--based systems for scientific workflows requires benchmarks reflecting authentic research complexity. Climate workflows demand domain-spocific API knowledge, multi-step rcasoning across data acquisition and analysis, integration with specialized tools, and scientifically interpretable outputs. Existing benchmarks like DataSciBench (Zhang et al., 2025a) and MASSW (Zhang et al 2025b) focus structured data science tasks but lack the domain complexity and real-world API integration of climate rescarch.
To address this gap, we introduce LLMATE-AGENT--EENE across six climate phenomena. Each task requires executable code that retrieves data from operational APIs, processes multi-dimensional datascts, performs domain-appropriate analyses, and generates publication- quality reports. The benchmark evaluates code generation, workflow planning, error recovery under API constraints, and scientifie correctness.
comprising 85 end-lo-end workflow tasks

---

Published in Transactions on Machine Learning Research (04/2026)
Algorithm 1: ClimateAAeet Orchestration Workflow
Input: Climate restarch lask T
Output: Scientific report R
Initialize experiment directory and context Co;
// Task decomposition phase
P THANN-AGENT.decomosss// Co.plan P
// Sequential subtask execution with error recovery for <- 1 to [P| do
// Current subtask
8, P(i]
// Route to appropriate agent
A& select_agent(s.lypp
retry etrr__ount (
false
// Retry loop with maximum attempts while retry count < Rmax and not success do
10
if A = DATA- then
// Strategy 1: Yulti-candidate generation for downl
11
S <- Aaageenraate_caaee BatpieeaannaddsssssCCC.C...
12
foreach S do
result +- execute(c, exp_ dir)
14
if success then
C, 1ottte__conttt((........... success +- true
16
break
18
// Strategy 2 3: Iterative refinement semantic validation
19
c - Aş ge.geeerate(ss, Ce-
// Semantic validation
20
is yalid <- LLM_ [AM__validate((,s,
// Runtime execution
21
result + )
22
if result. then
23
C. - uuppate_conteet(C....eeeee result))
24
success + true
25
else
eтror history.appenddreult. (rrrr) retry_count count f- count +
28
27
// Extract final report from coding agent output [epport report]
28 R Ce..essuts(fnal
29 return R
planning, persistent context, and robustness by translating them into measurable workflow tasks. The following sections describe how these capabilities are reflected in task design and evaluated through unified protocol.
This benchmark operationalizes the three capability dimensions defined in 3
4.1 Design Principles and Capability Mapping
To empirically evaluate the three system capabilities introduced in $ SLNAAIE-AGRRNIBBBNOHHSS is con- structed around three guiding principles that mirror the design goals of CLIMATEAGENT:
8

---

Published in Transactions on Machine Learning Research (04/2026)
Coordinated Task Planning. Tasks require multi-stage decomposition - from data retrieval to anal- ysis and visualization - so that effective workflow planning can be distinguished from ad-hoc single-step reasoning.
Contextual Coordination. Many tasks contain cross-step dependencies, such derived variables, climatological baselines, or intermediate files that must be reused downstream, testing whether agent can maintain and propagate state across iterative reasoning and execution.
Adaptive Celf-Correction Tasks involve real-world API and tool interactions (e.g., ECMWF API, TempestExtrenent under realistic parameter constraints, stressing the system's ability to detect, recover, and adapt to execution or formatting errors.
4.2 Task Domains and Scientific Coverage
CUMATE--GEENT-ENNCHI spans six climate phenomena representing diverse analysis patterns:
Atmospheric Rivers (AR, 15 tasks) require computing integrated vapor transport (IVT) from multi- level wind and humidity fields, applying physical thresholds, identifying spatial regions, and tracking temporal evolution. These tests involve vertical integration, spatial pattern recognition, and trajectory analysis with wraparound coordinates.
Drought (DR, 15 tasks) compute multi-timescale indices (SPI, soil moisture anomalics) requiring 30- year climatological baselines, statistical standardization handling seasonal cycles, and categorical severity visualization. These evaluate temporal aggregation, statistical edge cases (zero variance, missing data), and standard visualization conventions.
Extreme Precipitation (EP, 15 tasks) analyze precipitation extremes through percentile metrics, spatial extent, and multi-day event evolution. Workflows aggregate sub-daily data, identify threshold exceedance (25-250 mm/day), and visualize progression. These assess temporal conversions, unit man- agement, and multi-pancl figures.
Heat Waves (HW, 10 tasks) identify prolonged high-temperaturee events using multiple frameworks (absolute/ percentile (absolute/percentlle thresholds, duration criteria, wet-bulb temperature). These tests include multi- criteria detection, boolean logic, binary mask handling, and custom colormaps.
Sca Surface Temperature (SST, 15 tasks) examine patterns, anomalics, and trends, including ENSO indices and marine heat waves. Workflows compute cliratolooical anomalies, identify warm/cold events, and analyze spatial-temporal evolution. These require ocean-atmosphere process understanding, regional indices (Niño 3.4), and diverging color schemes.
Tropical Cyclones (TC, 15 tasks) use Tempest TempestEExremne Extremes (Ullrich al. 2021 with ERA5 and IB- TrACS data for detection, tracking, and intensity analysis. These require dynamic parameter configura- tion, command-line tool integration, trajectory stitching, and multi-source validation testing tool 1/O, subprocess management, and error recovery.
These domains cover atmospheric (AR, TC), hydrological (DR, EP), land surface (HW), and occanic (SST) processes, spanning diverse scales and methodologies. AR and TC tasks test planning through multi-slage pipelines; DR and SST test context through climatological dependencies; EP and HW test robustness through complex thresholding and conversions.
4.3 Task Construction Methodology
Expert-Driven Design. Three atmospheric science graduate students from our institution (co-authors of this work) with combined expertise spanning synoptie meteorology, climate dynamics, and computational climate science systematically designed all tasks. The design process followed iterative methodology:
Domain Selection: Experts identified six phenomena that (a) represent diverse spatial and temporal scales (from daily precipitation extremes to scasonal ENSO patterns), (b) require different data sources and processing pipelines, (c) reflect common research workllows operational and academic climate science, and (d) present varying computational and algorithmic challenges.
9

---

Published in Transactions on Machine Learning Research (04/2026)
Task Diversification: Within each domain, designed tasks to maximize coverage of analysis patterns, data sources (ERA5, S2S forecasts, OISST, IBT+ACS), spatial domains (regional to global), and temporal scales (daily to monthly). Tasks explicitly avoid redundancy each presents unique requirements in data handling, statistical methods, or visualization conventions.
Real-World Grounding:
All tasks are based on actual research workflows employed in published climate studies or operational forecasting, For example, AR detection follows algorithms from Pan & Lu (2019) drought analysis implements W.MO-standardized SPI methodology, and TC tracking uses established Tempest Extremes configurations. This grounding ensures tasks reflect authentic scientific practice rather than artificial benchmarks.
Specification Refinement: Initial task descriptions underwent multiple revision cycles to climinate ambiguity while preserving implementation flexibility. All three experts reviewed each specification to ensure clarity, scientific accuracy, and feasibility.
This expert-driven curation ensures that each task not only reflects real-world research practice, but also targets specific capability dimensions introduced $3 The following subsection further stratifies these tasks according to their expected planning depth, context dependence, and robustness requirements.
4.4 Task Complexity Stratification
We stratify tasks by workflow complexity determined by subtask steps, data sources, external tools, and algorithmic sophistication:
Easy Tasks (n=25, 30%) require single-source acquisition with straightforward processing one API call (BRA5/OISST), standard xarray operations, basic statistics, single-panel visualization. These tasks lest. fundamental capabilities: API parameter construction, coordinate handling, unil conversions, and basic plotting.
Medium Tasks (n=30, 35%) involve multi-source data integration or multi-step workflows dinating multiple API calls, handling data heterogeneity (mismatched grids, resolutions, conventions). multi-stage pipelines (compute derived variable + identify events - track evolution), multi- fig- ures. Medium tasks evaluate the system's ability to maintain workflow coherence across dependent steps, manage intermediate outputs, and ensure consistency.
coor-
Hard Tasks (n=30, 35%) require external tool integration with dynamic parameterization using Extpests CDO where parameters depend on runtime-discovered data characteristies, manag- ing tool I/O, coordinating heterogeneous sources (forecasts + observations + tool outputs), implementing sophisticated algorithms (trajectory stitching, multi-criteria detection). These tasks test tool integration, subprocess management, and error recovery.
This distribution reflects realistic rescarch workloads and aligns with capability dimensions: easy tasks assess planning, medium tasks require context maintenance, hard tasks challenge robustness.
4.5 Evaluation Protocol
To assess end-to-end system performance on evaluation protocol aligned with the scientific communication standards of the climate community. This framework centers on a unified report score, which quantifies overall output quality on a 1 10 scale across four dimensions critical for scientific commuiicttio:
we design a multi-dimensional
Readability: Clarity, logical flow, accurate use of scientific terminology, and accessibility to the target audience.
Scientific Rigor:
Adherence methodological standards, appropriate statistical analyses, uncertainty quantification, and validity of result interpretation.
Coverage of all task requirements, inclusion of relevant. contextual information, and delivery of actionable insights.
Completeness:
10

---

Published in Transactions on Machine Learning Research (04/2026)
Visual Quality: Relevance and clarity of figures, appropriate visualization methods, accurate labeling, and professional presentation.
Following established practices in recent evaluation literature (Hada et al. 2024 Li et al. 2025 Zheng
et al. 2023), we adopt an LLM-based judging framework for report quality assessment. This approach has demonstrated high correlation with expert human evaluation while enabling scalable and consistent scoring across large benchmarks. Our evaluator employs GPT-4o's multimodal capabilities to assess both textual content and embedded visualizations, comparing system outputs against expert-generated references. This evaluation protocol provides the foundation for the quantitative and qualitative analyses presented in $5
5 Experiments
Building upon the system capabilities defined in and the benchmark and evaluation framework established in $ we now empirically examine how well CLIMATRAGENT fulfills its design objectives. Our experiments aim to answer the following three core questions:
Q1. Coordinated Task Planning: Does collaborative division of workflow among specialized agents lead to higher-quality scientific reports and more complete end-to-end workflows compared with standard zero-shot LLM reasoning?
Q2. Contextual Coordination: Can the system maintain cross-step dependencies and methodological consistency throughout multi-stage scientific analyses, ensuring coherent reasoning from data acquisition to interpretation?
Q3. Adaptive Self- CorfCoore:tton: Can our error detection and recovery mechanisms enable the system to autonomously identify failures, revise its plans, and continue exccution without human intervention?
To evaluate these hypotheses, compare CLIMATEAGENT with strong bascline models on the CLIMATE- AGENI-BENCH-85 benchmark ($4), conducting quantitative, qualitative, and ablation-based analyses for each capability dimension. We first describe our experimental protocol and baseline systems (55.) then
present quantitative results on task planning (5.22). qualitative analysis on context coordination 5.3 and
ablation studies demonstrating adaptive self-correction ($5.4).
5.1
Experimental Setup
We evaluate CLIMATEAGENT on the AGEAAA-EEAE benchmark using the evaluation protocol
defined in $4.5
We evaluate CLIMATRAGENT against two strong baselines:
GPT-5 Baseline: A sophisticated baseline leveraging GPT-5's intrinsic reasoning capabilities with ex- ecution validation. This system employs best-of-N sampling (N=4) to generate multiple code candidates per lask, executes them in a sandboxed environment, and selects the first successful solution. While capable of multi-step reasoning within single generations, this baseline lacks structured workflow decom- position, domain-specifie knowledge intogration, and iterative refinement mechanisms.
GitHub Copilot Agent Mode: An agentic baseline utilizing GitHub Copilot's conversational capabil- ities for multi-turn task execution. This system leverages Copilot's advanced code generation expertise (powered by OpenAI Codex (Chen et al. 2021)) combined with its ability to maintain conversational context, execute code iteratively, and provide refinements based on execution feedback. While represent- ing state-of-tho-art general-purpose coding assistance, Copilot relies on broad programming knowledge without the domain-specific climate science expertise, structured workflow decomposition, or specialized agent coordination that characterizes our approach.
Both bascline systems utilize GPT-5 as the foundational LLM, ensuring our comparison isolates the effects of specialized agent. coordination versus advanced single-model reasoning with execution validation.
11

---

Published in Transactions on Machine Learning Research (04/2026)
Table 1: End-to-end report quality (Report Score) by domain.
Domain
GPT-5
Copilot
CLIMATEAGENT
Atmospheric River (AR)
3.05
6.78
7.32
Drought (DR)
7.87
6.87
8.57
Extreme Precipitation (FP)
0.62
5.58
8.43
Heatwave (HW)
3.98
8.30
9.15
Sea Surface Temperalure (SST)
4.28
8.10
8.88
Tropical Cyclone (TC)
0.00
2.65
7.85
All Tasks
3.26
6.27
8.32
Table 2: Overall Performance Summary on lasks 1-10 scale.
All scores are averaged across all
System
Readability
Scientific Rigor
Completeness
Visual Quality
Report Quality
CLIMATEAGENT
8.40
8.72
7.75
8.41
8.32
Baseline (GPT-5)
3.48
3.41
2.8
3.34
3.26
Baseline (Copilot)
6.68
6.89
5.62
5.87
6.27
5.2 Experimental Results about Task Planning
To answer Q1, we assess how coordinated task planning impacts end-to-end report generation on CLIMATE- ACENT-BENCI-85 using the setup in and $5.1 For each of the six climate domains, systems must execute the full data-to-report workflow under the evaluation protocol of $4.5) We compare CLIMAtEAGENt against the GPT-5 and Copilot baselines, which share the same underlying LLM but lacks explicit multi-agent decomposition and socian-specialized roles. Results are shown in Tables and 2
Under the evaluation protocol in 4.5 higher scores Readability, Scient Rigor, Completeness, Visual Quality, and overall Report Quality can all be interpreted as downstream consequences of more effective task planning: more coherent dccomposition and delegation should yield clearer narratives (Rcadability), more appropriate methodological choices (Scientific Rigor), fuller coverage of required steps (Completeness) and better targeted figures (Visual Quality).
Experiment Results and Discussions. We summarize the main results and discussion below:
Overall Impact of Coordinated Planning. Across all 85 tasks, CLIMATEAGENT achieves the highest average Report Quality (8.32 vs. 6.27 for Copilot and 3.26 for GPT-5; Table 2 indicating that explicit division of labor among specialized agents leads to substantially better end-lo-end reports than single- model reasoning with execution validation. These gains are consistent across all six domains (Table D with particularly large margins in complex, multi-stage settings such as Extreme Precipitation (8.43 vs. 5.58 vs. 0.62) and Tropical Cyclones (7.85 2.65 0.00), where baselines frequently fail to complete multi-step analyses.
Completeness and Scientific Rigor Planning Outcomes. Beyond overall Report Quality, CLI- MATEAGENT also substantially outperforms baselines Completeness (7.75 5.62 vs. 2.80) and Scientific Rigor (8.72 vs. 6.89 vs. 3.41). Interpreted through the lens of Q1, higher Completeness reflects that the PLAN-AGENT decomposes each task into concrete subgoals and assigns them to spe- cialized AGENTS and CODING-AGENT discussion that commonly oceur in the GPT-5 and Copilot basclincs. Higher Scientific Rigor indicates that coordinated planning encourages a more disciplined methodological choices selecting appropri- ate datasets, applying physically meaningful aggregations, and justifying parameter choices in way
reducing missing figures, truncated analyses, and omitted
12

---

Published in Transactions on Machine Learning Research (04/2026)
that aligns with expert expectations - rather than the ad- inconsistent methods often observed in single-model workflows.
Readability and Visual Quality as Planning Effects. Coordinated planning also improves how results are communicated. As shown in Table 2 CLIMATEAGENT attains higher Readability (8.40 vs, 6.68 vs. 3.48) and Visual Quality (8.41 vs. 5.87 vs. 3.34) than both basclincs. From the perspective of Q1, these gains arise because the PLAN-AGENT explicilly anticipates the narrative and visual require- ments of each task- - specifying which diagnostics, figures, and explanatory sections are needed - and delegates them to specialized agents. This lcads to reports with clearer structure, better-matched figures, and consistent labeling that directly support the planned analytical storyline. In contrast, single-model baselines often generate plots and text in a more opportunistic, step-by-step manner, so narrative flow, figure selection, and captioning drift away from the original task specification, lowering both readability and visual quality, despite the successful execution of some individual steps.
Summarized Answer to Q1. Taken together, the improvements in Report Quality, Completeness, Scien- tific Rigor, Readability, and Visual Quality demonstrate that coordinated task planning is a key determinant of end-to-end performance on CLIMATE-AGENT-BENNEE tributing responsibilities across specialized agents, CIJMATEAAGNT produces higher-quality scientific reports and more complete end-to-end workflows than strong GPT-5 based single-model baselines. These quanti- tative findings support Q1, confirming that collaborative division of labor among specialized agents yields systematic gains over advanced single-model reasoning with execution validation.
By explicitly decomposing workflows and dis-
5.3 Experimental Results about Context Coordination
To answer Q2, we complement the quantitative report scores with a qualitative analysis of end-to-end work- flows. Because each report is the outcome of a multi-step climate analysis pipeline, any miscoordination between stages (e.g., inconsistent data sources, mismatched parameter choices, or failed intermediate steps) directly degrades report quality or even prevents report generation altogether. While the scores in $5.2 show how well each system performs overall, they do not reveal why a system succeeds or fails. By qualita- tively examining generated reports and their underlying execution traces, we can assess whether each system maintains the necossary cross-step dependencies and methodological consistency that define contextual co- ordination. Figure how each system maintains (or fails to maintain) contextual coordination in practice.
2 presents representative outpuls across the six climate domains, which we use to analyze
Experiment Results and Discussions. We enumerate the key observations and discussion below:
Baseline Competence in Simpler Domains. For simpler domains such as Drought (DR) and Heat- wave (HW), both baselines generally produce reasonable figures and narratives: the underlying workflows involve fewer stages, simpler data choices, and more direct mappings from task description to code. In these settings, Copilot substantially outperforms GPT-5, reflecting the benefit of iterative code refinement and exccution foedback for correcting obvious bugs and filling missing steps. However, even in these easy domains, baseline reports sometimes exhibit mild inconsistencies in variable naming, axis labeling, temporal aggregation, foreshadowing the more severe coordination failures that emerge as workflow complexity increases. In contrast, CLIMATEAAGNT maintains stable mapping between task specifi- cation, data selection, and visual presentation, even in these simpler cases, indicating that contextual coordination mechanisms are active across the full task spectrum.
Monolithic Scripts. As workflows become more complex in Atmospheric River (AR) and Sea Surface Temperature (SST) tasks, baseline systems increasingly default to monolithic scripts with limited explicit context management. Qualitative inspection of exccution traces shows a recurring pattern: changes made late in the script to fix an exception or adjust a plot overwrite earlier logie without updating dependent steps, so the final code no longer matches the original analytical intent. This leads to misaligned prepro- cessing, inconsistent temporal or spatial domains across figures, and missing climatological references SST tasks. CLIMATEAGENT, by contrast, decomposes these workflows into explicit substeps with clearly defined inputs and outputs, ensuring that updates to one stage (e.g., data filtering or anomaly com-
13

---

Published in Transactions on Machine Learning Research (04/2026)
(a) Baseline (GPT-5) (b) Baseline (Copilot)
(c) Ours
(d)
Reference
Figure 2: Qualitative comparison of generated figures for representative tasks. Each row corresponds to
a climate task: (1) Drought (DR), (2) Sea Surface Temperature (SST), (3) Extreme Precipitation (EP),
(4-5) Tropical Cyclone (TC), (6) Atmospheric River (AR). Columns: (a) Baseline (GPT-5), (b) Baseline (Copilot), (c) Ours, (d) Golden answer.
putation) can be propagated to downstream analyses and visualizations, thereby preserving cross-step dependencies required for contextual coordination.
Untracked External Proccssos. demanding domains such as Tropical Cyclone (TC) and Extreme Precipitation (EP), baseline failures are often driven by how external tools and processes are orchestrated. When GPT-5 or Copilot spawns new processes (e.g., calling TempestFxtteenes for TC tracking), return codes and intermediate outputs are rarely tracked rigorously. Downstream code frequently proceeds if upstream stages had succeeded, attempting to read missing files, operate partially written datasets, plot fields that were never computed. This results in runtime crashes, emply nonsensical visualizations, and incomplete reports, especially in TC tasks that rely on multi-step external toolchains. In contrast, CLIMATEAGENT treats each external invocation as an explicit workflow step, recording return statuses, verifying that expected artifacts exist and are well-formed, and halting or repairing the pipeline when upstream tools fail, thereby maintaining coherent execution state across process boundaries.
Missing Intormediate Validation. Across all domains, these issues are compounded by a pervasive lack of systematic validation of intermediale results in the baselines. Figures are often generated without checking whether the underlying data satisfy task requirements (e.g., correct spatial subset, sufficient temporal coverage, or non-degenerate statistics), leading to qualitatively poor or even empty plots when earlier steps quietly fail or return trivial outputs. misconfigured data request or misaligned coordinate system at the beginning of the workflow thus cascade into misleading uninformative visualizations
14

---

Published in Transactions on Machine Learning Research (04/2026)
at the end, with mechanism to detect or correct the deviation. CLIMATEAGENT mitigates this cross- domain failure mode by embedding validation hooks throughout the workflow - checking the dataset metadata, asserting non-empty and physically plausible fields, and aligning coordinate systems before plotting - so that cach stage both consumes and produces well-validated context, reinforcing contextual coordination from data acquisition through to final report generation.
Summarized Answer to Q2. These findings confirm that Contextual Coordination (Q2) is the critical diffcrentiator for robust scientific workflows. The analysis reveals that bascline failures in complex domains are rarely due to local coding errors, but rather structural inability to maintain state across long horizons and process boundaries. CI.MATEAAGRNT overcomes this by enforcing explicit, validated handoffs between agents, effectively treating intermediate artifacts as contracts. This persistent state management ensures that downstream execution remains strictly conditioned on upstream results, preventing the context drift that causes single-model baselines to lose the analytical thread in multi-stage tasks.
5.4 Ablation and Case Studies for Adaptive Celf-Correction
We turn to answer Q3 the system's ability to autonomously identify failures, revise its plans, and continue exccution without human intervention. This section analyzes common bascline failure modes and demonstrates how CLIMATEAGENT achieves robustness through multi-layered detection, recovery, and adap- tive replanning mechanisms. To validate this capability, aggregate performance metrics are insufficient; we must instead isolate the specific mechanisms failure and recovery. Therefore, we adopt a two-pronged ap- proach: (1) an ablation analysis of the distribution of errors in baselines define the failure modes our system must overcome, and (2) a case study tracing a complex recovery loop to demonstrate the self-correction mech- anism in action. Beyond these reprosentative self-correction case studies, provide dedicated failure-caso analysis (with logs, artifacts, root causes, and mitigations) in Appendix and component-level ablation analysis in Appendix
Ablation Analysis. We systematically classified GPT-5 baseline errors across 35 failed tasks, identifying
six primary categories (Table 3 Data/Array Shape or Key Errors (26%), Data Request Errors (17%).
Syntax/Indentatho Errors (11%), Timeout Errors (11%), Type Errors (11%), and Miscellaneous (23%). These failures predominantly result in incomplete or absent report generation, demonstrating direct LLM code synthesis limitations without system-level safeguards.
Table 3: Summary of Error Categories and Their Counts
Error Category
Count
Pata/Array Shape or Key Error
9
Data Request Error
6
Syntax Doutax/Inderny Indentation Error
Timeout Error
4
Type Error
Miscellancous
Our system incorporates several architectural and prompt-based interventions that directly address the failure modes observed in the baseline. Figures 3 illustrate how specialized DATA-AGENT and CODING- AGENT components validate dataset metadata, enforce typing, and iteratively repair code before execution, climinating the majority of such errors. We summarize the koy findings bolow:
Data/Array Shape Key Error: As illustrated in Figure baseline approaches often fail due to incorrect assumptions about data structure mismatched array dimensions. Our system addresses these issues: DATA-AGENTS extract and validate dataset metadata before code generation, ensuring that only available variables and correctly shaped dimensions are used. Furthermore, CODING-AGENTS perform L.MM-based code validation to verify data access patterns before execution, systematically preventing such shape and key errors.
15

---

Published in Transactions on Machine Learning Research (04/2026)
Baseline GPT-5 Error Example
Our System Success
# Incorrect boolean mask shape ivt_selected = ivt [;, mask]
# Ensures mask shape matches data shape mask = (ivt threshold) # 10 selection
ivt_selected = ivt [maak]
Figure 3: Comparison of array indexing: the baseline code fails due to shape mismatch in boolcan indexing, while our system validates shapes and applies correct masking.
Baseline GPT-5 Error Example
Our System Success
# Incorrect date range format # in API request request =
# Validates date range for each month request = "year": str( "month": month, "day": days,
date"; f"req_ssarr//////rqq.oo/">
Figure 4: Comparison of ERA5 data request formatting: the baseline code fails due to an invalid date range string, while our system programmatically generates and validates correct request. paramcters, preventing API errors.
Baseline GPT-5 Error Example
Our System Success
# Syntan error: unmatched ' dtxt += f} (bearing)"
# No syntam indentation errors.
Figure 5: Comparison of syntax handling: the baseline code fails with a syntax error due an unmatched brace, while our system's coding agent ensures only syntactically valid code executed.
Baseline GPT-5 Error Example
Our System Success
# TypeError: not alt arguments # converted during string formatting äf ["[["""]].. ".son"]aapplal (on
# Convert LON to numersc # before modulo operation tooouooriicceeee (df_fddfffffaa filt [" df [LONN"] % 360
Figure 6: Comparison of longitude alignment: the baseline code fails when non-numeric values present, while system ensures type safety before applying arithmetic operations.
Data Request Error: As ratud in Figure 4 baseline approaches fail when interacting with ERA5 ECMWF APIs, often due to improperly formatted requests or insufficient parameter validation. Our DATA-AGENTS employ automated metadata extraction and validation routines before code generation, utilizing LLM-driven logic dynamically select valid variables, date ranges, and request options, refer- encing up-to-date dataset metadata to construct compliant APT calls.
Syntax/ Indentation Error: Figure highlights how baseline LLM -LM-generated code frequently encoun- syntax and indentation problems that halt exccution. Our RODING-AGENTR proactively check for such errors before running any code, using diagnostic feedback iteratively refine and correct the code, ensuring that only error-free scripts proceed to execution.
16

---

Published in Transactions on Machine Learning Research (04/2026)
Type Error: As demonstrated in Figure 6 baseline code often fails due to improper handling of data types. Our system integrates type validation and conversion directly into the workflow, with the CODING- AGENT automatically checking and enforcing correct data types guided by both prompt instructions and LLM-based code review - before any computation is performed.
Case Study. The tropical-cyclone task focusing on Typhoon Noru (SID: 2022264N17132) demonstrates the full value of our robustines mechanisms. This lask compares the observed historical track from the IBTrACS dataset against simulated track generated using ERA5 reanalysis data, producing meteorological map visualizing both tracks alongside summary report quantifying track differences and forecast accuracy.
The baseline GPT-5 code fails to complete the workflow, terminating with:
ERROR: Failed to select longest track. single positional indezer out-of-bounde
This failure is due to incomplete or improperly parsed track data.
In contrast, our system executes a robust, agent-based workflow:
Decomposes the TC analysis into 10 explicit subtasks, including: (1) reading and process- ing IBTrACS data, (2) determining ERA5 download parameters, (3) downloading ERA5 reanalysis data, (4) computing Tempest Extremes detection parameters, (5) running Netect (6) rumning StitchN- odes, (7) extracting the longest simulated track, (8) visualizing meteorological fields, (9) extracting central pressure, and (10) generating the final Markdown report.
PLAN-AGENT:
Utilizes metadata extraction and LLM-guided parameter selection to construct valid ERA5 APT requests. The agent automatically identifies the correct dataset treanalysis-era--ee sigaa---nnggeeeeeell and validates all required parameters (such as date ranges and arca) before making the API call.
DAEA-AGENT:
AGENT: Implements robust scripts for each downstream subtask, including dynamic parameter computation, file validation, and error handling. Each script checks for the existence and integrity of its outputs before passing control to the next step, ensuring that failures are caught early and reported with actionable diagnostics. Finally, the agent compiles the Markdown report, embedding the meteorological field plot and central pressure value.
Summarized Answer to Q3. These results substantiate Adaptive Self-Correction (Q3). The proper handling of errors, combined with the qualitative success in the Typhoon Noru case study, demonstrates that robustness in scientific agents camnot be achieved by LLM reasoning alone. Instead, it requires system hypothesis to be validated using metadata constraints,
architecture that treats code generation static analysis, and exccution feedback to autonomously correct the "hallucinations" that otherwise break long- horg-horizon scientific workflows.
6 Conclusion
We have introduced CLIMATEAGENT, an autonomous multi-agent system designed to orchestrate complex climate science workflows from high-level user prompts to comprehensive scientific reports. By leveraging a hierarchical architecture of specialized LLM-based agents, i.e., each responsible for planning, data acquisi- tion, analysis, and visualization, our system addresses the limitations of generic code-generation models and static scripting approaches. Extensive evaluation the strates that CLIMATEAGENT substantially outperforms advanced singlc-model basclines across all domains, particularly in lasks requiring multi-step reasoning, robust error handling, and domain-specific knowledge. Our results highlight the effectiveness of modular agent specialization, dynamic error recovery, and context- aware orchestration in enabling reliable, end-to-end automation of climate research workflows. This work advances the state-of ofetth--rrrtttoe in scientific workflow automation and paves the way for more accessible, effi- cient, and reprodueible climate science.
benchmark demon-
17

---

Published in Transactions on Machine Learning Research (04/2026)
Acknowledgments
This research is supported by the Hong Kong Research Grants Council's collaborative research funds (project no. C6032-21G), general research fund (project. nos. 16300424 & 1621 (82)) M. Lu also acknowledges the support by the Otto Poon Centre for Climate Resilience and Sustainability at HKUST. This work is part of the United Nations Educational Scientific and Cultural Organization (UNESCO) International Decade of Sciences for Sustainable Development (2024-33, and contribules to the Seamless Prediction and Services for Sustainable Natural and Built Environment (SEPRESS) Program (2025-30) 32), an initiative endorsed under this global framework.
References
Ilkay Altintas, Chad Berkley, Efrat Jacger, Matthew Jones, Bertram Ludascher, and Steve Mock. Kepler: An extensible system for design and execution of scientific workflows. Scientific and Stalisticnl Dalabase Management, pp. 423-424, 2004.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carric Cai, Michael Terry, et al. Program synthosis with large language models, 2021.
Rasmus Benestad, Kajsa. Parding. Andreas Dobler, and Abdelkader Mezghani. strategy to effectively make use of large volumes of climate data for climate change adaptation. Climate Services, 6:48 54, 2017.
Andres M. Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D. White, and Philippe Schwaller. Augmenting large language models with chemistry tools. Nature Machine Intelligence, 6(5))525-533) 2024. URL
doi: 10. (0008//42256-00--0000355
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, et al. Language models are few-shot learners. Advances in Newral Information Processing Systems, 33:1877-1901, 2020.
Roberto Buizza, Paul Poli, Michel Rixen, Magdalena Alonso- Balmasea, Michael Bosilovich, Stefan Brönni- mann, Gilbert Compo, Dick Dee, Franco Desiato, et al. Advancing global and regional reanalyses. Bulletin of the American Meteorological Society, E913) ES144, 2018.
CDS. Copernicus climate data store. https://cda cwwmate....acccccc..a cowernicus.....ooo eu/
Accessed: 2025-09-28.
Harrison Chase. Langchain: Building applications with large language models through composability. https: www..///llaagncaaaaaannnnnnnaaaa
2023. Accessed: 2025-08-31.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, et al. Evaluating large language models trained on code, 2021.
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi- Hsin Hung, Chen Qian, et al. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. In International Conference Representation Learning, volume 2024, pp. 20094-200 -00136, 2024.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, et al. Palm: scaling language modeling with pathways. J. Mach. Learn. Res., 24(1), 2023. ISSN 1532-4435.
Copernicus Climate Change Service (C3S). edsapi: A python library to retrieve data from the climate data 2019. Accessed: 2025-08-31.
store. http:////pppp.oooppojeeettt////t
Al. Crewai: Framework for multi-agent collaboration with llms. https://crewai.. .ev 2023. Accessed: 2025-08-31.
Ewa Deelman, Dennis Gannon, Matthew Shields, and Ian Taylor. Workflows and e-science: An overview of
workflow system features and capabilities. Future Generation Computer Systems, 25:528 540, 2009.
18

---

Published in Transactions on Machine Learning Research (04/2026)
ECMWF. European centre for medium-range weather forecasts. 2025-09-28.
https://www.ees ecmwf. int/
Accessed:
European Centre for Medium-Range Weather Forecasts (ECMWF). ecmwffapi-clint: Ecmwf data access python client, 2022. httpp:///itte http://w....c/ccceee/////tttaaaa
Accessed: 2025-08-31.
David Fuchs, Steven C. Sherwood, Abhnil Prasad, Kirill Trapeznkko,, and Jim Gimlett. TorchClim v1.0: deep-learning plugin for climate model physics. Geoscienlific model development, (7(14):5459-544 2024. doi: 10.5104/gmmmee 0000//mmdd17-55-2222222 URL
gmd- 5179-5559-220 2024
Dawei Gao, Zitao Zeng, Weirui Shi, Xuchen Liu, Wenjie Kuang, Xin Lan, Zhijian Chen, Zheng Zhang, Zhaoxuan Dong, Wenhao Zhang, Yuzhe Chen, Xian Li, Suncong Li, Chuang Yang, Junyuan Wang, Jinlin Li, Jiahui Qiu, Rui Hou, Hongxuan Chen, Jie Tang, and Hongming Zhang, AgentScope: A flexible yet robust multi-agent platform, 2024.
Nate Gruver, Mare Finzi, Shikai Qiu, and Andrew Gordon Wilson Large language models are zero-shot time series forecasters. In Advonces in Neural Information Processing Systems, volume 36, 2023.
Rishav Hada, Varun Gumma, Adrian de Wynter, Harshita Diddee, Mohamed Ahmed, Monojit Choudhury, Kalika Bali, and Sunayana Sitaram. Are large language model-based evaluators the solution to scaling up multilingual evaluation? In Yvette Graham and Matthew Purver (eds.), Findings of the Association for Computational Linguistics: EACL 2024, pp. 1051-1070, St. Julian's, ss, Malta, 2024. Association for Computational Linguistics. URL htttps::/aclaattollooo
owg//rg//////////./// 22224. findings-eacl.711
Hans Hersbach, Bill Bell, Paul Berristord, Shoji Hirahara, András Horányi, Joaquín Mluozz-Saaata Julien Nicolas, Carole Peubey, Raluca Radu, al. The eraō global reanalysis. Quarterly Journal of the Royal Meteorological Society, 146(730):1999) 2049, 2020.
Stephan Hoyer and Joseph J. Hamman. xarray: N-d labeled arrays and datasets in python. Journal of Open Research Software, 5:10, 2017.
Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. When can LLMs actually correct their own mistakes? A Critical Survey of Self-Correction of LLMs. Transactions of the Association for Computational Linguistics, 12:1417-1440, 2024. doi: 10.1162/tacl a 00713. URL attps://doi.org// 1162/tac1 a 00713
Ross King, Jem Rowland, Stephen Oliver, Michael Young, Wayne Aubrey, Emma Byrne, Maria Liakata, Magdalena Markham, Pinar Pir, et al. The aulomation of science. Science, 324:85 89, 2009.
Qintong Li, Leyang Cui, Lingpeng Kong, and Wei Bi. Exploring the reliability of large language models customized evaluators for diverse NLP lasks. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend AI-Khalifa, Barbara. Di Eugenio, and Steven Schockaert (eds.), Proceedings of the 31st Interna- tional Conference on Computational Linguistics, pp. 10325 10344, Abu Dhabi, UAE, 2025. Association for Computational Linguistics. URL
coling- main..a8nn
Hilary J Oliver, Matthew Shin, and Oliver Sanders. CYLC: a workflow engine for cycling systems. The Journal of Open Source Software, 3(27):737, 2018. doi: 10.2 10.21105/joss.00733..
Jonathan Overpeck, Gerald Meehl, Sandrine Bony, and David Easterling. Climate data challenges in the
21st century. Science, 331:700 702, 2011.
Mengxin Pan and Mengqian Lu. A novel atmospheric river identification algorithm. Water Resources Research, 5567):6069 6087, 2019. doi: 10. 1029 10.1029/2018 2208870024007. URL htttps://gguubb..o onlinelibrarr wiley.com www.o///////////////.cooooaae//e http://00000000/2211000000000000
Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael Bern- stein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST 223, New York, NY, USA, 2023. URL
Association for Computing Machinery. ISBN 9798440701320. doi: 10.1145/3586183.00 3606763.
19

---

Published in Transactions on Machine Learning Research (04/2026)
HoloViz Project. Earthml: Machine learning and analysis tools for earth science, 2020. URL https: //earthll.ooloiiz.coo www.oolovv..... org Accessed: 2025-08-31.
Markus Reichstein, Gustan Camps-Valle Valls, Bjorn Stevens, Martin Jung, Joachim Denzler, Nuno Carvalhais, and F Prabhat. Deep learning and process understanding for data-driven earth system science. Nature, 566(7743):195-200 204, 2019.
Thomas Richards. Auto-gpt: An autonomous gpt-4 experiment, 2023. URL https://github... com/ Sigpfffccmt-GGaartttaattttt-rrr Accessed: 2025-08-31.
Mattia Righi, Bouwe Andela, Veronika Eyring, Axel Lauer, Valeriu Predoi, Manuel Schlund, Javier Vegas- Regidor, Lisa Bock, Björn Brötz, et al. Earth System Model Evaluation Tool (ESMValTool) v2.0 technical overview. Geoscientific model development, 13(3):1179 1199, 2020. doi: 10.5194/gmd 13- 13- 1020-002000
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: lan- guage agents with verbal reinforcement learning. In Proceedings of the 37th International Conference Newral Information Processing Systems, Advances in Neural Information Processing Systems, Red Hook, NY, USA, 2023. Curran Associates Inc.
Akshay Sridhar, Zeyuan Huang, Weimin Chen, Alistar White, Tom Beucler, Trevor James Phillips, Stephan Rasp, Venkatramani Balaji, Elizabeth A. Barnes, Timothy A. O'Brien, Oliver Fuhrer, Pierre Gentine, and Christopher S. Bretherton. ClimSin-Online: A large multi-scale dataset and framework for hybrid physics-mass machine learning climate modeling, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bash- lykov, Souryya Batra, Prajjwal Bhargava, el al. Llama 2: Open foundation and fine-tuned chat models, 2023.
Paul A. Ullrich, Colin M. Zarzycki, Elizabeth E. McClenny, Marielle C. Pinheiro, Alyssa M. Stansfield, and Kevin A. Reed. Tempostextremes v2.1: a community framework for feature detection, tracking, and analysis in large datasets. Geosci. Model Dec., 14:5023-500 2021. doi: 10.5194/ /gmd- 14-5023-2022
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Han- nanch Hajishirzi. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. tional Linguistics, 2020. doi: 10.18653/v1 /ttp///1/2200mm/// wwwnpmmminnn....mmmmmmmmmmmmmmem 6609.
Association for Computa-
Lei Wang, Chen Ma, Xucyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chon, Jiakai Tang, Xu Chen, et al. survey on large language model based autonomous agents. Frontiers of Compuler Science, 18(6), 2024. doi: 111077/117004-024-0222
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. Chain-of-thoughu prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088.
Hao Xue and Flora. D. Salim. PromptCast: A new prompt- based learning paradigm for time series forecasting. IEEE Transoctions on Knowledge and Data Engineering, 36:6851 16666, 2022. doi: 10.1109/TKDE.200 3220067.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models, 2022.
Dan Zhang, Sining Zhoubian, Min Cai, Fengzu Li, Lckang Yang, Wei Wang, Tianjiao Dong, Ziniu Hu, Jie Tang, and Yisong Yue. Datascibench: An Ilm agent benchmark for data science. arXiv, 2502.13897, 2025a.
20

---

Published in Transactions on Machine Learning Research (04/2026)
Xingjian Zhang, Yutong Xie, Jin Huang, Jinge Ma, Zhaoying Pan, Qijia Liu, Ziyang Xiong, Tolga Er- gen, Dongsub Shim, Honglak Lee, and Qiaozhu Mei. Massw: A new dataset and benchmark tasks for ai-assisted scientific workflows. In Pindings of the Association for Computational Linguistics: NAACL 2025, pp. 2773,3994, Albuquerque, New Mexico, 2025b. Association for Computational Linguistics. ISBN 979-8-89176-195- 195-7. doi: 10.18653/ /2205// 22222....iiinnnss.. naach.... URL https://aclanthologyy org/2025. mandings-nacl.122)
Yifan Zhang, Yang Yuan, and Andrew Chi-Chih Yao. Meta prompting for ai systems, 2025c. URL https:
7/arxiv.. org/ http//bb////1111//
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
21

---

Published in Transactions on Machine Learning Research (04/2026)
A
Agent Details
A.1 Plen-Agent: Task Decomposition and Agent Assignment
The PLAN-AGEES AGENT responsible for decomposing high-level user tasks into a sequence of actionable subtasks, each assigned to specialized agent within the CLIMATEAGENT system. Leveraging large language models, the PLAN-AGENT generates detailed, chronologically ordered plans that preserve all scientific and technical roquirements from the original user prompt. Each subtask includes explicit instructions, file paths, paramcter values, and workflow conventions to ensure downstream agents can execute their roles unambiguously and reproducibly.
Prompt Engineering. The PLAN-ACENT constructs prompts that enumerate available agents (data download, programming, visualization), file system conventions, and dataset constraints. instructs the LLM retain all relevant details from the user task, specify agent assignments, and output the plan structured JSON list. This approach ensures that each subtask is both specific and actionable, minimizing ambiguity and error propagation.
Plan- Agen-Agent LLM Prompt
You are a planning agent for modular climate gbrecasting/reportii system. The system has four main agents:
Adsapi_downlow dogeettaddaadnttt downloads climate data only using the edsapi library (for Coper- nicus Data Store datasets).
data doteaaddwladdd.coo agent: downloads climate data only using ecmwf-api-client library (for ECMWF S2S dataset only).
processes, analyzes, and computes on climate data (but does not down- load data). should generate hots/graphs that will be used in the final report.
programming agent:
visualization, oneattt:: generates the final report in Markdown format, including all re- quested plots, visualizations, and human-frient interpretation. The final report generation should always be handled by the visualization_agee
File System Data Conventions
Only download agents may write files to data/.
CODING-ACENTS must write all outputs (processed data, intermediate results, analysis out- puts, figures, etc.) to code_output//
Visualization agents should read from rode_output/.
All user-provided data must be referenced under ./user_provided__aa. data// (relative to the task root).
When generating subtasks, you MUST preserve much detail as possible from the original user task prompt. For each subtask, explicitly include all relevant parameters, file paths, scientific logic, and requirements from the original prompt. Do not summarize or omit details. If the original prompt specifies variables, thresholds, file formats, coordinate conventions, or workflow steps, these MUST be included in the subtask description. The goal for each subtask be as actionable and unambiguous as possible for downstream agents.
Critical:
Planning Guidelines
Break down the main task into small, logical, and chronologically ordered steps.
For each subtask, specify the agent, the action, and all required details (parameters, files, scientific logic, etc.).
22

---

Published in Transactions on Machine Learning Research (04/2026)
Be specific and unambiguous: name required data sources, variables, parameters, and outputs preciscly.
a subtask involves generating a plot/graph. specify the type of plot and the data to used.
If subtask is for the visualization agent, make sure the description clearly states that it is for the final report and should include all required plots and interpretation.
Only include agent types relevant to the task (do not include unused agent types).
Example of Generated Plan. Below is an example of plan generated by the PLAN-AGENT for an atmospheric river (AR) detection workflow. Each subtask is assigned to the appropriate agent and includes a detailed description the required actions, parameters, and output conventions.
Plan-Agent Output Example
Subtasks:
1. Create configuration file and directory structure for AR detection workflow Agent: programming_agee _gent
Create code_ www.ottttocooo.g..op... py with constants and paths, and ensure all directories exist.
Write run_metadata, json code_oouopal httplouuuuttuuuuussssaotmt _re__re with the configuration used.
2. Download ERA5 pressure-level u, v for 2022-12-19 to 2022-12-25 at 00:00 UTC Agent: cdsapi_ dww.lood_ Use cdsapi to request. Soamallyis-err--ppssuueeeeeee data for specific humidity, u and v wind components, at specified pressure levels and times. Save output Bata/eraa_rawweeee Baapa___p_______________ p1___u____2222212222222222 1000000000000 grib.
Compute daily IVT on a 1.5 x 1.5 grid for cach date in _ATE_RANGE
Agent: progrumming agent
Script: code_ catpputccompute ivt.py. Load downloaded data, regrid 1.5 grid, compute IVT, and save daily NetCDF files with compression.
Load monthly IVT threshold and run AR detection per day (extract, rotate, paths, trajectories, metrics)
Agent: progrumming agent
Script: code lade__uuttt/deectttt arraar_ppaaaa wtrpppathwaysppp py. Load threshold, apply rotation, compute anomaly, extract paths, generate trajectories, and outputs in hierarchical date folders.
Weekly aggregation of daily pathway presence and create frequency map image Agent: progrumming agent
Script: code output/visualize frequency. py. Aggregate daily pathway files, compute weekly frequency, subset to US West Coast, NetCDF, and plot frequency map using Cartopy.
Generate final Markdown report with the exact reference text and embedded image
Agent: visualization. agent
Create code_ www.opttttttepttt....t md referencing the frequency map image and containing the required summary and bullet points.
23

---

Published in Transactions on Machine Learning Research (04/2026)
A.2 Dgta-Agents: Prompt Engineering and Metadata Extraction
The DATA-AGENTS in CLIMATEAGENT, namely ECMWF-AGENT and CDSAPI-AGENT, are responsible for generating robust Python scripts to download climate datasets from ECMWF S2S and Copernicus Data Store (CDS), respectively. These agents leverage large language models (LLMs) for code synthesis and employ automated metadata extraction using browser automation to ensure parameter validity and dataset availability.
Prompt Engineering. Both agents utilize carefully designed prompts to instruct the LLM in generating executable download scripts. The prompts include explicit requirements for code structure, error handling, output directory usage, and metadata integration.
ECMWF-Agent Prompt Example. We attach the agent prompt for ECMWF below:
ECMWF-Agent LLM Prompt
You are an expert in ECMWF S2S data download. Given the following task, parameter info, and metadata JSONs, write Python script that downloads the required data using the ecnwf-api-client library.
Use only the available options in the metadata JSONs (see file names and their content summaries below).
The purpose of these metadata JSONs to reduce errors in the API calls of the generated Python code.
General Requirements for the Code:
The code must be modular, well-structured, and include clear, descriptive comments explain- ing each step and function.
Follow Python best practices for readability, maintainability, and efficiency.
Use appropriate scientific/data libraries (e.g., numpy, pandas, matplotlib, xarray, etc.).
All necessary imports must be included at the top of the script.
All code should be directly executable, with all necessary fields and values filled in.
The script must be runnable as: python [code_name]_p py (with no arguments). Do not require or parse any command-line arguments.
All downloaded data files must be saved in the 'data' subdirectory of the current task folder. Do not save data files elsewhere.
Do NOT use 'experiments/uun provided data' for any downloaded data outputs. That directory is reserved for user-prowt proviprovided data only.
All relative paths should be constructed relative to the directory the code running. Don't use absolute paths.
For efficiency, always batch all required parameters and levels into single AP1 call using / /'-separated lists for 'param' and 'levelist'. Do NOT make separate retrieve call for each parameter or level unless required by the API. Only loop over forecast type (cf/pf) or origin if absolutely necessary.
IMPORTANT:
Available parameters (with codes): (AVAILABLE_PANE PARANES_AAAAAALEA.E TEXT]
24

---

Published in Transactions on Machine Learning Research (04/2026)
IMPORTANT:
Always use the correct 'origin' code for the requested model/database
Special Instructions for Data Downloading:
If the subtask involves data download, you must use the ecmuf-api-client library or the provided download tool.
Only include the required fields below in the API call (do not add any others, especially 'area'): ):
class: s2
dataset: s2s
date: <date range for (illiieemmmddd version date for hindcast>
expver: prod
levelist: <level range> (only for pl, omit for sfc)
levtype: <sfc|pl>> (if requesting both, call API separately for each)
model: glob
origin: <origin>> (e.g. anso, ecmf, kwbe)
param: <parameter> (if requesting multiple parameters, call API separately for each)
step: <step range> (use a ''-sspaaateed list of step values)
stream: <enfolenth>> (enfo: realtime, enfh: hindcast)
time: '00:00:00'
lype: <cflpf> (ef: control, pf: perturbed)
target: <target file name>>
hdate: <yyyy-mm-dd> (only for hindcast, specify a list of hindcast initialization dates)
number: <number of ensemble members>> (only for perturbed hindcast, use / /'-separated list for multiple members)
Do not include any other fields.
Guideline for Setting 'date', 'hdate', and 'step' Fields in Requests:
1. Real-Time Forecast Setting: Use the operational model version available on the date.
2. Hindcast (Reforocast) Setting: Use the most recent available model version date strictly before the requested date.
'step' field: For avely-av parameters, use hour ranges representing 24-hour periods; for instaallaneous//cccuuubaee parameters, use single time steps.
After Downloading the Data:
Create/Update a README.md file in the data directory, listing all downloaded files and their descriptions.
For GRIB files, use efgrib to extract and include metadata summaries in the README.
For other file types, provide appropriate previews or summarics.
Metadata JSONs (file name, description, and content preview): (meta_block)
Task description: (task_ description)
Return only the Python code, with all explanations and context provided through code comments. Do not include any narrative or markdown outside the code block.
25

---

Published in Transactions on Machine Learning Research (04/2026)
CDSAPI-Agent Prompt Example: We attach the agent prompt for CDSAPI below:
CDSAPI-Agent LLM Prompt
You are expert in CDS (Copernicus Data Store) data download. Given the following task, write Python script that downloads the required data using the cdsapi library.
The code must be modular, well-structured, and include clear, descriptive comments explain- ing each step and function.
Use only the required fields for the cdsapi call (see https///ehlcciimste.opppnoooos auppii how-to for reference).
All necessary imports must be included at the top of the script.
All code should be directly executable.
The script must be runnable as: python [code [code_name]_a py (with no arguments). Do not require or parse any command-line arguments.
All downloaded data files and the README.m must be saved in the directory: [DATA_DIR], which will be the current working directory when the script is run.
The script must use the current working directory getccu)))) a provided variable for all output paths.
After downloading, create or update a README.md file in the data directory, listing the files and a brief description of their contents.
If the dataset or variable is not available, the script should print a clear error message.
Note that the cdsapi. Client only supports the retrieve method.
Aiient
At the end of the script, print to stdout single line containing a JSON array of the absolute paths all files that were downloaded by the script. For example: print (json (ttmpum"/ppap//////adaa"" (""/path/to//illel" "/path/to/filee"])
Do not print anything else to stdout after this line.
Task description: (task_ desckpeescription
Metadata: (metadata_str}
Return only the Python code, with all explanations and context provided through code comments, Do not include any narrative or markdown outside the code block.
Metadata Extraction via Chrome/Seleniume To dynamically identify available datasets, variables, and valid parameter ranges, both agents use Selenium with Chrome in headless mode. The agent navigates to the relevant data portal, interacts with web forms, and parses metadata (c.g., from JavaScript objects or HTML elements). Extracted metadata saved as JSON and provided to the LLM as context for code generation, ensuring that only valid options are used in download requests.
Outputs.
The agents produce several outputs for cach download task:
Generated Python Script: A modular, well-commented script that downloads the requested data using validated parameters.
README.md: A summary file listing downloaded files and their descriptions.
Metadata JSON: record of available dataset options and parameters.
26

---

Published in Transactions on Machine Learning Research (04/2026)
Example: README.nd file
ERA5 Pressure-Tevel Data Raw Download
Dataset: sauuullyii-ers--pressue-e-eee
Variables:
specific hitii_ttm u component of wind, "__comoonent_oo of
wfin wind
Pressure levels (hPa): 1000, 925, 850, 700, 500, 300, 200
Time: 00:00 UTC
Date range: 2022-03-24 through 2022-03-30
Format: GRIB
File path:
grib
These data are native ERA5 pressure-level fields suitable for IVT computation.
By combining LLM-driven code generation with aulomated meladata extraction, the DATA- AGENTS reduce errors due invalid parameters and improve reprodueibility. This approach enables the system to adapt to evolving data portals and ensures that download scripts remain robust and up-to-date.
Discussion.
27

---

Published in Transactions on Machine Learning Research (04/2026)
A.3 Boding-Agent (Programming): Data Processing and Analysis
The CODING-AGENT (Programming) is responsible for generating Python code to perform analysis and processing subtasks within the ClimateAgent workflow. This agent leverages large language models (LLMs) to syuthesize modular, well-documented scripts for scientific data analysis and pest-processing, based on the main user task and specific subtask descriptions.
Prompt Engineering. The agent constructs detailed prompts for the LLM, specifying requirements such as code modularity, use of scientific libraries (e.g., numpy, pandas, xarray, matplotlib), and strict file sys- tem conventions. The prompt instructs the LLM to avoid data download operations (handled by dedicated agents), save all outputs in the code cutpuu/ directory, and update or create README. file describ- ing generated outputs. Debugging instructions and error messages included in the prompt when code regeneration is required.
Coding-Agent (Programming) LLM Prompt
You are an expert Python programmer and agent developer. You are programming for predicting and lorecasting atmospheric phenomena in climatology. Downloads from ECMWF datasets are already handled by the DATA-AGENTS, Do not write code to download climate data. General Requirements for the Code:
The code must be modular, well-structured, and include clear, descriptive comments cxplain- ing each step and function.
Follow Python best practices for readability, maintainability, and efficiency.
Use appropriate scientific/data librarics (c.g., numpy, pandas, matplotlib, xarray, etc.).
All necessary imports must be included at the top of the script.
All code should be directly executable.
All output files must. be saved in the code_output/ directory under the current task root.
Do not write any files to the data/ directory.
All user-preet aser-provided data must be loaded from the directory ..
For every output file generated, also create or update README. md file in the output directory describing the outputs.
Plotting tip: Always place your colorbar or legend outside the main plot area and use tight_layout constrained_ layout for spacing.
Write a Python script that executes the given subtask. At the end of the script, include a test script validate the generated output. Return only the Python code, with all explanations and context provided through code comments. Do not include any narrative or markdown outside the code block.
Main task:
Subtask: {subtask}
Previous subtasks and codes: (previous_coder
Directory structure: edir_treet
README.md summary: (readme_summll summary}
Workflow and Error Recovery. For cach subtask, the agent generates multiple candidate scripts, val- idates their syntax, and ranks them using LLM-based code review for correctness, robustness, and clarity. If all candidates fail, the agent enters debug loop, providing error messages and previous code to the
28

---

Published in Transactions on Machine Learning Research (04/2026)
LLM for iterative refinement. Successful code execution triggers automatic updates to the README. file, documenting outputs and code changes.
Output. The following is an example of a README. md file automatically generated by the CODING-AGENT (Programming) after completing a subtask for weekly sea surface temperature (SST) analysis. This file documents the produced figures and reports, describes the contents and projection details, and provides metadata for reproducibility and further analysis.
Example: README.md for Analysis Output
Weekly SST and SST Anomaly Map
This directory contains the high-resolution two-panel map showing:
Weekly mean Sea Surface Temperature (SST) for June 12-18, 1997
Weekly mean SST Anomalies for the same period
File:
weekly_sse cartt__nmall_yaee wwp.mmapppppp pag: Two-panel PNG figure [12x8 in, 300 DPI]
The map PlateCarree projection (central longitude 1155, extent 120 E90,,, =30 latitude), with bold black lines at the equator and 180 meridian, and coastlines /land shaded in gray.
Discussion. By automating code generation, validation, and debugging, the programming agent stream- lines scientific analysis and ensures reproducibility, Its design enforces strict conventions for output manage- ment and documentation, facilitating transparent and collaborative climate rescarch workflows.
29

---

Published in Transactions on Machine Learning Research (04/2026)
A.4 Coding-Agent (Visualization): Report Generation and Plotting
The AGENNG-AGENT (Visualization) automates the creation of scientific reports and visualizations the final step in the ClimateAgent workflow. This agent synthesizes Markdown documents, figures, and summary files by leveraging large language models to interpret analysis outputs and generate publication-quaiity content. It ensures that all results are saved in standardized directories and that each output is accompanied by a descriptive README, md file for reproducibility.
Prompt Engineering, The visualization agent constructs prompts that specify the required report struc- ture, output file conventions, and documentation standards. The prompt instructs the LLM
Generate all requested plots and Markdown content specified in the subtask.
Save all outputs (figures, CSVs, Markdown) in the code_output/ directory.
For every output file, create or update a README md in its directory describing the outputs.
Ensure the final report is a Markdown file named Final_report md in code_output/, containing all relevant analysis and images.
Load any user-provided data from the standardized directory experinenta/uuserproooaee _atooddddaa/..a
Return only the Python code, with all explanations and context provided through code comments, and extrancous markdown narrative.
Coding-Agent (Visualization) LLM Prompt
You are responsible for fully exccuting the following risualization/rrportt subtask. Generate all required plots, Markdown, and outputs as specified, and ensure all results are saved and documented as described below.
Requirements:
Assume code is run from the task root [[outputddrr]]
Save all outputs (figures, CSVs, Markdown) in code_outpine
For every output file, create or update a README. md in its directory describing the outputs.
The final output must be Markdown report named containing all relevant analysis and images.
in code citee_outut,,
All user-provided data must be loaded from axperimentas/usee_provddee dataadddaaaaaaaaaaaaa
Return only the Python code, with all explanations and context provided through code comments. Do NOT include any narrative, markdown, or code block markers outside the code.
Subtask to execute: {subtask}
Main Task: mhain_tasky Context: Directory structure, previous subtasks, previous codes, and key README. md summaries.
Workflow and Error Recovery. For each visualization subtask, the agent generates multiple candidate scripts, validates their execution, and iteratively refines code in response to errors. It gathers context from previous analysis outputs, directory structure, and documentation to ensure consistency and completeness in the final report.
Output Examples. Below are two examples of final_report. md files automatically generated by the AGENN (Visualization) after completing different visualization subtasks. These reports demonstrate the agent's versatility in synthesizing diverse scientific findings from multi-day precipitation events to
30

---

Published in Transactions on Machine Learning Research (04/2026)
tropical cyclone meteorological analyses - into publication-ready markdown documents with embedded figures, spatial analysis, and narrative summaries for transparent communication and reproducibility.
31

---

Published in Transactions on Machine Learning Research (04/2026)
Example: final report.md for Extreme Precipitation Event.
Extreme Precipitation Event in the Greater Bay Area (2023-09-05 to 2023-09-10)
Introduction
This report summarizes an exceptional precipitation event that affected the China Greater Bay Area between September 5 and 10, 2023. Hourly total precipitation (tp) data obtained from the ERA5 reanalysis via the Copernicus Climate Data Store and aggregated to daily totals (mm). Spatial analysis and visualization were performed on the nalive ERA5 grid, highlighting regions exceeding common thresholds (25, 50, 100, 250 mm) and tracking the evolution of rainfall cores. Multi-Pancl Procipitation Map
2023-09-05
2023 02-05
2023- 08
237M
23"4
23%
230
22'
2274
22TN
100
23.5'
30
Tiall Fres Iotattiiiinnnn (mme)
22.5%
23.5"
33.5%
21.5'
21.94
113.5% 1135 L130S HPME 1195
112 1125 1135%
11455
1125% 1126
L135E
1105
116.7
Figure Daily total precipitation in the Greater Bay Area from September to 10, 2023. Stepped color intervals (0, 1, 10, 25, 50, 100, 250 mm) illustrate rainfall intensity. Major cities (Guangzhou, Shenzhen, Hong Kong) marked for reference. Event Evolution Narrative
Onset (Sep 5-6): The event began on September 5 with scattered moderate showers (10-25 mm) over western Guangdong. By the 6th, a convergence axis intensificd over inland hills, producing localized cores up to 50 mm (sky blue to royal blue shading) northeast of Guangzhou.
Intensification (Sep 7-8): On September 7, rainfall expanded east with cores exceeding 100 mm over the Pearl River Delta. The 8th marked the peak growth phase: a broad swath of hot pink (100-250 mm) stretched from central Shenzhen northward, triggering fash-flood warnings along tributaries.
Peak (Sep 9): September 9 saw the maximum daily accumulation, with dark red (>250 mn) cells persisting near coastal hills. Flood impacts were reported in suburban Foshan and low-lying areas of Dongguan, where drainage systems were overwhelmed.
Weakening (Sep 10): By the final day, precipitation waned and shifted southward, contracting 25 50 mm bands around Hong Kong's northern New Territories. The event concluded with isolated showers and rapid clearing. Conclusion
This multi-day event demonstrated classic inland-to-coastal propagation of heavy rainfall under synoptic forcing, with peak intensities exceeding 250 mm in localized cores. The spatial shift of maxima and associated flooding underscores the importance high-resolution reanalysis for regional hazard assessment.
32

---

Published in Transactions on Machine Learning Research (04/2026)
Example: final report.md for Tropical Cyclone Event.
Tropical Cyclone Meteorological Fields (SID: 2022264N17132)
Tropical Cyclone Meteorological Fields
SID: 2022264N17132,
Time: 2022-09-26 09:00:00
18TH
16.5%M
12TH
112.5EE 114T 1:5.5E
118.'E 120'
10
This chart shows a low-pressure system, with a central pressure approximately 991.8969 hPa,, accompanied by a distinct counterclockwise rotating wind field. Such cyclone may bring strong winds and heavy rainfall, requiring close monitoring of its path and intensity changes to prevent potential impacts on coastal areas maritime activities.
B ClimateAgent Implementation & Reproducibility
This appendix clarifies the concrete software realization of ClimateAgent and provides reproducibility de- tails requested by reviewers. In our wewpoorttggeerrteeeeeeeeaar
implementation, ClimateAgent is dingle-process Python orchestration system that coordinates modular agent classes (planning, data acquisition, programming, visualization). The orchestrator executes fixed, plan-driven workflow and persistently stores run state (context. json), intermediate artifacts, and logs to a per-run workspace under nav-approach/ete experiments/
hew-approcc///rrrrimmttt/////
B.1 System Architecture and Entry Points
Single-process orchestrator. ClimateAgent is implemented a single Python process whose controller is OrehestratingAgent (agents/orchestratts py). For each run, the orchestrator (i) creates task workspace directory task_aYYkka YYYYUMDD HHMMSS under wwr-approccheeppriieeett
(or resumes existing onc), (ii) runs PlanningAgent to obtain structured subtask list, (iii) routes cach subtask to specialized agent implementation, (iv) executes generated scripts as subprocesses, (v) applies bounded per-sublask retries, and (vi) snapshots the persistent workflow state to context. json after planning and after each subtask.
Modular agents as executable components. Each specialized agent is implemented as Python class that inherits BaseAgent and returns uniform result. dictionary (e.g., status, code, code tte_ptth,, result.
33

---

Published in Transactions on Machine Learning Research (04/2026)
error). Agents are instantiated on-demand inside the orchestrator and invoked sequentially the workflow progresses.
minimal runnable entry point is provided via main. pa...py. accepts a natural- language task prompt. from a text file (-prompt) and supports resumption and ablations via -task dir, "start_from_ie" (O-based), and -no-context (disable passing prior context into LLM prompts).
Runnable interface.
B.2 Runtime Scheduling, Routing, and Retry Policy
Plan-driven scheduling. The orchestrator calls PlanningAgent to generate plan JSON list of subtasks, Each subtask is dictionary with minimal schema:
subtask: short Litle;
agent: routing label;
description: natural-language instruction for downstream execution.
A Agent routing. Routing is implemented fixed mapping from the agent string label to executable agent class:
data data_downl download agent + ECMNFDowmloadAgent
adsapi_domllaae _agent CDSAPIDomlloddgget
programming_agent LLMProgrammingAgeent
visualization + LLMWissullizationnggerr
Bounded retries at the subtask level. For each subtask, the orchestrator applies bounded retry loop (default max_ 3ax_retrial=3). Failures are detected by (i) non-zero subprocess return codes when executing generated code or (ii) agent-returned status="error", After exhansting retries, the orchestrator records a failure message context.jant jaon (Appendix B.4 and stops the workflow.
Execution of generated scripts. Generated code is executed by writing it to temporary script file cimpp_scriptppp py inside the task workspace and invoking subprocess run with:
working directory: the task workspace (cwd=task_dde _irk;
I/O capture: gapture_output-True (text=True);
timeout: 6000 seconds per script;
interpreter: invoked as "python" (resolved via the system PATH).
This design allows the orchestrator to capture stdout/stderr and feed error traces back into subsequent attempts.
B.3 What We Mean by "Agent": Components vs. Role Prompting
A reviewer concern is whether ClimateAgent is merely single LLM "rolc-playing" multiple roles. Our implementation is a modular agentic workflow coordinated by an orchestration layer:
Each "agent" is an executable software component (Python class) with role-specific prompt, an explicit I/O contract (the BaseAgent protocol), and dedicated procedures (e.g., metadata scraping, candidate generation, debugging loops).
The orchestrator invokes agents as soparate calls at different workflow stages. For the Programming_agert and aisualizationaagen context passing (directory tree, prior codes, README summaries) be disabled with -o-conteet for ablation.
34

---

Published in Transactions on Machine Learning Research (04/2026)
Persistence and "memory." We do not rely implicit conversational memory the primary state. Instead, persistence is implemented via:
a serialized workflow context file ponntxxtjssnn)
on-disk artifacts in the task workspace (downloaded data, generated code, figures, reports, logs).
This externalized state is inspectable, reproducible, and supports explicit resume/recovery (Appendix B.4
B.4 Persistent Workflow Context (context. json)
Top-level schema. In the currett aew-approact/exxeriieetts implementation, context. json contains four core keys plus one optional key:
main task: string;
subtasks: list of subtask dictionaries (subtask, agent, description);
codes: mapping from 0-basod subtask index (string) to a selected code file path (typically absolute);
results: mapping from 0-based subtask index (string) to captured stdout (success) failure message;
user_date lel_ddaa___umaaa summary (optional): dictionary produced by UserDataInspoottAgen. http://mmmpppctteeooiinnnnttoo// in- cluding paths to code_ cade__utput/userr cettput/user_dat__ssuiee casta_summary and joon.
The optional user faar__dtt___g _aamary is included for fresh runs where the orchestrator performs the user-data inspection step; it may be absent for runs started from a user-specified -task firsk_dir without prior context.
Update policy and failure recording. The orchestrator writes montext.json (i) after plan generation and (ii) after each subtask finishes (success or failure). Failures are recorded as strings in results [idx] (e.g. "Failed after attempts...") rather than in a dedicated errors field.
Resume mechanism. Resuming is supported by loading context json from an existing -task_diil dir. To continue from an intermediate point, the user specifies -start_ forttffrom___e from_idx idx (0-based); the orchestrator skips subtasks with indices lower than this value and continues execution from the specified subtask.
Representative (sanitized) snippet.
"main task": "Produce the AR 100ºH-140% ...", "subtasks":
"subtask": "Download $2S forecast data for humidity and winds",
"agent": dbwnloaddagent",,
"description": "Use the ECMWF S2S dataset 2022-03-24
]. "codes": "0": h.../experiments/taak. YYYYMDDD_ wWW.Dp_inns///ttt//oooooooooggg
}.
"results": {
"O": "Submitting retrieval request: (... 'origin': 'ecmf', ...3"
},
"user data summary":
35

---

jaon (with output_da _iir set to <task dir>/data by the Published in Transactions on Machine Learning Research (04/2026)
"markdown": " ".../expprimentss/aakk YYYN/DDDYDDDDDDO.
BIRN//////ode_uuurppptteerrddddd summary... .dd", "json": ..../experiments/taskk / wwwpttuuuuutuuuuuueeuuueuee data summary. json", "summary": +
}
B.5 Dynamic API Introspection in Data Acquisition Agents
Valid request parameters (variables, levels, lead times, -ataset-specific fields) are constrained and may evolve. Our data acquisition agents therefore include dynamic API introspection to ground request generation on runtime metadata, rather than relying on the LLM to guess parameters.
ECMWF S2S (EOWFDownlosdkgent)) The ECMWF S2S download pipeline proceods as follows:
Parameter extraction: an LLM parses the user instruction into structured JSON specifying (parameter, origin, type, mode, date). In the current code, parameter extraction uses a stronger model (gpt-4.1).
2. Runtime metadata extraction: the agent uses Selenium-driven scraping (via wwtpaa/eeettttteeeeeeeeeee dataa_oom.ooo.egggt/tttttCCCCC lettirraatteecnm_mttttttttee metadata.py) to extract valid options (e.g., steps, available levels, and available model-version dates for hindcasts) from the ECMWF dataset pages. Metadata are saved under the task's data/ directory (type)](modee)) (mode). json and, for hindcasts, {mode] (selecsed_ socteeeddmmmee__eeeee vereeeee_mddeeeeeeieeoeeeee pearsio_ddate]] json).
as JSON (e.g., meta_foriging __pa___aaaaa_________
catpaasdddtee___rrrttee (ewtoriiiiini_paaaa__ee_e__e (param) (type]__________________________
3. Multi-candidate script generation: conditioned on the extracted metadata, the agent generates
n candidate download scripts in one LLM call (default model string o4-mini).
4. Validation and execution: candidates are syntax-checked via compile(...) the orchestrator executes candidates sequentially and accepts the first script with returncode==0.
CDS (CDSAPIDomloadAgeent)) The CDS download pipeline is:
Dataset selection: an LLM selects
a datasel
name
from
a
curated
list
(agents/cds (agants/cds_datasetss.o det.ddaatttsjjee. json).
2. Runtime metadata extraction: the agent uses Selenium to open the dataset download page and parse the form configuration from NEXT DATA__ saving JSON metadata file under <output pontput_drr>/datee orchestrator..
3. Multi-candidate script generation: the current implementation generates 4 candidate scripts per subtask, conditioned on the metadata, and syntax-checks them with compile(...)
Execution:
the agent can execute valid candidates concurrently (threads + subprocess; 600-second timeout per candidate) and selects the first successful run. Candidate scripts are instructed to print JSON array of downloaded file paths the last stdout line for robust parsing and cleanup.
B.6 Execution Artifacts and Traces
Workspace artifacts. Each run creates a task workspace (specified by -task dir or aulo-generated under experiments/). The workspace contains:
context. json: persistent workflow state (Appendix B.4
data/: downloaded data, scraped metadata JSONs, and auto-generated download scripts;
36

---

Published in Transactions on Machine Learning Research (04/2026)
code/: generated amalyii/vvisualizatnn scripts;
code_ output/: analysis outputs and figures, user vara_ warraasuumarrrrrra json, and the final report (typically Simal_report.md)
log/: orchestrator and agent logs (e.g., orcheatrating. log, programming_
visualization.
LLM call traces. LLM call metadata and token usage are recorded in a single JSON file code_out tet_ooouuutttoeeeeeeee usa.e. json. It stores an events list (onc entry per API call, including agent name, subtask index, attempl counters, stage labels, model identifier, token counts, and duration) plus a summary section aggregating totals and breakdowns by agent/subtask/stagg.
B.7 Benchmark- Systtem Interface
ClimatAAgent is reusable system that accepts natural-language climate workflow prompts input (o.g.. data acquisition, analysis, visualization, and report generation). Benchmark tasks (e.g,, prompl collec- tions) are executed by invoking the same runner; evaluation is implemented as a separate harness under eval_ _aa__scrptss///e (c.g., Seluuatereporta_ py reads evaluation_pail json and compares system vs. refer- ence reports).
Example commands.
Fresh run:
python main. py --prompt geth/to/task_pronge txt
Resume from an existing workspace starting at subtask k:
python main. py --prompt path/to/task_pronn Bacill_ppppttttee txt --task dir path/to/task_ie tttp///ttttttttntoaaa _HMMSS --start_ festar_frrom_t k
Ablation run without passing context into LLM prompts:
python main.py py --prompt path/tootask_prrompt txt --no-context
B.8 Autonomy Assumptions and Practical Deployment Notes
Autonomy definition. By "end-to-end autonomy," " we mean that once task prompt is provided and the runtime environment is configured, the system proceeds without human intervention: it plans, downloads data, generates and executes code, produces visualizations, and writes report, using bounded retries and error-driven regeneration.
Required pre-configuration. Autonomous execution assumes:
LLM access credentials available via environment variables (the code reads OPENAI_API_KEI
CDS/ECMWF credentials configured locally (e.g., cdsapi and ecmvf-api-client credential files);
Selenium-capable browser tooling available for metadata scraping (headless Chrome via Sclenium WebDriver);
required Python packages installed and network access enabled.
37

---

Published in Transactions on Machine Learning Research (04/2026)
Failure handling and recovery.
Robustness is achieved by:
bounded per-subtask retries max_retrial);
multi-candidate generation for download/code/reppon steps (defaults: ECMWF = CDS n
programming/ visaaiiigiiiulliiiiinn 4), with syntax checks and error-driven regeneration;
runtime error capture via subprocess stdout/stderr, propagated subsequent attempts;
persistent state snapshots (context. json) enabling resume via -task dir and -start feeart_from__a _ix.
Security and isolation. The current implementation does not sandbox generated code beyond subpro- cess execution. Scripts run with the permissions of the local user environment, using the Python interpreter resolved from PATH. We recommend safer deployment, practices (e.g., containerization, least-privilege creden- tials, and network restrictions) for production use.
Model configuration. Default model identifiers are configured per agent in code: PlanningAgent, hiMProorammingAgente and L.Wiisualizatiiongeee default to gpt-5; download-code generation defaults to o4-mini; and ECMWF parameter extraction uses gpt-4.1. All recorded model identifiers and token usage statistics are savod in
usageu json for reproducibility.
C Executable Examples and Traces.
Task requirements "task_SST___ 1). This task analyzes monthly mean sca surface temperature (SST) and SST anomalies in the tropical Pacific during the mature phase of the 2002-2000 2003 EI Niño using NOAA OISST v2.1 (AVHRR-only) daily NetCDF files for
-0000001-00000--1111 The required workflow is: (i) download daily files into data/ (skip present; warn and continue on failures); (ii) process local files on the native OISST grid compute monthly means of and anom over 30%S-30% and E20-E-2909 (wrapping longitudes to 0-360" needed); (iii) save processed outputs as NetCDF; (iv) create publication-quality two-pancl map (SST and anomaly) using Cartopy PlateCarrce with central central_longite longi mongitude=-1555 1555, specified contour
land/ /on//oastlines, labeled gridlines, and bold equator and 180 meridian; (v) write 250-400 word report embedding the figure and documenting data source, methods, and time window.
Agentic decomposition and responsibilities. The orchestrator baabeestrating_geentt decomposes the task into four subtasks and delegates them to specialized agents:
S1 Download (programming_ _gentt): loop over 2003-01-01.2203--1--1, construct NOAA URLs, download into data/, skip existing, continue on failures.
S2 Process (programming_aggetee open dath:/distt--wirr--000000000 nc, select sur- face layer, wrap longitudes if needed, subset to the required domain, compute monthly mcans, write two NetCDF files.
S3 Plot (programming_aggee load monthly means, generate two-panel Cartopy figure with required styling, validate PNG output.
S4 Report : generate code cidd__outut/ffiall cttpat//iinalreprttttot embedding the PNG and update code code_ output/README.... md.
A lightweight fre-ftight step by user_date wwspect_ageenene records data summary artifact (not used by the OISST pipeline itself).
38

---

Published in Transactions on Machine Learning Research (04/2026)
Subtask
Agent
Arlifacts
Attempts, failures, and fix
S1
Down- programming agent 31 daily NetCDF files in data/
4 code candidates jected due to SyntaxError: unterminated string literal. The first executed download script timed out (120 s). A debug regeneration fixed this by adding explicit per-request time- outs and streaming downloads; it then com- pleted and validated the expected 31 files.
generated; 2 were
load
$2 Process
programming_ agent jan2003_sst _sst___sttttt.....
Succeeded on first execution. The pipeline wraps longitudes to 0 360 when needed, sub- sets to [-30, 30] latitude and [120, 290 longi- tude, computes the mean over time, writes out- puts, and validates variable/dimmentin pres- ence.
mean.nc,
jan2003 1e2000_anoo__onthi_e monthmyn
S3 Plot
programming_ jan2003_ss$
Initial plotting failed with TypeError: Input z must be 2D, not 3D (extra singleton di- mension). Two debug regenerations later, the fix was to equeeze fields and print dims /ssapess before contouring: the figure was then generaled and validated.
anom_ tet__tto_paneeeppp
the loaded
S4 Report
visualization, agent
final @inal_report.. md
Two attempts failed with FilellotFunnderor:: Output directory not found: code output. The final fix added
ensure directory() step and successfully wrote the report and updated the README.
Table 4: SST trace summary (sources under task_SST____ 100og//
Executable example (end-to-end).
cd task_SST_1
python3 Bttpe//aa______662_8_8_8_8__8 python3 code/subtask (76f29dB_candd) py
python3 code/subtask, python3 code/subtask, ccde//ubbaast,__6666111111222 debp___
Expected trace highlights (selected stdout lines saved in logs):
[INFO] Saved SST monthly mean to code_ atte__uutputtjan2033_____ http://aaaanno_ssssssooototyoooo [INFO] Saved anomaly monthly mean to code output//jnn2033_amg
Batp:000__nmm_aaaee hranano0oaanooomoohlllyyoooooem mean. ncc [DEBUC] SST dims: ('lat', 'lon'), shape: (240, 680)
[INFO] Figure saved to code_ aate__uuputjjaann000 sst__sstt_ssssoee ancm__amm_ttt
two two waano_tto_paaeele_______aa .ng
Trace overview (attempts, failures, fixes).
Selected failure traces (minimal excerpts). S1 (task_SST_L1_ wwwSSST//////gg///////oe pewgrramiinnaaentt...
generation/runtime issues
Syntax error: unterminated string literal timed out after 120 seconds Buubtaskk11c896079c0_ .py)
S3 dimensionality error (task SST Bttg__/11111//A/BBBttbb.gg. TypeError: Input must be 2D, not
S4 missing output directory (task Beask_STT_iie SSTT_///og/viuualzzaaiiio log):
39

---

Published in Transactions on Machine Learning Research (04/2026)
FilellotFounError::
Output directory not found: code_outping
Final artifacts.
The run produces the following deliverables under task_SSI, 1ea___8T__ccodd_utpppe
jan2003_ sat_sstt_moos cest_monthly_mee _me_n.nc nc (monthly mean absolute SST), jan2003_ Brrp:o00__aommmmoma_ wtt_aaaaoooootl__meeaaa
(monthly mean SST anomaly);
Jan2003_ _tto3_sst_a____ anom_ two potooppaeel.ppp.. (two-panel map; x000x3600, 300 DPI);
mnaalreport.oo md (250 400 word summary report embedding the figure).
Nonthly Anomaly (203 (023-20-15)
23%
11
32%
107C
162
1000
-20
Figure 7: Monthly mean SST (top) and SST anomaly (bottom) for January 2003 from NOAA OISST v2.1 (olH) averaged over 2003-01-01- 01-01-20033-1-3) and plotted over S, 200EE220015
Final result report (generated text). The figure(Figure 7 above presents a two-panel map of monthly mean sea surface temperature (SST) and SST anomalies for January 2003 over the tropical Pacific (30'S 30 E20" E),* centered on 2003-01-15. In the upper panel, absolute SST values range from about 18 C in the eastern basin to over C in the western warm pool. The highest temperatures (>28* C) form a continnous eastward extension along the equator, peaking near 160 W. Cooler waters (<24 C) occupy the eastern Pacific near South America and the subtropical gyres north and south the equator.
The lower panel shows SST anomalies relative to the 1971 2000 climatology. A pronounced El Niño signature emerges: positive anomalies up to + span from the dateline (~ 180 E) eastward to around 240 concentrated within + of the equator. The warm anomaly tongue extends approximately from 140 E to 260 Opposing negative anomalies -0.5 to CCC appear in the western margin the tropical Pacific and in higher-latitudee subtropical regions, indicating compensating cooling outside the equatorial zone. Overall, anomalies exceed +2 across broad central-eastemn band, confirming the mature phase of the 2002-20 2003 El Niño.
Data were sourced from NOAA OISST v2.1 AVHRR-only daily NetCDF files for January 31, 2003. Files were downloaded locally processed using xarray: surface fields sat and anom were combined and averaged over the month. Subsetting used native longitudes (0 (0-360") re-wrapped to 120 F-0 and latitudes -30 to +30%.
40

---

Published in Transactions on Machine Learning Research (04/2026)
Visualization employedd matplotlib and carlopy (PlateCarree projection with central Saatral_ooggituue=111355 using sequential colormap for SST C-32C, 190 intervals) and a diverging colormap for anomalies (levels -4"C to +4*C).
D Component-level Ablation Analysis
The reviewer requests component-level evidence that attributes performance gains to specific parts of CLI- MATEAGENT. Because CLIMATEAGENT is an end-to-end executable workflow, planning and data acquisition) are necessory for executability: removing them entirely would collapse the pipeline and does not yield a meaningful rumnable baseline. Therefore, in this section we provide (i) direct ablations where the system remains executable (e.g., Contextual Coordination on/off; Sec. D.2), and (ii) trace-based component attribution for Coordinated Task Planning (CTP), using concrete plans, artifact chains, and per-subtask attempt logs to isolate the role CTP plays in long-horizon completion and error localization.
components (notably
D.1 Ablation Analysis on Coordinated Task Planning (CTP)
What CTP does in our implementation. Coordinated Task Planning (CTP) is the orchestration mechanism that transforms a high-level scientific request into an ordered list of subtasks, assigns each subtask to sapability-maccher agent (download vs. programming vs. visualization), and enforces explicit artifact hand-offs filenames/patiin and expected formats) between subtasks. In system, the resulting subtask plan is persisted in context. json, and execution is driven by deterministic loop over these subtasks with per-subtask retries recorded in log/ hegg//rchsstrtinngg agenta.geentttae log (see Appendix B.4 B.6 for implementation details and traces).
Why we use trace-based attribution for CTP. CTP cannot be "turned off" without making the workflow non-executable (e.g., there is no downstream routing without subtask list). Instead of non- runnable ablation, we attribute CTP's contribution via obscrvable exccution cvidence:
Plan validity: whether the plan covers the essential scientific stages (data acquisition compu- lation visualizzation/reporiing in an expert-reasonable order.
Artifact-chain explicitness: whether intermediate outputs are explicitly named and reused (e.g..
GRIB/NetCDF intermediates feeding downstream steps), enabling reproducibility.
Failure localization: whether failures are contained to a single subtask and resolved via local
retries without discarding previously completed stages.
Capability boundary compliance: whether tasks that require separation of responsibilities (c.g.. generating a downloader vs. executing downloads into data/) are handled via explicit hand-offs.
Representative long tasks. To ground the above signals, we report three representative long-horizon tasks (6-7 subtasks) that require multi-stage coordination and produce complete artifact chains: pask_AR__ (AR delection weekly frequency map, US West Coast), task_AR, AR_14 (same workflow for East Asia; includes permission-awwre download separation), and task DR_6 (SPI-based drought severity mapping over CONUS). Table summarizes their subtask counts, total attempts (including retries), agent switches, and wall-clock times, derived from context. json and orchestrator logs.
Evidence from the traces. Across these tasks, CTP produces plans that (i) explicitly scparate data acquisition, computation, and reporting, and (ii) establish a concrete artifact chain that enables downstream stages to consume intermediate outputs deterministically. Moreover, when errors occur, the orchestrator retrics the failing subtask while presorving already-producet artifacts, demonstrating localized recovery rather than restarting the entire workflow, Below we summarize the CTP evidence for each representative task.
41

---

Published in Transactions on Machine Learning Research (04/2026)
Task
#Subtasks
#Attempts
Agent switches
Wall-clock
task AR_
6
6
27m34s
task_ AR_14 14
8
47m12s
task_DR_6
7
26m21s
Table CTP trace summary for three representative long-horizon tasks (from * #//oo//orchstrratngg..geee atgptttiinnngggnggg.......
#/context, json and
Example 1: task_AR_2 (AR detection + weekly frequency map; subtasks)
Plan validity and artifact chain. The plan explicitly separates setup/config, data acquisition, scientific computation, and reporting, with named hand-offs: GRIB downloads daily IVT NetCDEs daily AR presence/pathway outputs + weekly aggregation map PNG report. These artifacts written under a structured directory (code_outppl output/outputs, ar_fre/) and referenced by subsequent subtasks, supporting reproducibility and deterministic execution.
Failure localization. When a downstream stage fails, the orchestrator retries only that subtask (recorded in orchestrating betteraatinnnggett..oe 1og) and reuses previously generated intermediates rather than restarting the pipeline, illustrating the intended CTP behavior for long-horizon robustness.
Example 2: _ (AR detection + weekly frequency map; subtasks)
Capability boundary compliance. This task requires coordination-citical split: a programming agent generates downloader script, while dedicated download agent erecutes it to write into data/. CTP makes this hand-off explicit in the plan (separate subtasks with explicit artifacts), enabling policy-compliant, reproducible workflow in which the download script is checked into the workspace and the data-write step is isolated.
Failure localization. The orchestrator retries a failing scientific subtask without invalidating the completed download and intermediate products, illustrating localized recovery in multi-stage artifact chain.
Example 3: task_DR_ (SPI drought severity mapping over CONUS; subtasks)
Plan validity aligned with domain workflow. The drought pipeline decomposed into statistically meaningful stages (download + crop climatology split + monthly stats + SPI map report). This decomposition reduces the risk of silent methodological errors (e.g., mixing climatology and target windows) by making intermediate stages explicit. and inspectable.
Artifact-chain explicitness and localized retries. Intermediate products (cropped dataset, climatology statistics, SPI field) are saved with explicit filenames and consumed downstream. When failures occur, retries target a single stage, preserving earlier artifacts and preventing unnecessary recomputation.
Takeaway. These traces provide component-level attribution for CTP: it yields expert-reasonate decom- positions, explicit artifact chains, capability-aware routing, and localized recovery behavior in long-horizon climate workflows. This complements the direct on/off ablation for Contextual Coordination in Sec. D.2 which quantifies quality drops when cross-step context alignment is removed.
D.2 Ablation Analysis on Contextual Coordination
Contextual Coordination is the mechanism that maintains cross-step consistency by (i) propagating struc- tured task state (e.g,, key file paths, intermediate outpuls, and constraints) between subtasks and agents, and (ii) enforcing that downstream steps consume the actual artifacts produced upstream rather than re- generating or guessing them. This is particularly important for long-horizon workflows where subtle context drift (c.g., mismatched filenames, inconsistent regions/time windows, re-derived parameters) can silently degrade scientific correctness cause late-stage failures.
42

---

Published in Transactions on Machine Learning Research (04/2026)
With Contextual Coordination
w/o Contextual Coordination
Task ID Category
Read.
Sci.
Comp,
Vis.
Report
Read.
Sci.
Comp.
Vis.
Report
ARI
AR
6
5
6.5
5
3
4
4.5
DRI
DR
9
8.0
8
6
7.0
EP1
EP
8
9
7
9
8.25
7
8
6
6.5
HW1
HW
8
8.5
9
8.3
SSTT
SST
9
10
9
8
9.0
6
5
4
5.5
TC1
ТС
8
8.5
eTT
eTr
erT
err
err
Ablation on Contextual Coordination. Vis.=Visual Quality, ovport= report score. "err" indicates the run failed to produce valid end-to-end output.
Table 6:
Read.-Readabilit,, Sci.=Scientific Rigor,
Ablation setup. We compare two settings six representative tasks (one per domain):
With Contextual Coordination: the orchestrator passes forward the persisted workflow state and key artifact pointers, enabling agents reference previously produced outputs and maintain consistent assumptions across subtasks.
w/o Contextual Coordination: we remove this cross-step context propagation, forcing each subtask proceed with limited knowledge of prior artifacts and decisions, which increases the risk of drift and brittle interfaces.
Evaluation. We evaluate report quality using the same rubric dimensions as in the main paper: Readabil- ity, Scientific Rigor, Completeness, Visual Quality, and the aggregated Report Score. We additionally record whether the produces valid final output chain (required figure + report). Table reports results.
Results and attribution. Removing Contextual Coordination causes consistent degradations in report quality across domains, with particularly large drops on tasks that require careful cross-step reuse of inter- mediate scientific artifacts.
Large quality drops artifact-heavy workflows. SST1 shows substantial decline in overall Report Score (9.0 - 5.5), accompanied by sharp reductions in Scientific Rigor and Completeness. This is consistent with context drift causing downstream steps to mis-handle intermediate datasets, variable choices, or time /reginn scttings when prior docisions are not propagated.
Cross-step drift affects scientific correctness signals. In AR1 and EP1, the largest decreases occur in Scientific Rigor Completeness and Visual Quality, reflecting that downstream code and plots are more likely to deviate from upstream constraints (e.g., required thresholds, plotting levels, or spatial aggregation definitions) when the system cannot reliably reference the correct intermediate artifacts.
Hard failure in long-horizon pipelines. TC1 fails entirely ("err") without contextual coordina- tion, indicating that brittle multi-step dependencies (e.g., intermediate text rext/figures written in earlier subtasks) can break the final stage when artifact paths and working-directory assumptions are not consistently maintained.
Takeaway, This ablation provides direct evidence that Contextual Coordination is a key contributor to end-to-end robustness and report quality in CLIMATEAGENT. By maintaining consistent state and artifact references across agents and subtasks, reduces context drift, prevents late-stage contract breaks, and improves scientifie completeness and rigor in the generated reports.
43

---

Published in Transactions on Machine Learning Research (04/2026)
Table 7: Task stats and errors (short cols)
Name
Subtask
ProgRet
DataRet
VizRet
ReqEi
ShpKey
MiscErr
SyuErr
Timeout
TypeErr
task AR 4
task AR
6
0
0
0
0
0
0
task AR
5
task AR 4
7
8
0
0
0
0
3
0
task 5
5
3
0
0
2
4
task DR
4
task DR 2
4
1
0
7
0
0
7
1
0
task DR
6
task DR 4
4
1
11
1
0
O
task DR_
5
0
task EP
1
2
task EP
4
2
2
2
task EP
task EP
4
3
2
task EP
4
43
22
20
task HW
task HW 2
2
0
0
0
task IIW
2
task HW
2
0
0
task HW
2
task SST
4
00000000
2
2
task _Ss
4
10
000000000
3
2
task SST
4
task SST 4
4
1
0
0
0
task SST
4
6
1
5
D.3 Ablation Analysis on Adaptive Self-Correction
Adaptive Self- Correction (ASC) is the mechanism that improves CLIMATEAGENT robustness by iteratively recovering from execution-time failures. our implementation, ASC operates through bounded retry-and- regenerate loops: when a subtask fails, the responsible agent (programming/ htwpa//mmiiingaaaaasssaaaaaataaa revised candidates conditioned on observed error messages and re-executes until success or the retry bud- get is exhausted. This behavior is essential in long-horizon climate workflows where failures arise from heterogencous I/O backends, evolving APIs, and brittle codc/plotting details.
generates
Why traco-based attribution (instcad full removal). In an end-to-ond exccutable pipeline, ASC is tightly coupled with the orchestrator's runtime semantics: subtasks are sequentially dependent, and fail- ures are handled by retries and regeneration. Rather than introducing additional experimental variants in this revision, provide trace-based component attribution grounded in the errors that were actually encountered and the recovery events that were required for successful completion. Under the orchestrator's fail-stop behavior at the subtask level, any subtask that required at least one recovery would have terminated at its first failure without ASC, thereby preventing downstream subtasks (and thus the final report) from being produced.
Signals from exccution artifacts and logs. We summarize ASC behavior with two log-derivee signals:
Retry volume by stage: the number of retries required for each lask in the Programming, Data, and Visualization stages (ProgRet, DataRet, VizRet).
Recovered crror taxonomy: counts of observed error classes, including request/API er- rors (ReqErr), shape/key errors (ShpKey), miscellancous runtime errors (MiscErr). syntax errors (SynErr), timeouts (Timeout), and type errors (TypeErr).
44

---

Published in Transactions on Machine Learning Research (04/2026)
Table 7 reports these statistics aggregated from execution logs across representative tasks.
Findings. Two consistent patterns emergee from Table 7
(1) Recovery is frequently necessary for completion. Many tasks exhibit non-zero retry counts (e.g., ProgRet> 0 and/or VizRet> 0), indicating that the first generated candidate is often insufficient and that successful end-to-end runs depend on itcrative correction. Because subtasks are sequentially dependent, a single unrecovered failure would block all downstream stages and prevent producing the final figure and report.
(2) Failure modes are heterogencous and stage-specific. The recovered error taxonomy reveals that different stages fail in different ways:
Programming-staag failures (o.g. ShpKey, TypeErr, MiscErr) reflect common scientific- computing issues such as coordinate/dimmensio mismatches, variable naming assumptions, or xarray berkend/enginn mismatches.
Visualization-stage failures (often reflected in elevated VizRet and SynErr frequently stem from brittle artifact references (paths (iltis/fillmmmes) and plotting boilerplate issues; ASC enables iterative fixes without rerunning the entire workflow.
Data-stage failures (captured by DataRet and ReqErr) are driven by external service instability, request/cre crquest/crodential issues, or transient connoctivity problems; ASC mitigates these via retrics and regenerated download candidates.
Interpretation and linkage to concrete failures. These trace-based statistics attribute concrete role to ASC: it resolves diverse runtime failures that occur in practice and would otherwise terminale subtasks un- der fail-stop execution semantics. While final report scores reflect output quality, ASC primarily contributes by improving robustness and completion under realistic conditions (external APIs, heterogencous filc formats, and fragile artifact interfaces). We further connect these error classes to concrete examples with logs and intermediate artifacts in Appendix
Takeaway. Adaptive Self-Correction is key robustness component in CLIMATEAGENT, enabling recovery from syntactic, runtime, data 1/O, and external-service failures that common in long-horizon climate workflows and that would otherwise block end-to-end execution.
E
Dedicated analysis of failure cases
Given the complexity of the multi-agent workflow (data acquisition + scientific computation - visualiza- tion/ dlo//reporting) failures typically arise from (i) external data services, (ii) heterogencous file formats and coordinate conventions, (iii) runtime/resoures constraints, and (iv) brittle artifact contracts (paths/names) across agents. Below we analyze four representative cases of ClIMATEAGENT, including log evidence, inter- mediate artifacts, underlying causes, and mitigations done by agents.
E.1
Case 1: External data service failure (connection error during download)
Where it happens. Data acquisition stage (download agent), task_AR_1.
Symptom / impact. The workflow cannot proceed because the required forecast file is missing; the orchestrator retries.
Log evidence. The orchestrator records a connection error on the first attempt:
task_AN task_ARR//oog//orree htgp///oggoorchssttriiigg..ggg log:entt.ogg:: 18-20 Attempt for subtask
Subtask failed: Connection error.
Attempt 2 for subtask 1
45

---

Published in Transactions on Machine Learning Research (04/2026)
Intermediate artifacts.
Expected (after success): 2ataaaAA_/aaat/2222_2.1222222222
Download implementation: task AR Baiik_R__/daat//aee h/data//dwwlooaddd///aaoa/////// http:/oaddddsrpppptttteettteeet
Underlying cause. Reliance external APT (ECMWF Web APT) introduces transient network service failures; the system has control over availability.
Mitigation / resolution.
Retry with backoff and idempotency: retries should include exponential backoff and resume- safe downloads.
Preflight credential and endpoint checks: detect missing credentials or endpoint issues carly with lightweight request.
Cache-first behavior: if the expected file exists and is valid (non-empty, readable), skip re- download.
E.2
Case 2: Data format and coordinate heterogeneity (xarray engine + concat failures)
Where it happens. Scientific computation stage (programming agent), task_AR_ _______11 subtask 2.
Symptom/ / impact. Multiple candidates fail during dataset loading: (i) xarray open mfdataset cannot infer concatenation order for threshold files, and (fi) the forecast file cannot be opened using default NetCDF backends (requires GRIB/cfgrib).
Log evidence. Threshold concatenation failure:
task AR ha/lo//proggram/iee sess_agenn..lc ValueError: Could not find any dimension coordinates to use to order the datasets for concatenation
Forecast file backend/engee mismatch (default engines fail; efgrib succeeds):
task_AR_ Bat.AR///og//rrrramigg wwwpiiinnnggggg.neoo agent. 1og: 10364
Engine 'default' failed: did not find match in any of xarray's currently installed IO backends [...]
Engine 'netcdf4' failed: NetCDF: Unknown file format: s...s2s IAPCAS_ .ncc'
.nc
Opened S25 with efgrne='cfgrib'
Intermediate artifacts.
Forecast file: task_AR ////aaaaa22___APAA__002222233332
User thresholds directory:
http:///ww._.aaaapppeedeedd.__// thres/.
Candidate implementations: task AR hts_1///dd//uutaskk22 p.t.sstbbbbbaa__.p........p
Underlying cause. "Same extension, different reality": files named .nc may be NetCDF or GRIB-derived containers depending on the download pipeline and libraries available; additionally, multi-file threshold datasots may not share coordinates ncoded for boroiine=by_ccorree concatonation.
Mitigation / resolution.
Robust I/O strategy: attempt multiple engines; explicitly use engine="cfgrib" with indexpath="" when GRIB-like.
46

---

Published in Transactions on Machine Learning Research (04/2026)
Coordinate standardization: normalize time/lat/lon/levell names and ensure time is di- mension before slicing.
Threshold ingestion that does not assume concat coords: open files one-by-one, extract/ar avv- erage the threshold variable, and regrid/align explicitly (or fail fast with clear diagnostic about which coordinate is missing).
Validation gates: before heavy computation, assert non-emply spatial /tme subsets and coordinate consistency between forecast and thresholds.
E.3 Case 3: Runtime constraints (timeouts + invalid code candidates)
Where happens. Early-stage programming tasks with heavy I/O or expensive operations (example: Sask_SST_L subtask 1).
Symptom impact. Some LLM-gennrated candidates fail immediately due to syntax errors; others time out under the orchestrator's execution limit, causing retries and delayed completion.
Log evidence.
task_ Bask_SST____ htt///////pogrraam
agent.log:
Syntax error: unterminated string literal (...) (subtask_1_....cco .py) cand3.pppp Exception during execution (attempt 1): ... timed out after 120 seconds
Syntax error: unterminated string literal (...)
Intermediate artifacts.
Candidate scripts: task_SST___ /tt.TT///coe//bbbbaa.._...806000
py.
Partially produced outputs (after eventual success): task fask_SST_aa__ Sas__ST__/dee 1/dat// downloads and task_SSt (aTT__/code__uppppe NetCDF outputs.
Underlying cause. Two distinct issues: (i) generative code may be syntactically invalid, and (ii) even valid code may exceed a fixed wall-clock budget due to large downloads, slow per-file loops, or non-vectorized computations.
Mitigation / resolution.
Pre-execution lint/syntax gate: run fast parser check (python -m py_ py_compile) before full exccution to avoid wasting the timcout budget.
Chunked and resumable I/O: download in chunks with checkpointing: avoid re-downloading existing files.
Performance-aware
prompting and templates: prefer known-efficient patterns (vectorization, xarray operations, bounded spatial windows) and avoid Python-level loops over grids.
Adaptive timeouts: allow longer budgets for known-heavy subtasks, or add progress heartbeats so the orchestrator can distinguish a hang from a slow but healthy job.
E.4 Case 4: Artifact contract break (path/working diieerrrinngddrrcoorry mismatch halts final reporting)
Where it happens. Reporting stage (visualization agent), task_ TC_ subtask 10.
Symptom / impact. The workflow fails after three attempts at the final reporting step because the visualization agent cannot. find an intermediate file that does exist, but at a different relative path than assumed.
Log evidence. Subtask successfully writes code coweeootpteetteeeeooooocooooceet wwtppptt//oorrra.peeseueeeeeeee txt, but subtask 10 searches under code/code ctcc/oodooouue cttp:/oode___puttteeeeooe and fails repeatedly:
47

---

Published in Transactions on Machine Learning Research (04/2026)
task TC_ hoopa///....ooorassstaatagggggee :lo.:gentttt.:::330.3
Subtask 9 succeeded.
Subtask 10/10: Read httpppppppprrrttaaa.pssseeee... Subtask 10 failed with error:
file not found ... wtw.eeeeecc.eeo...........aaaaar 2022
Intermediate artifacts.
Produced by subtask 9 task_ Ti_k__T___e_ /eaat///_ecdddeppttt///aaaaaarrr 20200051100
Produced earlier (exists): task_ TC TC_1/code htt.://ooeeoupputtteeeeeeeeeeeee outpppt//atterroooooolllllleella fields 2023036812117_202 2100 0207 _te__1100ppp00 .ng.
Missing due to path mismatch: the report file task task_TC_ Biia_____rrrportr22233333 20120631122220331122211 not gener- ated.
Underlying cause. A brittle contract between agents regarding the working directory and relative paths. The visualization agent implicitly ran as if its CWD were Dask_TC_1/comt while the artifact was written relative to the task root.
Mitigation / resolution.
Enforce a single task-root CWD: the orchestrator should run all agent scripts from the task root and forbid ad-hoc CWD changes.
Resolve paths from an explicit root: use a declared TASK ROOT and construct paths via Path (TASK_RROT) /"aak___0O))"cood___aeee ("oder)//coo__outpptt"/////
Preflight artifact checks: before generating reports, validate required inputs exist (and are non- empty) and emit a clear, single error message with the checked paths.
Contract tests: add small automated checks that the expected filenames cacaames/locaatee llocaioos match the previous subtask outputs.
F Human Expert vs. LLM-as-Judge Agreement
To validate the reliability of our LLM-as-a-judge evaluation, we conduct human expert assessment on the same set of generated reports and compare domain-expert scores with the LLM judge scores. For each climate domain, we compule the mean absolute difference between the human expert score and the LLM judge score (lower is better, indicating stronger agreement).
F.1
Agreement Results Across Domains
Table summarizes the expert- LLM score gaps across the six domains in CLIMATE-AGENT-BENN
Domain
AR
DR.
EP
HW
SST
TC
Mean absolute score difference [[eexprtt
8LLM
1.9167
0.3250
0.5500
0.1250
0.4750
0.4750
Table 8:
Human expert LLM-as-judge agreement by domain. Values are mean absolute score differences;
smaller values indicate closer agreement.
Overall, the agreement is high in five out of six domains: DR, EP, HW, SST, and TC all exhibit gaps around ~0.3 0.55, suggesting that the LLM judge largely tracks expert preferences in these settings. In contrast, AR shows a substantially larger discrepancy (1.9167). This indicates that the Atmospheric involving stricter visualization requirements and more muanced spatial diagnostics can amplify differences between what an expert prioritizes and what an LLM judge infers from the report and figures alone. We therefore perform targeted qualitative analysis of the AR domain below.
River
48

---

Published in Transactions on Machine Learning Research (04/2026)
F.2 Analyzing the Most Complex Task: Atmospheric River Example
We analyze two representative AR cases to better understand the source of disagreement between the expert and the LLM judge. In both cases, we compare the Al Agent output to Specialist-produceed reference.
frequency
2022-03-24
InRialized on: 2022-03-24
Validity date: 2022-03-31 to 2022-05-06
4314
21
25'94
130/W
Figure 8: Case 1. Left: Output from the Al Agent. Right: Created by Specialist.
Case 1.
AR frequency ERAS reanalysis data
Date: 2022 12 12 2022-12 18
B8l Mil HAR (2 Maa
259
33016
3,35%
140PG
18576
Figure Case 2. Left: Output from the AI Agent. Right: Created by Specialist
Case 2.
Comments. Across the two cases, the written organization and narrative framework are broadly similar; the primary difference lies in the visual quality of the AI Agent -Case omits critical information, whereas Case reproduces the main spatial patterns reasonably well.
Readability. In both cases, the text uses a region-by-regien by-region, bullet-point format with clear logic and generally appropriate scientific terminology. The descriptions of data values and goographic locations are explicit and accessible to the intended expert audience. However, the phrasing and terminology relatively repetitive, with limited linguistic variety.
49

---

Published in Transactions on Machine Learning Research (04/2026)
The data sources are clearly stated and credible, and the interpretation of spatial patterns is generally sound, covering the major regional distributions of AR frequency. Nonetheless, neither case discusses forecast uncertainty or potential model biases, yielding comparable, moderate level of rigor overall.
Scientific Rigor.
Completeness. Both narratives cover the standard elements of "region-intensity hotspots/ holspots/corridon'' corridors" and follow a similar structure, but neither includes an assessment of potential impacts or implications.
Visual Quality. In Case 1, the AI Agent figure shows an obvious mmission/anomajj (the principal high-frequency band and hotspots are not depicted), which substantially weakens if not invali- dates-the evidential basis for the subsequent interpretation. In Case 2, the AI Agent figure is largely consistent with the Specialist's 's map (a pronounced AR band and an offshore high-value core are evident); remaining differences are minor in extent, thickness, or position and fall within acceptable plotting variability.
Takeaway. These examples suggest that the larger expert-LLMM gap in AR is driven primarily by visual- evidence scnsitivity: domain experts penalize figure omissions or physically implausible spatial patterns more strongly, whereas an LLM judge may overweight the narrative structure when the text appears coherent. This motivates (i) stronger visualization validation checks and (fi) incorporating small amount of expert auditing for visually critical tasks.
G Computation Cost
This section quantifies the computational overhead of CLIMATE.AGNT in terms of (i) LLM token usage and (ii) end-to-end wall-clock time. Since CLAMATEAGENT uses multi-candidate generation (e.g., n=8 in data agents) and iterative debugging/retin /retries, reporting these costs important for interpreting robustness gains.
Per-agent token/time breakdown. Table g reports the breakdown by domain (AR/DR/ EP HHW/ HEP/HW/STT and by agent type: Planning (Plan), Programming (Prog), and Visualization (Viz). Across the five rep- resentative domains, CLIMATEAGENT consumes total of 664,896 input tokens and 938,365 output tokens (1,603,261 total tokens), with aggregated wall-clock time of 8,094,272 ms (>< 134.9 min). Wall-clock time includes end-to-end execution (tool calls, downloads, Python script execution), not only LLM inference.
Where the overhead comes from. As expected, the dominant cost the Programming agent, which performs iterative code generation and debugging: Prog accounts for 729,331 729,331/938, /931/938,,65 = 77.7% of output tokens and most of the wall-clock duration across tasks. Planning is comparatively wight (one call per task in these runs), while Visualization can incur additional calls when report synthesis figure embedding needs refinement (c.g., HW).
Comparison to a single-pass GPT-5 baseline (tokens). To contextnalize the overhead of multi-step agentic execution, Table 10 compares output tokens (OtTk) between CLIMATEAGENT and a GPT-5 base- line for Task 1 in each domain. Summed over the five domains, CLIMATEAGENT produces 938,365 output tokens versus 176,220 output tokens for the GPT-5 baseline (Task 1), reflecting the additional intermedi- ate reasoning, multi-candidate generation, and iterative debugging in CLIMATEAGENT. We emphasize that this overhead directly supports higher robustness in tool-augmented, long- lorg-horizon workflows by reducing the probability of unrecoverable failures.
H Comparison to a General-Purpost Multi-Agent Framework (LangGraph DeepAgent)
Background and motivation. Our main experiments compare CLIMATEAGENT against strong generic baselines (e.g., single-pass GPT-5 workflow and GitHub Copilot). However, these comparisons do not fully
50

---

Published in Transactions on Machine Learning Research (04/2026)
Table 9: CLIMATEAGENT computational cost and wall-clock time breakdown by domain and agent. DlanPPoog Viz denote the Bhaniing/Prorramming /iingg///isaaiiiiiinn agents. SubT = number of subtasks as- signed to the agent; Invk number of LLM invocations; InTk/ O:Tk = input/output tokens; Dur = wall- clock duration in milliseconds for that agent; TotO total output tokens summed over Plan+Prog++ Viz;
TotD total duration (ms) summed over Plan+Prog+
Plan
Prog
Total
Task
SubT
Invk
InTKk:
OrTh
Dur
SubT
Invk
InTk
OrTk
Dur
SubT
Invk
Tn/Tk:
O:Tk
Dur
TotO
TorD
AR
1
9630
6016
103138
4
193785
233101
1810652
3481
25769
138872
265216
2052562
DR
8337
6291
94427
3
90042
123661
1408820
2366
26971
90090
156923
1602337
7185
105741
125339
1020384
1211
37952
172654
1232562
HW
1
8022
5025
90652
1
4
31018
38964
331431
7589
4/129
312029
88118
761112
SST
7830
6045
144748
3
16
177812
203966
2005904
1791
45443
241747
265454
2442399
Table 10: Output-token (OtTk) comparison between CLIMATEAGENT and GPT-5 baseline for Task 1 in each domain. OtTk (CLIMATEAGENT) corresponds to TotO in Table (total output tokens summed over Mlan+Pro+VViz) GPT-5 baseline reports total output lokens for the single-pass run.
Task
CLIMATEAGENT OLTk GPT-5 OtTk
AR
265216
62461
DR
156923
30343
EP
172654
35489
HW
88118 18
11358
SST
255454
36569
Total
938365
176220
isolate whether the observed gains come from our climale-specific design choices (specialized data agents, explicit persistence, artifact contracts, and targeted self-correction), or simply from using any multi-step agentic workflow. To address this, we include a comparison against widely used general-purpose multi- agent framework built for tool-using, long-horizon tasks: DeepAgent implemented with LangGraph (via the deepagents package). DeepAgent is prompted with the same tools and required to satisfy the same outpul/eveee evaluation contract as CLIMATEAGENT, enabling a controlled assessment of whether domain- specialized design is necessary beyond generic multi-agent orchestration.
H.1 DeepAgent as General-Purpose Multi-Agent Baseline
We use DeepAgent as our general-purpose multi-agent baseline. DeepAgent is an LLM-driven supervisor implemented as a LangGraph runnable that natively supports planning, tool use, and hierarchical delega- tion. In our setting, the DeepAgent supervisor can: (i) maintain explicit planning state (c.g., a TODO list via default planning tools such write, todos/read, r (ii) call tools (Python functions exposed LangChain tools), (iii) operate over scoped filesystem backend, and (iv) delegate work to named subagents.
Hierarchical delegation via native subagents. At DeepAgent creation time, we define set of named subagents, each with a description, system prompt, and a restricted subset of tools. During execution, the supervisor decides when to delegate by calling the native subagent tool (shown as task in our traces). Each subagent then runs its own tool-calling loop (within the same filesystem scope) and returns result to the supervisor. Conceptually:
Supervisor - tasb(Suuagent) + (project tools) Artifacts.
Fairness constraints: same tools, same output contract. To make the comparison con- trolled and low-risk, we enforee the same evaluator-facing artifacts and directory layout as CLI- MATEAGENT: both systems write into
51

---

Published in Transactions on Machine Learning Research (04/2026)
and must produce non-empty code_ cadd__ouputt/iiaal_aaaaa Srtpptt/Eiaal_pport..ttt md. Pilesystemmackend (root bobootddr=ttttt dir), preventing reads/wrr wwrds/writes outside single experiment folder. We also avoid exposing a general shell/execute tool; instead we provide a controlled python_rup run tool that executes Python under task dir with a timcout.
We scope side effects with
Project tools exposed to DeepAgent. We expose small, explicit tool set that mirrors the capabilities used by CLIMATEAGENT while remaining framework-agnostic:
init task dir, inspect_ _uettt__u______ user_data data
context_ppachl (deep-merge updates into context.json.
register_codee register_resull (update context, json codees' and context. context.jsonl jaapattssssooorrrslta ['results']]
python_rum (execute Python file code string under task_ dir, with timeout)
validate_contrace (checks required keys and rode_outpurr/tinaareerrr........
get_cds ____c_ddddtssttss (returns curated CDS dataset identifiers)
(optional
run_ sin_downlodd_ cds (task dir, subtask, attempt) run__dwmload_eee ecmwf (task dir, subtask, attempt)
wrappers)
and
These tools allow DeepAgent manage state, create/register artifacts, run controlled code, and (optionally) reuse the same download wrappers CLIMATEAGENT, while leaving high-level planning and delegation to the general-purpose framework.
Named subagents. We define four named subagents with restricted tool subsets to reflect common roles in tool-using workflows:
planner: produces and updates a structured subtasks list in context... json (via context_patch)
downloader: acquires datasets under data/ and registers scripts/resuu in context.
analyst: writes analysis scripts under code/ and produces intermediate artifacts under codel_output..
reporter: produces code_ tote_oottutttiinlrepooo. and verifies the output contract (via palidate_contract)
Observability and traces. To support qualitative diagnosis of generic-agent behavior, the DeepAgent baseline emits structured traces: task_ fook_//loo/deee deep haailipnatttieiiieeeeeeieeeeeeee log (runner logs), task #/log/ hheg//dep__uue native trace. crace.jsonl (structured events such as messages tool calls/ /alls/subagent start-end/TODO updates), and task_ native_trace....
* #/log/deep /log//dep_nattie___ trace. txt (a human-readable timeline). These traces enable analysis of whether errors
arise from generic orchestration (c.g., planning drift, inefficient tool use, brittle path assumptions) versus domain-specific mechanisms implemented in CLIMATEAGENT..
In the next subscction, we report preliminary experiments comparing CLIMATEAGENT to DeepAgent under identical tool access and the same outpul contract.
H.2 Preliminary Experiments: ClimateAgent vs. DeepAgent under the Same Tools and Output Contract
We conducted preliminary experiments comparing CLAMATEAGENT against DeepAgent under identical tool
access and the same evaluator-facing output contract (Sec. H.1 Table I1 summarizes report-quality scores for six representative domains.
52

---

Published in Transactions on Machine Learning Research (04/2026)
Table 11: Report quality comparison between CLIMATEAGENT and DeepAgent. gent run failed or did not produce a valid report for scoring).
indicates err (DeepA-
CUMATEAGENT
DeepAgent
Category
Read.
Rigor
Comp.
Vis.
Score
Read.
Rigor
Comp-
Vis.
Score
AR
7
5
8
6.50
8
6
5
6.50
DR
8.00
EP
8
7
9
8.25
HW
9
8
8.50
9
9
8
8.30
SST
10
8
9.00
9
9
9
8.75
TC
9
8
8,50
Overall outcome. DeepAgent successfully produced scorable reports for 3/6 tasks (AR, HW, SST), while 3/6 runs failed (err) and did not generate a valid report for scoring (DR, EP, TC). In contrast, CLIMATEAGENT produced valid reports across all six tasks.
Failure modes of a general-purpose framework. The three DoepAgent failures are qualitatively con- sistent with known brittleness generic multi-step agents in tool-angmented scientific workflows:
DR: tool invocation error. The run failed due to incorrect tool usage (tool-call schema/argumest mismatch), preventing the workflow from progressing to subsequent steps.
EP: data download error. The failed during data acquisition, reflecting difficulty in constructing valid requests against constrained, evolving climate data APIs.
file-path error. The run failed at a later stage due to a btth//orking-direette mismatch, i.e., a brittle artifact contract between steps (an intermediate file existed but was referenced under an incorrect relative path).
TC:
These failures directly motivate two core design choices in CLIMATEAGENT: (i) Data-Agents with dy- namic API introspection (querying the latest. valid parameters / dataset metadata at runtime re- duce invalid-paramcter and format-mismatch errors during download), and (ii) Adaptive Caff-Correction, which uses bounded retries and error-conditioned regeneration to recover from runtime failures (including tool/AP? errors and artifact-contract issues).
On the three tasks where DeepAgent completed end-to-end execution (AR, HW, SST), DeepAgent scores are slightly below CLIMATEAGENT in overall report score and/or key dimensions (Table 11.. For example, DeepAgent matches CLIMATEAGENT on AR report score but trails on visual quality, and it is modestly lower than CLIMATEAGENT on HW and SST overall score. While preliminary, this gap suggests that beyond generic multi-step orchestration, domain-specific knowledge integration (e.g., climate-awan aware data handling conventions, robust scientific computation patterns, and task-specific validation /formatting constraints) contributes to higher scientific rigor ccompleteness and more reliable end-to-end outpuls.
Quality comparison on successful runs.
Takcaway. Taken together, these preliminary tests indicate that improvements are not simply due to adopting any multi-step agentic workflow. A general-purpose multi-agent framework with the same tools can still fail due to tool-call brittleness, constrained data. APIs, and cettaactt-oottatt/peeooooo /ptth issues, whereas CLIMATEAGENT'S domain-specializan components (dynamic APT introspection and adaptive self-corroction) improve both completion reliability and report quality under the same output contract.
53


Reviews:
[{"review_id": "pRlXHiA4rq", "reviewer": "Reviewer_EjuM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces CLIMATEAGENT, an autonomous multi-agent framework designed to execute end-to-end climate data science workflows, from data acquisition to analysis and report generation. The work is motivated by the observation that generic LLM-based agents struggle in climate science due to the heterogeneity of data sources, domain-specific APIs, and methodological constraints, while manual workflows remain time-consuming and do not scale. To support systematic evaluation, the authors also introduce CLIMATE-AGENT-BENCH-85, a manually curated benchmark of 85 real-world climate science tasks spanning multiple phenomena and requiring full end-to-end execution.\n\nStrengths: \n1. The paper is easy to read and well structured. The overall system architecture is clearly described, and the roles of the different agents are explained with enough details.\n2. The use of specialized agents for planning, data acquisition, and coding is well justified given the complexity and heterogeneity of climate science workflows. The authors do a good job of explaining the difficulties of the climate science domain for end-to-end workflows, even for readers who are not experts in this domain.\n3. The experimental section provides both quantitative results and qualitative discussion that help explain why, at a high level, the proposed multi-agent design leads to better end-to-end performance compared to generic LLM-based agents.\n4. CLIMATE-AGENT-BENCH-85 is manually curated and targets end-to-end report generation rather than isolated subtasks. This fills a clear gap in the current evaluation landscape for climate science automation and represents, in my view, a major contribution of the paper.\n\nWeaknesses: \n1. While the paper provides thorough system-level analysis and discussion of performance gains, it does not include ablation studies that isolate the contributions of individual components. Additional analysis attributing improvements to specific components would strengthen the empirical claims and deepen the understanding of the system.\n2. The system is evaluated exclusively on CLIMATE-AGENT-BENCH-85, which is introduced in this work. While the benchmark itself is valuable, the paper does not clearly discuss whether other climate science benchmarks or task suites exist that could be used for complementary evaluation, nor does it explicitly justify why evaluation is restricted to this single benchmark. As a result, it is difficult to assess how representative the reported performance is compared to other works in this domain. \n3. The reported performance improvements over baselines such as GitHub-Copilot and a GPT-5 are clear, but their practical significance is difficult to interpret. These baselines are generic coding or agentic tools rather than climate-specific workflow systems, and the paper should make clearer why climate-focused agentic approaches cannot be used as baselines. \n4. Report quality is assessed exclusively using an LLM-as-a-judge framework. Given that CLIMATE-AGENT-BENCH-85 is a central contribution and that the benchmark size is moderate, the evaluation would be significantly strengthened by including some degree of human expert assessment. As it stands, the scalar scores are somewhat difficult to interpret scientifically.\n5. An analysis on system failures is missing.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The main claims of the paper are supported by clear and generally convincing evidence. The system architecture is described in sufficient detail to justify the design choices, and the experimental results on the proposed benchmark demonstrate improved end-to-end performance over strong generic baselines. The authors complement quantitative results with qualitative analyses and discussions that help explain why the proposed multi-agent approach is effective.", "audience_yn": "Yes", "audience_explanation": "The paper addresses a relevant problem for the TMLR community, at the intersection of LLM-based agents, scientific workflow automation, and climate science. Furthermore, this work introduces a new benchmark that can be used, after addressing the reported weaknesses, to compare different systems in end-to-end climate science tasks.", "requested_changes": "Critical changes: \n- Address Weakness #4: report quality is currently assessed exclusively using an LLM-as-a-judge framework. Given that CLIMATE-AGENT-BENCH-85 is a central contribution and that the benchmark size is moderate, the evaluation would benefit from additional validation. \n- Address Weakness #3: make clearer why other climate-specific workflows are not used as baselines. \n\nSuggested changes to strengthen the work: \n- Address Weakness #1: include component-level ablation or analysis.\n- Address Weakness #2: either expand the evaluation to other climate-specific benchmarks to prove that the system would work, or explain more clearly why this cannot be done. \n- Address Weakness #5: given the complexity of the overall system, the work would be strengthened by a dedicated analysis of failure cases, including scenarios in which the system fails to complete tasks or produces suboptimal outputs, and a discussion of the underlying causes.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MWF23lIdFA", "reviewer": "Reviewer_RBNs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "\\subsection*{Summary of Contributions}\n\nThe paper presents a structured and well-motivated approach to automating climate science workflows using large language models. Its main contributions can be summarized as follows:\n\n\\begin{itemize}\n    \\item \\textbf{Conceptual Multi-Agent Framework.}  \n    The paper proposes \\textit{ClimateAgent}, a conceptual multi-agent orchestration framework that decomposes complex climate science workflows into specialized roles, including planning, orchestration, data acquisition, coding, and visualization. The framework emphasizes modularity, task decomposition, and role specialization tailored to climate data analysis.\n\n    \\item \\textbf{Formalization of Climate Workflows as Sequential State Transitions.}  \n    The authors formalize end-to-end climate analysis as a sequential transformation of a persistent workflow context, providing a clear abstraction for reasoning about planning, execution, and error recovery in long-horizon scientific workflows.\n\n    \\item \\textbf{Robustness-Oriented Design Principles.}  \n    The work highlights three core capabilities—coordinated task planning, contextual coordination, and adaptive self-correction—and illustrates how these principles can mitigate common failure modes of LLM-based code generation in scientific settings.\n\n    \\item \\textbf{Climate-Agent-Bench-85 Benchmark.}  \n    The paper introduces a new benchmark consisting of 85 realistic, end-to-end climate science tasks spanning six major climate domains. The benchmark is carefully curated by domain experts and reflects real-world data access, processing, and analysis challenges.\n\n    \\item \\textbf{Comprehensive Empirical Evaluation.}  \n    Through extensive experiments, the paper demonstrates that a system following the proposed agentic design principles substantially outperforms strong GPT-5–based baselines and GitHub Copilot in terms of report quality, completeness, and robustness on the proposed benchmark.\n\n\\end{itemize}\n\nOverall, the paper contributes a compelling architectural blueprint and evaluation framework for agent-based automation in climate science, supported by large-scale empirical evidence, though the concrete implementation details of the proposed system are not fully specified.", "claims_and_evidence_yn": "No", "claims_explanation": "\\subsection*{Major Concern: Lack of Concrete Implementation of \\textit{ClimateAgent}}\n\nWhile the manuscript consistently claims the introduction of an autonomous multi-agent system termed \\textit{ClimateAgent}, it remains unclear where and how this system is concretely implemented beyond a conceptual design. The paper provides detailed descriptions of agent roles (Plan-Agent, Orchestrate-Agent, Data-Agent, Coding-Agent), but does not sufficiently specify whether these agents correspond to independent software components, persistent LLM instances, or logical prompt roles executed sequentially within a single control loop.\n\n\\paragraph{Implementation Ambiguity.}\nThe manuscript lacks a clear description of the software realization of \\textit{ClimateAgent}. It is not specified whether agents are implemented as separate processes, modular classes, state machines, or simply as prompt templates. While Algorithm~1 outlines a high-level orchestration procedure, it remains abstract and does not clarify how agents are instantiated, scheduled, or managed at runtime.\n\n\\paragraph{Agent vs.\\ Prompt Role Distinction.}\nIt is ambiguous whether each agent represents a truly independent agent with persistent memory and execution state, or whether the system consists of a single LLM assuming different roles via prompting. This distinction is critical for evaluating the novelty and validity of the claimed multi-agent architecture, as role-based prompting does not necessarily constitute a genuine multi-agent system.\n\n\\paragraph{Lack of Implementation Evidence.}\nThe manuscript does not provide a code repository, implementation appendix, or execution trace demonstrating the existence of a functioning \\textit{ClimateAgent} system. While the benchmark results are extensive, they primarily evaluate output quality rather than the system mechanics themselves, leaving uncertainty about reproducibility and deployability.\n\n\\paragraph{Unclear Context Persistence Mechanism.}\nThe paper claims persistent workflow context (e.g., serialized JSON state), yet no concrete example or schema is shown. It is therefore unclear how context is stored, updated, validated, or recovered during execution, and how this mechanism differs from standard conversational memory.\n\n\\paragraph{Benchmark-System Relationship.}\nAlthough \\textit{Climate-Agent-Bench-85} is well-defined, it is not explicitly stated whether \\textit{ClimateAgent} is a reusable system that can be independently applied to this benchmark, or a bespoke experimental pipeline constructed specifically for the reported evaluation. This raises concerns regarding generalizability and reuse by the community.\n\n\\paragraph{Autonomy Claims.}\nThe manuscript claims fully autonomous, end-to-end execution without human intervention, yet does not clarify how practical constraints such as API credentials, rate limits, tool installation, or unexpected runtime failures are handled. These omissions make it difficult to assess whether the reported autonomy is achievable in real-world settings.\n\n\\paragraph{Summary.}\nOverall, \\textit{ClimateAgent} currently appears more as a conceptual orchestration framework than a clearly instantiated agentic system. The paper would benefit substantially from an explicit description of the system implementation, including agent instantiation, execution flow, context management, and a minimal reproducible example or code release. Therefore I would reject the draft.", "audience_yn": "No", "audience_explanation": "While the paper addresses an important application domain (climate science) and explores agent-based automation using large language models, the primary contribution is largely conceptual and application-specific rather than methodological. The work does not introduce new learning theory, model architectures, or generalizable machine learning techniques that would be of broad interest to the core TMLR audience.\n\nMuch of the paper focuses on system design choices, workflow orchestration, and domain-specific benchmarking for climate data analysis. Although these aspects may be valuable to practitioners working at the intersection of climate science and AI systems, they are less aligned with TMLR’s emphasis on foundational advances in machine learning research.\n\nAs a result, while the paper may be of interest to a specialized subset of researchers in scientific workflow automation or applied AI for climate science, it is unlikely that a substantial portion of TMLR’s general readership would find the findings directly relevant.", "requested_changes": "\\subsection*{Requested Changes}\n\nTo strengthen the contribution and make the claims of an autonomous multi-agent system verifiable and reusable, the following changes are required:\n\n\\begin{itemize}\n    \\item \\textbf{Concrete Software Implementation.}  \n    The authors should provide a concrete implementation of \\textit{ClimateAgent} as a runnable software system or toolbox. This implementation should explicitly instantiate the proposed agents (Plan-Agent, Orchestrate-Agent, Data-Agent, Coding-Agent) as executable components rather than conceptual roles.\n\n    \\item \\textbf{Use of Established Agent Frameworks.}  \n    Implementing the system using existing agent frameworks (e.g., LangChain, AutoAgent, CrewAI, or comparable multi-agent toolkits) would substantially improve clarity, reproducibility, and credibility. Clearly specifying how agents communicate, store context, and recover from errors is essential.\n\n    \\item \\textbf{Explicit LLM Backend Specification.}  \n    The manuscript should clearly document which large language models are used (e.g., LLaMA-family models, GPT-based models, or other open-weight LLMs), how they are accessed, and how model choice affects system behavior. If multiple backends are supported, this should be demonstrated explicitly.\n\n    \\item \\textbf{Executable Examples and Traces.}  \n    At least one end-to-end executable example should be provided, including the full agent interaction trace (e.g., context evolution, retries, API introspection steps). This would allow readers to verify that the system operates as claimed.\n\n    \\item \\textbf{Public Code Release.}  \n    A public code repository with installation instructions, configuration files, and minimal reproducible scripts should be released. This is particularly important given the scale of the benchmark and the strong performance claims.\n\n    \\item \\textbf{Clarification of Autonomy Assumptions.}  \n    The authors should clarify practical assumptions required for autonomy, such as API credentials, rate limits, tool installation, and failure handling. This clarification would help distinguish between conceptual autonomy and deployable autonomy.\n\n\\end{itemize}\n\nOverall, providing a concrete, reproducible software implementation—ideally grounded in established agent frameworks—would significantly strengthen the paper and align its claims with verifiable system behavior.", "broader_impact_concerns": "without providing a software package / or a toolbox is not possible!", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sXHp7lMsAa", "reviewer": "Reviewer_H7nP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents CLIMATEAGENT, a multi-agent framework for executing complex climate data science workflows. It uses a hierarchical architecture with specialized agents for planning, orchestration, data access, and coding to address domain-specific data heterogeneity, evolving APIs, and long-horizon execution failures. The authors also introduce CLIMATE-AGENT-BENCH-85, a benchmark of 85 expert-curated real-world climate tasks across six domains. Experiments show that CLIMATEAGENT achieves promising task completion and generates higher-quality scientific reports than GPT-5 and GitHub Copilot.\n\n### Strengths\n- The Adaptive Self-Correction strategy is a strong practical contribution. The paper clearly maps distinct failure modes (e.g., invalid API parameters, logical errors in code) to targeted recovery strategies, such as multi-candidate generation (m=8) for parameter discovery and iterative refinement for code correction. This design effectively addresses the brittleness of long-horizon scientific workflows.\n- Clear Modular Architecture: The decomposition into specialized roles (Plan vs. Data vs. Coding) effectively isolates context and expertise, preventing the context drift often seen in monolithic agent workflows.\n- The benchmark itself is likely to be a valuable community resource.\n\n### Weaknesses\n- Limited Baseline Comparisons: the evaluation compares CLIMATEAGENT primarily against single-model baselines (GPT-5 and GitHub Copilot). As a result, it remains unclear whether the observed gains stem from the specific architectural innovations or simply from the general advantages of a multi-agent system over a zero-shot LLM. \n- Incremental Novelty: while the paper frames execution robustness as a major open challenge, many proposed components, such as browser-based parameter discovery, reflection loops, and candidate sampling, closely resemble established engineering practices in general-purpose coding agents. The work’s primary contribution appears to be thoughtful system integration and domain adaptation rather than fundamentally new AI methodologies.\n- Cost: the computational and monetary costs of dynamic introspection and multi-candidate execution are not quantified, leaving questions about scalability.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are generally well-supported by clear and accurate evidence, though the strength of the convincing factor is slightly limited by baseline choices.\n\n- The claim of superior performance is supported by Table 2\n- The efficacy of Adaptive Self-Correction is convincingly evidenced by an ablation analysis that explicitly maps 35 baseline failure cases (e.g., Data Request Errors) to specific code fixes generated by the system.\n- Qualitative evidence (Figure 2) effectively demonstrates the system's ability to maintain cross-step dependencies in complex tasks where baselines fail to produce valid visualizations.", "audience_yn": "Yes", "audience_explanation": "While the paper is heavily applied, its contributions to robust agent design, benchmarking methodologies, and tool-use strategies make it relevant to a wide swath of the TMLR audience, particularly those focused on systems, evaluation, and applied ML.", "requested_changes": "- Baseline Comparison: The current evaluation compares the proposed multi-agent system against single-model baselines (GPT-5 and GitHub Copilot). To validate the claim that the specific hierarchical architecture of CLIMATEAGENT is superior, please include a comparison (or at minimum, a detailed qualitative discussion supported by preliminary tests) against a general-purpose multi-agent framework prompted with the same tools. This is necessary to determine if the performance gains stem from your specific design choices or simply from using any multi-step agentic workflow.\n- Cost and Latency Analysis: Please add an analysis of the computational cost (e.g., token usage/inference cost) and wall-clock time for CLIMATEAGENT compared to the baselines. The use of multi-candidate generation (m=8) and iterative debugging implies a significant resource overhead that should be quantified.\n- Clarification of Related Work: Refine the claim regarding the \"gap between high-level planning and execution robustness\" in the Related Work section. Please explicitly distinguish your contribution from existing general-purpose self-correction/reflexion frameworks (e.g., Reflexion, self-debugging agents) and explain why those specific mechanisms are insufficient for climate APIs without your proposed architecture.", "broader_impact_concerns": "no", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)