Zephyrus: An Agentic Framework for Weather Science
Authors: Sumanth Varambally
Marshall Fisher
Jas Thakker
Yiwei Chen
Zhirui Xia
Yasaman Jafari
Ruijia Niu
Manas Jain
Veeramakali Vignesh Manivannan
Zachary Novack
Luyu Han
Srikar Eranky
Salva Rühling Cachay
Taylor Berg-Kirkpatrick
Duncan Watson-Parris
Yian Ma
Rose Yu
Venue: ICLR 2026 Poster (2026)
Keywords: Agents
Large Language Models
Weather Science
Code Generation
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We built an AI weather assistant that lets scientists explore meteorological data through natural conversation, and created a benchmark to evaluate LLMs for weather science.

Abstract: Foundation models for weather science are pre-trained on vast amounts of structured numerical data and outperform traditional weather forecasting systems. However, these models lack language-based reasoning capabilities, limiting their utility in interactive scientific workflows. Large language models (LLMs) excel at understanding and generating text but cannot reason about high-dimensional meteorological datasets. We bridge this gap by building the first agentic framework for weather science. Our framework includes a Python code-based environment for agents (ZephyrusWorld) to interact with weather data, featuring tools including a WeatherBench 2 dataset indexer, geolocator for geocoding from natural language, weather forecasting, climate simulation capabilities, and a climatology module for querying precomputed climatological statistics (e.g., means, extremes, and quantiles) across multiple timescales. We design Zephyrus, a multi-turn LLM-based weather agent that iteratively analyzes weather datasets, observes results, and refines its approach through conversational feedback loops. We accompany the agent with a new benchmark, ZephyrusBench, with a scalable data generation pipeline that constructs diverse question-answer pairs across weather-related tasks, from basic lookups to advanced forecasting, extreme event detection, and counterfactual reasoning. Experiments on this benchmark demonstrate the strong performance of Zephyrus agents over text-only baselines, outperforming them by up to 44 percentage points in correctness. However, the hard tasks are still difficult even with frontier LLMs, highlighting the challenging nature of our benchmark and suggesting room for future development. Our codebase and benchmark are available at \url{https://github.com/Rose-STL-Lab/Zephyrus}.

Published as a conference paper at ICLR 2026
ZEPHYRUS: AN AGENTIC FRAMEWORK FOR WEATHER SCIENCE
Sumanth Varambally, Marshall Fisher, Jas Thakker, Yiwei Chen, Zhirui Xia, Yasaman Jafari, Ruijia Niu, Manas Jain, Veeramakali Vignesh Manivannan Zachary Novack, Luyu Han, Srikar Eranky, Salva Rühling Cachay Taylor Berg- Kirkpatrrk,,, Duncan Watson-Parris, Yi-An Ma, Rose Yu UC San Diego
ABSTRACT
Foundation models for weather science are pre-trained on vast amounts of strue- tured numerical data and outperform traditional weather forecasting systems. How- ever, these models lack language-base
reasoning capabilities, limiting their utility in interactive scientific workflows. Large language models (LLMs) excel at un- derstanding and generating text but cannot reason about high-dimensional meteo- rological datasets. We bridge this gap by building the first agentic framework for weather science. Our framework includes a Python agents SARHHHYRRWRRLLDD to interact with weather data, featuring tools including a WeatherBench 2 dataset indexer, geolocator for geocoding from natural language, weather forecasting, climate simulation capabilities, and climatology module for querying precomputed climatological statistics (c.g., means, extremes, and quantiles) across multiple timescales. We design ZEPHYRUS, multi-turn LLM- based weather agent that iteratively analyzes weather datasets, observes results, and refines its approach through conversational feedback loops. We accompany the agent with new benchmark, ZEPHYRUS with scalable data genera- tion pipeline that constructs diverse question-answer pairs across weather-related tasks, from basic lookups to advanced forecasting, extreme event detection, and counterfactual reasoning. Experiments on this benchmark demonstrate the strong performance of ZEPHY RUS agents over text-only baselines, outperforming them by up to 44 percentage points correctness. However, the hard tasks are still difficult even with frontier LLMs, highlighting the challenging nature of our benchmark and suggesting room for future development. Our codebase and benchmark are available https: //githb. com/Rose STL Lab/ Sae--Lab//epphh /epphyus
environment for
INTRODUCTION
Large language models (LLMs) have demonstrated remarkable capabilities across diverse scientific domains (Birhane et al. 2023), revolutionizing fields from drug discovery (Zheng et al.) 2024; Wu ct al., 2024b) and materials science (Lei et al. 2024 Jablonka et al.. 2023) to network biology (Theodoris et al. 2023). These models excel at processing textual content such as scientific literature, source code (Jiang al. 2024), and structured data tables (Zhang al. 2024). However, their application to domains requiring reasoning over high-dimensional numerical data remains limited (Wang et al. 2024).
Meteorology offers a compelling yet challenging case study, as combining natural language reasoning with complex atmospheric data has the potential to greatly advance weather research. Weather prediction is a critical scientific challenge, with profound implications spanning agriculture, disaster preparedness, transportation, and energy management (Alley al.) 2019). The field has witnessed remarkable progress through machine learning approaches. with foundation models (Nguyen et al. 2023, Kurth et al. 2023] Lam et al., 2023 Bi al.) 2023) Nguyen et al., (2024) now achieving state-of-the-an aht- performance in medium- raage forecasting, often surpassing traditional physics- numerical simulations (Molteni al. 1996: Bauer et al. 2015). However, current weather models
"Correspondence: sva svarambal 11 1 edu, roseyu@ucad.ed.

---

Published as a conference paper at ICLR 2026
Prompe: Presented below is data illustrating the glokal weather fields, collected rotal duration hours, with messurement taken bour intervals.
ZephyrusWorld
prstecstor, fenoustery exnutatety otintsteend seteet extrone weacher emencs
Code Block
webli "vaciabloa""
WeatherBench 2 dataset
tan tenesouture's (popetientica" tapesscic hamtdity: "Capacattrne. sseat.
JCM
Lif sareatly teimn teaioost
Zephyrus
Code Execution Server
Simulator
Based on the aforemenntoned please
prosiide ansver the following
Exccution Results
question:
Aseertain if an cxrreme warter event
presendly occurting within given time-frame, affirmatiee, detail the nature of the event, specifyying the affected country regions.
Forecaster
Climatoloyy
Geolocator
Board on the provided daea, the following carneme weathereentee Tropical syclone impacting Mesics, with the provinces Dassca, Tahasco, and Veracraz belng queifiially affected.
events are curmently underway:
Response
Figure 1: Overview: We develop ZEPHYRUS, an agentic framework for weather science. Given query, the LLM-based agent ZEPHYRUS writes a code block which is sent to the code execution server. The server orchestrates several tools to execute the code block and returns the execution results to the agent. The agent either decides to execute more code to refine its output or respond back to the user. Refer to Appendix A.3 for the full prompt.
operate exclusively on structured numerical datasets such as reanalysis data, cannot incorporate valuable alternative modalities like textual weather bulletins field station reports, and crucially, lack interactive natural language interfaces for querying or reasoning.
Weather science workflows require substantial technical expertise orchestrate complex ecosystems of tools, datasets, and models. Researchers must navigate disparate data sources, integrate outputs from multiple forecasting systems, combine observational datasets with model predictions, and coordinate between different computational environments and APIs. This dependency on extensive technical knowledge creates barriers for domain non-experts, limiting broader participation in weather science. Traditional meteorological workflows therefore require expert interpretation to translate computational outputs into actionable insights, increasing costs and limiting their utility in human the-loop decision-support systems.
Multimodal LLMs can handle data from diverse modalities and offer a potential pathway to address
these challenges. Models capable of jointly processing text with images (Wang al.) 2022 Alayrac
al.
2022 Li et al. 2022 Liu al.] (023c)), video (Zhao et al., 2002 Zhang et al. |023] Cheng
et al. Wu et al. 2024a; 2025
2024 Lin et al. 2024 Zhang et al. 2025), and audio (Chu et al. 2023 Défossez al. 2024
Doh
Doh et al. 2025 Ghosh et al., 2025) have shown impressive cross-modal
reasoning abilities. Yet atmospheric data poses unique challenges: its spatiotemporal, multi-channel structure fundamentally different from conventional modalities, requiring specialized approaches for effective integration with language models. Initial attempts to bridge this gap have shown promise but remain limited in scope. Early vision-language approaches to meteorology (Chen et al.) 2024a Li al., 20244] Ma et al. (224)) have focused on narrow applications like extreme weather prediction using restricted variable subsets, falling short of general-purpose meteorological reasoning. More recent multimodal reather-language models /Varambally et al., 2025) demonstrate the potential of this direction but still fail to match established baselines across many important meteorological tasks. This persistent gap highlights a fundamental challenge: despite significant progress in both weather foundation models and L.MMs, no existing system successfully unifies meteorological data with natural language reasoning for broad, interactive scientific applications.
We address this challenge by first introducing an agentic environment that enables LLMS to interact programmatically with metcorological data and models. We setup ZEPHYRUS W WORLD, a comprehen- sive execution environment that exposes weather-focusee capabilities through easy-to-use Python APIs. The system includes interfaces to the WeatherBench 2 dataset (Rasp et al. 2024), geo-query functionality for translating between coordinates and named locations, state-of of ofie--of-the---e the-art forecasting models (Nguyen et al. 2024). a physics- based climate simulator based on JCM (Davenport et al.) 2026), and a climatology tool that provides precomputed statisties such mean, median, extreme
2

---

Published as a conference paper at ICLR 2026
values, and quantiles from historical weather data. A FastAPI backend parallelizes code execution from -LM-ggeerated queries.
We then develop two workflows of increasing sophistication within this agentic framework. ZEPHYRUS- ZEPHYRUS-DIRECT DIRECT generates Python code a single step to solve weather problems directly framework
(Gao et al. 2023). SEPHYYUS-RRFLECTTIR
employs iterative iiecution-refineene 2023b): it executes code to manipulate weather data, analyzes the results, and refines Yao et al.
both code and output before providing a final answer. Both approaches can automatically detect and correct errors produced during code execution. Figure I gives an overview of our agentic pipeline.
To systematically evaluate these approaches, we construct ZEPHYRUSBENCEE a comprehensive benchmark built on ERA5 reanalysis data (Hersbach et al., 2020) from WeatherBench 2 (Rasp) and semi-synthetic tasks spanning 2230 question- answer pairs across 49 distinct tasks. Tasks range from basic data lookups and forecasting challenging research problems involving extreme event detection, forecast report generation, and prediction and counterfactual analysis. We also implement robust evaluation schemes to assess the scientific accuracy of all generated answers across diverse meteorological reasoning tasks. We summarize our key contributions below.
al. 2024). The benchmark combines numan-authoon
We develop ZEPHHYRUSWORLJ an agentic environment providing unified Python APIs for meteorological data, forecasting models, climate simulation, and climatology tools.
We introduce two code-generaate systems that leverage REPHYRUSWORLD ZEPHYRUS- DIRECT for single-step code generation and ZEPIYYRUS-RERLECTVVV for iterative execution-reffnemeen workflows to solve open-ended meteorological problems.
We curate ZEPHYRUSBENCH, a challenging weather reasoning benchmark with 2230 question-answar pairs across 49 meteorological task types.
Our evaluation shows that LLM agents achieve encouraging results on the benchmark, suggesting that they can be effective assistants to weather scientists.
2 RELATED WORK
Agentic frameworks for scientific discovery. Agentic frameworks implement the per- ceive-reason-plate plann-act loop by pairing LLMs with tools, memory, and feedback to pursue long-horizon goals. Core patterns include interleaving reasoning with tool calls (ReAct (Yao et al., 2023al), self-critique with episodic memory (Reflexion [Shinn et al. (2023)), and self-supervised learning of API use (Toolformer (Schick et al. (023)). Seneral-purposs libraries such AutoGen provide a standard interface for multi-agent conversation and tool invocation, making these patterns reusable across tasks (Wu et al. 2024c).
In many scientific applications, these frameworks appear as domain agents and self self-driving labs. In chemistry, ChemCrow couples LLM controller with curated set of expert tools for synthesis and analysis (Bran et al.) 2024). while Coscientist integrates retrieval, code execution, and laboratory APIs plan and experiments end-to-end (Boiko al. (023). Biomedical agents extend the approach across literature, databases, and analysis workflows (c.g., Biomni (Huang et al. (2055). Despite these advances across multiple scientific domains, weather science remains largely unexplored territory for agentic approaches.
General-Purpose Vision-Language Models. Multi-modal vision language models (Li al. 202 I
Alayyac et al., 2022 Li et al. 2022; 2023 Liu et al. 2023c b a 2024) demonstrate strong visual reasoning capabilities on general purposs evaluation benchmarks. However, adapting these models for applications in weather science presents considerable difficulties. Standard VLM architectures assume RGB visual inputs and exhibit weaknesses in quantitative analysis tasks (Lu et al.) 2024 Yue et al. (2024). Metcorological data presents fundamentally different challenges. They comprise of high-dimensional, structured atmospheric measurements which require specialized integration approaches for language model compatibility. While weather-language hybrid models (Varambally et al. 2025) seem promising, they underperform relative to domain- -aeiiiicccificcc baselines across critical meteorological applications.
Weather Foundation Models. Neural network based weather forecasting systems (Lam et al. 2023
Price et al.
2025 Bi et al. 2023 Pathak al. 2022 Nguyen al. 2023 Bodnar et al. 2024
ggyenn
3

---

Published as a conference paper at ICLR 2026
Request server/worker Rs Reqqone from sever/murier
Worker
Tool Pool 1
Zephyrus
Code Exccution
Server
JCM
Worker
Tool Pool
Code Execution Server. ZEPHYRUS sends parallel requests the server, which distributes them to available workers. Each worker acquires resources from tool pools, loads datasets, injects tools into the execution environment, executes code, and returns results errors to the agent.
Figure 2:
et al. 2024) have revolutionized meteorological prediction by demonstrating superior performance compared to conventional physics-based approaches (Molteni et al.) T996) while being significantly more computationally efficient. Nevertheless, these architectures are predominantly trained for forecasting. In particular, they do not support conversational interfaces or cross-domain, multimodal reasoning capabilities.
Multimodal Weather Datasets. Recent research has developed several multimodal frameworks that combine weather observations with textual information. These include the Terra collection (Chen et al. 2024b), which integrates geographical imagery with descriptive text for general earth observation, and ClimatelQA (Chen et al. (024a), which focuses extreme weather detection through wind measurement analysis. Similarly, WeatherQA (Ma al., 2024) specializes in severe weather interpretation using remote sensing data and expert commentary, while CLLMate (Li al.) 2024) connects media reports with ERA5 observations for weather event classification. Despite these valuable contributions, existing frameworks narrow in scope. They concentrate narrow applications or utilize only small subsets of atmospheric variables. This approach overlooks a fundamental characteristic of atmospheric dynamics: weather systems involve complex multi-scale interactions across numerous meteorological parameters. To address these limitations, our benchmark incorporates diverse weather reasoning tasks, both numan-implemeent and semi -ytthettttti generated, that span across most WcatherBench2 data channels.
ZEPHYRUS: AN AGENTIC FRAMEWORK FOR WEATHER SCIENCE
3.1 ZEPHYRUSWORLD THE AGENTIC ENVIRONMENT
The fragmented nature of weather science tools makes challenging for LLMs effectively leverage them for scientific tasks. To address this, we introduce ZEPHYRUSWORII environment that unifies weather science capabilities from diverse tools through clean Pythonic interface. Given question, we leverage LLMs' ability (Gao et al., 2023 Jimenez et al., 2024) to generate Python code and execute it sandboxed environment. The output is then fed back the model along with any execution errors. We design high-level APIs for the tools for ease of use, and include documentation extracted from the docstrings in the models context at inference time.
a comprehensive agentic
The environment encompasses several essential weather science tools:
1. WeatherBench 2 Data Indexer. The environment provides the model access to the data through
the xarray dataset interface.
2. Geolocator. This tool provides comprehensive geospatial functionality for weather data analysis. It handles forward geocoding (place names to coordinates) and reverse geocoding (coordinates to location names) using the Natural Earth dataset (Natural Earth 2024). Key operations include finding geographic features at specific coordinates, retrieving boolean masks and area-weighted maps for regions, listing sublocations, and calculating geodistances. Built using geopandas and shapely, it maintains precomputed spatial caches for fast lookups.
3. Forecaster. We incorporate the Stormer model (Nguyen et al. 2024), a transformer based neural weather prediction system trained WeatherBench 2. We chose it for its strong performance
4

---

Published as a conference paper at ICLR 2026
short to medium range forecasts while being orders of magnitude more efficient than traditional numerical models. Our implementation abstracts checkpoint loading and preprocessing, providing a simple interface to run forecasts from arbitrary atmospheric initial conditions and return outputs as xa xarray datasets.
4. Simulator. We use the JCM simulator (Davenport al., 2026), intermediate complexity atmospheric model built on NeuralGCM's dynamical core (Kochkov et al. 2024)). It incorporates physical parameterizations from the SPEEDY Fortran model (Molteni et al. 1996), including radiation, moist physics (clouds and convection), and vertical and horizontal diffusion. We use the default T32 configuration (approximately 3.5 resolution) with vertical layers. Built on JAX, we can run 5-day simulations in only ~ 25s on A100 GPU.
5. Climatology. We precompute climatological baseline products for WeatherBench variables across multiple aggregation timescales for the refernece period 1979-2000. Our tool exposes simple query interface to retrieve summary statistics such as mean, extrema, standard deviation, and empirical quantiles. Supported timescales include all-time seasonal, monthly, daily (day-of-year), and six-hourly climatologies.
Code Execution Server. ZEPHYRUS WORLD requires system capable of handling multiple weather analysis tasks simultaneously without resource conflicts. We implement a FastAPI-based server-client architecture where clients send code execution requests dedicated execution server application that processes them in parallel. The system maintains resource pools for each tool component to prevent contention and enable true parallelism. Each pool contains one more instances of the above tools. A resource manager implements acquire/relcase semantics to ensure each execution thread has exclusive access to a complete set of tools while preventing deadlocks. Each execution follows strict protocol: acquire resources from pools, load requested datasets, inject tool instances into the execution environment, and execute user code with timeout protection. The system captures all outputs and error information, which are sent back to the client for further processing by the agent. Figure provides an overview.
3.2 THE ZEPHYRUS FAMILY OF WEATHER AGENTS
We design agentic systems that leverage ZEPHYRUSWORLD solve complex meteorological tasks. We construct prompts containing comprehensive documentation of ZEPHYRUS WORLDtools, variable descriptions, units, and coordinate systems. The models generate Python functions using these tools to solve the given questions, which execute on ZEPHYRUS SETRRSSWORLDY WORLD's code execution server. Any execution errors or timeouts are returned to the models, which regenerate code until the error is resolved. We implement two distinct systems that differ in their execution strategy and refinement approach. Both systems intentionally maintain simple designs to isolate and measure the agentic capabilities of LLMs for solving weather science problems.
ZEPHYRUS- DIRECT generates a complete Python solution in one attempt and reports the execution output as the final answer. This model runs the error-correcton loop for maximum of 20 times.
REPHYRUS-REFLECHEE REFLECTIVE implements a multi- -urn workflow that alternates between code gener- ation and execution phases. The agent executes individual code blocks and receives the output as observations. The execution results are fed back to the LLM,, which analyzes the observations and decides on the next step. This ReAct-style (Yao et al. (2023b)) iterative process enables the model to assess the scientific plausibility of outputs, identify anomalies or mistakes in results, and refine subsequent code blocks to address logical errors. We run the interaction loop for maximum of 20 times per question.
The complete prompts for both systems are presented in Appendix A.3
4 ZEPHYRUSBENCHE A COMPREHENSIVE WEATHER BENCHMARK
Weather science problems require analyzing complex atmospheric patterns, modeling trends, and combining data from multiple sources. We introduce ZEPHYRUSBENCH a comprehensive bench- mark that evaluates how effectively LLMs can assist in real-world meteorological workflows. The benchmark comprises 49 distinct meteorological tasks with answers derived from curated weather reports and human-generated or verified code.
5

---

Published as a conference paper at ICLR 2026
4.1 DATASET CURATION
We design our tasks based on the ERA5 reanalysis dataset (Hersbach et al., (020), specifically from WeatherBench 2 (Rasp et al. 2024). The dataset provides global atmospheric data from 1979 to 2022. We use 1.5 spatial resolution with 6-hourly temporal resolution.
The capabilities measured by our curated tasks range from basic data lookups and computations to more advanced problems involving forecasting, challenging research problems including extreme event detection, forecast report generation, prediction analysis, and counterfactual reasoning. We design tasks with increasing difficulty levels (Easy, Medium, Hard) based on the complexity of tool usage required to answer them, from simple single-step data queries to multi-step analytical workflows. Table 2 lists the full set of tasks in ZEPHYRUSBENCH, while table providee overview of the different task types.
For each task-type, we define natural language templates with placeholders such as location, variable, and time window. To create task-specific examples, these placeholders are filled by randomly sampling inputs, and the corresponding ground truth is computed deterministically using human- written or human-verified synthetic code applied to the raw ERA5 data. Figure 7 shows an example template, and sample generated from it.
Using our framework, we construct a benchmark dataset comprising 2230 test samples spread across 49 tasks. For a detailed breakdown of dataset statisties, please refer to Appendix A.1 We provide more details about how the tasks are implemented in the subsequent sections.
Difficulty
Human-Gen. Tasks
Human--Gnn. Samples
Synthetic Tasks
Synthetic Samples
Total Tasks
Total Samples
Easy
793
Medium
4
182
29
290
33
472
Hard
200
Total
19
1,740
30
490
49
2,230
Table 1: ZEPHYRUSBENCH Statistics: Number of unique tasks and samples, grouped by difficulty and generation method.
4.1.1 AUDDN TASKS
The human- -uman--gneeated tasks span across the Easy, Medium and Hard difficulty levels and represent realistic meteorological queries curated in conjunction with a domain expert. For each task, a graduate student created question template and wrote Python code to answer the query. Easy tasks focus on basic data retrieval operations like finding extrema, querying specific values, and identifying locations with particular weather conditions. future weather predictions at specific locations and times, and/or implementing complex data analysis pipelines. Hard tasks demand substantial meteorological expertise and mirror real-world operational workflows. These include extreme weather event detection, comprehensive weather assessments, and generation of detailed forecast discussions that span regional global scales. For instance, ENSO outlook reports require synthesizing complex interactions between multiple atmospheric and oceanic variables to produce coherent, scientifically grounded forecasts. We source the -eeerrteeee weather discussion reports from several online sources, such as the NOAA website" and IRI Seasonal Climate (orreasts/Outllookf]] For extreme weather event tasks, we use records from the EM-DAT international disaster database (Delforge et al., 2025), matching event entries by date and location to the ERAS data.
tasks introduce forecasting elements, asking for
4.1.2 SEMI-SYNTHERS TASK GENERATION
To increase task diversity, we implement a semi-synthetic pipeline that transforms unstructured weather-related text into verifiable benchmark tasks. FigureBprooiids an overview of the procedure. The process begins with claim extraction agent that analyzes weather texts from NOAA meterologi- cal reports, using an L.M (specifically, gpt-4 4. to identify scientifically meaningful observational
https:
sc-
pmdepd
httpps:/iig httpo////hiiiillllaaaaa/a///arsa
seasonal imatimmateaee int.imattttt-tttte.eee

---

Published as a conference paper at ICLR 2026
Hammn Vertfol
Weather Text Corpus
Extraction Agent
Simple, Verifiable Claims
LLM
Reusable Templates
WeatherBench dataset
Haxno Vellal
Synthetic
Task
Data
Coding Agent
Verifier
Synthesized together
Figure 3: Semi-synthetic task generation pipeline: Weather-related texts processed by claim extraction agent to identify scientifically meaningful observational claims. Verified claims transformed into reusable templates and manually reviewed. Code is generated by LLM and verified by humans to validate each sample from a template against ERA5 meteorological data. We combine the verifier code with the templates and WeatherBench data to produce novel samples.
claims about weather phenomena. The agent focuses on quantifiable changes, trends, extremes, and relationships between variables.
These claims are then converted into question templates where we can substitute different locations, time periods, and weather variables generate multiple benchmark examples from each original claim. For each template, an LLM writes a verification code block that can validate any instance generated from that template against the ERA5 data. This verification step ensures that the generated questions are not only linguistically coherent but also scientifically accurate when tested against actual meteorological observations. We generate multiple candidate instances from each template through this approach. Finally, manually review them for scientific interest and code correctness. this way, we generate 30 distinct human-validated synthetic task types.
We also include semi-synthetic meteorological claim verification task to test whether models are capable of validating claims extracted from metcorological reports against the weather data; more details this task are included in Appendix A.1.2
4.2 EVALUATION METRICS
Since all our benchmark questions are designed around weather tasks with objectively correct answers, we develop an evaluation pipeline that can assess the scientific correctness the answers produced by the models. The model answers fall into the following primary categories: numerical, temporal, boolean, spatial lloation-bbsed and descriptive. Given that model outputs are in natural language, we evaluate them through a multi-stage process:
1. Extraction: Extract structured answer from the model's natural language response using LLM (gpt-4. 11044.1-mng 1-mini). The expected format varies by question type.
2. Verification: Programmatically verify that the extracted answer is well-formed. If it is not, re- attempt extraction up to three times. At this stage, we merely assess whether or not the response has valid answer the given question, and not its correctness.
3. Scoring: Apply scoring methods specific to the type of question, which are detailed below.
Numerical Answers. For numerical responses, we record the Standarized Median Absolute Error be- tween the predicted and reference values, standardized by the standard deviation of the corresponding variable in the dataset. This enables us compare across variables with different scales and units. In addition, we also report the 25%, 75% and 99% quantiles the standarized absolute error to provide more complete picture of the error distribution. We quantiles rather than means because large outliers can significantly skew mean values, obscuring typical model performance patterns.
Tise-based Answers. We evaluate tasks with time values responses using Median Absolute Error. We omit the standarization step, since all the answers are in the same units (that is, hours). Like the numerical answers case, also report the 25%, 75% and 99% quantiles.
7

---

Published as a conference paper at ICLR 2026
Boolean Answers. For tasks with binary True/False responses, evaluate predictions using F1 score and correctness.
Location-basee Answers. For questions whose answers geographic locations, first match the extracted location name to one of the expected entries from the NaturalEarth dataset (e.g.. mapping the _ootttyyccoeeeerrrree library (Stadler (017). For other geographic entities such as continents and water bodies, we apply fuzzy string matching (Bachmann et 2023), accepting matches above predefined similarity threshold.
"USAA" to "United States of America"). For countries,
To quantitatively assess the geographic deviation between predicted and reference locations, we employ the Earth Mover's Distance (EMI)) (Monge 1781) primary evaluation metric. We hegin by generating surface area-weighted masks over a reference locations. These masks are normalized to form probability distributions. To account for the curvature of the Earth, we compute pairwise distances between grid points using geodesic distance. The EMD is then calculated using the POT library (Flamary al. 200T). As a complementary metric, we also report Location Accuracy, which simply measures whether the predicted and reference location strings are an exact match.
grid for both the predicted and
Descriptive Answers. To evaluate descriptive answers, we extract individual discussion points from both the model's 's response and the reference answer. We then classify each extracted claim from the model's response either SUPPORTED, REFUTED, NEUTRAL against the reference answer, obtaining logit scores from the language model and applying softmax normalization. Similarly, we perform the same procedure for claims from the reference text compared against the model response.
We then define two complementary metrics, precision and recall. We define precision as the validity of the model's 's claims by computing the proportion that supported rather than refuted by the reference answer, excluding neutral classifications.
Eies Pmodel vee Pee Supported,) Precision Sie P modeel-ver (Supporrt)) + Eies Prmudet-rnet (Refuted,)
where 5= {i: P mmddll-sref -eii--ereeeuuttt (Neutral)) < 0.5) and Fre (enntt--er(Suppotteed (Supported,) denotes the probability that model claim i is supported by the reference answer, calculated from the model logits.
Recall measures coverage by evaluating how well the model response addresses the reference claims, computed as the average support probability across all reference points:
Recall
1 N
Σ
Pref- (Supported.)
where N is the number of reference claims and Prer- (Supported)) denotes the probability that reference claim is supported by the model answer.
2 Precision Recall
Finally, we define the discussion score as the score Precision + Recall
Extreme Weather Tasks. In order to evaluate the extreme-weather tasks, we report two metrics: (I) Fl score, which only assesses whether the model correctly predicts the occurrence of extreme event anywhere in the world, without considering event type or exact location. (2) EMD, which measures the agreement between the reference and predicted list of countries.
Correctness. For case of presentation, we define correctness criteria that vary depending on the task type. Rather than requiring exact matches, consider answer correct if it falls within acceptable range of the target response. The precise criteria for determining correctness for each task type detailed in Appendix A.2
EXPERIMENTAL RESULTS
We evaluate model performance across all task types from Section using five LLM backbones:
three proprietary models OpenAl GPT-5.2, GPT-5-Mini, Google Gemini 2.5 Flash (Comanici et al. (025). and two open- open-source models OpenAl gpt-oss-tteeeeo 120b (Agarwal et al.) (025) and Sw A3B Thi -in--iiiiinnn2500 All the thinking models are evaluated with the reasoning
8

---

Published as a conference paper at ICLR 2026
Aggregate Correctness Across All Tasks, ZephyrusBench
% Overall Correctned
58.9%
58.5%
56.2%.
56.0%
51.2%
50
40
30
20
17.0%
16.4%
10
gpl-5.2
gpt-5-mini
gpt-oss- 120b
qwen3-30b
gemini- 2.5-faash
LLM Zephyrus-Direct
Replyrus-Rellettive
Text-Only-LLMM
Figure 4: Percentage of questions in the complete dataset answered correctly by each LLM and model type. Definitions of correctness for each question type are detailed in Appendix A.2
Standardized Error Distribution for Numerical Tasks
gpt-5.2
gwen3-30b
Density
1.0
Density
0.4
0.5
0.2
0.0
0.0
10
101
102
10
2
10
101
102
Standardized Absolute Error
Standardized Absolute Error -
Zephyrus-Reflective
Zephyrus-Dire
Text-Only
Geospatial Task Performance
Report Generation Performance
% Location Accuracy
90
80
Discussion Score
0.25
70
0.20
60
0.16
50
0.10
30
0.10
20
10.2%
0.05
0.06
10
0.00
gpt-5.2
qwen3-30b
gpt-5.2
qwen3-30b
LLM
LLM Text-Only-LLMM
Rephyrus-Reflectvvs
Zephyrus-Direct
Figure 5: Plots showing (top) error distribution on numerical tasks (bottom-left) location accuracy (bottom-right) discussion scores for GPT-5.2 and Qwen3-30B.
effort set to hi gh. We compare three experimental settings: (1) text-only baseline that attempts to answer weather reasoning questions using only natural language metadata without access struc- tured weather data numerical inputs, (2) ZEPHYRUS-DIRECT. and (3) -EFFRRSSREEEE The text-only baseline measures the extent to which models can utilize their prior meteorological knowledge.
The correctness results across all models and tasks are presented in Figure We observe that the ZephyrUS agents significantly outperform the text-only baseline across all models, demonstrating the agentic framework's 's ability to effectively ground answers by leveraging meteorological data from WeatherBench. For GPT-5-Mini, ZEPHYRUS-DIRER DIRECT and ZEPHYRUS- RERHHRRSS-REER RUR-RRUSR-RRROOE REF. achieve 58.5%

---

Published as a conference paper at ICLR 2026
and 61.2% correctness respectively, compared only 17% for the text-only baseline. This substantial improvement holds consistently across other LLMs, with ZEPHYRUS agents achieving 77.-44.2%% higher conrectness than their text-only counterparts. With the OpenAl models, the multi-tum execute- observe-solution framework implemented in ZEPHYRUS- TERRRREEEEECIIEEE enables it to outperform SEPHYRUS-DIRECT by 0.8-2.7% However, these gains are not consistent, with ZEPHYRUS-DIREE outperforming REPPLLLLLLLLSSRSLLEIIIIIEEEEEIEE Qwen3-30B and Gemini 2.5 Flash.
Figure ennables fine-grained analysis of error distributions for numerical tasks, location accuracy, and discussion scores for descriptive answers using GPT-5.2 and Qwen3-30B as the LLMs. The agents particularly excel numerical and location prediction tasks, achieving substantially lower Standarized Absolute Errors and higher location accuracies compared to text-only baselines. For location prediction, ZAPHYRUS- REFLECTIVE with GPT-5.2 achieves strong performance with 90.4% SAAPHYRRSRRRFFFETTVV
accuracy. Once again, the reflective variant enjoys small benefit in performance over the Direct approach. The difference between ZEPHYRUS- DIREECT and on numerical tasks for Qwen3-30B, while both variants perform similarly with GPT-5.2.
WEWERRUSRREEEEIVVE is significant
However, all models struggle with the challenging task of generating textual weather reports. The best performing model (ZEPHYRUS- ZAEPHYRUS-REFLECTIVV with GPT-5.2) only achieves díscussion score of 0.27. Nevertheless, ZEPHYRUS-REPLECTIVV demonstrates significant advantages over both ZEPHYRUS: DIRECT and text-only variants for these descriptive tasks. While the text-only variant lacks access to meteorological information, ZEPHYRUS- DIREEC produces rigid answers by directly outputting program results, making it ill-suited for nuanced textual generation. The exeaute-observe-solutio framework in ZEPHYRUS-REFLECTIE REFLECTIVE proves more effective.
Performance breakdown by difficulty level reveals interesting patterns (detailed results in Appendices and A..4410 On easy tasks, which primarily involve data analysis questions, ZEPHYRUS agents perform well with 76.9%% correctness. Medium difficulty tasks show moderate performance with 49.3-63.5% correctness. However, on hard tasks, all models struggle significantly, with ZEPHYRUS agents achieving 14.2%-37. .7% correctness. This suggests that while current LLMs can effectively solve simple data analysis problems that arise in meteorology, they do not yet possess the capability to reason about weather phenomena directly from data even when provided with relevant tools. For generating meteorological discussions and forecasts for the continental United States, models show promise with ZEPHYRUS- REPILYRUS-REFLECTTI REFLECTIVE + GPT-5.2 achieving an average discussion score of 0.43. This contrasts sharply with global climate forecasting tasks spanning three months, where all models fail completely, highlighting the current limitations in long-term, aarge-scale weather reasoning.
5.1 ABLATION STUDIES AND ADDITIONAL RESULTS
We conduct several ablation studies to better understand agent behavior and failure modes. These in- clude analysis of representative failure cases (Appendix [A.4I]) I. tool-use and runtime error-correction statisties (Appendix [A.4.2), the impact of withholding individual tools from the agent's tool set (Appendix A.4.3), the performance upper bound when ground-truth forecasts are provided directly (Appendix A.4.4), and run-to-run variance across repeated trials (Appendix A.4.5).
CONCLUSION
We tackled the challenging problem of enabling LLMs to reason over high-dimensional weather data by developing, to our knowledge, the first agentic model for meteorology. Our contributions include: (1) ZEPHYRUS WORLD, an agentic environment with comprehensive meteorological tools, (2) the ZEPHYRUS family of agents that leverage these tools, and (3) a scalable data pipeline producing a large, diverse benchmark dataset CEEPHYRUSBENCHE
Our empirical evaluation shows that the agentic framework enables effective reasoning about meteorological data, significantly outperforming text-only baselines. The agents excel at most tasks but struggle with complex challenges like forecast report generation. Looking ahead, ZEPUYRUSWORKI candidate to incorporate new tools, data sources, and domain sppeiffc workflows, positioning it as unified platform to support related fields such as hydrology, geosensing, and climate science. More broadly, our framework serves as a sandbox for developing and refining agentic workflows beyond meteorology. Future work could leverage more diverse and larger-scale datasets to further improve the scientific accuracy and generalizability of trained agent models.
modular design makes it a good
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
Sumanth and Zachary would like to thank Chinmay Talegaonkar for several useful discussions about the work. This work was supported in part by the U.S. Army Research Office under Army-ECASE award I9F-NN--7--
the U.S. Department Of Energy, Office of Science, Google Academic Research Award. LARPA HAYSTAC Program. NSF Grants #2205093, #2146343, #21 2224444, and CCF-2112665, CDC-RFA-FT-23-002 as well as DARPA AIE FoundSci and DARPA YFA.
REFERENCES
Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss- 120b & gpt-oss-20b model card. arXiv preprint wrXhv:2508.1092 2025.
Jean- San-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc. Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning, Advances in neural information processing systems, 35:23716-23736 2022.
Richard B Alley, Kerry A Emanuel, and Fuqing Zhang. Advances in weather prediction. Science, 363(6425):342-344, 2019.
Max Bachmann, layday. Georgia Kokkinou, Jeppe Fihl-Pearson, dj. Henry Schreiner, Moshe Sherman, Michat Górny, pekkarr, Delfini, Dan Hess, Guy Rosin, Hugo Le Moine, Kwuang Tang, Nicolas Renkamp, Trenton H, glynn, and odidev. rapidfuzz/rrpidduuzz Release 3.6.1, December 2023.
URL http::ttp//sottt //oo:.//doi.o .rg/ 10. 02t.552111zzoo... 10440201
Peter Bauer, Alan Thorpe, and Gilbert Brunet. The quiet revolution of numerical weather prediction. Nature, 525(7567):47-55, 2015.
Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. Accurate medium- range global weather forecasting with 3d neural networks. Nature, 619(7970):533-338 2023,
Abeba Birhane, Atoosa Kasirzadeh, David Leslie, and Sandra Wachter. Science in the age of large language models. Nature Reviews Physics, 5(5):277-280, 2023.
Cristian Bodnar, Wessel P Bruinsma, Ana Lucic, Megan Stanley, Johannes Brandstetter, Patrick Garvan, Maik Riechert, Jonathan Weyn, Haiyu Dong. Anna Vaughan, et al. Aurora: foundation model of the atmosphere. arXiv preprint arXi::2405.13063 2024.
Daniil A Boiko, Robert MacKnight, Ben Kline, Gabe Gomes, al. Autonomous chemical research with large language models. Nature, 624:570-578, 2023.
Andres M Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D) White, Philippe Schwaller, et al. Augmenting large language models with chemistry tools. Nature Machine Intelligence, 6(5): 525-535, 2024.
Jian Chen, Peilin Zhou, Yining Hua, Dading Chong, Meng Cao, Yaowei Li, Zixuan Yuan, Bing Zhu, and Junwei Liang. Vision-language models meet meteorology: Developing models for extreme weather events detection with heatmaps. arXiv preprint arXiv:2406.09835 2024a.
Wei Chen, Xixuan Hao, Yuankai Wu, and Yuxuan Liang. Terra: multimodal spatio- dataset spanning the earth. Advances in Neural Information Processing Systems, 37:66329-66356. 2024b.
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-tent patialltemporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.074777 2024.
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang. Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07911 2023.
11

---

Published as a conference paper at ICLR 2026
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 2025.
Ellen H Davenport, J Varan Madan, Rebecca Gjini, Jared Brzenski, Nick Ho, Tien- Yiao Hsu, Yueshan Liang, Zhixing Liu, Veeramakali Manivannan, Eric Pham, et al. Jem vl. 0: A differentiable. intermediate- complelxttt atmospheric model. EGUsphere, 2026:1-20, 2026.
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.00057 2024.
D. Delforge, V. Wathelet, R. Below, C. Lanfredi Sofia, M. Tonnelier, J. A. F. van Loenhout, and N. Speybroeck. EM-DAT: The Emergency Events Database. International Journal of Disaster Risk Reduction, pp. 105509, 2025. doi: 20d1 105509. URL httptttss//oo..cco
1016/ 1. ijrr 2025. 105509
Seungheon Doh, Keunwoo Choi, and Juhan Nam. Talkplay: Multimodal music recommendation with large language models. arXiv preprint arXiv:2502.13713, 2025.
Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, levgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. Pot: Python optimal transport. Journal of Machine Learning Research, 22(78):1-8, 2021. URL URRptttp wwwp:/rrrggpppppprsseeeeeeeeee 2ttpeeerr//2222---555 html
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference Machine Learning, pp. 10764-10799. PMLR, 2023.
Sreyan Ghosh, Zhifeng Kong, Sonal Kumar, S Sakshi, Jaehyeon Kim, Wei Ping, Rafael Valle, Dinesh Manocha, and Bryan Catanzaro. Audio flamingo 2: An audio-language model with long-audio 2025.
understanding and expert reasoning abilities. arXiv preprini
Hans Hersbach, Bill Bell, Paul Berrisford, Shoji Hirahara, András Horányi, Joaquín Julien Nicolas, Carole Peubey, Raluca Radu, Dinand Schepers, al. The era5 global reanalysis. Quarterly journal of the royal meteorological society, 146(730):1999-0049 2020.
Kexin Huang, Serena Zhang, Hanchen Wang, Yuanhao Qu, Yingzhou Lu, et al. Biomni: A general- purpose biomedical ai agent. bioRxiv, 2025. doi: 10. 10.1101/2025.05.00.66677
Kevin Maik Jablonka, Qianxiang Ai, Alexander Al-Feghali, Shruti Badhwar, Joshua Bocarsly, Andres M Bran, Stefan Bringuier. L Catherine Brinson, Kamal Choudhary, Defne Circi, al. 14 examples of how Iims can transform materials science and chemistry: a reflection on a large language model hackathon. Digital discovery, 2(5):1233-122 1250, 2023.
Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. survey on large language models for code generation. arXiv preprint 240.v:240060055555 2024.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-hench: Can language models resolve real-world github issues? In The Twelfth International Conference Learning Representations, 2024.
Dmitrii Kochkov, Janni Yuval, Ian Langmore. Peter Norgaard, Jamie Smith, Griffin Mooers. Milan Klöwer, James Lottes, Stephan Rasp. Peter Düben, et al. Neural general circulation models for weather and climate. Nature, 632(8027): 10600277::000-066 1060-1066, 2024.
Thorsten Kurth, Shashank Subramanian, Peter Harrington, Jaideep Pathak, Morteza Mardani, David Hall, Andrea Miele, Karthik Kashinath, and Anima Anandkumar, Fourcastnet: Accelerating global high-resolution weather forecasting using adaptive fourier neural operators. Proceedings of the platform for advanced scientific computing conference, pp. 1-11, 2023.
12

---

Published as a conference paper at ICLR 2026
Remi Lam, Alvaro Alet, Suman Ravuri, Timo Ewalds, Zach Eaton-Rosen, Weihua Hu, et al. Learning skillful medium-range global weather forecasting. Science, 382(6677)1141--1421 2023.
Matthew Willson, Peter Wirnsberger, Meire Fortunato, Ferran
Ge Lei, Ronan Docherty, and Samuel J Cooper. Materials science in the era of large language models: perspective. Digital Discovery, 3(7:11257-1222, 2024.
Haobo Li, Zhaowei Wang, Jiachen Wang, Alexis Kai Hon Lau, and Huamin Qu. Cllmate: multimodal Ilm for weather and climate events forecasting. arXiv preprint arXiv:2409.19055 2024.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven
Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 14:9694-9705, 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre- training for unified vision- language understanding and generation. In International conference on machine learning, pp. -288-119000 PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning. pp. 17730-197422 PMLR, 2023.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui. Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. In Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pp. 5971-59978 2024.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 11134899-334911 2023c.
Haotian Liu, Chunyuan Li, Yuheng Li. Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https:/ //
lava-next//
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In The Twelfth International Conference on Learning Representations, 2024.
Chengqian Ma, Zhanxiang Hua, Alexandra Anderson-Frey. Vikram Iyer, Xin Liu, and Lianhui Qin. Weatherqa: Can multimodal language models reason about severe weather? arXiv preprint arXiv:2406. 1117, 2024.
Franco Molteni, Roberto Buizza, Tim N Palmer, and Thomas Petroliagis. The eemwf ensemble prediction system: Methodology and validation. Quarterly journal of the royal meteomlogical society, 122(529))73-119, 1996.
Gaspard Monge. Mémoire sur la théorie des déblais et des remblais. Mem. Math. Phys. Acad. Royale Sei., pp. 666-704, 1781.
Natural Earth. Natural earth hww.hh/pps/www..ttttlaleeerrbaae 2024-11-15.
2024. Accessed:
Tung Nguyen, Johannes Brandstetter, Ashish Kapoor, Jayesh K Gupta, and Aditya Grover. Climax: A foundation model for weather and climate. arXiv preprint arXiv:2301.10344 2023.
Tung Nguyen, Rohan Shah, Hritik Bansal, Troy Arcomano, Romit Maulik, Rao Kotamarthi, Ian Foster, Sandeep Madireddy, and Aditya Grover. Scaling transformer neural networks for skillful and reliable medium-range weather forecasting. Advances in Neural Information Processing Systems, 37:68740068877 2024.
13

---

Published as a conference paper at ICLR 2026
Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay. Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli,, et al. Fourcast- net: A global data-driven tigh-resolution weather model using adaptive fourier neural operators. arXiv preprint arXiv:2202. / 2022.
Ilan Price, Alvaro Sanchez-Gonzalee Ferran Alet, Tom R Andersson, Andrew El-Kadi, Dominic Masters, Timo Ewalds, Jacklynn Stott, Shakir Mohamed, Peter Battaglia, et al. Probabilistic weather forecasting with machine learning. Nature, 637(8044):84-99,, 2025.
Stephan Rasp, Stephan Hoyer, Alexander Merose, Ian Langmore, Peter Battaglia, Tyler Russell, Alvaro Sanchez-Gonzaler Vivian Yang. Rob Carver, Shreya Agrawal, et al. Weatherbench 2: A benchmark for the next generation of data-driven global weather models, Journal of Advances in Modeling Earth Systems, 1660):2003350004011 2024.
Timo Schick, Jane Dwivedi-Ye Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36:68539-68551 I, 2023.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. 36:8634-6652, 2023. URL /ttppst///oooeeeee
neur wwtprrppppppppppprpeeeeeepeeee 1ttp:/fflle///ae//200/20
Konstantin Stadler. The country converter coco
python package for converting country names between different classification schemes. Journal of Open Source Sofhware, 2(16):332, 2017. doi:
10.21105/joss.0033 00332. URL /ttp::://o.....r/1...2. /tt.://000.11..../......./..//.. 00332
Christina Theodoris, Ling Xiao, Anant Chopra, Mark Chaffin, Zeina R Al Sayed, Matthew C Hill, Helene Mantineo, Elizabeth M Brydon, Zexian Zeng. X Shirley Liu, et al. Transfer learning enables predictions in network biology. Nature, 61867965)6116-64,, 2023.
Sumanth Varambally, Veeramakali Vignesh Manivannan, Yasaman Jafari, Luyu Han, Zachary Novack, Zhirui Xia, Salva Rühling Cachay. Srikar Eranky, Ruijia Niu, Taylor Berg- Kirr-Kiik,atrick, Duncan Watson-Parris, Yian Ma, and Rose Yu. Aquilon: Towards building multimodal weather LLMs. In net/
ICML 2025 Workshop on Assessing World Models, 2025. URL
Eorum??d=KVxOWEYA
Lei Wang, Shan Dong, Yuhui Xu, Hanze Dong, Yalu Wang, Amrita Saha, Re-Peng Lim, Caiming Xiong, and Doyen Sahoo. Mathhay: An automated benchmark for long-context mathematical reasoning in Ilms. arXiv preprint arXiv:2410.04696 2024.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. In International Conference on Learning Representations, 2022.
Junda Wu, Zachary Novack, Amit Namburi, Jiaheng Dai, Hao-Wes Dong. Zhouhang Xie, Carol Chen, and Julian McAuley. Futga: Towards fine-gran fine-grained music understanding through temporally- enhanced generative augmentation. arXiv preprint 24Xiv:2407.20444 2024a.
Junda WU, Zachary Novack, Amit Namburi, Jiaheng Dai, Hao- Heo-WennDo Dong, Zhouhang Xie, Carol Chen, and Julian McAuley. Futga-mir: Enhancing fine-grained and temporally-aware music understanding with music information retrieval. In International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2025.
Kehan Wu, Yingce Xia, Pan Deng, Renhe Liu, Yuan Zhang, Han Guo, Yumeng Cui, Qizhi Pei, Lijun Wu, Shufang Xie, et al. Tamgen: drug design with target-aware molecule generation through a chemical language model. Nature Communications, 15(1),3360, 2024h.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang. Shaokun Zhang. Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen gen LLM applications via multi-agent conversations. In First Conference on Language Modeling, 2024c. URL
bet/forum?ic
BAakY
14

---

Published as a conference paper at ICLR 2026
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023a.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICIRR, 2023b.
Xiang Yue, Yuansheng Ni, Kai Zhang. Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9556-9567, 2024.
Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501. 2025.
Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction-tuned audio-visual language model for video understanding. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 543-553, 2023.
Xiaokang Zhang, Jing Zhang, Zeyao Ma, Yang Li, Bohan Zhang, Guanlin Li, Zijun Yao, Kangli Xu, Jinchang Zhou, Daniel Zhang-Li, et al. Tablellm: Enabling tabular data manipulation by Ilms in real office usage scenarios. arXiv preprint arXiv:2403. 19310, 2024.
Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar. Learning video representations from large language models. In arXiv preprint arXiv:2212.04500 2022.
Yizhen Zheng, Huan Yee Koh, Maddie Yang, Li Li, Lauren T May, Geoffrey Webb, Shirui Pan, and George Church. Large language models in drug discovery and development: From disease mechanisms clinical trials. arXiv preprini 2xXiv:240900488/ 2024.
A APPENDIX
A.1 DATASET DETAILS
detaels all the tasks in ZEPHYRUSBENCH, and reepors the number of samples generated grouped by difficulty and type.
For weather tasks, we leverage the ERA5 reanalysis dataset (Hersbach et al. 2020), specifically from WeatherBench 2 (Rasp et al., 2024). which provides global atmospheric data from 1979 to 2022. We use 1.5 spatial resolution with 6-hourly temporal resolution, and include 4 surface variables and atmospheric variables at 13 pressure levels. For each task-type, we define natural language templates with placeholders such location, variable, and time window. For example, Task 1 is defined as 'Which average (variable] ??". To create task-specific examples, these placeholders are filled by randomly sampled inputs. Ground truth answers are obtained by deterministic procedure: we apply uuman-written or human- verified synthetic code the raw ERA5 data and other supplementary data.
experi
the (extremum _d rect i inn)
A.1.1 HUMAN-GENERATED TASKS
Tasks through rely entirely on the raw ERA5 data. Tasks 1-5 include basic data lookups and computations, while Tasks 6 and 7 involve simple comparisons with the climatological quantities. Tasks 8-9 are forecasting questions; however, we use the ground-truth answer as the response while curating the dataset. We extensively use the Geolocator tool map natural language names masks that are applied to spatiotemporal data.
For Tasks 11, 12 and 48, which involve extreme event detection, international disaster database (Delforge et al., 2025), matching event entries by date and location to ERA5 data. We consider the metereological events, comprising storms, heat and cold waves.
records from the EM-DAT
15

---

Published as a conference paper at ICLR 2026
ID Natural Language Description
Answer Type
Difficulty
Type
Which goographic teature experienced the highest/lowest average value of weather variable
Location
Easy
Human
2 What is the mnin/maxvvernge/eedee value of a weather variable at specific location
Numerical
Easy
Human
3 Which sublocation has highest/lowest recorded variable value
Location
Easy
Human
4 How many hours from start did a lncation experience extrermum
Temporal
Fasy
Human
What is the weather variable value location specific time Identify geofeatures where variable differs from mean by N standard deviations
Numerical
Easy
Human
6
List locations
Fasy
Human
Identify geofestures where variable lies outsäde climatological percentile envelope
List locations
Easy
Human
8
What will the variable be at location after time interval (forecast)
Numerical
Medium
Human
9
When will location experience its extremum in future period (forecast)
Tempural
Medium
Human
10
Which geofeatures are experiencing exceedance in variable values
List locations
Medium
Human
11
Identify extreme weather events that will
the next. hours (forecust)
List locations
Hard
Human
12
Check if extreme weather events are currently happening
List of locations
Hard
Human
13
Which goographic features experienced unusual weather anomalies compared to bascline
List of locations
Medium
Human
14
Does maximum weather variable occur at same or adjacent grid point as another variable (forccast)
Yes/No
Medium
Synthetic
15
Does maximam weather variable region remain lower future maximum (forecast)
Yes/No
Medium
Synthetic
16
Does maximum weather variable occur higher latitude than in another region (forccast) Does mean weather variable in one region exceed another hy specified amount (forecast) Does mean weather variable exceed threshold while maximum of another stays below
Yes/No
Medium
Synthetic
17
Yes/No
Medium
Synthetic
18
Yes/No
Medium
Synthetic
(forecast)
19
Does mean weather variahle within region exceed specified threshold (forecast) Does weather variable exceed threshold within any part of region (forecast) Does weather variable exceed threshold in more grid points in one region than another (forecast)
Yes/No
Medium
Synthttic
Yes/No
Medium
Synthetic
21
Yes/No
Medium
Synhhttic
22
Docs area where weather variable excoods threshold cover more than percentage of region (forecast)
YesNo
Medium
Synthetic
23
Does arca-averaged weather variable excced threshold while another stays below (fore- cast)
Yes/oo
Medium
Synthetic
24
Does maximum weather variable in one region exceed threshold while another stays below (forccast)
Yes/No
Medium
Synthetic
25
Does maximum weather variahle within region exceed specified threshold (forecast)
Yes/No Yes/No
Medium
Synthetic
26
Does maximum weather variable occur at latitude farther north than in another region (forecast)
Medium
Synthttic
27
Does muximumn weather variable stay ahove threshold while amother stays below (fore- cast)
Yes/No
Medium
Synthttic
28
Does muximum weather variable in region exceed another by specified amount (Torecast)
Yes/No
Medium
Synthttic
29
Does minimum weather variable within region remain above threshold (forecast)
Yes/No
Medium
Synthetic
30
What is the area where multiple weather variables exceed their percentile values (forecast)
Numerical
Medium
Synthetic
31
What is the
where weather variable excoeds its median value (forecast)
Numerical
Medium
Synthetic
What is displacement berween centroids areas with ahove-median values (forecast)
Numerical
Medium
Synthetic
33
What the distance between controids of maximum weather variable value arcas
Numerical
Medium
Synthetic
(forecast)
34
What is the maximum difference in weather variable between grid points within region
Numerical
Medium
Synthetic
(forecast)
35
What is the minimum weather variable where another variablle exceeds median (forecast) is the difference between maximum weather variables in two regions (forecast) What is the difference in arca-weighted mean weather variable between two regions (Forccat))
Numerical Numerical Numerical
Medium Medium Medium
Synthetic
Synthetic Synthetic
37
38
What is the displacement of minimum weather variable location after time window (forecast)
Numerical
Medium
Syathetic
39
What is the latitudo difference between controids of high weather variablc areas (forecast)
Numerical Numerical Numerical Numerical Description
Medium
Syathetic
40
What is the maxinum weather variable difference between two regions (forecast)
Medium
Synthetic
41
What is the mean weather variable where another variahle excends mediaan (forecast)
Medium
Synthetic
42
What is the weather variable value where another variable reaches maximum (forecast) Generate comprehensive glohal climate forecast for temperature and precipitation for
Medium Hard
Synthetic
43
Human
next 3 months (Forecast)
44
Provide detailed meteorological discussioo and forecast for continental United States (Torecast)
Description
Hard
Human
45
Generate ENSO climate update and outlook based on atmospheric data (forecast) How will weather variable change after specilied time given intervention (counterfac- tual)
Description Numerical
Hard Hard
Human
46
Human
47
What the value of the input parameter of the simulator model that produces the simulation output
Numerical
Hard
Human
48
Where is target disaster currently happening
List of locations
Hard
Human
Check whether the given claim extracted meteornlogical report supported by the data
Yes/No
Synthetic
Table 2: Complete set of Weather Tasks, grouped by difficulty.
The anomaly detection tasks (Tasks 6, 10 and 13) capture the model's ability to compare global data with precomputed quantile statistics derived from a historical reference period and accessed through the Climatology tool. Locations where the recent values significantly exceed or fall below the reference quantile are flagged as anomalous. We then use the Geolocator to map flagged grid points to natural language region names.
Report generation tasks (ID 43, 44, 45) are designed to evaluate forecasting and interpretation capabilities based on ERA5 atmospheric datasets. They all use global weather fields over the given
16

---

Published as a conference paper at ICLR 2026
time duration context. Task 43 requires generating a comprehensive global forecast report for temperature and precipitation for a three month horizon into the future. The task instructs the report to be structured into separate sections for precipitation and temperature, and to provide region-specific forecasts. Task 44 focuses on the continental United States, where the model must provide a detailed meteorological discussion and forecast, including current weather system positions and movements, temperature trends and expected changes over the coming days, precipitation patterns and likelihood of significant events, pressure system evolution and impacts, and notable atmospheric features such as fronts and jet stream positioning. Task 45 requires an ENSO (EI update and outlook. Models are tasked to analyze atmospheric variables to assess the current ENSO phase, evaluate strength and persistence indicators, forecast evolution over the next 3 6 months, and discuss global implications using uncertainty-awarr language and standard ENSO terminology. Ground truth reports for these tasks are obtained from authoritative government and research institutes, which provide validated assessments of global and regional climate outlooks and ENSO conditions. The answer sources for these three tasks are NOAA Global Climate Reports, NOAA National Weather Service Area Forecast Discussions, and WMO ENSO Reports, respectively.
Oscillation) climate
Task 46 and Task 47 both rely on the JAX-GCM simulator, an intermediate-coopple model built on NeuralGCM's dynamical core (Kochkov et al.) 2024). Task 46 assesses the causal impact of localized perturbations on atmospheric states. To obtain each specific sample, a variable, location, and perturbation magnitude are first sampled, and a Gaussian mask is applied to induce the desired perturbation the chosen location. The simulator is then run twice, once starting from the unperturbed initial state and once with the imposed perturbation. At the specified simulation end time, the target variable from both simulations is extracted and compared, with the difference quantifying the perturbation's 's impact.
atmospheric
Task 47 is a black- optimization task based on the simulator. The input consists of two components: (i) a segment of recent global data spanning a specified duration and interval, and (ii) simulated data generated by the simulator. In the simulation, we vary one input parameter of the model by sampling its value randomly from the range [0, 1]. then save the resulting simulation output. The objective of the task is to estimate the original value of the underlying input parameter from observable simulation outputs. Since the climate simulator is presented as a black box, the model must infer the parameter solely from the input-outpun mapping, which can be highly nonlinear and sensitive to small parameter changes. By evaluating the model on novel simulator outputs, we benchmark its general handling of a domain -specfffc optimization problem. Performance is assessed by comparing the predicted and ground-truth parameter values.
A.1.2 METEOROLOGICAL CLAIM VERIFICATION
In the Meteorological Claim Verification task (Task 49), extract metereological claims from NOAA monthly meteorological reports (1988-2024) into timestamped text using an LLM (gpt-4 4. 1 1-mini). We then select individual claims and pair them with the 24-hour slice of Weath- erBench2 data corresponding to the report's date. Negative instances are generated by systematically negating claims using an LLM. All examples are human-verifies to ensure clarity, verifiability, and correctness of negation. Figure demonstrates an example from this task.
A.2 DEFINITION OF TASK CORRECTNESS
Different task types in ZEPHYRUSBENCE ENCH are evaluated using relevant metrics. To create a unified definition of correctness, we employ the following requirements for each metric type:
Numerical: Standardized difference lύ-vl < 0.05, where is the standard deviation of the relevant task variable in the WeatherBench2 dataset.
Dissaneeevre//Coortntteemmaaaa Relative error
lø vl lul
0.05. For true values of O,
we require [0] < 0.05.
Location: Exact locations string match, using fuzzy string matching logic.
Extreme Weather/Anomaly Earth Mover's Distance (EMD) score < 100 km. If both true and predicted values are empty lists, the answer is considered correct.
Boolean: Exact match between model answer and ground truth boolean value.
17

---

Published as a conference paper at ICLR 2026
The following data shows meteorological conditions over 24 hour period:
The following data shows meneorological conditions hour period:
Ptype': wbZ', vvariales'' ['mean_ses _ew_leeelpeeessuue pressure'
Ptype' wbZ, 'variables' ['mean_sets Je.seellreresur''' '10=_u_cen sccmmoonnnn offtttt___ffoeeee of_w,,, '10m Y_ "_componeett.... _f_________ "i_dind,,, remperature,, geoppoenntill, "ppaific_hhmidiry ''emperature, u, component, of wind', component of wind']. time indices' "7917979188:11 'start jdx': 79179)
'10m_ _t_com_onent_cc_cm of wind',
'10m_ "_component, of wind', "2m temperature', poopotential, 'specific_ Sate__hityyyy. 'emperatune', un component, of wind', 'v _.ompponnt, of wind"), time indicess: 47700:44701::1; start start_idx' idx': 47007)
following question: Doss this data support provided metcorological claim? Answer with True or False.
Based the provided data,
Based on the provided data, answer the following question: Docs this data support the provided metcorological claim? Answer with True or False
Claim: very strong in excess of 150 knots will be across the central U.S. with fivorable left exit region the New England
Claim: Dry conditions with minimal precipitation expected across western Washington and into Columbia.
Positive Clamm Type
Neguive Claim Type
Original Clam:: A very strong in excess knoas acrous the central U.S. with favomalle left region the over New England.
Original Claim, Huwv precipitatinn expected Washinntto Beitish Columbis.
Figure 6: (left) Positive claim and (right) negative example for meteorological claim verification
'hee following data shows snapshot of the global weather (dats)
The fallowing data shows snapshor of global weather fields.
("espe's "abs", toseisbler's l'raan tite copponen sindt, 130e, pooppmmntt sind', iin tespes sturet, "peopetential"" "spactrie busidity", toonperstire ture, te to component of vineta tv conponent or wond ttime indiowats "s4746:26147c11)
Based on the above data, answer the fillwing question:
Which (gcofcature) experienced the (extrmmmm, direction] average (variable]? "" "Based on the provided dama, (answer) experienced the (sstremum, dircction] average (variable) over the specifid tine- period, with an average (variable] fanwwer numeric)."
Based the above dara, answer the following question: Which continent experienced the highest sverage Surface remperatur??
Basod on the provided data, Africa experienced the highest average Surfice temperature over specified time-period, average Surface temperature 303.5 K.
Example Template
Generated Sample
Figure 7: (left) Example template from which samples are generated and (right) sample generated using the template.
Discussion: Overall discussion score > 0.5.
Time: Exact match required (absolute enror 0.0).
A.3
MODEL PROMPTS
We the following core Instruction prompt for REFEERRTUUVRREECEIIVE
18

---

Published as a conference paper at ICLR 2026
Repfytus-Reflective Instruction Prompt
You are an AT weather expert agenz. You will use an Interact ive onding envi ronment with funstions, data,
user's
each turn, you should Firat provide your thinking and reasoning given the conversation hietery Iwhich might include output from executed code within Aftar Snseerattcnn<<cbberreess
do exactly 11 Write code based on problem and/or ooservation Your code should be enclosed using "sexecute>" Lagr
of the following:
example; You must end the code black with </execute> tag.
seturn "Ieelo World! </execule>, IMPORTANT:
When you think you have solution ready, directly provide solution that adheres to the required format for the given cask the
Your solution should be enclosed using "Esolution>" tag, for example: The answer is <solution> 4/solution>. IMPORTANT:
end the solution block with </solution> tag, When anawering numerical ques:ions, always use SI base unit (atandard units of problem specifically some quest ions require to answer hours. Erclose ONLY the rinal anawer the question in these Lags, do NOT include any other infermation.
measurement) unless
example,
In each response, you must include <execute> or <solutior> tag. Not both at the same output
Do not gererate code csolution>, Do not respond with measayes without any Lags, No empty messages.
<execute>,
outaide
Geolocator Documentalion:
The detailed documentation for the Usolocator class, including available methods, is 4 provided ded below:
(eeclostto__dccmmntttiiee
Porecaster API Jocumentation:
Corecaster__dccumnnttee
The Forecaster can reliably forezast mest 2 weeks into the future.
IXFORTANT: the questien about future, you object
toxs use Forecaster
the question and solve the task. the input data =wwill not.* contain the ansker to questions about the future.
Simulator Documentation:
taimulator_ddeummntattti
The simulator provides stmospharic modeling and can usad for climate simulations, answaring counterfactuala, sensitivity atuctes, generating syathet weather
dalá.
years; NOT create loops er multiple simulator instances. Set total_time to the desired curation and call simolatel) once.
can extended periods
call.
VariaoLe Descrigtions:
comprehensive description every contained xarray given [var_desc]
here:
Dataset Keya Exp anation:
Aa explanation of what each key in the datasets represents is provided below: (keys] inuntinued!
19

---

Published as a conference paper at ICLR 2026
Zepletus-Reflective Instruction Prompt (cont.)
(cont (nsed)
Unita:
Always the following SI units when seasoning and coding: (units_desc)
Answer in SI urits unleas the problem specifically asks for different unit. 4 example, some questions may require hours.
Ti Trcices:
You should N00 slice the provided dataset according to the time indices. The datasets alresdy sliced the ccrrect time indices. For any question that aaks about the time offset, only provide the time indices relative to the provided
If the question. aska for the time offset, return the anawer in houzs from. the initial index.
For example, if the queation asks about dataset with time interval 6 houra and time indices 12345:12351:1, and you think the answer is index 12350, you should return 30 hours.
Do NOT saturn the time index zo a timastamp or datstime objact.
Loecution code :equicementsje*
The code BJST
defined
function calied 'run'.
The "run" Cunction should accept Tour parametera:
s. list of one or more xarray datasets.
Geolosator objezt Iwhich
with
of predefined helpful functiens).
G. Forecaster abjeat twhich comes with a set of predefined halpful functions). with
predefined helpful functions). 30 NOT write any code outside of the "rua" funetien.
Simulator object (whics
IMPORTANT:
The Geolocator object is already construczed and passed in as "geclocator'. "xr.cpen_éataacee
**eover open files,
import Geolocator.
Tf' you are acbsetting, make sure to aubset carefully constderimg runztrre Tt la too the
you are subsetLing oves multiple cimensions (e.g. spatially and temporallyl, make sure to apply the snaller subset operation first.
xarray
code should clearly the provided datasets and tools to produce the correct result.
follewing these detailed instructions,
Coordinate System:
ReatherDench2 (WB2)
uses equiangular grid with following
spesifications:
Laticude: grid points ranging -90 +90* 1.5* incremenzs
tongituce: 240 grid points ranging from 0* to 358.5" in 1.5* increments The latitude coordinates are: [-90, -88.5, -87, .... 87, 88.5, 30] [2, 1.5, 3, .... 355.5, 357, 358.5]
The longl coordinates
v+0ther Roquirements::*
Jnner rcumstances should you loop over grld pointa 11.e, you should Nor loop latitudes and longitides!, bus rather try leverage vectorized operations, built-in functions or the Geolocator class as appropriate. This is a kay rement ANYWHERE in your generated code, Ensure that you call and use the functions from the Geolovator objecs correctly as per documentation.
bO NOT loop
latitudea
..Question:** [question)
For the reflective stage of ZEPHIYRUS-RERLECIVVE, we use the following Observation prompt:
RefferuseReflectivee Observation Prompt
The executed code produced the oucput above. Reason about your next step and either (1) output the final result based this observation. Enclose your Lsnlution></ao> tons tags., or (2) generste another code block to execute, Rrolose texecute><<execucee Lags.
in
youz If you choose to give solution, enclose ONLY the finsl aaswer to the questien in these include any other
tags, You shou' é exacute code if you think you need more information bafore providing final anawer.
For ZEPHYRUS-DIRECT, the following direct Instruction prompt:
20

---

needed inside your Published as a conference paper at ICLR 2026
Zephyrus-Direct Instruction Prompt
Your objective la to write Python funation cal led 'run' that so solves apect fied APIs. furction should designed according to the foll fowlowing guidelinea:
problem using provided
Function Definition:
The function must named
Tt should accept four parametera:
xarray datasele.
5. Geolocator object (which comes with set of predetined halpful funot fone).ens)...
G. Porecaster objest Iwhich comes with a set of predefined helpful Iunctions), predefined helpful functions).
Simulator object (which with
Data nescri potonsp
Variable Descriptions:
A comprehensive descrigtion of every vasiable contained in the xarzay datasets is given
here:
(var_desc)
Dataset Reys Exglamation:
explanation of what each key in the dataseta represents is provided below: (keys)
Units:
Always use the following SI units when seasoning and coding: (units_dese)
Time Indices:
The dataseta provided have been conversed from using time dimenaton so st simple integer step represents hours in the original
indices starting dazaset.
Not slice provided
according
provided indices.
dataseta siready sliced to the correct indices. Foz any question that aaks about the time offset, only provide the time indices relative to the provided datoset.
If the question. asks for the time offset, you should return the answer in hours from the index.
initial
Foz example, if the questien asks about datases with time interval hours and indices 5, you return hours.
0:6:1, and you
tha answer
Do NOT return the index directly.
Toolset
are given access to following "run" function:
Geolocator Documentation:
detailed documentation Geolocatoz class, including available methods, provided below:
boonloestor_ddccuantattet
Forecaster APT Tocumentation:
Forecaster can reliably
most weeks future.
IMFORTANT: of the question is about the future, you kouill need toxs use the Forecaster
object anawer the question and solve the task. The input data rwilll not+. contain the ansver questions about the 4 future.
(continued)
21

---

Published as a conference paper at ICLR 2026
Zephyrus-Direct Instruction Prompt (cont.)
(cont (nse))
Simelator APT nocumentat:ion:
The Simulator providea atmosphoric modeling and usad for alimate simulations, anawering countertactuala, sensitivity atucles, or generating synthet weather
The Sinulator can handle extended time periocs (monhhs years) in STNCLP call, no NOT create loops or multiple sinulator instances. Set Lotal_time to the desized curation and sall simulate!)
4.
Task Detaile:
The function should procesa the datasets using the pertinent variables specified within the question.
Jnder circunstances should you
the points li.e. you should NoT loop over lstitudes and longitudes), but rather try to leverage vactorized operations, Geolocato"
built-in Functions
appropriate, This key requirement. DO NOT loup over the latitudes and longitudes ANYWIERE in your genezaled code. functions
Ensure that you ita documentation.
objecz correctly per
Keturning the
final should returned by function.
Make sure to encapsulate your run function in triple backticks for elarity. For example:
def run(...):
"Halle"
Lime rather timestamp or datetime object. For example, return '5 hours" instead of 2022-01-01 05:30:00',
value,
reture
Always the in the unit the used in the wescherbench dataset.
Lo not convert any units.
6. Problem Statement:
fellowing these detailed instructions, your code should clearly use the previded
datasets Toolset APIS produce the carrect result.
The specliic question that function needa to answer 1s provided as the end of this prompt: (question)
We use the following prompt for the text-only LLM:
Text-Only LLM Prompt
Provide answer to the following queation. not complain about ineufficien: data. brief, end provide a 1-2 sentence answer. Always provide your
in SI units. question. mate answars.
Do not g' ve approx maprox'mate
Lime offset, you should retura answer
initial time index.
For example, if the question asks about a datase: with time interval houra and indices 12345:12351:1, and you think the hours.
is index 12350, you should return 30
time index Provide a concrete final answer, It is unacoeptable to aay "Kithout daca I cannot answer question", or, "I sorry cannot provide this answez". iquestion]
timestamp datatime objact.
22

---

Published as a conference paper at ICLR 2026
Based provided data, the following eatreme weataer ereutfe) might expected
An eurreme weather event forec ssted an cuto tropical storm (soup kow is projected devekgr track across North Pactiic into the Bering Sex within the nest 92 hours.
Storm hupann happen the next 42 hours
country Umited Seates Areeicaa in approsincatly the net 42 to 66 hours. Spocifically the regionls) that might get affected are. Bevamont, Fort Arthur arew (Jeffarson dierics, Texas prosinee). Suga Land (Fort Bend diariit, lesss province), Smith disrrict (Misissipi provinee), Dallas, Harris distries Texss provinoe), Los Angele cistrict (California province), Alabema, Arkansas, Flonda, Goopla, Loateans, Marylan,, Missari, North Carelina, OCahomm, South Candina, limmae,, Virginis, Wos Viggina prwinoss
Countris/tegion
the stom''s path:
Unittd States of America Alask (cspocially the Alcatian Islnnd sontweor//seel Alssk potenrial impacts excending towzrd the Gulf of Alaska).
Rusix Kamchatka region (poteotial imspaces
Bering Sea, North Pacitic Occam, Prcife/ Bering Sea side).
Primary athocted
Gulf of Alaska.
Ground Truth
Model Answer
Figure 8: Example failure mode of REPPIYRUS-REFLECTIE (GPT-5.2) on the extreme weather event detection task. The expected ground truth (left) is storm in southern USA, while the model (right) predicts a storm in Alaska and Russia.
A.4
ADDITIONAL RESULTS
A.4.1 EXAMPLE FAILURE MODE
We illustrate an example failure mode of ZZAHYYUS-REFLECTTO (GPT-5.2) on a challenging extreme weather event prediction task. The task is forecast whether any extreme weather event will develop in the 42 hours following the end of the data window (December 13, 2003, 06:00am UTC), and to identify the affected countries and regions. The ground truth is that a general storm is expected to affect large swathe of the continental United States, including Texas (Jefferson, Fort Bend, Harris, and Dallas districts), California (Los Angeles), Mississippi (Smith district), and the states of Alabama, Arkansas, Florida, Georgia, Louisiana, Maryland, Missouri, North Carolina, Oklahoma, South Carolina, Tennessee, Virginia, and West Virginia, with the event expected approximately 42 to 66 hours out. Figure 8 illustrates the expected and predicted model answers.
We analyze the agent's behavior across its four turns to understand how it arrived at its final answer.
Turn 1. The agent uses the forecaster to produce 7-step (42-hour) autoregressive forecast using the Forecaster tool. At each step, it extracts the global MSLP minimum and the global maximum 10m wind speed, then checks whether these exceed hardcoded absolute thresholds (MSLP < 98,000 Pa and wind 2 25 m/s, or wind 2 30 m/s). No extreme event is detected, as wind speeds peak at approximately 21 m/s, just below the threshold.
Turn 2. The agent retries with loosened thresholds, now flagging a storm if MSLP < 96,000 Pa and wind > 20.8 m/s, or if wind 2 25 m/s any point. While the MSLP values (peaking approximately 95,200 Pa) satisfy the pressure condition, the global wind maximum of 21 m/s does not co-occur with the deepest pressure at the same timestep, and again no extreme event detected.
The agent switches to a climatology-relatiee criterion. It retrieves the all-time 1st percentile MSLP from the climatology object at the grid point nearest to the forecast pressure minimum, and flags an extreme event if the forecast falls below that local threshold. This criterion is now satisfied (forecast 95,220 Pa vs. climatological 1st percentile 96,839 Pa), and the storm classified as an extra-tropical storm tracking across the North Pacific into the Bering Sea. However, when the agent samples the gale-wing wind footprint (wind 2 17 m/s) to identify affected land regions, finds only water bodies: the North Pacific Ocean, the Norwegian Sea, and the Labrador Sea.
Turn 3.
Turn Noticing the absence of any land regions in its output, the agent attempts to recover by manually probing set of hardcoded coordinates near the storm track, covering the Aleutian Islands, southwest Alaska, and the Kamehatka Peninsula, using the Geolocator. This returns Alaska (United States) and Kamchatka (Russia) as the nearest land regions, along with the Bering Sea and Gulf of Alaska as relevant water bodies.
Final Answer. The agent incorrectly reports an extra-tropical storm centered over the Bering Sea, peaking at 24 hours, with Alaska and Kamchatka identified as the regions at risk. The ground truth answer, general storm affecting more than 15 states across the continental United States, had a comparatively modest pressure signature and never registered as the global MSLP minimum during the forecast window. Because the agent's detection pipeline was anchored throughout the
23

---

Published as a conference paper at ICLR 2026
Agumt gpb-522 quolocator- Mous Toal Calls per linkk Senchmark Question ID
700
Question
Agent gpt. forecaster
per
Menn calle per tank.
Beachmark Question 1D
Aguni Aguni-opt-S opt-5.2 aimulaler- Mear Toal Calls per Task by Benchmark Question ID
Moxan tood calls pour tank
Agent clmatology
Tool Denchmark Onestion
Figure 9: Tool usage statistics for ZEPHYRUS REFLECTIVE (GPT-5.2) grouped by Question ID.
Muddl
Total
Calle/Takk
Clonectal
Altemppi
152,273
ZunEvRUs-Rmm RerVEnC--R...
Rxeeaster
Simulzaor
41.0
4.9
2,682
1,082
0,5
1.2
7.6
170
1.4
Climatolgyy
20.1
3,535
1.6
ZenevaLa DUxECT
Porecaster
Simulator
48.1
5.5
1,060
321
0.0
0.1
6.1
137
1.1
Climatolayy
25.9
3,972
1.5
Table Tool usage/calls and aggregate runtime error statistics for ZZEPYRUS--RELEECTIS REFLECTIVE (GPT. 5.2) on ZEPHYRUSBENCH.
single deepest pressure center on the globe, the far more consequential continental system was never examined.
A.4.2 TOOL USE AND ERROR-CORRECTIO STATISTICS
In this section, we examine the tool-usage patterns of ZEPHYRUS and examine the run-time error correction statisties. Table 3 details the aggregate tool-use statistics and runtime-error correction
24

---

Published as a conference paper at ICLR 2026
Tools
% Correct
Easy
Medium
Hard
Location Ace
EMD
Valid %
All Tools
56.7%
86.3%
62.0%
28.7%
76.8%
1,472
99.2%
Exclude Climatology
50.8%
74.1%
58.6%
27.2%
74,8%
2,870
97.9%
Exclude Forecaster
44.3%
84.9%
28.4%
16.0%
73.7%
1.744
93.4%
Exclude Geolocator
37.2%
42.9%
49.5%
27.0%
22.2%
4,497
98.5%
Exclude Simulator
55.6%
87.0% 13.4%
62.5%
25.4%
73.7%
1,489
96.9%
Exclude Datusct, Forecaster, Simulator
18.6%
37.0%
15.2%
29.3%
4,922
91.2%
Table 4: Tool ablation study on ZephyrusBench.
Tools
% Correct
Valid %
All Tools
33.3%
100.0%
Exclude Climatology
38.1 %
100.0% %
Exclude Forecaster
0.00%
71.4%
Exclude Geolocator
33.3%
100.0% %
Exclude Simulator
38.1% %
100,0 %
Exclude Dataset, Forecaster, Simulator
0.00%
90.5%
Table 5: Tool ablation for Template 8: Forecast Value After Time Interval.
statistics for ZEPHYRUS--DIRFE and -EFERRRUSRRFEEEIIVEEE on ZEPHYRUSBENCE Figure 9 details the average number of tool calls segmented by question ID..
As expected from tool calls and uses more turns average than ZEPIYYRUS-DIRECT Notably, despite significant runtime error rate (6.167.%% tasks encountered at least one error), both ZEPHYRUS DIREET and REPHYYUS-REPLLCTIVE REFLECTIVE successfully self-correcter in all cases, with task exceeding the 20-attempt limit.
drection-obseeraliie answeer loop, ZEPHYRUS- REEERRRTVVEEEEEEVVE makes more
Examining the tool use segmented by question ID, observe that most of the tool use aligns with the expected behavior. The Geolocator used for most of the tasks (>75% for REPHYRUS-REFLECTEE and almost 90% of the tasks for ZEPHYRUS SEPHYRUS-DIRECT) For the remaining tasks, observed that the LLM uses an approximate memorized latitude and longitude to answer questions. We find that the forecaster and climatology tools are extensively used in tasks 43, 44 and 45, which free-form report generation tasks with no preset procedure. One notable exception is task 46, counterfactual question answering task, where the simulator is invoked, explaining the model's poor performance on that task. Overall, these patterns suggest that ZEPHYRUS makes reasonable tool selection decisions across most of the benchmark.
A.4.3 ABLATION WITH EXCLUDED TOOLS
To test the effect of different tools on model performance, we evaluated SEOPHYRUS-REFLECTIE with a "leave-one-ous test for each tool. We also evaluated the model with only the climatology and geolocator available, as baseline to test how well the REPHYYUSBENCHHqeStie answered from historical climatology. Test runs were conducted with LLM got gpt-oss-1220 120 50% subsample of ZEPHYRUSBENCHL
could be
The results of this ablation are largely intuitive: performance decreases when tools are removed, especially tools critical for a large number of tasks, like the forecaster or geolocator. We do observe very slight uptick in performance on Easy and Medium tasks when the climate simulator tool is removed. To understand this phenomenon, we examined few sample tasks where performance improved without the simulator; these shown in Table 5 and T These are forecasting tasks, which are exceedingly difficult (0% correctness) when the forecaster tool removed. When both available, however, ZEPHYRUS CEEEYRRUS-RELEECCCCE REFLECTIVE performs
the forecaster and the simulator than when only the forecaster is available. This result suggests that the model sometimes reaches for the wrong tool and tries use the climate simulator for short-term forecasting, even though tool prompt instructions specifically recommend the forecaster. This highlights the shortcoming of current LLM agents in instruction following and intelligent tool use ability, especially on long-contex and reasoning tasks.
25

---

Published as a conference paper at ICLR 2026
Tools
% Correct
Valid %
All Tools
33.3%
100.0% %
Exclude Climatology
33.3%
100.0% %
Exclude Forecaster
0.00%
83.3%
Exclude Geolocator
0.00%
100.0% %
Exclude Simulator
50.0% %
100.0% %
Exclude Dataset, Forecaster, Simulator
0.00%
100.0%
Table 6: Tool ablation for Template 37: Difference in Forecast Area- Weighted Mean.
Forecaster
& Correct
Easy
Mediom
Hard
SAE (Median)
Location Acc
EMD
Valid %
Extreme F1
Boolean FI
Ground- Truth
63.3%
86.3%
83.2%
34.7%
0.00
77.8%
1,841
99.1%
0.04
0.58
Stormer
60.5%
37.8%
87.9%
1,486
Table 7: Ablation: Ground-Trie vs. Stormer forecaster on ZephyrusBench.
A.4.4 ABLATION WITH GROUND-TRUTH FORECASTER
Table 7 contains a comparison of running ZEPHYRUS-REFLECTTE with the default Stormer (Nguyen et al. 2024) forecaster tool against "Ground-Truie forecaster setting, which returns the exact ground- truth WeatherBench2 data through the tool API. The test was conducted using L.M gpt-oss- 12bb on 50% subsample of ZEPHYRUSBENCIL.
Using the Ground-Truth forecaster, model performance increases significantly (60.5% to 83.2% correctness) on Medium tasks, many of which straightforward forecast-and-return questions. On Easy tasks, which do not require forecasting to solve correctly, performance does not improve. Most interestingly. model performance also does not improve on Hard tasks. Hard tasks. like report generation, require additional capability in order to convert accurate forecast data into a correct task answer. This result highlights the improvements in weather science reasoning and domain knowledge still necessary for LLMs to succeed thoroughly ZEPHYRUSBENCIL.
A.4.5 VARIANCE ACROSS MULTIPLE RUNS
In order to ensure that the stochasticity in sampling from LLMs does not have a major impact on the scores of ZEPHYRUSBENCH, we ran ZEPHYRUS- REFFECTIVE three times with identical settings, using LLM gpt-oss-12 of this variance test are presented Table 8. From this test we establish that the variance overall benchmark performance induced by L.MM sampling stochasticity is low.
on a 50% random subsample of ZEPHYRUSBENCH, Results
A.4.6 PERFORMANCE BY DIFFICULTY LEVEL
We report the difficulty wise breakdown of model performance on ZEPHYRUSBENCH in Figure I0 The difficulty levels for the tasks defined Table
Mon
Correct
Easy
Medium
Hand
SAR (Median)
Lecatioa Ace
FMB
Vali 9
Extreme P1
Bootean P1
56.79
99.2%
0.00
0.51
Ruy
$5.6%
62.55
26.25
0m
90.2%
1321
90.5%
0.57
Rus
$6.25
$1.0%
$9.7%
24.95
004
73.7%
1330
05.65
0.57
Table 8: Per-run metrics across 3 independent runs, with cross-rum variance summary (mean + std) for ZEPHYRUS- ZAPHYRUS-REFLECTIVE on ZEPHYRUSBENCH.
26

---

Published as a conference paper at ICLR 2026
Easy Difficulty Tasks
% Questions Correct
88.7% 90.9%%
97..5%
84.2%
80
76.27%
70
60
50
30
20
10
10.0%
10.9%
8.0%
8.3%
gpl-5.2
gpl-5-mini
gpt-oss-12
qwen3-30b
gemini- 2.5-fassh
LLM Zephyrus-Direct
Repleres-Reflective
Text-Only-LLM
Medium Difficulty Tasks
% Questions Correct
60
60.2%
63.5%
37.0%
40
10.2%
30
20
10
gpt-5.2
gpt-G-mini
gpt-oss-120b LLM
qwen3-30b
gemini-2.55 2.5-flash
Zephyrus-Relecttie
Zephyrus-Dies Direct
Text-Only-LLM
Hard Difficulty Tasks
40
% Questions Correct
34.55%
30.7%
31.5%
26.7%
26.3%
22.9%
20
11.3%
14.4%
14.2%
14.6%
gpt-5.2
gpt-G-mini
gpt-oss- 12ss--10bb
qwen3-30b
gemini- 2.5-flash
LLM
Rephyrus-Reflective
Zephyrus-Direct
Text-Only-LLMM
Figure 10: Performance ZEPHYRUSBENCCH segmented by difficulty level.
27

---

Published as a conference paper at ICLR 2026
A.4.7 DETAILED PERFORMANCE METRICS
We include detailed performance metrics from running several LLMs across all three modes on the
entire ZEPIYRUSBENCI dataset in Tables TO and
Model
L.MM
SAE (Q25) (L)
SAF (Q50) (1)
SAE: (Q75) (U)
SAF (Q99) (J)
ZEPHYRUS REFLECTIVE
9pt-3.2
0.00
0.02
0.09
0.95
ZEPHYRUS DIRECT
9pt-5.2
0.00
0,01
0.08
472.8
Text Only LLM
9pL-5.2
0.24
0.33
0.67
17.5
ZEPHYRUS-REPLECTIV
gpt-b-mini
0.00
0.00
0.06
0.96
ZEPHYRUS-DIIRS DISUSE
gpt-5-mini
0.00
0.07
0.94
Texx Only LLM
gpt-5-mini
0.23
0.32
0.59
4.87
ZEPHYRUS- CEEUUS--REEETIVVI
gemini-2.5-E1ie
0.00
0.02
0.14
5.38
SEPLYRUS-DIREG
gemini-2.5-fles
0.00
0.02
0.09
1.00
Text Only LLM
gemini-2.5-
0.24
0.34
0.93
77.4
gpt-oss-1200
0.00
0.03
0.12
1.56
ZEPHYRUS-DIRG
-pt-oss-1200
0.00
0,03
0.11
1.27
Text Only LLM
3pt-oss-1203
0.24
0.30
0.37
9.03
REFEEEEVEE
qwen3-30b
0.00
0.02
0.13
9.70
ZEPHYRUS DIRRECT Text Only LLM
gwen3-30b
0.00
0.03
0.18
18.1
qwen3-30b
0.23
0.49
7.25
165,992
Table Output validity and error metric quantiles for numerical tasks. SAE stands for standardized absolute error, the absolute error divided by the standard deviation of the relevant variable in the data. Q25, Q50, Q75, and Q99 respectively represent the 25th, 50th, 75th, and 99th percentile of error across all numerical dataset examples.
28

---

Published as a conference paper at ICLR 2026
Model
LLM
AE (Q25) (0)
AE (Q50) (1)
AE (Q75) (L)
AE (Q99) (U)
ZEPHYRUS-REFLECTVV
gpt-5.2
0.00
0,00
12.0
99.7
18PHYRUS-DIREC
9pt-5.2
0.00
0.00
12.0
99.7
Text Only LLM
gpt-5.2
6,00
18.0
36.0
135.7
ZEPHYRUS- SEHHYRUS-REFLCTTIN REFLECTIVE
gpt-S-mini
0.00
0.00
3.00
73.4
ZEPHYRUS-DIRES
gpt-5-ming
0.00
0.00
6.00
66.0
Text Only LLM
3pt-5-mini
12.0
36.0
103.4
ZEPHYRUS REFLECTIVE
semini-2.5-flan
0.00
0.00
0.00
105.8
JEPHYRUS-DIREO
S-mini-2.5-flant
0.00
160.1
Text Only LLM
gemini-2.5-
6.00
18.0
42.0
129.1
ZEPHYRUS- REFLECTIVE
apt-oss-120b
0.00
45.7
ZEPHYRUS- DIIEED
spt-oss-120b
0.00
0.00
0.00
39.8
Text Only LLM
3pt-oss-120b
6.00
18.0
36.0
114.0
ZEPHYRUS- REFLECTIVE
gw0n3-30b
0.00
0.00
6.00
87.6
ZEPHYRUS- D-REECT
qwen3-302
0.00
0.00
0.00
60.0
Text Only LLM
G-en3-30b
6.00
24.0
54.0
151.4
Table 10:
Absolute error quantiles for time tasks, in units of hours.
Model
L.M
Location Accuracy (%
EMD (km)
Extreme Weather (%)
ZEPHYRUS REFI ECTIVE
gpt-5.2
90.4
1,452
0,00
ZEMIYRUS-DIRECT Text Only L.M
qpt-9.2
1,277
9pt-5.2
18.2
5,445
0,00
ZEPHYRUS REFL ECTIVE ZEPHYRUS- DIS-II
opt-5-mini
87.9
1,486
0.00
9pt-5-mini
88.4
1,353
1.83
Text Only LLM
9pt-5-mini
18.2
5,772
gemini-2.55-11am
ZEPHYRUS RUS-DIRECT
gemini-2.5 2.5-flash
76.3
1,890
1.79
Only LLM
gemini-2.5-flash
3,750
ZEPHYRUS-REIE SEARRRUSSRREEEETEE ECTIVE
cpt-oss-120b
78.3
1,472
0.00
ZAMIYRUS-DIRGCT
gpt oss 120b
1,870
Text Only LLM
opt-o5s-120b
9.09
2,823
0.00
7EPHYRUS-RR REFI ECTIVE ZEMIYRUS-DIREC" Text Text Only LLM
3-303
67.2
1,597
0.00
quen3 30b
60.6
1,370
qwen3-30b
13.1
3,899
0.00
Table 11: Location metrics for location answer-based questions. EMD stands for Earth Mover's Distance.
Model
LLM
% Correct (1)
Valid Outputs (1)
Discussion Score (1)
Boolean F1 (1)
ZEPHYRUS REFLECTIVE
9pt-5.2
58.9
98.9
0.27
58.0
ZEPHYRUS-DIREER Text Only LLM
9p2-5.2
57.7
99.5
0.16
58.5
9pt-5.2
17.6
97.5
0.10
42.5
ZEPHYRUS REFLECTIVE
spa-S-mini
0.25
ZEPHYRUS- DIREECT Text Only LLM
sp2-5-mini
58.5
99.2
0.09
61.5
ops-S-mini
17.0
98.6
0.07
53.0
ZEPHYRUS REFLECTIVE
gemini-2.S-Elass
52.5
97.2
0.10
61.8
ZEPHYRUS DIRECT
gemini-2.5-Eles
56.0
99.1
0.07
61.6
Only
gemini-2.5-Elam
ZEPHYRUS REFLECTIVE
opt-oss-12Cb epc-o9s-12Cb
56.2
99.1
0.11
59.3
ZEPHYRUS DIRECT Text Only LLM
98.3
0.08
57.1
cpc-oss-12Cb
13.3
68.8
0.02
25.0
ZEPHYRUS. REFLECTIVE
gxen3-30b
44.2
83.6
0.06
48.6
FAPHYRUS-DRRCTT DIRECT Text Only LLM
gxen3-30b
96.3
0.06
3x0n3-30b
16.4
93.0
0.03
8.97
Table 12: Overall percentage of questions correct, percentage of valid outputs, numerical score (0-1) for discussion questions, and F] score for boolean questions.
29

---

Published as a conference paper at ICLR 2026
A.4.8 PERFORMANCE METRICS, HUMAN-GENERATED TASKS
We also include top-level performance metries broken specifically for the 19 human-generated
tasks in ZEPIYYUSBENCI dataset. The results are reported in Tables 13
14
15
and
T6
Model
L.M
SAE (Q25) (U)
SAE (Q50) (J)
SAE (Q75) (U)
SAF (Q99) (J)
ZEPHYRUS- REFLECTIVE
9pt-5.2
0.00
0.01
0.07
0.60
SEPHYRUS-DIREC
gpt-3.2
0.00
0.00
0.07
2,164
Text Only LLM
9yL-5.2
0.24
0.32
0.58
4.60
ZEPHYRUS- REFLECTIVE
gpt-5-mini
0.00
0.00
0.05
0.44
ZEPHYRUS- -EPRECTS-DRREC
qpt-5-mini
0.00
0.00
0.06
0.42
Only
gpt-5-mini
0.23
0.31
0.47
2.84
ZEPHYRUS- REFLECTIVE
gemini-2.5-Clash
0.00
0.02
0.12
3.82
ZEPHYRUS- DISEEI
gemin1-2.5-El.
0.00
0.01
0.08
0.56
Text Only LLM
gemin1-2.5-rlas
0.25
0.33
0.70
476.1
gpt gpt-oss- 120b
0.00
0.02
0.10
1.34
ZEPHYRUS- DIRECT
apt-oss-1200
0.00
0.02
0.09
Text Only LLM
3pt-oss-120b
0.24
0.30
0.36
7.38
ZEPHYRUS- REFLECTIVE
qwen3-30b
0.00
0.00
0.10
14.3
ZEPHYRUS DIRECT
qwen3-30b
0.00
0.03
0.16
19.7
Text Only LLM
gwen3-30b
0.23
0.40
5.43
108,610
Table 13: Output validity and error metric quantiles for numerical tasks. SAE stands for standardized absolute error, the absolute error divided by the standard deviation of the relevant variable in the data. Q25, Q50, Q75, and Q99 respectively represent the 25th, 50th, 75th, and 99th percentile of error across all numerical dataset examples in auman-generated tasks.
30

---

Published as a conference paper at ICLR 2026
Model
LLM
AE (Q25) (0)
AE (Q50) (1)
AE (Q75) (L)
AE (Q99) (U)
ZEPHYRUS-REFLECTVV
gpt-5.2
0.00
0,00
12.0
99.7
18PHYRUS-DIREC
9pt-5.2
0.00
0.00
12.0
99.7
Text Only LLM
gpt-5.2
6,00
18.0
36.0
135.7
ZEPHYRUS- SEHHYRUS-REFECTTIN REFLECTIVE
gpt-S-mini
0.00
0.00
3.00
73.4
ZEPHYRUS-DIRES
gpt-5-ming
0.00
0.00
6.00
66.0
Text Only LLM
3pt-5-mini
12.0
36.0
103.4
ZEPHYRUS REFLECTIVE
semini-2.5-flam
0.00
0.00
0.00
105.8
ZEPHYRUS-DIRE DIIRECT
S-mini-2.5-flant
0.00
160.1
Text Only LLM
gemin1-2.5-flas
6.00
18.0
42.0
129.1
ZEPHYRUS- REFLECTIVE
apt-oss-120b
0.00
45.7
ZEPHYRUS- DIIRE-
spt-oss-120b
0.00
0.00
0.00
39.8
Text Only LLM
3pi-oss-120b
6.00
18.0
36.0
114.0
ZEPHYRUS- REFLECTIVE
gw0n3-30b
0.00
0.00
6.00
87.6
ZEPHYRUS- DDDRECT Text Only LLM
gwen3-30p
0.00
0.00
0.00
60.0
3-en3-30b
6.00
24.0
54.0
151.4
Table 14: Absolute error quantiles for time tasks, units of hours, on human-generatee tasks in human- human-generated tasks.
Model
L.M
Location Accuracy (%)(T)
(km) (L)
ZEPHYRUS R REFLECTIVE
gpt-5.2
90.4
1,452
0.00
ZEPHYRUS-DIREE Text Only LLM
gpt-5,
89.9
1,277
gpt-5.2
18.2
5.445
0.00
ZEPHYRUS-RIIE CEEISSRRFFEEEE REFLECTIVE
gpt-b-mini
87.9
1,486
0.00
ZEPUYRUS-DDE Text Ouly LLM
9pt-5-22
88.4
1,353
spt-S-mini
18.2
5.772
0.00
ZEPHYRUS-
remini-2.b--ias
1,651
ZEMIYRUS- RUS-DRUS
gemini 2.5 flash
76.3
1,890
1.79
Text Only LLM
remini-2.5-fles.
16.2
3.750
ZAPHYRUS RERI ECTIVE
gpt cpt-088-120s
78.3
1,472
ZEMIYRUS- -DRRECT Text Only LLM
gpt ppt--os--10 os8-120b
67.2
1,870
0.00
gpt 3pt-oss-122
ZEPHYRUS REFL ECTIVE ZEMIYRUS-DIRR Text Only LLM
3-303
67.2
1,597
0.00
qwen3-30b
1,370
g-en3-30b
13.1
3,899
0.00
Table 15: Location metrics for location waweerbbased questions in human generated tasks. EMD stands for Earth mover's Distance.
Model
LLM
Correct % Valid Outputs
Discussion Score
ZEPHYRUS REFLECTIVE
gpt-5.2
57.5
98.7
0.27
ZEPHYRUS- DIREECT
gpt-5.2
56.0
99.3
0.16
Text Only LLM
gpt-5.2
8.91
97.3
0.10
ZEPHYRUS- REFLECTIVE
opt-S-mini
60.4
99.0
0.25
REPHYRUS-DIRECT
opt-5-mini
57.1
99,1
0.09
Text Only L.M
gpt-5-mini
8.56
98.5
0.07
ZEPHYRUS-REFLECTIO REFLECTIVE
gemini-2.s-flan
50.9
97.8
0.10
SCHHYRUS-DIRECT
Semini-2.5-flea
54.1
98.9
0.07
Text Only LLM
gemini-2.5-f1em
8.28
89.1
0.06
ZEPHYRUS-
SSPHYRUS-REFLECIVV REFLECTIVE
3pt-oss-120b
54.5
98.9
0.11
ZEPHYRUS-DIEG DIREEC Text Only LLM
apt-oss-1200
53.1
97.8
0.08
apt-oss-120b
67.7
0.02
ZEPHYRUS- REFLECTIVE
qven3-30b
42.1
82.0
ZEPHYRUS- DDIRECT Text Only LLM
gwen3-30b
48.4
95.9
0.06
gwen3-30b
11.4
94.9
0.03
Table 16: Overall percentage of questions correct, percentage of valid outputs, and numerical score (0-1 for discussion questions in human human-generated tasks.
31

---

Published as a conference paper at ICLR 2026
A.4.9 PERFORMANCE METRICS, SEMI-SYNTHETIC TASKS
We also include top-level performance metries broken out specifically for the 30 semi-synthetic (LLM-generated, human-validated) tasks in ZEPIYYUSBENCI dataset. The results are reported in
Tables 17 and
Model
LLM
SAE (Q25) (J)
SAE (Q50) ())
SAE (Q75) (L)
SAE (Q99) (4)
ZEPHYRUS-REELEE REFLECTIVE
g9t-5.2
0.07
0.30
3.24
JEPHYRUS-DIREG Text Only LLM
gat-5.2
0.02
0.07
0.30
2.68
9pt-5.2
0.25
0.65
1.62
333.6
ZEPHYKUS-RERLE REFLETIVVE
9pt-5-mini
0.02
0.07
0.26
1.32
ZEPHYRUS- DIRECT
DIRRECC Text Only LLM
gpt-5-mini
0.02
0.07
0.24
1.41
9pt-5-mini
0.27
0.78
1.68
24.3
ZEPHYRUS- REFLECIVEE
gemini-2.5
0.03
0.10
0.34
9.47
ZEPHYRUS DIRECT
0.02
0.08
0.29
1.31
Text Only LLM
gemini-2.5-Elam
0.23
0.85
2.09
31.0
ZUPHYRUS-REFLECTIV
gpt-oss-12
0.02
0.07
0.29
1.89
ZEPHYRUS- DIRSEC
3pL-oss-120b
0.02
0.07
0.29
2.61
Text Only LLM
gpt gpt-oss-1200
0.28
1.02
1.92
16.2
ZEPHYRUS- REPHYRUS-REPLECIIO REFLECIVVE
mwen--30b
0.02
0.07
0.29
2.34
ZEPHYRUS DIRECT Only
gwen3-30b
0.02
0.11
0.36
4.36
mwen3-30b
0.40
3.39e+06
Table 17:
Output validity and error metric quantiles for semi-synthetically generated numerical tasks. SAE stands for standardized absolute error, the absolute error divided by the standard deviation of the relevant variable in the data. Q25, Q50, Q75, and Q99 respectively represent the 25th, 50th, 75th, and 99th percentile of error across all numerical dataset examples.
Model
LLM
% Correct (1)
% Valid Outputs (f)
Boolean Fl (%
ZEPHYRUS- ZEPHYRUS-RERLECIIVE REFLECTIVE
gpt-5.2
63.7
$8.0
ZEPHYRUS-DIRECT -DRRECT Text Only LJM
gpt-5.2
63.9
100.0
58.5
gpt-5.2
48.4
98.2
42.5
ZEPHYRUS-REFLECTII
gpt- gpt-5-mini
64.1
99.8
60.6
SEPHYRUS-DIREDD Text Only LLM
got-5-mini
63.7
99.4
61.5
gpt-5-mini
46.9
99.0
53.0
ZEPHYRUS-REN RERUUSS-REEEEIVEE
gemini-2. 5-ffaah
58.2
94.9
61.8
HEPHYRUS-DIRECH
gemini-2. 5-flash
62.9
99.8
61.6
Text Only LLM
2omini-2.5-flas
42.2
96.7
24.3
ZEPHYRUS-RERI CEUUS-REFLECTIVE ECTIVE
gpt-o8s-120b
62.2
100.0
59.3
ZEPHYRUS-DIIEE Text Only LLM
ept-oss-170b
63.5
100.0
57.1
opt-oss-1206
37.1
72.7
25.0
ZEPHYRUS-REE REFRRERT-EEEEEI
qwe:3-30b
51.4
89.4
48.6
SEHHYRUS-DIREGE Text Only LLM
quer3330bb 30b
61.2
98.0
59.8
3-0b:
34.1
86.5
8.97
Table 18: Overall percentage of questions correct, percentage of valid outputs, numerical score (0-1) for discussion questions, and Fl score for boolean questions on the semi-synthetically generated subset of tasks.
32

---

Published as a conference paper at ICLR 2026
A.4.10 PERFORMANCE METRICS BY TASK
At the most granular level of detail, provide performance metries of all models' performance on each unique template ID in ZEPIYYUSBENCI. The reported performance metrics are tailored individually to each question type. These results are reported in Tables 19 through 67
Model
LLM
Location Accuracy (%)(1)
EMD (km)(U)
gpt 5-mnn gpt-5-mini
96.8
179.8
ZEPHYRUS- TIRERR
100.0
0.00
Text Only LLM
gpt-5-mini
17.0
8,551
ZEPHYRUS
ZEPHYRUS-REFLECTHE REFLECTIVE
Semini-2.5-f1an.
83.0
1,132
ZEPHYRUS DIRECT
gemin1-2. 5-flashh
84.0
1,126
Text Only LLM
2amini-2.5-flame
18.1
6,372
ZEPHYRUS-REFI REFLECTIVE
gpt-oss-1205
83.0
911.2
HEPHYRUS-DIRECT
gpt-oss- 1atass-10bb 120b
88.3
889.3
Text Only LLM
1ot-oss-120b
13.8
6,954
DEPHYYRUS-EELLETII
gpt-5.2
100.0
0.00
ZEPHYRUS-DIEE DIRECT Text Only LLM
gpt-5.2 5.2
100.0
0.00
gpt-5.2
7,968
ZEPHYRUS-REFF ECTIVE
qwen3-300 306
76.6
1,502
ZEPHYRUS-DIREN
qwen3-30D
89.4
855.3
Text Only LLM
qwen3-30b
16.0
7,183
Table 19: Location prediction metrics for Template ID I: Which geographic feature experienced the highest/lowest average value of a weather variable
Model
LLM
SAE (Q25)(J)
SAE (Q50IC))
SAE (Q75)(1)
SAE (Q99)())
Err Std(J)
ZEPHYAUS-KEPLECTIVE
gpt-5-mini
0.00
0.00
0.00
0.13
20.7
ZEPHYRUS-DIRGE -DRRECT Text Only LLM
apt-b-mini
0.00)
0.00
0.00
0.06
5,34
gpt-S-mini
0.16
0.40
0.77
3.24
1,390
ZEPHYRUS-REIS REFLECEEE ECTIVE
-2mini22-5-flash
0.00
0.00
0.01
0.41
413.0
ZEPHYRUS-DIREE
remini-2.5-f1.en
0.00
0.00
0.00
0.39
422.0
Text Only LLM
gemini 22.-22..110. -laah
0.26
0.72
2.87
75.8
30,300
REFELUUVVEEEEEIIVEE
opt-oss-1206 12:088-120bb
0.00
0.00
0.01
0.47
413.0
JEPAYRUS-DIRERR
ept-oss-1200
0.00
0.00
0.00)
0.47
460.0
Text Only LLM
gpt-oss-120b
0.12
0.46
1.15
116.0
2,330
ZEPHYRUS REFLECTIVE
gpt-5.2
0.00
0.00
0.00
0.13
376.0
ZEPHYRUS-DIREE Text Only LLM
gpt-5.2
0.00
0.00
0.00
0.39
562.0
qpt-5.2
0.20
0.56
1.34
13.5
21,000
qwen3
0.00
0.00
0.03
391.0
ZEPHYRUS-DIIG DIRFCCT
qwen3-30b
0.00
0.00
0.02
0.48
271.0
Text Only LLM
qwen3-30b
0.26
0.92
6.86
76.2
39,000
Table 20: Standardized Absolute Error (SAE) quantiles for Template ID What is the min/ /tt///x//aee Everage/median. value of a weather variable at specific locatio
33

---

Published as a conference paper at ICLR 2026
Model
LLM
Location Accuracy (%)())
EMD (km)
SEPHYRUS-REFLECTIIE
gpt-5-mini
79.8
472.1
ZEPHYRUS DIRECT
gpt-5-mini
77.9
784.8
Text Only LLM
gpt-5-mini
19.2
3,611
SEHHYRUS-REFEECTIN
gemini-2.1. 5-flash
71.2
1,224
ZEPHYRUS DIRECT
gemini-2. 5-flash
69.2
1,442
Text Only LLM
(cmini-2.5-flee
14.4
2,962
EEEEEEVVVE ZEPHYRUS-REFT ECTEVE
gpt-oss-120b 120b
74.0
842.2
ZEPHYRUS-DIRECT Text Only LLM
ZEPHYRUS- DIRECT
gpt-oss-120b
48.1
2,290
gptcos--20b
4.80
3,290
JEPHYRUS-REFLER ECTIVE
gpt-5.2
81.7
403.6
ZEPHYRUS- -IRRETTTTT
gpt-5.2
80.8
211.2
Text Only LLM
gpt-5.2
19.2
3,196
ZEPHYRUS-REEEE REFLECTIVE
qwen3-30b
58.7
2,307
ZEPHYRUS-DIRES DIREECT Text Only LLM
qwen3-30b
34.6
2,656
Twen3-30b
10.6
3,312
Table 21: Location prediction metrics for Template II) 3: Which ch cublocatioon ion has the
hi highest/lowest. recorded variable va value
Model
LLM
AE (Q25KL)
AE (Q50XL)
AE (Q75)(L)
AE (Q99XL)
ZEPHYRUS-REFLECTIVE REFLECTIVE
pet-5-mini gpt-8-mini gpt-5-mini
0.00
0.00
0.00
82.9
ZEPHYRUS- -DRRCC Text Only LLM
0.00
0.00
0,00
73.0
6.00
18.0
42.0
109.0
ZEPHYRUS-RER REFLECTIVE
gemini-2. flash
125.0
ZEPHYRUS-DIRG
gemini-2. 5- aash
0.00
0.00
0.00
200.0
Text Only LLM
gemini-2.55 aash
24.0
48.0
138.0
ZEPHYRUS-REFANE REFLECTIVE
gpt-os8 120b
0.00
0.00
0.00
48.0
SEPHYRUS-DIRED
gpt-oss- 120b
0.00
0.00
0.00
37.2
Text Only LLM
gpt-o5s-120b
6.00
18.0
48.0
119.0
REFLECTVEE
g.2
0.00
0.00
12.0
103,0
ZEPHYRUS-DIIE -DRREC
gpt-5.2
12.0
105.0
Text Only LLM
gpt-5.2
6.00
24.0
48.0
138.0
SENYYRUS-REFECTIIR ZEPHYRUS- DIRECT
gw0b
0.00
0.00
0.00
90.0
qwen3-30b
0.00
0,00
0.00
60.7
Text Only LLM
qwen3-30b
12.0
30.0
72.0
156.0
Table 22: Absolute Eror (AE) quantiles for Template IDD 4: How many hours from start
did a location experience remum
Model
LLM
SAE (Q25)L)
SAE (Q50)()
SAE (Q75)()
SAE (Q99)())
Err Std(J)
ZEPHYRUS REFLECTIVE
gpt-5-mini
0.00
0.00
0.00
1.34
553.0
ZEPHYRUS-DIRECT
gpt-S-mini
0.00
0.00
0.00
0.36
551.0
Text Ouly LLM
gpt-5-mini
0.19
0.45
0.90
4.09
2,030
gemini 2.5 F1ash
0.00
0.00
0.00
2.41
634.0
ZEPHYRUS DIRECT
gemi gemini-2.5-2 -1--2-5-11aa
0.00
0.00
0.00)
0.42
2,890
Text Only LLM
gemini-2.b-2lasm
0.31
1.10
3.22
5,440
21,500
ZEPHYRUS REFLECTIVE
spt-oss-120b
0.00
0.00
0.01
1.46
6,970
ZEPHYRUS-DIREC DIRRECT
spt-oss-1205
0.00
0.00
0.01
0.37
343.0
Text Only LLM
spt-oss-120b
0.15
0.45
1.28
269.0
2,670
ZEPIVVRS-REELLCTIVEE
gpt 5.2
0.00
0.00
0.00
0.39
232.0
ZEPHYRUS DIRECTT
gpt-5.2
0.00
0.00
0.00
0.36
232.0
Text Only LLM
gpt-5.2
0.18
0.51
1.08
7.14
2,530
REFLECTIVE
qwen3-30b
2,060
ZEPHYRUS-DIRIC
qwen3 30b
0.00
2,900
Only gwen3-30b
173,000
38,700
Table 23: Standardized Absolute Error (SAE) quantiles for Template ID
what is the
weather variable value at location specific time
34

---

Published as a conference paper at ICLR 2026
Model
LLM
Correctness (()
EMD (km)())
IoU (% (()
gpt-5-min
100.0
0.00
100.0
ZEPHYRUS- DIRECT
gpt-5-mini
100.0
0.00
100.0
Text Only LLM
9pt-5-mini
20.0
7,580
0.00
ZEPHYRUS REFLECTIVE
gemini-2. gemin1-2.5-flast
1omini-2.5-flast
82.2
1,243
100.0
ZEPHYRUS- DIRRET
73.3
832.3
100.0
Text Only L.M
gemini-2.5-flast
37.8
10,864
0.00
ZEPHYRUS-REFLECTIV
gpt-oss-120b
100.0
0.00
100.0
ZEPHYRUS- DIRECT
gpt-oss-1206
100.0
0.00
100.0
Text Only L.M
apt-oss-120b
57.8
10,270
100.0
ZEPHYRUS- SEFRUUS-EFFEECIVEE SIPYYRSS-DIRECT ZEPHYRUS- DIRECT Text Only LLM
gpt-5.2
95.6
558.7
100.0
9pt-5.2
97.8
565.0
100.0
gpt-5.2
15.6
7,750
0.00
ZEPHYRUS- SETYRUS-REFLECIIE REFLECTIVE ZEPHYRUS- DIRECT
qwen3-30b
82.2
887.7
100.0
qwen3-30b
88.9
419.8
100.0
Text Only LLM
qwen3-303
46.7
10,569
0.00
Table 24: Extreme weather detection metrics for Template ID 6: Tdent geofeatures
where variable differs from mean by standard deviations
Model
LLM
Correctness (%)())
EMD (km)())
IoU (%)(t)
ZEPHYRUS-REFLECTEO ZEPHYRUS- DIRECT
gpt-5-mini
95.8
9.30
100.0
gpt-5-mini
100.0
0.00
100.0
Text Only LLM
gpt-5-mini
0.00
5,519
2.30
ZEPHYRUS- REFEETVVEE
gemini-2.55- f-ass
79.2
225.4
100.0
REPRERUS-DIRECT
1omini-2.5-flee
81.2
227.5
100.0
Text Only LLM
gemin1-2.5-flast
4.20
5,609
0.00
APPHYRUS-REFLECIVVE
gpt-oss-120b
100.0
0.70
100.0
ZEPHYRUS- DIRRECT
gpt-oss-120b
95.8
0,00
100.0
Text Only LLM
gpt-oss-120b
4.20
6,137
0.00
ZEPHYRUS-
ZEPHYRUS-REFLECHE REFLECTIVE REPHYRUS-DIRECT ZEPHYRUS- DIRECT
gpt-5.2
97.9
74.7
100.0
gpt-5.2
97.9
11.3
100.0
Text Only LLM
gpt-5.2
0.00
5,750
1.50
ZEPHYRUS-
SOHHYRUS-RELLETIVV REFLECTIVE ZEPHYRUS- DIRECT
qwen3-30b
83.3
292.6
100.0
qwen3-30b
87.5
100.0
Text Only LLM
qwen3-300
2.10
7,287
0.00
Table 25: Extreme weather detection metrics for Template ID 7: Identify geofeatures where variable lies outside cl climatologier
percentile envelope
Model
LLM
SAE (Q25K())
SAE (Q50IC))
SAE (Q75)()
SAE (Q99)L)
Err Std(J)
ZEHIIRUS-REELECTIVE ZUPHYRUS-DIRECT
gpt mini
0.01
0.06
0.12
0.67
173.0
gpt min1
0.04
0.07
0.17
1.36
180.0
Text Only L.M
gpt-5-mint
0.22
0.46
0.97
3.76
257.0
ZEMIYRUS-REFLECTVV
gemini-2. 2-5-11ash
0.01
0.05
0.10
2.12
31.1
ZEPHYRUS DIRECT
0.01
0.05
0.09
0.37
66.2
Text Ouly LLM
1omini-2.5--1..m
0.31
0.82
2.15
58.4
20,100
ZEPHYRUS- REFLECTIVE
gpt-oss-120b
0.01
0.06
0.10
0.67
182.0
ZOPIYRUS-DIRES
spt 03s 120b
0.01
0.06
0.11
0.67
1 186.0
Text Only L.M
ept-oss-(206
0.72
0.95
1.54
2.12
1,810
ZEHHYRUS-EELFCCOE
gpt-b.2
0.05
0.10
0.67
172.0
ZEPHYRUS DIRECT
gpt-5.2
0.01
0.06
0.11
0.67
173.0
Text Only LLM
5.2
0.28
0.66
1.67
3.22
432.0
ZAPHYRUS-KEPLEETIVE ZOPIYRUS-DIRECT Text Only L.M
gwen3-30b
0.54
205.0
qwen3
0.05
0.09
0.56
188.0
gwon3-30b
1.54
12.2
24.3
726,000
28,
Table 26: Standardized Absolute Error (SAE) quantiles for Template ID What will the variable be at a location after time interval (forecast)
35

---

Published as a conference paper at ICLR 2026
Model
LLM 9pt-5- 5-mi- gpt-5-mint spt-5-mini
AE (Q25KJ)
AE (Q50XL)
AE (Q75)(L)
AE (Q99XL)
ZEPHYRUS- SEHHYRUS-REFLECTVVE
0.00
6.00
39.2
7EPHYRUS-DIREE Text Only LLM
6.00
18.0
39.3
6.00
6.00
12.0
24.5
gemini 2.5 flash
0.00
0.00
6,00
39.2
ZEPHYRUS-DIRIE
gemini-2.3 22.5-flash
0.00
0.00
4.50
39.2
Text Only LLM
remini-2.5-flase
1.00
6.00
15.0
39.2
ZEPHYRUS-REFLECTIVV REFLECTIVE
gpt-oss- gpt-oss-120b
39.2
ZEPHYRUS- SIRRRUSS DIRECT Only
0.00
0.00
0.00
39.2
gpt-oss-120b
12.0
18.0
ZEPHYRUS- REFLEELLEEEE ECTIVE
gpt-5.2
0.00
0.00
12.0
39.2
SOPHYRUS-DIREG Text Only LLM
gpt-5.2
0.00
0.00
6.00
39.2
gpt-5.2
0.00
6.00
12.0
21.2
ZEPHYRUS- ZEPHYRUS-REPLECTVV REFLECIVEE
quen3-
0.00
6.00
21.0
39.7
ZEPHYRUS-DIREE DERRRUT Text Only LLM
qwen3-30b
0.00
0.00
12.0
42.0
qwen3-30b
6.00
12.0
18.0
45.2
Table 27: Absolute Error (AE) quantiles for Template ID 9: When will locat on
experience its extremum in future period (forecast) Model
LLM
Correctness (% (%)(t)
EMD (km)())
IoU (%)
REPHYRUS-RELLECCEE ZEPHYRUS- DIRECT Text Only LLM
gpt-5-mini
13.7
3,839
40.0
gpt-5-mini
13.7
3,210
45.5
gpt-5-mini
7.80
6,732
0.00
ZEPHYRUS- SETRRRSSEEFEECIIVE
gemini-2.5-fles
11.8
2,520
37.1
ZEPHYRUS- CIRERU
gemini-2.5-flie
15.7
2,693
37.1
Text Only LLM
gemini-2.5-flass
9.80
8,383
0.00
ZEPHYRUS- RETTRUSS-RFLEETIVEE
gpt-oss-120b
15.7
2,772
38.9
SHHYRRS-DDRRCT ZEPHYRUS- DIRECT
gpt-oss-120b
13.7
3,087
42.6
Text Only LLM
gpt-oss-120b
11.8
0.00
ZEPHYRUS- REFUUSSIFEEEE
gpt-5.2
13.7
4,644
34.4
ZEPHYRUS-DIRES
gpt-5.2
13.7
4,031
35.2
Text Only LLM
gpt-5.2
11.8
6,561
0.00
ZEPHYRUS-RELLECTE ZEPHYRUS DIRECT
qwen3-30b
11.8
3,582
$.00
qwen3-30b
13.7
3,359
17.1
Text Only LLM
qwen3-30b
9.80
8,215
0.00
Table 28: Extreme weather detection metries for Template ID 10: Which geofeatures
experiencing exceedance in variable va valuus Model
LLM
Correctness (%)
EMD (km)())
IoU (%)(1)
ZEPHYRUS-REFLNE REIIS-RRREEECEE ECTLECTIVE ZEPHYRUS- -IRRET
gpt 3pt-5-mini gpt-b-mini
17.1
9,422
0.00
20.0
8,410
0.00
Text Only LLM
spt-b-mini
28.6
0.00
REFELEUVVE TIREET
gemini-2.5-fLast
22.9
5,761
0.00
ZEPHYRUS- DIRECT
gemini-2.5-flase
17.1
8,696
0.00
Text Only LLM
gemini-2.5-f1es
28.6
0.00
ZEPHYRUS- REFLERRUVE
gpt-oss-120b
14.3
7,586
0.00
ZEPHYRUS- DIRECT
gpt-oss-120b
25.7
8,409
0.00
Text Only LLM
gpt-oss-120b
28.6
0.00
ZEPHYRUS- REEERRUURREEEEIIVEEE
gp
17.1
10,402
0.00
ZEPHYRUS- MIRERUTS-DIRET
gpt-5.2
28.6
11,612
0.00
Text Only LLM
gpt-5.2
28.6
0.00
ZEPHYRUS- REFLEETIVE
qwen3-30D 300
28.6
10,114
0.00
ZEPHYRUS- DIRECT
SESYRRRS-DIRETT Text Only LLM
qwen3-30a
25.7
11,073
0.00
qwen3-30b
28.6
0.00
Table 29: Extreme weather detection metrics for Template ID 11: weather events that will occur in the next hours (forecast)
Identify extreme
36

---

Published as a conference paper at ICLR 2026
Model
LLM
Correctness (%)(1)
EMD (km)()
IoU (% (%)(t)
gpt-5-min
8.30
6,912
0.00
ZEPHYRUS- DIRECT Text Only LLM
gpt-5-mini
8.30
7,871
0.00
9pt-5-mini
16.7
0.00
ZEPHYRUS REFLECTIVE
gemini-2.5-f25
14.6
7,462
0.00
ZEPHYRUS- DIRRET
gemin1-2.5-flast
4.20
6,824
0.00
Text Only L.M
gemini-2.5-flast
16.7
0.00
ZEPHYRUS-REFLECTVVV
gpt-oss-120b
8.30
9,589
0.00
ZEPHYRUS- DIRECT
gpt-oss-1206
6,20
8,357
0.00
Text Only L.M
opt-oss-120b
16.7
0.00
REFRUUSSEFFEECIVEE SIPYRRSS-DIRCCT ZEPHYRUS- DIRECT
gpt-5.2
12.5
7,556
0.00
9pt-5.2
10.4
9,214
0.00
Text Only LLM
gpt-5.2
16.7
0.00
ZEPHYRUS- REFLECTIVE CEEYRRSSRRELECTIIE ZEPHYRUS- DIRECT
qwen3-30b
16.7
8,183
0.00
qwen3-30b
16.7
7,785
0.00
Text Only LLM
qwen3-33
16.7
0.00
Table 30: Extreme weather detection metrics for Template ID 12: Check if extreme
weather events are currently happening
Model
LLM
Correctness (%)(1)
EMD (km)(J)
IoU (%)(1)
ZEPHYRRS-REFLLECTVV
9pt-5-mini
100.0
0.00
100.0
ZEPHYRUS-DII DIRREC Text Only LLM
gpt-5-mini
100.0
0.00
100.0
gpt-5-mini
4.80
6,248
2.00
ZEPHYRUS-REIT REFLECTIVE
gemini-2 1m.
71.4
1,136
100.0
ZEPHYRUS- DIRECT
DEPRERUS-DIRECT
gemini-2.5-flas3
85.7
559.8
100.0
Text Only LLM
semini-2.5-flase
4.80
7,554
0.00
ZEPHYRUS- REEEEUUUEEEEEEIIVEE
gpt-oss-120b
97.6
0.10
100.0
ZEPHYRUS- DIRECT
gpt-oss-120b
100.0
0.00
100.0
Text Only LLM
gpt-oss-120b
9.50
5,879
0.00
gpt-5.2
100.0
0.00
100.0
ZEPHYRUS-DIES
gpt-5.2
100.0
0.00
100.0
Text Only LLM
gpt-5.2
2.40
6,656
0.00
MINYRRUSRREFECCEOR ZEPHYRUS- DIRECT
qwen3-303 30b
76.2
598.4
100.0
-IRRET Text Only LLM
qwen3-30a
97.6
0.00
100.0
qwen3-30b
4.80
6,763
0.00
Table 31: Extreme weather detection metrics for Template ID 13:
Which geographic
features experienced unusual weather anomalies compared baseline ine
Model
LLM
Fl Score(t)
Precision (%)(t)
Recall (%)tt)
HEPHYRUS-REFLECTEE
9pt-5-mini
93.3
100.0
87.5
ZEPHYRUS-DIREE Text Only LLM
gpt-5-mini
93.3
100.0
87.5
gpt-5-mini
61.5
80.0
50.0
gemini-2.5 51222.51111. 5-flash
93.3
100.0
87.5
REPHYRUS-DIRER
gemini-2. 10-ii2.5-1100 5-555-f1hhh--
76.9
100.0
62.5
Text Only LLM
gemini-2.5 5-112255ffla. flash
50.0
75.0
37.5
ZEPHYRUS- SEWWYRUS-REFLECTVVEE
gpt-oss- 120b
93.3
100.0
87.5
DEPHYRUS-DIRECT
gpt-oss
76.9
100.0
62.5
Text Only LLM
gpt-oss-12Us
57.1
100.0
40.0
ZEPHYRUS- REFTIVVVEEE
gpt-5.2
76.9
100.0
62.5
ZEPHYRUS- DIREEC
gpt-5,2
100.0
87.5
Text Only LLM
g.t
76.9
100.0
62.5
ZEPHYRUS- SENRRRSSRRFLECTIVVE
qwen3- 30b
93.3
100.0
87.5
COHYRUS--IREER Text Only LLM
qwen3-308
94.1
88.9
100.0
mwen3-30b
75.0
100.0
60.0
Table 32: Boolean classification metrics for Template ID 14:
Does maximum weather variable occur at same or adjacent grid point as another variable (forecast)
37

---

Published as a conference paper at ICLR 2026
Model
LLM
F1 Score(t)
Precision (% (1)
Recall (1)
gpt-5-ming
85.7
75.0
100.0
ZEPHYRUS- DIRECT Text Only LLM
gpt-5-mini
85.7
75.0
100.0
3pt-5-mins
0.00
0.00
0.00
SEHHYRUSSRRFLECTIVN ZEPHYRUS- DIRRECT Text Only LLM
gemini-2 101i-2255.1a5.
85.7
75.0
100.0
85.7
75.0
100.0
Somin--2.5-llas:
0.00
0.00
0.00
ZEPHYRUS-REDH REFLECTIVE
gpt gpt-oss-11 1200
85.7
75.0
100.0
ZEPHYRUS-DIRGS Text Only LLM
gpt-oss-120a
85.7
75.0
100.0
opt-oss-120s
50.0
50.0
50.0
SEHYYRUS-REFFCCEO
gpt-5.2
85.7
75.0
100.0
SOPHYUS--DREEG
gpt-5.2
85.7
75.0
100.0
Text Only LLM
gpt-5.2
50.0
100.0
ZEPHYRUS- SEHYYRUS-RELECTIVE REFLECTIVE
qxen3-30b
85.7
75.0
100.0
ZEPHYRUS-DIREG Text Only LLM
qwen3-30b
100.0
100.0
100.0
qwen3-30b
0.00
0.00
0.00
Table 33: Boolean classification metrics for Template II) 15:
Does max fexiimon weather
variable region remain in lower than future maximum (forecast)
Model
LLM
Fl Score(t) Precision (%)())
Recall (%)(t)
ZEPHYRUS-REFLECTIVV
gpt-5-mint
100.0
100.0
100.0
ZEPHYRUS-DIES Text Only LLM
gpt-5-mini
100.0
100.0
100.0
9pt-5-mini
75.0
100.0
60.0
ZEPHYRUS- ZEPHYRUS-REFLECTIEE REFLECTIVE
gemini-2.5-2 5-flash
88.9
80.0
100.0
CIRRRT ZEPHYRUS- DIRECT Text Only L.M
100.0
100.0
100.0
gemini-2.5-f1es
33.3
100.0
20.0
ZEPHYRUS REFLECTIVE
gpt-oss-120s
100.0
100.0
100.0
ZEPHYRUS-DIRECT Text Only L.LM
gpt-oss- 1206
100.0
100.0
100.0
gpt-oss 1205
50.0
100.0
33.3
ZEPHYRUS- REFERUUVVEEEEE
gpt-5.2
100.0
100.0
100.0
ZEPHYRUS- SIPHERTS-DIRECT Text Only LLM
gpt-5.2
100.0
100.0
100.0
gpt-5.2
83.3
71.4
100.0
ZEPHYRUS- BEPHYRUS-RELECCIVV
qwen3-30b
50.0
100.0
33.3
qwen3-30b
75.0
100.0
60.0
Text Only LLM
@wcn3-30b
40.0
50.0
33.3
Table 34: Boolean classification metrics for Template ID 16:
Does maximum
weather variable occur at higher latitude than in another region (forecast)
Model
LLM
F1 Score(t)
Precision (%) (t)
Recall (%)(1)
ZEPHYRUS REFLECTIVE
9pt-5-mini
88.9
100.0
80.0
ZEPHYRUS- -IHREC Text Only LLM
gpt-5-mini
88.9
100.0
80.0
gpt-5-mini
62.5
83.3
50.0
ZEPHYRUS-REFLECTIVV
gemini-2 2-5-flash
100,0
100.0
100.0
ZEPHYRUS- EEPHERUS-DIRECT Text Only LLM
94.7
100.0
90.0
gemini-2.5-flast
46.2
100.0
30.0
ZEPHYRUS- REFLECTIVE
upt-oss-120s
94.7
100.0
90.0
ZEPHYRUS-DIISS Text Only LLM
gpt-oss-120b
94.7
100.0
90.0
gpt-oss-122
66.7
100.0
50.0
ZEPHYRUS- REFLECTIVE
REPHYRUS-RELLECIVE
gpt-5.2
90.0
90.0
90.0
ZEPHYRUS- DIRREC Text Only LLM
gpt-5.2
90.0
90.0
90.0
g.t
58.8
71.4
50.0
REFFETTIVE SEHHYRUS-DIREC" Text Only LLM
qwen3- 30b
87.5
87.5
gwen3-30b
90.0
90.0
90.0
gwcn3-30b
0.00
0.00
0.00
Table 35: Boolean classification metrics for Template ID 17: Does mean weather variable in one region exceed another by speci fied amount (forecast.)
38

---

Published as a conference paper at ICLR 2026
Model
LLM
F1 Score(t)
Precision (%)(1)
Recall (t)
9pt-5-min
0.00
0.00
0.00
ZEPHYRUS- DIRECT
gpt-5-mini
0.00
0.00
0.00
Text Only LLM
9pt-5-mint
0.00
0.00
0.00
gemini-2 10ai-2255.1aa.
0.00
0.00
0.00
ZEPHYRUS- DIRREC Text Only LLM
gemini-2. 10.55-f1as.
0.00
0.00
0.00
comini-2.5-flast
0.00
0.00
0.00
ZEPHYRUS-RETH REFLECTIVE
gpt-oss-122 1206
0.00
0.00
0.00
ZEPHYRUS-DIRES Text Only LLM
gpt-oss-120a
0.00
0.00
0.00
opt-oss-120s
0.00
0.00
0.00
SEPHYUS--REFLFOO
gpt-5,2
0.00
0.00
0.00
REPHYRUS-DIRED
gpt-5.2
0.00
0.00
0.00
Text Only LLM
gpt-5.2
0.00
0.00
0.00
ZEPHYRUS- SEHYYRUS-REFECTIVE REFLECTIVE
qwen3-30b
0.00
0.00
0.00
ZEPHYRUS-DIREG Text Only LLM
qwen3-30b
0.00
0.00
0.00
qwen3-30b
0.00
0.00
0.00
Table 36: Boolean classification metrics for Template ID 18: Does mean weather variable
exceed threshold while le maximum of another shays helow (forecast.)
Model
LLM
F1 Score(1) Precision (%)(1)
Recall (%)(t)
ZEPHYRSS-RELLECTIVV REPHYRUS-DIRECT
gpt-5-mini
100.0
100.0
100.0
9pt-5-mini
100.0
100.0
100.0
Text Only LLM
gpt-5-mini
0.00
0.00
0.00
ZEPHYRUS- REFUUUSIVEEEEE
100.0
100.0
100.0
SEHHYRSS-DIRCCE
femini-2.5-f1eee
100.0
100.0
100.0
Text Only LLM
gemini-2.5- flash
0.00
0.00
0.00
REPHYRUS-REFLECHE
gpt-oss-120b
100.0
100.0
100.0
ZEPHYRUS DIRRETTS-DIRECT
gpt-oss-1200
100.0
100.0
100.0
Text Only LLM
gpt-oss-120s
0.00
0.00
0.00
ZEPHYRUS REFLECTIVE
gpt-5.2
100.0
100.0
100.0
SEHHYRUS-DIRCCE Text Only LLM
gpt-5.2
100.0
100.0
100.0
gpt-5.2
0.00
0.00
0.00
ZEPHYRUS-REFLECTIV REFLECTIVE
qwen3-30b
100.0
100.0
100.0
REPHYRUS-DIRECT Text Only LLM
gwen3-30b
100.0
100.0
100.0
qwen3-30b
0.00
0.00
0.00
Table 37: Boolean classification metrics for Template 19: Does mean weather variable
within region exceed specified threshold (forecast)
39

---

Published as a conference paper at ICLR 2026
Model
LLM
F1 Score(t)
Precision (% (1)())
Recall (1)
gpt-5-ming
83.3
83.3
83.3
ZEPHYRUS- DIRECT
gpt-5-mini
83.3
83.3
83.3
Text Only LLM
3pt-5-mins
66.7
100.0
50.0
gemini-2 100--2255.1a5..
83.3
83.3
ZEPHYRUS- DIREETTS-DIRECT Text Only LLM
Sirini-2.5-flaa
83.3
83.3
83.3
1omini-2.5-flss:
50.0
100.0
33.3
ZEPHYRUS- REFLECTIVE
SEPHYRUS-RELLERR ZEPHYRUS-DIRGS Text Only LLM
gpt gpt-oss-11 1200
83.3
83.3
83.3
gpt-oss-120a
83.3
83.3
83.3
opt-oss-120s
28.6
100.0
16.7
SEHHYRUS-REFFCCEO
gpt-5.2
83.3
83.3
83.3
REPHYRUS-DIRED
gpt-5.2
83.3
83.3
83.3
Text Only LLM
gpt-5.2
83.3
83.3
83.3
ZEPHYRUS- REFLECTIVE
SOHHYRSS-RELECTIVE REPHYRUS-DIRECE Text Only LLM
qxen3-30b
80.0
80.0
80.0
qwen3-30b
83.3
83.3
83.3
qwen3-30b
0.00
0.00
0.00
Table 38: Boolean classification metrics for Template II) 20: Does weather variable
exceed threshold within any part of regi on (forecast)
Model
LLM
Fl Score(t) Precision (%)(t)
Recall (%)(t)
REFERUSS-EEEECTIVV ZEPHYRUS- DIRECT Text Only LLM
gpt-5-mini
100.0
100.0
100.0
gpt-5-mini
100.0
100.0
100.0
gpt-5-mini
88.9
80.0
100.0
ZEPHYRUS-
ZEPHYRUS-REFLECTIEE REFLECTIVE CIRRRT
comini-2.5-flee
100.0
100.0
100.0
ZEPHYRUS- DIRECT Text Only L.M
100,0
100.0
100.0
gemini-2.5-f1es
40.0
100.0
25.0
ZEPHYRUS REFLLCTIEE
gpt-oss-120s
100.0
100.0
100.0
ZEPHYRUS-DIRETT Text Only L.L
gpt-oss- 1206
100.0
100.0
100.0
gpt-oss 1205
0.00
0.00
ZEPHYRUS- REFEEUUVVEEEEE
gpt-5.2
100.0
100.0
100.0
ZEPHYRUS- DIPHERTS-DIRECT Text Only LLM
gpt-5.2
100.0
100.0
100.0
gpt-5.2
88.9
80.0
100.0
REETRRSS--FELEIIVEE ZEPHYRUS-DIREE Text Only LLM
qwen3-30b
85.7
100.0
75.0
qwen3-30b
100.0
100.0
100.0
@wcn3-30b
0.00
0.00
0.00
Table 39: Boolean classification metrics for Template ID 21:
Does weather variable
exceed threshold in more grid points in one region than another (forecast)
Model
LLM
F1 Score(t)
Precision (t))
Recall (%)
ZEPHYRUS REFLECTIVE
9pt-5-mini
100,0
100.0
100.0
ZEPHYRUS- -IDRECT Text Only LLM
gpt-5-mini
0.00
0.00
0.00
gpt-5-mini
0.00
0.00
0.00
ZEPHYRUS-REFLECTIVV
gemini-2 2-5-flash
100.0
100.0
100.0
ZEPHYRUS- EEPHERUS-DIRECT
gemini-2.5- 5-fash
100.0
100.0
100.0
Text Only LLM
gemini-2.5-flast
66.7
50.0
100.0
ZEPHYRUS- REFLECTIVE
gpt-oss-122
100.0
100.0
100.0
ZEPHYRUS-DIISS Text Only LLM
gpt-oss-120b
100.0
100.0
100.0
gpt-oss-122
33.3
20.0
100.0
ZEPHYRUS- REFLECTIVE
REPHYRSS-RELLETIVE
gpt-5.2
100.0
100.0
100.0
ZEPHYRUS- DIREEC Text Only LLM
gpt-5,2
100.0
100.0
100.0
gpt-
0.00
0.00
0.00
qwen3- 30b
100.0
100.0
100.0
SEPYYUSS-DREEC
gwen3-30b
100.0
100.0
100.0
Text Only LLM
gwcn3-30b
0.00
0.00
0.00
Table 40: Boolean classification metrics for Template ID 22: Does area where weather
variable exceeds threshold cover more than percentage of region (forecast)
40

---

Published as a conference paper at ICLR 2026
Model
LLM
F1 Score(t)
Precision (%)(1)
Recall (1)
gpt-5-ming
100.0
100.0
100.0
ZEPHYRUS- DIRECT Text Only LLM
gpt-5-mini
100.0
100.0
100.0
gpt-5-mins
66.7
100.0
50.0
SENDYRRUSREELECTIVE
gemini-2 100--22...1a5..
100.0
100.0
100.0
ZEPHYRUS- DIRRECT Text Only LLM
gemini-2.5-fin flasssh
100.0
100.0
100.0
Somini-2.5-flass
0.00
0.00
0.00
ZEPHYRUS-REDH REFLECTIVE
gpt gpt-oss-11 1200
100.0
100.0
100.0
ZEPHYRUS-DIRGS Text Only LLM
gpt-oss-120a
100.0
100.0
100.0
opt-oss-120s
66.7
100.0
50.0
SEHHYRUS-REF.CCOO
gpt-5.2
100.0
100.0
100.0
REPHYRUS-DIRED
gpt-5.2
100.0
100.0
100.0
Text Only LLM
gpt-5.2
66.7
100.0
ZEPHYRUS- REPHYRUS-RELEECIVEE REFLETIVE
qxen3-30b
100.0
100.0
100.0
ZEPHYRUS-DIREG Text Only LLM
qwen3-30b
66.7
100.0
50.0
qwen3-30b
0.0
0.0
0.0
Table 41: Boolean classification metrics for Template II) 23: Does area area-averaged weather
variable exceed threshold while another stays below wwwo:woooreeeeee (forecast)
Model
LLM
FI Score(1) Precision (%)(t)
Recall (%)t)
ZEPHYRUS-REFLECTIVE ZEPHYRUS- DIRECT Text Only LLM
gpt-5-mint
80.0
100.0
66.7
gpt-5-mini
72.7
80.0
66.7
gpt-5-mini
50.0
50.0
50.0
ZEPHYRUS-
ZEPHYRUS-REFLECTIEE REFLECTIVE TIRRRT
1omini-2.5-fl.cn
80.0
100.0
66.7
ZEPHYRUS- DIRECT Text Only L.M
80.0
100.0
66.7
gemini-2.5-f1em
28.6
33.3
25.0
ZEPHYRUS REFLECTEEFE
gpt-oss-120s
80.0
100.0
66.7
ZEPHYRUS-DIRECT Text Only L.LM
gpt-oss- 1206
80.0
100.0
66.7
gpt-oss 120b
75.0
50.0
ZEPHYRUS- REEERUUVVEEEEE
gpt-5.2
100.0
ZEPHYRUS- SIPHERTS-DIRECT Text Only LLM
gpt-5.2
80.0
100.0
66.7
gpt-5.2
83.3
83.3
83.3
MINYYRUS-RFFLCCIVEE ZEPHYRUS-DIREE Text Only LLM
qwen3-30b
75.0
100.0
60.0
qwen3-30b
80.0
100.0
66.7
@wcn3-30b
0.00
0.00
0.00
Table 42: Boolean classification metrics for Template ID 24:
Does maximum weather
variable in one region exceed threshold while another stays below (forecast)
Model
LLM
F1 Score(t)
Precision (% (t)
Recall (%)(1)
ZEPHYRUS- REFLECTIVE
9pt-5-mini
90.9
100.0
83.3
ZEPHYRUS- -IRRET Text Only LLM
gpt-5-mini
90.9
100.0
83.3
gpt-5-mini
83.3
83.3
83.3
ZEPHYRUS-REFLECTIVV
gemini-2 2.5-flash
88.9
100.0
80.0
ZEPHYRUS- EEPRERUS-DIRECT Text Only LLM
1arini-2.5-flan
90.9
100.0
83.3
gemini-2.5-flast
50.0
100.0
33.3
ZEPHYRUS- REFLECTIVE
opt-oss-122
90.9
100.0
83.3
ZEPHYRUS-DIISS Text Only LLM
gpt-oss-120s
90.9
100.0
83.3
gpt-oss-122
75.0
100.0
60.0
ZEPHYRUS- REFLECTIVE
REPHYRSS-RELLETIVE
gpt-5.2
90.9
100.0
83.3
ZEPHYRUS- -IRREC Text Only LLM
gpt-5,2
90.9
100.0
83.3
gpt
80.0
100.0
66.7
qwen3-300 30b
100.0
100.0
100.0
ZEPHYRUS-DIRECT
gwen3-30b
90.9
83.3
100.0
Text Only LLM
qwcn3-30b
40.0
100.0
25.0
Table 43: Boolean classification metrics for Template ID 25:
Does maximum weather
variable within region exceed speci fied threshold wwa.ddddooooooo (forecast)
41

---

Published as a conference paper at ICLR 2026
Model
LLM
F1 Score(t)
Precision (% (1))
Recall (1)
gpt-5-ming
80.0
100.0
66.7
ZEPHYRUS- DIRECT Text Only LLM
gpt-5-mini
90.9
100.0
83.3
3pt-5-mind
76.9
71.4
83.3
SENDYRRUSREELECTIVE ZEPHYRUS- DIPRET Text Only LLM
gemini-2, 10..-2.....a..
100.0
83.3
gemini-2.5-fin 10..5-f1.. flash
100.0
100.0
100.0
1mmini-2.5-llss:
66.7
75.0
60.0
ZEPHYRUS-REDH REFLECTIVE
gpt gpt-oss-11 1206 1200
80.0
100.0
66.7
ZEPHYRUS-DIRGS Text Only LLM
gpt-oss-120a
80.0
100.0
66.7
opt-oss-120s
60.0
60.0
60.0
SEHYYRUS-REFFCCEO
gpt-5.2
90.9
100.0
83.3
REPHYRUS-DIRED
gpt-5.2
90.9
100.0
83.3
Text Only LLM
gpt-5.2
72.7
80.0
ZEPHYRUS- SEHYYRSS-RELECTIVE REFLECTIVE
qxen3-30b
66.7
100.0
50.0
ZEPHYRUS-DIREG Text Only LLM
qwen3-30b
85.7
100.0
qwen3-30b
33.3
100.0
20.0
Table 44: Boolean classification metrics for Template II) 26:
Does maximum weather
variable occur at latitude farther north than in another region (forecast)
Model
LLM
Fl Score(t)
Precision (% (t)
Recall (%)(t)
ZEPHYRUS-REFLECHT ZEPHYRUS-DIRECT
9pt-5-mini
88.9
100.0
80.0
gpt-5-mini
88.9
100.0
80.0
Text Only LLM
gpt-5-mins
57.1
100.0
40.0
gemini-22 2--asssh
88.9
100.0
80.0
ZEPHYRUS-DIRECT
gemini-2. 522.555faa.. flash
100.0
80.0
Text Only LLM
gemini-2 2-
50.0
66.7
40.0
ZEPHYRUS- RENRUUSSS-EFFEEIIVEE
gpt--os-- 1205
100.0
80.0
ZEPHYRUS-DIes Text Only LLM
gpt- 1206
88.9
100.0
80.0
opt-oss-120p
57.1
66.7
50.0
ZEPHYRUS- SETRRUSSREFLECTVVE
gpt-5.2
88.9
100.0
80.0
EEPHYUSS-DIRRS Text Only LLM
g.t-5.2
75.0
100.0
60.0
gpt-5.2
60.0
60.0
60.0
ZEPHYRUS- CERUUS-REFLECTEE REFLECTIVE ZEPHYRUS- DIRECT Text Only LLM
@wen3-30b
75.0
100.0
60.0
gwen3-30b
57.1
100.0
40.0
qwen3-306
0.00
0.00
0.00
Table 45: Boolean classification metrics for Template II) 27:
Does maxi mum weather
variable stay above threshold while le another stays below (forecast) Model
LLM
F1 Score(t) Precision (t)
Recall (%)(1)
ZEPHYRUS REFLECTIVE
9pt-5-mini
75.0
75.0
75.0
ZEPHYRUS- -IPRETTS-DIRECT Text Only LLM
gpt-5-mini
88.9
80.0
100.0
gpt-5-mini
40.0
100.0
25.0
ZEPHYRUS-REFLECTIVV
gemini-2 5.
75.0
75.0
75.0
ZEPHYRUS- SEPHYRSS-DIRECT Text Only LLM
1iiini-2.5-flast
75.0
75.0
75.0
gemini-2.5-flas:
57.1
66.7
50.0
ZEPHYRUS- REFLECTIVE
upt-oss-120s
75.0
75.0
75.0
ZEPHYRUS-DIISS Text Only LLM
gpt-oss-120b
75.0
75.0
75.0
gpt-oss-122
0.00
0.00
0.00
ZEPHYRUS- REFLECTIVE
EAPHYRUS-REPLETIVE
gpt-5.2
75.0
75.0
75.0
ZEPHYRUS- DIRREC Text Only LLM
gpt-5.2
75.0
75.0
75.0
gpt-
66.7
100.0
50.0
REFLECTIVEE ZEPHYRUS-DIRER Text Only LLM
qwen3- 30b
85.7
100.0
75.0
gwen3-30b
75.0
75.0
75.0
@wcn3-30b
0.00
0.00
0.00
Table 46: Boolean classification metries for Template ID 28:
Does maximum
weather variable in one region exceed another by speci fied amount (forecast)
42

---

Published as a conference paper at ICLR 2026
Model
LLM
F1 Score(f)
Precision (%)())
Recall (1)
gpt-5-ming
90.9
83.3
100.0
ZEPHYRUS- DIRECT Text Only LLM
gpt-5-mini
90.9
83.3
100.0
gpt-5-mins
33.3
100.0
20.0
SEAYYRUS-REFLECTIVEE
gemini-2 1001-2255.1a5..
71.4
100.0
ZEPHYRUS- DIRRET Text Only LLM
gemini-2.5-
90.9
83.3
100.0
1omini-2.5-llas:
33.3
100.0
20.0
ZEPHYRUS-REDH REFLECTIVE
gpt gpt-oss-11 1200
90.9
83.3
100.0
ZEPHYRUS-DIRGS Text Only LLM
gpt-oss-120a
90.9
83.3
100.0
opt-oss-120s
85.7
100.0
75.0
SEPHYRUS-REFLFOE
gpt-5.2
90.9
83.3
100.0
REPHYRUS-DIRED
gpt-5.2
90.9
83.3
100.0
Text Only LLM
gpt-5.2
75.0
100.0
60.0
ZEPHYRUS- SERRRUSREEEEEIIVEE
qxen3-30b
90.9
83.3
100.0
REPHYRUS-DIREC Text Only LLM
qwen3-30b
83.3
100.0
qwen3-30b
50.0
100.0
33.3
Table 47:
Boolean classification metrics for Template II) 29:
Does minimum weather
variable within region remain above threshold (forecast)
Model
LLM
MAE (median)(L)
ZEPHYRUS-REFLEETVVV
ZEPHYRUS- REFLECTIVE
gpt-5-mini
191,000
ZEPHYRUS- RUPRRETT Text Only LLM
gpt-5-mini
182,000
gpt-5-mini
249,000
ZEPHYRUS-REFLLECIVV
ZEPHYRUS- REFLECTIVE
5-flash
183,000
ZEPHYRUS- DIREET Text Only LLM
1iiini-2.5-flese gemini-2.5-flaas
182,000
150,000
ZEPHYRUS-REFLECIVV
ZEPHYRUS- REFLECTIVE
gpt-css-120b
3,860
ZEPHYRUS-DIREE Text Only LLM
gpt-css- 1200
0.00
gpt-css- 120b
1,84c+06
ZEPHYRUS- RUS-REFLECTIVE
9pt-5.2
0.00
SERIYRUS-DRRECE
gpt-
0.00
Text Only LLM
9p2
174,000
REU-HYRUS-REEEECCE ZEPHYRUS- DIRECT Text Only LLM
qwen3 30b
0.00
qwen3-30b 30b
232,000
qwen3-30s 300b
182,000
Table 48: Performance metries for Template ID What is the area where muitiple
weather variables exceed their percentile values (forecast)
Model
LLM
MAE (median)(()
REF-RREEEEEIIVEE DESRECTS-DIRECT
gpt-5-ming
103,000
gpt-5-mini
29,200
Text Only LLM
gpt-5-mini
349,000
ZEPHYRUS- RU-HIVRURRRELEETIEEE
gemi ni..iii-25---- 5-flash
23,800
ZEPHYRUS- DIRRED
gemini -2. 5-flash
19,200
Text Only LLM
gemini-2.5-1 5-
20,400
ZEPHYRUS-
SEAPHYRUS-RFFRCTTVVE REFLECTIVE
gpt-css- 220
13,600
TEPHYRUS-DIRECE Text Only LLM
gpt-css- 120b gpt-oss-120s
13,600
4.26e+12
TEPHYRUS-REES REFLECTIVE
gp2-5.2
13,600
ZEPHYRUS-DIREE DIRECTT Text Only LLM
gpt-5.2
13,600
gpt-
15,800
ZEPHYRUS REFLEEEEVEEEVVEE RIPRECTS-DIRECT Text Only LLM
gwen3-30b
16,000
ZEPHYRUS- DIRECT
qwen3-30s
13,600
qwen3-2 30b
19,100
Table 49: Performance metrics for Template ID 31: What is the area where weather variable exceeds its median value (forecast)
43

---

Published as a conference paper at ICLR 2026
Model
LLM
MAE (median)(
SEPHHYUSS-RELECCIE DEPRERUS-DIRECT
gpt-5-mini
149.0
gpt-5-mini
96.0
Text Only LLM
gpt-5-mini
293.0
ZEPHYRUS- 2EPLYRRS-REFLLECTVVE REFLETIVE
gemini 5-flash
171.0
ZEPHYRUS- DIRUET Text Only LLM
Scmini-2.5-4 5-flash
163.0
5-flash
2,640
RUPPYYUS-REFLEEEE
gpt-cas- 120b
103.0
ZEPHYRUS- DIRECT
gpt-css-120b
99.9
ZEPHYRUS- REFLECTIVV
g.t-5.2
104.0
HEPHYRUS-DRREGH
gpt-5.2
104.0
Text Only LLM
gpt-5.2
531.0
HEPHYRUS-REFLECTVV ZEPHYRUS- REFLECTIVE
qwen3-305
197.0
ZEPHYRUS-DII RUS-DIRECT Text Only LLM
qwen3-30b
163.0
qwen3-303
11,600
Table 50: Performance metrics for Template ID 32: What. is the displacement between
centroids of areas with above-median values (forecast)
Model
LLM
MAE (median)(L)
HEPHYRUS-REPLECTII
gpt-5-mini
107.0
ZEPHYRUS-DIRECE Text Only LLM
gpt-5-mini
137.0
gpt-5-mini
1.91c+06
ZEPHYRUS- REFLEEEETVV
gemini 5-fash
112.0
ZEPHYRUS-DIREE Text Only LLM
gemini 12.mni22.55 22. 5-lash
164.0
gemini-2.5-flas
426.0
gpt-css-11 11sss-100bb 120b
158.0
TEPHYRUS-DIREGE
gpt-oss- 120b
163.0
Text Only LLM
gpt-oss-10bb
122.0
REPHYRUS-REEH REFLECTIVE TEPHYRUS-DIRECT Text Only LLM
gpt-5.2
138.0
gpt-5.2
107.0
183.0
ZEPHYRUS- REFLECTIVE
qwen3-30b
107.0
ZEPHYRUS- RUR--DECCTT Text Only LLM
qwen3-300 30b
136.0
qwen3-22 30b
9,110
Table 51: Performance metrics for Template ID 33: What is the distance between
centroids of maximum weather vari aable value areas (forecast)
Model
L.M
SAE (Q25)J)
SAE (Q50IJ)
SAE (Q75)()
SAF (Q99)()
Err Std(.)
SEMIYYRSS-RELLECTIV
min1
0.00
0.02
0.09
0.95
15.1
LEPHYRUS-DIREC Text Only LLM
got-S-mini
1.01
15.1
gpt-S-mini
0.21
0.67
1.26
2.72
155.0
REPHYRUS-RELL REREEEEEEEEEE
gemini-2.b-tlaas
0.00
0.02
0.09
0.95
15.1
ZUPIYRUS-DIRES
gemini 2.5 flash
15.1
Text Only LLM
gemini-2.5-f1ash.
0.05
0.47
1.41
5.30
$10.0
ZEPHYRUS. REFLECTIVE
ept-oss-i200
0.00
0.02
0.09
0.95
15.1
ZEPHYRUS-DIIG Text Only LLM
ept-oss-1203
15.1
gpt oss-oss- oss--ss---- oss-120b
0.16
0.21
0.50
1.24
459.0
ZEPHYRUS- REFLECTIVE
gpt-5.2
0.00
0.02
0.09
0.95
15.1
ZEPHYRUS. Text Only LLM
gpt-5.2
gpt-5.2
0.35
0.58
4.62
311.0
32.0
ZEPHYRUS-K REFU-REFLEETIVE
qwen3-30b
0.00
0.02
0.09
0.95
15.1
gwen3-30b
15.1
Text Only LLM
qwen3-30b
0.07
0.17
1.40
7.1
203.0
Table 52: Standardized Absolute Error (SAE) quantiles for Template ID 34: What is the maximum difference in weather variable between grid points within region (forecast)
44

---

Published as a conference paper at ICLR 2026
Model
LLM
SAE (Q25)])
SAE (Q50)()
SAE (Q75)()
SAE (Q99)())
Err Std(
ZEPHYRUS- REFLECTIVE
ZEPAYRUS--RPPLECIVVE
gpt-5-mini
0.58
ZEPHYRUS DIRECT Text Only LLM
gpt-5-mini
0.00
0.03
0.03
0.13
0.61
qpt-b-min1
0.02
0.09
0,50
1.14
3.07
REFEECIVVEEE
Remini-2.555 3-5ahhhhh
0.58
ZEPHYRUS DIRECT
gemini-2.5-$lame
0.01
0.03
0.03
0.13
0.58
Text Only LLM
gomini-2.55-1aa
0.17
0.31
1.00
4.25
17.5
ZEPHYRUS REFLECTIVE
spt-oss-1205
0.03
0.03
0.13
0.58
3pt-oss-1206
0.01
0.03
0.03
0,13
0.58
Text Only LLM
spt 088 120b
0.30
0.30
0.30
0.30
0.00
ZEPHYRUS-
ZEPAYRUS-REPLECTIVE REFLECTIVE
gpt-5.2
0.58
ZEPHYRUS DIRECT Text Only LLM
gpt-5.2
0.01
0.03
0.03
0.13
0.58
gpt-b.2
0.10
0.39
0.92
3.01
12.7
ZEPIYRUS-REPLECT
quen3 30b
0.56
ZEPHYRUS DIRECT
qwen3-30b
0.01
0.03
0.03
0.13
0.58
Text Only LLM
qwan3-30b
7.88
15.2
15.8
17.7
110.0
Table 53: Standardized Absolute Error (SAE) quantiles for Template ID 35:
what is
the minimum weather variable where another variable exceeds median (forecast)
Model
LLM
SAE (Q25KJ)
SAE (Q50K))
SAE (Q75K()
SAE (Q99)()) Std(.)
ZEPHYRUS-RER SERNYRUSSRRELFEOE REFLECTIVE ZOPIYRUS-DIRECT
qpt-b-mini gpt mini
0.10
0.23
0.45
1.54
5.84
0.16
0.23
0.35
1.82
6.80
Only
gpt-5-mini
20.5
ZEPHYRUS REFLECTIVE
0.15
0.25
0.75
1.49
6.42
ZEPHYRUS- DIRRECT
gemini-2.b-tlen laah
0.23
0.33
0.56
1.48
5,50
Text Only LLM
gemini
0.72
2.04
2.76
18.2
SEHHYRUS-REPLECTIN
ept-oss-1206
0.11
0.23
0.50
1,55
5,93
ZEPHYRUS-DIES -D-RERETT
opt-oss-122
0.13
0.27
0.91
5.10
18.3
Only
spt-oss-120b
2.00
7.77
ZEPHYRUS- REFL ECTIVE
gpt-5.2
0.15
0.44
0.85
5.18
19.0
ZOPHYRUS-DIRG DIRREC
gpt gpt-5.2 5.2
0.13
0.44
0.96
5,10
18.2
Only
gpt-5.2
0.39
300.0
14.9
ZEPHYRUS REFLECTIVE
qwen3-30b
0.20
0.33
0.52
1,80
6.55
ZEPHYRUS- DIRECT
qwen3-30b
0.24
0.38
1.07
6.28
23.1
Text Only LLM
qwen3
2.13
19.3
98.5
Table 54: Standardized Absolute Error (SAE) quantiles for Template ID 36: What is the difference between maximum weather variables in two regions (forecast)
45

---

Published as a conference paper at ICLR 2026
Model
LLM
SAE (Q25)])
SAE (Q50)()
SAE (Q75)()
SAE (Q99)())
Err Std(
ZEPHYRUS- ZEPAYBUS--RPPLECIIVE
REFLECTIVE
gpt-5-mini
0.04
0.07
0.67
238.0
ZEPHYRUS DIRECT Text Only LLM
gpt-5-mini
0.04
0.07
0.21
0.67
238.0
apt-b-min1
0.25
1.03
2.09
5.79
1,010
REPHYRUS-REELEECIVE
remini-2.55 2-5ahhh
0.04
0.06
0.24
0.49
247.0
ZEPHYRUS DIRECT
gemini-2.5-$lam
0.04
0.12
0.32
0.68
250.0
Text Only LLM
gomini-2.55-1aa
0.41
1.04
2.23
30.9
1,130
ZEPHYRUS REFLECTIVE
spt-oss-1205
0.06
0.12
0.33
0.67
238.0
EEPRRRUS-DREEC
3pt-oss-1206
0.04
0.96
0.21
0.67
238.0
Text Only LLM
spt 088 120b
0.26
0.99
1.94
2.58
4.31
ZEPHYRUS- ZEPAYRUS-REPLECTIV REFLECTIVE
gpt-5.2
0.04
0.07
0.67
2.38.0
ZEPHYRUS DIRECT Text Only LLM
gpt-5.2
0.04
0.07
0.21
0.67
238.0
gpt-b.2
0.53
0.77
1.35
19.5
772.0
ZEPHYRUS- REPLEEVVVE
quen3- 30b
0.04
0.06
0.67
238.0
ZEPHYRUS DIRECT Text Only LLM
qwen3-30b
0.04
0.10
0.36
3.10
237.0
qwan3-30b
0.23
2.16
30.6
31,500
724.0
Table 55: Standardized Absolute Error (SAE) quantiles for Template ID 37:
what: is
the di difference in area weeea-weihted mean weather variable between two regions (forecast)
Model
LIM
MAE (median)(.)
ZEPHYRUS-REFLECTIV REFLECTIVE
gpt-5-mini gpt-b-mini
36.2
ZEPHYRUS- RUS-DIRECT Text Only LLM
36.2
gpt-5-mini
258.0
RUSHYRUS-REFLETEE REFLECTIVE
135.0
ZEPHYRUS-DIKE Text Only LLM
gemini-2.5-Elash
73,9
gemini-2.5-
275.0
ZEPHYRUS-REEHE REFLECIVEE
ppt-css-120s
73.9
ZEPHYRUS-DIRES Text Only LLM
gpt-oss-1 120b
73.8
gpt-css-120b
1,990
SEEHHYRUSRRFFLECTEE
gpt-5.2
36.2
ZEPHYRUS RUS-DIRECT
9pt-5.2
36.2
Text Only LLM
gpt-5.2
51.9
ZEPHYRUS- REFEVVVE
gwen3-30b
302.0
ZEPHYRUS DRUE-DI
qwen3-30b
43.8
Text Only LLM
qwen3-303
258.0
Table 56: Performance metrics for Template ID 38: What is the displacement of minimum weather variable location after time window (forecast)
46

---

Published as a conference paper at ICLR 2026
Model
LLM
MAE (median)(
gpt-5-mini gpt-5-mini gpt-5-mini
5.19
DEPRERUS-DIRECT
4.38
Text Only LLM
8.66
ZEPHYRUS- REFEEEEEEVVVEVEEE
gemini 22 5-flash
4.13
ZEPHYRUS- DIRUET
S 5-flash
2.49
Text Only LLM
gemini-2.
9.54
RUSHYRUSSREELEER REFLECTIVE
gpt-cas- 120b
6.82
ZEPHYRUS RUS-DIRECT
gpt-css-120b
0,88
Text Only LLM
gptcos--10b
21.2
ZEPHYRUS- REF-VVEEEE
gpt-5.2
3.83
ZEPHYRUS-DIEE DIS-DI
gpt-5.2
2.04
Text Only LLM
3.24
ZEPHYRUS- ZEPHYRUS-REFLECTIV REFLECTIVE TEPHYRUS-DIRECE Text Only LLM
qwen3-30b
4.93
qwen3-30b
5.30
qwen3-305
21.2
Table 57: Performance metrics for Template ID) 39: What is the latitude di f ference between centroids of high weather variable areas (forecast)
Model
L.M
SAE (Q25K() (AE(())))
SAE (Q75K()
SAE (Q9))()
Err Std(.)
ZEPAYAUS-REFLLECIVEE
ZEPHYRUS- REFLECTIVE
9pt-5-mini
0.01
0.14
0.51
0.95
28.3
ZEPHYRUS DIRECT
gpt-S-mini
0.10
0.51
0.95
28.3
Text Only LLM
opt-b-mini
0.53
1.16
2.99
3.79
2,880
ZAPAYRUS-KELLECIIVE REFLECTIVE gemini-2. 5-Elash gemini-2 -2.5flas
0.07
0.28
0.58
0.90
19.8
ZEPHYRUS DIRECT
0.07
0.13
0.28
0.56
28.4
Text Only L.M
2emini-2.5-flass
0,28
0.45
1.52
3,28
2,760
RENIYRUS-REPLLCTIVE
spt 120b spt-oss-1203 spt-oss-120b
0.01
0.14
0.51
2.55
28.2
ZEPHYRUS- DDDRECT
0.14
28.3
Text Only LLM
1.83
1.91
9.95
17.7
10.3
ZEPHYRUS-REFL RETEIVIVEE
qpt-b.2
0.01
0.09
0.51
0.80
28.3
ZUPHYRUS-DIRIC Text Only LLM
g.t-5.2 5.2
28.3
gpt-5.2
0.14
0.80
2.00
21.1
1,460
ZEPHYRUS- REFL ECTIVE ZEPHYRUS-DIRG Text Only LLM
qwen3-30b
0.01
0.14
0.51
0.95
28.3
quen3 30b
28.3
qwen3-30b
0.39
1.73
3.06
1.61c=06
2,660
Table 58: Standardized Absolute Error (SAE) quantiles for Template ID 40: What is the max immm weather variable difference between two regions (forecast)
47

---

Published as a conference paper at ICLR 2026
Model
LLM
SAE (Q25)])
SAE (Q50)()
SAE (Q75)()
SAE (Q99)())
Err Std(.)
ZEPHYRUS- ZEPAYBUS--RPPLECIVV REFLECTIVE
gpt-5-mini
0.00
0.02
0.07
5.84
ZEPHYRUS DIRECT Text Only LLM
gpt-5-mini
0.00
0.02
0.07
0.21
5.84
apt-b-min1
0.44
0.70
1.95
32.9
259.0
ZEPHYRUS- REFEECIVVEEE
gemini-2.2. 2255-
0.04
0.18
4.51
10.7
3,290
ZEPHYRUS DIRECT
Lemini-2.5-$lame
0.00
0.02
0.07
0.21
5.84
Text Only LLM
gemini- comini-2.5-flass
0.47
0.99
1.90
3.96
28.8
ZEPHYRUS REFLECTIVE
spt-oss-1205
0.04
0.11
1.50
6.31
ZEPHYRUS- DEPRRRT--DREETT
ept=oss-1200
0.01
0.04
0.11
1.50
6.31
Text Only LLM
spt 038 120b
0.94
0.94
0.94
0.94
0.00
ZEPHYRUS-
ZEPAYRUS-REPLECTIVE REFLECTIVE
gpt-5.2
0.00
0.02
0.07
5.84
ZEPHYRUS DIRECT Text Only LLM
gpt-5.2
0,02
0.04
0.07
0.21
6.19
gpt-b.2
0.26
0.46
1.33
2.70
172.0
ZEPHYRUS- ZEPIYAUS-RUPLECIVE
quen3 30b
0.07
5.84
ZEPHYRUS DIRECT
qwen3-30b
0.02
0,06
0.15
1.49
6.52
Text Only LLM
qwan3-30b
0.98
7.41
20.0
875,000
307.0
Table 59: Standardized Absolute Error (SAE) quantiles for Template ID 41: What is the mean
weather variable where another variable exceeds median (forecast)
Model
LLM
SAE (Q25K))
SAE (Q50K))
SAE (Q75)()
SAE (Q99)(L)
Err Std(.)
SEHHYRRS-RRFLCCTVV ZUPIYRUS-DIRECT
gpt-b-mini gpt gpt-5-mini
0.06
0.11
0.36
1.19
251.0
0.14
0.36
L19
352.0
Text Ouly LLM
0.40
0.59
1.23
17.6
1,990
ZEPHYRUS- REFLECTIVE
gemini-2.5--flash
0.06
0.15
0.38
0.97
270.0
ZEPHYRUS-DIRES DDDRECT
çemini-2.9-tlle
0.06
0.14
0.36
1.19
252.0
Text Only LLM
gemini-22 2.5 51ash
0.55
1.20
2.42
29.2
58,500
ZEPHYRUS REFLECTIVE
spt-oss-120b
0.06
0.14
0.36
1.19
250.0
ZEPHYRUS DIRECT
3pt-oss-1206
0.06
0.36
1.18
251.0
Text Only LLM
spt (EE--08--2000
2,70
4.38
6.06
7.68
0.00
ZEPHYRUS- REFLEEVVVVVEE
gpt-5.2
0.06
0.36
1.19
251.0
ZEPHYRUS DIRECT
gpt-5.2
0.06
0.11
0.36
1.19
251.0
Text Only LLM
gpt-5.2
0.25
0.79
1.15
1.71
1,760
REPHYAUS-REFLECTIVE ZEPHYRUS DERRUT Text Only LLM
quen3 30b
0.18
3.12
366.0
qwen3-30b
0.09
0.17
0.41
1.31
372.0
gwen3-30b
4.58
13.7
28.6
63,500
68,300
Table 60: Standardized Absolute Error (SAE) quantiles for Template ID 42:
What is the weather variable value where another variable reaches maximum (forecast)
Model
LLM
Score (mean)(t)
Score (median)(
ZEPHYRUS-REFLECTIG REFLECTIVE
gpt gpt-5-na b-t-b-miii gpt-5-ming
0.02
0.00
SOPHYRUS-DIREGS Text Only LLM
0.00
0.00
gpt-b-mini
0.00
0.00
WOPHYRUS-REFLECIIVG
gemini-2.5-f1es
0.00
0.00
ZEPHYRUS- ZEPHYRUS-DIRECT DIRRECT Text Only LLM
0.00
0.00
comini-2.5-flas
0.00
0.00
gpt-oss-120b
0.00
0.00
ZEPHYRUS-DIRER -DRRCC
gpt-oss-120b
0.00
0.00
Text Only LLM
gpt-oss-120b
0.00
0.00
REPHYRUS-REFLECEE ECTECTIVVE
gpt-5.2
0.04
0.00
ZEPHYRUS-DIREE -IRRECT Text Only LLM
g.t-5.2
0.00
gpt-5.2
0.00
0.00
ZEPHYRUS- REFLEECTIVE
qwen 3- 3-30b 306
0.00
SOPHYRUS-DIREER Text Only LLM
3-30b
0.00
0.00
gwen3-30b
0.00
0.00
Table 61: Discussion metries for Template ID 43:
Generate comprehensive
global climate forecast for temperature and precipitation for next months (forecast)
48

---

Published as a conference paper at ICLR 2026
Model
LLM
Score (mean)(1)
Score (median)(
SEPPIYRUS-RELLECTIEE
gpt-5-mine gpt-5-mini
0.40
0.42
SEHHYRUS-DIREGE Text Only LLM
0.12
0.10
gpt-5-mini
0.09
0.04
SEHHYRUS-REFLECTIV ZEPHYRUS-DIRCS DIRECT
gemini gemini-2.5- 5-
1...ni-2.5----- 5-flash
0.15
0.11
0.09
Text Only LLM
gemini-2.5-Ei 5-flash
0.07
0.00
RAPHYRUS-REFLECCE ECTIVE
gpt-oss-1205
0.16
0.09
ZEPHYRUS- DIREETT
gpt-oss-120b
0.10
0.04
Text Only LLM
gpt-os--200
0.03
0.00
HAPHYRUS-REFLFCOE
gpt-5.2
0.43
0.46
SETHYRUS-DIRECH
gpt-5.2
0.26
0.30
Text Only LLM
gpt-5.2
0.14
0.12
qwen3-30b
0.10
0.02
ZEPHYRUS-DIRER
qwen3-30b
0.03
Text Only LLM
qwen3-30b
0.03
0.00
Table 62:
Discussion score
metrics for Template II) 44:
Provide detai led United
meteorological discussion and forecast for States (forecast)
Model
LLM
Score (mean)(1)
Score (median)(1)
ZEPHYRUS-REFLECTIVE
gpt opt-5-mini opt-5-mini
0.43
0.41
ZEPHYRUS-DIRECT Text Only LLM
0.25
0.22
gpt-5-mini
0.19
0.13
ZEPHYRUS-REELECEE
gemini -emini-2.5-flase
0.17
0.08
ZEPHYRUS-DIRECT
gemini-2.5--f
0.10
Text Only LLM
gemini-2.5 -2.5-flssh
0.13
0.05
gpt-oss-1200
0.25
ZEPHYRUS-DIRES
gpt-css-1209 120b
0.21
0.18
Text Only LLM
gpt-oss-120b
0.07
0.00
SOPHYRRSS-REFECTIVV
gpt-5.2
0.47
0.49
STTHYRUS-DIRECH Text Only LLM
gpt-5.2
0.28
0.27
gpt-5.2
0.26
0.29
ZEPHYRUS- REFLECTIVE ZEPHYRUS- DIRECT Text Only LLM
qwen3-30b
0.11
0.02
swen3-30D
0.11
0.03
qwen3-30b
0.11
Table 63: Discussion score metrics for Template ID 45: Generate ENSO climate update and outlook based atmospheric data (forecast)
Model
LLM
SAE (Q25)()
SAE (Q50(()
SAE (Q75)()
SAE (Q99)()
Err Std(.)
ZEPHYRUS-REFLECTION REFLECTIVE
gpt-5-mini
0.00
0.04
0.09
0.36
292.0
gpt-5-mi
357.0
Text Only LLM
opt-8-mini
0.24
0.29
0.34
0.40
549.0
JEPHYRUS-KEFLBECK REFLECTIVE
gemini-2.55 5-flash
0.02
0.08
0.22
7.84
7,330
gcmini-2.55f1..
416.0
Text Only L.M
0.24
0.29
0.34
01.40
535.0
ZEMIYRUS-REr.DCTH
gpt 03s 120b
0.03
0.07
0.19
1.02
352.0
ept-oss-120a
0.03
286.0
Text Only LLM
spt-oss-120b
0.24
0.29
0.34
0..0
$49.0
SOPHYYUS-RRELECHE ZEPHYRUS-DIRECT
gpt-5.2
0.02
0.05
0.17
0.69
449.0
g.t-5.2
0.05
4,710
2.57c406
Text Ouly LLM
gpt-5.2
0.24
0.29
0.34
0.40
$49.0
7EPHYRUS-REF REFL ECTIVE ZAPIYRUS-DIRECE Text Only LLM
qwan3-30b
0.05
0.16
0.39
22.4
25,500
quen3 30b
0.04
0.13
0.30
1.170
$7,900
qwen3-30b
0.00
0.28
0.36
10,500
372.0
Table 64: Standardized Absolute Error (SAE) quantiles for Template ID 46:
How will weather variable change after specified time given an intervention (counterfactual)
49

---

Published as a conference paper at ICLR 2026
Model
LLM
MAE (median)(
ZEPHYRUS- RUFEETEE
gpt-5-mini gpt-5-mini gpt-5-mini
0.12
ZEPHYRUS- REPRERUS Text Only LLM
0.15
0.27
ZEPHYRUS- REFFEEERVEEEEE
gemini 22 5-flash
0.08
ZEPHYRUS- DIRRET
1.iini-2.5--- 5-flash -
0.07
Text Only LLM
gemini-2.
0.24
RUSHRRUSSRRELEER EEFLECTIVV
gpt-cas- 120b
0.22
ZEPHYRUS RUS-DIRECT
gpt-css-120b
0.24
Text Only LLM
gptcoss-10b
0.30
ZEPHYRUS- REF-VEEEEE
gpt-5.2
0.20
ZEPHYRUS-DIEE DIUS-DIRECT
gpt-5.2
0.28
Text Only LLM
gpt-5.2
0.29
ZEPHYRUS- RUSTRRUUE-RRREEEE REFLECTIVE
qwen3-30b
0.36
HAPHYRUS-DIRECT Text Only LLM
qwen3-303
0.37
3wen3-30b
0.26
Table 65: Performance metrics for Template II) 47: What is the value of the input parameter of the simulator model that produces the simulation output
Model
LIM
Correctness (%)(1)
EMD (km)(L)
ToU (% ((t)
ZEPHYRUS-RERI EETLECTVVE
3pt-5-ming mi
14.3
8,626
0.00
HEPHYRUS-DIREGG Text Only LLM
gpt-5-mini
11.9
6,572
0.00
gpt-5-mini
28.6
6,699
0.00
ZEPHYRUS- REFUITIVEEE
gemini-2.5-flee
19.0
5,409
0.00
ZEPHYRUS- DIRECT
gemini-2.5-flast
11.9
5,932
0.00
Text Only LLM
gemini-2.5-fles
28.6
0.00
ZEPHYRUS- REFLECTIVE
gpt-oss-120b
14.3
5,795
0.00
ZEPHYRUS- TEPHYRUS-DIRECT
gpt-oss-120b
14.3
5,918
0.00
Text Only LLM
gpt-oss-120b
28.6
0.00
ZEPHYRUS-RERT REFLECTIVE
gpt-5.2
19.0
6,832
0.00
ZEPHYRUS- DIRECC
gp
19.0
7,553
0.00
Text Only LLM
gpt-5.2
28.6
0.00
JEPHYRUS-REFLECHE ECTIVE ZEPHYRUS- TIRTET Text Only LLM
qwen3-30b
23.8
4,443
0.00
qwen3-30b
19.0
6,910
0.00
qwen3
28.6
0.00
Table 66: Extreme weather detection metrics for Template ID 48: Where is target disaster currently happening
50

---

Published as a conference paper at ICLR 2026
Model
LLM
F1 Score(t)
Precision (%)()
Recall (%)(1)
HEPHYRUS-REFLECTIV
gpt-5-mint
32.4
61.1
22.0
ZEPRYRUS-DIRECT
gpt-5-mini
33.1
66.7
22.0
Text Only LLM
9pt-5-mini
49.5
47.0
CETHYRUS-REFLECTIOE
gemini-2 5- Flash
37.3
51.9
29.2
ZEPHYRUS-DIRS
gemini-2. 5- flash
62.2
23.0
Text Only LLM
gemini-2.5-121 flash
7.10
33.3
4.00
ZEPHYRUS-REFLECTIN
gpt -pt-oss-122 120b 9pt-oss-12000
51.2
21.0
JEPHYRUS-DIRENS
26.5
50.0
18.0
Text Only LLM
gpt-oss-1200
0.00
0.00
0.00
ZEPHYRUS-
ZEPHYRUS-REPLECTIV REFLECTIVE
gpt-5.2
26.2
56.7
17.0
ZEPHYRUS-DISE Text Only LLM
gpt-5.2
26.2
56.7
17.0
gpt-5.2
7.00
26.7
4.00
ZEPHYRUS- ZEPHYRUS-REFLECTEE R
qwen3-30b
15.6
29.4
10.6
ZEPHYRUS- -IRRRU Text Only I.M
qwen3-300 30b
55.8
24.2
qwen3-300
0.00
0.00
0.00
Table 67: Boolean classification metrics for Template ID 49:
Check whether the given
claim extracted from meteorological report is supported by the data
51


Reviews:
[{"review_id": "v2ou6ryt5z", "reviewer": "Reviewer_GNmT", "summary": "This paper presents Zephyrus, an AI agent that can perform tasks in weather science such as identifying weather events from data, an API / code execution server and a new benchmark.\n\nThe LLM based agent is connected to a code execution server, that provides access to several weather science related APIs.\nThe agent uses either single step code generation/execution or multistep, with several execution / result steps before producing the result\n\nThe benchmark is produced from two types of task templates: \n\n* manual (grad student) written pairs of template + reference implementation\n* semi-synthetic: mines patterns from weather related texts to produce templates and solution code. Templates are human verified\n\nActual benchmark examples are generated by setting variables such as time and location in the \ntemplates.\n\nEvaluation is done conditional on the answer type, for example locations are resolved and compared with earth movers distance. Text based answer are compared after deconstruction into sets of claims.\n\nExperimental results confirm, as expected, that allowing several agentic steps improves results and that any access to code execution is much better than a non-agentic baseline. An interesting outcome is that this difference gets smaller as the tasks get harder.", "questions": "From what I understand, all problems for a template can be solved by a similar generated code with filled in variables. Does this mean that the performance of the evaluated models correlated strongly within a template cluster?\n\nCan you provide more statistics on how the APIs were used? Which problems occured, if code errors were correctly fixed from the feedback, etc.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Significant contributions: Agent, Framework + Benchmark\n\nOverall well written paper with good related work section", "weaknesses": "While the overall number of examples in the benchmark is over 2k, these are based on only 46 templates. It is not clear to this reviewer whether having several examples per task improves evaluation strength.\n\nGiven that this work evaluates a complex agentic system, it would be interesting to see more in-depth analysis of the behaviour and comparison between common problems between models\n\nGiven that the number of examples is not too high, evaluating top line models like GPT5 or o3, Claude Opus or Gemini Pro would be interesting and not too cost intensive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rcxtiKaCst", "reviewer": "Reviewer_M2k6", "summary": "This paper introduces an agentic framework - Zephyrus - to enable LLMs to interact with weather data via many datasets/tools to address weather science tasks. They also introduces a benchmark (ZephyrusBench) to evaluate how well frontier LLMs can assist with whether science queries/tasks both directly and within their agentic framework. Their agentic framework includes access to different tools to address weather queries including observation data (from ERA5 via WeatherBench2), Geolocator, forecasting models, simulation models, as well as a practical and fast code execution environment. \n\nTo create the ZephyrusBench benchmark, they had 2 approaches one that involved human annotation and another that used a semi-synthetic pipeline to generate queries. For the human generated, graduate students come up with 15 task types (5 types each in Easy, Medium, Hard category) and associated question template and solution code. They then substitute the variables in the templates with different parameters e.g. location etc. to generate a wide number (~1.8k) of queries. For the semi-synthetic pipeline they extract claims from papers to formulate queries/tasks and create 31 task types (30 medium, 1 hard) and similar to the human version, substitute variables to create ~300 questions.\n\nThey compare performance of the frontier-LLMs in 3 settings: (1) text-only: here the LLM does not get  access to any any external resources/tools, (2) Zephyrus-direct: here the LLM gets to generate the entire code for the solution in a single response turn, but is allowed to get execution feedback and correct errors upto 5 times. (3) Zephyrus-reflective: here the LLM gets to build out the code solution in chunks in a multi-turn fashion, getting the execution outputs for each code chunk which would allow the LLM to plan and break the task into components (and they allow error correction for upto 20 times).", "questions": "Please address the weaknesses.\n\n1. Since the questions are templated based on tasks, do you see similar performance on most questions of a task type?\n\n2. Can you separate evaluation for synthetic queries vs human queries.\n\n3. Having built an interesting semi-synthetic task/query generation pipeline, why not generate more synthetic tasks/questions? were there any issues? Is there value in having more queries?\n\n4. Related to weakness-3. It appears that there may not be quite as much of a difference between the Zephyrus-direct and Zephyrus-reflective agents and the small difference we see may be attributed to differences in # of LLM queries and error-corrections allowed in both settings. So, question: what if you gave Zephyrus-direct ~100 error correction attempts (to compensate both increasing LLM queries and opportunities for error feedback), would it be on-par or better than Zephyrus-reflective?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* Good weather science agentic framework contribution: The paper introduces a practical agentic framework for weather science that integrates a number of different tools for getting observation data, Geolocator, forecasting models, simulation models, as well as a fast execution environment to run model generated code to compute the solution.\n\n* Interesting and large benchmark to evaluate weather science agents. The paper has a thoughtful approach to curating a set of 46 tasks /templates and over 2100 questions. They used both human annotation to generate queries and solutions, and also an interesting semi-synthetic pipeline to generate some of the questions (see summary).\n\n* Their evaluation set up to compare frontier LLMs without access to external resources, and with access to their agentic framework under different computational resources settings seems sound and the results are meaningful clearly highlighting the value of their dataset and the agentic framework.\n\n* the paper is written well and is easy to follow.", "weaknesses": "1. The benchmark creation has humans generate templates and task types to enable semi-automated creation of questions (with suitable solutions). I wonder if this perhaps reduces the diversity that one might observe in realistic settings which may deviate substantially from the templates,\n\n2. The results could be analyzed a few additional ways e.g. based on synthetic tasks and human tasks, and also by 2-3 task-types to see if these chracterizations of the benchmark can provide more insights.\n\n3. It's not entirely clear that there is much of a difference between Zephyris-direct and Zephyrus-reflective agentic settings. Perhaps the small difference could be that Zephyrus-direct gets to make fewer model queries and lower amount of error correction. It's not exactly a major weakness but a more interesting experiment would be to see if the agentic environment would infact benefit from planning as part of the agentic framework or if current \"reasoning\"/\"thinking\" models are capable of doing the reasoning fully by themselves.  \n\n4. It would be nice to see at least one open source LLM performance in the agentic framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xDLiDSDPsf", "reviewer": "Reviewer_gtMv", "summary": "The paper introduces Zephyrus, an “agentic” framework designed to enable large language models (LLMs) to interact programmatically with meteorological data. The system includes tools to directly interact with weather and geographic data, and forecasting and simulator models.\n\nThe authors also curate a weather reasoning benchmark with 2062 question answer pairs in 46 meteorological tasks. The authors show that their system Zephyrus-Reflective which allows observations and reflections is better than Zephyrus-Direct which only allows the system to execute in one step. Both these systems are better than a text-only baseline on easy and medium tasks but do not show improvement on hard tasks.", "questions": "What is the prompt for the text-only baseline?\n\nCan you explain why text-only seems to be better in Table 16, 17, and 18? The performance trends also appear inconsistent between different models.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1. The paper clearly articulates a gap between numerical weather foundation models and language-based reasoning systems. Positioning the work at this intersection is timely and relevant, especially given the rising interest in LLM-augmented science workflows.\n\nS2. The paper presents a very reasonable set of tools and agentic systems (Zephyrus-Reflective and Zephyrus-Direct) for reasoning over meteorological questions and data and is a solid first step in this direction.\n\nS3. The dataset is well-constructed and leverages a good mix of human expertise and LLM aided construction.", "weaknesses": "W1.  The authors do not report error bars or statistical significance. This makes it hard to assess whether observed performance differences are meaningful or consistent across runs.\n\nW2. The observation that adding tools leads to better performance or allowing observations and actions in a multiple steps improves performance is not novel. From my perspective, the main contribution of the work is the incorporation of relevant and helpful tools in the agentic workflow to begin with. Thus, it would be more insightful to see how different baseline tools or the inclusion/exclusion of tools impact performance in Zephyrus-Reflective. \n\nW3. The analysis could be strengthened to understand why the agent succeeds or fails. Given that models all perform roughly equal on the harder subset, are we limited by the tool? agentic function calling? or the LLM reasoning? Such an analysis could make more clear what is needed to improve.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WfMtUuuePn", "reviewer": "Reviewer_nucp", "summary": "This is a very. nice study that creates two contributions that are novel and difficult: the first (zephyrusworld) is a system for evaluating weather focused tasks by LLM agents. This is difficult because of the data compute nature of these questions. The second is the\nbenchmark built on ERA5 combining questions from humans and synthetic creation to cover a wide range of weather probelms with varying difficulty. I'm impressed especially with the hard questions-- this goes beyond what is usually done. Graduate students wrote python code to solve the problems. The authors used these together to evaluate a set of frontier LLMs on the tasks.", "questions": "I dont understand how your choice of solvers (GCM simulator, geolocator,Forecaster) affect the scores on the tasks. Which of these were important for which tasks? \n\nThere could be more ablation studies showing why the scores are what they are. What if you delete tools? \n\nStronger baselines -- What about a LLM given access only to climatology? ie to what extent is performance due to memorized patterns versus real skill with data analysis. Are there other good baselines?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The strength is the difficulty of the tasks that are created both for the execution environment and for the benchmark itself. Both of these are impressive. These problems are not easy to set up or to create. Expertise is required for both. It is this difficulty that differentiates this paper from the large number of  other benchmarking papers.  I also like the creativity of the tasks--particularly meteorological claim verification, looking for data to support the claims from weather reports.\n\nThe evaluation metrics chosen are quantitative and rigorous (eg not employing LLM as judges etc)\n\nThe paper also contains a substantive supplementary information that gives lots of detail on tasks, performance breakdowns, etc.", "weaknesses": "the number of different tasks is pretty limited -- 46 total of which 30 are synthetic. Yet I acknowledge how hard it is to create tasks and so this is an understandable weakness. I think there is a lot of creativity in the question generation.\n\nAnother weakness is that the benchmark was built to a large extent for teh set of tools that the LLM has access to. Thus I can't tell how real the results are: what if the LLM were driving a real GCM, would the results be different? What matters for the quality of ther esults?\n\n10 or so of the tasks are yes/no questions -- evaluation is much easier here, one can just guess ;-) based on climolotagical prior\n\nBaselines aren't given other than the raw text LLM. What about a LLM given access only to climatology? ie to what extent is performance due to memorized patterns versus real skill with data analysis", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)