EarthSE: A Benchmark Evaluating Earth Scientific Exploration Capability for Large Language Models
Authors: Wanghan Xu
Xiangyu Zhao
Yuhao Zhou
Xiaoyu Yue
Ben Fei
Fenghua Ling
Wenlong Zhang
LEI BAI
Venue: ICLR 2026 Poster (2026)
Keywords: large language model
science
earth
benchmark
Primary area: datasets and benchmarks

Abstract: Advancements in Large Language Models (LLMs) drive interest in scientific applications, necessitating specialized benchmarks such as Earth science. Existing benchmarks either present a general science focus devoid of Earth science specificity or cover isolated subdomains, lacking holistic evaluation. Furthermore, current benchmarks typically neglect the assessment of LLMs' capabilities in open-ended scientific exploration. In this paper, we present a comprehensive and professional benchmark for the Earth sciences, designed to evaluate the capabilities of LLMs in scientific exploration within this domain, spanning from fundamental to advanced levels. Leveraging a corpus of 100,000 research papers, we first construct two Question Answering (QA) datasets: Earth-Iron, which offers extensive question coverage for broad assessment, and Earth-Silver, which features a higher level of difficulty to evaluate professional depth. These datasets encompass five Earth spheres, 114 disciplines, and 11 task categories, assessing foundational knowledge crucial for scientific exploration. Most notably, we introduce Earth-Gold with new metrics, a dataset comprising open-ended multi-turn dialogues specifically designed to evaluate the depth and diversity of LLMs in scientific exploration, including methodology induction, limitation analysis, and concept proposal. Extensive experiments reveal limitations in 11 leading LLMs across different domains and tasks, highlighting considerable room for improvement in their scientific exploration capabilities. The data is available on https://huggingface.co/ai-earth.

Published as a conference paper at ICLR 2026
EARTHSE: A BENCHMARK EVALUATING EARTH SCI- ENTIFIC EXPLORATION CAPABILITY FOR LARGE LAN- GUAGE MODELS
Wanghan Xu Xiangyu Zhao Yuhao Zhou', Xiaoyu Yue², Ben Feiz.4
Fenghua Ling2. Wenlong Zhang2 Lei Lei Bai²
'Shanghai Jiao Tong University SShanghai Artificial Intelligence Laboratory
'The Hong Kong Polytechnic University "The Chinese University of Hong Kong 'Corresponding author. (zhangwenlong, bailei ) @ppilab. org. cn
Hydrosptess
Knowloodge QA
Q
Cryosphere
Blosphere
Experimenial
Design
Analyada
S
Calsuition
Cloude
1.5
Grc-3
Eicendue
Tomn
Uene-11100
Cintion
Explananorr
Atsomphare
Lithoppere
Uone 11.000
Oven 23-72
Toxt Utiication
Relation Extrastin
Figure 1: Cross-domain Evaluation of Mainstream LLMs on EarthSE. (a) EarthSE evaluates the capabilities of different LLMs in Earth's 's five spheres. (b) The multi-task evaluation demonstrates pronounced capability limitations in tasks such as calculation and term explanation.
ABSTRACT
Advancements in Large Language Models (LLMs) drive interest scientific ap- plications, necessitating specialized benchmarks such Earth science. Existing benchmarks either present a general science focus devoid of Earth science speci- ficity cover isolated subdomains, lacking holistic evaluation. Furthermore, current benchmarks typically neglect the assessment of LLMs' capabilities open-ended scientific exploration. In this paper, present comprehensive and professional benchmark for the Earth sciences, designed to evaluate the capabilities of L.LMs in scientific exploration within this domain, spanning from fundamental advanced levels. Leveraging corpus of 100,000 research papers, we first construct Question Answering (QA) datasets: Earth-Iron, which offers extensive ques- tion coverage for broad assessment, and Earth-Silver, which features higher level of difficulty evaluate professional depth. These datasets encompass five Earth spheres, 14 disciplines, and 11 task categories, assessing foundational knowledge crucial for scientific exploration. Most notably, we introduce Earth-Gold with new metrics, dataset comprising open-ended multi-turn dialogues specifically designed to evaluate the depth and diversity of LLMs in scientific exploration, including methodology induction, limitation analysis, and concept proposal. Extensive exper- iments reveal limitations in 11 leading LLMS across different domains and tasks, highlighting considerable room for improvement in their scientific exploration pabilities. The data is available on httpt::://// httt:s://huggingffacc. co eeart
"This work done during his internship Shanghai Artificial Intelligence Laboratory.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
In recent years, the rapid development of large language models (LLMs) has continuously pushed the boundaries of their capabilities, demonstrating remarkable performance in general knowledge tasks eL al. 2025). Against this backdrop. researchers are actively expanding the applications of LLMs to specialized and challenging scientífic subfields (Zhang et al.) 2023 Bi al. 2023). This trend not only enhances the reasoning ability of LLMs for complex scientific problems but also holds potential for Al-assisted scientific discovery (Lu et al. 2024; Yamada et al.)
2025). To systematically evaluate the performance mainstream LLMs in scientific tasks, multiple
science-oriented benchmarks (Wang et al. 2023 Rein et al. 2024) have been proposed.
However, a comprehensive benchmark for evaluating LLMS the critical field of Earth science remains conspicuously absent. Existing benchmarks predominantly fall into two categories: a) general science benchmarks (c.g.. ScienceQA (Lu et al., 2022). SciBench (Wang et al. 2023))
which lack the necessary specificity and depth in Earth science, often featuring questions of common-sense nature. b) single-subdomain benchmarks (e.g., ClimaQA (Manivannan et al. 2024),
OceanBench (Bi et al. 2023)) that concentrate on particular areas such as climate or ocean science, thus failing to encompass the broad and interdisciplinary spectrum of Earth science. Furthermore, the prevalent questton--nnswerin
(QA) format in most benchmarks overlooks the evaluation of LLMs in open-ended scientific exploration tasks. Consequently, the construction of comprehensive and specialized Earth science benchmark that incorporates assessments of scientific exploration capabilities represents an underexplored yet vital area of research.
In this paper, we present EarthSE, comprehensive and specialized benchmark dataset for Earth science that uniquely incorporates evaluations of scientific exploration capabilities. To ensure both sufficient scale and high quality, we curate a corpus of over 100,000 Earth science academic papers as our primary data source. Through semantic analysis of titles and keywords, we categorize these papers into five major spheres (Manaham) 2006) and 114 uub-disipplins. Leveraging the publication venue and citation counts, we further stratify a subset of 10,000 papers into three distinct levels. The first level paper collection constitutes the largest portion, emphasizing comprehensiveness. The second level focuses on papers from high-impact journals, prioritizing specialized knowledge. The third level comprises highly cited papers, specifically designed for evaluating scientific exploration.
Scientific exploration demands multi-level competencies, spanning fundamental domain knowledge mastery to advanced critical reflection and innovative improvements. The former, amenable to explicit evaluation criteria, suits question-answerine assessed through open-ended multi-turn dialogues. Correspondingly, we develop two pipelines: one for QA data using predefined task formulations to directly generate high-quality pairs from papers, and another for dialogue data structuring papers to extract scientific inquiry workflows for multi-turn dialogues. Both pipelines include automated cleaning and human verification for quality.
formats. The latter, lacking unified standards, is better
Ecology and Biosciences Restoration Ecology
Hydrology
Polar Science Glaciology
Atmospheric Sciences Metcorology
Ecohydrology
Biogeochemistry
Groundwater Hydrology
Polar Occanography
Solar Energy
Landscape Ecology
Hydrological Measurement
Polar Climate Science
Atmospheric Remote Atmospheric Physics Atmospheric Chemistry
Population Ecology
Regional Hydrology
Permafrost Science Science
Ecosystem Ecology
Hydrological Meteorology
Polar Ecology
Biogcography
Hydrological Physies
Hydrometcorology
Regional Ecology
Hydrogeology
Oceanography
Air Quality and Pollution
Soil Ecology
Ocean Physics
Weather Forecast
Environmental
Solid Earth Geophysics
Ocean Chemistry
Urban Ecology
Gcomagnetism
Occan Gcology
Geology
Seismology
Ocean Biology
Sedimentology
Climatology
Volcanology
Ocean Remote
Quatemary Geology
Palcoclimatology
Remote Sensing Geology
Ocean Environment
Mineralogy and Petrology
Figure 2: EarthSE Covers 114 Disciplines in Earth Sciences.
2

---

Published as a conference paper at ICLR 2026
Consequently, release three distinct datasets, each with unique characteristics. Earth-Iron (QA) encompasses 4133 questions across 11 tasks and 1 114 sub-disciplines (Figure emphasizing foun- dational and broad scientific exploration capabilities. Earth-Silver (QA) prioriizzes high-difficulty, specialized knowledge. Earth-Gold (dialogue) focuses on evaluating advanced scientific exploration capabilities such as summarization, reflection, and innovation within open-ended dialogues.
We summarize the contributions of this paper as follows:
We built two QA datasets to evaluate the fundamental capabilities of scientific exploration: Earth-Iron, containing 4133 questions that span 114 subfields for broad assessment, and Earth-Silver, presenting more challenging and professional inquiries for deeper evaluation.
We innovatively propose Earth-Gold and new metric (i.e., SES) assess advanced capabilities of scientific exploration (e.g.. methodology induction, limitation analysis, and concept proposal) through open-ended multi-turn dialogue.
Our systematic evaluation of 11 leading LLMs across these datasets reveals significant deficiencies in complex Earth science reasoning and open-ended scientific exploration.
RELATED WORK
General Science Benchmark. In recent years, LLMs have increasingly focused on scientific reasoning (Li et al. 2025). leading to the development of range of benchmarks designed to assess model performance across diverse scientific domains (Thirunavukarasu
al.) 2023] Zhang al..
purpose science benchmarks such as ScienceQA (Lu et al. (022), SeiBench (Wang et 2023) and MMLU-Pro (Wang et al. (024bb) cover disciplines from elementary to graduate levels. For instance, ScienceQA includes over 21,000 multi-modal questions spanning natural, social, and language sciences. MMLU-Pro extends MMLU (Hendrycks et al. 2020) with broader college-level subjects to evaluate deeper reasoning. SciBench emphasize college-level problem solving, drawing from canonical textbooks. Expert-curated datasets like GPQA (Rein et al. 2024). SuperGPQA (Du et al., 2025). and HLE (Phan et al.) 2025) offer high-quality, tine-graind domain expertise. However, these benchmarks primarily prioritize breadth over depth, often underrepresenting Earth sciences. As unique and complex discipline covering five spheres, Earth science requires more targeted evaluation.
2024).
carduate-levee questions to assess
Specialized Subject Benchmark. In addition general benchmarks, several efforts have focused on subdomains within Earth sciences. For example, OceanGPT (Bi et al., [2023) introduces Ocean- Bench, benchmark for oceanographic tasks such as QA, fact verification, and applied writing. ClimaQA (Manivannan et al., 2024) and Climate Watch (Kraus et al., 2023) assess reasoning over climate data and models. ClimateBERT (Wehersinke al. 2021) and Clima'Text (Varini al. 2020) support stance detection and document classification in climate discourse. Geological (Deng et al.) 2024) tasks are addressed by GeoBench, which supports entity recognition and relation extraction, and SeafloorAI (Nguyen et al. 2024), which evaluates multimodal reasoning over sedimentary and geomorphological features. While these benchmarks are domain-relevan each focuses on a narrow component of the Earth, lacking unified evaluation across the full Earth science spectrum.
Moreover, the majority of current benchmarks emphasize question answering, focusing on factual recall and reasoning over established knowledge. Yet, critical frontier lies in enabling LLMs to support scientific exploration and discovery. a more open-ended capability involving summary and limitation analysis of existing methods, and hypothesis generation.
EARTHSE: EARTH SCIENCE EXPLORATION BENCHMARK
Overview. Scientific exploration requires capabilities from basic Earth science knowledge to advanced reflection and proposing new methods. Starting from 100,000 papers in the Earth sciences, we constructed three datasets. Earth-Iron (QA) contains 4133 questions covering 114 sub-disciplines, focusing on a comprehensive evaluation fundamental scientific exploration abilities. Earth-Silver (QA) features more difficult and challenging questions, focusing more specialized Earth science knowledge. Earth-Gold (dialogue) evaluates advanced scientific exploration abilities through open-ended dialogue, focusing reflecting existing problems and proposing approaches.
3

---

Published as a conference paper at ICLR 2026
Table 1: Comparison between Existing Benchmarks and EarthSE. Comparison across (1) question volume, (2) data sources, (3) Earth sphere coverage, (4) graduate-level difficulty, (5) multiple subsets and (6) scientific exploration assessment. EarthSE uniquely offers comprehensive domain coverage, professional difficulty, and evaluation of scientific exploration in Earth sciences.
Benchmark
#Ques.
Earth Cover
Grud-Diff
Multi-ubb
Scientific Exploration
ScienceOA
2022
21,208
High School Courses
MMLU-Paa Wang st
2024bb
12,032
Exam Questions
SeiRench
Wang
X123
869
10 Texthooks
GPOA RRiin al. 2004
448
Expert Curuted
(8)
2023
12,426
Domain Texes
ClimaQA
Manivannan
2024
3502
18 Textbooks
GeoBench
Dong
2439
Exam Questions
EarthSE
4133
100,000 Earth Science Papers
selection
QA Construction
QA Cleaning
Passa 100,000 Pepers
Text Extraction
Knowlodge QA Calculaiin
toak proimpt
Rule based gerbled characters, extra references...
67
Toal isage Experiment
prompe
Jourzal solection
Code penension
poper Concens
Semoaticn based tricievani answers, intornect solutions...
Earth-Iroa Comprehansive
Category Clavification
Please selece task construct Question and Andwer based on the followeing peper.
Solving Steps: [1) Second.. Ans: Werr Ocean Water...
Ques: Lyphoon formattoo
Difficulty Screaning
paper keywords
S
Espert chuchhngg
hydrosphere
10,000 Papers
strooshhre
Paper Structure S
Dialogue Constraction
Expert Review
Barth-Sllver Challenging
Citation filler
cryosphere
M, methors
Map kig M,. In
& & & & Specific Specific methods Complese veferences Specific new mechad Spesific limitatiioss
blosphere
Lg Limitationa of Mn
Can you andlyze...
Box(2018) proposed
lithosphere
M,
Design new mechod?
Earth
Sclentifie Exploration
somantic similerit
bg Limitations of Mj
possibbl way...
1,000 Papers
Earth-Iron consinction procoss
Earli-Siverr process
Earth .iol process
Figure 3: Construction Process of EarthSE. Automated construction of three-tiered benchmarks Irant Earth-Silver, Earth-Gold) Gold) from 100K papers, enabling comprehensive evaluation of knowledge coverage, professional proficiency, and scientific exploration capabilities.
3.1 TASK DEFINITION FROM FOUNDATIONAL TO ADVANCED
Foundational Science Task Definition. To comprehensively evaluate the capabilities of LLMs in Earth exploration tasks, spanning from foundational advanced levels, defined 11 fundamental research tasks, as detailed in Table 2 These tasks cover broad spectrum, ranging from basic understanding and reasoning advanced research skills. These capabilities form the foundation for LLMs to conduct scientific exploration. For example, the "Dataset" task specifically focuses on datasets within the Earth sciences. Given the significant diversity of data (such seismic wave data and two-dimensional remote sensing images) across the numerous sub-disciplines of Earth science, enhancing L.LMs' understanding of these varied datasets crucial.
-immennsiia
Scientific Exploration Task Definition. Beyond these 11 fundamental scientific exploration tasks. we have also defined scientific discovery task formulated as open- open-ended dialogue. Through sys- tematic analysis of these papers, we identify a recurrent research pattern: "analyzing limitations of existing work -> proposing novel methods" (Yang et al.) 2024b). This inspires our formaliza- tion of scientific exploration as an iterative self-negation process, mathematically expressed as: LLM(M', L'), where M denotes methodology. L represents limitation analysis of M. and indicates dialogue turns. This recursive framework simulates the human scientific process of critically examining prior work's constraints and progressively improving upon them. Conse- quently, models with genuine scientific discovery potential must demonstrate robust self self-critique and self-improvement capabilities (Li 2024
4

---

Published as a conference paper at ICLR 2026
Table 2: Foundational Task Categories and Descriptions.
Task
Description
Term Explanation Heiman 2025
Requires delining technical concepts
Unnerrstandngg
(e.g.. "Define medium--angge weather forocast' explain significance")
Knowledge
Xu ct 2024
Requires factual explarations detailed descriptions distributions charactoristics
(eg.. "Describe the distribution fishery resources the East China. Sea").
Fact Verification Zhang & Gao.
2025
Involves validating clainss' accuracy
(e.g. "Verity whether the reported sca level rise data is conrect").
Analysis (Cai al.
2024
Demands logical reasoning to draw conclusions from data (e-g. 'Determine which evidence supports given conclusion" ").
Rezsoning
Relation
analysis interconnections hetween
(e.g. 'Summariiee ecological relationships
plants")
Calculation Stephun al.
2024
Involves multi step mathematical operations with numerical answers
(e.g. "Calculate annual precipitaiion from the dataset").
Tool Utilization Yuan et al.
2014
Involves eocommending docnan-spocifie methodologics (e.g. "Suggest atmospheric models for weather prediction") )
Literature Citation Ryun al.
2024
Demands proper academic references
Research
(e.g. "List key publications on occan circulation including citations (Ravari et al., 2021)").
Taylor
2022
Roquires introducing recommending
(egg. 'Recemmend ERAS (Hersaach 2020 mediumm-rnp mannnn--eee--ee weather forscasting")
Experimental Design
Chen al.
2024
Needs detailed methodological planting
(eg.. "Design un experiment to investigate soil moisture's 's impuct photocynthesis")
Code Generation Ga
2023
Invalves writing functional code
(egg. "Visualize rainfall data using Python's Matpletlib library (Bisong 2019 ))
3.2
PAPER CORPUS COLLECTION
The study uses Earth science academic papers as data because: (a) their dense, high professional knowledge facilitates quality data creation; (b) their structured format aligns with general scientific discovery processes. During collection, we obtain and convert 100,000 PDFs to structured JSON using MinerU (Wang al. 2024a). Semantic similarity (Devika et al. 2021) abstracts and Earth sphere keywords (see Figure classifies papers into five Earth spheres, as detailed in Table
We define the initial collection of 100,000 papers the base dataset Phese. From this collection, first select a subset Pmj comprising 10,000 papers published in high-impact Earth science journals (see Table 7 for details). We then extract the top 10% most cited papers from Pij form the high-citation core dataset Phe with 1.000 papers. The entire selection process maintains strict balance across all five spheres, with detailed distributions provided Table 3 Figure 4 further illustrates the citation distribution patterns across different spheres in the high-citation dataset Phe- Puuse Phj are used to construct Earth-Iron, Earth-Silver, Earth-Gold respectively, as depicted in Figure
500
Table Number of Papers on the Five Spheres of Earth.
400
Citatoos
300
Earth Sub-domain
Pbase
Pij
Pho
Biosphere 21,248
1,554
201
Lithosphere
22,820
2,357
236
100
Atmosphere
24,213
2,401
240
Hydrosphere
23,425
2,254
226
Cryosphere
11,402
1,969
217
Biosphere
Lithosphere Atmosphere
Hydrosphere
Cryosphere
Total
103,108
10,535
1,120
Figure 4: Citations of Papers in Phe.
3.3 FARTH-IRON/SILVER QA BENCHMARK FOR FOUNDATIONAL TASKS
Earth-Irow Iron and Earth-Silver constructed from Prese and Pny respectively, using the same pipeline as shown in the first row of Figure Characteristically, Earth-Iron offers a large and comprehensive set of questions, while Sarth-Silver features more difficult and specialized questions.
5

---

Published as a conference paper at ICLR 2026
Task Selection, Before QA construction, small LLM analyzes each paper's abstract to determine the most suitable task type from foundational tasks shown in Table for QA generation. This content-awary assignment enhances question relevance. For instance, papers containing substantial numerical results preferentially generate computation questions.
QA Construction. In the question generation phase, paper content and task prompts are fed into GPT-4 (Achiam et al. (2023) to produce QA pairs. To ensure answer accuracy, the model is required to provide complete Shain-of-Thougee
(CoT) (Wei et al., (022) reasoning that includes both the final answer and its derivation process. These CoT' not only form integral parts of the questions but also serve as critical references for subsequent data cleaning, effectively mitigating errors caused by LLM hallucinations (Wei al. 2024).
Data Cleaning. The data cleaning process employs dual-phase strategy: (a) rule-based pri- mary cleaning for formatting issues (missing options, irregular answers, improper references); (b) semantics-base based advanced cleaning using LLMs to detect deeper problems (multiple correct options, irrelevan/incooreet answers). Throughout this process, CoT reasoning plays a pivotal role, as its explicit step-by-step derivation provides reliable evidence for error detection.
After data cleaning, we performed difficulty screening using mainstream LLMs. Based on testing results from both proprictary and open-source models, we removed questions with accuracy rates exceeding 80% to ensure sufficient overall challenge. For questions with accuracy rates between 60% and 80%, human experts determined their retention based on the question's 's value.
3.4 EARTH-GOLD: OPEN-END DIALOGUE BENCHMARK FOR EXPLORATION TASK
Most existing benchmarks predominantly employ auestion-answeele formats, which, while effective for knowledge assessment, fail to capture the open-ended exploration paradigm characteristic of genuine scientific research. To evaluate LLMs' capabilities in open scientific exploration, we construct the Earth-Gold dataset derived from Phc. collection of 1,000 highly-cited Earth science papers, as shown in the second row Figure 3 These papers exemplify superior scientific reasoning patterns, making them ideal prototypes for studying scientific exploration behaviors.
Paper Structurization Following the scientific exploration task definition in Section 3.1 decompose each paper Phe into four structured components:
Existing Method Summary (M*): Extracted from related work sections, providing com- prehensive synthesis of current methodologies.
Limitation Analysis (L"): Derived from motivation sections, identifying precise shortcom- ings that constitute the starting point for new research.
Novel Method Proposal (M*): Abstracted from methods sections, capturing the core innovations.
New Method Limitations (L²): Distilled from discussion sections, anticipating potential constraints of proposed solutions.
Dialogue Construction With M T.O. M and I extracted, we employ GPT-4 to generate two-turn dialogues simulating human-AT collaborative scientific exploration. The first turn requires the AI assistant to summarize existing methods and critically analyze their limitations given research direction; the second turn directs the assistant to propose improved methods addressing these limitations while objectively assessing the new methods' potential constraints. This dialogue structure authentically replicates human ideation processes assisted by AI, demanding both extensive scientific knowledge and critical thinking abilities from LLMs.
Human Expert Validation Following dialogue generation, domain experts conduct rigorous quality control using multi-criteria evaluation: a) Information Density Scoring: Higher scores are assigned to dialogues containing specific analytical methods, datasets, or well-defined events (e.g., earthquakes, typhoons). while vague discussions receive lower scores. b) Methodological Quality Assessment: Proposed methods receive higher scores if they systematically enumerated and

---

Published as a conference paper at ICLR 2026
concretely implementable. Non-specific proposals are filtered out. Through this stringent validation process, we curate the final Earth-Gold dataset, which exclusively comprises high-quality scientific exploration dialogues, thus establishing a new benchmark for evaluating LLMs' research capabilities.
New Metric: Scientific Exploration Score (SES). Earth-Gold evaluates open-ended scientific dialogue, where standard correctness metrics inadequate due to the inherent diversity of valid sci- entific exploration beyond definitive answers. To address this, we innovatively measure performance using retention rate and diversity. For each user question the dialogues, the LLM generates M diverse responses (lemperature=0.6) (Peeperkorn et al. 2024). and then we compute:
Retention Rate (r): GPT-4 ranks M generated answers and the reference answer by reflective depth and innovation. Retention rate (i 1)/M (where i is the reference answer rank) quantifies the model's preservation of high-quality scientific reasoning. 0, it means that all LLM answers are worse than the reference answers.
Diversity (d): We compute embeddings v; for each response using sentence- traence-tranfformrr. then derive the mean vector v. The average cosine similarity between and v; is calculated as Equation I Since lower similarity indicates higher diversity, define d = 1/3.
M
1
Σ 8 M
cos (v, v), where v M
Σ 1=1
The Scientific Exploration Score (SES) combines retention (r) and diversity (d) multiplicatively:
SES d. Since is relatively close to normalize & to [0.9, 1] in actual calculations for better comparison, that is, SES = 10x Higher SES values indicate superior open- open-ended scientific thinking, reflecting both answer quality (retention) and conceptual breadth (diversity).
4 EXPERIMENT
4.1 EXPERIMENTAL SETUP
This paper introduces three evaluation datasets: Earth-Iron and Earth-Silver as question answering (QA) datasets, and Earth-Gold as open-ended dialogue dataset. The QA datasets incorporate question formats: multiple-choice (MC), fill-inthhe-llnk (FIB), true/false (TF), and free-response (FR) questions. For MC, FIB, and TF questions, questions, we employ the win rate (WR) against the reference answer evaluated by GPT-4, and semantic similarity (SS) metrics.
accuracy (ACC) the metric: for FR
Accuracy (Acc.) For closed-form questions (multiple-choice, fill-in-the-blank, and true/false) with deterministic correct answers, calculate accuracy as the exact match rate between model outputs and reference solutions.
Win Rate (WR) Bi al. 2023) For free-response questions, we employ GPT-4 an impartial judge to compare model answers against reference answers along three dimensions: relevance (degree of topical aligmment, penalizing tangential responses), scientific rigor (logical consistency and factual correctness), and specificity (depth of targeted analysis, penalizing generic statements). The evaluation protocol ensures fairness by blinding GPT-4 answer origins. Win rate quantifies the percentage of cases where the evaluated model's response surpasses the reference.
Semantic Similarity (SS) DDevika et al. 2021) For free-response questions, we additionally measure the conceptual alignment between model outputs and reference answers using sentence beddings. Specifically, we utilize sentence- sentence-transfone transformers to project responses into 384-dimelli semantic space, then compute their cosine similarity as the semantie similarity metric.
For Earth-Gold, we use the SES defined Section 3.4 to evaluate the performance of scientific exploration dialogues.
7

---

Published as a conference paper at ICLR 2026
Table 4: Evaluation Metries for Leading Models on the Earth-Iron and Earth-Silver Bench- marks. For MC (Multiple Choice), TF (True/False), and FIB (Fill-in-the-llane questions, Accuracy (ACC) used as the evaluation metric, reported percentage (%). For FR (Free Response) questions, Win Rate (WR), Semantic Similarity (SS) are employed as evaluation metrics.
Earth Iron (More Comprehensive)
Earth Silver (More Challenging)
Model
MC²
TF
FIB:
FR-WR1
FR-SS
MC
TF
FIB1
FR-WR]
FR-SS1
Llama-3 1omaa-311-80
Grattafiori et al.
2024
59.41
74.36
2.52
13.70
0.76
36,00
$4.00
2.02
4.40
0.72
1lama-3.1-700
Gratiation et 2027
91.56
87.91
6.63
61.85
0.80
56.00
63.60
4,00
18.40
0.80
Qwen-2.5-72B
Yang ctal
2024a
92.42
86.26
11.96
92.05
0.79
53.60
64.40
9.20
44.40
0.78
V3
Li al 2024
93:40
81.14
18.99
97.60
0.81
58,00
56.40
12.80
75.20
0.81
GPT-46 OpenAT
2024
93.28
88.28 28
19.12
82.00
0.81
55,60
69.60
18.40
23.00
0.80
1.3-aal
Team ct
2024
90.83
75.82
13.65
95.60
0.79
54,40
44.80
8.00
62.80
0.78
G.miai-2.0-Flas
Team
2023
92.67
87.55
14.69
77.10
0.77
54,40
72.40
11.60
34,80
0.75
Gemini-2.5-Flasb
G.Comanici
2025
93.15
77.84
17.02
95.81
0.75
58,00
55,60
13:65
74.30
0.75
Claude-3-S-Haal
Kurokawa
2024
91.08
83.52
12.48
12.05
0.79
56.80
60.80
9.60
4.40
0.77
Clunde-3-7SSomet
(Lim
94.01
20.68
0.79
Grok-3 de Carvalho
Welgang.
21.85
70:40
Category: Atmosphere
Format: Free-response
Task: Culculation
the density fly ash particles 1800 kg/m², m², the surface area of the particle is 1.6 x 10 m² incident velucity 1.5 m/s, and the viscosity liquid 0.001 Pa s under dry conditions, calculate the Stokes number using the formula St prin where is the particle density, is the particle radius, vi is the incident velocity, and is the viscosity of the liquid.
Step 1:
Calculate partiele radius
4
10
Step 2: Calculate Stokes number
St
ArM
1800x2x18
5400
Answer: 5400
0,001
DeekSeck-V V3:
Llama-3, 1-405B:
1. Find particle radius (r):
Surface Area 4.y where is the radius of the sphere.
Surface area of sphere: 4my?
We rearrange the formula to solve for sqri(Surface Area 4x)
2. 1.6 10 2
Plugging in the value for the surface arca, get:
3.
16x10-1
V4 x 10 2 1
r= 2 x 10"
Now that we have the particle radius, we can calculate the
2. Calculate Stokes number (St):
Stokes number using the formula:
1. Formula: St
St = prv,
Plugging in the values, we get:
1800x2<10 XX15
2. St
5400
St 1800 kg/m² x 2 10 m 1.5 m/s
Figure 5: Case Study from Earth-Silver. Some models make mistakes when applying formulas in multiple steps, resulting in low accuracy in calculation questions.
4.2 EARTH-IRON/SILVER: ASSESSING BROAD FOUNDATIONAL CAPABILITIES
Earth-Iron comprehensive QA benchmark consisting of 4133 questions spanning the Earth sciences domain. Table comparative performance metrics across these question formats for various L.LMs. Most models perform well on multiple-choice questions but struggle with fill- in-the-blank tasks. In free-response questions, performance varies significantly; some models (e.g., Claude-3.5) provide overly general answers, resulting in poor Win Rate against the reference answers.
We compute each model's overall competency across Earth's 's five spheres, as detailed in Figure [ This figure visually illustrates the capability distribution of different models across the five spheres. Notably, Grok-3 achieves oftt-ooftthear (SOTA) performance across all spheres. While most models exhibit a relatively balanced capability distribution, such as DeepSeek- V3, Gemini-2.5. some show specific weaknesses, for example, GPT-40 in the cryosphere.
8

---

Published as a conference paper at ICLR 2026
The right panel in Figure Tiauureeilllsttttet Iillustaate the capabilities of different models across the 11 fundamental scientific tasks. Most models exhibit a similar capability distribution, particularly the SOTA LLMs, likely due to similar training paradigms. Notably, most models perform relatively poorly the calculation task. Additionally, the term explanation metric is generally low, indicating a potential weakness in LLMs' understanding of specialized Earth science terminology.
Compared to Lronh-Iron, Carth-Silved is constructed from a higher-quality corpus of original papers, resulting in a significantly more challenging dataset. The experimental results on the right side of T Sable|ssho that most LLMs achieve only 54% accuracy on multiple-choiee questions (compared to 25% random chance rate). This not only reflects the high difficulty of Earth-Silvee but also indicates considerable room for improvement in the performance of current LLMs within the Earth sciences. tasks, the average performance of L.LMs notably low, at just 11%.
Figure 5 presents representative examples from these challenging tasks alongside model responses. The case study analysis reveals frequent formula misuse and computational errors Earth science- specific calculations, indicating critical knowledge gaps in domain-specifie reasoning.
4.3 EARTII-GOLD: EVALUATING OPEN-END SCIENCE EXPLORATION CAPABILITY
Earth-Gold, a core innovation, evaluates LLMs in open scientific exploration dialogues using our novel Scientific Exploration Score (SES),
Table 5: Scientific Exploration Capabilities of Main- stream Models on Earth-Gold. Earth-Gold assesses LLMs' research potential through open-ended scientific dialogues, employing three metries: Retention (propor- tion of responses outperforming references), Diversity (measure of divergent thinking), and their composite Sci- entific Exploration Score (SES).
which assesses divergent thinking (diversity d) and answer quality (retention rate r). Table shows performance with each model generat- ing M = responses for r and calculation. The results indicate that most models achieve a retention rate of less than 50%, suggesting that over half of the generated responses un- derperform the reference answers. Regarding the diversity metric, most models exhibit low diversity scores. This indicates that when re- sponding to open-ended questions, LLMs tend to generate similar answers across multiple at- tempts, which significantly contrasts with the divergent thinking characteristic of human sci- entists during scientific exploration.
Model
Retention (%)
Diversity
Llama-3.1
8.00
3.9813
0.3301
Llama-3.1-708
11.78
1.4891
0,2453
Qwen-2.5-72B
7.11
1.7158
0.1375
DeepSeck-V3 V3
38.00
1.6942
0.6599
GPT-46
9..4
".0347
0.0981
Gemini-1.
19.67
1.4437
0,1989
Gemini-2.0
18.22
2.6290
0.6505
Gemini-2.5
50.56
2.7016
1.3710
Claude-3.5
14.67
1.5517
0.2396
Claude-3.7
31.89
1.7130
0,5465
Grok-3
17.22
1.5284
0.2727
Our analysis identifies three primary issues in these subpar responses: (1) overly generic content lacking specific details, (2) non-specific analyses of limitations, and (3) excessively broad proposed solutions. Figure 6 provides a visual comparison between high-quality and low-quality responses.
4.4 COT GUIDANCE ENHANCES PERFORMANCE ON CHALLENGING QUESTIONS
FIB ACC in Table 4 show lower accuracy, likely due to their inher- difficulty compared to MC/TF, Since our question construction generates step-by-step explanations resembling CoT reasoning, we investigate if providing LLMs with initial CoT steps during inference improves performance these challenging questions.
Table 6: The Impact of Using CoT Guidance on FIB ACC.
Inspired CoT Steps
FIB (56)
DeekSeek V3
"O (Baselinee)
12,60
Table O illustrates the model's accuracy with varying numbers of provided CoTT steps. As is evident, increasing the number of CoT steps offered as hints significantly improves the model's accuracy. This suggests that for more challenging questions, even without modifying the model's 's fundamental capabilities, leveraging CoT- augmented reasoning performance. This finding provides a promising foundation for inference-time scaling strategies.
21.60
2
29.60
45.60
GPT-40
(Baseline)
18.40
inference time substantially boost
2
38. 80
3
55.60

---

Published as a conference paper at ICLR 2026
I'm interested in exploring changes and variability Arctic sea ice thickness, volume, and multiyear coverage from 1958 to 2018. Could you summarize key prior research th(s area highlight their limitations?
Reference answer:
answer:
Certainly. Research in this field has relied submarine missions (since 1958) for early ice thickness estimates, later supplemented by satellite observations like ICESat and CryoSat- (CS-2). Multlyear (MY!) coverage been mapped using scatterometer data from QuikSCAT and ASCAT...
Research Arctic sea ice from 1958 to 2018 indicates significant declines in thickness, volume, and multiyear ice coverage, driven climate change. Key studies utillzed satellite models, revealing
like reduced multiyear ice and thinner overall ice...
Given these limitations, what alternative methodology would you recommend, and what challenges might introduce?
Reference answer:
Bad answer:
Monthly gridded
fields could then
recommend using a combination remote sensing, in situ measurements, and advanced modeling techniques like machine learning. This approach could enhance spatial and temporal resolution but may introduce challenges such integrating diverse data sources, ensuring consistency.
paired with MY1 coverage maps to analyze However, this method has drawbackss, the retiance two temporally overlapping but independently derived datasets (MY! and volume) may introduce artifacts trend attribution, particularly for pre-satellite eras.
Figure 6: Case Study from Earth-Gold. Earth-Gold used evaluate the performance of LLMs in open scientific exploration. For the same user questions, the reference answers contain more details, such as the yellow text, while the bad answers are very general.
5 CONCLUSION
The increasing interest in leveraging Large Language Models (LLMs) for scientific applications un- derscores the need for specialized benchmarks, particularly in fundamental yet multifaceted domains like Earth science, which current general or fragmented benchmarks inadequately address, especially regarding open-ended scientific exploration. To this end, this paper introduces comprehensive and professional Earth science benchmark comprising two QA datasets, Earth-Iron and Earth-Silver, and novel open-ended dialogue dataset, Earth-Gold, built upon a 100,000-paper corpus. Errth-Iron provides broad foundational assessment across five Earth spheres, 114 disciplines, and 11 tasks, while Earth-Silver offers challenging, professional-leve questions. Notably, Earth-Goll Gold evaluates advanced scientific exploration abilities through multi-turn dialogues. Experiments show LLMs perform reasonably on basic Earth-Iron QA but significantly worse on challenging Varth-Silver. Farth-Gold reveals below low retention and diversity in open-ended dialogues. These quantitative results highlight current LLMs' limitations in Earth science knowledge depth and genuine scientific exploration, indicating significant room for improvement.
The limitation this work lies in the fact that it does not integrate the tasks to create a more complex task chain, highlighting gap in achieving automated scientific discovery.
ETHICS AND REPRODUCIBILITY STATEMENT
The data used in this study soureed from OpendataLab (He et al. 2024), and were collected in strict accordance with relevant guidelines, avoiding any sources that restrict copying or redistribution. This work aims establish benchmark for evaluating the scientific discovery capabilities of large language models in Earth sciences. We believe that this research not only promotes the responsible of artificial intelligence in scientific investigations but also enhances public understanding of Earth sciences. All data are released anonymously, with the full dataset publicly available on Anonymous HuggingFace ftttps://hegiingface. co/ earth). The details of data processing are described in detail in the Appendix.
ACKNOWLEDGMENTS
This work is supported by Shanghai Artificial Intelligence Laboratory.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXiv:2303.087771 2023.
Zhen Bi, Ningyu Zhang, Yida Xue, Yixin Ou, Daxiong Ji, Guozhou Zheng, and Huajun Chen. Oceangpt: A large language model for ocean science tasks. arXiv preprint arXiv:2310.02031,
2023.
Ekaba Bisong. Matplotlib and seaborn. In Building machine learning and deep learning models google cloud platform: A comprehensive guide for beginners, pp. 151-165. Springer, 2019.
Courtni Byun, Piper Vasicek, and Kevin Seppi. This reference does not exist: an exploration of Ilm citation accuracy and relevance. In Proceedings of the Third Workshop on Bridging Human- Computer Interaction and Natural Language Processing, pp. 28 39, 2024.
Hengxing Cai, Xiaochen Cai, Junhan Chang, Sihang Li, Lin Yao, Changxin Wang. Zhifeng Gao, Hongshuai Wang, Yongge Li, Mujie Lin, al. Sciassess: Benchmarking Ilm proficiency in scientific literature analysis. arXiv preprint arXiv:2403.01976, 2024.
Yanxi Chen, Yaliang Li, Bolin Ding, and Jingren Zhou. On the design and analysis of Ilm-based algorithms. arXiv preprint MXXY::2477.14788 2024.
Murillo Edson de Carvalho Souza and Li Weigang. Grok, gemini, chatgpt and deepseck: Comparison and applications in conversational artificial intelligence. INTELIGENCIA ARTIFICIAL, 2(1), 2025.
Cheng Deng, Tianhang Zhang, Zhongmou He, Qiyuan Chen, Yuanyuan Shi, Yi Xu, Luoyi Fu, Weinan Zhang, Xinbing Wang, Chenghu Zhou, et al. K2: foundation language model for geoscience knowledge understanding and utilization. Proceedings the 17th ACM International Conference on Web Search and Data Mining, pp. 161-170, 2024.
R Devika, Subramaniyaswamy Vairavasundaram, C Sakthi Jay Mahenthar, Vijayakumar Varadarajan, and Ketan Kotecha. deep learning model based bert and sentence transformer for semantic keyphrase extraction on big social data. IEEE Access, 2-165222-16520 2021.
Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, et al. Supergpqa: Scaling 11m evaluation across 285 graduate disciplines. arXiv preprint arXiv:2502. 14739, 2025.
Eugene Garfield et al. The impact factor. Current contents, 25(20)33-7, 1994.
G.Comanici et al. Gemini 2.5: Pushing the frontier with advanced reasoning. arXiv preprint arXiv:2507.06261. 2025. URL.ttts://axxxvv.oo wttp://w.wrgab///5....o.oo
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama herd of models. arXiv preprint arXiv:2407.2178. 2024.
Qiuhan Gu. Llm-based code generation method for golang compiler testing. In Proceedings of the 31 ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. pp. 2201-2203, 2023.
Conghui He, Wei Li, Zhenjiang Jin, Chao Xu, Bin Wang, and Dahua Lin. Opendatalab: Empowering general artificial intelligence with open datasets. arXiv preprint arXiv:2407.13773, 2024.
Alice Heiman. The accuracy, robustness, and readability of llm-generated sustainability-relate word definitions. arXiv preprint arXiv:2502.00916, 2025.
Dan Hendrycks, Collin Burss, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.0338 03300, 2020.
11

---

Published as a conference paper at ICLR 2026
Hans Hersbach, Bill Bell, Paul Berrisford, Shoji Hirahara, András Horányi, Joaquín Muñoz-Sabater Julien Nicolas, Carole Peubey, Raluca Radu, Dinand Schepers, et al. The era5 global reanalysis. Quarterly journal of the royal meteorological society, 146(730):1999-2004 2020.
Mathias Kraus, Julia Anna Bingler, Markus Leippold, Tobias Schimanski, Chiara Colesanti Senni, Dominik Stammbach, Saeid Ashraf Vaghefi, and Nicolas Webersinke. Enhancing large language models with elimate resources. arXiv preprint arXiv:2304.00/ 16, 2023.
Ryo Kurokawa, Yuji Ohizumi, Jun Kanzawa, Mariko Kurokawa, Yuki Sonoda, Yuta Nakamura, Takao Kiguchi, Wataru Gonoi, and Osamu Abe. Diagnostic performances of claude 3 opus and claude 3.5 sonnet from patient history and key images in radiologyåAZs åAlJdiagnosis pleaseaAí cases. Japanese Journal of Radiology, pp. 1-4, 2024.
Qianxi Li. Iterative large language models evolution through self-critique. 2024.
Xingzuo Li, Kehai Chen, Yunfei Long, and Min Zhang. Llm with relation classifier for document-level relation extraction. arXiv preprint 238.::2880..8888 2024.
Yunxin Li, Zhenyu I. Liu, Zitao Li, Xuanyu Zhang. Zhenran Xu, Xinyu Chen, Haoyuan Shi, Shenyuan Jiang. Xintong Wang, Jifang Wang, et al. Perception, reason, think, and plan: survey on large multimodal reasoning models. arXiv preprint arXiv:2505.0492 2025.
Bryan Lim, Ishith Seth, Molly Maxwell, Roberto Cuomo, Richard Ross, and Warren M Rozen. Eval- uating the efficacy of large language models in generating medical documentation: A comparative study of chatgpt-4, chatgpt-4o, and claude. Aesthetic Plastic Surgery, pp. 1-12, 2025.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint 19439 2024.
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408. 06292, 2024.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2522 2022.
Stanley E Manahan. The five environmental spheres. In Environmental Science and Technology, pp. 59 84. CRC Press, 2006.
Veeramakali Vignesh Manivannan, Yasaman Jafari, Srikar Eranky, Spencer Ho, Rose Yu, Duncan Watson-Parris, Yian Ma, Leon Bergen, and Taylor Birg-Kirkpatrick. Climaqa: An automated evaluation framework for climate foundation models. arXiv preprint arXiv:2410.1670. 2024.
Kien Nguyen, Fengchun Qiao, Arthur Trembanis, and Xi Peng. Seafloorai: A large-scale vision- language dataset for seafloor geological survey. Advances Neural Information Processing Systems, 37:22107-22123, 2024.
OpenAl. Gpt-4o system card. http::///eaaaa.ccce/////////rtts 2024.
Max Peeperkomn, Tom Kouwenhoven, Dan Brown, and Anna Jordanous. Is temperature the creativity parameter of large language models? arXiv preprint 2.KV::2405500499 2024.
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity's last exam. arXiv preprint arXiv:2501. 14249, 2025.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: graduate-level google-pree proof q&a benchmark. In First Conference Language Modeling, 2024.
12

---

Published as a conference paper at ICLR 2026
Michael D Skarlinski, Sam Cox, Jon M Laurent, James D Braza, Michaela Hinks, Michael J Hammerling, Manvitha Ponnapati, Samuel Rodriques. and Andrew D White. Language agents achieve superhuman synthesis of scientific knowledge. arXiv preprint arXiv: 2404:2240. 2024.
Andreas Stephan, Dawei Zhu, Matthias Aßenmacher, Xiaoyu Shen, and Benjamin Roth. From calculation to adjudication: Examining Ilm judges on mathematical reasoning tasks. arXiv preprint arXiv:2409.00 0044099041663 148, 2024.
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprini arXiv:221 1.09085, 2022.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptin Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, Katie Millican, al. Gemini: a family of highly capable multimodal models. arXiv preprint 2xXi:2332.1//800 2023.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530. 2024.
Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 29(8): 1930-1940, 2023.
Francesco S Varini, Jordan Boyd-Graber, Massimiliano Ciaramita, and Markus Leippold. Climatext: A dataset for climate change topic detection. arXiv preprint arXiv:2012.00483, 2020.
Bin Wang, Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang, et al. Mineru: An open-source solution for precise document content extraction. arXiv preprint arXiv:2409. 18839, 2024a.
Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scientific problem-solving abilities of large language models. arXiv preprint urXiv:2307.10633 2023.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024b.
Nicolas Webersinke, Mathias Kraus, Julia Anna Bingler, and Markus Leippold. Climatebert: A pretrained language model for climate-related text. arXiv preprint arXiv:2110.12010 2021.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting clicits reasoning in large language models. Advances in 2022.
neural information processing systems,
Jiaheng Wei, Yuanshun Yao, -aanFFracoii Ton, Hongyi Guo, Andrew Estornell, and Yang Liu. Measuring and reducing Ilm hallucination without gold-standard answers. arXiv preprint arXiv:2402.1044 104 2024.
Yao Xu. Shizhu He, Jiabei Chen, Zihao Wang, Yangqiu Song, Hanghang Tong, Guang Liu, Kang Liu, and Jun Zhao. Generate-on-gre -nggaph: Treat 11m both agent and kg in incomplete knowledge graph question answering. arXiv preprint arXiv:2404. /4741, 2024.
Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search. arXiv preprint wwXivv2504.08006 2025.
An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 1555 2024a.
13

---

Published as a conference paper at ICLR 2026
Zonglin Yang, Wanhao Liu, Ben Gao, Tong Xie, Yuqiang Li, Wanli Ouyang, Soujanya Poria, Erik Cambria, and Dongzhan Zhou. Moose-chem: Large language models for rediscovering unseen chemistry scientific hypotheses. arXiv preprint arXiv:2410.07076, 2024b.
Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, et al. Justice or prejudice? quantifying biases in Ilm-as-a-judge. arXiv preprint arXiv:2410.02736, 2024.
Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Yongliang Shen, Ren Kan, Dongsheng Li, and Deqing Yang. Easytool: Enhancing Ilm-based agents with concise tool instruction. arXiv preprint arXiv:2401.066 24020061,,, 2024.
Di Zhang, Wei Liu, Qian Tan, Jingdan Chen, Hang Yan, Yuliang Yan, Jiatong Li, Weiran Huang, Xiangyu Yue, Wanli Ouyang, et al. Chemllm: A chemical large language model. arXiv preprint arXiv:2402.06852, 2024.
Xuan Zhang and Wei Gao. Towards Ilm-based fact verification on news claims with hierarchical step-by-step prompting method. arXiv preprint arXiv:2310.00305, 2023.
Yifan Zhang, Cheng Wei, Shangyou Wu, Zhengting He, and Wenhao Yu. Geogpt: Understanding and processing geospatial tasks through an autonomous gpt. arXiv preprint wIXi:2307.009300 2023.
14

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A.1
PAPER SELECTION AND TIERED CLASSIFICATION
The proposed benchmark derives from 100,000 papers downloaded from OpenDataL (He et al. 2024 which undergo tiered classification process resulting in three distinct paper collections:
Base paper collection (Poase)
High-journallimpact papers (Ri)
High-citation papers (Phe)
The Phase collection is constructed through semantic similarity analysis of paper titles and keywords. Our methodology employs both positive and negative keywords to enhance selection precision. Negative keywords effectively filter semantically related but thematically irrelevant papers. For instance, when using "earth" as positive keyword, astronomy-related papers may incorrectly appear relevant. To address this, we introduce "cosmos" as a negative keyword, requiring papers to maintain similarity score below specified threshold (0.1) with negative keywords while exceeding minimum threshold (0.2) with positive keywords. This tual-threshold approach yields the final Phase collection of 100,000 papers.
Using an analogous positive and negative keyword approach, we accurately classify papers into five Earth science subdomains (hydrosphere, biosphere, lithosphere, atmosphere, and cryosphere). The complete domain-specific keyword system is presented Figure]
Positive and Negative Keywords
Overall Positive Keywords Earth, Earth system, hydrosphere, biosphere, lithosphere, atmosphere, cryosphere. Overall Negative Keywords cell biology, virus, pharmaceuticals, chemistry, physics, astronomy, food science, proteins, microbiology.
Hydrosphere Positive Keywords
water cycle, ocean, rivers, lakes, groundwater, ice caps, aquifers, precipitation, evaporation, humidity.
Hydrosphere Negative Keywords
chemistry, universe, planets, astronomy, astrophysics, space, stars, galaxy, cosmology.
Biosphere Positive Keywords
ecosystem, biodiversity, habitat, species, biomes, ecological balance, carbon cycle.
Biosphere Negative Keywords
cell biology, chemistry, medicine, pharmacology, microbiology, biochemistry, toxicology, pathology, clinical.
Lithosphere Positive Keywords earthquake, tectonic plates, earth's crust, minerals, rocks, soil, sediments, mountains, volcanoes, landforms, geological processes. Lithosphere Negative Keywords ancient texts, archacology, culture, history, artifacts, civilization, prchistoric, mythology, anthropology.
Atmosphere Positive Keywords
stratosphere, troposphere, weather, climate, greenhouse gases, ozone layer, air pressure, humidity, winds, carbon dioxide, temperature. Atmosphere Negative Keywords universe, galaxy. astronomy, astrophysics, space, stars, planets, cosmology, black holes, nebula, solar system.
15

---

Published as a conference paper at ICLR 2026
Cryosphere Positive Keywords
glaciers, ice sheets, sea ice, permafrost, snowpack, icebergs, frozen ground, climate change, albedo effect, polar regions.
Cryosphere Negative Keywords frozen food, ice cream, refrigeration, freezing, cold storage, ice cubes, food preservation, chilling, frost.
Figure 7: Positive and Negative Keywords.
From the initial paper collection Phuses we select subset of papers Phy published in high-quality
journals based on multiple criteria including impact factor (IF) (Garfield et al. 1994), disciplinary ranking, and expert assessment. Representative journals are listed Table with the complete journal inventory provided in Supplementary Materials.
Table 7: Partial List of Selected Journal Titles.
CELL
NATURE
SCIENCE
NATURE POTECHNOLOGY
NATURE CELL BIOLOGY
NATURE CHEMICAL BIOLOGY
NATURE CHEMISTRY GENETICS
NATURE CLIMATE CHANGE
COMMUNICATIONS
NATURE GROSCIENCE
NATURE IMMUNOLOGY
NATURE MATERIALS
NATURE MEDICINE
NATURE METHODS
NATURE VANDTECHNOLOGYY
NATURE NEUROSCIENCE
NATURE PHOTONICS
NATURE PHYSICS
CITY
ONCOL
Finally, we select the 1000 papers Phe with the highest number of citations from Phj to form collection. The distribution of papers in each collection shown in Table
A.2 QA CONSTRUCTION
When we construct QA questions, the prompts we use will vary depending on the format and task of the question. For questions of different formats, we use the following prompts.
Prompt for Different Question Formats
Free Form QA
Free-form questions and answers allow for open-ended responses. These questions typically require detailed explanations, recommendations, or descriptions. The answers can vary in length and structure, depending the complexity of the question. For example, question about improving the marine environment may require a list of measures with explanations. Format Rules: Answers can be in any format, including sentences, lists, or paragraphs. There are no strict restrictions on length or structure.
Multiple Choice
Multiple-choice questions provide question followed by several answer choices, typically labeled with letters (e.g., A, B, C, D). Only one choice is correct. The task is to select the correct choice by providing the corresponding letter. Format Rules: The question requires a prompt along with four options,
of which is the correct answer. The answer must be a single letter corresponding to the correct choice (e.g.. 'A', 'B', 'C', or 'D'). The answer should not include the full text of the choice.
True False
True/false questions require determining whether given statement is correct or incorrect.
The answer must be either 'True' or 'False", depending on the accuracy of the statement. Format Rules: The answer must be exactly 'True' or 'False". No additional text or explanations are allowed.
16

---

Published as a conference paper at ICLR 2026
Free Form QA
Fill-in-the-blamt questions provide sentence or statement with one or more missing words, phrases number. The task is to complete the sentence by filling in the blank(s) with the correct word(s). Format Rules: The answer must be the exact word or phrase that fits the blank. No additional text or explanations are allowed unless explicitly requested.
Figure 8: Prompt for Different Question Formats.
The prompts for different tasks shown in Table 2 The complete QA prompt is shown in Figure
Complete QA Construction Prompt
System Prompt
You highly skilled scientist with deep expertise in reading and analyzing scientific literature. Below is research paper that you will carefully examine. Your task to generate well-structured question and answer based the provided instructions. Ensure that your output is precise, relevant, and adheres to the specified guidelines. Instructions
1. Format: <Format Prompt>
Task: <Task Prompt>
3. Answer Explanation: Each question must be accompanied by a corresponding answer explanation.
4. Output Format: The output should be structured dictionary, including the following
keys:
question: The generated question.
The correct answer to the question. A list containing the explanation(s) for the answer. 5. Select One Specific Detail
answer:
explanation:
Do not generalize the entire paper into question. Instead, carefully select the most relevant and specific part of the paper (e.g.. key finding, methodological detail, or discussion point) and craft detailed question around The question should highly specific and require nuanced understanding of the paper to answer. Avoid generating general or overly simplistic questions.
6. Content Relevance and Accuracy
Ensure that the questions are directly derived from the content of the paper. Avoid generating questions that are irrelevant based on incorrect facts. The questions should accurately reflect the findings, methodologies, or discussions presented in the paper.
7. Independent and Complete Question, Answer, and Explanation
Self-contained: The question, answer, and explanation must be self-contained and complete. Do not phrases such "this article" "this paper" "according the article" "according the paper" or similar references to the paper.
Avoid Personal Pronouns: Additionally, avoid using personal pronouns like "we" or "our." The question, answer, and explanation must stand alone and be understandable without additional context.
Avoid Analysis of the Paper: Do not include any analysis of the paper in the questions, answers, explanations. Instead, transform the analysis into independent statements that are self-sufficient and do not rely on the paper for context.
Example
"question": "The generated question based on the paper.", "answer": "The corresponding answer the question.", "explanation":
"Step 1: Explanation for the first step.". ",
"Step 2: Explanation for the second step." "Step Explanation for the third step."
17

---

Published as a conference paper at ICLR 2026
]
Figure Complete QA Construction Prompt.
After constructing the QA pair, we will also refine the QA. The main purpose is to remove extra references that may exist the QA, such as mentioning papers or non-existent figures in the question. The refine prompt is as follows.
QA Refine Prompt
System Prompt
Please modify the following question, answer, and explanation to remove any expressions related to "paper." "article." "study," " or similar references. Ensure that the question is complete and can be answered directly based on the explanation provided, without requiring any additional context knowledge of the paper. After modification, maintain the original dictionary format and ensure that the meaning of the questions, answers, and explanations remains unchanged.
Instructions
Remove References the Paper:
Eliminate any phrases such "this paper," "the article," "according to the study," similar references.
Ensure the question, answer, and explanation self-contained and do rely external sources for understanding.
Maintain Clarity and Completeness:
The question should be clear, specific, and able to stand alone, without referencing the paper.
The answer should directly address the question without referencing the paper.
The explanation should provide sufficient detail to justify the answer, using independent and self-sufficient statements, without referencing the paper.
3. Preserve the Dictionary Format:
Keep the output the original dictionary format, including the keys 'question', 'answer', and 'explanation'.
Ensure that the meaning of the questions, answers, and explanations remains unchanged.
Figure 10: QA Refine Prompt.
A.3 SCIENTIFIC EXPLORATION DIALOGUE CONSTRUCTION
The Earth-Gold dataset construction from the high-citation paper collection Phe involves a two-phase processing pipeline:
1. Paper Structuring Phase: Decompose each paper into four core components:
Summary of existing methods (My)
Limitations of existing methods (To)
Proposed new methods (M1)
Potential limitations of new methods (L,)
2. Dialogue Generation Phase: Generate two-turn scientific exploration dialogues
The paper structuring prompt is presented as follows:
18

---

Published as a conference paper at ICLR 2026
Paper Structuring Prompt
System Prompt
You are an Al research assistant with expertise in analyzing and structuring academic papers. Your task is to extract and organize the content of a research paper into five specific sections: research direction, methods of previous work, limitations of previous work, method of this work, and limitation of this work. Follow the instructions carefully and output only a dictionary (dict) specified.
Instructions
Task Description:
Extract and categorize the content of the provided research paper into the following five
sections:
1) Research direction
2) Methods of previous work
3) Limitations of previous work
4) Method of this work
5) Limitation of this work
Exclude all other sections or content not related to these five categories.
2. Output Format:
The output must be a Python dictionary (dict) with the following structure:
"research direction": "xxx", "methods of previous work"; "xxx", "limitations of previous work": "xxx", "method of this work": "xxx", "limitation of this work": "xxx"
-Replace ""xxx" with the extracted content for each section.
Extraction Rules:
research direction: Identify the primary focus, goals, Avoid using terms such as "paper," "study," "work," or "thesis." Instead, directly state the research direction in concise and general manner.
of investigation in the paper.
methods of previous work: Summarize the methodologies or approaches used in prior research relevant the paper.
limitations of previous work: Highlight the shortcomings, gaps, or challenges in previous research.
method of this work: Describe the methodology or approach proposed or used in the current paper.
limitation of this work: Identify and summarize any limitations or weaknesses in the current work's approach.
4. Important Notes:
Strictly adhere to the five sections outlined above.
Do not include any additional text, explanations, commentary outside the dictionary.
Ensure the output is concise, clear, and directly relevant to the specified sections.
Elaborate the content with much detail possible, retaining specific numerical
values, dataset names, method names, author names, etc., and ensure the content is highly professional and information-rich. Example
'research direction": "xxx", "methods of previous work": "xxx", "limitations of previous work": "xxx",
"method of this work": "xxx". "limitation of this work": "xxx"
Figure 11: Paper Structuring Prompt.
19

---

Published as a conference paper at ICLR 2026
The structured paper content enables the generation of two progressive dialogue rounds:
Round The LLM summarizes existing methods (Mn) and analyzes their limitations (Lo). This round evaluates the model's understanding of domain-specific research methodologies.
Round 2: Building upon Round 1, the LLM proposes innovative methods (M1) while
reflecting potential limitations (L,). This round assesses the model's scientific innovation capability and critical reflection skills.
The dialogue construction prompt is presented follows:
Dialogue Construction Prompt
System Prompt
You are an AI research assistant specializing in refining supervised fine-tuning (SFT) data for large language models. Your task is to enhance the quality and linguistic diversity of the provided SFT data while preserving its original structure and content. Instructions
1. Task Description:
Polish the text in the SFT data, including both the "user" and "assistant" parts.
Ensure the polished text is clear, concise, and linguistically diverse while maintaining the original meaning and intent.
2. Refinement Requirements:
Articulate the logic with utmost clarity, employing logical conjunctions to underscore logical
relationships where pertinent.
Elaborate on the content with as much detail as possible, retaining specific numerical values, dataset names, method names, author names, etc., and ensure the content is highly professional and information-rice
While retaining the original meaning the sentence, appropriately add or modify make the unsmooth conversation content smooth. Constraints:
Do not add, remove, alter the number of dialogue turns.
Preserve the original structure and role labels ("user" and "assistant").
Use variety of linguistic styles, vocabulary, and phrasing to increase the diversity of the text.
4. Output Requirements:
Output the modified SFT data as a dialogue list in the same ISON format as the original. Ensure the polished text is natural, engaging, and suitable for training large language models. Example
("role": "user", "content": "<INPUTI> Could you provide an overview of related work and discuss their key limitations?""..
("role": "assistant", "content": "Certainly. The related works include the following. <IN- PUT2> However, these approaches face several limitations. <INPUT3>"), ("role": "user", "content": "Given these existing works and their limitations, can you propose a new method and evaluate its potential drawbacks?").
I "rrole: "assistant". "content": "Building these foundations, the proposed method is structured as follows. <INPUT4> Despite its advantages, this method has certain limitations. <INPUT5>")
Figure 12: Dialogue Construction Prompt.
The <INPUTI> to <INPUT5> the prompt are replaced by the values the dictionary obtained in paper structuring.
20

---

Published as a conference paper at ICLR 2026
A.4 EXPERT ANNOTATION GUIDELINES
To ensure that all benchmark questions possess scientific value, specificity, and challenge, we recruited experts with Master's s or PhD degrees in Earth sciences (e.g., meteorology, geology) evaluate each candidate question. Experts scored each question along six dimensions:
1. Scientific Method:
Does the question involve concrete scientific methods such data processing, analysis, or experiments?
2. Data Source:
Does the question include specific datasets or experimental data?
3. Temporal or Spatial Context:
Does the question reference a specific time or location, such as an earthquake, typhoon, or drought event?
4. Literature Citation: Does the question reference specific scientific paper?
5. Method Details: If a new method is proposed, is it described in sufficient detail (step-by-step procedures, method names, etc.)?
6. Limitation Analysis: Does the question include a logical, in-depth discussion of method limitations?
Each dimension was scored scale of 0 to 2 (0 = not present, present but insufficiently detailed, 2 = very specific). We retained only those questions for which at least three dimensions received score greater than 0 and the total score was at least 5. This scoring system ensures that retained questions are scientifically valuable, specific, and challenging.
Table 8: Expert Evaluation Dimensions and Scoring Criteria
Dimension
Description
Score Range
Scientific Method
Does the question involve data analysis or experiments? Are datasets or sources explicitly specified?
0-2
Data Source
0-2
Temporal Spatial Context
Does reference specific scientific event or location?
0-2
Literature Citation
Is scientific paper cited?
0)-
Method Details
Are methods described in steps or named specifically?
0-2
Limitation Analysis
Is there in-depth discussion of limitations?
0-2
A.5 LLM DATA CONSTRUCTION AND EVALUATION: BIASES AND LIMITATIONS
Although there some bias in using LLM for data construction (Skarlinski et al.) 2024) and evalua- tion (Ye et al.] 2024), we will try to reduce this bias much as possible from the following three perspectives.
Reconstruction-basee Generation Instead of Free Creation. The construction process relies on reconstruction from original papers rather than free-form generation. The LLM extracts and reorganizes content from the source material, ensuring scientific accuracy and factual fidelity. In this context, all outputs in Earth-Gold are fully manually verified. For Earth-Iron and Earth- Silver, a combination of model difficulty calibration and human inspection ensures quality control. Specifically, during difficulty calibration, we let LLMs attempt the questions: besides filtering out questions that most models answer correctly, this process identifies questions that all models answer incorrectly. Such questions may either be genuinely difficult flawed; therefore, human experts review and remove questions that contain errors, while retaining those that are correctly constructed but challenging. All remaining questions have at least one model successfully answering them, ensuring that the question construction is reliable and scientifically sound.
Fine-grained Structured Construction Process. When structuring the papers, we consider their sectional organization and employ rule-based parsing combined with LLM refinement to efficiently and accurately structure the documents. Specifically, MinerU is used to parse PDF papers and output Markdown format. Using Markdown syntax, extract first- and second-level headings and assign content to common sections such as abstract, introduction, related work, method, experiment, discussion (optional), and conclusion. Papers that do not fit these rules manually structured
21

---

Published as a conference paper at ICLR 2026
by human experts. Subsequently, apply LLM refinement to produce the four parts defined in Section 3.4, namely (M L .M¹ L'). where (M' LD) (existing work) is primarily extracted from related work, and (M' _ (new work) is derived from method, experiment, and discussion sections. Summarizing sections such as abstract, introduction, and conclusion are provided as context, ensuring that the model captures the overall research direction while focusing specific sections for accurate summarization.
Contrast-based Evaluation Ensures Relative Objectivity and Consistency. In evaluation, the computation of win rate and retention rate uses LLMs. For free-response questions, win rate calculated by comparing model responses against reference answers. Retention rate derived from ranking the responses relative to the reference answers. Importantly, both metrics rely on comparative ranking rather than direct LLM scoring, which can introduce bias and is difficult to quantify. Comparative ranking provides relatively objective and consistent assessment across different models.
A.6 QA EXAMPLES
QA Example 1 (Multiple Choice)
Question
What factor plays pivotal role in the formation of sapropel S1 in the Mediterranean, and what evidence suggests its influence during the last interglacial insolation maximum?
A) Increased wind stress and evidence from high sedimentation rates.
B) Monsoon-fueled freshwater input from the Nile River and synchronized timing of sapropel
S1 deposition with wetter conditions in North Africa.
C) Enhanced salinity from increased evaporation and evidence from salinity isotope ratios.
D) Elevated ocean temperatures and evidence from fossil coral records. Explanation
Monsoon- fueled freshwater input, particularly from the Nile River, significantly affected sapropel S1 formation.
The timing of sapropel S1 deposition in the eastern Mediterranean closely coincided with periods of increased Nile River discharge due to enhanced monsoon strength over North Africa.
Evidence suggests these monsoonal shifts resulted in wetter conditions. which aligned with the insolation maxima during the interglacial, promoting the deposition organic-rich sapropels.
Answer
Figure 13: QA Example 1.
QA Example 2 (True or False)
Question
Atmospheric humidity in Nanjing's 's urban canopy layer demonstrates significant nighttime differences among local climate zones, particularly due to moisture deficits during colder months, and shows clear seasonal patterns in humidity ratios with greater discrepancies in warmer months than colder months. Explanation
Step 1: Significant differences in atmospheric humidity among local climate zones are observed during nighttime, highlighting variations in humidity ratios across seasons.
Step 2: Negative humidity ratio values (moisture deficits) are more frequent during colder months, while positive values (moisture excess) are noted more often warmer months. Step 3: Seasonal patterns are more pronounced in humidity ratio differences, showing greater discrepancies during warmer months, supporting the statement's claim about seasonal variations.
22

---

Published as a conference paper at ICLR 2026
Answer
True
Figure 14: QA Example 2.
QA Example 3 (Fill in the Blanks)
Question
In the comparison between SP-CCSM4 and CCSM4, the difference in projected ENSO-shear relationships is attributed to the varying intensity and spatial extent of anomalous westerlies at upper levels during El NINO events. Specifically, while the climatology of easterly flow at simulated in CCSM4 is weaker and retreated eastward compared to SP-CCSM4, the projected westerly anomalies due to ENSO are SP-CCSM4.
CCSM4 relative to those in
Explanation
Step 1: The climatology of easterly flow at 200 hPa noted to be weaker and shifted eastward in CCSM4 compared to SP-CCSM4.
Step 2: During El NINO events, CCSM4 projects stronger westerly anomalies than SP-
CCSM4 at upper atmospheric levels, contributing to variability in ENSO-shear relationships. Step 3: These differences in the intensity and spatial distribution of westerly anomalies directly influence the projected ENSO-shear relationships, highlighting the distinct behavior between the two models response to ENSO warmer climate. Answer
200 hpa, stronger
Figure 15: QA Example 3.
QA Example 4 (Free QA)
Question
How does the redistribution of acrosol particles captured through nucleation and impaction scavenging into various drop size categories affect the dynamies and ultimate removal process of aerosol particles in the cloud environment, and what implications do the stochastic collision- coalescence process and drop growth mechanisms have on the distribution and contamination levels observed in precipitation-sizet drops? Explanation
Step 1: Aerosol particles captured in cloud drops through nucleation and impaction scav- enging, with their subsequent redistribution depending on the stochastic collision-coalesconn processes of drop interactions.
Step 2: As smaller drops merge and grow, aerosol particle mass becomes concentrated in larger precipitation-sizer drops, which are more effective in removal due their increased mass and falling speed.
The stochastic collision-coalescenn process also results in varying contamination levels, with smaller drops being highly contaminated due to their increased pcrosol-to-watte mass ratios, while larger drops ensure bulk removal of aerosol mass, significantly affecting the efficiency of wet deposition mechanisms in clouds. Answer
Step 3:
The redistribution of acrosol particles captured via nucleation and impaction scavenging significantly alters the dynamics acrosol removal by moving the acrosol mass into larger drop size categories, thereby concentrating aerosol particles in precipitation-sized drops. During stochastic collision-coalessennee processes, smaller cloud drops merge and grow, carrying the aerosol particle with them, leading coupling between water mass and aerosol mass growth. This results in a redistribution where the main aerosol mass shifts to drops of precipitation size, which more effective removing particles from the cloud
23

---

Published as a conference paper at ICLR 2026
environment. Additionally, smaller drops within the cloud exhibit higher contamination levels due to their higher aerosol-to-water mass ratio, while larger precipitation drops accumulate the majority of aerosol mass, thereby ensuring efficient removal from the atmosphere.
Figure 16: QA Example 4.
A.7 EXAMPLES OF SCIENTIFIC EXPLORATION DIALOGUES
Earth-Gold: Scientific Exploration Dialogue Example
User
I am interested in the study of sedimentary cycles and their depositional rates extensive geological time spans, with a particular emphasis on how the likelihood of sedimentary rock destruction changes with age. Could you provide comprehensive overview of the existing literature in this area, highlighting the principal limitations of these studies? Assistant
Absolutely. The primary works in this domain include Gregor's s seminal approach, where he aimed estimate the global denudation rates across various geological epochs. He derived depositional rates by analyzing the mass-age distribution of current sediments. Nonetheless, these methodologies encounter certain restrictions. For instance, Gregor himself cast doubt on his findings regarding depositional rates during the pre- rre-Carboniferous phases of the Phanerozoic eon. His assumption uniform probability of destruction for sedimentary rocks, regardless of their age, failed to consider fluctuations in the half-lives of rocks over time.
User
Considering these existing studies and their constraints, could you suggest an innovative approach and discuss its potential drawbacks?
Assistant
Leveraging the insights from previous studies, propose novel approach which follows this outline: recharting Gregor's data using semilogarithmic graphs, converting the data into units of grams per year, and fitting these values as exponential functions. This would be complemented by modeling sedimentary cycles that exhibit different half-lives to account for mass-age relationships. Despite innovative aspects, this approach is not without limitations. While it successfully elucidates Gregor's 's mass-age associations, it lacks uniqueness and posits steady total sedimentary mass across cycles, which might overlook some of the more intricate geological phenomena.
Figure 17: Scientific Exploration Dialogue Example 1.
Earth-Goll Scientific Exploration Dialogue Example
User
Please examine the interplay and responsiveness of the convective boundary layer (CBL) under diverse atmospheric and oceanic conditions, including sea surface temperature (SST), surface wind speed, and tropospheric moisture levels. Could you provide summary of existing research and highlight their main shortcomings? Assistant
Certainly. Related studies include the following. Betts (1973) introduced closure model for the subcloud layer that utilized an adapted subsidence approach and energy balance closures consistent with Sarachik's 's tropical model (1978). Nonetheless, these methods exhibit several limitations. For instance, models by Betts and Sarachik were based on assumptions that might not thoroughly represent the coupling dynamics between the CBL and varying atmospheric parameters.
24

---

Published as a conference paper at ICLR 2026
User
Considering these previous studies and their limitations, could you propose novel approach and assess its potential limitations?
Assistant
Drawing inspiration from these constructs, the proposed approach is defined as follows. This work formulates detailed model to analyze the CBL equilibrium by evaluating uncoupled solutions influenced by SST, wind speed, and tropospheric variations, along with assessments of coupled troposphere solutions and their responsiveness changes. Despite its benefits, this method does contain some limitations. The model might encounter challenges in accurately forecasting the CBL's behavior under extreme climatic conditions when integrating intricate feedback mechanisms not entirely accounted for in the theoretical framework.
Figure 18: Scientific Exploration Dialogue Example 2.
A.8 USAGE OF LANGUAGE MODELS
We utilized a large language model (LLM) to aid in the preparation of this manuscript. Its use was limited editorial tasks, including proofreading for typographical errors, correcting grammar, and improving the clarity and readability of the text.
25


Reviews:
[{"review_id": "eKpJsq6CXb", "reviewer": "Reviewer_h6Tz", "summary": "The paper introduces a suite of 3 new LLM benchmarks, two QA benchmarks around Earth science. \nThere are two QA style datasets and a multi-turn conversation benchmark. \n\nThe benchmark is based on 100k papers. Papers are categorized for field/sub-field.\nThe authors select subsets of size 100k, 10k, 1k by impact/popularity.\n\n\nThe QA pairs are constructed using LLMs. Each question/answer pair is built around a task category. The tasks include understanding but also reasoning and research tasks, such as experimental design and code generation.\nFor each paper tasks are selected to match the abstract.\nToo simple examples are filtered.\nExperts reviewed the remaining questions, keeping \n\n\nFor the multi-turn dialog they model papers as a transition from( existing methods and their limitations) to (novel method and its constraints). They extract these from the top 1k most cited papers. \nThen they auto-generate two turn dialogues of summary / proposal with the respective limitations.\nAgain, the examples are reviewed by experts.\n\nTo evaluate the multi-turn task, the paper proposes a novel metric based on ranking several samples from an LLM against the gold example along with a diversity metric.\n\nA slightly dated set of models is used in experiments showing significant differences between models, particularly for the multi-step task.\n\nThere are some issues with this paper that I am hopeful can be fixed.", "questions": "Is there an expert review of only the questions or also of the answers? How were the experts recruited / compensated? What is the acceptance / rejection rate? What is the agreement for these decisions?\n\n\nSmaller issues / improvements\n\nPlease check references for the models, e.g.\n* Not sure Islam & Moushi is the right reference for GPT-4o\n* Gemini is cited as \"Team et al\" ?\n\nThe model names need to be more explicit, e.g. Gemini 2.0 Flash or Claude 3.7 Sonnet\n\nIn Figure 3, \"a task that best suits your construction…\" did you want to say qualification?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Addressed a variety of scientific tasks, particularly beyond reasoning/extraction\n\nLarge corpus of papers used.\n\nPaper is well written and has very little fluff.\n\nHeadroom for evaluation\n\nExpert verification of at least parts of the data", "weaknesses": "GPT-4o was used in several steps of the generation process which might lead to bias. It doesn't look like a problem from the results but this should at least be explicitly acknowledged.\n\nThere is little detail on \"human experts determined their retention based on the question's value\". Can you tell a bit more about this process?\n\nEvaluation of the Free Response QA tasks could be extended beyond embedding similarity, e.g. by using a judge model or using rubrics.\n\nIt is unclear if diversity should be evaluated by temperature sampling. Without calibration, that parameter might do different things with different models, as evidenced by results in Table 5\n\nEvaluation of some more recent models with thinking enabled", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Uf69B7Ccqr", "reviewer": "Reviewer_aJ8Z", "summary": "EarthSE introduces a benchmark for evaluating LLMs’ Earth science exploration capabilities. It includes three datasets: (1) Earth-Iron, (2) Earth-Silver, and (3) Earth-Gold. These datasets target different levels of knowledge, from broad assessment (Earth-Iron) to higher difficulty requiring professional understanding (Earth-Silver), and finally open-ended scientific exploration through dialogue (Earth-Gold). The authors then analyze these capabilities in leading LLMs and find that they perform reasonably well on Earth-Iron but struggle with Earth-Silver and show low retention and diversity on Earth-Gold tasks.", "questions": "(1) In lines 455-460, it is mentioned that the initial CoT steps from the question construction step are provided during inference to see whether that helps with performance on more challenging question types. However, it is not clear to me why you didn’t instead ask the model to generate CoT reasoning while answering the question. Providing the CoT steps from the question construction step might leak knowledge about the answer that it shouldn’t, and that the model wouldn’t normally have access to.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "EarthSE introduces a benchmark with various difficulty levels and broad coverage across subfields (114 disciplines and 11 tasks), as well as multi-turn, open-ended dialogues for scientific exploration and discovery. To build the QA datasets, the authors leverage around 100,000 research papers and categorize them by journal impact, citation count, and topical focus. They prioritize high-quality sources and exploit paper structure aligned with the scientific discovery process, followed by extensive data cleaning and expert validation. They also introduce additional metrics for assessing Earth-Gold, evaluate multiple LLMs, and report interesting, important insights.", "weaknesses": "(1) Earth-Gold uses a fixed two-turn format. Real scientific exploration often needs longer iterative chains. It would be interesting to see the results with more turns.\n\n(2) The inference-time “initial CoT steps” taken from question construction seem to boost FIB accuracy, but may leak answer-related cues a model wouldn’t get at test time. It would be interesting to see the self-generated CoT results as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "876Wm7w6Yo", "reviewer": "Reviewer_zDSD", "summary": "This paper presents EarthSE, a QA dataset benchmarking the earth science exploration capability of LLMs. These questions are sourced from 100k Earth Science papers, and are constructed into three subsets (Iron, Silver, and Gold). The authors decomposed each paper and then used GPT-4 to generate dialogues, followed by using human expert validation to ensure the dialogue quality. On several LLMs, the authors found that CoT guidance enhances the performances on challenging questions.", "questions": "- How is the paper decomposed into the structured components? Is it also using GPT-4?  \n- The experts scored the dialogue, but did they score a subset of the QA questions in the benchmark?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This is the first science QA dataset for earth science that is derived from this magnitude of academic papers.  \n- This paper presents a principled approach to construct benchmark datasets from earth science publications.", "weaknesses": "- Using one metric (SES) to assess advanced capabilities of scientific exploration (e.g., methodology induction, limitation analysis, and concept proposal) seems a bit too on reductionist side to me. Looking at the two components of the SES metric (retention rate and diversity), I became less convinced about the utility of this metric to the intended capabilities to evaluate.  \n- How are the Iron and the Silver subsets divided? The boundary seems a bit arbitrary to me.  \n- Similarly, the decision to make Iron & Silver in QA formats while making Gold in dialogue format also appears very arbitrary to me.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MoAnbLQqIJ", "reviewer": "Reviewer_XoSt", "summary": "This paper proposes a new set of benchmarks that deal with Earth Science, a topic that is missed by current LLM benchmarks. The contribution is two QA datasets, an easier Earth-Bronze and a more challenging Earth-Silver, with an additional scientific exploration Earth-Gold dataset. All in all, over 100,000 papers were considered for the benchmark construction. The paper additionally proposes a new metric to be used to judge LLM performance on scientific exploration, which is used on Earth-Gold. Lastly, 11 LLMs are evaluated on the 3 proposed benchmarks.", "questions": "1. How are biases and noise from using LLMs for both question generation and evaluation and metrics controlled? Is it possible that even that the data cleaning step, which uses LLMs, also exhibits biased/erroneous behavior?\n2. In open ended tasks, what happens when more than one answer is reasonable? Are there quantitative metrics that show this not to be an issue? \n3. Given the near-saturated results on Earth-Bronze/Silver, why are these benchmarks still worth further exploration?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Extending LLMs to topics unexplored by current benchmarks, like Earth Science is important\n- The benchmarks are comprehensive, taking into account a pool of more than 100,000 papers. The end result features a large set of 114 disciplines, which is relatively large compared to other similar benchmarks\n- The paper is clear and easy to understand", "weaknesses": "- LLMs are used in both benchmark construction and in evaluation (retention, win rate). Given that LLMs have been shown to be biased in various ways [1], using them to deconstruct papers into parts and then come up with questions, and then judge proposed answers seems prone to noise, biases and errors. Even with expert curation, the sheer volume of the base set of papers raises concerns with the regards about the amount of errors that can be caught. \n- The nature of the tasks means they're hard to define well. For example, the example tasks given in Table 2's research section are open ended and I could see human experts giving varied and even contradictory answers. \n- Looking at Earth-Iron/Silver: these feature more well defined answers, but on the other hand, are close to being saturated. Apart from fill in the blank, the best models get 60/70/80% on the other categories. This is approaching saturation, which raises questions about the benchmark’s continued relevance and value to the community.\n\nMinor things:\n- Figure 2 should probably be a table\n- In line 238 in the caption of Table 2, $P_{hj}$ appears twice\n- I think that [2] is very relevant and should be included in the discussion or in Table 1, or both. \n\n[1] Ye, Jiayi, et al. \"Justice or prejudice? quantifying biases in llm-as-a-judge.\" arXiv preprint arXiv:2410.02736 (2024).\n\n[2] Skarlinski, Michael D., et al. \"Language agents achieve superhuman synthesis of scientific knowledge.\" arXiv preprint arXiv:2409.13740 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)