ClimaQA: An Automated Evaluation Framework for Climate Question Answering Models
Authors: Veeramakali Vignesh Manivannan
Yasaman Jafari
Srikar Eranky
Spencer Ho
Rose Yu
Duncan Watson-Parris
Yian Ma
Leon Bergen
Taylor Berg-Kirkpatrick
Venue: ICLR 2025 Poster (2025)
Keywords: Climate Benchmark
Scientific Foundation Models
Scientific Question Answering
Large Language Models
Automated QA generation
Primary area: datasets and benchmarks
TL;DR: We develop ClimaGen, an automated framework for generating climate science QA datasets, and introduce ClimaQA-Gold and ClimaQA-Silver as benchmark datasets to evaluate and improve the performance of foundation models in climate science.

Abstract: The use of Large Language Models (LLMs) in climate science has recently gained significant attention. However, a critical issue remains: the lack of a comprehensive evaluation framework capable of assessing the quality and scientific validity of model outputs. To address this issue, we develop *ClimaGen* (Climate QA Generator), an adaptive learning framework that generates question-answer pairs from graduate textbooks with climate scientists in the loop. As a result, we present *ClimaQA-Gold*, an expert-annotated benchmark dataset alongside *ClimaQA-Silver*, a large-scale, comprehensive synthetic QA dataset for climate science. Finally, we develop evaluation strategies and compare different LLMs on our benchmarks. Our results offer novel insights into various approaches used to enhance knowledge of climate LLMs. ClimaQA’s source code is publicly available at https://github.com/Rose-STL-Lab/genie-climaqa

Published as a conference paper at ICLR 2025
CLIMAQA: AN AUTOMATED EVALUATION FRAME- WORK FOR CLIMATE QUESTION ANSWERING MODELS
Veeramakali Vignesh Manivannan, Yasaman Jafari, Srikar Eranky, Spencer Ho, Rose Yu, Duncan Watson-Parris, Yian Ma, Leon Bergen & Taylor Berg- Barg-Kirkpatrick. University of California, San Diego
{vmani vannan, ya jafari, seranky, s8ho, roseyv, dwat sonparris, yianma, lbergen, (berg))ucssd. edu
ABSTRACT
The use of Large Language Models (I.LMs) in climate science has recently gained significant attention. However, a critical issue remains: the lack of a compre- hensive evaluation framework capable of assessing the quality and scientific va- lidity of model outputs. To address this issue, we develop ClimaGen (Climate QA Generator), adaptive learning framework that generates question-answer pairs from graduate textbooks with climate scientists in the loop. As a result, we present ClimaQA-Gold, an expert annottan benchmark dataset alongside ClimuQA-Silver, a large-scale, comprehensive synthetic QA dataset for climate science. Finally, we develop evaluation strategies and compare different LLMs on our benchmarks. Our results offer novel insights into various approaches used to enhance knowledge of climate LLMs. ClimaQA's source code is publicly avail- able at hutptts:: http://////tttttto/o www.ttthbooooooooooooooooooooccc -aa. haaaggnniinieeeeeeeeeeeeeeaeenaa e- imaqa
INTRODUCTION
Climate change one of the most pressing global challenges today, with profound impacts on ecosystems, economies, and societies. In recent years, Large Language Models (LLMMs) have gained significant interest in climate science (Thulke et al.) [0244 Nguyen et al.) 2024] Cao et al. (2024) due to their potential to transform climate predictions and enable applications in climate policy analy- sis, environmental decision-making and public education. By improving LLMs' understanding of climate science, can empower stakeholders to make informed decisions, develop actionable solu- tions, and foster broader awareness of climate issues. However, while LLMs are powerful, they often fall short when it comes to answering technical questions requiring high precision such Whar is the net effect of Arctic stratus clouds on the Arctic climate? Even advanced models like GPT-4 ex- hibit epistemological inaccuracies in Climate Question-Answnn Answering (QA) tasks (Bulian al. 2024 raising concerns about their reliability in scientific workflows.
This highlights the need for domain-specific evaluation framework to assess the quality and validity of outputs generated by these models. Current benchmarks for Large Language Models (LLMs) pre- dominantly focus on linguistic accuracy or general factual correctness (Bai & Wang 2021), but they fail to address the unique demands of climate science, where factual rigor, domain- edge, and robust reasoning are essential. Although some work has explored the scientific evaluation of LL (Table D. tion generations. address this issue, we develop ClimaGen, an adaptive learning framework for creating benchmarks in collaboration with domain experts to evaluate scientific question-answering models, specifically for climate science but adaptable to other scientific disciplines, shown in Figure This enables achieve a balance between utilizing the efficiency of LLMs and the expertise of domain specialists.
knowl-
they either rely heavily on manual expert input or employ fully synthetic ques-
Using framework, we introduce novel benchmark for evaluating question- answering models in climate science across three scientific QA task forms: multiple-choice, freeform, and cloze. The questions designed with varying levels of complexity, challenging the models to demonstrate range of reasoning abilities from basic factual recall to scientific reasoning and scenario applications.
correspondence Rose Yu & Taylor Kink-Kiikat

---

Published as a conference paper at ICLR 2025
Vector 3B
Evoluster
Retrieved Contactt
Question
Seed Context
Irput Premet
QJt Peins Preposel
Filtened Pairs
Pronpt Assembier
Scientific
Generater
Praiminary
Symtheic Benchmark
Hunan
Evalattors
Principles
Framework
Figure 1: ClimaGen Our proposed Automated Benchmark Creation Framework. The QA gener- ation framework creates synthetic data from seed contexts extracted from graduate-level textbooks using LLMs to generate base-level guestion-answer pairs and evolve them by adding complexities to the same. These are validated by domain experts during the annotation process produce the semi- synthetic benchmark. The evaluator model is trained actively using the human-labeled examples in order completely automate the process.
The benchmark consists of two datasets: ClimaQA-Gold- - an expert-annotated dataset with a total of 566 questions validated by climate scientists, ensuring high scientific rigor, and ClimaQQ-Silke a large-caale synthetic dataset consisting of 3000 questions generated by our framework, providing substantial ground truth data for model fine-tuning at scale. Together, these datasets enable compre- hensive performance assessment of LLMs in climate science, specifically for scientific QA tasks. We evaluate several LLMs on our benchmark under different settings. We observe that most models struggle with reasoning-bes -based multiple-choice questions (MCQs) and Retrieval Augmented Gener- ation (RAG) (Lewis et al. (2000) significantly outperforms Continued Pre-training and Supervised Fine-tuning across different tasks.
In summary, our contributions are as follows:
Creation of publicly releasable datasets: expert-annotated (ClimaQA-Gold) and synthetic (ClimaQA-Silver), along with tailored evaluation metries facilitating both rigorous assess- ment and large-scale fine-tuning on 3 scientific QA task forms: multiple-choice, freeform, and cloze, with varying levels complexity.
Development of generalized adaptive learning framework (ClimaGen) for creating sci- entific benchmarks at scale in collaboration with domain experts for evaluation of natural language
models on scientific accuracy.
Evaluation of state-of-the-ard LLMs on climate science QA tasks, with insights into im- proving scientific accuracy.
Table 1: Comparison of scientific benchmarks. Automated indicates automatic creation, Validated shows expert validation, Multi-task represents multiple task types, and Multi-level represents ques- tions of varying complexity
Dataset
Domain
Source
Size
Automated Validated Multi Task Multi-Level
ScienceQA
Science
Hi-Scl Text
21000
X
Pira2
Ocean
Research
2250
xxx
x
SciQA
Comp Sci
ORKG
2500
Climate Crisis
Climate
None
20000
SeiQAG-24D
Science
Research
8531
ClimaQA-Gold
Climate
Grad Text
566
ClimaQA-Silver
Climate
Grad Text
3000
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
ScienceQA (Lu al., 2022) contains a vast collection of multimodal MCQs manually curated from high school textbooks. Pira2 (Pirozelli et al., 2024) consists of expert-created questions derived from research articles focused on oceans, the Brazilian coast, and climate change. The creation of these benchmarks required substantial manual effort. SciQA (Auer et al. 2023) innovatively gener- ates freeform QA pairs by leveraging hand-crafted queries on the Open Research Knowledge Graph (Jaradeh et al. 2019) primarily drawing from computer science literature. Although these pairs are factually accurate, they do not include an automatic evaluation method for generated responses. Climate Crisis QA (Zhu & Tiwari 2024) and SciQAG-24D (Wan et al. 2024) explore synthetic data generations using Large Language models. However, their approaches are prone to suffer from hallucinations and lack of scientific validity. To address this, we introduce gold-standard dataset, rigorously validated by domain experts, alongside a large-scale silver dataset whose generation pro- cess was guided by these expert validation labels. Moreover, existing benchmarks generally focus on a single QA format and lack scientifically aligned evaluation metrics. Our benchmark contains ques- tions of three distinct scientific QA task forms at varying levels of complexity. along with evaluation metrics tailored to them. Table presents comparison of various scientific benchmarks.
Previous work on automated MCQ generation has focused on selecting keywords and generating distractors based on contextual information. (A. Nwafor & E. Onyenwe (02II) utilize traditional NLP techniques, such as TF-IDF, for keyword extraction, while (Mehta et al. 2021 leverage BERT Devlin 2018) for summarization and keyword extraction, employing WordNet (Miller 1995) to generate distractors. (Das et al. (021) applies RAKE (Rose et al.] (2010) for keyword extraction and clustering methods for distractor generation. Other approaches, such as utilizing dependency trees (Afzal & Mitkov 2014). have also been explored for MCQ creation. These methods typically focus on generating MCQs with single-word answers. However, the recent advancements in L.MMs have enabled the creation of more complex MCQs with longer, detailed answer choices. (Meißnner et al.) 2024 demonstrate the automated generation of self-assessment quizzes using LLMs, while (Hang et al. 2024) explore self-refining prompting techniques for improved MCQ generation. Recent studies, including (Olney) (2023) and (Doughty et al. (202)) suggest that LLMs can generate MCQs comparable to those created by humans, though (Grévisse et al., (2024) emphasize the importance of human oversight to ensure the quality and pedagogical relevance of these questions.
3 CLIMAQA CLIMATE QUESTION ANSWERING BENCHMARK
The ClimaQA benchmark is built questions generated from graduate-level climate science text- books, ensuring alignment with the precise terminology and complex theories of the field. These textbooks provide a reliable source for generating both the expert-validated ClimaQA-Gold dataset and the synthetic ClimaQA-Silver dataset. By leveraging textbook content and combining it with expert review, ClimaQA facilitates rigorous evaluation and fine-tuning of LLMs across freeform, multiple-choice and cloze question-answe answering tasks in climate science. Our expert-validated dataset, ClimaQA Goldd ensures that the evaluation questions are accurate, relevant, and reflect the current understanding of climate science.
3.1 SCIENTIFIC QUESTION ANSWERING
To thoroughly evaluate model's 's ability to handle scientific questions, we create our benchmark dataset focus on the different complexities of scientific reasoning. The aim is to test the model's ability to engage with scientific concepts at different levels of understanding and scenario applica- tion.
Our benchmark consists of questions of three levels of complexity. The first level involves basic questions designed to test straightforward factual understanding. The second level introduces rea- soning, requiring the model to connect multiple scientific facts or principles. The third level involves hypothetical scenarios, testing the model's ability to apply scientific knowledge in unseen contexts. These questions challenge the model's 's scientific reasoning in different ways, from knowledge recall to advanced reasoning and problem-sole solving in dynamic contexts. A question from each level of complexity is shown in Figure 2 as example.
3

---

Published as a conference paper at ICLR 2025
Base
Question What is crucial factor to when collecting data for calibration purposes?
Options
a) Using different solution sources for each data set.
b) Consistency equipment setup and data collection procedures.
c) Changing the calibration locations frequently to avoid bias.
d) Varying the nebuliser type each calibration date.
-
Reasoming
Question Why consisteney equipment setup and data collection procedures considered crucial factor for collecting data for calibration purposes?
Options
a) It ensures that the calibration process completed faster.
b) It helps in minimizing errors and maintaining reliable and accurate measurements.
c) It helps identifying outliers the data sets more effectively.
d) It allows for easy integration of new equipment without affecting the calibration
results.
Answer
Hypothetical Scenario
Question How might the calibration accuracy be affected if the driers and DMA were changed
between different calibration sessions?
Options
a) The calibration accuracy would deteriorate due to inconsistent conditions.
b) The calibration accuracy would fluctuate depending type of nebuliser used.
c) The calibration accuracy would remain unaffected by the change in equipment.
d) The calibration accuracy would improve due the variability introduced.
Answer
Figure 2: Examples of Question Evolution. The first is the initial version of the generated question. The second is the enhanced version of the question that requires scientific reasoning answer. The third is the modified version of the question that involves hypothetical scenario. The contexts from the textbook data were used during the question evolution.
The questions come in three different task forms, demonstrated in Figure 3
MCQ:
The model selects correct answers from predefined options, assessing its factual accuracy and decision-making under constrained conditions.
Freeform: The model generates detailed, structured responses, testing its ability to reason logically and produce scientifically sound explanations.
Cloze: The model fills in blanks with appropriate scientific terms, evaluating its contextual understanding and use of Nomain-specite vocabulary.
Together, the benchmark shown in Table provides a robust framework for evaluating an LLM's proficiency in scientific reasoning, critical thinking, and applying knowledge in unseen scenarios.
3.2 EVALUATION METRICS
Although assessing multiple-choice question-answering is relatively simple, the other two tasks present more challenges. To address this, we propose and validate the following evaluation met- rics for freeform and cloze question-answering. A more detailed study to demonstrate the robustness these metrics can be found in the Appendix A.2.3 We use these metrics to report experimental results in Section
4

---

Published as a conference paper at ICLR 2025
Table 2: Contents of the ClimaQA dataset. Both ClimaQA-Gool and ClimaQA-SSllve include
lask-forms with varying levels of complexity for MCQ and Freeform.
Dataset
Task
Base
Reasoning Hypothetical
Total
MCQ
126
72
47
245
ClimaQA-Gold
Freeform
54
52
55
161
Cloze
160
MCQ
501
264
235
1000
CliveQA-Silver Freeform
507
241
252
1000
Cloze
1000
Free Form
Question What is the purpose of measurements at monitoring sites in relation to air pollutents acid rain?
Answer Measurements at monitoring sites were initially made to demonstrate long-
transport pollutants and inform policies mitigate acid rain associated damage to ecosystems and infrastructure.
Multiple Choice
Question Nhy wet deposition more efficient when the aerosol concentration high?
Options
a) Since high aerosol concentration reduces the aerosols spend the atmosphere.
b) As there are more aerosol particles available for removal by precipitation processes.
c) Due the larger size aerosol particles facilitating easier capture by
precipitation.
d) Because high aerosol concentration leads to increased collision and coagulation rates.
Answer
Cloze
Question The thickness of the layer between any two pressure surfaces is proportional to the mean virtual temperature of the layer, leading to direct relationship between the change <blank> height and the thickness Answer geopotential
intervening layer.
Figure 3: Examples of the three types of scientific question-answering tasks presented our bench- mark
3.2.1 FREEFORM QA
Various metrics are employed evaluate sentence similarity, ranging from surface-level compar- isons to deeper semantic analysis. Lexical metrics such as BLEU (Papineni et al. 2002). ROUGE (Lin 2004), and METEOR (Banerjee Lavie 2005) focus on exact word n- n-gram matching, ren- dering them useful for tasks where token overlap is crucial, such as machine translation. In contrast, semantic metrics like BERTScore (Zhang et all 2019), Word Mover's Distance (WMD) (Huang al. 2016), and Sentence-BERT (Reimers) 2019) more advanced, capturing the meanings of sentences through embeddings. These metrics are better suited for tasks that necessitate an under- standing of meaning, such as paraphrase detection. However, they may not adequately assess factual accuracy.
To measure the factual accuracy of generated answers relative to reference answers, propose the use of factual entailment classifier, reporting the confidence level as the factual accuracy score. Instruction-tunel models, such as GPT-4, have demonstrated superior performance textual

---

Published as a conference paper at ICLR 2025
tailment tasks and have shown the ability to generalize across various datasets (Sanyal et al. 2024). We employ the GPT--o-mini model with the prompt below for factual entailment. This method achieved 81 % zero-shot classification accuracy on the Climate-Fever dataset (Diggelmann et al, 2020) indicating the ability measure factual accuracy.
You are climate expert who annotates whether a given claim either SUPPORTS or REFUTES the presented evidence. You will be provided with the following input:
Evidence: (evidence)
Claim: (claim)
Respond with only one word: SUPPORTS if the claim supports the evidence and REFUTES
otherwise.
To use this for scoring freeform QA, the reference answer was used as the evidence and the generated answer was used as the claim. The confidence score was computed by applying sigmoid smoothing to the logit scores, with the lemperature parameter set to T 5. Note that the choice of does not alter the score trend; it was selected to optimally scale values between 0 and 1. If l, and 4, are the logit scores for SUPPORTS and REFUTES respectively, then
Factual Accuracy = SigmoidSmooth
1a +
Overall, we report three metrics for freeform question answering (QA): BLEU, BERTScore, and Factual Accuracy to evaluate different aspects of the generated answers.
3.2.2 CL.OZR QA
Performance on this task is typically evaluated using the exact match metric. However, this approach has limitations due to the existence of multiple correct answers. gen- erated answer may differ from the reference answer while remaining contextually and se- mantically valid; for instance, while 'point' and 'temperature" are semantically distinct terms, they can be contextually similar within phrases like 'freezing point' and ffrezzing temperature." This illustrates that semantic relationships can depend heavily on context. To address this challenge, we introduce metric that captures the semantic similarity between the generated answer and the ground-truth an- swer for a more nuanced assessment of model utilize context window to extract two phrases: one with the (blank) replaced by the reference answer and the other with the generated answer. The se-
0.6
Wrong
performance. Specifically,
0.1
Mean Phrase Similantty
066
mantic similarity is measured using cosine sim-
ilarity between the Universal Sentence Encoder Cer 2018) embeddings of these phrases.
0.5
To evaluate the robustness of this approach, we synthetically generated 219 cloze ques- tions with our framework (described in Section 4p. which were answered by the GPT-4o-mini model. We collected 32 questions where the generated answers did not exactly match the reference answer. These answers were then labeled as wrong or correct by domain ex- perts based on scientific and contextual accu- racy. We plotted the average cosine similarities of phrases for these questions shown in Figure 4 concluding that context window of size most effectively differentiates between correct and incorrect answers. This configuration yields the maximum difference scores while maintaining sufficiently high scores for correct answers. The cosine similarities were subsequently rescaled to emphasize these differences. If and e2 are the embeddings of the respective phrases mentioned above, then
Context Window Around <blank> Token
Figure 4: Mean Phrase Similarity for Correctly Answered and Incorrectly Answered Cloze Ques- tions
Phrase Similarity (CosineSimillaityyee,. 0.5)

---

Published as a conference paper at ICLR 2025
We report two metrics for cloze question answering (QA): Exact Match and Phrase Similarity.
4 CLIMAGEN AUTOMATED BENCHMARK CREATION
In this section, we describe ClimaGen as shown in Figure [ the framework used to create the ClimaQA dataset from Climate Textbooks. By leveraging RAG and prompt engineering, sys- tematically generate and refine questions of varying complexity levels. Domain expert annotators ensure quality, producing a semi-synthetic benchmark for the evaluation of Al systems in complex scientific inquiry. Additionally, we automate annotation by fine-tuning an LLM on human-laheled data, enabling the creation of large-scale synthetic dataset for fine-tuning tasks. The techniques discussed here could be generalized to aid in the semi-automatic production of benchmarks for other scientific fields aswell.
4.1 TEXTBOOK DATASET
LLMs typically pre-trained extensive general internet data, which often contains noise and misinformation. This limitation is particularly significant in fields like climate science, where a precise understanding of specialized terminology and concepts is crucial. To evaluate LLMs' profi- ciency in climate science, we employed graduate-level climate science textbooks as a reliable source of specialized knowledge, providing accurate scientific information that better represents the tech- nical terms and nuanced theories integral to this discipline. We collected 18 textbooks (Table 4 that broadly represent mixture of graduate and expert literature on the physical climate with particular focus on the role of aerosol in the climate system one of the critical sources of uncertainty in cli- mate projections. The content was extracted and preprocessed ensure cleanliness and relevance, making it suitable for downstream applications such as benchmark creation, continuous pre-training, and RAG. held-out set of textbooks (Figure carefully selected to represent varying levels of technical and qualitative depth across broad range of key topics in climate science, was utilized for the benchmark creation process.
4.2 QA GENERATION FRAMEWORK
Our QA generation pipeline begins by selecting a random seed 2000-character context chunk from the collected textbook data stored in vector database. Additional context chunks are retrieved based on cosine-sine simiiartiillitt scores, ensuring relevant information from multiple sources is included. These chunks are then augmented and passed to the generator L.LM for question-answed (QA) generation. Question generation principles, inspired by (Doughty al.) 2024), guide the prompt formulation, focusing on creating high-quality stems and distractors for MCQs, as well as refining questions by adding complexity as described in Appendix A.4
We used GPT-3.5-turbo as the generator model in our experiments. The model generates base- level questions and evolved variants with increasing complexity, such as multi-step reasoning and hypothetical scenarios, ensure diverse and comprehensive question set as shown in Figure 2 However, approximately 25% of the multiple-choice questions were incorreetly answered by the same model even when the contexts were provided, often due to scientifically inaccurate question- answer pairs, indicating self-inconsistency and uncertainty in the generation process.
While QA pairs ments intended for cloze questions were also generated, from which the scientific term to be masked would be chosen during the annotation phase. After generation, the questions undergo preliminary screening with both handcrafted and LLM-based (self-inconsisteency) audits to filter out potentially invalid QA pairs. The refined set of QA pairs is then passed to the annotation phase for further validation.
generated for multiple-choice and freeform questions, plain scientific state-
4.3 DOMAIN EXPERT ANNOTATION
One key challenge with synthetic data is ensuring its distribution closely mirrors real-world data, as deviations negatively impact downstream tasks like fine-tuning and evaluation. This issue arose when generating scientific questions with GPT-3.5-turbe which sometimes produced inaccurate or imprecise data probably due to limited understanding of domain-specific terminology.
7

---

fine-tuned separately for Published as a conference paper at ICLR 2025
To mitigate this, we developed an interactive web application that enables climate scientists to re- view and annotate the generated questions. For freeform and MCQs, the scientists validated the correctness of the content, while for cloze questions, they selected which scientific term mask, ensuring alignment with scientific standards. They also identified common reasons for rejecting generated QA pairs during validation, providing valuable insights into improvement of the QA gen- eration framework as discussed in Appendix [A.5. By combining human expertise with AI, curated 245 freeform, 161 MCQ, and 160 cloze questions, forming the ClimaQA-Gole dataset, re- viewed and validated by domain experts.
4.4 AUTOMATED ANNOTATION
To fully automate the review and annotation process, we develop an evaluator model by fine-tuning an LLM (GPTT44--inn) on expert-annotated data to validate and refine generated question-answer pairs. This removes the need for human intervention and enables scalable generation high-quality scientific question-answer pairs, especially for data-intensive tasks like fine-tuning.
Building on (Zhang et al. 2023) and (Zhang et al. 2024), which demonstrate that uncertainty-base active sampling improves supervised fine-tuning with limited labeled data, apply similar ap- proach. The evaluator model is fine-tuned as a classifier to label QA pairs as valid or invalid based on the given context descrihed in Appendix A.5.2 Uncertainty is measured by the classifier con- fidence scores, and samples with confidence above 0)85 are dropped with 50% probability to ensure learning from more representative examples. The evaluator models both MCQs and freeform questions
We observed that around 85% of multiple-choice questions (MCQs) and around 90% of freeform question-answers (QAs) were valid, indicating high-quality question generation. Experiments across different train-test splits show that the evaluator models enhance the quality of the generated MCQ question set by 10% and the Freeform question set by 5% as shown in Appendix [A.5.2 Additionally, we fine-tune a separate model to mark scientific terms from given statements as shown in Appendix A.5.3 automating the cloze annotation process. Using this framework, we generated 1000 freeform QAs, 1000 MCQs, and 1000 cloze questions, collectively forming the ClimaQA-Silve dataset, pro- duced synthetically at scale without manual intervention.
5 EXPERIMENTS
We aim to investigate the effectiveness of various adaptation techniques on this fine-grained scien- tific benchmark. Fine-tuning on raw text data within a target domain is common approach, and we seek to evaluate its effectiveness for addressing deep scientific questions. In addition, we evaluate other techniques, such as in-context learning and retrieval augmentation.
5.1 EXPERIMENTAL SETUP
We evaluate different families of LLMs on our benchmark. We use TogetherAl for performing inference on open source models like gemma-27b (Team al. 2024b). 1lama377b<Dubey al. 2024). and mixtral-8x22b (Jiang et al. 2024). We also evaluate OpenAl's (Achiam et al. 2023 gpt-3.5-turbo and gpt-4o.
We evaluate each of these models in 3 settings default, few-shot prompting (FS) 2020 and Retrieval Augmented Generation (RAG) (Lewis et al. 2020). For the MCQs, the models were prompted to output single letter representing the correct option, and the top-most token was chosen as the answer. For Freeform QA, the models were prompted to output concise answers with maximum of 2 sentences. For Cloze QA, the models were prompted to output single scientific word that best fits the blank with respect to the context around it.
We conduct further pre-training on graduate-level textbook data for both the LLaMA3. 1-8-Inttte-- and Mistral-7B- -0.--003-Inntruct (Jing et al. 2023) models. This pre-training was based on 13 distinct graduate textbooks that were not part of the muestion-generationn process. The objective was to enhance the model's climate knowledge without directly exposing it to the specific sources used for question generation, thereby reducing the risk of data contanination.
8

---

Published as a conference paper at ICLR 2025
Table 3: Performance analysis of various state-of-the-arr LLMs on MCQs and Cloze QA. While source represents the of books used for QA generation, held-oul represents the remaining set of books. Bold marks the max within model's variants and green highlights the overall column max.
Model
MCQ
Cloze
Base
Reason
Hypo
Overall
EM
PS
gemma-27b
81.75
72.22
82.98
79.18
49.38
0.87
gemma-27b-fs
80.95
77.78
82.98
80.41
52.50
0.88
monmma-27-rag--oureee
90.48
80.56
78.72
85.31
56.88
0.90
gemmaa277brrag-held-ouu
79.37
76.39
78.72
78.37
45.62
0.85
gpt-3.5-turbo
74.34
69.91
74.47
73.06
43.12
0.81
gpt-3.5-turbo-fs
76.98
74.54
76.60
76.19
43.75
0.78
gpt-3.5-turbop
3pt-3.5-turbo-rag-ouurre
80.42
80.09
77.30
79.73
68.75
0.92
1p--3---tubbo-rrg--helooot
70.63
71.30
69.50
70.61
39.38
0.81
gpt-4o
86.77
86.11
82.27
85.71
53.12 0.88
gpt-4o-fs
87.83
87.50
80.85
86.39
56.25 0.89
gpt-4o-rag-souree
95.77
91.67
86.52
92.79
71.88 0.94
gpt-4o-rag-helee
82.80
80.56
81.56
81.90
50.62 0.88
llama3-70b
84.92
80.56
82.98
83.27
38.75
0.82
llama3-70--is
82.54
81.94
82.98
82.45
48.12 0.85
Mamaa-70b-rag-so00
92.06
84.72
87.23
88.98
63.12 0.91
llama3-7000
1ama3--70-rag-held-aa
80.95
76.39
85.11
80.41
43.75 0.84
mixtral-8x226
80.16
79.17
80.85
80.00
35.62
0.75
mixtral-8x221 -x2tbffs
80.95
81.94
80.85
81.22
45.00
0.83
mixtral-8x22b-
90.48
80.56
76.60
84.90
45.00
0.78
mixxral-8Kx222--rg-helddd
80.16
73.61
74.47
77.14
28.12
0.65
mistral-7b
64.29
63.89
82.98
67.76
17.50
0.74
mistral-7b-fs
65.08
66.67
76.6
67.76
33.12
0.80
mistral-7b-rag-soune
92.86
84.72
85.11
88.98
57.5
0.88
mistral-7b-rag-helld-or
66.67
62.5
74.47
66.94
22.50
0.76
mistral-7b-op-held-out
72.22
62.5
74.47
69.8
21.25
0.76
mistral-7b-ft-silver
73.81
75.00
80.85
75.51
0.83 41.88
llama3.--bb
76.98
62.50
65.96
70.61
26.25
0.77
1lama3.1-8b-f
76.19
72.22
76.60
75.10
38.75
0.82
Ilama3. 1amaa.1-8b-rage
1-8b-rag-source
94.44
83.33
89.36
90.2
72.50
0.92
76.19
66.67
76.6
73.47
36.25
0.81
1lama3.1-43 1-8b-cp-held-out
77.78
75.00
72.34
75.92
30.00
0.77
1lama3.1 1-8b-ft-silven
74.6
70.83
72.34
73.06
51.25
0.85
gemini-1.5-
82.54
73.61
87.23
80.82
50.62
0.88
Siiiii-----ffash--o------- -ouuure
88.1
75.00
80.85
82.86
51.88 0.89
70.63
70.83
78.72
72.24
46.88
0.87
and Nistral-7B-v003-Innstuut on the ClimaQA- Silver dataset, which contains all three forms of MCQ, Freeform, and Cloze, in different complexity levels. We then evaluate the impact of this task-specific fine-tuning by assessing the models' per- formance the ClimaQA-Gold dataset. The details of the continued pre-training and fine-tuning procedure are explained in Appendix AA.3) We also leverage Gemini 1.5 (Team et al.) 2024a), with a context window of up to million tokens, pass entire textbook in context and answer questions based on that. Finally, we evaluate models on the ClimaQA-Silverr dataset and analyze its potential differences from ClimaQA-Gold Appendix A.2.2
Additionally, we fine-tune
5.2 RESULTS
We report the performance various models across different QA forms and complexities. Table shows the results for the MCQ and Cloze form questions, and free-form results are demonstrated in Appendix Table 5 We observe that most models struggle with reasoning MCQs compared to base and hypothetical questions. While some models perform poorly reasoning and hypothetical

---

Published as a conference paper at ICLR 2025
MCQs, they tend to generate stronger responses for the same type Freeform Questions, indicating improved performance when reasoning is emphasized over factual recall similar to observations of Chain of Thought experiments (Wei et al. 2022], as shown Figure
reasoning
recsoning
reasoning
reasoning
gi-3 Sturbo
germoett
90
04
0.00
0.75
8
base
sese
base
base
hypotrefital-
hypothotirat
hypotheftrat
hypoteefirat
(a) Accuracy
(h) BLEU
(c) BERTScore
(d) Factual Accuracy
Figure 5: Analysis of various LLMs under default setting different tasks and different complex- ities. The first figure shows accuracy of models in the MCQ task while the others show different metrics under the Freeform task
We examine the impact of providing relevant context through RAG when answering questions. We explore retrieval scenarios: one where the model retrieves from 13 books that were not used to generate the questions (rag-held-out), and another where it retrieves from the 5 books that the questions were derived from (rag-source). Retrieval from source textbooks consistently enhances performance across all tasks. For Gemini 1.5, we identify the book corresponding to the most relevant retrieved chunk under both the source and held-out settings, and include as in-context information to answer the question. Additional context from source books yields a slight improve- ment over the no-context baseline. However, both RAG and long-context scenarios, incorporating held-out books usually reduces performance, likely due to irrelevant distracting content.
Continued pertaining (cp-held-out) on the graduate textbooks leads to improved performance in both MCQs and Cloze Silver dataset further enhances performance, often producing the best results after RAG on the source textbooks (rag-source) in most scenarios. Furthermore, Few-shot prompting yields marginal improvements in most cases.
tasks. Additionally, fine-tuning (ft-silver) on the ClimaQA-
Finally, We observe that the BLEU and BERTScore metrics slightly biased towards the model that was used for QA-generation (gpt-3.5-turbo while this is not seen in the proposed Factual Accu- racy metric 5 Overall, GPT-4o dominates across tasks, demonstrating superior performance com- pared to other models in this evaluation set.
6 CONCLUSION
The ClimaQA benchmark offers a comprehensive framework for evaluating language models in climate question-ansserrngg addressing critical aspects such as reasoning, factual accuracy, and understanding of scientific terminology. By incorporating freeform, multiple-choice, and cloze task forms with different levels of complexity, the benchmark rigorously tests models across different dimensions of scientific inquiry. Furthermore, the use of advanced metrics, such as factual accuracy for freeform tasks and phrase similarity for cloze tasks, can provide a more nuanced assessment of model performance.
The automated benchmark generation framework (ClimaGen) integrates domain-specitfs textbooks and natural language understanding of LLMS along with human expertise to produce high-quality QA data at scale. However, the benchmark's reliance on only five textbooks limits the diversity of contexts, and the small size of the annotated dataset constrains the effectiveness of automated annotation. Addressing these limitations with a broader corpus and expanded annotation data will improve future benchmarks.
While models like GPT-4o performed well on reasoning- basedd tasks, the overall performance of models highlights the ongoing challenge of achieving consistent scientific accuracy. In conclusion, ClimaQA sets new standard for evaluating scientific question-answenig models, providing a foun- dation for future advancements in AI-driven climate research.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILIIT STATEMENT
The ClimaQA dataset, both gold and silver, publicly available at Hugging F While we cannot release the scraped textbook data due to copyright restrictions, the references to all textbooks used are provided in the appendixA.]]] allowing for reconstruction of this dataset. Our complete codebase, including data generation pipeline, web-Ul and model evaluation scripts, is available in our GitHub repository ?| The ClimaGen framework is fully reproducible; however, training the evaluator models requires domain expert inputs, which may introduce variability. Furthermore, reproduction of all parts of our framework requires appropriate API keys for external services.
ACKNOWLEDGEMEN
This work was supported in part by the U.S. Army Research Office under Army-ECASE award W91 DEI--N0F--000333 the U.S. Department Of Energy, Office of Science, TARPA HAYSTAC Program, and NSF Grants #2205093, #2146343, #2134274, CDC-RFA-FT-23-0066 DARPA AIE FoundSci and DARPA YFA. We thank Sophie Wynn, Varan Madan, and David Vishny for providing expert validation of the ClimaQA dataset.
REFERENCES
Chidinma Nwafor and Ikechukwu E. Onyenwe. An automated multiple-choice question gener- ation using natural language processing techniques. International Journal Natural Language Computing, 10(02):1-10, April 2021. ISSN 2319-4111. doi: 10.5121/7jll.222111000.. URL
htttp://dx: doi www.ooggg....... 5111::.1111iiinni.
10201
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303. 087774 2023.
Naveed Afzal and Ruslan Mitkov. Automatic generation of multiple choice questions using dependency-based semantic relations. Soft Computing, 1807):1299-2281, 2014. doi: 10.1007/ 500500-013-1141-4 URL http::
2ttp://000000000000000001----1-0 114-111--41
Sören Auer, Dante A. C. Barone, Cassiano Bartz, Eduardo G. Cortes, Mohamad Yaser Jaradeh, Oliver Karras, Manolis Koubarakis, Dmitry Mouromtsev, Dmitrii Pliukhin, Daniil Radyush, Ivan Shilin, Markus Stocker, and Eleni Tsalapati. The sciqa scientific question answering benchmark for scholarly knowledge. Scientific Reports, 13(1):7240, 2023. doi: 10.1038/ 000--002333333 -00-5380----33--2-222222-----0--
4411598-023333007-
Yang Bai and Daisy Zhe Wang. More than reading comprehension: A survey on datasets and metrics of textual question answering. arXiv preprint arXiv:2109.12264 2021.
Satanjeev Banerjee and Alon Lavie. Meteor:
An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
P.A. Baron and K. Willeke. Aerosol Measurement: Principles, Techniques, and Applications. A Iiley-Intersciencee publication. Wiley, 2001. ISBN 9780471356363 URL google. www.books.booooccooooooooooooe.o http://boookkk.ttt3saa.aaaa
Tom B Brown, Language models are few-shot learners. arXiv preprint arXiv:2005.14166 2020.
Jannis Bulian, Mike Schäfer, Afra Amini, Heidi Lam, Massimiliano Ciaramita, Ben Gaiarin, Michelle Chen Hübscher, Christian Buck, Niels G. Mede, Markus Leippold, and Nadine StrauB. Assessing large language models on climate information, 2024. URL https: wtt.:httpsss/aate /tapppss////x.....// orgg/
abs / 221/2331.00022 02932
htt.ass//uggiirrree..c///sssssss https://github.rt ht
imaQA
imaga

---

Published as a conference paper at ICLR 2025
Charles Cao, Jie Zhuang, and Qiang He. LLM-assisted modeling and simulations for public sector
decision-making: Bridging climate data and policy insights. In AAA1-2024 Workshop Public Sector s.Mss Algorithmic and Sociotechnical Design, 2024. URL /
Sttt/Forumm11ddu55555333
K.S. Carslaw. Aerosols and Climate, Elsevier, 2022. ISBN 9780128231722. URL https://
books gwwgle..oog.......o com bbooks/ooksss 7
D Cer. Universal sentence encoder. arXiv preprint arXiv:/800. /803. 1175, 2018.
Bidyut Das, Mukta Majumder, Santanu Phadikar, and Arif Ahmed Sekh. Multiple-choice question generation with auto geteeeeeeeteed distractors for computer-assistie educational assessment. Multi- media Tools and Applications, 80221)31107-31925 2021.
Carlos De Wasseige. Martin Tadoum, Eba' a Atyi, Charles Doumenge, et al. The forests of the congo basin-forests and climate change, 2015.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint wxXiv/810.0088000 2018.
Thomas Diggelmann, Jordan Boyd-Graber, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippold, Climate-fever: A dataset for verification of real-world climate claims. arXiv preprint arXiv:2012.00614 2020.
Jacob Doughty, Zipiao Wan, Anishka Bompelli, Jubahed Qayum, Taozhi Wang, Juran Zhang, Yujia Zheng, Aidan Doyle, Pragnya Sridhar, Arav Agarwal, et al. A comparative study of ai- te-geenraeed (gpt-4) and human-crafted mcqs in programming education. In Proceedings of the 26th Aus- Iralasian Computing Education Conference, pp. 114-123, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
L.M. Gelfand and S.V. Fomin. Calculus of Variations. Dover Books on Mathematics. Dover Pub- lications, 2012. ISBN 9780486135014, htttps://bookss google. cttpe//o.cooooooos.c.... [eC7AQAAQRAJ]
Christian Grévisse, Maria Angeliki S Paviou, and Jochen G Schneider. Docimological quality anal- ysis of lIm-generated multiple choice questions computer science and medicine. SN Computer Science, 5(5):636, 2024.
Ching Nam Hang. Chee Wei Tan, and Pei-Duo Yu. Meqgen: A large language model-driven mcq generator for personalized learning. IEEE Access, 2024.
D.L. Hartmann. Global Physical Climatology. International Geophysics. Elsevier Sci- ence, 2015. ISBN 9780080918624. URL htttps:/bbooks..... wtw.:oool....o/ooooo/ososaa com/bookssid
RsSCEAAAQBAJ
Heintzenberg and R.J. Charlson. Clouds in the Perturbed Climate System: Their Relation- ship to Energy Balance, Atmospheric Dynamics, and Precipitation. Strüngmann Forum reports. Mass., 2009. ISBN 9780262012874. URL bttpp::/bbokks www.ss.ooooo........o.o.... cow/books ?id=
VERUPAAACAAJ
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. ArXiv, abs/2106.09685. 2021. ltttps:::///ai http///pp..a......mmmma/mmmmam.. semant www.holar.cc.oooom htt.chhooa...og//ooppatttaaa5aa. 2354558009
Gao Huang, Chuan Guo, Matt J Kusner, Yu Sun, Fei Sha, and Kilian Q Weinberger. Supervised word mover's distance. Advances in neural information processing systems, 29, 2016.
P. Imkeller and J.S. von Storch. Stochastic Climate Models. Progress in Probability. Birkhäuser Basel, 2012. ISBN 9783034882873. URL htttp:///booee //books...../ google. com/books?id A8bZBWAAQBAJ
12

---

Published as a conference paper at ICLR 2025
M.Z. Jacobson. Fundamentals of Atmospheric Modeling. Fundamentals of Atmospheric Modeling. Cambridge University Press, 2005. ISBN 9780521548656, URL httpss://bokss: goog e.
Mohamad Yaser Jaradeh, Allard Oelen, Kheir Eddine Farfar, Manuel Prinz, Jennifer D' Souza, Gábor Kismihók, Markus Stocker, and Sören Auer. Open research knowledge graph: next generation infrastructure for semantic scholarly knowledge. In Proceedings of the 10th international confer- ence on knowledge capture, pp. 243-246, 2019.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anr Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William Sayed. Mistral 7b, 2023. URL https:
//arxiv.org/abbl 2330 06825
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.040S5 2024.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmentee genera- tion for pnowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33: 9459-9474, 2020.
Chwn Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
U. Lohmann, Lüönd, and F. Mahrt. An Introduction to Clouds: From the Microscale Climate. Cambridge University Press, 2016. ISBN 9781316586259, URL httpttts////oks..gtg google.
com/bookssid PDDDAAAAAOO
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022.
A.J. Majda and J. Harlim. Filtering Complex Turbulent Systems. Cambridge University Press, 2012. ISBN 9781107016668. URL lattts://bbbokks google www.bookeeeee.o.oooooo bidp//ooooksidJn. Pid=JEuJ
SoshBUC
Pritam Kumar Mehta, Prachi Jain, Chetan Makwana, and CM Raut. Automated mcq generator using natural language processing. Proceedings the the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applicutions, pp. 284-290, 2021.
Niklas Meißner, Sandro Speth, Julian Kieslinger, and Steffen Becker. Evalquiz-lim-based auto- mated generation of self-assessment quizzes in software engineering education. In Software En- gineering im Unterricht der Hochschulen 2024, pp. 53-64. Gesellschaft für Informatik eV,, 2024.
George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11) 1): 39-41, 1995.
Vincent Nguyen, Sarvnaz Karimi, Willow Hallgren, Ashley Harkin, and Mahesh Prakash. My climate advisor: An application of NLP in climate adaptation for agriculture. In Dominik Stammbach, Jingwei Ni, Tobias Schimanski, Kalyan Dutia, Alok Singh, Julia Bingler, Christophe Christiaen, Neetu Kushwaha, Veruska Muccione, Saeid A. Vaghefi, and Markus Leippold (eds.), Proceedings of the 1st Workshop on Natural Language Processing Meets Climate Change (Cli- mateNLP 2024), pp. 27 45, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL httph:://////t http:///aaaanttoooy..ooo / 2024. c wwwmatennlpl......mm
Andrew M Olney. Generating multiple choice questions from a textbook: Llms match human per- formance most metrics. In L.M@@ AIED, pp. 111-128, 2023.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
13

---

Published as a conference paper at ICLR 2025
A.C. Petersen. Simulating Nature: A Philosophical Study of Computer-Simulation Uncertainties and Their Role in Climate Science and Policy Advice, Second Edition. CRC Press, 2012. ISBN 9781466500679. htkl.htts:://oo..
www.book//////ooooooooooooo..oso books (
R.T. Pierrehumbert. Principles of Planetary Climate. Cambridge University Press, 2010. ISBN 9781 139495696 URL
google. wtt.e/eccoooooocccooooooo http:oossss-d---0_U_SSSSSV
Paulo Pirozelli, Marcos M. José, Igor Silveira, Flávio Nakasato, Sarajane M. Peres, Anarosa A. F. Brandão, Anna H. R. Costa, and Fabio G. Cozman. Benchmarks for Pirá 2.0, Reading Com- prehension Dataset about the Ocean, the Brazilian Coast, and Climate Change. Data Intelli-
gence, 6(1):29-633 63, 02 2024. ISSN 2641-435X. doi: 10.1162/dint 00245. URL https: //doi. 21tp//intt__ninnee
N Reimers. Sentence-hert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.1008 10084, 2019.
Stuart Rose, Dave Engel, Nick Cramer, and Wendy Cowley. Automatic keyword extraction from individual documents. Text mining: applications and theory, pp. 1-20, 2010.
Soumya Sanyal, Tianyi Xiao, Jiacheng Liu, Wenya Wang. and Xiang Ren. Are machines better at inference gaps in entailment verification, 2024.
complex reasoning? unveiling abs 22022 03686
URL httphhtts///xxxieeee /atorg/b///////////////oa//eaeer
D. Schultz. Eloquent Science: A Practical Guide to Becoming Better Writer, Speaker, and At- mospheric Scientist. SpringerLink Bücher. American Meteorological Society, 2013. ISBN 9781935704034. URL :estpss//bboks.
J.H. Seinfeld and S.N. Pandis. Atmospheric Chemistry and Physics: From Air Pollution to Cli- mate Change. Wiley, 2016. ISBN 9781119221133. URL https::/bocks... google. com/
books?1d=xopecwAA00
Gemini Team, Petko Georgiev. Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint Xix::2403.055.4 2024a.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models practical size. arXiv preprint w0..v:240.000122 2024b.
David Thulke, Yingbo Gao, Petrus Pelser, Rein Brune, Rricha Jalota, Floris Fok, Michael Ramos, Ian van Wyk, Abdallah Nasir, Hayden Goldstein, Taylor Tragemann, Katie Nguyen, Ariana Fowler, Andrew Stanco, Jon Gabriel. Jordan Taylor, Dean Moro, Evgenii Tsymbalov. Juliette de Waal, Evgeny Matusov, Mudar Yaghi, Mohammad Shihadah, Hermann Ney, Christian Dugast, Jonathan Dotan, and Daniel Erasmus. Climategpt: Towards synthesizing interdisciplinary re- search climate change, 2024. URLhttps: http:///axivvvoo org abs aaaaaaaaa..... 09640.00.99666
James Trembath. Airborne CCN measurements. The University of Manchester (United Kingdom), 2013.
J.M. Wallace and P.V. Hobbs. Atmospheric Science: An Introductory Survey. International Geo- physics. Academic Press, 2006. ISBN 9780080499536. URL bttts::/bbos google.
com womm/bookk221d=#22ww.ooooo
Yuwei Wan, Yixuan Liu, Aswathy Ajith, Clara Grazian, Bram Hoex, Wenjie Zhang, Chunyu Kit, Tong Xie, and lan Foster. Sciqag: A framework for auto-generated science question answering dataset with fine fine-grained evaluation, 2024. URL
R. Webster and M.A. Oliver. Geostatistics for Envirommental Scientists. Statistics in Practice. Wiley, 2007. ISBN 9780470517260. URL lattps://bot google. woo.gle..comboooseeee.oooo wwt.books/bicoccooooooo.oo...... KBwSyvIvNY8C
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi. Quoe V Le, Denny Zhou, et al. Chain-of thouht prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2444 -44337, 2022.
14

---

Published as a conference paper at ICLR 2025
D.S. Wilks. Statistical Methods the Atmospheric Sciences. Elsevier, 2019. ISBN 978001816657770
URL httpt:t//so/okss wttp:///oooooooooeee.... wwt. http:////oooooo/....aacaaaaaaaaa
Linhai Zhang, Jialong Wu, Deyu Zhou, and Guoqiang Xu. Star: Constraint lora with dynamic active learning for data-efficient fine-tuning of large language models. arXiv preprint arXiv:2403.01115. 2024.
Ruoyu Zhang, Yanzeng Li, Yongliang Ma, Ming Zhou, and Lei Zou. Llmaaa: Making large language models as active annotators. arXiv preprint arXiv:2310. 19596, 2023.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluat- ing text generation with bert. arXiv preprint arXiv:1904, 09675, 2019.
Hongyin Zhu and Prayag Tiwari. Climate change from large language models, 2024. URL https:
11985
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 TEXTBOOK DATASET
Table 4: List of climate textbooks used in this work along with the number of pages and the number of context chunks. The extracted data was preprocessed omit figures and tables. The content of each page was then split into overlapping context chunks of around 2000 characters.
Textbook
Pages
Chunks
Aerosol Measurement BBaron & Willeke 2001
937
1813
Aerosols and Climate (Carslaw 2022)
854
1766
Airborne CCN Measurements rembath 2013)
268
333
An Introduction to Clouds (Lohmann et al. (0066)
377
606
Atmospheric Chemistry and Physics (Seinfeld Pandis 2016)
1 1127
1789
Atmospheric Science (Wallace & Hobbs 2006)
495
1085
Calculus of Variations (Gelfand Fomm 2012)
239
266
Clouds in the Perturbed Climate System Heintzenberg & Charlson
2009
598
1060
Eloquent Science (Schultz 2013)
420
725
Filtering Complex Turbulent Systems MMjja & Harlim 2012)
358
475
Fundamental of Atmospheric Modeling (Jacobson 2005
826
1204
Geostatistics for Environmental Scientists (Webster & Oliver 2007
327
457
Global Physical Climatology (Hartmann) 2015)
481
715
Principles Of Planetary Climate (Prerrehumberi 2010
468
1096
Forests and Climate Change (De Wasseige et al. 2015)
119
250
Simulating Nature (Petersen 2012)
205
386
Statistical Methods in the Atmospheric Sciences (Wilks)
2019
806
1597
Stochastic Climate Models (Imkeller & von Storch 2012
411
594
To assess the proficiency of LLMs in climate science, we utilized graduate-level climate science textbooks as a reliable source of specialized knowledge. These textbooks were selected for their ac- curate and comprehensive representation of the technical terminology and nuanced theories integral to the field. The collection was curated from the virtual bookshelf of a professor in atmospheric physics and broadly represents mixture of graduate and expert textbooks the physical climate, with a particular focus on the role of aerosol in the climate system (which provides one of the key uncertainties in climate projections). The complete list of textbooks is provided in Table 4
Atmospheet Science
22.2%
Aerosols and Climate
51.3%
1.0%
Calculus of Variations
16.5%
9.1%
An Introduction to Clouds
Geostatistics for Environmental Scientists
Figure 6: The distribution ClimaQA-Gold questions over the corresponding source.
The five textbooks selected for question generation were carefully chosen to reflect different levels of breadth and depth (both technically and more qualitatively) across range of important topics in climate science. These include Atmospheric Science, a comprehensive graduate-leve textbook atmospheric science; Aerosols and Climate, brand-new and more detailed textbook the role of aerosol in the climate system, An Introduction to Clouds, another relatively new textbook pro- viding the latest research on the important role of clouds in the climate system; Geostatistics for
16

---

Published as a conference paper at ICLR 2025
Environmental Scientists, more technical graduate-level textbook on geostatistics for environmen- tal sciences; as well as Calculus of Variations, a classic calculus textbook to test the extent of the technical ability of the models. The exact distribution in Figure 6 The remaining textbooks were used in the rag-held-out experiments to test the models without directly exposing them to the sources of the benchmark.
questions from these textbooks is shown
A.2 MORE EXPERIMENTS
A.2.1 FREEFORM EVALUATION RESULTS
Table 5: Performance analysis of various state-of-the-art L.MSs on Freeform QA. Bold marks the max within model's variants and green highlights the overall column max.
Model
BLEU
BERTScore
Factual Acceuracy
B
R
H
O
B
R
H
O
B
R
H
O
gemma-27b
0.381
0.431
0.418
0.410
0,870
0.886
0.888
0.881
0.63
0.78
0.79
0.73
gemma-27b-fss
0.430
0.400
0.361
0.397
0.884
0.895
0.896
0.892
0.65
0.77
0.82
0.75
gemma-27b-rag-sourre
0.462
0.573
0.552
0.529
0.902
0.911
0.906 0.906
0.84
0.85
0.85
0.85
gemma-27b-rag-heldaan
0.349
0.496
0.524
0.456
0.867
0.889
0.896
0.885
0.58
0.73
0.77
0.69
gpt-3.5-turbo
0.453 0.515
0.579
0.516
0.886
0.902
0.907
0.899
0.63
0.77
0.79
0.73
gpt-3.5-turbo-fs
0.514 0.451
0.465
0.477
0.894
0.908
0.915
0.906
0.61
0.77
0.86
0.75
0.528
0.601
0.561
0.563
0.908
0.919
0.921 0.916
0.85
0.86
0.87
0.86
gp--3.-turro-rrg-held----
0.425
0.477
0.543
0.482
0.877
0.889 0.904
0.891
0.46
0.64
0.81
0.64
gpt-4o
0.458 0.483
0.500
0.480
0.887
0.896 0.902 0.895
0.67
0.80
0.81
0.76
gpt-4o-fs
0.502 0.430
0,411
0.448
0.898
0.906 0.914 0,906
0.66
0.82
0.84
077
spt-4o-rag-sourg
0.539 0.626
0.620 0.595
0.916
0.917 0.916 0.916
0.81
0.83
0.85
0.83
0.425 0.520
0.561
0.502
0.884
0.892
0.903
0.893
0.54
0.71
0.86
0.70
llama3-70b
0.312 0.469
0.510
0.430
0.873
0.892 0.898 0.888
0.65
0.66
0.85
0.72
llama3-70b-fs
0.456 0.557
0.585
0.532
0.889
0.903 0.909 0.900
0.64
0.68
0.78
0.70
Hamaa-70b-rag-aa
0.441
0.574
0.604
0.540
0.909
0.916
0.917
0.914
0.82
0.81
0.84
0.82
llama3-70b-rag-
0.341
0.475
0.519
0.445
0.872
0.890
0.902
0.888
0.45
0.62
0.79
0.62
mixtral-8x22b
0.374
0.516
0.525
0.471
0.877
0.897 0.900
0.891
0.67
0.74
0.85
0.75
mixtral-8x22b-fs
0.495
0.497
0.486
0.493
0.892
0.906 0.913 0.904
0.68
0.76
0.83
0.76
mixtral-8xx22--ag--sooe
0.458
0.598
0.610
0.555
0.905 0.916 0.912
0.911
0.78
0.82
0.86
0.82
mixtral-Bxx22brrg-hhell hollooo-
0.341
0.488
0.546
0.459
0.870
0.890 0.898 0.886
0.50
0.62
0.77 0.63
llama3.1-89 1-8b
0.387 0.467 0.505
0.453
0.872
0.889 0.899
0.887
0.59
0.65
0.75 0.66
llama.. 1--b-fs
0.399 0.484 0.532 0.472
0.877
0.894 0.905 0.892
0.59
0.68 0.75 0.67
Ilama3. 1-8b-r88-r-----
-oburee--ouuce source
0.509 0.550 0.534 0.531
0.905
0.920 0.911 0.912
0.77
0.80
0.83 0.80
Ilama3. 1-8b-rag-helldour
0.392 0.469 0.507 0.456
0.873
0.893 0.901 0.889
0.52
0.65 0.72 0.63
Ilama3. 1lama3.1-8b-gr 1-8b-cp-held-ood
0,420 0.455 0.496 0.457
0.876
0.892 0.902
0.890
0.53
0.70
0.71 0.65
1omma3.-8b-ftt-ilfe
0.436 0.469
0.533 0.480
0.882
0.896 0.904
0.894
0.53
0.59
0.77 0.63
mistral-7b
0.385 0.370
0.412 0.390
0.869
0.886 0.892
0.882
0.58
0.64
0.74 0.65
mistral-7b-fs
0.432 0.409 0.438 0.427
0.878
0.901 0.908 0.896
0.51
0.66
0.80 0.66
mistral-77-rag--surre
0.541 0.455 0.448 0.482
0.898
0.912 0.907 0.906
0.77
0.82
0.84 0.81
mistral--b-rag-heldd-oo
0.413 0.356 0.399 0.390
0.868
0.884 0.896 0.883
0.54
0.60 0.73 0.62
0.295 0.176
0.189
0.221
0.880
0.890 0.899 0.890
0.52
0.70
0.70 0.64
mistral-7b-ft-silve
0.520 0.386
0.434
0.447
0.892
0.900
0.906
0.899
0.40
0.61
0.82 0.61
17

---

Published as a conference paper at ICLR 2025
A.2.2 CLIMAQA-SHLVER EXPERIMENTS
In this section, we assess the models in their default settings on a subset of the ClimaQA-Silver dataset, consisting of 200 questions for each task type. The column-wiss trends observed in the results are largely consistent with those from the ClinaQA-Gold dataset. However, a notable dif- ference lies in the relative difficulty of the complexities in the MCQ task, scores do not vary significantly. This discrepancy highlights the intricate nature of the MCQ gener- ation process. Additionally, this variation may also be attributed to the relatively small number of questions in each column. Future work should focus on scaling the expert validation process to enhance the quality of the automated annotation pipeline, thereby addressing these challenges and improving overall dataset reliability.
though the overall
Table 6: Performance analysis of various LLMs on MCQs and Cloze QA in ClimaQA-Silve
Model
MCQ
Cloze EM
Base
Reason
Hypo Overall
PS
gemma-27b
78.00
84.91
76.6
79.5
50.00
0.85
gpt-3.5-turbo
76.00
75.47
72.34
75.0
38.00
0.78
gpt-4o
88.00
84.91
78.72
85.00
60.50
0.88
llama3-70b
85,00
79.25
65.96
79.00
44.00
0.83
mixtral-8x22b
80.00
81.13
82.98
81.00
33.00
0.67
Table 7: Performance analysis of various LLMs on Freeform QA in ClimaQA-Silver
Model
BLEU
BERTScore R
Factual Acceuracy
B
R
H
O
B
H
O
B
R
H
O
gemma-27b
0.392
0.441
0.365
0.398
0.870
0.880
0.886
0.876
0.78
0.86
0.85
0.81
gpt-3.5-turbo
0.467
0.523
0.545
0.500
0.885
0.892
0.907 0.892
0.71
0.79
0.84
0.76
gpt-4o
0.440
0.493
0.491
0.465
0.880
0.887 0.902 0.887
0.80
0.88
0.86
0.84
1lama3-70b
0.335
0.474
0.569
0.425
0.874
0.888
0.904
0.885
0.77
0.82
0.75
0.78
mixtral-8x22b
0.394
0.516
0.485
0.447
0.877
0.890
0.904
0.887
0.80
0.88
0.83 0.82
A.2.3 CASE STUDY
In this section, present sample questions from the ClimaQA-Gold dataset along with responses from various LLMs and their corresponding evaluation scores.
Freeform QA
In this example, the reference answer discusses increased global emissions in the absence of air quality measures. While the outputs of most LLMs align with this reference, the response from gemmu-27b suggests a decline in global emissions. Note that both the Factual Accuracy and BLEU metrics penalize this response, whereas the BERTScore does not.
Quesition: How might the global emission estimates have been different if air quality mea- sures introduced mainly targeted at large, coal-burning installations before 2000?
Reference Answer: Without the introduction of air quality measures mainly targeted at large, coal-burning installations before 2000, the global emission estimates, particularly for SO2, may not have shown a decrease hefore 2000. The emissions could have continued to rise or remained at higher levels due to the ongoing industrial growth and lack of regulations to curb pollution from such sources.
18

---

Published as a conference paper at ICLR 2025
gpt-3.5-turbo: Global emission estimates would likely have been higher if air quality mea- sures targeting large, coal-buring installations were not introduced before 2000, as these mea- sures have helped reduce emissions of pollutants such as sulfur dioxide and particulate mat- ter.
Bleu: 0.485, Bert: 0.901, Factual Accuracy: 0.988
Without air quality measures targeting large, coal-burning installations before 2000, global emissions, particularly sulfur dioxide and particulate matter, would likely have been significantly higher, exacerbating air pollution and related health issues. This could have also contributed to more severe acid rain and potentially altered climate patterns due to increased atmospheric aerosols.
gpt-4o:
Bleu: 0.589, Bert: 0.898, Factual Accuracy: 0.982
1lama3-70b: If air quality measures were not introduced before 2000, global emission esti- mates would likely be higher, as coal-burning installations would have continued to emit large amounts of pollutants, including particulate matter, sulfur dioxide, and nitrogen oxides, con- tributing increased greenhouse gas emissions. This could have resulted in additional 1-2 gigatons of CO2 equivalent emissions per year, exacerbating climate change impacts.
Bleu: 0.542, Bert: 0.888, Factual Accuracy: 0.968
mixtral-8x22b: If air quality measures hadn't 't been introduced for large, ooal-burning instal- lations before 2000, global emission estimates would likely have been significantly higher due to unregulated emissions from these sources. This could have potentially accelerated global warming and worsened air quality on global scale.
Bleu: 0.546, Bert: 0.910, Factual Accuracy: 0.988
Global emission estimates would likely be significantly higher without those air quality measures. Reduced emissions from coal-burning power plants would have contributed less to the overall decline in global emissions observed since 2000.
gemma-27b:
Bleu: 0.364, Bert: 0.885, Factual Accuracy: 0.741
In the following example, the reference answer indicates that the distance from the sun influences surface pressure, with higher pressures during summer. Responses from gpt-3.5-nurbo and mixtral- 8x22b contradict this statement but still achieve high BLEU and BERTScore values. The Factual Accuracy metric, however, correctly identifies and penalizes this inconsistency. Notably, gpt-4o pro- vides a correct answer yet receives low BLEU score, BLEU is not robust against paraphrasing.
Quesition: How does the distance from the sun affect the surface pressure on Mars?
Reference Answer: The surface pressure on Mars varies seasonally from 7.0 hPa during the austral winter, when Mars is farthest from the sun, to 9.0 hPa during the austral summer.
The distance from the sun does not directly affect the surface pressure on Mars; instead, the surface pressure Mars is primarily influenced by the planet's atmospheric composition and temperature.
gpt-3.5-turbo:
Bleu: 0.388, Bert: 0.880, Factual Accuracy: 0.175
The distance from the Sun affects Mars' temperature, which in turn influences the sub- limation of its polar ice caps, impacting atmospheric pressure. During warmer periods, more CO2 sublimates, increasing surface pressure, while colder periods result in lower pressure.
gpt-4o:
Bleu: 0.253, Bert: 0.865, Factual Accuracy: 0.980
19

---

Published as a conference paper at ICLR 2025
The distance from the Sun affects the surface pressure on Mars by controlling the amount of atmospheric gas that freezes out or escapes to space, with more gas freezing out at greater distances. As result, Mars' surface pressure about 1% of Earth's s, due to its average distance from the Sun being about 1.5 times greater than Earth's.
llama3-70b:
Bleu: 0.262, Bert: 0.861, Factual Accuracy: 0.896
mixtral-8x22b:
The distance from the does significantly affect the surface pressure Mars. Instead, Mars' thin atmosphere and low gravity are the primary factors influencing its surface pressure.
Bleu: 0.403, Bert: 0.874, Factual Accuracy: 0.051
gemma-27b: Mars is farther from the sun than Earth, receiving less solar radiation. This weaker solar heating results in much lower surface pressure Mars compared to Earth.
Bleu: 0.458, Bert: 0.870, Factual Accuracy: 0.905
Interestingly, gemma--27 provides an irrelevant response that nevertheless receives high scores. While the response is factually accurate, fails address the question, illustrating limitations in existing evaluation metrics. This highlights the need for more fine-grained evaluation methodolo- gies, which future research should aim develop.
Cloze QA
In the following examples, the models generate answers that differ from the reference answer but remain correct within the given context. This is effectively captured by the phrase similarity metric, which assigns higher scores to responses that are contextually aligned with the reference answer.
Question: Aqueous-phase oxidation of (mask) to form sulfuric acid in cloud droplets is cru- cial process that contributes significantly to global sulfate mass and has long been recognized important for aerosols and climate.
Reference Answer: SO2
gpt-3.5-turbo: Sulfur, Phrase Similarity: 0.819
gpt-4o: sulfur dioxide, Phrase Similarity: 0.885
llama3-70b: Sulfur, Phrase Similarity: 0.819
mixtral-8x22b: sulfur dioxide, Phrase Similarity: 0.885
gemma-27b: sulfur dioxide, Phrase Similarity: 0.885
The rate of change in the air parcel's moist static energy due to (mask) is deter- mined by the difference between the moist static energies of the environment and the cloud air parcel.
Question:
Reference Answer: entrainment
gpt-3.5-turbo: evaporation, Phrase Similarity: 0.681
gpt-4o: entrainment, Phrase Similarity: 1.000
llama3-70b: entrainment, Phrase Similarity: 1.000
mixtral-8x22b: mixing, Phrase Similarity: 0.726
gemma-27b: entrainment, Phrase Similarity: 1.000
20

---

Published as a conference paper at ICLR 2025
A.3 TRAINING DETAILS
We used Llama3.J 1-8B and Mistral-7B-vDl our base models and performed continued pre-training and fine-tuning on them. We utilized the Low-Ruue Adaptation (LoRA) (Hu et al. 2021 technique for efficient continued pre-training and fine-tuning.
A.3.1 CONTINUED PRE-TRAINING ON GRADUATE TEXTBOOK DATA
We used 13 graduate textbooks as our training set, while the 5 textbooks used for question generation served as the validation set. A hyperparameter search was performed, guided by the cross-entropy loss on the validation set of textbooks. The final hyperparameters used for continued pre-training are detailed in Table g
Model
LoRA Rank
LoRA Alpha Epoch Count Learning Rate
Mis0ral-7b-v0.3
64
16
5e-5
Llama-3.1
16
16
2e-5
Table 8: Parameters used for graduate textbook continued pre-training
A.3.2 FINE-TUNING ON HLIAQQA-SLVER
We used the with the various question forms and complexities seen in the ClimaQA-Gold to see how this task- specific fine-tuning can affect the performance.
dataset fine-tune Llama3.1-8P and Mistral-7B-v0.3 on questions
For this purpose, we used the hyperparameters, shown in Table 9
Model
LoRA Rank LoRA Alpha Epoch Count Learning Rate
Mistral-7b-v0.3
16
16
5e-5
Llama-3.1 Al1-a-3.1-8b
16
16
5e-5
Table 9: Parameters used for question finetuning
A.4 CLIMAGEN PROMPTS
A.4.1 QA GENERATION
The following prompt was used to generate multiple-choice question-answer pair:
You are question paper setter creating multiple choice questions (MCQs) from a graduate- level climate science textbook.
MCQ Components:
Stem: The main question, scenario, or statement requiring completion. should clearly assess the intended knowledge.
2. Correct Answer: The indisputable correct response the stem.
3. Distractors: Three incorrect but plausible answers. They should be:
Related to the stem and correct answer.
Positively phrased and true statements that don't answer the stem.
Plausible but incorrect, without giving clues the correct answer.
Unique, each reflecting different misconceptions if possible.
MCQ Guidelines:
21

---

Published as a conference paper at ICLR 2025
Questions should be clear, concise, and free from unnecessary complexity or ambiguity.
2. Avoid overly long sentences and use consistent phrasing for repeated items.
3. Ensure questions are self-contained and provide necessary context.
4. Do not include phrases like "According to the provided context..."
5. Do not make any references the given context the question
6. Ensure that distractors do not overlap by reflecting different misconceptions on the topic.
7. Minimize clues that make the correct answer obvious.
8. Use "None of the Above" or "All of the Above" sparingly.
9. Each MCQ must have exactly four
choices (one correct, three distractors).
10. Questions should not rely on external figures or tables.
The user will provide main context and some retrieved contexts separated by as the input.
Use details from retrieved context only if they are relevant to your question.
You must output a single JSON object in the following format:
{
question: (question).
options: {
a: (optionl).
b: (oplion2).
(option3), d: (option4) }.
correct option:
Here is the correct answer. Replace it with the actual correct answer.
Make sure you return a valid JSON object.
The following prompt was used to generate a freeffom question- answer pair:
You are a question paper setter creating freeform questions (MCQs) from graduate-level climate science textbook. Your question must be related to provided context.
Please respect the following rules to generate the question:
The answer to the question should be found inside the provided context
The question must be self-contained
Do not include phrases such as "According to the provided context."
The user will provide one main context and some retrieved contexts separated by as the input.
Use details from retrieved context only if they are relevant to your question.
You must output a single JSON objectin the following format:
22

---

Published as a conference paper at ICLR 2025
question: (queslion),
answer: (answer)
Make sure you return a valid JSON object.
The following prompt was used generate scientific statement for cloze question question-answer answer genera- tion:
You are scientific annotator. Given scientific context from climate textbook, generate scientific statement based on the facts presented in the context.
Please respect the following rules to generate the statement:
Generate only single sentence
No external knowledge should be used refered in generating the statement
Do not use phrases like 'based on the provided context."
The user will provide one main context and some retrieved contexts seperated by as the input.
Use details from retrieved context only if they are relevant to your question.
You must output single JSON objectin the following format:
{
statement: (statement),
Make sure you return a valid JSON object.
A.4.2 COMPLEXITY ADDITION
The following prompt used to add reasoning complexity the base freeform question-answer pair.
Given a question-answer pair generated from the given context, Modify the question-answer pair to incorporate multi-step reasoning.
Please respect the following rules to generate the question:
Answering the new question should encourage applying knowledge from "Context" to deduce outcomes.
The question must be fully answerable from 'Context'.
No external knowledge should required to answer the new question
The question should not be dependent external things such figures tables
Do not use phrases like 'based on the provided context."
The user will provide the original question, context, and some retrieved contexts separated by as the input.
Use details from retrieved context only if they are relevant to your question.
You must output single JSON objectin the following format:
23

---

Published as a conference paper at ICLR 2025
{
question: (question),
answer: (answer)
Make sure return a valid JSON object.
The following prompt was used add hypothetical scenario complexity to the base freeform question-answer pair.
Given question-answer pair generated from the given context, Modify the question-answer pair to incorporate hypothetical or speculative scenario.
Please respect the following rules generate the question:
Answering the new question should encourage applying knowledge from 'Context" to deduce outcomes.
The new question must be fully answerable from 'Context'.
No external knowledge should be required answer the new question
The question should not be dependent on external things such as figures or tables
Do not use phrases like 'based the provided context.'
The user will provide the original question, context, and some retrieved contexts separated by as the input.
Use details from retrieved context only if they are relevant to your question.
You must output single JSON object in the following format:
{
question: (guestion),
answer: (answer)
Make sure you return a valid JSON object.
The same prompts with modified output format were used to add complexities to the MCQs as well.
A.5 ANNOTATION
A.5.1 EXPERT VALIDATION
During the validation process of the ClimaQA-Gold dataset for both multiple-choice and freeform questions, experts were asked to select predefined reasons for rejecting generated QA pair. They also had the option to provide a custom reason for rejection. The figure below illustrates the various types of errors made by the ClimaGen pipeline during the generation phase. While most rejection reasons highlight limitations the generator LLM, the category of bad context points to instances where the seed context used for QA generation inherently flawed or lacked meaningful infor- mation. Addressing this issue through improved preprocessing techniques a key area for future work.
24

---

Published as a conference paper at ICLR 2025
wrong answer
31.3%
context
29.9%
6.0%
other reason
14.9%
17.9%
not enough information
question doesn't make sense
Figure 7: The distribution of different reasons selected for scientific invalidity of generated QA pairs by domain experts.
A.5.2 AUTOMATED ANNOTATION MCQ AND FREEFORM
We used gpt-4o-mini as the base model for the Evaluator Model. It was fine-tuned for the classifica- tion task with the prompt below. The same prompt was used for both MCQ and freeform validation and 2 different models were created by fine-tuning the base model with respective datasets.
You are climate {guestion-type) question-answe validator that marks given question- answer pair as valid or invalid based on scientific accuracy with respect to the given context. You will be provided the following as the input:
Question: (question)
Answer: (answer)
Context: (contezt)
Respond with just one word VALID if the qa pair is scientifically accurate and INVALID otherwise
Dataset
Valid Count Invalid Count
Total Epochs
MCQ 245
47
292
3
Freeform
161
20
181
3
Table 10: Details of expert-validated dataset and fine-tuning
Since negative samples are dropped in large-scale data generation, precision used over accuracy as the key metric to measure the classifier's 's impact, as it represents the fraction of final output data that is valid.
NO19 RIFS Braik aod
95
Precision (N4
90
Prectsion M91
90
Foo-tred
255
80
70
Tral2
11113
75
Taal
Pial
Thbl
(a) MCQ
(b) Freeform
Figure 8: Precision across various train-test splits of labeled data under different conditions: validation (all data classified as positive), zero-shot validation, and fine-tuner validation. The test sets consisted of around 40 examples in each case.
25

---

Published as a conference paper at ICLR 2025
A.5.3 AUTOMATED ANNOTATION CI.OZE
To automate cloze annotation, we fine-tuned gpt-4o-mini with the following prompt to pick the scientific term to be masked. The dataset consisted of total of 160 expert- labeled examples.
You are climate cloze generator that marks scientific term from the given scientific state- ment to be masked for cloze question-answering The scientific term has to be a single word from the given statement that has significant impact if removed You will be provided the following as the input:
Statement: (statement)
Respond with just one word
26


Reviews:
[{"review_id": "dbHzFQxrf2", "reviewer": "Reviewer_vgwR", "summary": "The authors introduce ClimaQA, a question-answering benchmark designed to evaluate climate foundation models in handling scientific inquiries. This paper develops ClimaGen, a framework for generating question-answer pairs with climate experts in the loop, resulting in two datasets: ClimaQA-Gold (expert-validated) and ClimaQA-Silver (synthetic). The benchmark assesses models across multiple task forms (MCQ, freeform, and cloze) to gauge factual accuracy and scientific reasoning. The paper presents an analysis of several LLMs on ClimaQA.", "questions": "1. What is the potential for scaling the ClimaQA benchmark to other scientific domains? Could ClimaGen be adapted for other domains like environmental science or epidemiology, and what would be the main challenges?\n\n2. Have you considered additional metrics for freeform QA evaluation? While factual accuracy is addressed, additional metrics like coherence and logical flow might further enhance the assessment of model responses on freeform tasks.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "· Originality: ClimaQA stands out as a pioneering benchmark specifically tailored for climate science. The framework’s approach of combining domain expertise with synthetic QA generation represents an innovative solution to the lack of reliable climate QA datasets.\n\n· Quality: The benchmark’s rigorous development process, including the expert validation of ClimaQA-Gold and systematic generation of ClimaQA-Silver, ensures high-quality question-answer pairs that reflect real-world scientific standards. The choice of using diverse question formats (MCQ, freeform, cloze) is well-aligned with testing scientific reasoning skills.\n\n· Clarity: The paper is organized and highly readable, with clear descriptions of each dataset and evaluation metric. The figures illustrate the QA generation and annotation processes well, making the methodology transparent.", "weaknesses": "· Complexity of Generated Questions: While the QA generation pipeline is well-designed, some questions in ClimaQA-Silver may lack the complexity required to thoroughly test advanced climate models. Supplementing the dataset with more nuanced, multi-layered questions could improve its utility.\n\n· Dependence on Manual Annotation for ClimaQA-Gold: Although ClimaQA-Gold is an expertly annotated dataset, its reliance on manual validation limits scalability. Developing a semi-automated validation process or implementing peer review from multiple experts could reduce bias and improve reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uzGMZi9cii", "reviewer": "Reviewer_QKkY", "summary": "The authors present ClimaQA, a benchmark for language models, assessing skills in the area of climate science. The research introduces an automated pipeline (ClimaGen) to generate question-answer pairs from graduate textbooks, resulting in two datasets: ClimaQA-Gold (expert-validated) and ClimaQA-Silver (synthetic). The article presents three types of tasks: multiple-choice, freeform, and cloze with varying levels of complexity to challenge models on factual recall, scientific reasoning, and scenario application. For evaluation, the research develops dedicated metrics for freeform and cloze tasks and employs a fine-tuned LLM for the synthetic annotation process using expert-annotated data. The study evaluates several state-of-the-art large language models on the created benchmark under different settings, such as default configurations, few-shot prompting, and retrieval-augmented generation.", "questions": "- Could you provide a quantitative comparison with established climate benchmarks, specifically Climate Crisis and SciQAG-24D?\n- Would it be possible to present a comparative performance analysis, similar to Table 3, for the ClimaQA-Silver dataset to better understand potential quality differences between gold and silver datasets?\n- Has the research considered implementing a mixture-of-experts approach for the freeform evaluation process instead of relying solely on GPT4o-mini?\n- What were the selection criteria for the 18 textbooks, and how were they assessed for their relevance to recent advances in climate science?\n- What methodology was used to ensure that the set of 5 textbooks provided a balanced coverage of climate science topics?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The research addresses a lack of expert-annotated benchmarks in climate science and proposes a novel algorithmic framework to accelerate scientific benchmark creation through synthetic generation and expert validation.\n- The synthetic scaling approach enables continuous updates to the benchmark, accommodating new findings in climate science research.\n- The benchmark's incorporation of three distinct tasks (multiple-choice questions, freeform, and cloze) reduces potential biases arising from models' familiarity with specific question formats, unlike benchmarks limited to a single task type.\n- The implementation of three well-defined complexity levels enables a comprehensive assessment of models’ knowledge and application capabilities, allowing precise identification of areas for improvement and a better understanding of current model performance.", "weaknesses": "- Exclusive use of proprietary models (GPT3.5 for question generation and GPT4o-mini for freeform judgment) introduces potential bias favoring models from the same provider or those trained/fine-tuned on similar model outputs.\n- The proposed metrics demonstrate an inconsistent correlation with expected language model performance. For instance, GPT3.5 exhibits superior or comparable performance against newer models, including those from the same family (GPT4o), on BLEU and BERTScore metrics as shown in Figure 5, highlighting potential metric reliability concerns in model evaluation.\n- The research reports an 85% validity rate for generated multiple-choice questions in ClimaQA-Silver (Section 4.4), which may compromise the effectiveness of the evaluation. As several models on ClimaQA-Gold reach or exceed this performance level, the synthetic dataset may not provide reliable performance metrics. A comparative analysis similar to Table 3 for the synthetic dataset could illustrate this limitation.\n- The study lacks quantitative comparisons with existing climate benchmarks, such as Climate Crisis or SciQAG-24D.\n- The benchmark's reliance on textbook knowledge may introduce distribution bias in the evaluation process.\n- The research provides a limited analysis of the expert validation process and common question generation errors, which could offer valuable insights into potential biases in the synthetic benchmark version.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FWmyCHPNem", "reviewer": "Reviewer_hgtP", "summary": "The authors present a benchmark that they have developed for evaluating LLMs on question-answer tasks specialized for the domain of climate science. They developed two datasets: ClimaQA-Gold containing 502 questions that have been annotated by climate science experts, and ClimaQA-Silver containing 3000 questions generated by their framework for LLM-powered generation of question-answered pairs based on a corpus of graduate-level textbooks. Their benchmark contains three types of questions: \"free form\" which expects the answers to be free-form text, \"multiple choice\" where the answer is given as an indicator to one of the proposed answer choices, and \"cloze\" which contains certain parts of text marked as \"blank\" and the task is to find the correct words to fill in the blanks. Questions come in three levels of difficulty: \"base\" which focuses on simple information retrieval, \"reasoning\" which requires the connection of multiple facts to form a conclusion, and \"hypothetical scenario\" which requires the application of knowledge to novel contexts. They conduct an empirical evaluation of several popular LLMs (including extensions such as RAG). The scores that the models achieve form a wide range which demonstrates the utility of the benchmark for evaluating LLMs on QA tasks for the specific domain of choice.", "questions": "**(Q1)** In section 5.2 you mention that there are 5 books used to derive the corpus of questions, and an additional 13 books that are used for the RAG model as a retrieval corpus. Do we know if all relevant information between these two sets of books overlaps? In other words, do we know if the information in the 13 books is sufficient to support all the questions that were produced by using the 5 books?\n\n**(Q2)** In the introduction you mention \"Creation of publicly releasable datasets\". Could you please clarify what you mean by a \"releasable\" dataset?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**(S1)** The proposed benchmark represents a significant step in creating a rigorous evaluation for QA models on climate-specific knowledge and reasoning tasks.\n\n**(S2)** The authors provide both a human-curated corpus for final evaluation and an automatically augmented corpus for fine-tuning.\n\n**(S3)** The fact that the benchmark is able to stratify models based on their performance on this task demonstrates that it is challenging, which will help it become a reference point for future models that are built specifically for the climate science domain.", "weaknesses": "**(W1)** The domain that the authors focus on can be deemed as a very niche one. The authors could have tried to do a better job at providing concrete arguments in the introduction that underline the significance of this domain. Specifically, it would be good to explain why we need LLMs to be good at this specific domain. If they become better at this domain, which applications would this unlock?\n\n**(W2)** There are certain small issues with the presentation: (1) the authors use the term \"foundation models\" but probably just want to say either \"large language models\" or \"question-answering models\". Foundation models are pre-trained on large corpora and used mainly for transfer learning across different tasks, but many of them are not LLMs. It would help to use a more specific term. (2) Formulas presented in section 3 use some symbols that were not explicitly defined (e.g. $p_s$, $p_r$). (3) The numbers in Figure 4 seem to keep going up (albeit at a reducing pace), and yet the authors conclude that a \"context window of size 4 most effectively differentiates between correct and incorrect answers\". From the figure, it is unclear to me why this context window is optimal.\n\n**(W3)** This is a minor weakness, but the provided corpus is relatively modest, compared to other ones. I don't think this is too much of an issue given that it is probably large enough to sufficiently evaluate a QA model, and a small size makes it more manageable. That said, it would be good to provide some sense of how well the provided corpus covers the actual domain (e.g. by giving the readers a sense of the subdomains and the number of questions from each subdomain).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ic4w35wFB7", "reviewer": "Reviewer_wZYi", "summary": "The paper presents a framework that significantly enhances the reliability and scientific rigor of AI models in climate science by generating domain-specific QA benchmarks. With ClimaQA-Gold, an expert-validated dataset, and ClimaQA-Silver, a large-scale synthetic set, the framework tests foundation models across different complexity levels, addressing factual accuracy, scientific reasoning, and hypothetical scenarios. By leveraging retrieval-augmented generation (RAG) and innovative metrics like Factual Accuracy and Phrase Similarity, ClimaGen facilitates more accurate model evaluation and fine-tuning. While advanced models like GPT-4 show promising results, they still face challenges in complex scientific questions, highlighting ClimaGen's role in driving further advancements in specialized climate AI. An expanded corpus and annotated dataset could enhance its robustness and adaptability to broader applications in scientific inquiry.", "questions": "Suggested Improvements:\n\n- Increase the number and diversity of climate science sources, including a broader range of textbooks, research papers, and domain-specific databases. This would enhance the contextual diversity and robustness of the generated questions.\n- Expand the expert-validated dataset to strengthen the framework's reliability and support automated annotations in additional scientific fields.\n- Implement a tiered evaluation system that emphasizes model performance on reasoning and hypothetical questions separately, allowing for a deeper analysis of model strengths and weaknesses in scientific reasoning.\n- Further experiment with retrieval-augmented generation using a wider array of sources or explore weighted retrieval to reduce distractors and improve information relevance.\n- Introduce additional objective metrics to balance any biases introduced by BLEU or BERTScore, potentially incorporating domain-specific metrics tailored to scientific language accuracy and precision.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The ClimaGen framework is tailored to climate science, which enhances the reliability of AI applications in this field by addressing unique requirements for factual rigor, domain-specific knowledge, and scientific reasoning.\n- Combining an expert-validated gold-standard dataset with a large-scale synthetic dataset provides comprehensive resources for training and evaluating foundation models, supporting both high-quality benchmarking and scalability.\n- The benchmark evaluates models across three question complexity levels (factual recall, reasoning, hypothetical scenarios) and multiple task types (MCQ, freeform, and cloze), which ensures a well-rounded assessment of model performance in scientific inquiry.\n- Metrics like Factual Accuracy and Phrase Similarity for freeform and cloze tasks, respectively, offer a nuanced evaluation, considering both factual precision and contextual understanding.\n- The use of RAG improves models' performance on reasoning questions, demonstrating a promising adaptation technique for enhancing model outputs.", "weaknesses": "- The framework relies on a relatively small set of graduate-level climate textbooks, which may limit the diversity of contextual information and scientific perspectives in the generated questions.\n- The expert-annotated dataset (ClimaQA-Gold) is relatively small, which may restrict the scalability and generalizability of automated annotations in other scientific disciplines.\n- Although improvements were noted, the models often struggle with reasoning-based questions, suggesting that further work is needed to refine reasoning capabilities in foundation models.\n- The BLEU and BERTScore metrics show some bias towards the model used for question generation (GPT-3.5-turbo), which may affect the objectivity of the evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FUhgPw6Jpd", "reviewer": "Reviewer_mkPm", "summary": "This paper designs an automated pipeline for constructing a ClimaQA benchmark, then presents two datasets, ClimaQA-Gold and ClimaQA-Silver, and finally conducts an experimental evaluation of existing LLMs on this benchmark.", "questions": "- Q1: Why only three tasks are designed and what is the motivation? Why not design judgment questions or are there any other possible tasks needed to assess the quality of scientific validity of model outputs?\n- Q2: Both datasets of ClimaQA constructed in this paper are relatively small in size. What knowledge of climate is covered by these QA pairs? It shouldn't be very comprehensive, and I'm concerned that this only focuses on certain small branches of scientific knowledge. Is it possible to make some statistical analysis of the datasets to illustrate this?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- S1: This paper develops ClimaGen, an automated climate QA generator.\n- S2: This paper proposes two datasets and three question-answering tasks.\n- S3: Based on the ClimaQA benchmark, this paper presents some experiments to evaluate existing llms.", "weaknesses": "- W1: One of the core innovations of the paper is the tool ClimaGen that automates the construction of ClimaQA, which involves the QA principle and prompt assembler. From line345 it appears to be a direct reference to other articles, which the authors do not describe in detail. However, this part directly affects the quality of the input prompt, which determines the quality of the generated QA pairs. \n- W2: This paper proposes a benchmark to assess the quality and scientific validity of the output of climate base models, but there is a lack of experimental validation that the QA pairs in the benchmark are scientifically meaningful, i.e., how to prove that the constructed dataset is able to help future research on climate foundation models, so that the output is of higher quality and scientific validity. \n- W3: Lack of detailed information about the data source. The authors emphasize the data sources as 18 graduate-level textbooks, but give no detailed information about the data sources to prove it.\n- W4: The experimental section lacks a case study. There is no visual comparison between the actual performance of the existing LLMs on these tasks and the difference between the groundtruth.\n- W5: Writing flaws: lack of citations for some important ideas, such as in line32 and line35; inconsistency in font size of illustrations, such as Figure 6; lack of necessary explanations for the letters of the formulas, such as the equations about factual accuracy and phrase similarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)