CURIE: Evaluating LLMs on Multitask Scientific Long-Context Understanding and Reasoning
Authors: Hao Cui
Zahra Shamsi
Gowoon Cheon
Xuejian Ma
Shutong Li
Maria Tikhanovskaya
Peter Christian Norgaard
Nayantara Mudur
Martyna Beata Plomecka
Paul Raccuglia
Yasaman Bahri
Victor V. Albert
Pranesh Srinivasan
Haining Pan
Philippe Faist
Brian A Rohr
Michael J. Statt
Dan Morris
Drew Purves
Elise Kleeman
Ruth Alcantara
Matthew Abraham
Muqthar Mohammad
Ean Phing VanLee
Chenfei Jiang
Elizabeth Dorfman
Eun-Ah Kim
Michael Brenner
Sameera S Ponda
Subhashini Venugopalan
Venue: ICLR 2025 Poster (2025)
Keywords: science
LLMs
evaluation
benchmark
long-context
Primary area: foundation or frontier models, including LLMs
TL;DR: CURIE is a long context comprehension benchmark to asses the ability of LLMs in assisting in realistic scientific workflows requiring deep expertise.

Abstract: Scientific problem-solving involves synthesizing information while applying expert  knowledge.   We  introduce  CURIE,  a  scientific  long-Context  Understanding, Reasoning, and Information Extraction benchmark to measure the potential of Large Language Models (LLMs) in scientific problem-solving and assisting scientists in realistic workflows. This benchmark introduces ten challenging tasks with a total of 580 problems and solution pairs curated by experts in six disciplines - materials science, condensed matter physics, quantum computing, geo-spatial analysis, biodiversity, and proteins - covering both experimental and theoretical work-flows in science. We evaluate a range of closed and open LLMs on tasks in CURIE which requires domain expertise, comprehension of long in-context information,and multi-step reasoning.  While Gemini Flash 2.0 and Claude-3 show consistent high comprehension across domains, the popular GPT-4o and command-R+ fail dramatically on protein sequencing tasks. With the best performance at 32% there is much room for improvement for all models. We hope that insights gained from CURIE can guide the future development of LLMs in sciences. Links to the data and evaluation code are in https://github.com/google/curie

Published as a conference paper at ICLR 2025
CURIE: EVALUATING LLMs ON MULTITASK SCIENTIFIC LONG CONTEXT UNDERSTANDING AND REASONING
Hao Cui Zahra Shamsi' 1* Goweon Cheon Xuejian Ma' Shutong Li' Maria Tikhanovskaya
Peter Norgaard' Nayantara Mudur Martyna Plomecka Paul Raceuglia"
Yasaman Bahri'
Victor V. Albert44. Pranesh Srinivasan' Haining Pan Philippe Faist" Brian Rohr
Michael J. Statt Dan Morris' Drew Purves' Elise Kleeman Ruth Alcantara' Matthew Abraham
Muqthar Mohammad Ean Phing VanLee' Chenfei Jiang' Elizabeth Dorfman"
Eun-Ah Kim Michael P Brenner1.2 Viren Jain' Sameera Ponda' Subhashini Venugopalan
'Google, Haavard,, University of Zurich, *NIST, UMD College Park, Rutgerss "FU Berlin, MModelyst, CConell
ABSTRACT
Scientific problem-solving involves synthesizing information while applying ex- pert knowledge. We introduce CURIE, a scientific long-Context Understanding. Reasoning and Information Extraction benchmark to measure the potential of Large Language Models (LLMs) in scientific problem-solving and assisting scien- tists in realistic workflows. This benchmark introduces ten challenging tasks with a total of 580 problems and solution pairs curated by experts in six disciplines materials science, condensed matter physics, quantum computing, geospatial anal- ysis, biodiversity, and proteins covering both experimental and theoretical work- flows in science, We evaluate a range of closed and open LLMs on tasks in CURIE which requires domain expertise, comprehension of long in-context information, and multi-step reasoning. While Gemini Flash 2.0 and Claude-3 show consistent high comprehension across domains, the popular GPT-4o and command-R++ fail dramatically on protein sequencing tasks. With the best performance at 32% there is much room for improvement for all models. We hope that insights gained from CURIE can guide the future development LLMs in sciences. Evaluation code and data links in: https: ://::////a //ttttbttbb/../////// www. hamm//gogle//rriii
INTRODUCTION
The advancement of science relies on the ability to build upon the collective knowledge accumulated in scientific literature, requiring not only deep domain expertise and reasoning skills, but also the capacity to apply that knowledge within the context of a given problem. Large Language Models (LLMs) have already shown remarkable knowledge across wide spectrum of domains (Fig. lb). Recent benchmarks e.g., MMLU (Hendrycks et al., 2020), have demonstrated proficiency in varied subjects (e.g., mathematics, history, social sciences, law) with standardized exams (BAR, SAT etc.), commonsense reasoning (Zellers et al., 2019), coding (Chen et al., 2021; Jimenez et al., 2023), reading comprehension (Dua et al., 2019), arithmetic (Zhang et al., 2024; Lu al., 2023), and grade school scientific knowledge (Clark et al., 2018). However, surfacing knowledge to actively solving problems, the capacity to understand and reason about long- form, context-rich information is paramount. Recent advances in model architecture have seen dramatic increases in context windows from 8k to 32k, 128k, and 1M+ tokens, reflecting a growing recognition of this need.
LLMs transition from merely
This has led to development of benchmarks testing capabilities of LLMs on long document un- derstanding such as ZeroScrolls (Shaham al., 2023), Bamboo (Dong al., 2023) variety of tasks including summarization (Kryšciński et al., 2021), retrieval (Kamradt, 2023), multi-hop QA (Zhu al., 2024; Kočiský al., 2018), sorting sequences (Wang et al., 2024) and others (Bai et al., 2023). However, current LLM benchmarks on science e.g., PubmedQA (Jin et al., 2019), GPQA (Rein al., 2023) focus primarily on short sequence questions, with answers often in mul- tiple choice form. To address this gap, we introduce the scientific long-Cond Context Understanding, Reasoning, and Information Extraction benchmark (CURIE).
"equal technical contribution, worr done as a student researcher at Google "I LLad and corresponding author (vsubha shi n @goog] e. com)

---

Published as a conference paper at ICLR 2025
Gemini 2.0 Flash
MNLI: Knowlodge
GPQA: Expertise
UROP: Linguistis
CURIL: Se eceence + Long Context (ours)
Claude (Opus)
Gemini 1.5 Pro
80
Gemini 1.5 Flash
GPT-4a
60
Scoe
Gemini 1.0 Pro
Mixtral-8x7b
LongLLaMA
0.0
0.2 0.3
Average Score
Gemini 1.0 Pro
Gemini 1.5 Fro
Figure 1: (a) Average normalized performance of state-of-the-art LLMs across 10 tasks from six scientific domains in CURIE. (b) Comparing performance of different model versions supporting long-context windows on previous benchmarks testing Knowledge (MMLU), Linguistic (DROP), and Science expertise (GPQA), along with our new scientific long-context understanding CURIE benchmark, highlighting the difficulty of the tasks in the benchmark.
The CURTE benchmark introduces 10 tasks, with total of 580 input and solution pairs based on 429 research documents across six diverse scientific disciplines: materials science, theoretical condensed matter physics, quantum computing, geospatial analysis, biodiversity, and proteins - covering both experimental and theoretical aspects of scientific research. These tasks not only require deep domain understanding but also challenge models on their capacity to comprehend full-length scientific pa- pers for information extraction, concept tracking, aggregation, algebraic manipulation, multimodal understanding, and cross-domain expertise (e.g. generating code for theoretical calculations).
We the CURIE testbed to perform extensive evaluation and analysis of 8 state-of-the-ard open and closed weight models (see Fig. la) supporting context windows of 32k tokens or more. Among closed models, Gemini Flash 2.0 and Claude-3 perform consistently well across all disciplines, while Command-R+ does better amongst the open models. Our materials tasks, which require models to exhaustively retrieve and aggregate information spread through the document proved to be exceptionally challenging for all models. Notably, while GPT-4o does well on most tasks, it fares dramatically poorly on the protein sequencing task failing to stop generation and repeating subsequences leading to very low score. This repetition in subsequence is also seen in other open models. On the same task, the more recent Flash 2.0 provided correct python code to solve the problem half the time, and attempted to enumerate the sequence with lesser success other times.
While the CURIE benchmark is aimed at facilitating evaluation of scientific reasoning over long contexts, we hope the rich human annotations can serve the community in advancing planning, in- struction following, and evaluation of generated texts mixed and heterogeneous formats including dates, locations, numerical values, units, descriptors, domain specific terms, equations and code.
RELATED WORK
Science NLP tasks. There have been numerous datasets created to perform core NLP tasks sci- entific texts e.g. BLURB Gu et al. (2021). This includes (i) named entity recognition to annotate entities such disease names (Li al., 2016) material properties (Mavracic et al., 2021a), and relations such as disease-chemical interaction (Luan et al., 201 8); (ii) dependency parsing (Kim (PICO) annotation (Nye et al., 2018), (iv) text
al., 2003), (iii) participantiintereettoooo-ou classification such as citation intent classification (Jurgens et al., 2018; Cohan et al., 2019), paper domain classification (Beltagy al., 2019) from titles; (v) relation extraction for chemical-protein- disease annotation (Kringelum et al., 2016) or material structure and properties (Zhang et al.. 2023), (vi) information extraction e.g. material property values (Dong & Cole, 2022a; Polak & Morgan), and (vii) question answering (Jin et al., 2019; Krallinger et al., 2020). However all of these focus on inputs short length such as paper abstracts, sentences or text spans. They were curated for lan- guage models operating short contexts, though the task in the actual scientific workflow requires application on full documents. Of the recent LLM benchmarks, GPQA (Rein et al., 2023) focuses on evaluating scientific domain expertise in biology, physics, and chemistry, while MMLU (Hendrycks et al., 2020) covers a range of high school science. These too are limited to short questions and multiple choice answers.
Long context benchmarks. With the increase in context windows supported by the LLMs, there have been new benchmarks focusing on evaluating long context capabilties. ZeroScrolls (Shaham
2

---

Published as a conference paper at ICLR 2025
PDB
DFT-P OFT. C. MPH
Source Diverse Long
Benchmark
Domain
Tasks
Context
Trotehh Suazencing 184
Matorials
ZeroSCROLLS
various
Bamboo
various
Ruler
Ada-Leval
synthetic
lit., code QA,
CURIE
Condereed
Marter Fradics
HEE
14
BLURB
biomed
BIOGR
053
DFT
QASA
ML lit.
QA
Quartum Computing
Qusper
GPQA
science
MCQ
x
1981
147
OECC
CURIE
sei. lit.
196rd
(a) Comparison with other datasets.
(b) Distribution of tasks.
(c) Input word lengths per domain.
Figure 2: CURIE dataset. (a) CURIE introduces diverse long context tasks on scientific literature, (b) Distribution of 10 tasks and 429 research documents in six disciplines from which 580 examples were curated, and (c) Length of imput context in each domain (log scale), avg. is about 15k.
et al., 2023) covers summarization, question answering, aggregation which are now present in many newer benchmarks: NIAH (Kamradt, 2023) includes retrieval, LongBench (Bai al., 2023) in- cludes bilingual tasks, Bamboo (Dong et al., 2023) has textual entailment tasks amongst others and M4LE (Kwan et al., 2023) has tasks testing translation and classification, L-eval (An et al., 2023) includes a task on multi-document dialog, and Loogle (Li et al., 2023) includes a computation task. However, while of these benchmarks combine many existing datasets cover range of tasks none of them operate on data from the scientific domain. Further, for very long (100k+) contexts synthetically crafted data used, c.g. Ruler (Hsich et al., 2024) proposes synthetically created length adaptable tasks and Ada-Leval (Wang al., 2024) includes length adaptable sorting and QA tasks, where they add distractor texts to increase the context. These ignore data from scientific literature that's naturally complex and requires processing long context, see Fig. 2(a) for comparison.
Scientific literature. Of the tasks most relevant to scientific expertise, QASA (Lee et al., 2023) and QASPER (Dasigi et al., 2021) operate on full scientific paper, Machine Learning (MI.) and Natural Language Processing (NIP) papers respectively, however these focus solely on question answering, since expensive and labor-intensive collect tasks requiring expert knowledge. In our work we introduce ten new tasks curated from six disciplines, all annotated by experts( (with Ph.D. degrees) and requiring reasoning over long context information, on average about 15k words (Fig. 2c). The outputs also include structured and unstructured text averaging about 1k words (Fig. 4c).
CURIE DATASET AND TASKS
The CURIE benchmark consists of series of tasks that measure how well LLMs can assist in diverse scientific workflows, from synthesis of information towards final execution anchored on single scientific research documents. Each task in the benchmark: (1) a realistic task performed by scientific experts on domains requiring years of study, (2) has information relevant solve the given problem within the context provided (c.g. a full-length scientific paper. image/caption pair), and (3) ensures expert humans evaluate task performance, providing metrics that highlight the potential limitations current models. Summary of data, tasks, and metrics are in Appendix A.
Collection guidelines. We selected six domains requiring deep scientific expertise: materials sci- ence, theoretical condensed matter physics, quantum computing, geospatial analysis, biodiversity, and proteins. Within these, worked with 1-3 experts to define tasks representative of realistic scientific workflows, covering the following seven assessment categories: concept extraction, con- cept tracking (co-reeerence resolution), aggregation, algebraic manipulation, sunmarization, visual comprehension. and integrating expertise across domains. We focused on tasks that, if successful, could enable automation (Donoho, 2024) of a time intensive critical component of a workflow e.g- extraction of experimentally reported values towards curating database (Dong & Cole, 2022b), or generate code or calculations to fully reproduce computational or theoretical analyses. We worked with domain experts on 3 critical aspects of the task preparation: (I) sourcing papers representative of the task and domain; (2) creating ground truth answers that were accurate, nuanced and compre- hensive; and (3) identifying measures evaluate model responses against ground truth answers that properly captured salient features of the task which the experts also used for rating the difficulty of each example. Figure 2(b) shows the distribution and details of tasks in the CURIE benchmark.
3

---

Published as a conference paper at ICLR 2025
Cow dance of deping and SPAI OE poores bekkeror
Land welleeiriggh Lork Engine Claad Compuling
Density Functional Theory (DFT)
Hartres-Fock Derivation (HFD)
Quantum Correction Codes (QECC)
Geospatial analysis (GEO)
DFT-8: Idernily input structures. identify calcuiations params.
the outlined staps, and derive tomiltonian step
Extract information for all datasats used in the papar (source, variables, limelppatial ranges)
Ail a YAMI fie with the error correction code's propertios.
Write python code calculations
TT viter con oraur -Do0e tedorr aiitl
alayle
nober tppliste raalls
bethwerr" yaag eu TLA
sut codee eptimally
There
eve
Touth
Let MC Daate Aetem
sagi, 500.0
Ther, the Sefiend
vecisroppaionn
Seipe maane EnGaNN AE Br//00
1.00
shere the natrices V/A EV) are cts:
for Brrman sakee
"Migelt City,
relettors:
1in SoAEOL uni
perents:
Ce (4) 8G7
Figure 3: Examples of tasks in the CURIE benchmark. The DFT, HFD, QECC, and GEO tasks require the LLM to perform tasks on scientific papers (top blocks), described in the task snippets (in orange), extract, calculate, or aggregate information. Expected output (ground truth) snippets are shown in the blue blocks. (Only snippets of the query /outputs are shown for illustrative purposes.)
We provide a brief motivation and describe tasks for each of the domains. We identified papers with permissive licenses and worked with to experts in each domain to select specific papers, and to perform ground truth annotations. Data collection details are included in Appendix F.
Density Functional Theory Task (DFT). Density Functional Theory (DFT) is widely used frame- work for quantum mechanical modeling of materials, enabling first-priniiples predictions and val- idation of experimental findings. We define 3 tasks that measure the ability of LLMs reproduce DFT calculations from papers: (1) extracting input material structures (DFT-S); (2) identifying DFT calculation parameters associated with computation steps (DFT-P); and (3) translating computational steps essential for reproducing key results from the paper into functional code (DFT-C), Executing all these tasks successfully requires the LLM to comprehend domain-specific concepts, extract in- formation dispersed across different sections of the publication, and generate scientific code. Two Ph.Ds with expertise in DFT computations identified 75 papers and prepared solutions for the tasks.
Material Property Value Extraction (MPV). While historically material properties have been tab- ulated in books (Welsch et al., 1993; Kutz, 2002), the published literature is an untapped resource, with experimentally reported materials, structure information, properties, and processing conditions. Human curation of such data is time intensive and expensive, and rule-based automation is limited in scope (Swain & Cole, 2016) as it miss crucial processing details. However, prompt-based LLM extraction has shown promising carly results (Zheng et al., 2023) at sentence-level extractions. Our benchmark contains 17 scientific papers for exhaustively extracting material properties at the full document-level, annotated and verified by experts. The main task is to identify all instances of material properties mentioned in the text, including material name, descriptor and particular prop- erty, along with the passage or table where the property is described. For ablations, we consider variations to extract just a pre-specified set of properties (e.g. just refractive index and band gap).
Hartree-Fock Tasks (HFD, HFE). In condensed matter physics, Hartree-Fock mean-field theory framework for simplifying mathematical descriptions of interacting quantum systems. The framework begins with an interacting Hamiltonian and uses symbolic computations to reduce to a simpler non-interacting Hamiltonian. We construct two tasks: derivation (HFD) and extraction (HFE). HFD measures the ability of an LLM to derive the Fortree-Fook mean-fiell Hamiltonian for a quantum many-body system, motivated by prior work (Pan et al., 2024). Deriving the correct answer requires 13-19 reasoning steps, making it extremely challenging without expert oversight as the LLM needs to (i) identify the necessary steps to carry out the calculation and (ii) execute the steps, which requires symbolic computations and deep knowledge of theoretical physics. The
4

---

Published as a conference paper at ICLR 2025
Se protein
6F-
D6CC
Aniino Bold sequat
PFN
105% INSSFOKCCFUNN
LEOKELEGR
COTLOL SEA.LEEEAAAOOOOOONEELLLL
Ground Iruth wont
(a) An example from BIOGR.
(b) Illustration of the PDB task.
(c) Length of groundtruth outputs
Figure 4: (a) A map input from the BIOGR Task, (b) Example PDB task of reconstructing a protein's 's amino acid sequence from the 3D structure. (c) Length of ground truth per domain (avg. 954 words).
second simpler task, HFE, evaluates LLM's ability to identify and aggregate key equations from a research paper to extract the most general mean-field Hamiltonian. We have 53 papers (38 HFE, 15 HFD) with expert annotations including prompts for detailed reasoning.
Error Correction Zoo Task (QECC). The Error Correction (EC) Zoo 5.(Albert& & Faist, 2024) is an open source effort to build Wikipedia-like repository collecting and categorizing error correcting codes from the literature. EC codes are used to redundantly encode and store classical or quantum information so as protect from noise, with wide applications (e.g. in 5G communication and broadcast protocols). Creating an entry in the EC Zoo is a knowledge intensive process and requires listing the properties of given EC code along with any relations to other codes in literature. Each example contains many different types of information: a succinct non-technical summary, bespoke technical details, numerical parameters quantifying code performance, and non-trivial connections to other literature. The diversity in encoding schemes makes the task of selecting critical components for code-entries challenging to templatize. We construct a benchmark that tests the ability of LLMs to eurate the EC Zoo by taking given paper and asking it produce YAML code-entry file. Our benchmark consists of codes from 65 papers curated by experienced experts.
Geospatial Dataset Extraction (GEO). Geospatial analysts integrate various diverse datasets to answer complex questions. For example, a study of time-series snowmelt detection over Antarctica may combine satellite imagery, radar data, weather station temperature data, elevation/hopooraphyy information, etc (Liang et al., 2021). ). In this task, given a research paper, the LLM is required to identify all utilized datasets, including source websites, variable names, descriptions, time ranges and spatial ranges. This requires not only recognizing direct dataset references within the text, but also contextualization, comprehension and aggregation of information scattered throughout the paper, pushing the boundaries of long-contess understanding. Our benchmark includes 19 papers ranging across earth observation, economics, epidemiology and public health, along with detailed ground truth annotations necessary to reproduce each study.
Biodiversity Georeferencing Task (BIOGR). Critical geospatial information is often conveyed exclusively through maps, highlighting the need for powerful tools capable of interpreting such visual data. In this task we study the ability of multimodal LLMs to work with geographical maps. Specifically, we investigate the core capability of georeferencing. where, given image of map and its associated caption, the task is to determine the latitude/longitude bounding box encompassing the region displayed. A domain expert would often use multi-step process and specialized mapping tools (e.g.. QGIS, AreGIS), zooming in and switching between different imagery layers to find recognizable landmarks (river bends etc.) For this multimodal task, we assembled dataset of 138 map images (c.g., Fig. 4a) and captions from papers in ecology, of varying difficulty with ground truth labels for bounding boxes verified by domain experts.
Protein Sequence Reconstruction (PDB). This final task tests the ability of an LLM to extract meaning from three dimensional structure, associating the 3D structure of a protein with its se- quence. Given the 3D structural coordinates of a protein, provided in the Protein Data Bank (PDB), capturing the precise arrangement of atoms within complex molecule, we ask the LLM to re- construct the protein's amino acid sequence. To create the benchmark we curated the DB file to contain only coordinate information, stripped of any explicit functional annotation. This minimalist approach forces the LLM to rely solely its understanding of structural patterns to deduce the underlying amino acid sequence. We curated 64 structures with annotations for this task.
5

---

Published as a conference paper at ICLR 2025
0.9
LangtLala
0.8
Clocde (0ppua)
0.7
0.6
DF15
DFT-P
DFTC
MPV
HFD
HFE
QECC
GEO
BIOGR
FDB
Task
Figure 5: Per task normalized scores of various LLMs on the CURIE benchmark that measures performance of LLMs on 10 long-context tasks requiring expertise across six scientific disciplines. DFT-S, DFT-P, and MPV scored using LLMSim, while others use programmatic metrics.
Annotations. The annotations include ground truth solutions for each task, and difficulty rating for each example based on complexity of information extraction and reasoning (details in Appendix F). On the extraction tasks, such as MPV, HFE, GEO, and DFT-S, experts within each domain reviewed each others' work and reported a high rate of agreement.
4 EXPERIMENTS AND EVALUATION
4.1 EXPERIMENTAL SETUP
We evaluate the CURIE benchmark tasks several state-of-the-art LLMs supporting long-context windows. including five closed weight LLMs such as GPT-4o (OpenAl), Claude-3 Opus (An- thropic). Gemini 1.5 Pro (Reid al., 2024), and three open-weight LLMs. Mixtral (Jiang et al., 2024), Command-R-+ (Cohere). and LongI (Tworkowski et al., 2024). We follow stan- dard zeroshot prompt template across tasks, first describing the task the model needs to perform and the desired output format, and then providing the text of the full paper (except for LongLLaMa-3B full paper was provided first). the case of DFT and MPV tasks, we provide the output format in the context of an additional hand-crafted excerpt clarify expectation of formats for each field. The BIOGR task is multimodal, and for this we provide just the image and caption as input, rather than the full paper. Performance reported for each model each task using single run.
4.2 EVALUATION METRICS
For the tasks requiring generation of long English text responses (eight out of ten tasks), we the ROUGE-L (Lin, 2004) and BERTScore F1 (Zhang et al., 2019) metrics. For BIOGR and PDB other more appropriate quantitative metries. BIOGR uses Intersection-over Union (IoU), defined as the area of intersection between two bounding box regions (using latitude and longitude) divided by the union. This metric captures the similarity between two polygons, accounting for both location and size, while also being scale-invariams normalized distance eиror between the predicted and ground truth boxes Appendix Tab. 7. For the PDB task, we compare reconstructed sequences to ground truth sequences (Cock et al., 2009), using pairwise sequence alignment scored using the number of identities. The raw scores are normalized by the alignment length to account for potential length discrepancies, yielding the identity ratio (ID,) metric.
We also present additional metrics such as the
Computing average performance across all tasks in the CURIE benchmark. Appendix Table 4 reports the primary metric for each task used to compute the average performance. DFT-S, DFT-P, MPV use model-based Fl scores (Sec. 4.3), BIOGR uses IoU, PDB uses Dr, the rest use ROUGE- L. All metrics are normalized to [0.1]. Fig. shows performance of models par-task and Fig. I(a) shows average performance of models across all 10 tasks.
Human evaluations. Additionally, experts identified a set of rubrics for evaluating model responses and compared responses. E.g, for the tasks requiring extraction of information such DFT-S, DFT- and MPV, experts measured precision and recall of retrieved information relative to ground truth. Experts also rated 10-12% responses in each task with overall "good", "ok", "bad" rating.

---

Published as a conference paper at ICLR 2025
4.3 MODEL-BASED EVALUATION OF MIXED LONG FORM RESPONSES
Tasks in CURIE are varied and have ground truth annotations in mixed and heterogenous outputs. Evaluating free-form generation is challenging because answers often descriptive, and even when a format is specified as in most of our cases the response to each field can have differing forms e.g. in case of materials grid points may sometimes be specified as [p.q.]] and other times as pxqxr. Hence most existing knowledge related benchmarks (Hendrycks et al., 2020; Rein et al., 2023) lean towards multiple choice format for answers. While these allow for clean evaluation, this doesn't allow us to evaluate the full expressiveness of the model. Inspired by the ability of LLMs to evaluate natural language (Zheng et al., 2024; Liu et al., 2024; Kamalloo et al., 2023), three recent approaches, LAVE (Mañas et al., 2024), LIMA (Zhou et al., 2024) and Prometheus- Vision (Lee et al.. 2024), utilize the in-context capability of instruction- -astree LLMs to rate the candidate answers in 3-point, 6-point and 5-point Likert scales, respectively. While evaluation on Likert scales is suited for generated text, many of the outputs on our task have structured information that could benefit from more fine grained evaluation. So we propose two model-based evaluations (i) LLMSim a nuanced score for measuring similarity of elements in lists of dictionaries, which can then be used to compute precision and recall of elements, and (ti) LMScore an overall weighted score on a 3-point scale obtained by asking the LLM if the predictions match ground truth. We focus on LLMSim here since it provides a quantitative measure for structured outputs missing in previous work and discuss LMScore in Appendix D.
LLMSim is used compare similarity of dictionary elements to assist in comparison of sets of dictionaries. Our goal is to identify the number of ground truth dictionary items that have been retrieved correctly. So, we ask the LLM examine all of the predicted dictionaries and match and identifyy the predicted dictionary most similar to the each of the ground truth dictionaries. We use thain-of-though (CoT) prompt that asks the LLM to identify the predicted dictionary indices that correctly match each field (key) of the ground truth, and then select the predicted dictionary index most similar to the ground truth or output 'None'. Prompt and code are included in the supplement.
Concretely, suppose Dp is the set of predicted dictionaries, DG the set of ground truth dictionaries, ILMSim helps find the optimal matching M between the predicted dictionaries and each ground truth Dy Do:
LLMSim (Dp, D,)
{ [Nooe no match in values
Dp E Dp arg max s(fi, Dg) where ft represents the 7th field (key) in the dictionary and s(fi, Dy) is the similarity of the value of each field of D, with Dy- Given the matching, we then compute precision, recall and
Fl as
Pr= (Dp, D₂) E M
(g),.D)e M
Pr Re
Re
FI = 2
Dpl
Doo
Pr + Re
5 RESULTS
Main Results. Fig. shows the performance of all models averaged across all tasks in the CURIE benchmark. Claude-3 Opus is the best performing with consistent high performance across all tasks. Fig, 5 and Table 1 show task level performance of all models. The popular GPT-4o outperforms the others on GEO and BIOGR, however it's performance on PDB and HFD is surprisingly low. On closer inspection
found the GPT-4o model exhibited repetition in the outputs in the PDB task (see. Fig. 6), clipped responses in the MPV task, and failed to follow formatting instructions on the HFD task leading to lower performance. Overall though, most of the closed models had similar performance, and given the variability (e.g.. 25%-75% error bars around the mean in Fig. 5) the difference between them not significant. On several tasks there is considerable room for improvement, making CURIE an interesting benchmark for furthering model development.
Room for improvement. Fig. (bb compares performance of models from two different generations, Gemini 1.0 pro (32k) and Gemini 1.5 pro (IM+ context window) on popular benchmarks evaluating linguistic capability Dua et al. (2019), breadth of knowledge (Hendrycks et al., 2020), and expertise benchmark evaluating expertise
in science (Rein et al., 2023), alongside the performance with long-com long-context comprchension. We observe that there is considerable room for improvement on the types of realistic complex scientific tasks the CURIE benchmark provides.
7

---

Published as a conference paper at ICLR 2025
DFT
MPV
HFD
HEE
QECC
GBO
BIOGR loUl
PDB
Method
R-1.
B-F1
R-I.
B-F1
R-1.
B-F1
R-I.
B-F1
R-I.
B-F1
R-1.
B-F1
ID,
Zero-shot Open Weight LLMs
Mixtral
12.20
0.75
12.48
0.82
3,78
0.27
9.15
0.47
5.11
0.17
20.23
0.67
0.00
0.03
Command-R+
13.79
0.79
15.67
0.83
5.93
0.82
41.23
0.85
15.88
0.67
21.36
0.78
0.00
0.07
LongLLaMa
8.17
0.77
9.28
0.80
4.36
0.78
10.33
0.77
9.38
0.76
9.53
0.77
0.00
0.00
Zero-shot Closed Weight LLMs
Gemini 1.0 Pro
11.22
0.78
17.37
0.83
0.53
12.95
0.69
0.27
22.56
0.79
0.34
0.00
GPT-40
13.30
0.78
12.74
0.83
9.81
0.81
48.93
0.85
20.02
0.70
27.30
0.80
0.37
0.13
Gemini 1.5 Pro
13.66
0.78
31.54
0.82
18.86
0.84
39.56
0.71
20.07
0.76
27.24
0.74
0.42
0.29
Gemini 1.5 Flash
11.31
0.76
12.11
0.79
22.86
0.84
41.92
0.78
23.50
0.81
27.76
0.78
0.36
0.15
Gemini 2.0 Flash
14.18
0.79
14.72
0.84
22.52
0.84
50.06
0.87
22.36
0.79
29.69
0.79
0.49
0.56
Claude 3 (Opus)
13.78
0.80
15.86
0.83
16.82
0.83
49.10
0.87
21.81
0.72
28.66
0.79
0.39
0.29
Table 1: Results comparing performance of all models on all tasks based automated metries
R-L: Rouge-L, and F.F.:BerScooreFF. The avg. performance of all DFT tasks are reported under DFT. All models support context length of 32k or more. BIOGR has multimodal inputs which is unsupported by the chosen open models.
Blue highlights the highest values.
Ground Truth
"1d":
Gemini 1.5 Pro
0.478
ground_truth
MEEKTEF-DVIL -VI-EE--VVI---------EE A-LL-K----AMM--WWA VAV-I KAY RGA- VELLEAAAEEESPPAAAOORRROOAA EAAA:E/AAEOAAAAAEEEEEEEAEE-EEEEE
-A --EEE----- -1|1--1--11111
III
predicted
WWW.EAIERRRLLS---LLWWW------ ToL
WWW.EBLEWWSEEEEEO
Claude 3 Opus
ID 0.416
ground, AMEEKTEFD VILKAGGANNAAV KAVRGATOLOLKES
ESA PAALK EDVEKOD4EALK ELEAGG
predicted
.. FOFVILKA
EALEXO AAE KKPAALKSEVVSKD--A.
GPT-40
0.037
ground
MEEKTEFD-VIIL-KE KAA--AAAAA--A GA-
predicted
EKADLY EGA (VEEVAA repeated times)
Figure 6: Comparing responses on the PDB task. Measuring alignment using identity ratio (ID,) between the predicted amino acid sequence and groundtruth sequence for a given protein structure, Gemini 1.5 pro and Claude were better at predicting the sequence of amino acids, whereas GPT-4o collapsed into a mode of repetition, and Flash 2.0 generated code to solve the problem.
Human vs Model-based eval on retrieval. On the information retrieval tasks: DFT-S and DFT-P tasks which requires L.MS to retrieve material structures and DFT parameters from a given paper; as well as the MPV tasks requiring models to retrieve material property and values, we use I.L.Sim to compare the dictionaries of extracted material properties. Table 2 reports precision, recall and Fl scores computed after matching elements using LLMSim. We found the precision and recall to closely match those measured by human experts (on the Gemini 1.5 pro and GPT-4o models).
Human evaluation of model responses. We worked with experts in each domain evaluate pre- dictions generated by the models against ground truth responses on a 3-point scale identical to the proposed LMScore (in Appendix D). For each example, the expert "good" if it had few or no errors compared to the ground truth, "okay" if it had many minor errors, and "bad" if there were major emors.. We these human responses compare and correlate the newly proposed LMScore which is reported in the Appendix (App. Fig. 10 and Table 5). While LMScore appears be promising, requires further analysis prior wider usage.
asked to rate a response as
Performance vs. Difficulty Experts in each domain independently determined and rated the diffi- culty of answering each input example on 3-point scale as "easy", "medium", or "hard". In most cases, such as with the MPV, HFE, QECC, and DFT-S extraction and aggregation tasks, difficulty was determined based on how dispersed the information was within the paper. "Easy" examples had answers often within a section page, while "medium" cases could be spread across mul- tiple sections, and if the information required knowledge of specific literature outside of the given context the example the number of implementable functions to the total number of functions mentioned in the paper was used. For HFD, the number of reasoning steps and for GEO the number of datasets also fac-
rated "hard". Additionally for the DFT-C code generation task, the ratio of
8

---

Published as a conference paper at ICLR 2025
tored in, and for PDB the length of the sequence determined complexity. Fig. 7 reports performance of each model sliced by difficulty. Overall, models perform substantially better on easy examples compared to the medium and hard examples. Models appear to perform about the same on examples marked medium or hard. Though, one thing of note is that there are usually many more medium examples than hard examples across all tasks.
Enty
Medurm
0.1
0.0
Scree3b/N 3
medel
Figure 7: Avg. performance of models sliced by diffiicutyy of examples. Consistent with expec- tations, all models perform substantially better on easy examples except in the case of Mixtral. For most domains, experts independently converged measuring difficulty for each example based on how spread-out the requested information was within the context of the full paper.
DISCUSSION
Model responses lack robustness in instruction following. common observation across tasks was that, there is variability across model runs, even though average performance remained fairly constant. The variability is usually higher harder tasks. Instruction following remains a chal- lenge: Models often had pieces of the right answer, but were unable to consistently format it despite examples in the prompt. rare cases, even though we explicitly asked models to provide answers when they weren't t sure, they often refused to venture an educated guess. The model-based evalua- tion metrics were quite helpful in mitigating issues arising from lack of adherence to instructions.
Performance on retrieval. Fig. 5 and Table 1 report performance of the models on each task in each of the domains. Noticeably all models show high ROUGE-L scores on HFE which is a variation of the needle-in-a-heee seedle-in---haystace haysthhk problem where the model needs to extract related equations that might be spread throughout the paper. On tasks requiring exhaustive retrieval of multiple values and aggregation, e.g., DFT (see App. Fig. 14), MPV, and GEO, the models have considerably lower performance than just single value retrieval tasks (e.g., HFE).
Concept tracking, aggregation, summarization. On tasks requiring concept aggregation and tracking, e.g. DFT-P, GEO and QECC, experts found responses from some models quite promising. With DFT-P, Claude-3 uppeared to understand the purpose of DFT calculations better and grouped relevant parameters to appropriate functions (see App. Fig.8). On QECC, the experts noted that the summaries generated by the LLM tended be succinct while also including multitude of key informational "nuggets" and quantitative measurements, While not all of these were correct or im- portant, experts noted that would be easier to exclude the wrong bits (after examination) but harder to extract and comb out such details from the paper. On the GEO task, the closed models did well
DFTS Rec.
DFT-P
MPV
MPV-non-trivial
MPV-specific Rec.
Model
Pr.
FI
Pr.
Rec.
FI
Pr.
Roc.
F1
Pr.
Rec.
Fl
Pr.
F1
Zeru-shot Open Weight LLMs
Mixtral
24.96
23.30
22.67
9.12
6.13
7.09
31.86
23.29
22.82
29.70
21.14
22.31
22.20
35.05
22.64
Command--RR
32.19
6.92
5.41
22.64
20.80
3.87
6.31
4.52
18.18
LongI.LaMa
1.26
1.47
1.36
2.99
3.95
3.13
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Zero-shof Closed Weight LLMs
Gemini Pro
11.19
12.62
10.93
23.1
21.01
20.56
31.28
32.92
31.00
36.41
34.92
31.78
24.86
38.76
23.26
GPT-4o
36.96
29.50
30.63
27.93
19.66
22.13
39.22
24.14
25.90
45.10
24.08
30.05
32.35
21.77
22.97
Gemini 1.5 Pro
36.04
33.67
32.11
23.67
16.53
19.00
23.86
38.36
26.85
31.74
42.60
30.08
25.00
31.34
24.48
Gemini Flash
33,07
35.28
17.91
23.16
15.82
17.76
Gemini 2.0 Flash
31.38
40.46
32.39
8.22
7.74
7.68
35.84
46.56
36.99
30.81
47.76
33.79
26.48
33.64
24.37
Claude (Opus)
40.45
32.89
33.76
27.26
17.17
19.87
41.35
35.60
34.04
45.64
43.67
38.32
32.18
47.06
31.48
Table 2: Retrieval performance using LLMSim On tasks requiring exhaustive retrieval of infor- mation we use LLMSim and compute Precision, Recall, and Fl scores on each document and report the mean. We also include ablations for the MPV task where we ask the LLM to retrieve non-trivial or specific property values (refractive index and optical bandgap) for materials.

---

Published as a conference paper at ICLR 2025
Claude 3 Opus
GPT-4o
cere are the tecalls of the OFT celociations in the paper in the spesifled JEEN fornot:
funttionu "create. oft wtt..et.pppccetrrppp..." "Ipay"
"uacctiona" p9a1
Aoposeto 21"..11'
aoftward" "olaa
Tenarsy: porry_.oonrrggnnt hal"
foror_.oeverreens" 9.81 diik".
1.6.
"otterainfe fooad stacing A. vor enthied used deseer effect of cerer cores or valence ecectrone. persseac siao of storae (139) platse uses, with store consurition
"other, oober.antorattan"" uree spacang or 1 A. vi rethes, exerisents) poraneter of s at 4 for ou extrax, stoes three outernost stanse Lavers alloned ralan, perandie toundary contations directione."
lavere
surtaoa stono).
fapection rome wtw.ol.o....roooooooo. www.rraattrooprrooer....o...
"haection new "create, elf, www.eterr......ccmmmmmm rime"
(2,2,1), (2,4.1), 12,6.1), (8,4.1) (4.6 1J",
"Functioni" nest
"Comergenoe tested with respect kossinte, grod spacting Biieiing energles tites free anrrg witian wwiikr..
atter matten"
"-ppssns"" "[,,2,11
surfane
"other Infor wittan: oree spacing of 8. na nothod, paraneter of a 51 A for DJ estrix, stons in three auternost lapers allomet ralax, periodle toundry contitions
rome" cae wateuaatiosss funtrssslenssss..ttt..
and directiare."
Farest "exteed 7861, S76C, x0n-072". k-peinte" "Kos"
"unctton, new "create dtt.aarraaeterra.......
"atter, into wetterr: "hatted imsect of acchenge-tormenn melectto nunettored are inclexton
"unetionn"" -Je IF2",
nise. viPromaeee Linling eneratee for delected und rise villans.
overkinds
posets" "1.2..1"
scl persodso 6335. stents planes reparisentsi lattice
scchnne-corte antto fenttionll 12,7. 1) s-point orin, and ferce consengence critersa horport
porancter
Ou mecrke. ctoms three lapers allomed na2az, and diract:one."
coneitions
coevergenne mezzring, smgast of exchange correiation fanctional was teated. fecatte like eneraw estoff sere not penvi ded
2ria spacing.
Figure 8: Example of model outputs for the DFT-P parameter identification task. Claude-3 Opus appears to understand the purpose of the calculations better than the other models and avoids unnec- essary repetition. Claude-3 correctly (green) identifies that there one set of DFT parameters used in the actual study as well as two more set of parameters which are used for convergence testing.
to extract some of the important datasets with the correct spatial and temporal ranges (Fig.25) but performance degrades when multiple datasets are used to cover larger spatial extent (App. Fig.27). Overall, carefully engineered prompts and agentic workflows could be effective on such tasks.
Closed vs Open models. One thing of note is that on QECC, DFT, and MPV extraction and ag- gregation tasks, the Command-R+ open weights model which uses retrieval-augmentee approaches shows performance similar to the closed weight models. The evals on BIOGR highlight that open models are yet to support both multimodal and long-context capabilities which can enable more scientific applications. On PDB, Mixtral performed higher than GPT-4o which is quite surprising. Both LongLLaMA and Command-R++ failed to produce any sort of FASTA format the PDB task. They either failed to fully understand the task, or missed steps during aggregation.
Reasoning in code. On the PDB task, remarkably, the more recent Gemini Flash 2.0 model, not only understood the task but seemed to realize the task is better solved by using programming and wrote python function (which executed correctly) about half of the time. On other half of the PDB inputs, Flash 2.0 decided to aggregate and generate the amino acid sequence similar to other models.
Overall, across tasks, model performances have room for improvement and we discuss specific examples and failure cases in appendix F for each task.
CONCLUSION
In this work we introduce the CURIE benchmark. A series of tasks designed to measure the ability of LLMs on scientific reasoning and understanding long-contexts. Our main contributions are (i) A new benchmark of 580 examples from 429 research papers that can assess LLMs on comprehension of long-context information from across six scientific disciplines requiring deep expertise. (ii) 10 realistic tasks combining concept retrieval and extraction, concept tracking, aggregation, algebraic manipulation, and expertise across multiple domains measure capability of models on different aspects of scientific workflows. (iit) We propose model-based evaluation metrics to address the challenge of automatically evaluating complex mixed-format heterogeneous outputs and compare them with human evaluations. (iv) We share guidelines for curating such multi-step tasks and eval- uating annotation quality of such complex answers. We hope the diverse tasks and rich annotations in the CURIE benchmark serve the community in not only evaluating LLMs their scientific problem solving abilities but also advance research on scientific planning, instruction following, and evaluation of generated texts containing information of diverse types and formats.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
We make the data, prompts, and code available in https://gitthtn com/ google/ progle//crii e/curie under the Apache 2.0 license. Our dataset is available under CC-BY license. The dataset includes full text of the papers, the ground truth annotations, prompts (which includes prompts used to elicit model responses, as well the prompts used for evaluation using the proposed LMScore and LLMSim metrics), model responses and code for evaluations. The appendix includes additional information on each the tasks, annotation procedures, and examples of model outputs and failure modes.
DISCLAIMER
V.V.A. participated only as subject-matter expert for the QECC task. Certain equipment, instru- ments, software, or materials are identified in this paper in order to specify the experimental pro- cedure adequately. Such identification is not intended to imply recommendation endorsement of any product or service by NIST. nor is it intended to imply that the materials or equipment identified necessarily the best available for the purpose.
ACKNOWLEDGEMENTS
N.M. acknowledges support from the National Science Foundation under Cooperative Agreement PHY2019786 (The NSF AI Institute for Artificial Intelligence and Fundamental Interactions).
REFERENCES
Materialsproject.org,
https://docs.materiian http:///lcsmmmmiiaappeee/oo/oooe http:///wwwooooooooooooooooooo/o
2017.
Victor V. Albert and Philippe Faist (eds.). The Error Correction Zoo. 2024. URL https:// errorcorrectonnooooo
Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models. arXiv preprint arXiv:2307. 1088, 2023.
Anthropic. The Claude 3 Model Family: Opus, Sonnet, Haiku. Available online at: https://
www-cdn. ant.aaaicccccooooo comt//o.c..cooooooooooooooo..oct Model_ Model__arr_cllaude__ p.f..
Yushi Bai, Xin Ly, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.1450 2023.
Iz Beltagy, Kyle Lo, and Arman Cohan. Scibert: pretrained language model for scientific text. EMNLP, 2019.
Thomas Camara, Harold Ollivier, and Jean-Pierre Tillich.
A class of quantum Idpe codes: con- struction and performances under iterative decoding. 2007 IEEE International Symposium on Information Theory, pp. 811-815, 2007. URL https:// //as:///pp...// www.nt
org/
Corpus II:
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprini arXiv:2107.0337 2021.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafford. Think you have Solved Question Answering? Try ARC, the A12 Reasoning Challenge. arXiv, Abs//883.00547,, 2018. URL https:///// /api......//// sepiiseene semant icscholas /
CorpusTD:392281
11

---

Published as a conference paper at ICLR 2025
Peter JA Cock, Tiago Antao, Jeffrey T Chang, Brad A Chapman, Cymon Cox, Andrew Dalke, Iddo Friedberg, Thomas Hamelryck, Frank Kauff, Bartek Wilczynski, et al. Biopython: freely available python tools for computational molecular biology and bioinformatics. Bioinformatics, 25(11): 2009.
Arman Cohan, Waleed Ammar, Madeleine Van Zuylen, and Field Cady. Structural scaffolds for citation intent classification in scientific publications. arXiv preprint arXiv: vrXivv/904.00 1904.01604, 2019.
Cohere. Introducing Command R+: Scalable L.M Built for Business. Available online
https: cooere. wttochere..co//e... com/'o 1 og/ command- ruslus------- mas-mmiiee crosoft-azure
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. A dataset of information-seekingg questions and answers anchored in research papers. In NAACI, pp. 4599- 4610, 2021.
Qingyang Dong and Jacqueline M Cole. Auto-generated database of semiconductor band gaps using chemdataextracttor. Scientific Data, 9(1):193, 2022a.
Qingyang Dong and Jacqueline M Cole. Auto-generated database of semiconductor band gaps using chemdataextractor, Scientificc Data, 9(1):193, 2022b.
Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. Bamboo: A comprehensive benchmark for evaluating long text modeling capacities of large language models. arXiv preprint 1.X1::2309.13345, 2023.
David Donoho. Data science the singularity. Harvard Data Science Review, 6(1), 2024.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv 2019.
preprint
Alexander Dunn, John Dagdelen, Nicholas Walker, Sanghoon Lee, Andrew S Rosen, Gerbrand Ceder, Kristin Persson, and Anubhav Jain. Structured information extraction from complex sci- entific text with fine-tuned large language models. arXiv preprint arXiv:2212.0522 2022.
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH), 3(1): 1-23, 2021.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.00 03300, 2020.
Cheng-Ping Hsich, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.00660 2024.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04408: 2024.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: dataset for biomedical research question answering. In EMNLP, pp. 2567-2577 2019.
David Jurgens, Srijan Kumar, Raine Hoover, Dan McFarland, and Dan Jurafsky. Measuring the evolution of a scientific field through citation frames. Transactions of the Association for Com- putational Linguistics, 6:391-406 2018.
12

---

Published as a conference paper at ICLR 2025
Ehsan Kamalloo, Nouha Dziri, Charles LA Clarke, and Davood Rafiei. Evaluating open-domain question answering in the era of large language models. arXiv preprint arXiv:2305.06984, 2023.
Gregory Kamradt.
Needle in a haystack www.ingkaamaatttALiiiseeeeeeeeee
pressure 2023.
testing
LLMs.
J-D Kim, Tomoko Ohta, Yuka Tateisi, and Jun'ichi Tsujüi. Genia corpus-a semantically annotated corpus for bio-textmining, Bioinformatics, ([suppl_1):i180-11 2003.
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:3 2018.
Martin Krallinger, Anastasia Krithara, Anastasios Nentidis, Georgios Paliouras, and Marta Villegas. Bioasq at clef2020: Large-scale biomedical semantic indexing and question answering. In Ad- vances in Information Retrieval: 42nd European Conference IR Research, ECIR 2020, Lisbon, Portugal, April 14-17, 17, 2020, Proceedings, Part 11 42, pp. 550-556. Springer, 2020.
Jens Kringelum, Sonny Kim Kjaerulff, Søren Brunak, Ole Lund, Tudor I Oprea, and Olivier Taboureau. Chemprot-3.0: global chemical biology diseases mapping. Database, 2016:bav 123,
2016.
Wojciech Kryšciński, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. Booksum: collection of datasets for long-form narrative summarization. arXiv preprint arXiv:2105.08209, 2021.
Myer Kutz. Handbook of materials selection. John Wiley & Sons, 2002.
Wai-Chung Kwan, Xingshan Zeng, Yufei Wang. Yusen Sun, Liangyou Li, Lifeng Shang, Qun Liu, and Kam-Fai Wong. M4le: A multi-ability multi-range multi-task multi-domain long-context evaluation benchmark for large language models. arXiv preprint arXiv:2310. 19240, 2023.
Seongyun Lee, Seungone Kim, Sue Hyun Park, Geewook Kim, and Minjoon Seo. Prometheus- vision: Vasion-language model as judge for fine-gran fine-grained evaluation. arXiv preprint arXiv:2401.06591, 2024.
Yoonjoo Lee, Kyungjae Lee, Sunghyun Park, Dasol Hwang, Jachyeon Kim, Hong-in Lec, and Moontae Lee. Qasa: advanced question answering scientific articles. In ICML, pp. 19036- 19052. PMLR, 2023.
Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Al- lan Peter Davis, Carolyn Mattingly, Thomas C Wiegers, and Zhiyong Lu. Biocreative edr task corpus: a resource for chemical disease relation extraction. Database, 2016, 2016.
Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. Loogle: Can long- contexttext language models understand long contexts? arXiv preprint XrXiv:231 A0ii:2111.0444 04499, 2023.
Dong Liang, Huadong Guo, Lu Zhang, Yun Cheng, Qi Zhu, and Xuting Liu. Time-series snowmelt detection the antarctic using sentinel-1 sar images google earth engine. Remote Sensing of Environment, 256: 12318, 2021.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.0225. 2023.
Yi Luan, Luheng He, Mari Ostendorf, and Hannanch Hajishirzi. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. EMNLP, 2018.
13

---

Published as a conference paper at ICLR 2025
Oscar Mañas, Benno Krojer, and Aishwarya Agrawal. Improving automatic vqa evaluation using large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 38, pp. 4171-4177 2024.
Juraj Mavracic, Callum Court, Taketomo Isazawa, Stephen R Elliiot, and Jacqueline M Cole. Chemdataextractor 2.0: Autopopulated ontologies for materials science. Journal of Chemical Information and Modeling, 61(9):4280-428,, 2021a.
Juraj Mavracic, Callum Court, Taketomo Isazawa, Stephen R Elliott, and Jacqueline M Cole. Chemdataextractor 2.0: Autopopulated ontologies for materials science. Journal of Chemical Information and Modeling, 61()):480-4229, 2021b.
James Mayfield, Eugene Yang. Dawn Lawrie, Sean MacAvaney, Paul McNamee, Douglas W. Oard, Luca Soldaini, Ian Sohoroff, Orion Weller, Efsun Kayi, Kate Sanders, Mare Mason, and Noah Hibbler. On the Evaluation of Machine-GGnnratte Reports, May 2024. URL httpt://arxii org/abs/ 22054 00982. arXiv:2405.00m: [cs].
Nicolas C. Menicucci. Fault-tolerant measurement-basee quantum computing with continuous- variable cluster states. Phys. Rev. Lett., 112: Mar 2014. doi: Ro..ee.. 122.24 URL https: htta://linke /t///iink..aa/// aps 120504.
/t0o//10... http://.....////eeeeeettttte 112.
Benjamin Nye, Junyi Jessy Li. Roma Patel, Yinfei Yang, lain Marshall, Ani Nenkova, and Byron C Wallace. A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature. In Proceedings of the conference. Association for Computational Linguisties. Meeting, volume 2018, pp. 197. NIH Public Access, 2018.
OpenAl. Hello GPT-4o.
https://openaa caaeeeaai.ccmmmmim com/index/ index/ Available online at:
hel helloo-pt-4o/.
Haining Pan, Nayantara Mudur, Will Taranto, Maria Tikhanovskaya, Subhashini Venugopalan, Yasaman Bahri, Michael P. Brenner, and Eun-Ah Kim. Quantum many-body physics calcula- tions with large language models. arxiv 2403.03154, 2024.
Maciej Polak and Dane Morgan. Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering -Example of ChatGPT. URL https: //aaxiv.org/pde..... /2303 05352 pdf.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530. 2024.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel Bowman. Gpqa: A graduate-level google-prood q&a bench- mark. arXiv preprint arXiv:2311. 12022, 2023.
Ari Seifter, Alison Schwarzwalder, Kate Geis, and John Aucott. The utility of google trends for cpidemiological research: Lyme disease as an example. Geospatial health, 4(2): 137, 2010.
Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. Zeroscrolls: A zero-shot benchmark for long text understanding. arXiv preprint 27114 2023.
Matthew C Swain and Jacqueline M Cole. Chemdataextractor: toolkit for automated extraction of chemical information from the scientific literature. Journal of chemical information and model- ing, 56(1)) 189181948111 1904, 2016.
Szymon Tworkowski, Konrad Staniszewski, Mikolaj Pacek. Yuhuai Wu, Henryk Michalewski. and Piotr Milos. Focused transformer: Contrastive training for context scaling. Advances in Neural Information Processing Systems, 36, 2024.
Chonghua Wang, Haodong Duan, Songyang Zhang, Dahua Lin, and Kai Chen. Ada-leval: Eval- uating long-context llms with length-un addatable benchmarks. arXiv preprint arXiv:2404.06488 2024.
14

---

Published as a conference paper at ICLR 2025
Gerhard Welsch, Rodney Boyer, and EW Collings. Materials properties handbook: titanium alloys. ASM international, 1993.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint 1905::900.077855 2019.
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai Chang, Peng Gao, et al. MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? arXiv preprint arXiv:2403. 14624, 2024.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluat- ing text generation with bert. arXiv preprint arXiv: 2019.
Xiang Zhang, Zichun Zhou, Chen Ming, and Yi-Yang Sun. Gpt-assisted learning of structure- property relationships by graph neural networks: Application to rare-earth doped phosphors. arXiv preprint arXiv:2306. 14238, 2023.
Yuzhou Zhang, Gabriel Milinovich, Zhiwei Xu, Hilary Bambrick, Kerrie Mengersen, Shilu Tong, and Wenbiao Hu. Monitoring pertussis infections using internet search queries. Scientific reports, 7(1):10437, 2017.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging alm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
Zhiling Zheng, Oufan Zhang, Christian Borgs, Jennifer T Chayes, and Omar M Yaghi. Chat- gpt chemistry assistant for text mining and prediction of mof synthesis. arXiv preprint arXiv:2306. 11296, 2023.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
Andrew Zhu, Alyssa Hwang, Liam Dugan, and Chris Callison-Burch. Fanoutqa: Multi-hop, multi- document question answering for large language models. arXiv preprint arXh::2402. 2024.
15

---

Published as a conference paper at ICLR 2025
A DATASET STATISTICS AND TASKS OVERVIEW.
In Fig. 2 (b), Fig. (c), and Fig. 4 (c) we present the distribution of the CURIE dataset terms of papers, and distribution of lengths of inputs and outputs per domain. Here, in Table. 3 we consolidate and present the number examples in each task and the average length the inputs and ground truth outputs. Table 4 presents brief description of the tasks, the capabilities necessary for the task, format of output, and both programmatic and LLM-based metries that are used to evaluate performance on the task.
Avg. length (# words) output (ground truth)
Domain
Task
papers
examples
input
Material Science
DFT-S
74
74
5818
232
Material Science
DFT-P
74
5818
132
Material Science
DFT-C
74
74
5818
1742
Material Science
MPV
17
17
1687
2188
Condensed Matter Physics
HFD
15
15
5385
1422
Condensed Matter Physics
HFF
38
38
8472
111
Quantum Computing
QECC
65
65
19913
207
Geospatial
GEO
19
19
7802
808
Biodiversity
BIOGR
137
138
20
Protein Sequencing
PDB
64
64
44028
Table 3: Statisties of the CURIE dataset. We report the the number of papers and examples for each task in each of the domains and also include the average length the input and ground truth outputs in words.
Output Format
Primary Eval. Metric
Task
Brief description
Capability
DFT-S
Extract input material structures for DFT calculations.
entity recognition, concept tracking concept extraction, tracking, aggregation concept aggregation, coding entity recognition, concept extraction, tracking concept extraction, alg. manipulation, reasoning concept extraction
ISON
LLMSim-FI
DFT-P
Extract parameters for DFT
ISON
L.MSSi-FFI
calculations.
DFT-C
Write functional code for DFT
TEXT
ROUGE-L
computations.
MPV
Identify all instances of materials,their properties, and
JSON
LLMSim-FI
descriptors.
HFD
Derive the Hartree-Fock mean-field
TEXT
ROUGE-L
Hamiltonian for quantum
many-body system.
HIFE
Extract the most general
TEXT (latex equation) YAML
ROUGE-L
mncan-field Hamiltonian.
QECC
Create YAML with the Error
concept aggregation, summarization concept extraction,
ROUGE-L
Correction Code's properties.
GEO
Extract information for all
ISON
ROUGE-L
geospatial datasets used along with the spatial and temporal extents.
aggregation
BIOGR
Determine the latitude, longitude bounding box encompassing the region in the map image. Reconstruct protein's amino acid sequence form the 3D structure.
visual comprehension, reasoning
JSON (lat., lon. co-ordinates)
Intersection-
over-union (loU)
PDB
tracking, aggregation reasoning
Code or TEXT (seq.)
Identity ratio (ID,)
Table 4: Task details and capabilities required. A brief description of the tasks, capabilities assessed, output format, and the primary evaluation metric (used in Fig. and Fig. 5).
16

---

Published as a conference paper at ICLR 2025
CURIE: Science + Long Context (ours)
MathVista: Reasoning in Visual Context
ZeroScrolls: Long Context
MMLU-pro: Understanding
GPQA: Science Expertise
RULER: Long Context
100
80
60
Score
40
20
0
Claude (Opus) Gemini 1.5 Pro
GPT-4o
Figure 9: Comparing model performances on scientific QA and long-context benchmarks. We compare the performance of the top 3 closed models on CURIE against, 2 science QA bench- marks GPQA, Vista, improved language understanding benchmark MMLU-pro, and long-context benchmarks Zero-Scrolls and RULER.
B MODEL PERFORMANCES ON SCIENCE VS LONG-CONTEXT BENCHMARKS
In Fig. 9 we observe how performance of models on the CURIE benchmark correlate or compare with performance other scientific QA benchmarks and long- congecontext benchmarks. We specifically examine the top-3 models on CURIE (Claude-3, Gemini 1.5 Pro, and GPT-4o). We observe that performance of models CURIE is more in-line with the realistic ZeroScrolls long-context bench- mark and also the GPQA scientific question answering benchmark. CURIE appears harder benchmark than both Zero-Scrolls and GPQA.
C ANNOTATOR AGREEMENTS.
For all tasks we worked with experts to establish consistent output format over couple of initial iterations. We then had pilot phase followed by some adjudication clarify outputs. To evaluate annotator agreements, for retrieval tasks, we had annotators examine each other's s work after the adjudication. For the HFE and DFT-S tasks the agreement was near perfect (> 95%). There were minor differences in the exact phrasing chemical formula notation used. For MPV, we had two rounds of feedback and were then able to get an agreement over 80% on the materials to extracted comprehensively. For GEO, the agreement was again high after an initial pilot phase and discussion, agreement was over 90% on the datasets, spatial and time ranges. There were minor differences in the exact phrasing of the responses but these were discounted e.g. "for each year between 2012- 2015" vs "2012,2013.201.20055
D LMSCORE: A COARSE MODEL-BASED EVALUATION METRIC.
Evaluation on Likert scales provide a quick coarse signal of the quality the responses on gener- ation tasks. With LMScore propose an overall weighted score on 3-point scale obtained by asking the LLM if the predictions match ground truth and using the model's confidence (log-probs scores) get weighted score. Specifically, given the ground truth and predicted responses, the model to check if the predicted responses match the ground truth, and ask the model to output "good" (if the prediction has few minor errors), "okay" (if there are many minor errors), and "bad" if
ask
17

---

Published as a conference paper at ICLR 2025
there major emrrr. Instead of using the model generated response directly, compute score based on the model log-likelihood values. Iy represents the tokens for the categories we are interested in, 2₁ (bad, ok, good), and wy the corresponding weights want to assign each category, w, E {0, 0.5, }. then
LMScore
p(ae) x
t=
p(z;) is computed by renormalize the probabilities of the tokens by considering a so softma(() opera- tion on the log-probabilities of the tokens: (lbad, Lok: (good]). ). We consider an uncased version of the tokens and treat 'ok' and "okay" equivalently. We use GPT-4o as the LLM. When the tokens are not present in the top-5 log probabilities we compute an approximation based on the probability mass of tokens not present the top-5, We make the code for computation available
supplement.
task _TTaadtaaa task DFT_structun
task GEO
task HFE
task HFD
task QECC
1.00
GPT
0.50
0.25
0.00
Bad
Okay Good
Bad
Okay Good
Okay Good Human Evaluation
Bad
Okay Good Human Evaluation
Bad
Bad Human Evaluetion
Okay Good
Bad Human Evaluation
Okay Good
Human Evalustion
Human Evalustion
Figure 10: Correlation of GPT-4o based LMScore metric with human evaluations, Across tasks in domains where ROUGE-L is the primary evaluation metric, LMScore appears to be promising alternative to ROUGE.
DFT
MPV
HFD LMS
HFF R-L
QECC LMS
GEO
Method
R-L
LMS
R-L.
LMS
R-L
LMS
R-L
R-L
LMS
Zero-shot Open Weight LLMs
Mixtral
7.43
0.08
13.43
0.18
3.78
0.05
9.15
0.11
3.10
0.06
20.23
0.19
Command-R++
5.12
0.06
11.22
0.37
15.81
0.17
41.23
0.31
17.23
0.16
21.36
0.15
LangLLaMa
4.61
0.02
6.03
0.06
0.01
10.33
0.00
8.67
9.53
0.00
Zero-shot Closed- Weight LLMs
Gemini 1.0 Pro
10.52
0.06
13.69
0.30
18.10
0.02
36.60
0.10
17.48
0.05
28.44
0.18
GPT-4o
9.05
0.12
13.52
0.25
9.81
0.06
48.93
0.47
19.51
0.16
31.47
0.30
Claude (Opus)
0.11
15.54
0.34
16.82
0.13
49.10
0.50
20.54
28.63
0.24
Gemini 1.5 Flash
10.62
0.13
14.14
0.43
17.53
0.21
43.94
0.39
17.40
0.22
28.94
0.29
Gemini 1.5 Pro
10.83
0.12
14.12
0.41
18.86
0.14
39.56
0.33
18.03
0.23
30.25
0.30
Table 5: Results comparing performance of all models on all tasks based on the ROUGE-L metric
and the proposed GPT-4o based LMScore (LMS) metric. DFT reports avg. across all DFT tasks.
Human vs. Model-based 3-point evaluations We worked with experts in each domain to evaluate predictions generated by the models against ground truth responses on 3-point scale identical to the proposed LMScore. For each example, the expert was asked to rate a response as "good" if had few there were major errors. We use these human responses to compare and correlate the newly proposed LMScore which is reported in Fig. 10.
compared to the ground truth, "okay" if it had many minor errors, and "bad" if
Table shows performance of the proposed LMScore (LMS) for all models across all tasks. We observe that LMScore values show trends similar to what is observed in ROUGE-L (R-L). Table 6
E LIMITATIONS
This work focused on select set domains and a narrow set of tasks with high quality annotations, thus limiting the scale. Increasing the scale of examples across tasks would provide more robust
18

---

Published as a conference paper at ICLR 2025
DFT-C
DFT-S LLMSim FI
DFT-
DFT-AVG
Method
R-L
LMS
R-L
LMS
LLMSim F1
R-L
LMS
R-L
LMS
Zero-shot Open Weight LLMs
Mixtral
7.43
0.09
25.61
14.03
0.05
7.87
15.16
0.10
7.43
0.08
Command-R +
5.12
34.47
18.68
0.03
6.97
17.59
0.09
5.12
0.06
LongLLaMa
4.61
0.03
1.96
8.68
0.01
5.02
11.23
0.02
4.61
0.02
Zero-shot Closed- Weight L.Mss
Gemini 1.0
6.57
0.10
42.95
12.42
0.02
9.66
14.67
0.05
10.52
0.06
GPT-4o
9.05
0.16
33.30
15.70
0.04
24.76
15.17
0.15
9.05
0.12
Claude 3 (Opus)
8.38
0.16
36.28
16,87
0.05
21.81
16.10
0.13
8.38
0.11
Gemini 1.5 Flash
10.07
0.22
39.34
10.96
0.05
21.36
12.90
0.13
10.62
0.13
Gemini 1.5 Pro
10.83
0.21
36.99
14.55
0.03
20.89
15.59
0.12
10.83
0.12
Table 6: Results comparing performance of all models on the DFT tasks based on the ROUGE-L
metric (R-L), LLMSim (F1) and the proposed GPT-4o based LMScore (LMS) metric. highest score and Yellow second highest.
Blue is
evaluation benchmark. With the fast pace of language model advancements, evaluating the generated text responses on such complex tasks is challenging even with high quality human annotations. In particular, just based on instructions and output format provided in the prompt, existing automated evaluation metrics Rouge-I. and BERTS can be unforgiving resulting in low scores for responses that look different but might still be reasonable. While we propose model based evaluation metrics, these are still far from perfect and provides room for more creative strategies. Further, we primarily evaluate models in the zero-shot and two-shot settings (discussed in the supplement) and we invite researchers to explore retrieval augmented generation and chained prompting strategies that evaluate the models on planning and task decomposition.
Another limitation is that, it is difficult to obtain human performance the benchmark. These tasks require extensive expertise and it's unlikely that any one person has sufficient expertise across these domains. On the other hand, hiring an expert for each of the domains and evaluating their performance of the task would double the cost of annotation, so in this work we rely on annotator adjudication to come up with the final answers.
F DETAILED DESCRIPTIONS OF TASKS AND DATA
F.1 DENSITY FUNCTIONAL THEORY (DFT) TASK
Density functional theory (DFT) provides robust framework for quantum mechanical modeling of materials, enabling first-principles predictions and validation of experimental findings. Despite its widespread use and success in materials science, DFT calculations remain largely inaccessible to most experimental researchers, typically requiring a specialized PhD-level training in computational materials science. This knowledge gap arises from confluence of factors: the underlying quantum mechanical formalism; intricacies of numerical implementation and software-specific parameters; and the nuanced interpretation of results. While the theoretical foundations often covered in elec- tive graduate coursework, the practical application of DFT necessitates years of specialized training develop the intuition required navigate software complexities, select appropriate parameters and ensure the physical validity of results.
This benchmark represents critical first step towards harnessing LLMs for automating complex scientific workflows. While considerable interest exists in leveraging Al to accelerate scientific dis- covery, concerns persist regarding the accuracy and reliability of L.MSs in this context. Evaluating the performance of LLMs on intricate scientific tasks, such as DFT calculations, presents signif- icant challenge due to the domain expertise required for assessment. To address this, we introduce comprehensive framework for evaluating each stage of the DFT workflow from initial planning to final execution, ultimately aiming towards the complete automation of DFT calculations. Such automation would empower broader range of scientists including materials scientists, chemists and physicists to perform complex calculations without the need for extensive specialized training, enriching the theoretical foundations of experimental work and accelerating discovery.
19

---

Published as a conference paper at ICLR 2025
To assess the ability of large language models (LLMs) generate DFT workflows, we propose benchmark task requiring the translation of materials science papers into executable Python code, leveraging established domain-specifit libraries. This benchmark encapsulates three core subtasks: (1) iddntifying the computational steps essential for reproducibility; (2) extracting metadata, includ- ing input structures and DFT parameters; and (3) translating these steps and metadata into functional code. Subtask (1) evaluates the LLM's comprehension of domain-specific concepts and ability to plan complex procedures. Subtask (2) assesses its capacity extract pertinent information within potentially expansive context, where relevant details may be dispersed across different sections of the publication. Finally, subtask (3) gauges the LLM's s aptitude for scientific coding which requires not only proficiency in specific coding language and libraries but also deep understanding of the underlying scientific principles.
F.1.1 DFT DATASET COLLECTION DETAILS
We identified set of ~ 200 papers from the S2ORC corpus' that mentioned DFT computations in the abstract. We hired two expert annotators, who hold PhD degrees specifically in Materials Science and work directly DFT computations to annotate the papers. The annotators identified 74 papers which had DFT calculations to validate experimental results and also included a mix of theoretical understanding. Some of the papers that were discarded included papers that were proposing new DFT computation methods, or papers that were large focused on theory with just a passing mention of DFT computations. The annotators were asked to annotate all information necessary reproduce the DFT computations in the paper.
The annotators started by identifying the structures studied in the paper. They identified the graph of the computational steps carried out along with the inputs and outputs that went into each compu- tation. They used the inputs and outputs to perform a topological sorting of the functions create a computational graph of the different DFT computations done in the paper. They then implemented the functions the DFT computation in python code. In cases where the paper did not have suffi cient information, they marked such functions with placeholders noting that additional information was necessary and missing from the paper to fully reproduce that specific step. The annotators also identified analysis steps, and include code to perform the analysis functions. The annotators provide final code that includes the structure metadata, DFT params, and general helper functions as well as specific functions for each of the computation steps, and final block of code that executes the computations and the analysis in the sequence performed in the paper.
F.1 DFT TASK DETAILS
Our dataset is composed of three parts: calculation metadata for input structures, DFT computation functions and parameters, and code.
Metadata for input structures and DFT functions We capture the details of the inputs to each DFT calculation run in the workflow in two different types of metadata: structure metadata for the input structure, and DFT parameters for the DFT calculation settings. These subtasks are named DFT-S and DFT-P respectively and prompts are shown in Fig. 11 and Fig. 12.
Code for reproducing the calculation workflow We write python code that reproduces the calcula- tion workflow specified by the computation graph and the calculation metadata. The code is written in python, and we primarily use the Atomic Simulation Environment (ASE) library for setting up, manipulating, running and analyzing DFT calculations. We assume that the DFT software used in the original paper are available to run, and call the DFT software from ase. The prompt for this shown in Fig. 13.
S20RCC: https: //gi thmb. lenai /
20

---

Published as a conference paper at ICLR 2025
DFT-S Prompt
materials scientist ist would like reproduce the Der calculations from a paper.
They want to identify the input structures, and gather as much information about the structures possible.
output a list of the distinct input structures information with fields "id", "commen_name", "scientific_name" "type", "composition", "description", "vacuum.x"" "vacuumyy", "vacuum.z", "supercel1", "cas nuabeer,, "lattice.a", "lattice.b, "lattice.c", "space group", "orientation", in", "isomer name".
Make sure to identify all input structures,
The "id" field is just a string of format serruture.mtaddtt.fmmm where [number] starts from 1 and indicates the order the structure appears in the excerpt.
The "description" field should capture that is not captured by the other fields. Make sure to write your output as a list of dictionaries, NOT A BULLETED LIST. The "common-name" field should have the cenron name of the material and "scientific.name" should have the formal scientific matorial.
relevant information
the
The "type" field should point out whether the material is a molecule, a protein, a bulk crystal structure, a thin film or something else.
If the structure has vacuum around it, please include the thickness of vacuum layer in three directions with units in these fields: "vacuum x", "vacuum y", "vacuum 2".
The fields "Lattice.a", "lattice_b", "Lattice_e" correspond to the lattice parameters of the unit cell. Include units in these fields as well. Leave them blank lattice parameters are missing.
The "supercell" field should tell how many times the unit cell is repeated in the [x, 2] directicns, like "2x2x2". If the text indicates the material structure is from material project and provides "mp_id", please include that in the field "mp-id".
If the structure has multiple isomers and the text menticns which is used, plcase indicate that in the field "isomer.name". If any information is relevant but missing, input "UNKNOWN" in the field. If any information is irrelevant and mi missing, input None the field.
The "description" Field should capture all relevant information that is not captured by the other fields. Make sure to write output 1ist of dictionaries, NOT BULLETED LIST.
Example excerpt:
[Excerpt and additional electronic configuration details abbreviated]
Example output format:
"id": Etaructure.mttadatal11 "common name": "STO",
"scientific_name" "Strentium titanate",
[Additional fieids abbreviated: type, composition, all parameters) "description"; "pure SrTi03" [Additional structures abbreviated]]
Here is the paper: ([text]]
Identify the input structures, and gather much information about the structures as possible. Use the format from the example output.
Figure 11: Prompt for extracting input structures used in DFT calculations (DFT-S Task).
21

---

Published as a conference paper at ICLR 2025
DFT-P prompt
materials scientist would like reproduce the DFT calculations from a paper.
They want detailed information about each of the DFT calculations in the paper.
Output list of the distinct DFT parameters with fields including, but not limited to, "software", "functional", "k-points-grid", "pseudopotentials""
"basis-set", "energy-cutoff", "energy.convergence" "torre..onvergence" "relaxed.nuclei""
weelaxed.unit.cell. "spin", "hubbard
If structure relaxed, please set "relexednuclei" or "relaxed.unit.cel11
to 1.0 (corresponding to true).
If spin is involved in the calculation, "spin" to 1.0 Include the units these fields applicable. If any information is relevant but missing, input "NaN" the field.
If any information is irrelevant and missing, input "NaN" in the field.
Make sure write your output in JSON format.
Include any related information to the that has not been covered into the "other.information field.
Use the following format:
"funetion.name": wwwsshtt.futttiondssssetteo.eeee "software"; "Specify which software was used, e.g. vasp, gaussian, castep, qe, dmol, orca, wein2k", "functional"; "Sunctional name", "k-points": "k-point grid as [x,y,2
"energy-outofff: "energy cutoff in eV mentioned, else NaN", "enerrsy.convergencee. "energy convergence if mentioned, else NaN", "forreescovvrrgence" "force convergence if mentioned, else NaN", "relaxed nuclei": "1.0 if nuclei is mentianed to be relaxed, 0.0 if ment oned be fixed, else NaN",
"relaxed unit cell": "1.0 if it is mentioned to be relaxed, 0.0 if if mentioned not relaxed or else NaN ", "spin": "1.0 is spin considered, 0.0 is not considered, else NaN" "hubbard_u"; "NaN if not mentioned",
"other_information": "Any other relevant information the calculation." }
(Example excerpt abbreviated]
[Example output: Single DFT calculation with VASP, HSE06, key parameters shown]
Here is the paper: ([text]}
Using the specified format, extract and list out the details of all the calculations the paper.
Answer:
Figure 12: Prompt for extracting DFT calculation parameters (DFT-P Task).
22

---

Published as a conference paper at ICLR 2025
DFT-C prompt
am a computational materials scientist, and would like to reproduce the DET calculations from a paper.
Write python code to reproduce all DFT calculations from paper. You have access to the library ase, and any DFT software used in the paper.
Whenever possible, make the code medular by write functions for each step of the calculation
(i.e. set up the unit cell, run DFT, find the total energy from the DFT calculation, plot the band structure, ...) and calling the functions, instead of writing one long body of code.
Make sure all input structures, DET calculations, and calculation outputs such as energies, density of states or band gap are accounted for.
This is the paper I'd like to write the python code for: ([text]}
Output:
Figure 13: Prompt for generating Python code to reproduce DFT calculations (DFT-C Task).
Ground Truth
Gemini 1.5 Pro
BPT-4a
"censon, zone "anpto axp-',
aver" "chragge"
"satt letratmers aecasar, Coscotttloooeee Teckpdrocettoo
"N ii-eceityyin ceetyyiieeee 3 mestrri-11 1-phe--------------- 2- pyrasslan-3
www)-a-aettsee slayrine
oc15) 4 erimeettovrann"
"comssuaiiin" "suk". cryystai.cr
solonale".
coppestion:: "Li8111224""
"Sesttt Jagand: derived
"astaes ccnmoaa 1 "CT "cruowede" "Agand"
Frns ydoweetur
"comeon rame "[Ne (mpp (pppi--apppp001111 (D1))
'As": "structure netaders 2",
"etrarture caatsdetr..""
|MMIK0 (4 Liceyy withyy phhng) 2:p
"cueesffi hwt.://ww..........oooooeeeeeee. cioncelloriee
"tspe": C "srolated solacatar,
-eiinoocii pprttee "solacide"
"crystal.or. Lisolatee"s "soteouse",
Epperspt spttet naailT Togand serivl der derdeative and untoontioerdree
cecoriptian: "snurt cade lagres
"stracturs mecadets
Claude Opus
comme "npta-ispt
"sd": "structure awordate 17,
(3l thyl. phool pioe prizoli I-attesvtoortt""
'conosdttaan: "1211 221022
Typa "alladdaa"
"actentafic rave's aodty 2 piph
eny 2------mmazannn d-ore) -at-----nnnntttrraaeee
fres 4 ceri 5 metwvi 2 e-erroeeion are
"ype"" oo
"cccrraation City racolone Sptsli base
"anomr "Sarrse selr Fnvttoaaiann
'ST:: "structure rntabica 4" sstantsrio, "1aassse"
"dscrtgtsan" "Melyoderss stretratcl complex ligand"
"tyae" mileouia"
cerucnure, pecatern,
tenorsption:: cantttrpppaalpppeeeeee conoles
La iicunaee" 1 www.lollyleeerrrngoooooo
"eor artuuu
Me (n-corramy) wtwreyy---teeeeeeeee wwwe---oerrtt---e-ee wtw.eeetttttttttttyyly... hotpey/l---yyannneeee innd-
one)
tyshoyy) ligunds"
153": "srnetun straeten_eetattr...
salybeerue () ceagine with 2307
"compastionn "100000"
scuanss san "183300""
"mnttrroylaall (teeene)))))))))))
comples made ssing arotp assh ligare"
ceptorage coxtprcer/mmm/seeeeerr carales
4seerstton"
with arop-eeal 3igee""
Figure 14: Model responses the DFT-S structure metadata extraction task. Claude-3 Opus extracts accurate structures (green) relevant to the DFT computation whereas the other models do not precisely identify the exact structures that go into the DFT computation and tend repeat entries.
F.1.3 DFT TASKS: RESULTS AND ANALYSIS
Structure Metadata. (DFT-S) Many of the models did good job of picking out relevant in- formation about the chemical structures from the text. Qualitatively, the claude-3-opus-202444 20240229 model outperforms the others in its ability to understand how many structures are described in the paper. For example, on record 2023-09-22 1--000922-3-bttt 33bof900c3ff44314133deg rectly identifies two structures that are present in the text. The other models output many more struc-
2aude=3-opuu-22244229 20240229 cor-
23

---

Published as a conference paper at ICLR 2025
tures with repeated information. The same is true for records 2023-09- and 00t.-22227b446e2377711o.o
DFT Paramaters Metadata. (DFT-P) Again, all models do reasonable job of identifying rele- vant information from the text and structuring it properly as prompted. However, claude-3-opus- 20240229 appears to understand the purpose of the calculations better than the other models, and it avoids unnecessary repetition For example, in record 2023-09-22 09-22-07b4466623397 ccbb8500, claude- 3-opus-20240229 correctly identifies that there is one set of DFT parameters used in the actual study as well as two more sets of parameters used for convergence testing. Based on how it names each parameter set, it is clear that it understands the purpose of each parameter set. For example, it named parameter set "DFT-paramcters www.eeeeerrrcooeeeeeeeeeee testss"""eee""""" as opposed to the other models' more generic names, like GPT-4o's "create-dft-paraae safft-prraeeteesrpphee However, its predisposition to brevity has its drawbacks as well. For example in record 20240229 incorrectly finds only one set of DFT parameters. Meanwhile, gemini-1.5 1.5-flash-lae latest correctly identifies the two sets of parameters actually present in the paper. All other models repro- duce the same sets parameters multiple times with different names.
claude-3-opus-
Code (DFT-C), Gemini- 1.anini-1.5-flassii 15-flash-lattees and tixtral-8x7b-32766 do much worse than the others. E.g. record 6023-09-22-06 15cca98637765543333333 Gemini-1.5 Semini-1.5-flash--lle 1.5--flash-latest returns simple constant values like "return 0" and "retum [0]" where other models implement more useful functions. Mixtral appeared not to finish writing a script at all. The others arguably performed similarly. Claude3 was the best overall.
F.2 MATERIAL, PROPERTY, VALUE EXTRACTION TASK (MPV)
Modern scientific research articles contain unstructured natural language descriptions of materials, their structure, processing, and properties. This wealth of information is often difficult to access due to the unstructured nature of the text. Collection and standardization of such unstructured informa- tion at scale would benefit materials researchers and accelerate new materials discovery. Existing resources, such as human-curated tables or handbooks, are typically limited in scope and often lack crucial details regarding material structure and processing. This lack of context can lead poor pre- dictability of material properties. While cule-based approaches have been widely employed Swain & Cole (2016); Mavracic et al. (2021b); Dong & Cole (2022b), they often fail capture critical information hidden within the broader context of the text. Dataset curated by computational meth- ods is a good alternative with structured information, however computational results could not be generalized to the prediction of experimental measurements mp (2017) due to the inherent limitation in the theory itself (e.g. density functional theory (DFT)).
With LLM, prompt-based extraction from the scientific literature is emerging method that could result better accuracy at lower amount of human effort. Early LLM-basee property extraction studies utilized fine-tuning Dunn al. (2022), but there have been series of papers using prompt engincering with success, such as extraction of critical cooling rates of metal- lic glasses, yield strengths of high-entropy alloys, emission wavelengths of phosphors synthesis parameters metal organic frameworks Zhang et al. (2023); Zheng et al. (2023); Polak & Morgan.
new extraction
F.2.1 MPV: DATASET COLLECTION DETAILS
We identified set of 17 papers from the S2ORC corpus". We hired expert annotators to annotate the papers and extract all material names, material descriptors, property studied along with property descriptors, and values corresponding to the properties as mentioned in the paper. The annotators were also asked to provide the source sentence to help validate the extractions.
F.2.2 MPV: TASK DETAILS
We perform 3 sets of tasks.
Extraction of all properties. We ask the model to extract all materials, properties and values mentioned in the paper. Prompt for this is shown in Fig. 15. This is identical to the task description given to the annotators. This task also requires that the final output includes the material descriptor, property descriptor and source sentence corresponding to the descriptors and values being extracted. This is the main MPV task reported in the results in Table and Figure 5.
S22RRC https::/p//........ //g/ com/ 111 1 /s2orc
24

---

Published as a conference paper at ICLR 2025
MPV prompt
You materials scientist. Your goal is find and extract all numeric values or numeric value ranges of material preperties mentioned or tabulated in given paper text.
The output should be in JSON format:
"material": "<material name>", mnaterrial.descripteen "<material descriptor>",
"material passagee: "<souree passage from which the material name and descriptor were identified>", "material.sourcetablee
"csource table from which the material name and descriptor were identified>")
"property name": "<property name>",
"property.deccripte descriptor"; "<property descriptor>",
"property.soe source passaaee."" "<source passage from which material name, property and property descriptor were identified>", Dpropettyssourcetaalle" "<source table from which material name, property and property descriptor were identified>", "lowivalue" "<low value>",
"high.value"; "<high value>", "value_units": "<value units>", "value.souroe.pasee
"<source passage from which material
property values were identified>", "value source table": "<source table from which material property values were identisied>", }
An example [Example: Single entry for HfOz with refractive index 1.84]
output as follows:
There are certain rules need to follow:
1. Be thorough! Don't miss even single numeric value.
2. Avoid null low.value and high.value. simply skip the entry if you cannot extract the numeric value.
3. If value is only extractable from a figure, not text or table, simply skip that entry.
whatever Tow_value and high.value actually exists in the source passage or referred table.
4. Make
5. If one passage contains muitiple materials or properties,
record all of then as separate entries.
6. If different values of them separate entries.
menticned for the material
property, record
7. Material, property, and value may or may not be menticned in the same passage.
This below is the excerpt from paper in LaTeX format published in a materials science journal. Excerpt:
text:
[[text]]
with the above excerpt, the goal is to extract numeric val uue or numeric value ranges of material properties from the above paper.
Please only output your in the exact format shown above without any presix. Output:
Figure 15: Prompt for extracting material property values from text (MPV Task).
25

---

Published as a conference paper at ICLR 2025
Extraction of specific properties. The second task do is to test performance when narrow down specific properties of interest. We specifically ask the model to extract materials, property values where refractive index and bandgap properties are being studied. This is scoped task, and we do this specifically to evaluate the models on narrower well defined task. As with the task of extracting all material property values, ask the model to also include the material descriptor, property descriptor and source sentence corresponding to the descriptors and values being extracted.
tightly
Extraction of non-trivial properties. We also evaluate a minor variation of the first task where we ask the model extract all non-trivial properties of materials and their values. These properties in some sense would be considered to be important contributions of the experimental work reported in the particular research paper. This is to avoid any confusion between generic properties materials that are known prior to given research paper vs. those specifically studied or identified newly in research paper. This task is useful extracting just the additional information necessary to populate a database of materials, properties and values.
F.2.3 MPV: DIFFICULITY RATING LOGIC
When rating the difficulty of each example for the MPV task, the experts marked an example as "Easy" the answer could be found in the original sentence without modification. Examples were marked "Medium" if the answer was not very obvious, and the descriptors are very important and cannot be missed. Examples were marked "Hard" if the answer was derived from multiple sen- tences, and sometimes from tables and figures and reasoning was necessary to determine the right descriptors or values.
Ground Truth
GPT-40
naterial: "2n0". groperty_nase "direct bandgap",
naterial: "Zn0" property nane: "direst bandgap",
volue: "3.3", high-value: "9.3", volue, unita: "eV",
lon, value: "3.3", high_value: "3.3", volue, unita: "eV",
naterial: "2n0",
naterial: "2n0" property nose: "exciton binding energy" lon, value: "60", high.value: veluejunits: "me"",
property.nases naae: "exciton binding energy", lowuvalue: "68". high_valoe: valce,units: "wel",
naterial: "Zn0".
naterial: "Zn0"
property.nem "lattice parameterino-axte lattice constant",
property "grain value "135". high_value: "175", units: "m""
Jow.velue: "0.518", high_value: "8.518", value.units: "ns".
}
]
Claude-3 Opus
Gemini Pro
naterial: "Zn0"
raterial: "2n0"
propertyy.nane 'direct bandgap". tol.valu: '3.3", high.value: '3,3", value_units: "ev",
property nane: "direct bandgap".
valve:::: "3.3", high.value: "3.3", value_unste: "el",
property_nane: "oxcston energy". low.value: '60", high.value: *., valuejunits: "rev",
"exciton binding energy". lon.value: "68", high_value: valueiunits: 'rev",
naterial: "2n0"
natertal: "2no"
property name: grain sizelmeessured
property "wavelengthint""
high.value: "236", value_units:
"248", high_value: '248',
Example from the MPV task where Claude-3 Opus, Gemini 1.5 pro, and GPT-4o re- called some important material properties and values but failed to capture all relevant properties and focused instead on some trivial properties. (paper id: 53519111)
Figure 16:
F.2.4 MPV: RESULTS AND ANALYSIS
Both precision and recall rate were used in the evaluation process. A tuple of material, material descriptor, property, value were used for matching and then measuring if property value is aceu-
26

---

Published as a conference paper at ICLR 2025
Ground Truth
MPV task. paper_ id: "15804005"
naterial: 2+f0=aabx2+/aubb""
synonyme: deseriptor: "rilesinatonic layer
GPT-40
seposttionnntiickees 9.
"eaterial": "Wto2".
raterial. source, passage; :
"esterial descriptor "filasnaatonic layer deposition', "saterial_ tource_
raterial, source table: "1",
"Vafntus stlicate films with pure Hf02 and $102 samples as references nere fabricated by
pr property. nome: "refractive indeximninindex of refraction".
property_ descraptor "Ravelangth 558 miinphoton energy gy 2:26
stontc layer deposition (ALD) in this work.".
eVirtaussian dispersion model".
"eaterial, source. table":
pasaage: "Figure andex funotion photon
property.
the refractive
"property. ty..tsee"
all camples
www.rrpttrrrr" "Wovelength sse neirphottn energy 2.26 #V'nGaussian dispersion sadel".
deduced Fron the analysis of the SE results. The value for
H#f==subb2</uub seasured at 550-re (2.26 ev) wavelength is 1,84, which is similar with the previous report [28).", property.souree.te table:
"property. courceepassage "The n value for hr02 neasured
at 550-ne (2.26 av) wavelength is 1.84, which is similar with
the previous report [20].
losevalue: '1.84".
"property source table":
valie.unttss:
hhigh_value", "1.84",
value, wwwrcesuucseeeeeee passagee "Figure 2n shows the refractive index
"value units'
a function of photon energy for all samples deduced fron the analysia of the SE results. The value for irfosubb-<//ubb seasured at 550-rm (2.28 av) wavelength is 1.84, which is similar with the previous report [20].". waloe_source_table
"value. source_passage": 'The n value for Hr02 neasured at 558-ns (2.26 wi) wavelength is 1.84, which in similar with the
previous report [28)
"value, hatpu__ooure__tabbee ),
Claude-3 Opus
Gemini 1.5 Pro
"saterial"; "Hf0esubr2-1/aan /saubbbb
"asterial": "#fD=aub-241///ub> petersal.descriptor "filas" "saterial sal source, passage". : "Figure 2a shows the refractive index n as function photon energy for all samples as results.
"ssterial.cescripp" descriptor" "filasinatonic layer
depositionin Thicknnnnh 9, 88 me
"ssterial_ tource passage" "Pure iffossubee//tubb.
SiD=sub-241/ /sube, and five groups HfSiD--subxe\//ube saeples conpositions were prepared.
deduced
analysis of
value reatured 550-m (2.26 snvelength previous report
different
iefo-sub-2e1/
"saterial. table"
which similar
"prsperty_aaee" "refractive indexten/nindes. of refraction",
cnterial_ssuree table": "*,
"property
name": soopertyynaaee....
"refractive indexien".
sesgerty_cescriptnn "Mavelangth 550 natophoton energy 2.26 eVinSaussian dispersion model".
"property weseryyyeeeptrrrrro " "Wavelangth 558 nevophoton energy
2.26 eV",
propertyyssurra passagg"" "Figure 2a shows the rafractive camples as for
"proparty, ty. sourcoureeeaae tyy.soree_passaee" "Figure 2a shows the refractive deduced function photon energy
as function photon energy analysis the
results. samples as value i#f0<subo241// /ubb> reatured at 550-m (2.26 eV) wavelength is 1.84, which is similar with the previous report [28] "property sourcee table"
The iif@esubb2el//sie /subo neasured 550-ns (2.26 ev) wavelength 1.84, which is similar with the previous report [28].", table":
deduced
analysis
pproperty.
source, souuret
-Low value": "1.84".
"Low value". "1.84".
"high_vall valua"v '1.84".
"high_
"1.84".
units":
unite":
the refractive all camples deduced fron the analysis of the SE results. The value for HrfD-sub-ee//eie /ebbe neasured at 55p-ns (2.26 ev) vavelength is 1,84, which is similar with the previous report [28).". mallun_source_ter
"valus, pasrceepasiaae": "Figure function of photon
refractive "Figure for camples index function photon
"value,
index
deduced fron the analysis of the SE results. The n value for Vetesuubbe///aabb neasured at 558-m (2:26 eV) wavelength is 1.84, which is similar with the previous report [28] "value_sourca" sourcentabbee"" ).
Example from the MPV task where all models correctly identify and extract material, property and other information but have slight differences. (paper id: 15804005)
Figure 17:
rate. The model output and ground truth are only considered as a match if these four fields are semantically equivalent. Exact match is often impossible because of the existence of alternative names for material and properties. The material descriptors could also be part of the material name, which makes the exact match very difficult. For example, "TiO2 nanocrystal" as material name is equivalent to "TiO," " as material name and "nanocrystal" as material descriptor.
In the context of model evaluation, we once again incorporate LLM models into the workflow in order to analyze the level of correspondence between the material propertics extracted by the previ- ous LLM and the material properties annotated by humans. This process is referred to the model evaluation. During the model evaluation, both the material properties annotated by humans and those extracted by the LLM are fed into the LLM. Prompt engineering is utilized to enable the LLM models to identify matches based on specific attributes: material names, property names, low and high values, and unit of measurement. To ensure comprehensive matching, the model to exhibit leniency: this allows it to encompass properties that, despite lacking identical names, convey the same meaning (c.g.. Indium Nitride vs. InN) or differ in the order of units (c.g.. 100cm² vs. 1 x 10ºcm²). Furthermore, we emphasize the importance for the LLM to pay attention to descriptors of materials and properties, as these may indicate a significant difference in the form or state of the material. During the model evaluation process, the source passages are not taken into consideration while instructing the LLM. This decision made because the same properties can be extracted from different passages. Additionally, it is not uncommon for different annotators to select different sections of the same passages or to use different lengths as their source passages. Conse- quently, the source passages are only utilized as a reference for human attention when identifying and analyzing issues of evaluations.
specifically instruct
27

---

Published as a conference paper at ICLR 2025
By correlating each human-anman annotated material property with matching or empty LLM -utput prop- erty, we can compute the precision and recall of the model evaluation, as illustrated in Table
The results of the model evaluation provide valuable insights into the potential transformation of the entire task into fully automated pipelines all by LLM, with the LLM doing both the material property extraction and extraction result evaluation tasks.
is also worth mentioning that the LLM outputs also include trivial material- proppett entities such as sample thickness. Human annotators omit this information due to the lack of general interest to materials science community. Thus we explicitly asked LLM to only include non-trivial material property entities in the outputs and the precision is significantly improved while the recall rate is similar, as shown in the impy-non-trivial" column of Table 3. We further limit the properties to "bandgap" and "refractive index" as these two properties are the most common material properties reported in scientific literature. The results can be seen in the "mpv-specific" column of Table 3: the precision is significantly higher than the regular mpv task due to less ambiguity on whether property should be included. But the recall rate is reduced, which is also related to the filtering we applied in the prompt.
F.3 HARTREE-FOCK MEAN-FIELD THEORY TASKS (HFD, HFE)
In this section we provide additional details about the datasets, detailed task description and analysis of the results for the condensed matter physics tasks. Condensed matter physics often utilizes the Hartree-Focd Fock method, which involves deriving a Hartree-Fock (mean-field) Hamiltonian.
F.3.1 HFD AND HFE TASK DETAILS
We select two tasks pertaining to Hartree-Fock mean-field theory.
Hartree-Fock Derivation (HFD). The first task, HFD, involves analytically deriving the Hartree- Fock Hamiltonian through a series of 13 to 19 intricate steps. Each step demands a deep under- standing of the physical system and requires specific mathematical operations. Due the com- plexity of this task, we employ template approach. The template provides general structure for each step and includes placeholders for key components, The specific task for the Large Language Model (LLM) is towards filling these placeholders which would allow for analytic derivation of Hamiltonian each step. We evaluate the L.LM's performance by comparing the responses for the placeholders with the ground truth written by an expert. Aside from filling in the placeholders at each step, we also ask the model to derive Hamiltonian at each step and compare it to the ground truth Hamiltonian. The entire prompt used for this task contains about 6k words.
Hartree-Fock Extraction (HFE), Recognizing the challenges inherent in HFD, we introduce the Hartree-Fock extraction task (HFE) soning required in HFD. This task involves extracting a Harrtre-Foock Hamiltonian from scientific papers that contain this Hamiltonian. If the Hamiltonian contains some terms that are defined in the paper, we ask the model to extract those terms as well. While HFE is less demanding than HFD, it presents its specific to different cases being studied. So we ask the LLM to fill in template with an exact Hamiltonian expression and all the terms that are necessary to describe it. We note that Hamil- tonian, in most cases, contains several terms that are written in different parts of the paper. For example, it usually contains non-interacting term (H₂) as well as interacting H terms. While the non-interacting term is usually unique, the interacting term can take on various forms in given paper and, therefore, the L.LM must choose which interacting Hamiltonian is the most general.
simpler comprehension task opposed to the deep rea-
challenges,
single paper may contain multiple Hartree-Fock Hamiltonians
By dividing the process into these two tasks, we can systematically evaluate the capabilities of LLMS in tackling complex physics problems, HFD allows us to assess the model's ability to perform step- by-step complex, scientific analytical derivation, while HFE focuses reasoning and information extraction. The prompt for HFE task is shown Fig. 18 and a condensed version for HFD Fig. 19.
F.3.2 HARTRER- FOCK DATASET COLLECTION
We have two datasets for the tasks Mlartree-Fock derivation and Hartree-Fock extraction).
28

---

Published as a conference paper at ICLR 2025
HFE prompt
You are physicist. You are reading a paper that has explicit equation (or equatiens) for the general Hartree-Fock or mean-field Hamiltonian. There are might be several Hertree-Fock Hamiltonians in the paper. You should return the one that is the most general. Return this Hamiltonian.
Print out each equation explicitely instead of citing it. Print out terms in the Hamiltonian if they are present in the paper,
including intergrals.
Do not explain the Hamiltonian or the terms Return the Hamiltonian in the following format: "The general Hartree-Pock Hami Cominto is ((the Hartree-Fock where ((include all terms in the Hamiltonian]] Be concise. Do not explain constants.
mean-field Hami Itoniann))))
PAPER:
[[text]
YOUR RESFONSE:
Figure 18: Prompt extracting the general Hartree-Fock Hamiltonian from a paper (HFE Task)
The dataset for the HFD task consisted the papers used in Pan al. (2024). These were hand selected by expert post-doctoral scientists intimately familiar with the derivations in the work. 15 papers were selected and reasoning steps from solving the quantum many body system was identified for each paper. Pan et al. (2024) also provides prompts and includes derivations to compare responses from language model. The contribution of our work is a generalized version of the template created by Pan et al. (2024). We create format that is consistent across all papers so that evaluation can be automated. The ground truth for task includes all the derivations associated with each of the reasoning steps annotated in Pan et al. (2024).
HFD.
The papers for the HFE dataset were selected from those that contained Hartree-Ford Hamil- tonian, discovered using the arXiv advanced search. Specifically, performed advanced search on arXiv with the following parameters: (i) We set Classification: Physics: Condensed Matter; (ii) set include cross list 'True'; (iái) look for 'Hartree-Fock'
HFE.
key words in abstract. We then selected papers in reverse chronological order (from most recent onwards) and filtered to get total of about 50-80 papers. An expert doctoral candidate in the field then filtered papers where the Hartree-Fock Hamiltonians were present in the paper directly and also extracted and aggregated the equations form across the papers to create ground truth evaluation set. The final dataset for the HFE task consisted of 38 papers.
F.3.3 DIFFICULTY RATING LOGIC
Two experts marked each example with the difficulty ratings using the following rubric.
HartreeFock Extraction task. Easy: If the non-interacting term Hy and interacting terms HEF are on the same page or in the same section and the paper does not have many formulas, so the Hamiltonian easy to find then was marked "easy". Medium: If Ho and Hur are in different parts of the paper the example was marked "medium". Hard: If the paper contains many formulas, or several HF Hamiltonians and it requires time or involves logic to find the right Hamiltonian the paper then it was marked "hard".
HartreeFock Derivation task. For the derivation task, the experts evaluated the non non-interacting term and the interaction term separately either "easy" or "hard". If both are "hard" then the ample was marked as "hard". both are "casy", ; then the example was marked as "easy". Otherwise, the example marked "medium".
29

---

Published as a conference paper at ICLR 2025
HFD prompt
You are materials scientist. You provided with paper and the STEPS derive a Fartree-Fock Hamiltonian.
Be concise. Some STEPS are optional. You should decide whether or not to perform the step based on your understanding of the paper.
In the STEPS below, you will need deduct necessary information from the paper in order to fill in the placeholders ().
the questions
You should print Hamiltonian at each step whenever it is requested to derive.
Placeholders Explanation:
1. The { placehelders are intended for string subst i cons.
2. The [A[B} placeholders are intended for string subat tutions
with either
3. The [text|None) brackets denote optienal strings (text).
System Choice Before Derivation:
Before starting the derivation, you should make choice among these three possibilities for the system based on the paper: This is very important. The choice is strongly correlated with the description of the problem in the paper.
Which Hami tonian does the paper study? Choose one:
1. Continuum version, single-particle
2. Continuum version, second-quantized
3. Lattice version
Note: not print out the choice. Rather, follow steps based
on the choice.
Output Format:
Respord in valid JSON-formatted string. Ensure your respense follows this format:
"Step": "1",
"task": "Construct Kinetic Hami Itonian",
"answer"; "Hamiltonian
[Additional steps follow the format]
STEPS:
Step 1:
Construct Kinetic Hamiltonian
Express Hamiltonian using proper operators and summations.
Step 2:
Define Terms in Kinetic Hamiltonian
Choose di dispersion: parabolic, rirac, cos-like. [Example dispersions abbreviated]
Step 3:
Construct Potential//nteractiia Hamiltonian Add diagonal and off-diagonal terms. [Detailed potential terms abbreviated]
Step 4: Convert to Second Quantized Form
Step 5: Apply Wick's Theorem
[Additional steps and examples abbreviated]
Here is the paper: [[text]}
Figure 19; Prompt for deriving the Hartree Fock Hamiltonian (HFD Task).
30

---

Published as a conference paper at ICLR 2025
Ground Truth
Gemini 1.5 Pro
"Step": "1", "bask": "Conatruct Xinetic Maniltorian (continuus
"response_tee text""
Json
vacalon, single-pacticle"
{
"ceail : "zeal", "definition of variables": m=lll
'Step": "1",
"task": "Construct Kinetic Hamiltonian (continuus version; single-particia)"
sepcesents # valleys, botton Layer (D) top layer (0),
"systen" "AB-stacked MoTo2/MBo2",
"degreen of Freedon"; "Layer (differentiating
rreal [nomentun" "real",
MoToy and WSee) and the combined apin-valley degree", "kinetic symbol": My",
"single-particee pewpteiel/seecoddeeererrrpeereee jecond-quntizzd":
"single-particle
of_erreeoo""" "spin and valley degrees of "degrees.of. freedon
"kinetic, symbol"; "HB"
"Step": *2",
"dispersion "dispersion.syybol"" "(h*k))(22n""
"cask": "Define each term Kinetic momiltonian (cuntinuum veraion]
"amnihilation.op ''"".
"creation, op "gt"
"Eneray disperaion"s EsKA, EsKas K-Kh, E-Ke
"entire, space ffrsster1llouun_zon
entire_space
"definition of variables": : x= (1,0j at corser of the moir"e Brillouin sone, (ml the reat electzon masa) ,
ddfimitionn.e of. of.vaaraales".. "k-10 is the momentun
(m) (..55.
operator, n is the effective noss, ptu(pt-(t, (t,t).tt_ vT 17.(".1).4 (...(--)))) and & are the creation and anniiilation operators. Here. +,- represent +K and -k valley, respectively."
{ "Step": '2" "task": "Define each tern in Kinetic Hamiltonian (continuun version)..
GPT-4o
"response, Json
"Step" "1"
[
Claude Opus
"Construct Kinetic Haviltonian (continuun version, becondd-uantizeed
"reepoose_teee 'Here foonattrd output for
"systen": "continuus version".
the first based the paper:
somentus" 'mosentus",
"single-particle pwwrrtcce/second--eeee iscond-quantizedd:
{
"second-quantzzed""
"Step": "1"
"degrees_ of_eeddo="" "valley and layer",
"task" "Construct Kinetic Hamiltonian (continuum
"kinetic sysbol":
version, single-particle)"
"dispersion_ syybbol""""""" "E(K)",
"systen" : "AB-stacked
"anndhslation.
"degrees_of of_eedfof"'' "spin and valley",
"creation_cn "o.e
'dispersion. symbol":
"entire_spacel Frrrst_rriloounn...
first,
A (rs
(k-T)) A(()) ( Va
2mg
2m
"kinetic_symbol"
"valley index",
"layer
"step": "2"
"task": "Define each tern in Kinetic Hamiltonian
(continuun version)
"2" (continuum version)",
"Energy dis.perssson"
Kinetic Hamsitoniian
(k) 2mp
ELr(k) = (k=
2m
"Energy_ dispryydddppeioonn
"E(K)".
"parabolic| DDirac "parabolic", "electrons hholes" "holes", "somentum_shift""
Figure 20: Model responses for an example the HFD task.
F.3.4
HF TASKS: RESULTS AND ANALYSIS
HFD. As described in the main text, the derivation task is challenging, and expect poor LLM performance, Fig. 5 shows that Gemini 1.5 Pro performs better than other models. Gemini 1.5 Flash, Gemini 1.0 Pro, Claude 3, and Command R Plus also achieved good scores. GPT-4o and LongLLaMa performed the worst. We note that GPT-4o's score was affected by a formatting issue. Although we instructed the model provide the output in ISON format, GPT-4o's output included the words "key" and "value," which were not present in the ground truth. Despite this, the average score for the HFD task across models is much lower than for the HFE task, which human expert considers simpler.
HFE. The extraction task does not involve extensive instructions and is considered simpler than HFD. As shown in Fig. 5, all models (GPT-40, Claude 3, Gemini 1.5 Flash, Gemini 1.5 Pro, Gemini 1.0 Pro, and Command R Plus, listed in descending order of performance) performed consistently well on this task. However, LongLLaMa performed significantly worse than any other model. Over- all, the average score for this task is much better than the score for the HFD task, as expected. This difference in performance is more evident when using the proposed LMScore as seen in
31

---

Published as a conference paper at ICLR 2025
Ground Truth
Gemini 1.5 Pro
"arxiv idft: 21111"9661",,
"response_ree "exptsse__ext'.. text":
"The general Hartree-Fock Hamiltonian
"Haniltonian"s "The general Hartree-Fock
is
Hamiltonian is
HMP En Es (-1))att- (Man(n)
Harr En (L
+ +Tn(m)e, (n)c 1)) +[a(())c(()))(()) h.. 2NC
[n(m)c + 1)] -Ln(()c((()))())) the + +NC
where
J|l + (-1)atoóg)
staggered
Jan(n)
JitR + (-1)"öR),
columnar
Claude 3 Opus
J(R++ + (-1)utagaRh
staggered
Tan(n)
J(nR (-1)"
columnar
"response_text" text": 'The general Hartree-Fock
Jin(n) = J
Haniltonian
En [2a [ + +Tx(a)d(()c(((++t) (Ln(n)e((()2((()) ha 2NC
Har
GPT-40
where
"response. text":
Jan(n)
Jlln + (-1)atobn)
staggered
HMP E (-1)"to-1 ---ntoo-((ndneeeeeeeaaaaae
J(R + (-1)"6k),
columnar
Jhn+ + (-1)utmaanl JhR + (-1)"
staggered
(m)( 1)) (Ln(()(((())((())) the 2NC
TaR(n)
columnar
where
J(t (-1)uto6n),
staggered
JiR(1) J1 2
Jan(n)
Jil + (-1)"ānl-
columnar
renormalized paraneters
JbR + (-1)ato
staggered
Tan(n)
tR + 82n)),
J(i + (-1)"
columnar
5R = 6(1 21I+ n)),
Jin(n) = 2111+211)
7 7-2(P- 8pmp),
( = 1 2(K
6n = 5(1 2(1+2(( 2(K++n))
JaR -26(P- mp)
on = 2(P - -p))
The constant tern is
daR 26(P mp)
C 2 p2 + (1² 1/)) + (KCn+ Pmp) J
C p2 + 62 - 28 Pmp) + J
Figure 21: Model responses for an example in the HFE task.
Fig. 10, which presents the human evaluation of LLM performance, showing better results for the HFE task than the HFD task.
F.4 (QUANTUM) ERROR CORRECTION Zoo TASK (QECC)
The Error Correction Zoo &. (EC (Albert & Faist, 2024) is an open source effort to build Wikipedia-like repository collecting and categorizing error correcting codes from the literature. Such codes are used to redundantly encode and store classical or quantum information so to pro- tect from noise, with wide applications (e.g., 5G communication and broadcast protocols, quantum communication and computing). Currently, the construction and maintenance of the 700 is carried out manually by a "Zookeeper", who updates the ECC Zoo's contents whenever a paper introducing new code appears on the arXiv repository.
An entry the EC Zoo consists of a YAML file, listing the properties of a given EC code along with any relations to other codes. Each entry contains many different types of information: succinet non-technical summary, bespoke technical details, numerical parameters quantifying code perfor- mance, and non-trivial connections to other literature. Due the diversity of encoding schemes, code entries can be very different, and there is no one-size- alll template. We construct a bench- mark that tests the ability of LLMSs curate the EC Zoo.
32

---

Published as a conference paper at ICLR 2025
F.4.1 QECC: TASK DETAILS
The goal of the QECC task is to produce the YAML file similar to the EC Zoo code entry given paper and appropriately designed prompt description of the task. The models output is then compared against the human-curated ground truth. Prompting an LLM to provide such an entry tests its ability correctly extract both the relevant information Mayfield et al. (2024) and the context.
Figure 22 shows the prompt for this task which lists various available YAML key-value pairs taken from a template code YAML file".
F.4.2 QECC: DATASET COLLECTION
We curated the benchmark from two experts in the domain who have prior experience curating en- tries for the ECZoo. Our benchmark consists codes from 65 arXiv papers ranging from 1997 to 2024. Each paper introduces a particular code code family, running the gamut of classical and quantum coding theory and including codes relevant to high entrrey/a/yrrriccleeee physics, phases of quan- tum matter, theoretical computer science, electrical engineering. and quantum engineering. More specifically, the 65 codes include 15 codes realizing phases of quantum matter, 14 codes designed for continuous-variabhe quantum systems, tum devices, 7 general qubit and 4 general qudit code families, 4 classical codes, 2 fermionic codes, 2 randomized code constructions, 2 codes for transmission of classical information over quantum channels, 2 codes designed for magic-state distillation, and I code describing holography.
high-perti segh-perfrrming codes designed for qubit-based quan-
F.4.3 QECC RESULTS ANALYSIS
All LLM-generated error-correcting code YAML files were evaluated by the Rouge metric, and sample of about 20 was carefully read by a human expert. The Rouge metric indicates that LLMs performed relatively well overall.
The human-expert reading of the LLM-generated files affirms that LLMs can effectively summarize technical scientific papers. The summaries tend be succinet, while at the same time flagging multitude of key informational "nuggets" and quantitative metrics relevant to a code. The required YAML formatting rules tended be respected. The performance was consistent across the sample and independent of the underlying discipline of the paper.
In some cases, an LLM flagged informational "nuggets" that were not (but should have been) in given ground-truth EC Zoo entry. For example, an LLM mentioned a numerical parameter relevant to the performance of a code that was buried in the paper and that was missing from the ground truth. As another example, an L.M pointed out that particular code was independently discovered by another cited work. Such statements could only be obtained upon very careful human reading, which takes much longer than rumning the text through an LLM.
On the other hand, since useful nuggets are not always (and need not be) technically correct, cor- rection from an expert is necessary at this stage. The observed advantage of LLMs is that manual correction is faster than a manual identification of all relevant nuggets.
The level of technical detail in the output files tended to be low, with many LLMs tending to avoid recasting technical details of the paper in pedagogical fashion (see Sec. F.4.4 for an exception). It remains unclear whether this be mitigated by prompt engineering if further training is required.
Much of the text in typical LLM- generated entry consisted of extraneous information that ranged from superficial "fluff" to complete hallucination. In this case, this information was relatively easy spot by the expert and did not impede evaluation efforts. It remains be seen whether this ability to fish out relevant nuggets from the sea of boilerplate holds true for other scenarios.
In summary, current LLMs are beneficial for synthesizing literature for the EC Zoo. More often than not, the LLM thoroughly flagged key aspects of a paper and left only the (faster) process of verification to the human. Running paper through an LLM produces a draft EC Zoo entry whose
Phttps:/ / / thub. httpb//cc//crocorreeeeeee www.o///w.o..oocooooooo
ym1
33

---

Published as a conference paper at ICLR 2025
QECC prompt
Fill in YAML file for the code described in the attached paper according the prescription defined the YAML template below. Fields are to be filled only if they are directly relevant to the code introduced in the paper. Be sure to extract any quantitative data like thresholds and code rates. Above all, be concise! If you cannot explain something technical in detail, do not try to explain it. If something not detailed in the paper, not mention it.
This is a code entry in the errer correction 200. http:///ht.aaccccceerrrrreeeteee #*с***********
UTF-8 encoding, AMS-TeX
.... , cite with cite(arXivit.
(Additional documentation comments abbreviated)
code_id: no.spaces_lower.cass lowercase only physical: gubits # one of: bits, qubits, qudits, etc. logical: qubits
Code parameter: ((2'r-1,r, {6} & e.g., ((),
name: 'Coce-name'
introduced:
cite(doi:....
description:
Brief description, references.
(Additicnal description paragraphs abbreviated)
protection: 'Protects against ...!
features: Include only if specifically mentioned [Multiple encoder examples atbreviated to:]
encoders:
['Process description')
[Multiple decoder examples abbreviated to:]
decoders: l'Syndrome measurements'
threshold: ['Error rates')
relations: i Optional
parents: [code.id: parent.code]
.meta:
changelog:
date: "YYYY-MM-DD'
Here is the paper [[text]}
Figure 22: Prompt for generating YAML code entries in the Error Correction Zoo Task (QECC).
comprehensive part can be verified and whose superficial part can be deleted, with both jobs done relatively quickly for this task scenario.
F.4.4 QECC TYPICAL MODEL RESPONSE EXAMPLEES
As we analyze the typical model responses, we also present components the code entry to give sense for the aspects considered by the human expert when evaluating the model responses. We
34

---

Published as a conference paper at ICLR 2025
go through the output for the code described in Ref. Menicucci (2014). The output below is Gemini 1.5 Pro, with GPT-4o, Claude 3, and Gemini 1.5 Pro producing similar outputs for the same input. Inputting other papers yields outputs with similar features as those covered below.
The header of the output file is generally formatted in the correct YAML format, containing a guess of the code id key, the underlying physical alphabet used to construct the code, the alphabet used for the logical information, and the code name (with some superfluous information removed). Almost all LLMs respect the desired key-value YAML format throughout the file, and will not mention this from now on.
code id: ev ov_cluster_gh physical: qubits (encoded contimious-variabe oscillators) logical: qubits
name: "KPP- qubits CV cluster states' rnducd: "ciite 1311. 3538)'
Next comes concise description of the code, typical strength of LLMs. The summary is succinet and well done, while being sufficiently vague so as not incorrect. Two quantitative informa- tional nuggets flagged: the code's 20.5dB threshold and 12.6% error probability.
description:
This code describes fault-tolerant measurement baaed quantum computation scheme using contimoussvvriable (CV) cluster states and GKP-encoded qubits.
The scheme relies on concatenating GKP qubit-level error-correcting code. If the initial squeezing in the cluster state and the GKP states is above threshold (20.5 dB for typical threshold of 10"-6), the error rate on the encoded qubits falls below the fault-tolerance threshold the concatenated code, enabling fault-tolerant computation.
correction with
Magic state distillation of encoded Hadamard eigenstates with an error probability of 12.68 and success probability of 2/3 allows for universal computation.
protection: 'Protects against Pauli noise introduced by finite squeezing in the CV cluster state and CKP encoding.'
Next, the LLM produces list of various features of the code, instructed. The features to be listed depend on what the paper covers, but LLMs tend to provide text for all available features, irrespective of whether they discussed in the paper not. If the paper discusses particular feature, the LLM flags the relevant information, e.g., the code's 20.5 dB threshold. If the paper does mention feature, LLMs tend to put boilerplate for the value in the corresponding key, which ranges from summary similar to those above to a hallucination. This effect is likely to be mitigated by prompt adjustments.
features:
encoders:
GKPP encoding using sharp Gaussians.'
transversal_gates 'All clifford gates are transversal.'
general_gates:
Universal gate set achieved through magic state distillation of encoded Hadamard eigenstates."
decoders:
GGKP error correction using
measurements.'
fault_tolerance tolerance:
fault
/Transversal Clifford gates are fault-tolerant when the initial
35

---

Published as a conference paper at ICLR 2025
squeezing is above the threshold.' 'Concatenation with a qubit-level error-correcting code provides fault tolerance for the entire computation.
threshold:
'0..5 dB squeezing threshold for fault-tolerast computation with typical concatenated code threshold of 10"-6.'
notes:
"The squeezing threshold can be generalized to include
additional noise sources like photon loss and detector inefficiency..
'The scheme presented is not necessarily optimal, and further research may lead to better encoding methods and improved thresholds."
Finally, the LLM presents relations between the primary code extracted from the paper and other codes. Relations form a large portion of the EC Zoo and are difficult to spot because they may be mentioned only briefly somewhere deep in the body of the paper. Using LLMS to flag them saves reading time. In this case, both notable relations are extracted.
relations:
parents:
code_id: cv_cluster_sted state
detail: 'CV cluster state is the underlying physical resource.'
cousins:
code_id: id: gkp code
detail: GGK code is used for encoding qubits in CV osci llators
Attempt at technical details
All LLMs tend to not delve into technical detail, so pedagogy was rare. A notable exception is this attempt by Claude 3 to technically explain codes from Ref. Camara et al. (2007). We present the value the description key, which includes subsections and LaTeX-formatted equations.
descriptions
The authors present two methods for constructing quantum LDPC codes. Both approaches are based representing the generators of the stabilizer group as a graph with simple local rules that ensure commutativity.
\subsection (Generic $ (a,
construction)
The generic construction produces (a, S quantum LDPC codes, meaning each qubit is involved in $aS parity checks and each parity check involves $b$ qubits. It starts by choosing group $ (G) and two subgroups $\mathsf(H)$ and $\mathsf:K)$ The cosets of $ (1mattsf(H)s correspond to qubits and the cosets of $ $1mathsf(K) to parity checks.
Generators (G)) are divided into sets $G_ $G_lomegas and $G_ (bbar((oeega))); with certain commutation and overlap properties, to define the Tanner graph.
CubbscctionnCCnssrutttiii of a (4, s-reqular code)
The $ (4, 8) r-rgular construction uses 3-step procedure. First it constructs a 4-cycle graph as a Cayley graph. Then a ((4,8)$-regula Tanner graph is obtained based the 4-cycle graph. Finally, the edges are labeled to satisfy the commutation relations. Each qubit is involved in two $ $'oomega and two s\bar ("omega) parity checks.
F.5 GEOSPATIAL DATASET EXTRACTION TASK (GEO)
key aspect of geospatial analysis is integrating various diverse datasets together to answer complex questions. For example, study of time-series snowmelt detection Antarctica may combine
36

---

Published as a conference paper at ICLR 2025
Cluude Opus
code, cet reacchona".
introddesd:: :11 121111111
Introdacel
"Iette, T
Logital: "spiits""
fiee Lulti "fusbor of class of queat
1xte: Weschsng code
581 117783"
on suth pur inc, tuch edga ko lebeled led 3 y or The stenilizer gereratore Paull operators around asch alsqurtts, and speraters that of Fuul: operators alann pams connacting paire verstoss.
description:
sanrthin ap codes fannty qutte eoee Titer trivalent
mecortprirn
crivelant Latties with qusits on L' ven tiaes
scobilizer defaned
for riva-qi8tt vereton of cods
the Eitser nzneyoars model
realicet Lente
"Braiding facts foor denorsitaated
evat
sleouette operators
SHO. bacanss the vercsces
operaters alore cercein patils the Lattice.
The codes are defored by finding matching. lastise & pars
in ard Statllizer game cettrrr are then given by for the udbtts alors Use aath
po nonte::
VE (nxthb1133a3)) anyon modaa, eame as
The resuliun, codo recliss the
the of Poats operati time between wast along each Iine chase owm the adponent in line have are gaven by the preeas of Parl: specetore type
cade "cobat stabilize""
aiadiarly surface quanmue ocapurationn the codes zro
preposo for defects that berave the jojanss modes.
code id: "cuantan douple abelian
chu ploquctte, 0g35n
by the Vil (1 copoligpoal arder phease the ieeeycend wssel coadderat
furcrres:
recoders:
The resltring
uode]
codes is
nh cxpliest cosstero are prouited bat the
501 (veamte O-418 it so stom teot The codes sontps-acton actin toronasa modee.
stabilleer gensratora cire.
coustes: I
Vnnactcte ledting caddes veru irapline
oste_ad: "kipeer, honeyocak""
transvrrsal www.laggeee "rrenoerrat clatford peces dopending on lotrace strecture."
code sossill of Use nutohirg oades, delared on square osuttee:
perentss:
code. Lo: curtace, code
The
dhose of the kitaey loncrusad model
teatt todenneee "Subbaaeer iouureents are correcting.
"Expected to hore drreholks
cok-iddi color
cassler codes, threstulde auplicttly caleslateed.
color
constructed
by arbxddting AurTaca code with nanishtng
Figure 23: Examples of outputs on the QECC task. All models do a reasonable job in extracting and describing the code. Claude-3 does a good attempt to recover the technical details. (paper id: 1501.07779)
satellite imagery. radar data, weather station temperature data, elevation/tooograpphy information, etc Liang et al. (2021). ). An epidemiological study on infectious disease spread may combine data from the CDC with people dynamics and behaviours (c.g.. census, zoning, mobility patterns, Google Trends searches, etc.) Seifter et al. (2010); Zhang et al. (2017). Geospatial analysts leverage their extensive domain expertise to choose representative datasets and then build models to achieve new insights conclusions that be used to inform policy, analyze economic or environmental im- pacts, etc. The goal of our GEO task is to help evaluate how well LLMs can assist with Geo-Analyst workflows.
F.5.1 GEO TASK DETAILS
In this task we study the ability of LLMs to extract all the relevant information regarding datasets used in given research paper, including source websites, variable names, descriptions, time ranges and spatial ranges. This task presents a significant challenge to LLMs because requires not only recognizing direct dataset references within the text, but also contextualization, comprehension and aggregation of information scattered throughout the paper, pushing the boundaries of long-context understanding. Prompt for this is shown in Fig. 24.
F.5.2 GEO DATASET COLLECTION
Our benchmark consists of 19 papersranging across earth observation, economics, epidemiology and public health, along with ground truth annotations describing the datasets used in each study. To source this dataset we pulled from various sources and repositories (e.g.. DigitalCommons, Sci- enceDirect, MDPI, NIH, Arxiv, etc.), and included only articles with open licenses (e.g.. CC BY 4.0, CC BY-NC 4.0, MDPI AcenAs etc.) While we started out with more papers, not all of them had permissive licenses thus resulting variety of domains, as well as varying degrees of dataset extraction difficulty and literature quality. For example, some studies had only a few datasets or variables and others many, some had complex time ranges like "Jan-Apr for every year from 2012-2015", some had mix of spatial ranges like "every state in the United States, and also 9 countries in Western Europe", some articles were well written and others quite poor, some had information about the datasets cleanly written in one section and others had the information scattered throughout the introduction, problem statement, analyses, and even figure captions, some articles were missing key pieces of information altogether (e.g., g.. garding time range, spatial range, or source), but knowledgeable domain expert can correctly guess what these should be if they were familiar with the datasets and study topics.
smaller set of 19 final papers. We wanted to represent a
37

---

Published as a conference paper at ICLR 2025
GEO prompt
Given paper, please gather the following information and put in JSON format.
Here is the JSON format:
"paper.title"; spaper.title>,
"paper.link": <paper.link>,
"datasets": [[
"datasst.name": <dataset.name>,
"cataset.website.rr.oo
[<dataset websitt..r
"variables": If
"variable.name" <variable.name>,
"description": <description>,
"time_range":
"start.date": <start date>,
"end.date"; <end.date>
}
"spatial.range"; rspatial.range> ,,
]1,
"notes": <notes>
<paper <paper-title> title> is the paper title. <paper 1ink> is the paper link. For ALL datasets used in the paper: <dataset.name> is the <dataeet.wosstee.o..ooroer
dataset.
is link to the dataset.
<variables> is the list of variables used in dataset. Be thorough and descriptive.
<variable_nane is list of all the names of variables. <description> is the description variable.
Example: if data includes all tweets that include set of keywords between two dates, explain this with enough detail that the dataset could be reproduced if we had access to the raw twects. For example if the dataset is based google trends, you need to include search
categories used
FORMAT: VARTABLE,1, VARTARLE,2, VARTABLR_3, <time range> is list of all time ranges of the variabies. <start date> is the start date of the time range (format: yy-mm-dd).
<end.date> the end date the time range (format: wy-mnd-dd) Note: Make sure to give all the time ranges in a list format.
<spatial is list of all spatial ranges of variables. Note: Make sure to specify all locations with enough detail that the dataset can be exactly reprocuced. So if dataset includes data from 196 counties in the US, please give explicitly the names of all of the counties.
Note: Make sure to give the format.
all the locations in list
Example: County vs State vs Country vs Census Block. <notes> add a note if you have issues filling out any of the fields.
Please copy directly
paper.
Please be concise but have enough detail reproduce.
Make sure you generate the JSON.
Here is the paper: [[text]]
Figure 24: Prompt for extracting datasets from geospatial research papers (GEO Task).
38

---

Published as a conference paper at ICLR 2025
To annotate the papers in the dataset, worked with two annotators who carefully captured the dataset details for each paper in a standardized JSON format. If key information was missing, we asked the annotators to use domain expertise and common sense to make educated guess, while also adding detailed information in the notes section highlighting the shortcomings of the paper. The work of the annotators was verified by experts with Ph.Ds. Agreeing on reasonable format that correctly captured all details required to reproduce the work, while not being overly cumbersome to annotate or read, was challenging (e.g., sometimes it made sense to group variable names and time ranges, versus writing out the entire combinatorial set). After few iterations, we arrived at a reasonable representation that we believe human reader could understand clearly without ambiguity or error, while still being concise.
F.5.3 GEO: DIFFICULTY RATING LOGIC
Human evaluation consideration. Extracting structured dataset information from unstructured scientific text poses unique challenges for evaluation, so we first present the components considered for evaluation prior to describing the logic for measuring difficulty. At the dataset level, we evaluated each model's ability to accurately predict the number of datasets mentioned and identify dataset names using text similarity metries. At the variable level, evaluation focused on the model's accuracy in predicting the total number of variables, identifying individual variables based on a composite representation incorporating multiple attributes, and correctly extracting the time and spatial ranges of each variable.
Using the above rubrics. the difficulty of each example was rated as "casy" if the number datasets and the time and spatial ranges were more easily extracted from a single section of the paper. Examples were rated "hard" if the information was much more spread out and the annotators felt was tricky to have captured all aspects accurately. Other examples were rated "medium".
F.5.4 GEO TASK RESULTS ANALYSIS
Doing a full evaluation of all of the components described above allowed us to pay attention to the parts of the answers we cared about most, but to do this well one had to have similar formatting between the ground truth answer and the model response, and the models often got lot of the right pieces of the answer but in different format, or returned something that looked mostly correct but was not a parsable ISON, making programmatic metric calculations difficult. Other specific points about why this task was challenging to evaluate are: there were often multiple ways to format a correct answer, some parameters required exact matches (e.g., time range) and others semantic matches (like variable description), variable name and description were often muddled and salient information could be placed in either one, dataset name and variable name could sometimes be confused as well (sometimes it was reasonable to concatenate variable name, description, and dataset name and then do a semantic match on the whole set). The most success we had was with prompting an LLM to do the scoring using the LMScore, described in Appendix D. We found that LMScore scores captured the salient features that mattered most automatically. hiding a lot of this complexity. and for this task the human evaluation results correlated quite strongly with the model-based scores.
A few overall observations: there was variability across model runs for almost all models, even though average performance remained fairly constant. The variability was usually higher on the harder tasks. Instruction following remains a challenge: models often had pieces of the right an- swer, but were unable to consistently format their responses, often leading to unparsable JSONs. The model-based scoring approach cireumvented this challenge and also showed good correlation with human evaluations. ROUGE-L scores on the closed weight LLMs were all quite similar (in range [28, 32]), with GPT-40 performing slightly higher than the others. Open-weight model scores were lower (in the 20's for Mixtral and Command-R+ and; 10 for Long..LaMa). We explored a few different breakdowns of the task results to try to identify systematic correlations in model perfor- mance. The first breakdown, shown in Figure 26, plots score values versus the number of datasets in the paper, showing slight inverse correlation between score and number of datasets, although the trend is very small and not very consistent.
We also analyzed performance encompassing one continent the performance was higher than for compound spatial ranges involving multiple continents (see Figure 27). Overall, results for this task illustrate that there is much room
function of geographical spatial information. For studies
39

---

Published as a conference paper at ICLR 2025
Ground Truth
Gemini 1.5 Pro
paperr title": "CAN CDP BE PREDICTIVE oF FEATELETY RATES IN SREDEN
paper Link "hitpa://dkk /atgs///iiaaaaa talcomonn uni, httoorrtthhee//222 datanets"
_aper__ttlee "Can CDP Be Predictive of Fartslity itates in Seaden?
paper
http:://diggtaalrwwwre unl. www.hhonerrshheeee//8222
"dttsstt name". "Data Dammens: Swcden",
mesisstt_mbisseeee wtt.tttrrrrr.rreee porreoure"" "hotpe://dat http:://dttttmmmee www../oww........so........ wttp://aww../ooottr....... "variablee::
ddtasset_ nane" Fertslity rate data fron 1968 to 2019", wwwpeeeeeeeeetteeeeeeeeeeeee |'dntacommons.org "variables"
"voriable, nawe": "I lotal Fertalaty Hate of sweden". "deccription" "The lotal fertilaty Rate (IIR) of
populat ton as the average number chalcren that bern aoman
"vriabll name" "lotal Fertality sate (1FR)", "decortption" "Namber oetldrenn sonon is expected to
over her :iferime they
to exgerience the current through lifetime."
speoitiicc fertility
lsfetine."
"tine range":
Lime.range" :
"storz_date" 11968-01-8"".
"stert_date": 11968-81-00 "end date" "2815-12-311).
end date": "2816-12-31")
range"
"apatssl_range"" ["Sveden
"vrrieble, mase"; "Gross Donustic Product of Giedan", description" "Mesuur the value antedd created through persod, such, also services
countryy during procuction, or the total arount spent on final goods and services flasa seporta)."
production
the income
product (00F) for of 1958 through 2819" dataset Leboiite or or.source". "aatacomonns.rg" "variables"
"tise range":
"end_date": "2815-12-31"),
variable name": "GOP".
"spitial, range": "Seece"
deseription" "A netric of Sveden acononie
status
"tiee_rance" "1960-01-01"
date" 22919-12-811)
"notes" "Although the TFR datoset ventions catacomnons in cataset saction, the re femence section rent tione hacrolrande datweet unage Svecen fertality nate 1958-2022. racrolrendd. (n.d.), Retraeved Narch
"spotial_ range" 'SSveden ]
hetps: mxorotrents intt/ countrressst/eetee /ttereee//////eeeeeeeeeeereieeie fersetyyt roodd..d"" "eocep"
"atte" "Dould not find 1ink to the exsott datesets on datacommona .or.."
GPT-4o
Claude 3 Opus
ttitle "Can Predsetive Fertiliit Swadan? paper link": 'nstoa:/ unk sth /tatocathcaraa522
peper. title": "Can CEP Bi Predictive of Fertility Betes in Swedan?", Lint" Sitaa:/dkrttaaeemm und, hwwhhhoaaathaaeaaaaa. dxtasete"
datasete":
dataset, "Fortalaty hote bata",
dacaset, nane".
Forcility rate data for seedon", "dataset t, www.seteeerrreeeeeeeeeee prttturorrr : httlias./dltassummoo.
dataset deteset www.ite.errroo..o seouree" ('atiansi www.s//aaaasommm....... variables
"variables"
"vrriable m "m
"Tocal Fareslity Rate (IFR)', "dscraption" 'The nambrr of children a noman is
"variable, name": "Total Fartility Rate (TFR)', deseription" 'The number of children a woran is expected to have in her Lsfetiie Seeden
expected have in her lifetine, collected annuslly fro
"ttme_range"
"stert_tato": "1969-61-01",
"start. date"s 1966-61-91",
*ond "2910-12
and date": "2010-12-51").
spatial, range" "Dedenn ]
"spatial_ range" ['Sweden")
"dstaset, 1, nane". GGoss Sumestic Product (GIP) data for
pntaxet..nee _nam"" "Greus Doneetiae Product (CDP) Data", www.ettttoottttooooooooo ittrasi//aaraooommnn vartables
Seeden
"dataset. wwwsstte.rr.cuurre of /ttoos// ooo"),
vartable_nasr
"Gross Berestic Product (GOP)
"variable name" "Gross Dosestic Product (GBP)" "description"
"dsccription"
gross
value goods servicees
product Sveden 1369 2019. netric
produced Swedes. used
Swodon'
Sweden's a cconcaie status.",
coononde status"
"time Irange of
"ties range" 4
"start Latee" "1968-01-81",
"stert =start_dtte": "1960-01-41"
13--1-))
spattial, range" ['Sweden']
"spotial_ range" ["Sweden]
"The datasets used detacsmmons.com org and cover the period from 1968 to 2010 for Sveden. The varsables include Totil Ferrility Bate (TFR) and Gross Domestic Product (63P).
"netes
sourced froe
"etees: "The explores the reletionshis between and fertility ratis (TFR) in Sweden over 58 year period froe 1968 to 2019. The specific counties or regions kithan Sweden are not spenified, so the spatial range is lirited te just the country of shole.")
paperic id: "c0000"
Figure 25: Example from the GEO task where all closed models Gemini 1.5 pro, GPT-4o, and Claude-3 Opus extracted the correct spatial and time range for the datasets. They provided a less specific link than what human annotators provided and used slightly different format where the models treated each variable as a separate dataset whereas human considered the dataset to be a single source but with different variables. Despite this slight difference the model's response is still accurate and useful. (paper id: 00000)
for improvement for LLMs to be able to automatically extract dataset information from the literature, especially in some of these more complex compound cases.
40

---

Published as a conference paper at ICLR 2025
Figure 26: Score value versus number of datasets in the paper. There papers with fewer datasets have substantially higher performance, but there isn't a linear degradation in performance as the number of datasets studied the paper increases perhaps because when many datasets are used in the paper they are still tied to some common sources.
Figure 27: Score value versus geographical/spatial range studied in the paper. The scores for papers containing datasets covering multiple continents in their spatial extents (the 4 bars the right) have noticeably lower performance compared to those covering the spatial extent within a single continent. Further, spatial ranges in Europe and Antartica seem better scoped than other continents, this might be worth verifying with a large sampling of papers.
41

---

Published as a conference paper at ICLR 2025
F.6 BIODIVERSITY GEOREFERENCING TASK (BIOGR)
In this task we study the ability of multi-modal LLMs to work with geographical maps. Specifically, we investigate the core capability georeferencing, where, given an image map and its associ- ated caption, the task is to determine the latitude/longiude bounding box encompassing the region displayed. Figure 28 shows examples images from the dataset georeferenced against geographi- cal maps. This section provides additional details regarding the dataset creation, task definition, and evaluations for the biodiversity georeferencing task.
Figure 28: Illustrations of map georeferencing. showing images from the dataset superimposed geographical maps in their correct georeferenced locations.
F.6.1 BIOGR DATASET COLLECTION
To source this dataset we worked with domain experts in ecology to find articles that were represen- tative of the domain, paying specific attention to the map figures. We pulled from various sources and repositories (e.g.. bioRxiv. ScienceDirect, Nature, etc.), and included only articles with open licenses (e.g., CC BY 4.0 or CC BY-NC 4.0). We wanted to represent a variety of map features often displayed these figures, as well as varying degrees of georeferencing difficulty (e.g., some regions are larger and some smaller, some mention locations explicitly in the image and/or caption, some have recognizable shapes at the continent or country level, some contain multiple repeated instances of the displayed region, some have insets or coordinates displayed to help the reader derstand the spatial information, etc. Fig. 29 shows a few illustrative examples). We assembled dataset of 24 map images and captions, extracted from 24 papers, and worked with domain
42

---

Published as a conference paper at ICLR 2025
pert to acquire the ground truth latitude/longitudee bounding box for each image. We also asked the domain expert to rate the individual tasks using a difficulty scale from 1-10 (1 being the eas- iest and 10 the hardest) which we simplify to the easy, medium, hard ratings with other tasks. portion of the BIOGR dataset consisting of 14 examples with additional metadata including original pdf documents from which the figures sourced is available https//ggihubb.om/googgl
rescarch/ccolog--eoreeecee
Figure 29: sample of map images from the Biodiversity Georeferencing Task dataset.
F.6.2 BIOGR TASK DETAILS
The goal for the BIOGR task is to accurate extract the spatial coordinates (latitude and longitude) for a location that specified in the map images. To do this, we configured standard prompt, and passed the prompt, along with each image and caption pair, to each of the multi-modal models evaluated. The prompt used is shown in Fig. 30
BIOGR prompt
For the following figure and caption, please return WGS84 latitude and longitude bounding box coordinates of the map in the image.
If there only one answer.
multiple maps of the same region, please just return
If two areas are represented, and one is an inset of the other, return the smaller of the two areas.
If you are not sure, please guess at answer anyway. I' d rather have
than no Make sure to return decimal coordinates in range [-90, 90] for latitude and (-180, 180] for longitude.
all.
Please put your answer in the following JSON format:
"n": <wett>,
"s": <south>,
"E": <east>,
"N": <north>
Here is the image and caption. [[toxt]]
Figure 30: Prompt for the Biodiversity Georeferencing Task (BIOGR).
Given this prompt, the image, and the caption, the selected model would provide a text response. Even though explicitly instructed not some models would provide text preamble prior to the JSON bounding box answer. In these cases we filtered the response to extract the JSON bounding box answer. In all instances, all models format suggested in the prompt, so we did not have to handle cases where the answer was provided but not in the requested format. For illustration purposes, sample ground truth answer looks like:
able format their answer in the requested JSON
43

---

Published as a conference paper at ICLR 2025
"N":-114.472617555 "S" "s"::4.229991077 "E" 1GM:-113.29917766 "N" :b::440002526648
GPT-4a
WT: 168
STC -7.75,
Ground Truch
Claude
183.2
182 ST -1
+1
189
Gomini 1.5 Pro
"NT's
101.35
-2.5
**"
Figure 31: Responses from the different models for an example from the BIOGR task. Here, Claude- 3 (bluc) predicts the coordinates closest to the ground truth (green) bounding box locations indicated by the red dot in the original map image (top-left). GPT-4o (purple) and Claude-3 Opus generally performed well across examples on this task while Gemini 1.5 pro (orange) was bit less precise. (record id: 556058.2)
F.6.3 BIOGR RESULTS ANALYSIS
For this task we consider evaluation metrics. Intersection-over-UNni (IoU), which is defindd as the area of intersection between two bounding box regions divided by the area of union. IoU is popular metric for bounding box tasks, and captures the similarity between two polygons, ac- counting for both location and size, while also being scale-invariant. For simplicity, we computed IoU using longitude and latitude as X and Y axes directly versus converting into meters". Ex- plicitly, we used "W" and "E" as our minimum and maximum X-coordinates, and "S" and "N" as minimum and maximum Y-coordinates respectively. Normalized Distance Error measures the distance between the centers of the two bounding box regions (in kilometers) when placed on the surface of the Earth, and this is normalized relative the distance of the half-diagonal of the ground truth bounding box. This metric helps measure how far off the predicted box is when the prediction doesn't 't overlap with the ground truth box. Relative box size. This is the relative size of the pre- dicted box (distance of the half diagonal) with respect to the ground truth box half- nalf-diagonal, which indicative of the error in the predicted box size.
When performing this task, over the full image/caption set and 3 model runs, the average IoU with each model ranged between 0.42 and 0.54 (Table 1), with GPT-4o performing slightly higher than the other models. The median results, with 25% and 75% error bars, are shown in Figure 5, with medians ranging between 0.3 and 0.6 and error bars showing wide variability accounting for both task and model variability. To put these numbers in context, the loU metric ranges between 0 and 1, and any value greater than 0 denotes at least some region overlap, which implies that the model has at least geospatial knowledge. Values between 0.2 and 0.5 have reasonable region overlap, and
A more precise loU calculation would convert to physical distance, which would equally penalize hor- izontal and vertical emors, and would account for the fact that the physical distance between two longitude lines varies with latitude: however, for small regions this difference is minor and the scale-invariant IoU metric captures the bounding box overlap reasonably well.
44

---

Published as a conference paper at ICLR 2025
values greater than 0.8 have very good georeferencing performance (see Figure 32 for illustration of loU).
Less WoneEr
SARP
Gula
Hid
Peary ther
MICHIGAN
Medio Minauuee
Det
Orde Sepdk
Chicago
Mere
(a) loU = 0.07
(b) loU = 0.8
Figure 32: An illustration of Intersection-over- -vio-nnoonn (IoU) metric for biodiversity georeferencing.
Model
IoU
Normalized Distance Error
Relative Box Size
Claude 3 (Opus)
0.39
6.15
1.39
Gemini 1.5 Pro
0.42
14.28
1.20
Gemini 2.0 Flash
0.49
3.03
3.05
Gemini 1.5 Flash
0.36
6.09
1.69
GPT-4o
0.37
1.43
1.22
Gemini 1.0 Pro
0.34
12.37
1.30
Table 7: Performance of models on BIOGR based on IoU, Normalized Distance Error, and Rela- tive Box Size. t indicates higher is better, + indicates lower is better. Blue highlights the highest values across closed models.
One observation is that, even though we explicitly asked the models provide answers even if they weren't I sure, there were instances where models refused to estimate any coordinates. We also noted that there was some variability in model responses over the 3 runs for each image, and the variability was higher on the harder examples in the dataset. In additional experiments with Gemini 1.5 Pro, reducing the temperature of the model reduced variability but also increased the percentage of cases where the model refused to provide answer. These stochastic and robustness performance issues must be taken into account when building systems that use LLMs.
To analyze how different features in the image/caption pair contributed to the LLM's ability to per- form the task, we ran further experiments with Gemini 1.5 Pro, where we passed in only the image without the caption, and only the caption with image. On the image only runs the average IoUs and model non-responses remained roughly the same, but on the caption only runs the loUs de- creased and the model non-responses increased significantly, highlighting that the image is much more important than the caption for this multimodal task. We also computed average loUs over various subsets of the dataset, shown in Figure 33 on the left. The subset of images with coordinates shown on the map performed better than average, which is reasonable since multi-modal LLMs are trained on similar bounding box object detection tasks the pixel level. and having numerical scale present in the image that the model can reference increases performance. The larger regions were significantly easier for the model than smaller regions, partly because the chances of overlap are higher and partly because larger regions often had recognizable shapes or well-known loca- tions mentioned. One result that was surprising was that, in this set results, the images of small regions without insets performed better than the ones with insets, which is counter-intuitive. On closer inspection, the images in this set with small regions and without insets either had coordinates displayed, were really well-known regions (e.g., maps of known cities), explaining the higher per- formance. Increasing the size the dataset and adding more representation of images with insets and without coordinates well-known shapes/locations would help generate more comprehensive
45

---

Published as a conference paper at ICLR 2025
statistical results for the inset subset. Our suspicion is that these models are not currently paying a lot of attention to the image insets, like human georeferencer would, and that there is room for improvement in teaching these models to process that information well. We plan on studying this further in future work. Finally, the right side of Figure 33 shows loUs for all the tasks and models plotted against the domain expert's difficulty rating, suggesting that the tasks that were harder for the domain expert were also more challenging for the model (GPT-4o had the cleanest correlation trend, whereas the other models had more variability in their performance).
lou Task Oificity Estimate
Choude (0pol
Gonis to
Gurra GFT/4g
0.5
HamTn Cuficuity Sereran
Figure 33: On a subset of 24 BIOGR examples, the left chart shows a breakdown of IoU for various data subsets. On the right we show IoU versus task difficulty ratings provided by a domain expert.
Finally, we note that we discarded some examples as out of scope when creating the dataset not because of difficulty, but because they did not facilitate clear query and a single accuracy metric (e.g. multiple unrelated images in the same figure, multiple study areas within a bigger region, cropped and concatenated regions that were not actually geographically contiguous, etc.). However, system that automatically parses the literature extract information will have to deal with these real-world examples as well. For future directions we would like to go deeper and build benchmarks that help study the ability of LLMs to extract more detailed information from maps (c.g. polygons of surveyed areas, points of interest highlighted in the legend, geographical features and insights that correlate to the data presented, Q&A pairs that capture what a domain expert would ask, etc.). Being able to work well with maps has wide applicability to many scientific domains (e.g. geospatial analysis, public health, epidemiology, etc.) so it's encouraging to see how much spatial information is already captured by these models today. One final note is that a domain expert usually uses a variety of tools their disposal perform these tasks (e.g.. using AreGIS, QGIS, Google Maps, etc.), rather than memorizing a large library of maps. It would be interesting to teach LLMs to integrate these geospatial tools as part of their workflows and analyze how much performance can increase via this agentic approach.
F.7 PROTEIN SEQUENCE RECONSTRUCTION TASK (PDB)
This task assesses the ability of large language models (LLMs) extract biologically relevant infor- mation from structured textual data. Specifically, we challenge the models to reconstruct the amino acid sequence of protein given its dinensiinnssinal structure in PDB format. The dataset consists of 64 PDB files representing diverse proteins with varying sizes, functions, and originating species. Each example comprises a single chain to maintain task focus and complexity.
Solving this problem necessitates an intricate understanding of the PDB format and the ability to parse it accurately. The model must first recognize the hierarchical organization PDB files, iden- tifying sections containing atomic coordinates and residue information. Subsequently, it needs to extract the three-letter amino acid codes for each residue, traversing the entire structure to ensure completeness. Finally, the model should leverage its knowledge of amino acid IUPAC Data to map the three-letter codes to their corresponding single-letter representations, ultimately producing a FASTA formatted sequence.
46

---

Published as a conference paper at ICLR 2025
F.7.1 PDB DATASET COLLECTION
An expert in the domain identified 64 proteins from the Protein Data Bank The proteins span variety of species (15), sequence lengths, and protein functions. The ground truth amino acid sequences are also available the data bank and can also be computed programmatically.
F.7.2 PDB TASK DETAILS
For this task, we passed into the models PDB text describing the three-dimensional structure of protein, along with standard prompt asking the model to return the amino acid sequence in FASTA format. The prompt is shown in Fig. 34.
PDB prompt
You are a computational biologist and want you recenstruct protein's s amino acid sequence from its tertiary structure. The input is PDB that is a textual format describing the three-dimens onal structures of protein.
Return the amino acid sequence in the standard FASTA format, which starts with definition line with the greater than (>) line, followed by the single-letter coces for all amino acids in the second line.
Make sure the amino acid sequence is in the second line. If there an unknown amino acid in the structure, put "x" in the sequence.
Make sure you go through the whole structure and get all the amino acids.
No explanation is needed.
below are the tertiary structure:
[[text]]
Figure 34: Prompt for reconstructing protein amino acid sequences from tertiary structures (PDB Task).
F.7.3 PDB RESULTS ANALYSIS
Evaluating the reconstructed sequences involves comparing them to ground truth sequences obtained using the Biopython library. We employ pairwise sequence alignment determine the optimal correspondence between predicted and ground truth sequences. The alignment is then scored using standard metries including the number gaps, identities, and mismatches. These raw scores are normalized by the alignment length to account for potential length discrepancies. We primarily report the identity ratio, defined as the number of identities divided by the aligmment length, as the primary performance metric. Unlike other tasks, model-based evaluation is not employed here due to the straightforward nature of exact evaluation and the limited ability of current L.LMs to accurately assess sequence similarity. ROUGE score, typically used for text summarization. is also unsuitable for capturing the nuances of protein sequence similarity.
Our experiments revealed several failure modes, highlighting the challenges LLMs face in process- ing long and structured biological data. Frequently, the models struggled to accommodate the entire PDB file within their limited context windows. Even when the context window sufficed, the mod- els often failed to traverse the complete structure, leading to incomplete sequences, particularly for larger proteins. Another recurring issue the incorrect handling of consecutive identical amino acids, often resulting in undercounting. Additionally, we observed instances where the model mis- takenly uses the first letter of the three-letter amino acid name instead of employing the established three-to-one letter mapping, despite this mapping being a fundamental concept in bioinformatics
hhtps//www.resbborg
47

---

Published as a conference paper at ICLR 2025
and likely encountered during the model's training. One noticeable trend was that the identity protein sequence became longer, which is reasonable, although we note that for
tio dropped as the the shorter sequence length proteins there was large variability in performance (see Figure 35 for details).
0.6
value
0.4
0.2
0.0
(51.999, 68.01
78.0)
pess 174.8
(17..0 190.8)
0..00. 223.01
223.0 238.0)
(230.00 0. 28000
1960.0 0. 266.00
100
10100
1DEEEE (100))
10 10355 00
200 1O000
(68
Pert
190.00
Sequence_engie bins
Figure 35: Plot of identity ratio versus protein sequence length, showing that performance drops as the protein sequence length increases.
48


Reviews:
[{"review_id": "8HQcwfKDyP", "reviewer": "Reviewer_rvRN", "summary": "This paper introduces CURIE, a scientific long-context reasoning and information retrieval (IR) benchmark. It contains ten tasks across six scientific domains, all designed to be challenging yet realistic. A wide range of open-source and closed-source large language models (LLMs) are tested, and Claude-3 consistently outperforms other models, including GPT-4.", "questions": "- I don’t fully understand how LLMsim works. If the goal is to identify the number of dictionaries that are correctly retrieved, why not use an exact match or other metrics commonly used for evaluating retrieval, such as nDCG?\n\n- For each paper, is there only one question asked? The paper is not entirely clear on this point, and it raises the question of whether each paper corresponds to only one question or several questions (similar to Qasper).\n\n- In Figure 4(c), does the average length of 954 words refer to the ground truth, or does it refer to the average length of the model-generated responses?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- **Dataset Design**: The dataset is well-designed, covering ten complex tasks (even complex for humans) across six scientific fields. It targets the realistic problems faced by scientists.\n\n- **Annotation Process**: The annotation process is thoroughly explained, and the motivation for selecting each sub-task is clearly articulated.", "weaknesses": "**Dataset Size Too Small**:\n\n- “The CURIE benchmark encompasses 434 examples across ten tasks curated from 273 research papers across six diverse scientific disciplines.” The scale of the dataset is somewhat small. Specifically, tasks like PDB and CEO have only 21 and 19 examples, respectively. This limited size may lack statistical significance when comparing models at the sub-task level, making it difficult to determine the reliability of the results given the small sample sizes.\n\n**Issues with LMScore**:\n\n- As mentioned in Appendix A, GPT-4 is used as the language model (LM) to compute LMScore. It would be better to use an open-source model, as GPT models are constantly being updated, which may alter evaluation scores over time. Even when specifying a GPT version, it may become deprecated, making it hard to replicate the results. \n\n- Additionally, in Figure 9, the results do not convincingly show that LMScore has a high correlation with human judgment. Therefore, it may be challenging to conclude that LMScore can replace ROUGE.\n\n**Presentation Could Be Clearer**:\n\n- It would be beneficial to include a table that specifically lists the number of questions, the number of questions under each task, and the average number of tokens (or words) in queries, documents, and ground truths. Although this information is scattered across Figure 2(b)(c) and Figure 4(c), consolidating it into a table would enhance clarity.\n\n- Line 269: “On the extraction tasks, such as MPV, HFE, GEO, and DFT-S, experts within each domain reviewed each other’s work and reported a high rate of agreement.” It would be better to provide statistics on the inter-agreement between annotators.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1UCE6iaPHe", "reviewer": "Reviewer_EWc3", "summary": "The paper introduces the CURIE benchmark, designed to evaluate large language models (LLMs) on multitask scientific problem-solving, focusing on long-context understanding and reasoning. CURIE encompasses ten challenging tasks across six scientific disciplines. These tasks require comprehension of extensive context and domain expertise.\n\nThe authors assess various LLMs, highlighting that while Claude-3 performs well across domains, models like GPT-4o struggle with specific tasks, such as protein sequencing. The study finds that current models still have significant room for improvement.", "questions": "1. Could the authors explain more about how you define the input and output format?\n2. What is your method to evaluate the free-form generation?\n3. What are the approaches for quality control?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed benchmark focuses on long-context understanding and reasoning in scientific domains, which is relatively unexplored and meaningful for advancing AI4science.\n- The paper provides many details and discussion towards the tasks and experiments.", "weaknesses": "1. The presentation of this paper is not clear enough. While the authors spent much space introducing the domains and tasks, the input and output (e.g., format, content) are not explicitly defined, which makes it difficult to understand how the authors evaluate the models.\n2. The evaluation metrics may not fairly show the performance. The reliance on ROUGE-L and BERTScore may not fully capture the complexity of scientific reasoning. While the authors propose metrics like LLMSim and LMScore, how they extract the answers from open-formatted model outputs and compare the potentially heterogenous answers (e.g., different field name, spelling difference, number/string, with/without extra text) with the gold standard is not clear to me. If not carefully handled, the experimental results might not be reliable.\n3. The experiment analysis does not provide many deep and insightful observations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gx4sJBGUHh", "reviewer": "Reviewer_zECm", "summary": "This paper introduces a benchmark called Curie for evaluating Large Language Models (LLMs) on science-related tasks in the disciplines of material science, condensed matter physics, quantum computing, geospatial analysis, biodiversity, and proteins. The benchmark includes 434 examples taken from 273 research papers related to the previously mentioned scientific disciplines. Based on those examples, tasks were created and used to evaluate several LLMs.", "questions": "- Have the authors considered inserting the task description into the LLM prompts without providing the paper? How would that compare with the CURIE method? \n- Have the authors considered adding more tasks from a wider range of disciplines as future work?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written.\n- The paper provides a helpful benchmark for evaluating LLMs in science-related tasks.", "weaknesses": "- There is no comparison with the baseline case of just inserting the task description into the LLM prompts without providing the paper. It would be useful to see what the results would be in that case.\n- Only 6 very specific disciplines (material science, condensed matter physics, quantum computing, geospatial analysis, biodiversity, and proteins) are considered in this dataset. This limits the scientific domain in which LLMs can be tested.\n- The dataset construction requires human annotations. This makes scaling up the dataset challenging. \n- Typo line 339: \"precison\" -> \"precision\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hn1p43lvMa", "reviewer": "Reviewer_dZb5", "summary": "The paper introduces CURIE, a benchmark for evaluating LLMs on scientific problem-solving that requires understanding long-context information and multi-step reasoning across six disciplines: materials science, condensed matter physics, quantum computing, geospatial analysis, biodiversity, and proteins. CURIE includes 434 tasks derived from 273 scientific papers, covering realistic scientific workflows such as information extraction, concept tracking, aggregation, algebraic manipulation, and multimodal understanding. This benchmark evaluates eight state-of-the-art models, revealing significant performance gaps, especially in tasks like protein sequencing. The paper aims to foster advances in LLM capabilities for scientific applications.", "questions": "See \"weakness\". I've listed my questions there.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The paper proposed a new long-context scienceQA benchmark CURIE, containing 434 examples from 273 science literatures in different disciplines. The tasks are carefully collected and benchmarked with the help of domain experts.\n\n2. The paper provided a detailed experimental analysis and many meaningful discussions about the challenges LLMs face in long-context scientific reasoning, especially regarding tasks like protein sequencing and geospatial data extraction, where complex, multi-step reasoning and domain-specific knowledge are crucial. The appendix also included many details for benchmark construction, examples, and case studies, helping readers better understand the annotation process and challenges of the tasks.\n\n3. The paper is well-written and easy to follow. I really appreciate the experts' efforts for annotation and the authors' contribution for conducting this research, and believe this benchmark would be beneficial for both developing better LLMs and better usage of LLM for scientific research.", "weaknesses": "1. I'm not sure if LLMs could have seen some of these research papers, which might affect the effectiveness of evaluation results on CURIE. A figure showing the comparison / correlation of performance between CURIE and other scientificQA / long-context QA benchmark would resolve this concern more clearly.\n\n2. I think one of the challenge of such benchmarks should be the difficulty of scaling up? Due to the static manner and limited scale, public papers might be included within future (or existing) corpus for LLM training. I wonder if the authors considered to somehow automatically scale up the dataset, or keep updating the examples?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yexIKV0gWP", "reviewer": "Reviewer_CbxB", "summary": "This paper introduces a new benchmark for long context understanding, reasoning and information extraction of different scientific tasks and evaluates the current state-of-the-art LLMs on it. It has 10 tasks split on 6 disciplines which are labeled by domain experts. \nThe authors also introduce a new LLM-based evaluation metric and use it along with evaluation established metrics. They also employed domain experts to evaluate the LLM predictions on a sample of the dataset.", "questions": "- Have you considered making more specific questions? For example, in Figure 3 the DFT example where you ask to identify all the input structures and this expects a big dictionary, can you make multiple questions instead for each key of the dictionary? \nWould it make sense to provide only the keys of the dictionary and ask it to complete?\n- When you say 434 examples curated from 273 research papers, what would an example entail? \n- Since this benchmark is also about evaluating models in long contexts, a plot of the model performance as a function of the context length would be interesting  \n- Figure 18 needs bigger font size for the axis labels", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This work introduces a novel dataset with challenging tasks from different disciplines and evaluates different state-of-the-art LLMs and shows where they lack. I appreciate the effort that has went into building this dataset. I understand that this required time from domain-experts from different disciplines to label it. They have employed domain experts to make sure they have a reliable evaluation on the predictions of the dataset\n- Having benchmarks for complex scientific domains is important for the improvement of language models\n- The paper is well written and the message is conveyed clearly", "weaknesses": "- For some tasks there are not many examples\n- As you mention, these answers are open-ended and it’s hard to evaluate the model’s output. A model may have a correct output but have used different set of steps or a different notation to answer. Having Latex as the ground-truth will also make it harder to evaluate. This will be less of an issue in the extraction tasks and more of an issue in tasks that require planning/reasoning. I understand that you use ROUGE-L/BERTScore F1, but I still think the performance may be unreliable for some of these tasks\n- LLMSim/LMScore which follow LLM as a judge paradigm can also have their shortcomings. I would suggest adding multiple choices where it makes sense in this dataset. Even though you won’t be evaluating the expressiveness of the language model, it will be easier to judge the LLM’s knowledge using accuracy as a score which is interpretable\n- Asking for the LLM to come up with an open-ended dictionary will make it harder to evaluate and map the generated dictionary keys to the ground truth keys as you do with LLMSim\n- Since this dataset is cross-domain and requires such a specialized knowledge for each domain, it would be challenging to build a good model that performs consistently good across the different domains. Instead, a dataset that is focused on a specific domain and is more thorough would be better", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)