CURIE: Evaluating LLMs on Multitask Scientific Long-Context Understanding and Reasoning

Authors: Hao Cui, Zahra Shamsi, Gowoon Cheon, Xuejian Ma, Shutong Li, Maria Tikhanovskaya, Peter Christian Norgaard, Nayantara Mudur, Martyna Beata Plomecka, Paul Raccuglia, Yasaman Bahri, Victor V. Albert, Pranesh Srinivasan, Haining Pan, Philippe Faist, Brian A Rohr, Michael J. Statt, Dan Morris, Drew Purves, Elise Kleeman, Ruth Alcantara, Matthew Abraham, Muqthar Mohammad, Ean Phing VanLee, Chenfei Jiang, Elizabeth Dorfman, Eun-Ah Kim, Michael Brenner, Sameera S Ponda, Subhashini Venugopalan
Primary area: foundation or frontier models, including LLMs
Keywords: science, LLMs, evaluation, benchmark, long-context

Abstract:
Scientific problem-solving involves synthesizing information while applying expert  knowledge.   We  introduce  CURIE,  a  scientific  long-Context  Understanding, Reasoning, and Information Extraction benchmark to measure the potential of Large Language Models (LLMs) in scientific problem-solving and assisting scientists in realistic workflows. This benchmark introduces ten challenging tasks with a total of 580 problems and solution pairs curated by experts in six disciplines - materials science, condensed matter physics, quantum computing, geo-spatial analysis, biodiversity, and proteins - covering both experimental and theoretical work-flows in science. We evaluate a range of closed and open LLMs on tasks in CURIE which requires domain expertise, comprehension of long in-context information,and multi-step reasoning.  While Gemini Flash 2.0 and Claude-3 show consistent high comprehension across domains, the popular GPT-4o and command-R+ fail dramatically on protein sequencing tasks. With the best performance at 32% there is much room for improvement for all models. We hope that insights gained from CURIE can guide the future development of LLMs in sciences. Links to the data and evaluation code are in https://github.com/google/curie

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730563329458

thread_type:
official_review

<thread_object>
creator: reviewer_cbxb

<content>
summary:
This paper introduces a new benchmark for long context understanding, reasoning and information extraction of different scientific tasks and evaluates the current state-of-the-art LLMs on it. It has 10 tasks split on 6 disciplines which are labeled by domain experts. 
The authors also introduce a new LLM-based evaluation metric and use it along with evaluation established metrics. They also employed domain experts to evaluate the LLM predictions on a sample of the dataset.

strengths:
- This work introduces a novel dataset with challenging tasks from different disciplines and evaluates different state-of-the-art LLMs and shows where they lack. I appreciate the effort that has went into building this dataset. I understand that this required time from domain-experts from different disciplines to label it. They have employed domain experts to make sure they have a reliable evaluation on the predictions of the dataset
- Having benchmarks for complex scientific domains is important for the improvement of language models
- The paper is well written and the message is conveyed clearly

weaknesses:
- For some tasks there are not many examples
- As you mention, these answers are open-ended and it’s hard to evaluate the model’s output. A model may have a correct output but have used different set of steps or a different notation to answer. Having Latex as the ground-truth will also make it harder to evaluate. This will be less of an issue in the extraction tasks and more of an issue in tasks that require planning/reasoning. I understand that you use ROUGE-L/BERTScore F1, but I still think the performance may be unreliable for some of these tasks
- LLMSim/LMScore which follow LLM as a judge paradigm can also have their shortcomings. I would suggest adding multiple choices where it makes sense in this dataset. Even though you won’t be evaluating the expressiveness of the language model, it will be easier to judge the LLM’s knowledge using accuracy as a score which is interpretable
- Asking for the LLM to come up with an open-ended dictionary will make it harder to evaluate and map the generated dictionary keys to the ground truth keys as you do with LLMSim
- Since this dataset is cross-domain and requires such a specialized knowledge for each domain, it would be challenging to build a good model that performs consistently good across the different domains. Instead, a dataset that is focused on a specific domain and is more thorough would be better

questions:
- Have you considered making more specific questions? For example, in Figure 3 the DFT example where you ask to identify all the input structures and this expects a big dictionary, can you make multiple questions instead for each key of the dictionary? 
Would it make sense to provide only the keys of the dictionary and ask it to complete?
- When you say 434 examples curated from 273 research papers, what would an example entail? 
- Since this benchmark is also about evaluating models in long contexts, a plot of the model performance as a function of the context length would be interesting  
- Figure 18 needs bigger font size for the axis labels
</content>
</thread_object>

</thread>
<thread>
creation date:
1730607456623

thread_type:
official_review

<thread_object>
creator: reviewer_dzb5

<content>
summary:
The paper introduces CURIE, a benchmark for evaluating LLMs on scientific problem-solving that requires understanding long-context information and multi-step reasoning across six disciplines: materials science, condensed matter physics, quantum computing, geospatial analysis, biodiversity, and proteins. CURIE includes 434 tasks derived from 273 scientific papers, covering realistic scientific workflows such as information extraction, concept tracking, aggregation, algebraic manipulation, and multimodal understanding. This benchmark evaluates eight state-of-the-art models, revealing significant performance gaps, especially in tasks like protein sequencing. The paper aims to foster advances in LLM capabilities for scientific applications.

strengths:
1. The paper proposed a new long-context scienceQA benchmark CURIE, containing 434 examples from 273 science literatures in different disciplines. The tasks are carefully collected and benchmarked with the help of domain experts.

2. The paper provided a detailed experimental analysis and many meaningful discussions about the challenges LLMs face in long-context scientific reasoning, especially regarding tasks like protein sequencing and geospatial data extraction, where complex, multi-step reasoning and domain-specific knowledge are crucial. The appendix also included many details for benchmark construction, examples, and case studies, helping readers better understand the annotation process and challenges of the tasks.

3. The paper is well-written and easy to follow. I really appreciate the experts' efforts for annotation and the authors' contribution for conducting this research, and believe this benchmark would be beneficial for both developing better LLMs and better usage of LLM for scientific research.

weaknesses:
1. I'm not sure if LLMs could have seen some of these research papers, which might affect the effectiveness of evaluation results on CURIE. A figure showing the comparison / correlation of performance between CURIE and other scientificQA / long-context QA benchmark would resolve this concern more clearly.

2. I think one of the challenge of such benchmarks should be the difficulty of scaling up? Due to the static manner and limited scale, public papers might be included within future (or existing) corpus for LLM training. I wonder if the authors considered to somehow automatically scale up the dataset, or keep updating the examples?

questions:
See "weakness". I've listed my questions there.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730678516751

thread_type:
official_review

<thread_object>
creator: reviewer_zecm

<content>
summary:
This paper introduces a benchmark called Curie for evaluating Large Language Models (LLMs) on science-related tasks in the disciplines of material science, condensed matter physics, quantum computing, geospatial analysis, biodiversity, and proteins. The benchmark includes 434 examples taken from 273 research papers related to the previously mentioned scientific disciplines. Based on those examples, tasks were created and used to evaluate several LLMs.

strengths:
- The paper is well-written.
- The paper provides a helpful benchmark for evaluating LLMs in science-related tasks.

weaknesses:
- There is no comparison with the baseline case of just inserting the task description into the LLM prompts without providing the paper. It would be useful to see what the results would be in that case.
- Only 6 very specific disciplines (material science, condensed matter physics, quantum computing, geospatial analysis, biodiversity, and proteins) are considered in this dataset. This limits the scientific domain in which LLMs can be tested.
- The dataset construction requires human annotations. This makes scaling up the dataset challenging. 
- Typo line 339: "precison" -> "precision"

questions:
- Have the authors considered inserting the task description into the LLM prompts without providing the paper? How would that compare with the CURIE method? 
- Have the authors considered adding more tasks from a wider range of disciplines as future work?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730697996703

thread_type:
official_review

<thread_object>
creator: reviewer_ewc3

<content>
summary:
The paper introduces the CURIE benchmark, designed to evaluate large language models (LLMs) on multitask scientific problem-solving, focusing on long-context understanding and reasoning. CURIE encompasses ten challenging tasks across six scientific disciplines. These tasks require comprehension of extensive context and domain expertise.

The authors assess various LLMs, highlighting that while Claude-3 performs well across domains, models like GPT-4o struggle with specific tasks, such as protein sequencing. The study finds that current models still have significant room for improvement.

strengths:
- The proposed benchmark focuses on long-context understanding and reasoning in scientific domains, which is relatively unexplored and meaningful for advancing AI4science.
- The paper provides many details and discussion towards the tasks and experiments.

weaknesses:
1. The presentation of this paper is not clear enough. While the authors spent much space introducing the domains and tasks, the input and output (e.g., format, content) are not explicitly defined, which makes it difficult to understand how the authors evaluate the models.
2. The evaluation metrics may not fairly show the performance. The reliance on ROUGE-L and BERTScore may not fully capture the complexity of scientific reasoning. While the authors propose metrics like LLMSim and LMScore, how they extract the answers from open-formatted model outputs and compare the potentially heterogenous answers (e.g., different field name, spelling difference, number/string, with/without extra text) with the gold standard is not clear to me. If not carefully handled, the experimental results might not be reliable.
3. The experiment analysis does not provide many deep and insightful observations.

questions:
1. Could the authors explain more about how you define the input and output format?
2. What is your method to evaluate the free-form generation?
3. What are the approaches for quality control?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730717577728

thread_type:
official_review

<thread_object>
creator: reviewer_rvrn

<content>
summary:
This paper introduces CURIE, a scientific long-context reasoning and information retrieval (IR) benchmark. It contains ten tasks across six scientific domains, all designed to be challenging yet realistic. A wide range of open-source and closed-source large language models (LLMs) are tested, and Claude-3 consistently outperforms other models, including GPT-4.

strengths:
- **Dataset Design**: The dataset is well-designed, covering ten complex tasks (even complex for humans) across six scientific fields. It targets the realistic problems faced by scientists.

- **Annotation Process**: The annotation process is thoroughly explained, and the motivation for selecting each sub-task is clearly articulated.

weaknesses:
**Dataset Size Too Small**:

- “The CURIE benchmark encompasses 434 examples across ten tasks curated from 273 research papers across six diverse scientific disciplines.” The scale of the dataset is somewhat small. Specifically, tasks like PDB and CEO have only 21 and 19 examples, respectively. This limited size may lack statistical significance when comparing models at the sub-task level, making it difficult to determine the reliability of the results given the small sample sizes.

**Issues with LMScore**:

- As mentioned in Appendix A, GPT-4 is used as the language model (LM) to compute LMScore. It would be better to use an open-source model, as GPT models are constantly being updated, which may alter evaluation scores over time. Even when specifying a GPT version, it may become deprecated, making it hard to replicate the results. 

- Additionally, in Figure 9, the results do not convincingly show that LMScore has a high correlation with human judgment. Therefore, it may be challenging to conclude that LMScore can replace ROUGE.

**Presentation Could Be Clearer**:

- It would be beneficial to include a table that specifically lists the number of questions, the number of questions under each task, and the average number of tokens (or words) in queries, documents, and ground truths. Although this information is scattered across Figure 2(b)(c) and Figure 4(c), consolidating it into a table would enhance clarity.

- Line 269: “On the extraction tasks, such as MPV, HFE, GEO, and DFT-S, experts within each domain reviewed each other’s work and reported a high rate of agreement.” It would be better to provide statistics on the inter-agreement between annotators.

questions:
- I don’t fully understand how LLMsim works. If the goal is to identify the number of dictionaries that are correctly retrieved, why not use an exact match or other metrics commonly used for evaluating retrieval, such as nDCG?

- For each paper, is there only one question asked? The paper is not entirely clear on this point, and it raises the question of whether each paper corresponds to only one question or several questions (similar to Qasper).

- In Figure 4(c), does the average length of 954 words refer to the ground truth, or does it refer to the average length of the model-generated responses?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732320216554

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
### We thank all reviewers for their positive comments as well as suggestions and feedback to help improve our work.

### All reviewers agree that a benchmark like CURIE for complex scientific domains is important (CbxB), helpful (zECm), a meaningful advance for AI4Science (EWc3), and would be beneficial for both developing better LLMs and better usage of LLMs for scientific research (dZb5). The reviewers also noted that the dataset is well-designed (rvRN) and carefully-collected (dZb5) with the help of domain experts (CbxB, dZb5), and the annotation process is thoroughly explained (rvRN) with the appendix including many details of the construction (dZb5). The reviewers also appreciated that the motivation for the sub-tasks were clearly articulated (rvRN) and the paper is well-written (CbxB, dZb5, zECm) and provides detailed experimental analysis and many meaningful discussions (CbxB).

Reviewers also included suggestions and requested clarifications. We include responses for some comments noted by multiple reviewers below.

**Number of examples, data size** (rvRN, CbxB)
We agree it would be good to have more examples for some tasks to provide stronger statistical signal. In fact, we did have more papers for GEO and BIOGR domains but had to remove them closer to submission when we discovered they had licenses that prevented re-distribution of the papers. Since submission, *we have gathered several more annotations for the BIOGR task, and are working to add more examples to the PDB tasks* as well to increase each to near 50 to provide a stronger statistical signal. We are still in the process of annotations and running evaluations. Just as a note of comparison, the now popular GPQA (https://arxiv.org/abs/2311.12022) dataset which tests expert level knowledge in biology, chemistry and physics contains 448 multiple choice questions; whereas our dataset covers disciplines requiring narrow expertise and has a wider range and complexity in responses which makes such a dataset more time and resource intensive to create. Thus, we believe the dataset and in particular annotations will have value despite the size.

**Evaluation of open-ended responses, and LLMSim** (rvRN, EWc3, CdxB)

We agree with reviewers that evaluation of open-ended responses is challenging. The advantage of ROUGE-L, BERTScore and LMScore metrics are that they do not require parsing of model output text and we agree these have limitations.

We note that for tasks where a structured output is preferred (e.g. GEO, DFT, MPV) our prompts provide the required JSON response format, and most LLM models – closed and open models – appear to adhere to instructions somewhat well to produce a JSON response in most cases. Our proposed LLMSim metric is used for some of these tasks with JSON responses. In the evaluation code included in the supplement zip file,  we incorporate several heuristics to give additional leeway when model responses do not conform to this format and do a best effort match between ground truth elements and model responses by making several calls to the LLM. 

Overall, for the retrieval tasks such as DFT-S, DFT-P, MPV, we recommend LLMSim since it is designed and better suited for these retrieval tasks. For summarization and aggregation tasks such as HFE, HFD, QECC, and DFT-C, the existing ROUGE and BERTScore metrics are appropriate as is. We use LMScore as an additional model based evaluation to provide signal, and is not necessarily meant to replace other metrics just yet. For BIOGR, we  use Intersection Over Union; and for PDB we use ID_{ratio} which are both designed by domain experts and appropriate metrics well suited for the tasks.


**Automatic scaling of dataset** (zECm, dZb5)

We agree such dataset creation requires human (and actually expert) annotations and scaling this is a challenge. CURIE is a new benchmark that fills in the gap in existing literature and curate a range of problem solving tasks in science. For instance, early works such as [1], which aimed to look at the impact of LLMs for science were a collection of anecdotal examples noted by experts. In contrast, our work aims to provide a concrete measurable benchmark. Similarly, the very recent FrontierMath [2] benchmark was also created with the help of expert mathematicians and professors coming together, and required human annotation. As we progress, we believe we might be able to identify datasets and tasks from experts in different scientific fields that might enable easier scaling. For instance, the PDB task in our dataset is quite amenable to scaling since there is a programmatic way to generate the solutions and the evaluation is also much more concrete. With newer agentic frameworks developing, we will think more about how examples may be automatically updated and verified for some of these tasks. We can include this as a future direction.


[1] https://arxiv.org/abs/2311.07361

[2] FrontierMath: https://arxiv.org/abs/2411.04872
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary of the paper:
This paper introduces CURIE, a benchmark designed to evaluate LLMs on scientific problem-solving tasks that require long-context understanding and multi-step reasoning across six disciplines: materials science, condensed matter physics, quantum computing, geospatial analysis, biodiversity, and proteins. CURIE consists of ten challenging tasks derived from 434 examples taken from 273 research papers, emphasizing realistic scientific workflows such as information extraction and concept tracking. The authors assess various LLMs, noting that Claude3 consistently outperforms others, including GPT4, in specific tasks like protein sequencing. The study identifies significant performance gaps across LLMs, highlighting the need for improvements in LLM capabilities for scientific applications. Additionally, a new LLM-based evaluation metric is introduced, along with evaluations conducted by domain experts on LLM predictions.

Strengths of the paper:
- Comprehensive Benchmark: Introduces the CURIE benchmark with 434 examples from 273 science literature, validated by domain experts. Covers ten complex tasks across six scientific fields, targeting realistic problems faced by scientists.
- Contribution to Scientific Research: Provides a valuable tool for the development and application of language models in scientific contexts.
- Detailed Experimental Analysis: Assesses various LLMs and discusses challenges faced by LLMs in long-context reasoning, highlighting their limitations and areas for improvement in scientific reasoning tasks.

Weaknesses of the paper:
- Small Dataset Size (Reviewers rvRN, CbxB): CURIE consists of only 434 examples, with certain tasks like GEO and BIOGR having as few as 19-21 examples, limiting statistical significance and reliability at the sub-task level.
- Challenges in Dataset Scaling (Reviewers zECm, dZb5): The requirement for human annotations makes it difficult to scale the dataset effectively, potentially hindering future research. Additionally, concerns exist about whether LLMs may have been trained on the same research papers included in the CURIE benchmark, affecting evaluation outcomes.
- Evaluation Difficulties with Open-ended Tasks (Reviewers rvRN, EWc3, CdxB): For open-ended tasks, the variability in model responses makes it challenging to assess correctness, particularly when using different notations or steps. The evaluation relies on GPT-4 to compute LMScore, which may vary over time due to updates. Additionally, the correlation between LMScore and human judgment is not convincingly established, raising doubts about its effectiveness compared to ROUGE.
- Lack of Clarity in Presentation (Reviewers rvRN, EWc3): The paper could use tables to make the presentation clearer.

Reasons for the decision:
After considering the rebuttal, I believe that most of the concerns raised have been adequately addressed by the authors, as outlined in the reviewer discussion box. All reviewers agree that this paper should be accepted at ICLR. Upon careful reflection, I find this paper to be a valuable resource, as it delves into an unexplored area within AI for science. It has the potential to engage the broader community and make an impact. However, I recommend that the authors release updated benchmarks with additional annotations for the BIOGR and PDB tasks for the camera-ready version of the paper.