CURIE: Evaluating LLMs On Multitask Scientific Long Context
  Understanding and Reasoning
Authors: name: Hao Cui, name: Zahra Shamsi, name: Gowoon Cheon, name: Xuejian Ma, name: Shutong Li, name: Maria Tikhanovskaya, name: Peter Norgaard, name: Nayantara Mudur, name: Martyna Plomecka, name: Paul Raccuglia, name: Yasaman Bahri, name: Victor V. Albert, name: Pranesh Srinivasan, name: Haining Pan, name: Philippe Faist, name: Brian Rohr, name: Michael J. Statt, name: Dan Morris, name: Drew Purves, name: Elise Kleeman, name: Ruth Alcantara, name: Matthew Abraham, name: Muqthar Mohammad, name: Ean Phing VanLee, name: Chenfei Jiang, name: Elizabeth Dorfman, name: Eun-Ah Kim, name: Michael P Brenner, name: Viren Jain, name: Sameera Ponda, name: Subhashini Venugopalan

Scientific problem-solving involves synthesizing information while applying
expert knowledge. We introduce CURIE, a scientific long-Context
Understanding,Reasoning and Information Extraction benchmark to measure the
potential of Large Language Models (LLMs) in scientific problem-solving and
assisting scientists in realistic workflows. This benchmark introduces ten
challenging tasks with a total of 580 problems and solution pairs curated by
experts in six disciplines - materials science, condensed matter physics,
quantum computing, geospatial analysis, biodiversity, and proteins - covering
both experimental and theoretical work-flows in science. We evaluate a range of
closed and open LLMs on tasks in CURIE which requires domain expertise,
comprehension of long in-context information,and multi-step reasoning. While
Gemini Flash 2.0 and Claude-3 show consistent high comprehension across
domains, the popular GPT-4o and command-R+ fail dramatically on protein
sequencing tasks. With the best performance at 32% there is much room for
improvement for all models. We hope that insights gained from CURIE can guide
the future development of LLMs in sciences. Evaluation code and data are in
https://github.com/google/curie