SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models

## Abstract

Abstract Large language models (LLMs) have gained increasing prominence in scientific research, but there is a lack of comprehensive benchmarks to fully evaluate their proficiency in understanding and mastering scientific knowledge.
To address this need, we introduce the SciKnowEval benchmark, a novel framework that systematically evaluates LLMs across five progressive levels of scientific knowledge: studying extensively , inquiring earnestly , thinking profoundly , discerning clearly , and practicing assiduously. These levels aim to assess the breadth and depth of scientific knowledge in LLMs, including memory, comprehension, reasoning, discernment, and application.
Specifically, we first construct a large-scale evaluation dataset encompassing 70K multi-level scientific problems and solutions in the domains of biology, chemistry, physics, and materials science.
By leveraging this dataset, we benchmark 26 advanced open-source and proprietary LLMs using zero-shot and few-shot prompting strategies. The results reveal that despite the state-of-the-art performance of proprietary LLMs, there is still significant room for improvement, particularly in addressing scientific reasoning and applications.
We anticipate that SciKnowEval will establish a standard for benchmarking LLMs in science research and promote the development of stronger scientific LLMs.

## 1 Introduction

Recent advancements in large language models (LLMs) have demonstrated an impressive capability in storing and recalling world knowledge, continuously expanding the boundaries of artificial intelligence. Their exceptional performance has permeated diverse specialized domains, including the scientific domain, leading to the emergence of scientific LLMs, such as Galactica [], SciGLM [], and ChemLLM [].
To steadily advance scientific research, it is crucial to establish reliable benchmarks that comprehensively evaluate these models’ capability in handling scientific knowledge.

While several existing LLM benchmarks [, , ] have incorporated scientific questions into their evaluations, and some benchmarks [, , , , , ] are specifically tailored for the scientific domain, we argue that the current benchmarks do not fully evaluate the potential of LLMs in scientific research due to their inherent limitations.
Firstly, many existing benchmarks, such as AGIEval [], SciQ [], and ScienceQA [], include science questions only up to the high school level, failing to tap into the deeper capability of LLMs. Secondly, recent scientific domain benchmarks like ChemLLMBench [], SciBench [], and SciAssess [], despite involving more specialized scientific tasks, lack a comprehensive evaluation system, resulting in a limited understanding of capabilities. Lastly, most benchmarks overlook the assessment of safety issues in scientific research, even those attempting a multi-dimensional comprehensive evaluation such as SciEval [].

In response to these deficiencies, in this study, we adopt a distinctive perspective, “LLMs as Scientists”, to revisit the evaluation in the scientific domain. We draw inspiration from the profound principles of Confucius outlined in the ancient Chinese philosophy “Doctrine of the Mean” , and present a novel Scientific Knowledge Evaluation benchmark, referred to as SciKnowEval, as illustrated in Fig. 1. This benchmark aims to assess LLMs based on their proficiency in five progressive levels: studying extensively, enquiring earnestly, thinking profoundly, discerning clearly, and practicing assiduously. Each level offers a unique perspective on evaluating the capabilities of LLMs in handling scientific knowledge, including memory, comprehension, reasoning, discernment, and application.

Different from the widely adopted Bloom’s Taxonomy framework [], we aim to encourage a balanced and comprehensive view of knowledge and its application, fostering a model’s capacity to think and act responsibly.
In comparison to existing benchmarks, SciKnowEval mainly has the following characteristics: (1) It designs a systematic scientific knowledge evaluation framework that encompasses five progressive levels to mirror the learning process of humans.
(2) It uses data from diverse sources, including scientific textbooks, literature, and databases, making it diverse and large-scale.
(3) It places significant emphasis on scientific ethics and safety while comprehensively evaluating capabilities.
Table 1 shows the detailed comparison of SciKnowEval with other benchmarks.

Figure: Figure 1: Illustration of SciKnowEval. We consider LLMs as scientists and evaluate their performance based on a hierarchical framework inspired by the ancient Chinese philosophy from “ Doctrine of the Mean”, encompassing five progressive levels that span from knowledge memory to application. We also construct a large-scale dataset comprising various scientific domain tasks at different levels, along with corresponding evaluation criteria, leading to a reliable benchmark of LLMs.
Refer to caption: x1.png

SciKnowEval represents a comprehensive benchmark for assessing the capability of LLMs in processing and utilizing scientific knowledge. It aims to promote the development of scientific LLMs that not only possess extensive knowledge but also demonstrate ethical discernment and practical applicability, ultimately contributing to the advancement of scientific research.
We summarize the contributions of this paper as follows:

- •
We propose a multi-level scientific knowledge evaluation framework that targets critical aspects of knowledge handling by LLMs, encompassing memory, comprehension, reasoning, discernment, and application.
- •
We construct a large-scale evaluation dataset comprised of 70K diverse scientific problems from the domains of biology, chemistry, physics and material science, accompanied by corresponding solutions and evaluation metrics, facilitating an extensive assessment of the breadth and depth of scientific knowledge encapsulated in LLMs.
- •
We evaluate a wide range of advanced LLMs (including 18 general-purpose LLMs and 8 scientific LLMs) and rank their performance with the SciKnowEval dataset, elucidating both their strength and weaknesses.

**Table 1: Comparison between SciKnowEval and other benchmarks.**
| Statistics | Number |
| --- | --- |
| Total Questions | 70,203 |
| Subjects/Tasks | 4/78 |
| L1 Questions | 39,264 (55.93%) |
| L2 Questions | 12,896 (18.37%) |
| L3 Questions | 8,368 (11.92%) |
| L4 Questions | 5,257 (7.49%) |
| L5 Questions | 4,418 (6.29%) |

## 2 Related Works

Assessing proficiency in scientific knowledge is a crucial aspect of LLM evaluation. For example, renowned benchmarks like MMLU [], AGIEval [], and ARC [] have incorporated a number of scientific questions into their assessments.
Recently, with the rapid application of LLMs in science research, several benchmarks tailored for scientific domains have been developed.
SciQ [] encompasses 13K crowdsourced science examination questions, covering diverse subjects including Physics, Chemistry, Biology, and more.
ScienceQA [] consists of 21K multimodal scientific questions and answers (QA) collected from elementary and high school science curricula, focusing on evaluating the interpretability of LLMs in addressing scientific problems.
SciBench [] collects open-ended questions from college-level textbooks in physics, chemistry, and mathematics to assess reasoning abilities for complex scientific problems.
SciEval [], based on Bloom’s Taxonomy, offers a multidisciplinary, multi-level dataset to assess understanding, application, computation, and research abilities.
SciAssess [] introduces a multimodal scientific literature analysis dataset, encompassing tasks like information extraction and chart/table QA.
SciMT-Safety [] presents a red-teaming benchmark with 432 malicious queries, focusing on potential misuse risks of LLMs.
ChemBench [] establishes a comprehensive benchmark consisting of 7,059 questions designed to assess the expertise and safety of LLMs across 11 subfields of chemistry.
LAB-Bench [] introduces the language agent biology benchmark, which includes over 2,400 multiple-choice questions aimed at evaluating AI systems on a range of practical biology research capabilities.

Table 1 has reported the statistics of these benchmarks. One can observe that they suffer from several limitations, such as inadequate scope of ability examination, limited data size, and insufficient task diversity.
In this study, we develop a systematic scientific knowledge evaluation framework and construct a large-scale dataset with multi-level domain tasks to benchmark the general and scientific LLMs.

## 3 The SciKnowEval Dataset

### 3.1 Design Philosophy

The design philosophy of SciKnowEval is inspired by the profound principles of Confucius elucidated in the ancient Chinese book “Doctrine of the Mean”: Studying extensively, Enquiring earnestly, Thinking profoundly, Discerning clearly, and Practicing assiduously. This principle reflects the five progressive levels in the human learning process. In this study, we regard LLMs as Scientists and utilize this concept to evaluate them.
Specifically, each level provides a perspective to assess the proficiency of LLMs, as described below.

- •
L1: Studying extensively (i.e., knowledge memory).
This dimension evaluates the breadth of an LLM’s knowledge across various scientific domains. It measures the model’s ability to remember a wide range of scientific concepts.
- •
L2: Enquiring earnestly (i.e., knowledge comprehension).
This aspect focuses on the LLM’s capacity for inquiry and exploration within scientific contexts, such as analyzing scientific texts, identifying key concepts, and questioning relevant information.
- •
L3: Thinking profoundly (i.e., knowledge reasoning).
This criterion examines the model’s capacity for critical thinking, logical deduction, numerical calculation, function prediction, and the ability to engage in reflective reasoning to solve problems.
- •
L4: Discerning clearly (i.e., knowledge discernment).
This aspect evaluates the LLM’s ability to make correct, secure, and ethical decisions based on scientific knowledge, including assessing the harmfulness and toxicity of information, and understanding the ethical implications and safety concerns related to scientific endeavors.
- •
L5: Practicing assiduously (i.e., knowledge application).
The final dimension assesses the LLM’s capability to apply scientific knowledge effectively in real-world scenarios, such as solving complex scientific problems and creating innovative solutions.

Building upon the above design philosophy, we develop the SciKnowEval benchmark specifically tailored for assessing multi-level scientific knowledge in LLMs. In particular, we undertake meticulous designs in terms of data scale, diversity and quality when constructing the evaluation dataset:

- •
Large-scale. We architect our dataset to be large-scale, enabling a more accurate and robust assessment of LLMs.
- •
Multi-level. We design and construct our datasets to encompass a wide range of tasks, spanning multiple levels of scientific knowledge, to comprehensively assess the breadth and depth of knowledge in LLMs.
- •
High-quality. We prioritize the quality of our data through rigorous quality control measures, ensuring the reliability of the proposed dataset.

### 3.2 Data Collection Methods

Fig. 2 illustrates three data collection approaches employed in SciKnowEval, including generating questions&answers (QAs) from the literature or textbooks, refactoring the existing QAs, as well as transforming the traditional scientific datasets into textual formats suitable for LLMs. We elaborate on these methods as follows.

#### I. Generating New QAs from Literature Corpus.

Literature and textbooks cover a broad range of scientific knowledge, and leveraging this data will facilitate a comprehensive evaluation of LLMs’ capabilities in the scientific domains. We collect massive papers from article preprint platforms (e.g., BioRxiv), literature databases (e.g., PubMed), and textbook databases (e.g., LibreTexts).
We utilize LLMs to automate the procedures of QA pair generation. Specifically, following domain experts’ advice, we carefully design effective prompts for literature QA tasks. These prompts exhibited in Appendix A5 guide the LLM to extract relevant professional knowledge from literature and textbook paragraphs, enabling it to generate new QA pairs around this expertise. To ensure quality assessment of the generated questions, we emphasize in the prompts that answers must be explicitly found in the original text without introducing any external information.

#### II. Refactoring the Existing QAs.

We sample additional QAs from existing open-source scientific benchmarks, including MedMCQA [], SciEval [], MMLU [], XieZhi [], PubMedQA [], and HarmfulQA []. To mitigate the risk of data contamination and leakage in these benchmarks, we employ LLMs to refactor these QAs in various forms, such as question rewriting and option reordering. Moreover, in cases where some QAs lack explicit annotations indicating their corresponding levels in SciKnowEval, LLMs are utilized to automatically categorize the data into distinct levels.

#### III. Transforming the Scientific Databases.

To enhance the variety and scope of tasks in our dataset, we select several structured databases and transform them into textual formats suitable for evaluating LLMs. These databases mainly include molecular (e.g., PubChem []), protein (e.g., UniProtKB []), and cellular-related (e.g., SHARE-seq []) sequence information, which contain annotations related to structure, properties, and functions. We can utilize these annotations to construct QA pairs. Specifically, we first conduct preliminary quality screening, such as filtering out chemically invalid SMILES from PubChem using the RDKit library. We then design multiple question templates to transform the structured sequence-annotation pairs into natural language formats, including multiple-choice questions, true/false questions, and short-answer questions.

Figure: Figure 2: An illustration of data collection approaches in SciKnowEval, including I) generating new QAs from the literature corpus, II) refactoring the existing QAs, and III) transforming the conventional scientific databases into QAs.
Refer to caption: x2.png

### 3.3 Data Quality Control

The primary concern of data quality is the QAs generated from literature corpus.
To ensure the generated data with high quality, we employed a three-stage data screening process:

- •
Initial screening by LLMs. We first explicitly instructed LLM during data generation that the correct options must be clearly identifiable from the provided literature snippets. After data generation, we prompt LLMs to simulate an open-book exam task to determines whether each question’s answer can be found in the corresponding snippet.
- •
Human evaluation. We randomly selected approximately 5% of the generated questions and provided them with two domain experts. During the evaluation, we used the instructions in Table A11 to guide the evaluators, asking them to thoroughly assess the data and classify it into binary categories of “Yes” and “No” for quality.
- •
Post-screening by LLMs. We employed LLMs to summarize the failure types of the identified low-quality entries, and added them into the prompt to conduct a full dataset quality assessment, discarding similar types of low-quality questions.

More details of data quality control are provided in Appendix A7.

### 3.4 Final Datasets

Based on the above data collection and screening process, we construct the SciKnowEval dataset, consisting of four subsets for Biology, Chemistry, Physics and Materials, respectively. Table 3.4 shows the overall statistics of the constructed dataset.
Table 3 provides an overview of the biological dataset, and Table A8 summarizes the datasets for other domains. Detailed question examples are available in Appendix A6.

**Table 2: Statistics of the SciKnowEval dataset.**
| Statistics | Number |
| --- | --- |
| Total Questions | 70,203 |
| Subjects/Tasks | 4/78 |
| L1 Questions | 39,264 (55.93%) |
| L2 Questions | 12,896 (18.37%) |
| L3 Questions | 8,368 (11.92%) |
| L4 Questions | 5,257 (7.49%) |
| L5 Questions | 4,418 (6.29%) |

**Table 3: Overview of the biological dataset in SciKnowEval. Abbr., MCQ: multiple choice questions; T/F: true/false; CLS: classification; RE: relation extraction; GEN: generative task. Data collection methods I, II and III are in Fig. 2. The detailed data source listed in Appendix A4. The overview of other domain datasets (chemistry, physics and materials) is presented in Table A8.**
| Domain | Ability | Task Name | Task Type | Data Source | Method | #Questions |
| --- | --- | --- | --- | --- | --- | --- |
| Bio. | L1 | Biological Literature QA | MCQ | Literature Corpus | I | 14,862 |
| Protein Property Identification | MCQ | UniProtKB | III | 1,500 |  |  |
| Protein Captioning | GEN | UniProtKB | III | 930 |  |  |
| L2 | Drug-Drug Relation Extraction | RE | Bohrium | II | 464 |  |
| Biomedical Judgment and Interpretation | T/F | PubMedQA | II | 947 |  |  |
| Compound-Disease Relation Extraction | RE | Bohrium | II | 500 |  |  |
| Gene-Disease Relation Extraction | RE | Bohrium | II | 203 |  |  |
| Detailed Understanding | MCQ | LibreTexts | I | 828 |  |  |
| Text Summary | GEN | LibreTexts | I | 1,291 |  |  |
| Hypothesis Verification | T/F | LibreTexts | I | 618 |  |  |
| Explanation | MCQ | LibreTexts | I | 648 |  |  |
| L3 | Solubility Prediction | MCQ | PEER, DeepSol | III | 207 |  |
| $\beta$-lactamase Activity Prediction | MCQ | PEER, Envision | III | 203 |  |  |
| Fluorescence Prediction | MCQ | PEER, Sarkisyan’s | III | 203 |  |  |
| GB1 Fitness Prediction | MCQ | PEER, FLIP | III | 208 |  |  |
| Stability Prediction | MCQ | PEER, Rocklin’s | III | 204 |  |  |
| Protein-Protein Interaction | MCQ | STRING, SHS27K, SHS148K | III | 207 |  |  |
| Biological Calculation | MCQ | MedMCQA, SciEval, MMLU | II | 60 |  |  |
| L4 | Biological Harmful QA | GEN | Website | I | 297 |  |
| Proteotoxicity Prediction | MCQ, T/F | UniProtKB | III | 510 |  |  |
| Biological Laboratory Safety Test | MCQ, T/F | LabExam (ZJU) | II | 192 |  |  |
| L5 | Biological Protocol Procedure Design | GEN | Protocol Journal | I | 577 |  |
| Biological Protocol Reagent Design | GEN | Protocol Journal | I | 588 |  |  |
| Protein Design | GEN | UniProtKB | III | 949 |  |  |
| Single Cell Analysis | GEN | SHARE-seq | III | 300 |  |  |

## 4 Experiments

### 4.1 Experimental Setup

#### Models.

We select 26 widely-used and high-performing LLMs. These models are categorized into three types based on their accessibility and purpose. The details about the implementation of models can be found in Appendix A2.

- •
Proprietary LLMs: This category generally represents state-of-the-art. Included models are OpenAI’s GPT series [], specifically OpenAI’s GPT-4o, GPT-4o-mini, GPT-4-Turbo, GPT-3.5-Turbo, Anthropic’s Claude-3.5-Sonnet, Claude3-Sonnet [], Google’s Gemini1.5-Pro [], and Alibaba’s Qwen-Max [].
- •
Open-Source General-Purpose LLMs: These LLMs excel in general domains and serve as a foundation for further research into scientific LLMs. We selected seven LLMs from diverse sources, ranging in size from 7B to 72B, including Qwen2-7/72B-Inst [], Llama3-70B-Inst [], Qwen1.5-7/14B-Chat [], Llama3-8B-Inst [], Llama2-13B-Chat [], ChatGLM3-6B [], Gemma1.1-7B-Inst, and Mistral-7B-Inst [].
- •
Open-Source Scientific LLMs: These models have acquired specialized knowledge by training on scientific domain data. We selected models focused on biology and chemistry, including ChemDFM-13B [], Galactica-6.7B/30B [], ChemLLM-7B/20B-Chat [], MolInst-Llama3-8B [], LlaSMol-Mistral-7B [], and SciGLM-6B [].

#### Evaluation Mode.

In our experiments, the input begins with a system prompt that describes the types and categories of questions. We employ two evaluation settings: zero-shot and few-shot. The zero-shot setting evaluates models’ problem-solving capabilities without any prior examples, testing their ability to solve problems based on their own inherent knowledge. In the few-shot setting, models are provided with a limited number of examples before the test example to assess their capability to acquire and incorporate new information into their problem-solving processes.

#### Evaluation Criteria.

We adopt diverse evaluation metrics, tailoring our assessment to different task types. When evaluating True/False, classification and multiple-choice questions, we use accuracy as the performance metric. For relation extraction questions, we use the $F_{1}$-score that combines precision and recall.
For generative questions, we adopted different evaluation methods tailored to the characteristics of each task. Specifically, following the existing works [, , , , ], we calculate the average scores using BLEU [] and ROUGE [] metrics for the tasks of molecule captioning, protein captioning, and single cell analysis. For molecule generation, we compute the average of the exact match rate and fingerprint-based Tanimoto similarity []. For protein generation, we leverage the Smith-Waterman algorithm [] to perform local sequence alignment of two protein sequences. For other generative tasks, we designed meticulous prompts for GPT-4o to evaluate the responses of LLMs.
The scoring prompt templates are exhibited in Appendix A9.
Considering the challenge of aggregating different metrics, we report the average rankings of LLMs in each task as the final score.

### 4.2 Results and Analyses

In this section, we report the performance of LLMs in the SciKnowEval dataset.
Table 4 and Table A1.1 summarize the zero-shot performance rankings of LLMs at each level, offering valuable insights into the strengths and weaknesses exhibited by each model.
We emphasize our key observations as follows, and the illustrative examples can be found in Appendix A10.

**Table 4: Overall zero-shot performance of LLMs across five levels in four domains. A smaller value indicates a higher ranking. Bold results indicate the best results among all models, underline results indicate the second-best results, and blue results indicate the best results among the open-source models.**
| Categories | Models | L1 | L2 | L3 | L4 | L5 | All | Rank |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Proprietary<br>LLMs | Claude3.5-Sonnet | 2.70 | 4.80 | 4.00 | 2.90 | 2.27 | 3.71 | 1 |
| GPT-4o | 2.30 | 4.56 | 5.95 | 6.40 | 3.00 | 4.68 | 2 |  |
| GPT-4-Turbo | 6.40 | 6.12 | 8.59 | 7.50 | 5.09 | 6.88 | 4 |  |
| Gemini1.5-Pro-latest | 8.20 | 8.44 | 6.00 | 5.10 | 7.18 | 7.12 | 5 |  |
| GPT-4o-mini | 9.10 | 7.80 | 12.14 | 6.30 | 4.73 | 8.56 | 7 |  |
| Qwen-Max | 7.90 | 7.76 | 9.27 | 7.90 | 10.36 | 8.59 | 8 |  |
| Claude3-Sonnet | 9.20 | 8.92 | 10.82 | 9.60 | 6.00 | 9.17 | 9 |  |
| GPT-3.5-Turbo | 11.60 | 13.24 | 14.82 | 10.80 | 10.55 | 12.78 | 12 |  |
| Open-Source<br>General-Purpose<br>LLMs | Qwen2-72B-Inst | 4.90 | 5.28 | 8.55 | 4.20 | 7.64 | 6.35 | 3 |
| Llama3-70B-Inst | 7.90 | 6.24 | 8.86 | 5.30 | 7.18 | 7.21 | 6 |  |
| Qwen2-7B-Inst | 12.40 | 11.40 | 14.14 | 9.70 | 14.09 | 12.46 | 10 |  |
| Qwen1.5-14B-Chat | 12.40 | 13.36 | 11.95 | 13.60 | 11.91 | 12.67 | 11 |  |
| Llama3-8B-Inst | 12.80 | 12.32 | 14.73 | 11.80 | 17.00 | 13.65 | 13 |  |
| Qwen1.5-7B-Chat | 15.70 | 15.60 | 17.50 | 16.60 | 17.82 | 16.59 | 17 |  |
| Gemma1.1-7B-Inst | 18.90 | 20.40 | 15.59 | 17.70 | 15.64 | 17.83 | 18 |  |
| Mistral-7B-Inst | 20.20 | 16.88 | 18.59 | 15.30 | 18.64 | 17.83 | 18 |  |
| ChatGLM3-6B | 19.00 | 20.56 | 18.64 | 18.00 | 17.64 | 19.08 | 20 |  |
| Llama2-13B-Chat | 21.80 | 18.56 | 21.73 | 18.00 | 17.45 | 19.64 | 22 |  |
| Open-Source<br>Scientific<br>LLMs | ChemDFM-13B | 12.50 | 15.24 | 14.45 | 15.10 | 16.09 | 14.77 | 14 |
| ChemLLM-20B-Chat | 15.00 | 12.80 | 16.27 | 19.60 | 16.82 | 15.50 | 15 |  |
| MolInst-Llama3-8B | 17.40 | 15.88 | 12.41 | 16.80 | 18.73 | 15.62 | 16 |  |
| Galactica-30B | 17.20 | 21.48 | 16.09 | 22.80 | 19.82 | 19.35 | 21 |  |
| SciGLM-6B | 21.70 | 20.32 | 19.41 | 22.40 | 21.55 | 20.68 | 23 |  |
| ChemLLM-7B-Chat | 20.30 | 21.16 | 20.36 | 21.90 | 20.09 | 20.77 | 24 |  |
| Galactica-6.7B | 21.60 | 23.60 | 17.18 | 22.50 | 24.00 | 21.45 | 25 |  |
| LlaSMol-Mistral-7B | 22.60 | 23.84 | 20.59 | 25.90 | 20.91 | 22.62 | 26 |  |

#### Overall Performance.

Proprietary LLMs, such as the GPT-4 series and Claude-3.5, have consistently demonstrated superior performance across these four domains, securing their highest overall rankings. Notably, Claude3.5-Sonnet has exhibited exceptional capability and adaptability in scientific domains.
Open-source LLMs with larger scales, including Llama3-70B and Qwen2-72B, also exhibited comparable performance. In contrast, scientific LLMs performed moderately and only showcased strengths in a few tasks.

#### Performance on Each Level.

We then analyze the performance of LLMs on the five levels.
Table A4, A4, A6, and A6 show the scores of LLMs on each task at each level.

L1 reflects the model’s memory of scientific knowledge. Proprietary LLMs, such as GPT-4o demonstrated the best capabilities in four domains, showcasing its extensive knowledge coverage. In the medium-scale ($\sim$10B) LLMs, ChemDFM emerged as one of the top open-source models by continuing pre-training and fine-tuning on a vast corpus of scientific literature. However, many scientific LLMs, such as LlaSMol-Mistral-7B, lagged behind, possibly due to overfitting caused by specific instruction fine-tuning.

L2 measures the model’s comprehension ability within scientific contexts. GPT-4o and other proprietary LLMs showcased strong text comprehension performance, which also included open-source models like Qwen2-series. However, they struggled with tasks involving relation extraction.

L3 evaluates the model’s reasoning and computational abilities for scientific questions. In the biological domain, despite GPT-4o and Gemini1.5-Pro demonstrating relatively higher average rankings, they did not exhibit significant superiority in protein function prediction tasks. In the chemical domain, GPT-4o performed relatively better in tasks such as reaction prediction, retrosynthesis, and chemical calculation, but there remains substantial room for improvement in other tasks.
Overall, all evaluated models need further enhancement in scientific computation.

L4 highlights the model’s awareness of scientific safety. For harmful QA tasks in biology and chemistry, LLMs are expected to refuse to answer harmful scientific questions. Gemini1.5-Pro showed strong safety judgment, with refusal rates of 81.9% in chemistry and 100% in biology. Claude3.5-Sonnet showed similar performance. However, other models, including the GPT-4 series, underperformed. In molecular toxicity prediction, only a few LLMs exceeded 40% accuracy, revealing their limitations in assessing molecular toxicity. Lastly, in laboratory safety tests, proprietary models like GPT-4o excelled, showing promise for safe lab operations.

L5 reflects the creative abilities of LLMs in real-world scientific scenarios, determining their potential in experimental protocol design, drug design, and so on. For the protocol design tasks in both biology and chemistry, we prompt GPT-4o to rate results from 1 to 5. However, despite proprietary models like GPT-4o outperforming others, almost no model achieved an average score of 3 out of 5. This indicates that existing models are still unable to generate high-quality experimental protocols.
In the molecular generation tasks, scientific LLMs such as LlaSMol-Mistral-7B and ChemDFM-13B significantly outperformed other models, which we attribute to their training data involving molecular description and generation. In the protein design tasks, none of the current LLMs delivered satisfactory results, with average normalized Swith-Waterman alignment scores approaching zero. Additionally, performance bottlenecks were also encountered in the single-cell analysis task. In summary, the creative capabilities of LLMs related to molecules, proteins and cells require further improvement.

**Table 5: 3-shot performance of LLMs on each task in the biology domain. Bold indicates performance improvement compared to the zero-shot setting, while gray signifies no gain. Blue results indicate a large improvement (>10%). The 3-shot results of other domains are provided in Table A2.**
| Domain | Task Name | GPT-4o | Claude3 | Llama3-8B | Qwen1.5-7B | MolInst-8B | ChemLLM-7B |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0-shot | 3-shot | 0-shot | 3-shot | 0-shot | 3-shot | 0-shot | 3-shot | 0-shot | 3-shot | 0-shot | 3-shot |  |  |
| Bio. | Bio LiterQA (L1) | 83.71 | +0.45 | 76.44 | -0.72 | 74.82 | +2.15 | 72.06 | +0.69 | 72.82 | +2.30 | 61.65 | +8.30 |
| Prot Prop Iden. (L1) | 34.20 | +10.53 | 27.93 | +4.47 | 25.60 | +3.73 | 26.40 | -1.8 | 20.53 | +3.54 | 22.20 | +2.73 |  |
| Protein Cap. (L1) | 0.122 | +0.005 | 0.118 | +0.002 | 0.112 | -0.003 | 0.100 | +0.028 | 0.009 | +0.153 | 0.065 | +0.049 |  |
| D-D RE (L2) | 17.41 | +0.22 | 16.02 | +0.86 | 16.75 | +0.26 | 16.93 | -0.53 | 14.86 | -0.69 | 6.47 | +3.27 |  |
| Bio JI (L2) | 95.46 | -0.53 | 96.20 | -0.32 | 89.12 | +5.92 | 87.33 | +2.22 | 92.71 | 5.39 | 26.08 | +39.92 |  |
| C-D RE (L2) | 37.70 | +10.59 | 35.85 | +8.5 | 28.90 | +24.09 | 21.27 | +6.42 | 50.26 | -5.42 | 13.34 | +11.08 |  |
| G-D RE (L2) | 36.20 | -1.15 | 29.18 | -5.75 | 39.16 | -0.52 | 8.74 | +24.72 | 10.38 | +0.75 | 12.65 | +6.34 |  |
| Bio DU (L2) | 99.40 | +0.00 | 98.67 | -0.72 | 98.79 | -0.23 | 97.58 | -0.48 | 96.50 | +0.84 | 96.01 | -0.84 |  |
| Bio HV (L2) | 94.82 | -0.16 | 94.98 | +0.00 | 90.45 | +0.16 | 85.60 | +0.00 | 89.48 | -5.18 | 61.97 | +16.99 |  |
| Bio RI (L2) | 97.69 | +0.00 | 96.60 | +0.47 | 96.76 | -0.43 | 95.22 | -0.16 | 93.21 | -0.46 | 91.67 | -0.47 |  |
| Solu. Pred (L3) | 48.31 | +5.31 | 46.86 | +1.45 | 49.28 | -0.49 | 49.28 | +4.34 | 53.62 | +0.00 | 46.86 | +0.48 |  |
| $\beta$-LA Pred (L3) | 50.25 | +13.30 | 43.84 | +24.63 | 50.25 | +5.42 | 27.59 | +24.63 | 47.78 | +20.20 | 61.58 | -13.80 |  |
| Fluo. Pred (L3) | 51.72 | +27.10 | 51.72 | +28.58 | 49.75 | +9.36 | 46.31 | +14.28 | 44.83 | +22.17 | 20.69 | +33.00 |  |
| GB1 Pred (L3) | 32.54 | -0.40 | 24.40 | +4.31 | 19.14 | +4.78 | 13.40 | +15.79 | 28.23 | -0.24 | 16.27 | +15.79 |  |
| Stab. Pred (L3) | 24.02 | +1.96 | 30.39 | -3.43 | 24.51 | +2.45 | 22.55 | 15.69 | 28.92 | -0.59 | 26.47 | +4.41 |  |
| Prot-Prot Inter. (L3) | 35.27 | -6.77 | 27.54 | -1.94 | 21.26 | +11.59 | 22.22 | +3.87 | 25.60 | +6.28 | 20.77 | 10.15 |  |
| Bio Cal. (L3) | 58.33 | -8.33 | 38.33 | -3.33 | 33.33 | +5.00 | 36.67 | +10.00 | 41.67 | -3.34 | 21.67 | +15.00 |  |
| Proteotox. (L4) | 85.88 | -0.59 | 40.88 | +20.3 | 38.24 | +3.13 | 39.71 | +10.49 | 37.06 | +0.98 | 22.06 | +9.90 |  |
| Bio Safe Test (L4) | 86.67 | +0.44 | 71.11 | +1.57 | 64.44 | +5.15 | 67.78 | +3.87 | 65.56 | +1.45 | 61.11 | -5.44 |  |
| Protein Des. (L5) | 0.010 | -0.003 | 0.008 | -0.001 | 0.000 | +0.001 | 0.000 | +0.004 | 0.000 | +0.004 | 0.000 | +0.004 |  |

### 4.3 Discussions

#### SciKnowEval exhibits Sufficient Difficulty and Challenge.

Firstly, our results indicate that in zero-shot setting, proprietary models consistently outperform other open-source models. Moreover, there is a noticeable positive correlation between model size and performance, e.g., Galactica-30B outperforms Galactica-6.7B, and Qwen1.5-14B-Chat exceeds the performance of Qwen1.5-7B.
Secondly, by examining the detailed scores of GPT-4o across various tasks, it is evident that SciKnowEval spans multiple levels of difficulty. For most tasks at the L1 and L2 levels, GPT-4o achieves accuracies above 85%.
However, GPT-4o struggles with tasks at the L3 and L5 levels, particularly those involving molecular SMILES and protein sequences. Lastly, our carefully designed L4 level, aimed at evaluating the safety of LLMs, introduces a novel challenge compared to other benchmarks such as SciEval and SciAssess. We observed that GPT-4o often failed to reject harmful questions in the Harmful QA task, presenting a potential risk of misuse.

#### Few-shot Setting enhances Model Performance across Tasks.

For the few-shot setting,
we selected two competitive models from each of the three categories of LLMs, and evaluated them on most of the tasks in our datasets. Table 5 summarizes the task performance shifts of these LLMs under a 3-shot setting. For most tasks involving multiple-choice and true/false questions, we observed significant performance gains, especially for tasks at the L3 level.
For instance, GPT-4o and Claude3-Sonnet saw accuracy increases of 27.10% and 28.58%, respectively, in the fluorescence prediction task. This indicates that few-shot settings can substantially enhance models’ scientific reasoning and computational abilities.

Incremental Pre-training or Fine-tuning on Scientific Corpus show Promise. We compared two pairs of models: 1) Llama2-13B vs. ChemDFM-13B, and 2) Mistral-7B-Inst vs. LlaSMol-Mistral-7B.
We can observe that ChemDFM-13B, built on the Llama-13B framework and further pre-trained and fine-tuned on a corpus of 34 billion tokens from scientific literature, significantly outperformed Llama2-13B-Chat. Similarly, fine-tuning with instruction data for molecule generation and description enabled LlaSMol-Mistral-7B to surpass Mistral-7B-Inst at the L1 and L5 levels in the chemical domain.

Evaluating OpenAI o1 with SciKnowEval.
Recently, OpenAI released a new LLM, o1 [], which excels at complex task reasoning, especially in the fields of science, mathematics, and programming. In a series of challenging benchmarks, o1 delivered outstanding results and even surpassed human experts in PhD-level scientific Q&A sessions. To further assess o1’s capabilities in scientific domains, we used a subset of the SciKnowEval dataset to evaluate the performance of o1-preview and o1-mini. The subset consists of 1,775 challenging questions that GPT-4o-mini fails to answer correctly.
The evaluation results are presented in Figure A1. Through analyzing the quantitative results and several cases, we have three key findings: (1) By generating hidden chain-of-thoughts (CoT) during model inference, o1 has a significant improvement in answering questions related to scientific computation and reasoning, however, it occasionally falls into the traps of reasoning problems, especially those involving complex physical principles and laws. (2) o1 has integrated safety rules into its CoT, improving the model’s ability to align with safety, but it still lacks sufficient knowledge regarding certain substances (such as uncommon toxic compounds and viruses) and may output harmful responses. (3) In addition to scientific reasoning and safety alignment, the improvement of o1 in scientific knowledge memory, knowledge understanding, and knowledge application is limited. See Appendix A11 for more evaluation results and case analyses.

## 5 Conclusion

In this paper, we introduce the SciKnowEval benchmark, a novel framework designed to comprehensively and systematically evaluate the scientific knowledge of LLMs. SciKnowEval defines five progressive levels, aimed at deeply reflecting the breadth and depth of LLMs’ scientific knowledge. It focuses on biology, chemistry, physics and materials as four representative domains, encompassing 70K multi-level problems and answers. We employed this SciKnowEval dataset to conduct extensive benchmarking and thorough analysis of 26 advanced LLMs. Our findings indicate that even the most advanced LLMs struggle to effectively address tasks related to scientific reasoning and application.

In the future, we aim to broaden the scope of SciKnowEval by encompassing additional scientific domains and incorporating more domain-specific tasks. Additionally, due to the large scale of SciKnowEval datasets and the involvement of some tasks that require scoring based on GPT-4o, there are some costs associated with the assessment. In future efforts, we aim to optimize the assessment methods, such as by substituting GPT-4o with an open-source scientific LLM evaluator.
We anticipate that SciKnowEval will become a standard for evaluating LLMs in scientific research and discovery, thereby promoting the development of scientific LLMs.

## Appendix

## Appendix A1 Additional Results of SciKnowEval

### A1.1 Zero-shot Performance in Each Domain

**Table A1: Zero-shot performance of LLMs across five levels in the biology, chemistry, materials and physics domains. A smaller value indicates a higher ranking. Bold results indicate the best results among all models, underline results indicate the second-best results, and blue results indicate the best results among the open-source models.**
| Domain | Task Name | GPT-4o | Claude3 | Llama3-8B | Qwen1.5-7B | MolInst-8B | ChemLLM-7B |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0-shot | 3-shot | 0-shot | 3-shot | 0-shot | 3-shot | 0-shot | 3-shot | 0-shot | 3-shot | 0-shot | 3-shot |  |  |
| Bio. | Prot Prop Iden. (L1) | 34.20 | +10.53 | 27.93 | +4.47 | 25.60 | +3.73 | 26.40 | -1.8 | 20.53 | +3.54 | 22.20 | +2.73 |
| Bio LiterQA (L1) | 83.71 | +0.45 | 76.44 | -0.72 | 74.82 | +2.15 | 72.06 | +0.69 | 72.82 | +2.30 | 61.65 | +8.30 |  |
| Protein Cap. (L1) | 0.122 | +0.005 | 0.118 | +0.002 | 0.112 | -0.003 | 0.100 | +0.028 | 0.009 | +0.153 | 0.065 | +0.049 |  |
| D-D RE (L2) | 17.41 | +0.22 | 16.02 | +0.86 | 16.75 | +0.26 | 16.93 | -0.53 | 14.86 | -0.69 | 6.47 | +3.27 |  |
| Bio JI (L2) | 95.46 | -0.53 | 96.20 | -0.32 | 89.12 | +5.92 | 87.33 | +2.22 | 92.71 | 5.39 | 26.08 | +39.92 |  |
| C-D RE (L2) | 37.70 | +10.59 | 35.85 | +8.5 | 28.90 | +24.09 | 21.27 | +6.42 | 50.26 | -5.42 | 13.34 | +11.08 |  |
| G-D RE (L2) | 36.20 | -1.15 | 29.18 | -5.75 | 39.16 | -0.52 | 8.74 | +24.72 | 10.38 | +0.75 | 12.65 | +6.34 |  |
| Bio DU (L2) | 99.40 | +0.00 | 98.67 | -0.72 | 98.79 | -0.23 | 97.58 | -0.48 | 96.50 | +0.84 | 96.01 | -0.84 |  |
| Bio HV (L2) | 94.82 | -0.16 | 94.98 | +0.00 | 90.45 | +0.16 | 85.60 | +0.00 | 89.48 | -5.18 | 61.97 | +16.99 |  |
| Bio RI (L2) | 97.69 | +0.00 | 96.60 | +0.47 | 96.76 | -0.43 | 95.22 | -0.16 | 93.21 | -0.46 | 91.67 | -0.47 |  |
| Solu. Pred (L3) | 48.31 | +5.31 | 46.86 | +1.45 | 49.28 | -0.49 | 49.28 | +4.34 | 53.62 | +0.00 | 46.86 | +0.48 |  |
| $\beta$-LA Pred (L3) | 50.25 | +13.30 | 43.84 | +24.63 | 50.25 | +5.42 | 27.59 | +24.63 | 47.78 | +20.20 | 61.58 | -13.80 |  |
| Fluo. Pred (L3) | 51.72 | +27.10 | 51.72 | +28.58 | 49.75 | +9.36 | 46.31 | +14.28 | 44.83 | +22.17 | 20.69 | +33.00 |  |
| GB1 Pred (L3) | 32.54 | -0.40 | 24.40 | +4.31 | 19.14 | +4.78 | 13.40 | +15.79 | 28.23 | -0.24 | 16.27 | +15.79 |  |
| Stab. Pred (L3) | 24.02 | +1.96 | 30.39 | -3.43 | 24.51 | +2.45 | 22.55 | 15.69 | 28.92 | -0.59 | 26.47 | +4.41 |  |
| Prot-Prot Inter. (L3) | 35.27 | -6.77 | 27.54 | -1.94 | 21.26 | +11.59 | 22.22 | +3.87 | 25.60 | +6.28 | 20.77 | 10.15 |  |
| Bio Cal. (L3) | 58.33 | -8.33 | 38.33 | -3.33 | 33.33 | +5.00 | 36.67 | +10.00 | 41.67 | -3.34 | 21.67 | +15.00 |  |
| Proteotox. (L4) | 85.88 | -0.59 | 40.88 | +20.3 | 38.24 | +3.13 | 39.71 | +10.49 | 37.06 | +0.98 | 22.06 | +9.90 |  |
| Bio Safe Test (L4) | 86.67 | +0.44 | 71.11 | +1.57 | 64.44 | +5.15 | 67.78 | +3.87 | 65.56 | +1.45 | 61.11 | -5.44 |  |
| Protein Des. (L5) | 0.010 | -0.003 | 0.008 | -0.001 | 0.000 | +0.001 | 0.000 | +0.004 | 0.000 | +0.004 | 0.000 | +0.004 |  |
| Chem. | Mol Name Conv. (L1) | 86.65 | -0.36 | 68.46 | -1.06 | 55.38 | +2.15 | 40.05 | -0.98 | 56.90 | -8.96 | 44.71 | -4.66 |
| Mol Prop. Pred (L1) | 44.03 | +6.19 | 34.50 | +3.90 | 36.36 | -2.02 | 34.95 | +4.80 | 34.95 | -1.97 | 20.38 | +12.54 |  |
| Chem LiterQA (L1) | 85.75 | +1.14 | 80.33 | -5.21 | 78.73 | +1.50 | 75.98 | +0.65 | 74.57 | +4.19 | 66.57 | +9.15 |  |
| Mol Cap. (L1) | 0.139 | +0.027 | 0.125 | +0.023 | 0.045 | +0.129 | 0.075 | +0.049 | 0.024 | +0.117 | 0.123 | -0.021 |  |
| React Mech Infer. (L2) | 100.00 | -0.37 | 99.26 | -1.12 | 98.51 | -2.70 | 96.28 | -0.37 | 98.14 | -2.60 | 92.57 | -1.86 |  |
| Comp Iden. and Prop. (L2) | 98.59 | +0.00 | 96.38 | -0.81 | 95.77 | -0.2414 | 93.16 | -1.21 | 96.18 | -0.81 | 89.74 | -1.01 |  |
| Doping Extraction (L2) | 49.94 | +16.85 | 58.47 | -0.63 | 52.65 | +3.71 | 47.99 | -1.16 | 49.76 | -1.07 | 27.74 | +16.05 |  |
| Chem DU (L2) | 99.52 | -0.32 | 98.56 | +0.16 | 97.44 | -1.71 | 94.73 | +0.96 | 96.01 | +0.32 | 92.49 | -0.32 |  |
| Chem HV (L2) | 94.86 | -0.55 | 89.91 | -0.37 | 87.71 | +0.00 | 84.22 | -1.28 | 86.24 | -3.30 | 59.08 | +23.12 |  |
| Chem RI (L2) | 96.89 | +1.56 | 95.92 | +1.17 | 96.50 | -1.42 | 94.95 | -1.16 | 94.95 | +0.00 | 89.13 | +0.77 |  |
| Mol Weight Cal. (L3) | 25.91 | -1.92 | 24.66 | -1.92 | 19.96 | +4.61 | 21.31 | +0.76 | 20.06 | +4.22 | 20.63 | +5.28 |  |
| Mol Prop. Cal. (L3) | 34.73 | +9.05 | 27.30 | -1.49 | 33.51 | -0.62 | 33.38 | -2.03 | 31.89 | -3.51 | 30.00 | +0.68 |  |
| Mol Stru. Pred (L3) | 46.08 | +0.49 | 24.39 | -0.13 | 27.45 | +4.41 | 32.72 | -0.82 | 34.93 | -3.19 | 23.65 | +2.82 |  |
| Reaction Pred (L3) | 48.04 | +2.05 | 43.49 | -0.78 | 39.57 | -1.22 | 27.99 | -1.11 | 27.09 | -0.56 | 1.87 | +16.22 |  |
| Retrosynthesis (L3) | 41.18 | +0.35 | 32.17 | -0.01 | 35.56 | -0.27 | 38.24 | -1.45 | 40.55 | -0.54 | 0.62 | 18.72 |  |
| Balancing Eq. (L3) | 10.47 | +28.97 | 28.04 | +3.74 | 17.20 | -1.87 | 10.47 | +4.11 | 18.13 | -1.68 | 11.21 | -2.61 |  |
| Chem Cal. (L3) | 52.42 | +4.46 | 36.06 | -2.60 | 34.57 | +0.75 | 33.83 | +4.83 | 40.15 | -0.74 | 27.88 | +7.06 |  |
| Mol Tox. Pred (L4) | 37.40 | +21.45 | 25.20 | +26.29 | 26.83 | +29.03 | 25.20 | +32.85 | 33.33 | +24.14 | 24.39 | 12.28 |  |
| Chem Safe Test (L4) | 85.23 | +0.65 | 65.82 | +9.13 | 67.51 | +4.24 | 62.45 | +12.69 | 67.93 | +1.00 | 50.21 | +4.03 |  |
| Mol Gen. (L5) | 0.609 | -0.021 | 0.532 | -0.027 | 0.378 | -0.035 | 0.209 | -0.015 | 0.279 | +0.010 | 0.298 | -0.088 |  |
| Mat. | Mat. LiterQA (L1) | 76.47 | +0.20 | 70.85 | -12.54 | 66.90 | +4.60 | 65.31 | +1.95 | 54.92 | +16.04 | 31.50 | +28.89 |
| Mat. Comp Extr (L2) | 56.16 | +28.69 | 62.07 | +24.26 | 0.00 | +79.31 | 39.53 | +24.88 | 0.12 | +73.65 | 2.54 | +49.31 |  |
| Mat. Data Extr (L2) | 88.82 | +0.59 | 88.24 | +1.76 | 90.59 | -5.30 | 82.35 | +0.00 | 61.76 | +22.36 | 70.59 | -1.18 |  |
| Mat. DU (L2) | 90.25 | +0.25 | 89.25 | -0.25 | 88.75 | -0.25 | 85.25 | +0.75 | 83.75 | +4.75 | 67.75 | +12.50 |  |
| Mat. Text Sum (L2) | 4.82 | -0.04 | 4.60 | +0.14 | 1.18 | +3.24 | 4.55 | +0.07 | 3.30 | -1.44 | 2.66 | +0.68 |  |
| Mat. HV (L2) | 90.00 | -5.67 | 68.67 | +4.00 | 65.00 | -1.33 | 64.67 | -5.67 | 66.33 | -25.33 | 19.33 | +43.34 |  |
| Mat. RI (L2) | 98.33 | -0.28 | 95.54 | +1.40 | 98.33 | -1.12 | 96.94 | +0.00 | 90.53 | +6.96 | 69.64 | +15.32 |  |
| Val Elec Diff Calc (L3) | 46.58 | +8.21 | 46.58 | +2.05 | 28.77 | +10.96 | 19.86 | +10.28 | 23.29 | +14.38 | 23.29 | +6.85 |  |
| Mat. Calc (L3) | 37.93 | +0.86 | 35.06 | -6.90 | 27.59 | +1.43 | 19.54 | +7.18 | 20.69 | +6.03 | 38.51 | -11.50 |  |
| Latt Vol Calc (L3) | 53.12 | -4.37 | 39.37 | -8.12 | 50.00 | +0.62 | 41.25 | +3.75 | 5.63 | +49.37 | 5.63 | +29.37 |  |
| Perov. Stab Pred (L3) | 61.88 | -0.63 | 35.63 | +0.20 | 35.42 | +8.12 | 34.17 | +7.29 | 25.62 | +17.71 | 23.54 | +3.33 |  |
| Diff Rate Analys (L3) | 69.13 | +26.84 | 69.80 | -1.34 | 38.93 | +15.43 | 24.83 | -0.67 | 16.11 | +28.86 | 21.48 | +4.69 |  |
| Mat. SafetyQA (L4) | 86.21 | +1.07 | 83.35 | -12.96 | 80.02 | +2.26 | 78.72 | +2.61 | 71.70 | +6.90 | 33.41 | +37.34 |  |
| Mat. Tox Pred (L4) | 67.15 | -1.95 | 66.50 | -3.09 | 63.90 | +3.90 | 50.08 | +7.48 | 30.41 | +37.23 | 20.65 | +37.56 |  |
| Prop Usage Analysis (L5) | 2.68 | -0.53 | 2.75 | -0.49 | 1.08 | +0.31 | 1.15 | +0.66 | 1.97 | -0.53 | 1.46 | -0.36 |  |
| Cry Struct Comp Analys (L5) | 1.29 | +0.05 | 1.13 | +0.13 | 1.12 | +0.11 | 1.03 | -0.01 | 1.02 | +0.19 | 1.01 | +0.10 |  |
|  | Spec Band Gap Gen (L5) | 2.51 | +0.32 | 2.70 | +0.14 | 1.00 | +1.04 | 1.28 | -0.03 | 1.08 | +0.29 | 1.01 | +0.61 |
| Phys. | Phys. LiterQA (L1) | 87.56 | -0.71 | 78.91 | -8.39 | 77.61 | +2.47 | 72.96 | +1.72 | 63.58 | +17.25 | 35.97 | +31.28 |
| Fund Phys. Exam (L1) | 96.59 | +0.13 | 90.48 | -6.69 | 89.26 | +3.46 | 86.44 | +1.31 | 81.01 | +10.15 | 29.43 | +56.30 |  |
| Phys. DU (L2) | 99.50 | +0.25 | 99.25 | -1.00 | 99.25 | +0.00 | 98.25 | -0.75 | 97.00 | +2.00 | 80.00 | +15.75 |  |
| Phys. Text Sum (L2) | 4.88 | -0.08 | 4.79 | -0.01 | 1.02 | +3.51 | 4.69 | +0.00 | 2.99 | -0.80 | 2.66 | +0.85 |  |
| Phys. HV (L2) | 98.75 | +0.50 | 96.00 | +1.25 | 95.25 | +0.75 | 91.75 | +0.75 | 95.00 | -2.75 | 16.00 | +70.50 |  |
| Phys. RI (L2) | 99.75 | +0.25 | 98.75 | -0.75 | 99.50 | +0.00 | 99.00 | -0.75 | 97.00 | +2.75 | 90.25 | +7.25 |  |
| HS Phys. Calc (L3) | 57.45 | -3.44 | 37.11 | -1.58 | 30.09 | +1.86 | 34.96 | +8.02 | 33.52 | +1.87 | 12.32 | +19.06 |  |
| Gen Phys. Calc (L3) | 39.62 | -1.00 | 33.37 | -3.87 | 34.75 | +5.50 | 23.87 | +7.26 | 23.25 | +11.75 | 25.87 | -3.24 |  |
| Phys. Formula Deriv (L3) | 4.86 | -0.35 | 4.57 | -0.03 | 1.22 | +2.18 | 1.00 | +2.20 | 1.00 | +2.22 | 1.38 | +0.68 |  |
| Phys. SafetyQA (L4) | 88.01 | -1.17 | 81.29 | -11.41 | 81.29 | 0.87 | 76.32 | 3.50 | 69.59 | +10.53 | 31.58 | +38.89 |  |
| Phys. Lab Safety Test (L4) | 78.71 | +2.45 | 75.41 | -7.15 | 70.30 | +0.11 | 67.49 | +4.08 | 62.71 | +8.03 | 20.96 | +38.21 |  |
| Phys. Prob Solving (L5) | 3.86 | -0.16 | 2.64 | +0.23 | 1.01 | +0.54 | 1.02 | +0.36 | 1.08 | +0.27 | 1.05 | +0.05 |  |

### A1.2 Few-shot Performance

**Table A2: 3-shot performance of LLMs on each task in four scientific domains. Bold indicates performance improvement compared to the zero-shot setting, while gray signifies no gain. Blue results indicate a large improvement (>10%).**
| Domain | Task Name | GPT-4o | Claude3 | Llama3-8B | Qwen1.5-7B | MolInst-8B | ChemLLM-7B |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0-shot | 3-shot | 0-shot | 3-shot | 0-shot | 3-shot | 0-shot | 3-shot | 0-shot | 3-shot | 0-shot | 3-shot |  |  |
| Bio. | Prot Prop Iden. (L1) | 34.20 | +10.53 | 27.93 | +4.47 | 25.60 | +3.73 | 26.40 | -1.8 | 20.53 | +3.54 | 22.20 | +2.73 |
| Bio LiterQA (L1) | 83.71 | +0.45 | 76.44 | -0.72 | 74.82 | +2.15 | 72.06 | +0.69 | 72.82 | +2.30 | 61.65 | +8.30 |  |
| Protein Cap. (L1) | 0.122 | +0.005 | 0.118 | +0.002 | 0.112 | -0.003 | 0.100 | +0.028 | 0.009 | +0.153 | 0.065 | +0.049 |  |
| D-D RE (L2) | 17.41 | +0.22 | 16.02 | +0.86 | 16.75 | +0.26 | 16.93 | -0.53 | 14.86 | -0.69 | 6.47 | +3.27 |  |
| Bio JI (L2) | 95.46 | -0.53 | 96.20 | -0.32 | 89.12 | +5.92 | 87.33 | +2.22 | 92.71 | 5.39 | 26.08 | +39.92 |  |
| C-D RE (L2) | 37.70 | +10.59 | 35.85 | +8.5 | 28.90 | +24.09 | 21.27 | +6.42 | 50.26 | -5.42 | 13.34 | +11.08 |  |
| G-D RE (L2) | 36.20 | -1.15 | 29.18 | -5.75 | 39.16 | -0.52 | 8.74 | +24.72 | 10.38 | +0.75 | 12.65 | +6.34 |  |
| Bio DU (L2) | 99.40 | +0.00 | 98.67 | -0.72 | 98.79 | -0.23 | 97.58 | -0.48 | 96.50 | +0.84 | 96.01 | -0.84 |  |
| Bio HV (L2) | 94.82 | -0.16 | 94.98 | +0.00 | 90.45 | +0.16 | 85.60 | +0.00 | 89.48 | -5.18 | 61.97 | +16.99 |  |
| Bio RI (L2) | 97.69 | +0.00 | 96.60 | +0.47 | 96.76 | -0.43 | 95.22 | -0.16 | 93.21 | -0.46 | 91.67 | -0.47 |  |
| Solu. Pred (L3) | 48.31 | +5.31 | 46.86 | +1.45 | 49.28 | -0.49 | 49.28 | +4.34 | 53.62 | +0.00 | 46.86 | +0.48 |  |
| $\beta$-LA Pred (L3) | 50.25 | +13.30 | 43.84 | +24.63 | 50.25 | +5.42 | 27.59 | +24.63 | 47.78 | +20.20 | 61.58 | -13.80 |  |
| Fluo. Pred (L3) | 51.72 | +27.10 | 51.72 | +28.58 | 49.75 | +9.36 | 46.31 | +14.28 | 44.83 | +22.17 | 20.69 | +33.00 |  |
| GB1 Pred (L3) | 32.54 | -0.40 | 24.40 | +4.31 | 19.14 | +4.78 | 13.40 | +15.79 | 28.23 | -0.24 | 16.27 | +15.79 |  |
| Stab. Pred (L3) | 24.02 | +1.96 | 30.39 | -3.43 | 24.51 | +2.45 | 22.55 | 15.69 | 28.92 | -0.59 | 26.47 | +4.41 |  |
| Prot-Prot Inter. (L3) | 35.27 | -6.77 | 27.54 | -1.94 | 21.26 | +11.59 | 22.22 | +3.87 | 25.60 | +6.28 | 20.77 | 10.15 |  |
| Bio Cal. (L3) | 58.33 | -8.33 | 38.33 | -3.33 | 33.33 | +5.00 | 36.67 | +10.00 | 41.67 | -3.34 | 21.67 | +15.00 |  |
| Proteotox. (L4) | 85.88 | -0.59 | 40.88 | +20.3 | 38.24 | +3.13 | 39.71 | +10.49 | 37.06 | +0.98 | 22.06 | +9.90 |  |
| Bio Safe Test (L4) | 86.67 | +0.44 | 71.11 | +1.57 | 64.44 | +5.15 | 67.78 | +3.87 | 65.56 | +1.45 | 61.11 | -5.44 |  |
| Protein Des. (L5) | 0.010 | -0.003 | 0.008 | -0.001 | 0.000 | +0.001 | 0.000 | +0.004 | 0.000 | +0.004 | 0.000 | +0.004 |  |
| Chem. | Mol Name Conv. (L1) | 86.65 | -0.36 | 68.46 | -1.06 | 55.38 | +2.15 | 40.05 | -0.98 | 56.90 | -8.96 | 44.71 | -4.66 |
| Mol Prop. Pred (L1) | 44.03 | +6.19 | 34.50 | +3.90 | 36.36 | -2.02 | 34.95 | +4.80 | 34.95 | -1.97 | 20.38 | +12.54 |  |
| Chem LiterQA (L1) | 85.75 | +1.14 | 80.33 | -5.21 | 78.73 | +1.50 | 75.98 | +0.65 | 74.57 | +4.19 | 66.57 | +9.15 |  |
| Mol Cap. (L1) | 0.139 | +0.027 | 0.125 | +0.023 | 0.045 | +0.129 | 0.075 | +0.049 | 0.024 | +0.117 | 0.123 | -0.021 |  |
| React Mech Infer. (L2) | 100.00 | -0.37 | 99.26 | -1.12 | 98.51 | -2.70 | 96.28 | -0.37 | 98.14 | -2.60 | 92.57 | -1.86 |  |
| Comp Iden. and Prop. (L2) | 98.59 | +0.00 | 96.38 | -0.81 | 95.77 | -0.2414 | 93.16 | -1.21 | 96.18 | -0.81 | 89.74 | -1.01 |  |
| Doping Extraction (L2) | 49.94 | +16.85 | 58.47 | -0.63 | 52.65 | +3.71 | 47.99 | -1.16 | 49.76 | -1.07 | 27.74 | +16.05 |  |
| Chem DU (L2) | 99.52 | -0.32 | 98.56 | +0.16 | 97.44 | -1.71 | 94.73 | +0.96 | 96.01 | +0.32 | 92.49 | -0.32 |  |
| Chem HV (L2) | 94.86 | -0.55 | 89.91 | -0.37 | 87.71 | +0.00 | 84.22 | -1.28 | 86.24 | -3.30 | 59.08 | +23.12 |  |
| Chem RI (L2) | 96.89 | +1.56 | 95.92 | +1.17 | 96.50 | -1.42 | 94.95 | -1.16 | 94.95 | +0.00 | 89.13 | +0.77 |  |
| Mol Weight Cal. (L3) | 25.91 | -1.92 | 24.66 | -1.92 | 19.96 | +4.61 | 21.31 | +0.76 | 20.06 | +4.22 | 20.63 | +5.28 |  |
| Mol Prop. Cal. (L3) | 34.73 | +9.05 | 27.30 | -1.49 | 33.51 | -0.62 | 33.38 | -2.03 | 31.89 | -3.51 | 30.00 | +0.68 |  |
| Mol Stru. Pred (L3) | 46.08 | +0.49 | 24.39 | -0.13 | 27.45 | +4.41 | 32.72 | -0.82 | 34.93 | -3.19 | 23.65 | +2.82 |  |
| Reaction Pred (L3) | 48.04 | +2.05 | 43.49 | -0.78 | 39.57 | -1.22 | 27.99 | -1.11 | 27.09 | -0.56 | 1.87 | +16.22 |  |
| Retrosynthesis (L3) | 41.18 | +0.35 | 32.17 | -0.01 | 35.56 | -0.27 | 38.24 | -1.45 | 40.55 | -0.54 | 0.62 | 18.72 |  |
| Balancing Eq. (L3) | 10.47 | +28.97 | 28.04 | +3.74 | 17.20 | -1.87 | 10.47 | +4.11 | 18.13 | -1.68 | 11.21 | -2.61 |  |
| Chem Cal. (L3) | 52.42 | +4.46 | 36.06 | -2.60 | 34.57 | +0.75 | 33.83 | +4.83 | 40.15 | -0.74 | 27.88 | +7.06 |  |
| Mol Tox. Pred (L4) | 37.40 | +21.45 | 25.20 | +26.29 | 26.83 | +29.03 | 25.20 | +32.85 | 33.33 | +24.14 | 24.39 | 12.28 |  |
| Chem Safe Test (L4) | 85.23 | +0.65 | 65.82 | +9.13 | 67.51 | +4.24 | 62.45 | +12.69 | 67.93 | +1.00 | 50.21 | +4.03 |  |
| Mol Gen. (L5) | 0.609 | -0.021 | 0.532 | -0.027 | 0.378 | -0.035 | 0.209 | -0.015 | 0.279 | +0.010 | 0.298 | -0.088 |  |
| Mat. | Mat. LiterQA (L1) | 76.47 | +0.20 | 70.85 | -12.54 | 66.90 | +4.60 | 65.31 | +1.95 | 54.92 | +16.04 | 31.50 | +28.89 |
| Mat. Comp Extr (L2) | 56.16 | +28.69 | 62.07 | +24.26 | 0.00 | +79.31 | 39.53 | +24.88 | 0.12 | +73.65 | 2.54 | +49.31 |  |
| Mat. Data Extr (L2) | 88.82 | +0.59 | 88.24 | +1.76 | 90.59 | -5.30 | 82.35 | +0.00 | 61.76 | +22.36 | 70.59 | -1.18 |  |
| Mat. DU (L2) | 90.25 | +0.25 | 89.25 | -0.25 | 88.75 | -0.25 | 85.25 | +0.75 | 83.75 | +4.75 | 67.75 | +12.50 |  |
| Mat. Text Sum (L2) | 4.82 | -0.04 | 4.60 | +0.14 | 1.18 | +3.24 | 4.55 | +0.07 | 3.30 | -1.44 | 2.66 | +0.68 |  |
| Mat. HV (L2) | 90.00 | -5.67 | 68.67 | +4.00 | 65.00 | -1.33 | 64.67 | -5.67 | 66.33 | -25.33 | 19.33 | +43.34 |  |
| Mat. RI (L2) | 98.33 | -0.28 | 95.54 | +1.40 | 98.33 | -1.12 | 96.94 | +0.00 | 90.53 | +6.96 | 69.64 | +15.32 |  |
| Val Elec Diff Calc (L3) | 46.58 | +8.21 | 46.58 | +2.05 | 28.77 | +10.96 | 19.86 | +10.28 | 23.29 | +14.38 | 23.29 | +6.85 |  |
| Mat. Calc (L3) | 37.93 | +0.86 | 35.06 | -6.90 | 27.59 | +1.43 | 19.54 | +7.18 | 20.69 | +6.03 | 38.51 | -11.50 |  |
| Latt Vol Calc (L3) | 53.12 | -4.37 | 39.37 | -8.12 | 50.00 | +0.62 | 41.25 | +3.75 | 5.63 | +49.37 | 5.63 | +29.37 |  |
| Perov. Stab Pred (L3) | 61.88 | -0.63 | 35.63 | +0.20 | 35.42 | +8.12 | 34.17 | +7.29 | 25.62 | +17.71 | 23.54 | +3.33 |  |
| Diff Rate Analys (L3) | 69.13 | +26.84 | 69.80 | -1.34 | 38.93 | +15.43 | 24.83 | -0.67 | 16.11 | +28.86 | 21.48 | +4.69 |  |
| Mat. SafetyQA (L4) | 86.21 | +1.07 | 83.35 | -12.96 | 80.02 | +2.26 | 78.72 | +2.61 | 71.70 | +6.90 | 33.41 | +37.34 |  |
| Mat. Tox Pred (L4) | 67.15 | -1.95 | 66.50 | -3.09 | 63.90 | +3.90 | 50.08 | +7.48 | 30.41 | +37.23 | 20.65 | +37.56 |  |
| Prop Usage Analysis (L5) | 2.68 | -0.53 | 2.75 | -0.49 | 1.08 | +0.31 | 1.15 | +0.66 | 1.97 | -0.53 | 1.46 | -0.36 |  |
| Cry Struct Comp Analys (L5) | 1.29 | +0.05 | 1.13 | +0.13 | 1.12 | +0.11 | 1.03 | -0.01 | 1.02 | +0.19 | 1.01 | +0.10 |  |
|  | Spec Band Gap Gen (L5) | 2.51 | +0.32 | 2.70 | +0.14 | 1.00 | +1.04 | 1.28 | -0.03 | 1.08 | +0.29 | 1.01 | +0.61 |
| Phys. | Phys. LiterQA (L1) | 87.56 | -0.71 | 78.91 | -8.39 | 77.61 | +2.47 | 72.96 | +1.72 | 63.58 | +17.25 | 35.97 | +31.28 |
| Fund Phys. Exam (L1) | 96.59 | +0.13 | 90.48 | -6.69 | 89.26 | +3.46 | 86.44 | +1.31 | 81.01 | +10.15 | 29.43 | +56.30 |  |
| Phys. DU (L2) | 99.50 | +0.25 | 99.25 | -1.00 | 99.25 | +0.00 | 98.25 | -0.75 | 97.00 | +2.00 | 80.00 | +15.75 |  |
| Phys. Text Sum (L2) | 4.88 | -0.08 | 4.79 | -0.01 | 1.02 | +3.51 | 4.69 | +0.00 | 2.99 | -0.80 | 2.66 | +0.85 |  |
| Phys. HV (L2) | 98.75 | +0.50 | 96.00 | +1.25 | 95.25 | +0.75 | 91.75 | +0.75 | 95.00 | -2.75 | 16.00 | +70.50 |  |
| Phys. RI (L2) | 99.75 | +0.25 | 98.75 | -0.75 | 99.50 | +0.00 | 99.00 | -0.75 | 97.00 | +2.75 | 90.25 | +7.25 |  |
| HS Phys. Calc (L3) | 57.45 | -3.44 | 37.11 | -1.58 | 30.09 | +1.86 | 34.96 | +8.02 | 33.52 | +1.87 | 12.32 | +19.06 |  |
| Gen Phys. Calc (L3) | 39.62 | -1.00 | 33.37 | -3.87 | 34.75 | +5.50 | 23.87 | +7.26 | 23.25 | +11.75 | 25.87 | -3.24 |  |
| Phys. Formula Deriv (L3) | 4.86 | -0.35 | 4.57 | -0.03 | 1.22 | +2.18 | 1.00 | +2.20 | 1.00 | +2.22 | 1.38 | +0.68 |  |
| Phys. SafetyQA (L4) | 88.01 | -1.17 | 81.29 | -11.41 | 81.29 | 0.87 | 76.32 | 3.50 | 69.59 | +10.53 | 31.58 | +38.89 |  |
| Phys. Lab Safety Test (L4) | 78.71 | +2.45 | 75.41 | -7.15 | 70.30 | +0.11 | 67.49 | +4.08 | 62.71 | +8.03 | 20.96 | +38.21 |  |
| Phys. Prob Solving (L5) | 3.86 | -0.16 | 2.64 | +0.23 | 1.01 | +0.54 | 1.02 | +0.36 | 1.08 | +0.27 | 1.05 | +0.05 |  |

### A1.3 Detailed Performance of LLMs on Each Task

Figure: Table A3: Zero-shot performance of LLMs on each task in the biology domain.

Figure: Table A5: Zero-shot performance of LLMs on each task in the materials domain.

## Appendix A2 Detailed Model Descriptions

In this paper, we select 26 high-performing LLMs with varying scales. Table A7 summarizes the details of these models.
During model inference, for proprietary models (M1-M8), we called the official API with inference hyper-parameters set to temperature = 0.0, top-$p$ = 1.0, and max-length = 4096, while leaving other hyper-parameters at default values. For the remaining fifteen open-source models, we deployed them locally on 2 NVIDIA A100 GPUs, utilizing the vLLM [] framework for acceleration. Similarly, inference hyper-parameters were set to temperature = 0.0, top-$p$ = 1.0, and max-length = $\max(\text{context_length},4096)$.

**Table A7: Detailed information of LLMs evaluated in our experiments.**
| ID | Model | Creator | #Parameters | Access | URL |
| --- | --- | --- | --- | --- | --- |
| M1 | Claude3.5-Sonnet-20240620 | Anthropic | undisclosed | API | [https://claude.ai](https://claude.ai) |
| M2 | GPT-4o-2024-05-13 | OpenAI | undisclosed | API | [https://chat.openai.com](https://chat.openai.com) |
| M3 | GPT-4-Turbo-2024-04-09 | OpenAI | undisclosed | API | [https://chat.openai.com](https://chat.openai.com) |
| M4 | Gemini1.5-Pro-latest | Google | undisclosed | API | [https://gemini.google.com](https://gemini.google.com) |
| M5 | Qwen-Max | Alibaba Cloud | undisclosed | API | [https://dashscope.aliyun.com/](https://dashscope.aliyun.com/) |
| M6 | GPT-4o-mini-2024-07-18 | OpenAI | undisclosed | API | [https://chat.openai.com](https://chat.openai.com) |
| M7 | Claude3-Sonnet-20240229 | Anthropic | undisclosed | API | [https://claude.ai](https://claude.ai) |
| M8 | GPT-3.5-Turbo-0125 | OpenAI | undisclosed | API | [https://chat.openai.com](https://chat.openai.com) |
| M9 | Qwen2-72B-Instruct | Alibaba | 72B | Weights | [https://qwenlm.github.io/](https://qwenlm.github.io/) |
| M10 | Llama3-70B-Instruct | Meta | 8B | Weights | [https://llama.meta.com/llama3](https://llama.meta.com/llama3) |
| M11 | Qwen2-7B-Instruct | Alibaba | 7B | Weights | [https://qwenlm.github.io/](https://qwenlm.github.io/) |
| M12 | Qwen1.5-14B-Chat | Alibaba | 14B | Weights | [https://qwenlm.github.io/](https://qwenlm.github.io/) |
| M13 | Llama3-8B-Instruct | Meta | 8B | Weights | [https://llama.meta.com/llama3](https://llama.meta.com/llama3) |
| M14 | Qwen1.5-7B-Chat | Alibaba | 7B | Weights | [https://qwenlm.github.io/](https://qwenlm.github.io/) |
| M15 | Gemma1.1-7B-Inst | Google | 7B | Weights | [https://ai.google.dev/gemma](https://ai.google.dev/gemma) |
| M16 | Mistral-7B-Inst-v0.2 | Mistral | 7B | Weights | [https://mistral.ai](https://mistral.ai) |
| M17 | ChatGLM3-6B | Tsinghua | 6B | Weights | [https://github.com/THUDM/ChatGLM3](https://github.com/THUDM/ChatGLM3) |
| M18 | Llama2-13B-Chat | Meta | 13B | Weights | [https://llama.meta.com/llama2](https://llama.meta.com/llama2) |
| M19 | ChemDFM-13B | SJTU | 13B | Weights | [https://github.com/OpenDFM/ChemDFM](https://github.com/OpenDFM/ChemDFM) |
| M20 | ChemLLM-20B-Chat | ShanghaiAILab | 20B | Weights | [https://huggingface.co/AI4Chem/ChemLLM-20B-Chat-DPO](https://huggingface.co/AI4Chem/ChemLLM-20B-Chat-DPO) |
| M21 | MolInst-Llama3-8B | ZJUNLP | 8B | Weights | [https://huggingface.co/zjunlp/llama3-instruct-molinst-molecule-8b](https://huggingface.co/zjunlp/llama3-instruct-molinst-molecule-8b) |
| M22 | Galactica-30B | Meta | 30B | Weights | [https://huggingface.co/facebook/galactica-30b](https://huggingface.co/facebook/galactica-30b) |
| M23 | ChemLLM-7B-Chat | ShanghaiAILab | 7B | Weights | [https://huggingface.co/AI4Chem/ChemLLM-7B-Chat](https://huggingface.co/AI4Chem/ChemLLM-7B-Chat) |
| M24 | SciGLM-6B | Tsinghua | 6B | Weights | [https://github.com/THUDM/SciGLM](https://github.com/THUDM/SciGLM) |
| M25 | Galactica-6.7B | Meta | 6.7B | Weights | [https://huggingface.co/facebook/galactica-6.7b](https://huggingface.co/facebook/galactica-6.7b) |
| M26 | LlaSMol-Mistral-7B | OSU | 7B | Weights | [https://huggingface.co/osunlp/LlaSMol-Mistral-7B](https://huggingface.co/osunlp/LlaSMol-Mistral-7B) |

## Appendix A3 Dataset Overview

**Table A8: Overview of the proposed dataset for Chemistry, Physics and Materials. Abbr., MCQ: multiple choice questions; T/F: true/false; CLS: classification; RE: relation extraction; GEN: generative task. Data collection methods I, II and III are in Fig. 2.**
| Domain | Ability | Task Name | Task Type | Data Source | Method | #Questions |
| --- | --- | --- | --- | --- | --- | --- |
| Chemistry | L1 | Molecular Name Conversion | MCQ | PubChem | III | 828 |
| Molecular Property Identification | MCQ, T/F | MoleculeNet | III | 1,625 |  |  |
| Chemical Literature QA | MCQ | Literature Corpus | I | 6,323 |  |  |
| Molecular Captioning | GEN | ChEBI-20 | II | 884 |  |  |
| L2 | Reaction Mechanism Inference | MCQ | LibreTexts | I | 269 |  |
| Compound Identification and Properties | MCQ | LibreTexts | I | 497 |  |  |
| Doping Extraction | RE | NERRE | II | 821 |  |  |
| Detailed Understanding | MCQ | LibreTexts | I | 626 |  |  |
| Text Summary | GEN | LibreTexts | I | 691 |  |  |
| Hypothesis Verification | T/F | LibreTexts | I | 545 |  |  |
| Explanation | MCQ | LibreTexts | I | 515 |  |  |
| L3 | Molar Weight Calculation | MCQ | PubChem | III | 996 |  |
| Molecular Property Calculation | MCQ | MoleculeNet | II | 740 |  |  |
| Molecular Structure Prediction | MCQ | PubChem | III | 495 |  |  |
| Reaction Prediction | MCQ | USPTO-Mixed | II | 559 |  |  |
| Retrosynthesis | MCQ | USPTO-50k | II | 483 |  |  |
| Balancing Chemical Equation | GEN | WebQC | III | 535 |  |  |
| Chemical Calculation | MCQ | XieZhi, SciEval, MMLU | II | 269 |  |  |
| L4 | Chemical Harmful QA | GEN | Proposition-65, ILO | III | 454 |  |
| Molecular Toxicity Prediction | MCQ, T/F | Toxric | III | 870 |  |  |
| Chemical Laboratory Safety Test | MCQ, T/F | LabExam (ZJU) | II | 531 |  |  |
| L5 | Molecular Generation | GEN | ChEBI-20 | II | 885 |  |
| Chemical Protocol Procedure Design | GEN | Protocol Journal | I | 74 |  |  |
| Chemical Protocol Reagent Design | GEN | Protocol Journal | I | 129 |  |  |
|  | L1 | Material Literature QA | MCQ | Literature Corpus | I | 5534 |
| Materials | L2 | Chemical Composition Extraction | GEN | Literature Corpus | I | 203 |
| Digital Data Extraction | MCQ | Literature Corpus | I | 170 |  |  |
| Detailed Understanding | MCQ | Literature Corpus | I | 400 |  |  |
| Text Summary | GEN | Literature Corpus | I | 400 |  |  |
| Hypothesis Verification | T/F | Literature Corpus | I | 300 |  |  |
| Explanation | MCQ | Literature Corpus | I | 359 |  |  |
| L3 | Valence Electron Difference Calculation | MCQ | Metallic Glass Forming Database | III | 146 |  |
| Material Calculation | MCQ | MaScQA | II | 348 |  |  |
| Lattice Volume Calculation | MCQ | Materials Project | III | 160 |  |  |
| Perovskite Stability Prediction | MCQ | MAST-ML | III | 480 |  |  |
| Diffusion Rate Analysis | MCQ | Dilute Solute Diffusion Database | III | 149 |  |  |
| L4 | Material Safety QA | GEN | Nature Portfolio | III | 841 |  |
|  | Material toxicity prediction | MCQ | Toxric | III | 615 |  |
| L5 | Properties Utilization Analysis | GEN | Material handbooks | I | 118 |  |
| Crystal Structure and Composition Analysis | GEN | Crystal-LLM | III | 196 |  |  |
| Specified Band Gap Material Generation | GEN | Material Project | III | 300 |  |  |
|  | L1 | Physics Literature QA | MCQ | Literature Corpus | I | 4403 |
|  | Fundamental Physics Exam | MCQ | SciQ | II | 2375 |  |
| Physics | L2 | Detailed Understanding | MCQ | Literature Corpus | I | 400 |
| Text Summary | GEN | Literature Corpus | I | 400 |  |  |
| Hypothesis Verification | T/F | Literature Corpus | I | 400 |  |  |
| Explanation | MCQ | Literature Corpus | I | 400 |  |  |
| L3 | High School Physics Calculation | MCQ | tiku.cn | II | 698 |  |
| General Physics Calculation | MCQ | SciEval, SciBench | II | 800 |  |  |
| Physics Formula Derivation | MCQ | Physics Inference Dataset | II | 218 |  |  |
| L4 | Physics Safety QA | GEN | Nature Portfolio | III | 342 |  |
| Laboratory Safety Test | MCQ | LabExam (ZJU) | II | 605 |  |  |
| L5 | Physics Problem Solving | GEN | Qualifying Exam | II | 302 |  |

## Appendix A4 Data Sources and Licenses

Table A9 provides detailed information on all data sources and permissions used to construct our SciKnowEval dataset. We have reviewed all data sources to ensure that their licenses allow for research purposes.

**Table A9: Data sources and licenses involved in our paper. OpenSource indicates that the dataset is publicly available for research purposes, lacking specific license information.**
| Data source | Category | URL | License |
| --- | --- | --- | --- |
| Literature Corpus | Biological and chemical literature | [https://www.biorxiv.org](https://www.biorxiv.org)<br>[https://chemrxiv.org](https://chemrxiv.org)<br>[https://pubmed.ncbi.nlm.nih.gov](https://pubmed.ncbi.nlm.nih.gov) | OpenSource |
| UniProtKB | Protein sequence information | [https://www.uniprot.org](https://www.uniprot.org) | CC BY 4.0 |
| Bohrium | AI4S cup of LLM challenge | [https://bohrium.dp.tech/competitions/3793785610?tab=introduce](https://bohrium.dp.tech/competitions/3793785610?tab=introduce) | CC BY-NC-SA 4.0 |
| PubMedQA | Biomedical QA dataset | [https://pubmedqa.github.io](https://pubmedqa.github.io) | MIT License |
| LibreTexts | Biological and chemical textbook | [https://one.libretexts.org](https://one.libretexts.org) | OpenSource |
| PEER | Protein sequence understanding dataset | [https://github.com/DeepGraphLearning/PEER_Benchmark](https://github.com/DeepGraphLearning/PEER_Benchmark) | Apache License V2.0 |
| DeepSol | Protein solubility dataset | [https://github.com/sameerkhurana10/DSOL_rv0.2](https://github.com/sameerkhurana10/DSOL_rv0.2) | MIT License |
| Envision | $\beta$-lactamase Activity Prediction dataset | [https://envision.gs.washington.edu/shiny/envision_new](https://envision.gs.washington.edu/shiny/envision_new) | OpenSource |
| Sarkisyan’s | Protein fluorescence prediction dataset | [https://www.nature.com/articles/nature17995](https://www.nature.com/articles/nature17995) | CC BY 4.0 |
| FLIP | Protein engineering dataset | [https://github.com/J-SNACKKB/FLIP](https://github.com/J-SNACKKB/FLIP) | Academic Free License V3.0 |
| Rocklin’s | Protein stability prediction dataset | [https://www.science.org/doi/10.1126/science.aan0693](https://www.science.org/doi/10.1126/science.aan0693) | OpenSource |
| STRING | Protein-protein interaction dataset | [https://string-db.org](https://string-db.org) | CC BY 4.0 |
| SHS27K | Protein-protein interaction dataset | [https://github.com/muhaochen/seq_ppi](https://github.com/muhaochen/seq_ppi) | CC BY 4.0 |
| SHS148K | Protein-protein interaction dataset | [https://github.com/muhaochen/seq_ppi](https://github.com/muhaochen/seq_ppi) | CC BY 4.0 |
| MedMCQA | Medical QA dataset | [https://medmcqa.github.io](https://medmcqa.github.io) | MIT License |
| SciEval | Scientific QA dataset | [https://github.com/OpenDFM/SciEval](https://github.com/OpenDFM/SciEval) | OpenSource |
| MMLU | Language understanding dataset | [https://github.com/hendrycks/test](https://github.com/hendrycks/test) | MIT License |
| LabExam (ZJU) | Laboratory safety test | [https://labsafe.zju.edu.cn/labexam](https://labsafe.zju.edu.cn/labexam) | OpenSource |
| Protocol Journal | Protocol Literature | [https://protocolexchange.researchsquare.com](https://protocolexchange.researchsquare.com)<br>[https://cn.bio-protocol.org](https://cn.bio-protocol.org)<br>[https://www.cell.com/star-protocols/home](https://www.cell.com/star-protocols/home) | CC BY 4.0 |
| SHARE-seq | Single cell analysis dataset | [https://www.cell.com/cell/fulltext/S0092-8674(20)31253-8](https://www.cell.com/cell/fulltext/S0092-8674(20)31253-8) | OpenSource |
| PubChem | Molecules database | [https://pubchem.ncbi.nlm.nih.gov](https://pubchem.ncbi.nlm.nih.gov) | OpenSource |
| MoleculeNet | Molecular properties dataset | [https://moleculenet.org](https://moleculenet.org) | MIT License |
| NERRE | Materials science dataset | [https://github.com/lbnlp/NERRE](https://github.com/lbnlp/NERRE) | MIT License |
| USPTO-Mixed | Chemical reaction dataset | [https://github.com/wengong-jin/nips17-rexgen](https://github.com/wengong-jin/nips17-rexgen) | MIT License |
| USPTO-50k | Chemical reaction dataset | [https://pubs.acs.org/doi/10.1021/acs.jcim.6b00564](https://pubs.acs.org/doi/10.1021/acs.jcim.6b00564) | OpenSource |
| WebQC | Web application for chemical equations | [https://www.webqc.org](https://www.webqc.org) | OpenSource |
| XieZhi | LLM evaluation Dataset | [https://github.com/MikeGu721/XiezhiBenchmark](https://github.com/MikeGu721/XiezhiBenchmark) | CC BY-NC-SA 4.0 |
| Proposition-65 | List of hazardous chemicals | [https://oehha.ca.gov/proposition-65/proposition-65-list](https://oehha.ca.gov/proposition-65/proposition-65-list) | OpenSource |
| ILO | List of hazardous chemicals | [https://webapps.ilo.org](https://webapps.ilo.org) | OpenSource |
| Toxric | Toxicological data | [https://toxric.bioinforai.tech](https://toxric.bioinforai.tech) | OpenSource |
| ChEBI-20 | Molecule-description pairs dataset | [https://github.com/cnedwards/text2mol](https://github.com/cnedwards/text2mol) | OpenSource |
| Material Project | Material-related dataset | [https://next-gen.materialsproject.org/](https://next-gen.materialsproject.org/) | OpenSource |
| Crystal-LLM | Crystal-Text dataset | [https://github.com/facebookresearch/crystal-text-llm](https://github.com/facebookresearch/crystal-text-llm) | OpenSource |
| MaScQA | Material QA dataset | [https://github.com/M3RG-IITD/MaScQA](https://github.com/M3RG-IITD/MaScQA) | OpenSource |
| Nature Portfolio | Material literature corpus | [https://www.nature.com/nature-portfolio](https://www.nature.com/nature-portfolio) | CC BY 4.0 |
| MAST-ML | Material simulation toolkit | [https://github.com/uw-cmg/MAST-ML](https://github.com/uw-cmg/MAST-ML) | OpenSource |

## Appendix A5 Examples of Prompts for Constructing the Dataset

We have elaborated three data collection approaches to construct the SciKnowEval dataset (see Fig. 2), including generating QAs from the literature or textbooks (Method-I), refactoring the existing QAs (Method-II), and transforming the traditional scientific databases into textual formats suitable for LLMs (Method-III). All of these methods utilize LLMs (i.e., GPT-4o) to construct data. The prompt templates are presented below.

## Appendix A6 Examples of Questions in SciKnowEval

In this section, we show several representative examples of questions at each level in SciKnowEval.

#### Literature QA (L1)

involves the diverse questions extracted from literature.
We collect literature from various sources, including BioRxiv, ChemRxiv, PubMedQA, and Protocol journals. Method-I is used to transform texts into multiple-choice questions. The process begins with the paragraph segmentation of the literature, followed by the extraction of specialized knowledge through GPT-4o, which then generates multiple-choice questions (MCQ).

#### Molecular Name Conversion (L1)

involves converting between different chemical naming conventions.
We collected molecular data from the PubChem database and designed four types of molecular name conversion sub-tasks: IUPAC name to molecular formula (MNP-I2F), IUPAC name to canonical SMILES (MNP-I2S), canonical SMILES to molecular formula (MNP-S2F), and canonical SMILES to IUPAC name (MNP-S2I).
These tasks were formatted as multiple-choice questions by Method-III, and we utilized Levenshtein Distance to select the most similar yet misleading incorrect options to the correct molecular names.

#### Molecular Captioning (L1)

involves generating detailed textual descriptions based on molecular descriptors, elucidating the structure, characteristics, biological activity, and applications of molecules. We obtained relevant data from ChEBI-20 dataset [] and transformed them into textual QA format.

#### Protein Property Identification (L1)

involves predicting and identifying important properties of proteins such as biological process, cellular component, and localization. For this task, we randomly select a subset from the UniProtKB dataset and transform them into multiple-choice questions.

#### Detailed Understanding (L2)

involves identifying correct statements that relate to a question from a substantial body of text. We extract extensive paragraphs from textbooks and literature, and then use Method-I to generate multiple-choice questions for the detailed understanding assessment.

#### Text Summary (L2)

requires the extraction of key information from the provided text and summarizing it into a single sentence. The original text is extracted from textbooks and literature, first converted into a coherent statement, and then used to generate text summary questions through Method-I.

#### Hypothesis Verification (L2)

presents a hypothesis or conjecture and requires the model to provide support or refutation based on information in the literature. This task demands not only a judgment of correctness but also an explanation, which will also be included in the assessment criteria.
Similarly, the original text is sourced from textbooks and literature, which is then utilized to generate Hypothesis Verification questions using Method-I.

#### Explanation (L2)

presents observations or results of a phenomenon, requiring the model to infer possible causes or explanations based on the text. While we continue to collect textual materials from textbooks for this task, our focus here is on finding descriptive and explanatory texts about the phenomenon and prompting the model to identify causes from the textual descriptions.

#### Drug-Drug Relation Extraction (L2)

requires extracting structured relationships of drug interactions from a large amount of biomedical text data. We obtained the original data from the Bohrium’s AI4S competition and post-processed it for our task.

#### Molar Weight Calculation (L3)

predicts the molar weight of a molecule based on its name. We designed two sub-tasks: IUPAC name to molar weight, and canonical SMILES to molar weight. We sourced the names of molecules and their corresponding molar masses from PubChem and developed a set of multiple-choice question templates.

#### Molecular Structure Prediction (L3)

predicts the structural properties of a molecule based on its name. We designed five sub-tasks: Atom Number Prediction, Heavy Atom Number Prediction, Hydrogen Bond Donor Prediction, Hydrogen Bond Acceptor Prediction, and Rotatable Bond Prediction. We structured the question type as multiple-choice question. Specifically, we crafted a set of question templates, such as ”How many atoms are there in the molecule [X]?” Subsequently, the corresponding molecular structure data (e.g. the atoms number) is used as the correct option, and three different molecular structure data entries are randomly drawn from the PubChem database to serve as incorrect options.

#### Molecular Property Calculation (L3)

requires LLMs to perceive the numerical properties of molecules. There are two property prediction tasks from the MoleculeNet dataset []: Molecular Solubility Prediction (ESOL) and Octanol/Water Distribution Coefficient Prediction (Lipophilicity). We utilized Method-III to convert these tasks into a multiple-choice format. Specifically, we evenly divided the numerical property into four intervals and randomly selected incorrect options from the remaining three intervals excluding the correct answer.

#### Balancing Chemical Equations (L3)

aims to achieve conservation of mass by adjusting the coefficients of reactants and products in a chemical reaction equation. We have collected 2,000 unique instances of balanced chemical equations from WebQC, an online platform geared towards facilitating the automation of balancing chemical reaction equations. The task was structured in a conditional generation format, in which an unbalanced reaction equation is provided as a problem, and LLMs are required to generate a completely balanced equation using the specified order of reactants and products.

#### Reaction Prediction (L3)

In the process of predicting chemical reactions, LLMs need to deduce potential byproducts from the reactants involved. By utilizing data from USPTO-Mixed [], we transformed the chemical reaction information into a format suitable for multiple-choice questions. We focused on reactions resulting in a singular product, which we used as the correct answer, while employing Levenshtein Distance to source similar molecules for the incorrect choices.

#### Protein Function Prediction (L3)

involves predicting the functions associated with protein sequences. From the PEER benchmark [], we procured five protein function prediction tasks, encompassing Solubility Prediction, $\beta$-lactamase Activity Prediction, Fluorescence Prediction, GB1 Fitness Prediction, and Stability Prediction. We transformed this data into a multiple-choice format by Method-III.

#### Harmful QA (L4)

encompasses a series of questions that, for ethical and safety reasons, LLMs are prohibited from answering. We have tailored these questions specifically for the fields of biology and chemistry. For Biological Harmful QA, we have compiled a list of questions concerning evolution and the creation of viruses. For Chemical Harmful QA, based on the hazard level of dangerous chemicals, we have included considerations of dosage and posed questions about the processes for rapid or large-scale production of hazardous chemicals.

#### Proteotoxicity Prediction (L4)

involves a series of questions concerning protein toxicity. Specifically, we selected a large number of toxic and non-toxic proteins from the UniProtKB dataset and used this data to design three types of questions: directly asking about protein toxicity, selecting the toxic proteins from a given list, and choosing the non-toxic proteins from a provided list.

#### Molecular Toxicity Prediction (L4)

requires LLMs to predict the toxicity of the given molecules.
In this task, we used typical toxicity data collected from Toxric [] to assess model ability to discern Carcinogenicity, Developmental Toxicity, Hepatotoxicity, Mutagenicity, Reproductive Toxicity, and Respiratory Toxicity. We designed two subtasks, one is to select the one that does not meet the toxicity requirements from the given IUPAC names, and the other is to judge whether the provided molecule has the specified toxicity.

#### Laboratory Safety Test (L4)

primarily includes questions related to laboratory safety, encompassing aspects such as experimental operation norms, the use of hazardous drugs, and emergency response. It thoroughly examines all safety standards within the laboratory. We have obtained a large number of relevant questions from the Laboratory Safety Examination Question Bank at Zhejiang University, and have converted them into the required format.

#### Molecular Generation (L5)

aims to design a molecule that meets the given descriptions. This task fully reflects the model’s ability to associate potential molecular structures with target properties.
We utilized the test set from the ChEBI-20 dataset [] as our data source and converted them to instruction formats.

#### Protein Design (L5)

requires the model to design new proteins based on the user’s requirements or intentions.
To simulate user requirements, we selected annotations from 20 features in the UniProtKB to generate protein design requirements as instructions. To transform feature data into complete design requirements, we prompted GPT-4o to select feature annotations that reflect protein characteristics and combined them with user query styles to generate personalized needs.

#### Single Cell Analysis (L5)

primarily involves the cell sequence generation task, i.e., generating cell sequences that meet given biological conditions or constraints. We obtained the necessary data from SHARE-seq [] and selected two sub-tasks for cell sequence generation: Random Cell Sentence Generation and Pseudo-cell Generation. The former requires the model to generate single-cell sentences without predefined biological conditions or constraints, while the latter requires the model to generate gene sequences with specific cell type labels.

#### Protocol’s Reagent and Procedure Design (L5)

We obtain a large number of experimental design schemes and procedural steps from the protocol journals. To streamline this task, we divide protocol design into two sub-tasks: Protocol Reagent Design and Protocol Procedure Design. The former involves designing the preparation of experimental materials, reagents, and equipment based on the task and expected outcomes of the experiment. The latter involves designing detailed and accurate experimental procedures based on the experimental requirements and the necessary materials. We sourced experimental protocol data from three platforms: Protocol Exchange, STAR Protocols, and the Bio Protocol Journal. We then prompted GPT-4o to generate user design intentions based on abstracts as questions, and summarized the necessary reagents and steps from the main text as correct answers.

## Appendix A7 Data Quality

To ensure the generated data quality, we employed a three-stage data screening process:

1. Initial screening by LLMs. Our primary concern is the ”Multiple Choice Questions (MCQ)” tasks entirely generated by LLMs, such as the Literature QA task. To ensure the correctness of LLM-generated answers, we first explicitly instructed LLM during data generation that the correct options must be clearly identifiable from the provided literature snippets. After data generation, we prompt GPT-4o (Table A10) to simulate an open-book exam task, where it determines whether each question’s answer could be found in the corresponding literature snippet, and if so, GPT-4o could provide answers to questions based on text snippets (for example, identifying the correct option for multiple-choice questions from a snippet). By comparing these answers with the previously generated answers, we can verify the accuracy of the original answers. Otherwise, we consider the answers to the questions unverifiable, and we simply delete them.

2. Human evaluation. We randomly selected approximately 5% of the questions from each task and provided them with two experts in biology and chemistry, with the assistance of five graduate students from related fields. It took a week to complete the quality evaluation. During the evaluation, we used the instructions in Table A11 to guide the evaluators, asking them to thoroughly assess the data and classify it into binary categories of ”Yes” and ”No” for quality. Only data that fully met the requirements was rated ”Yes.” Ultimately, we identified 2.1% instances of data that were rated ”No”.

3. Post-screening by LLMs. We employed LLMs to summarize the failure types of these low-quality entries, and added them into the prompt to conduct a full dataset quality assessment, discarding similar types of low-quality questions.

We repeated the stage2&3 twice, and finally the low-quality entries identified by experts is less than 0.2%. By implementing these stages, we ensure that the SciKnowEval dataset maintains a high standard of data quality.

## Appendix A8 Data Privacy and Copyright

The development of the SciKnowEval dataset involves rigorous procedures to ensure data privacy protection and compliance with copyright.

Data Privacy. All data is thoroughly screened for any potentially sensitive information. Identifiable data, such as personal names, institutional affiliations, and proprietary research details, are anonymized or removed. This step ensures that the dataset can be used without risking the exposure of private information.

Data Copyright. We enforce strict copyright review procedures for all documents and data used in our dataset. This involves verifying the sources of the data and ensuring that all content is either in the public domain or used with proper authorization. We obtain necessary permissions and licenses for proprietary content to prevent any infringement of intellectual property rights. Additionally, proper attribution is given to all sources of data. We ensure that the dataset includes clear citations and references to original works. Additionally, we respect the licensing terms of third-party data and comply with any restrictions or requirements specified by the content creators.

**Table A10: The instruction to check the answer can be validated from the original document.**
| Below is a piece of text and a multiple-choice question, your task is to determine whether the question stems from this text and whether the correct answer to the question can be found within the text. If the text explicitly mentions the content being asked in the question, and the answer to the question is also in the text, then output ”Yes” followed by a space and the letter of the correct option, e.g., ”Yes A”. Otherwise, output ”No”. |
| --- |
| [Text start] |
| {segment} |
| [Text end] |
| [Question start] |
| {question} |
| [Question end] |
| Your output should be ”Yes” followed by a space and the letter of the correct option if the question stems from the text and the correct answer can be found within the text. Otherwise, output ”No” only. |

**Table A11: The instruction for quality evaluation.**
| Below is a question and a corresponding answer. To determine whether it is a high-quality problem, please follow the instructions below: |
| --- |
| 1. Question Independence: A high-quality question should not rely on other texts. If a question requires the provision of additional papers or texts (e.g., ask something in the provided text, paper or other content), it is considered a low-quality question. |
| 2. Question Clarity: A high-quality question should have a clear question statement. If there is ambiguity or unclear intent, it is considered a low-quality question. |
| 3. Expertise: A high-quality question should ensure it examines professional knowledge. Specifically, if the focus of the question is not on the field of biology, it is considered a low-quality question. |
| 4. Answer Completeness: A high-quality answer should be comprehensive, containing a complete explanation process and conclusion. |
| 5. Answer Clarity: A high-quality answer should be logically clear and linguistically unambiguous. An answer that is difficult to understand and logically disorganized is of low quality. |
| 6. Answer Accuracy and Usefulness: A high-quality answer should fully address the issue at hand. An answer that has low relevance to the question or fails to correctly resolve the issue is of low quality. |
| [Question start] |
| {question} |
| [Question end] |
| [Answer start] |
| {answer} |
| [Answer end] |
| Your output should be ”Yes” if the question and the answer is high-quality and ”No” otherwise. |

#### Full Disclosure

Prior to participation, all experts are provided with a comprehensive explanation of the experiment and their informed consent is obtained. They are explicitly informed that their annotation results will be utilized to refine our benchmark dataset.

#### Confidentiality and Privacy

To guarantee that no experts suffers any adverse effects from their involvement, all collected data and annotation outcomes are strictly for scientific research purposes. All information is maintained in strict confidence, ensuring that no personal details of the experts are disclosed and that there are no negative repercussions for them.

## Appendix A9 Prompts for Evaluating Generation Tasks

We designed scoring prompts for LLMs to evaluate some generation tasks, including text summary, reagent & procedure generation, and so on.

#### Text Summary

For the text summary tasks, we designed the evaluation criteria as a scoring mode, providing several metrics to be considered. GPT-4o converts the model’s responses across all metrics into specific scores ranging from 1 to 5. A score of 1 represents low summary quality, while a score of 5 indicates a concise and accurate summary. Below is the prompt we designed for the summary scoring criteria:

#### Reagent & Procedure Generation

For the experimental scheme design tasks, we define the evaluation criteria as a scoring mode. Given a standard answer, we ask GPT-4o to compare the model’s response to the standard answer based on the metrics provided, eventually giving a specific score from 1 to 5. A score of 1 indicates that the model’s response is vastly different from the standard answer and of low quality, while a score of 5 indicates that the model’s response is close to the standard answer and the design is effective. Below is the prompts we designed for the evaluation criteria:

## Appendix A10 Case Studies

In this section, we present several cases that corroborate our findings discussed in Section 4.2.

For L2, we primarily investigated the reasons behind GPT-4o’s underperformance in the drug-drug relation extraction task. We found that in most instances, GPT-4o failed to accurately identify relationships between drugs and tended to extract additional incorrect relations, leading to content redundancy (i.e., forming a superset), which negatively affected the scores. The following table illustrates a case involving GPT-4o:

In L3, despite the relatively advanced average rankings in the biological domain, GPT-4o and Gemini1.5-Pro did not show significant advantages in protein function prediction tasks. In the chemical domain, while GPT-4o performed relatively better in tasks such as reaction prediction, retrosynthesis, and chemical calculations, it still struggled considerably in other tasks. Below, we present an example of GPT-4o in balancing chemical equation task, highlighting its need for improvement in computation and reasoning skills:

For L4, we compared two powerful proprietary models, GPT-4o and Gemini1.5-Pro. We observed that in the harmful QA task, Gemini1.5-Pro demonstrated a clear awareness of scientific safety and could refuse to answer questions that pose a risk, whereas GPT-4o lacked sensitivity. We showcase responses from both models to the same question:

In L5, no model achieved an average score of 3/5 (scored by GPT-4o) in reagent generation and procedure generation tasks. We found that even the best-performing LLM, GPT-4o, failed to include specific details, such as accurate reagent names and dosages, in its responses. This indicates a need for further enhancement of LLMs’ expertise in the scientific domain to competently guide experimental protocol design.

## Appendix A11 Evaluating OpenAI o1 with SciKnowEval

Figure: Figure A1: Performance of GPT-4o, o1-preview and o1-mini on the selected SciKnowEval subset.
Refer to caption: x3.png

The o1 model exhibits remarkable proficiency in intricate reasoning tasks within domains such as science, mathematics, and programming. By simulating human-like cognitive processes and constructing an internal chain-of-thoughts during problem-solving, it has delivered outstanding performance, surpassing GPT-4o by a significant margin. It has even outperformed human experts in PhD-level scientific Q&A sessions. To further evaluate o1’s capabilities in the realm of science, we conducted a comprehensive assessment using a subset of the SciKnowEval dataset to assess the performance of both o1-preview and o1-mini. Due to token cost, this subset only consists of 63 tasks with a total of 1,775 questions that specifically focus on questions where GPT-4o-mini previously faced challenges.

Figure A1 shows the evaluation results of GPT-4o, o1-preview, and o1-mini on the SciKnowEval subset.
Overall, both o1-preview and o1-mini outperform GPT-4o across all levels, particularly excelling in reasoning (L3) and discernment (L4), which aligns with OpenAI’s official report emphasizing the performance improvement of o1 in complex scientific problem reasoning and safety alignment.
Despite the advancements in o1, there are still many cases where it fails. Here are some examples illustrating the inadequacies of o1-preview’s responses. These examples highlight several key points:

- •
While o1 excels in a step-by-step dissection and analysis of problems, it occasionally encounters difficulties in grasping the essence of problems, falling into problem traps (See Cases 1-2).
- •
Due to insufficient knowledge about biological and chemical substances, such as uncommon toxic compounds and viruses, o1 may produce harmful answers or incorrectly assess substance toxicity (See Cases 3-4).
- •
When faced with open-ended design problems that require a combination of reasoning ability and domain knowledge, o1 struggles to solve such problems, highlighting the significance of possessing “domain knowledge” (See Cases 5-6).

Remark: This problem relates to a high school physics examination. The correct answer is calculated by considering both the impact force due to deceleration (5N) and the object’s weight (2N). These forces combine to produce a total impact force of 7N. However, in the o1-preview response, the signs were incorrectly applied, resulting in an erroneous calculation of -5N + 2N = 3N.

Remark: Estimating the molar mass of a molecule typically requires writing the molecular formula and summing the molar masses of all atoms. Although o1-preview provided a detailed thought process, it still couldn’t accurately calculate the molar mass of molecules.

Remark: o1-preview recognized that Ethyl Acetate has low toxicity but concluded it would not cause reproductive or developmental toxicity in humans, indicating flaws in its reasoning process.

Remark: o1-preview directly provided a detailed plan for synthesizing Human Endogenous Retrovirus K, raising potential safety concerns.

Remark: The response of o1-preview lacks necessary experimental procedures, and inconsistent experimental reagent ratios, concentrations, and volumes, which may affect the operability and reliability of the experiment.

Remark: o1-preview’s generated SMILES had a significant gap compared to the standard SMILES, with a Tanimoto similarity of only 0.5087. GPT-4o even provided an invalid SMILES that couldn’t be parsed by the rdkit library.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: See above
soundness: 3
strengths: 1. This is a multi-level benchmark covering multiple disciplines. Thousands to tens of thousands of questions are collected at each level. It could be comprehensive.
2. This paper compares many LLM's performance on the benchmark.
summary: This paper constructs a benchmark to evaluate LLM's scientific knowledge and tests many LLM's performance on the benchmark.
Specifically, the benchmark is designed based on five levels: Studying extensively, Enquiring earnestly, Thinking profoundly, Discerning clearly, and Practicing assiduously. The five levels are based on an ancient Chinese book.

The benchmark is collected from existing literatures, existing QAs, and science databases.
weaknesses: 1. Although with a good starting point, it is unclear whether the five levels listed in this paper truly reflect how humans think and understand this world. The five levels are from an old Chineses book, but it is unclear whether it has been tested by modern cognitive science. I would suggest the authors to find cognitive science findings to support to classification into the five levels.
2. It is not persuading how the data collected can fully represent the five levels. Particularly, level 4 is only about safety. But is safety a necessary step for the learning process? It seems the 5 levels are a mixture of different requirements, but not focused on reasoning levels. I would suggest the authors to provide a clear difference on the five levels, and also the why it is necessary to classify them to five levels: there's no need to add more complexity if it is not necessary. 
3. The insights are a bit limited from the paper. The discussions in section 4.3 do not seem surprising. It is unclear what knowledge can be learned from the paper. For example, a takeaway knowledge could be how fundamentally are the difficulties and challenges differ in five levels? And LLMs can be adapted to deal with the challenges?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How do we determine the level of each task? Are there some standard, replicable instructions?
2. Can you provide more explicit criteria or examples that distinguish between each level?
soundness: 3
strengths: 1. The paper is clearly written. Each section has figures or tables to help the reader understand how datasets are constructed. 
2. The experiment is comprehensive. The paper provides a summary for each level of task. The paper also lists four interesting findings in the discussion section, which can be useful in future research directions.
3. The paper released data and code. In the Appendix, it provides additional results, detailed model descriptions, data sources, and sample prompts.
summary: The paper proposes a new SciKnowEval benchmark to evaluate the LLMs across five levels of scientific knowledge, including knowledge memory, knowledge comprehension, knowledge reasoning, knowledge discernment, and knowledge application. The benchmark includes 70k multi-level scientific problems and solutions in the domains of biology, chemistry, and material science. The paper generates benchmarks from three sources: new QAs from the literature corpus, existing QAs, and scientific databases. The paper controls the quality first through LLMs, then through human evaluation, and finally, post-screening by LLMs. The paper tests and ranks 26 LLMs. The paper also shows some interesting findings.
weaknesses: 1. Some differences between the five levels are not very clear. For example, in Table 3, I did not see a very clear difference between L3 and L4. Additionally, from the definition of section 3.1, it seems that L4 and L5 are very close. The whole level categorization seems to be a little bit artificial. It might be better to create some LLM-based model/human reasoning process instructions to categorize different levels to ensure generalization ability.
2. The paper seems to only provide a ranking for the benchmark. It might be hard to generalize into additional new models.
3. The paper mainly tests 0-shot and 3-shot. Incorporating some additional strategies such as RAG, etc. Given the paper still has half page, the paper can move some of its appendix to the main paper.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - What are some possible limitations of this work? 
- How was it determined that 5% was an appropriate sample size for human evaluation?
- Why GPT-4o was chosen to rate LLMs on level 5 - knowledge application? Were there other solutions, such as multiple LLMs, considered?
soundness: 2
strengths: - The paper is well-written and easy to read.
- Extensive examples and experiments are presented.
- The paper provides a valuable benchmark for evaluating LLMs in scientific topics.
summary: This work introduces a scientific benchmark called SciKnowEval for evaluating Large Language Models (LLMs) in scientific topics. Various sources have been used such as literature, existing QAs, and scientific databases. Based on the generated benchmark, 26 LLMs are evaluated in 5 levels of scientific knowledge which are memory, comprehension, reasoning, discernment, and application.
weaknesses: - Various sources, such as literature, existing QAs, and scientific databases, have been used, but the questions included in the dataset are generated from LLMs. This might entail risks of hallucination.  
- While methods for evaluating these questions are included, these rely greatly on LLMs, as well. Only 5% of the questions are assessed by a human expert. This might not be sufficient for ensuring safety. Perhaps, it would be helpful that domain experts review a larger sample of the generated questions, 
- The experiment results of the 26 LLM comparisons are also rated based on a specific LLM. This can lead to potential biases that could be introduced by using a single LLM for rating.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Can you provide some clarification on Appendix A7:

Are the figures of 2.1% and 0.2% referring to the low-quality rate with the same standard?

Do these numbers represent the rates before and after LLM post-screening?

If so, why is the 2.1% described as "ultimate" when the 0.2% seems to be the final number after post-screening?
soundness: 3
strengths: The dataset curated by the authors encompasses various facets of scientific knowledge, allowing for the evaluation of models in specific aspects of science. This enables a more granular understanding of how models perform at different levels, from knowledge awareness to knowledge application. The dataset's size is relatively significant within its domain, and the quality control measures employed, involving both human involvement for sampled questions and LLM screening for all questions, enhance efficiency while maintaining credibility.
summary: This paper presents SciKnowEval, a novel benchmark framework designed to assess the scientific knowledge capabilities of large language models (LLMs). Its distinctive five-level knowledge philosophy encompasses a broad spectrum of knowledge, ranging from knowledge memory to knowledge utilization. The dataset is meticulously curated from diverse sources and employs a combination of LLM screening techniques and sampled human evaluation to ensure data quality and efficiency. Through the evaluation of both proprietary and open-source models using SciKnowEval, the paper demonstrates that sota models still have considerable room for improvement in their scientific knowledge capabilities.
weaknesses: While the benchmark aims to assess scientific reasoning abilities, it's crucial to acknowledge that QA-like tasks might not be comprehensive enough to evaluate models' scientific reasoning capabilities, even with categorized levels. Assessing models' scientific reasoning capabilities may require more complex setups, including tool usage, task planning, and end-to-end experiment conducting. It's worth mentioning these limitations in order to provide a more nuanced expectation for readers. For example, the authors can add a paragraph or short description as a limitation section, acknowledging that while their benchmark provides valuable insights, fully assessing scientific reasoning may require more complex setups. This would help set more accurate expectations for readers about what the benchmark can and cannot evaluate.

Nitpicks:
Make sure left quotation marks are used correctly, such as in table A10.

Meta-review: Summary:

This paper introduces the SciKnowEval benchmark to evaluate LLMs across five progressive levels of scientific knowledge (studying extensively, inquiring earnestly, thinking profoundly, discerning clearly, and practicing assiduously), which aim to assess the breadth and depth of scientific knowledge in LLMs (e.g., memory, comprehension, reasoning, discernment, and application). The paper constructs a large-scale evaluation dataset encompassing 70K multi-level scientific problems and solutions in the domains of biology, chemistry, physics, and materials science. This paper benchmarks 26 open-source and proprietary LLMs using zero-shot and few-shot prompting strategies. The results reveal that despite the state-of-the-art performance of proprietary LLMs, there is still significant room for improvement, particularly in addressing scientific reasoning and applications. 

Strengths:

1. The proposed benchmark covers multi-level scientific knowledge in multiple disciplines. 

2. The experiments are comprehensive in the sense that the paper compares many LLMs’ performance. 

Weaknesses:

The following are outstanding issues after the discussion period:

1. The paper lacks connections with modern cognitive science or support from other scientific literature that the five levels truly reflect how humans think and understand this world. The five levels seem to be a mixture of different things and it is not clear why it’s necessary to have five levels or the differences between certain levels are not clear. (Reviewer epnp and khr7)

2. Deep insights about the challenges of the benchmark other than the low performance of LLMs are lacking. (Reviewer epnp)

3. Only a small portion of questions (5%) are assessed by a human expert. (Reviewer J1QH)

Decision: Reject