ChemEval: A Multi-level and Fine-grained Chemical Capability Evaluation for Large Language Models
Authors: Yuqing Huang
Rongyang Zhang
Xuesong He
Xuyang Zhi
Hao Wang
Nuo Chen
Zongbo Liu
Xin Li
Feiyang Xu
Deguang Liu
Huadong Liang
YiLi
Jian Cui
Yin Xu
Shijin Wang
Qi Liu
Defu Lian
Guiquan Liu
Enhong Chen
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Benchmark
Chemical Knowledge Inference
Primary area: datasets and benchmarks

Abstract: The emergence of Large Language Models (LLMs) in chemistry marks a significant advancement in applying artificial intelligence to chemical sciences. While these models show promising potential, their effective application in chemistry demands sophisticated evaluation protocols that address the field's inherent complexities. To bridge this critical gap, we introduce ChemEval, an innovative hierarchical assessment framework specifically designed to evaluate LLMs' capabilities across chemical domains. Our methodology incorporates a distinctive four-tier progression system, spanning from basic chemical concepts to advanced theoretical principles. Sixty-two textual and multimodal tasks are designed to enable researchers to conduct fine-grained analysis of model capabilities and achieve precise evaluation via carefully crafted assessment protocols. The framework integrates carefully curated open-source datasets with expert-validated materials, ensuring both practical relevance and scientific rigor. In our experiments, we evaluated the performance of most main-stream LLMs using both zero-shot and few-shot approaches, with carefully designed examples and prompts. Results indicate that general-purpose LLMs, while proficient in understanding chemical literature and following instructions, struggle with tasks requiring deep chemical expertise. In contrast, chemical LLMs perform better in technical tasks but show limitations in general language processing. These findings highlight both the current limitations and future opportunities for LLMs in chemistry. Our research provides a systematic framework for advancing the application of artificial intelligence in chemical research, potentially facilitating new discoveries in the field.

Published as a conference paper at ICLR 2026
CHEMEVAL: A MULTI-LEVEL AND FINE-GRAINED CHEMICAL CAPABILITY EVALUATION FOR LARGE
LANGUAGE MODELS
Xuesong He', Xuyang Zhi', Hao Wang", Nuo Chen', Zongbo Liu', Xin Li!! Feiyang Xu². Deguang Liu', Huadong Liang', Yi Li?, Jian Cui', Yin Xu Shijin Wang", Qi Liu', Defu Lian', Guiquan Liu Enhong Chen 'University of Science and Technology of China 2iFLYTEK Co., Ltd {huangyug, z @mail www.uu.cc..c..m
Yuqing Huang'; Rongyang X
ABSTRACT
The emergence of Large Language Models (LLMs) in chemistry marks a signif- icant advancement in applying artificial intelligence to chemical sciences. While these models show promising potential, their effective application in chemistry demands sophisticated evaluation protocols that address the field's inherent com- plexities. To bridge this critical gap, we introduce CvemEval, an innovative hierar- chical assessment framework specifically designed to evaluate LLMs' capabilities across chemical domains. Our methodology incorporates a distinctive four-tier progression system, spanning from basic chemical concepts to advanced theoret- ical principles. Sixty-two textual and multimodal tasks are designed to enable researchers to conduct fine-grained analysis of model capabilities and achieve precise evaluation via carefully crafted assessment protocols. The framework integrates carefully curated open-source datasets with expert-validated materials, ensuring both practical relevance and scientific rigor. In our experiments, we eval- uated the performance of most main-stream LLMs using both zero-shot and few- shot approaches, with carefully designed examples and prompts. Results indicate that general-purpose L.Mss, while proficient in understanding chemical literature and following instructions, struggle with tasks requiring deep chemical expertise. In contrast, chemical L.LMs perform better in technical tasks but show limitations in general language processing. These findings highlight both the current limi- tations and future opportunities for LLMs in chemistry. Our research provides systematic framework for advancing the application of artificial intelligence in chemical research, potentially facilitating new discoveries in the field
INTRODUCTION
The advent of large language models has ushered in transformative in artificial intelligence, particularly within the domain of natural language processing. The expansive capabilities these models have not only redefined the boundaries of text generation and understanding (Brown al.
2020
Ouyang et al. 20022; Touvron et al. 2023 Achiam et al. [2023] Huang et al. |025] Gu et al., Liang al. (2225 but have also opened new avenues for various domains, such as recom-
2025
mendation (Shen et al. 2024) and scientific exploration (Beltagy et al. 2019; Hong et al. 2022 Bhatacharjee al.) (024). Researchers have adeptly employed LLMs to accelerate the pace of scientific research and instigate transformative shift in scientific research paradigms. The field of chemistry has notably profited from the integration and advancement of L.LMs (Yu al., 2024 Chen et al. 2024), becoming key area where these sophisticated technologies have delivered substantial advantages. The intricate nature chemical research, involving complex molecular interactions and reactions, presents unique challenges that LLMs can address through advanced pattern recognition and predictive analytics.
* "Both authors contributed equally this research.
1 'Corresoonding author.
'The code and data available at httpaa/tttteeeeeeoooooooo///tttr

---

Published as a conference paper at ICLR 2026
In order to systematically assess the capabilities of LLMs across various domains and identify areas for their potential enhancement, numerous benchmarking initiatives have been introduced. For in- stance, the MMLU (Hendrycks et al. 2020) covers 57 tasks spanning basic mathematics, American history, computer science, law, and other fields. The XieZhi (Gu et al. 2024) benchmark includes three major academic categories with 516 specific subjects. However, general benchmarks (Zhong) et al., 2023] Huang et al. 2024) often overlook detailed assessment of chemical knowledge. Al- though Sun et al. (2024) introduce SciEVAL framework for assessing the competencies of LLMs within the scientific domain, the chemistry-related tasks are overly simplistic and do not adequately capture the depth required. Regarding chemistry Nomain-specifee benchmarks, [uo et al. (2023) propose 8 chemical tasks aimed at assessing understanding, reasoning, and explanation abilities, but the benchmark consists of tasks derived from existing public datasets, which may be insufficient to capture the full spectrum of competencies needed for thorough chemical research. Other studies like (White et al., 2023 Liu et al. 2023) have similar problems. Moreover, existing benchmarks fail to address the capability of LLMs to extract chemical information from text and tables. This limitation prevents them from tackling key issues of interest to chemistry researchers and has not fully met the
specialized needs of chemistry.
In light of these considerations, we introduce ChemEval, a benchmark designed to address the gap in the hierarchical assessment framework for LLMs in chemistry by providing multi-dimensional evaluation. 1). Extensive tasks are included in ChemEval, which encompasses chemical tasks of interest to researchers that were not included in previous benchmarks. It has four levels, thirteen dimensions, and a total of 62 distinct tasks, covering a vast array of issues within the domain of chemical research. Notably, we innovatively introduce test sets related information extraction and inductive generation in chemistry. 2). Multimodal tasks are specifically designed to assess models' capabilities in understanding and reasoning across diverse themisstr--relatee data types, including text, molecular structure diagrams, and spectral images. 3). Domain experts in chemistry have meticulously crafted in-depth task datasets and prompts for ChemEval, partly addressing the previ- data in chemistry benchmarks. Compared to previous work, our study
ous lack of encompasses a broader range of tasks that are of actual concern in chemical research. It assesses models on graduated scale of capabilities, from general to domain-specific skills, to determine the model's proficiency. Our aim is to construct specialized tasks from the perspective of chemical re- searchers, thereby providing valuable insights for AI researchers and chemists, and improving large language models' effectiveness in chemical research.
For experiments, we conducted a highly detailed evaluation process, focusing on designing prompts that challenge LLMs, including O-shot and few-shot settings. We evaluated currently widely used LLMs. including both general LLMs and specialized chemical LLMs, and gained many meaningful insights. This fine-grained evaluation has revealed that though general LLMs excel in Literature Understanding tasks and possess great require a deeper understanding of molecular structures and scientific inference. On the other hand, specialized LLMs generally show improved chemical abilities even when their ability to understand literature and cestruction-follwwwe significant improvements in the way LLMs are trained and evaluated for chemical tasks. In addition, explored the impact of few-shot learning and model size on the performance of large language models and provided corresponding insights. We highlight the contributions of this paper as follows:
capability, they struggle with tasks that
capability is diminished. This finding underscores the need for
We have established an open-source benchmark for I.L.Ms in the field of chemistry, which provides a fine-grained evaluation of their mastery of chemical knowledge as well as their multimodal reasoning capabilities, filling the absence of a holistic benchmark that encompasses the diverse range of tasks within the chemical domain.
We set up 4 progressive levels and access 13 model capability dimensions through 62 tasks in ChemEval, which is developed through extensive discussions and collaborative design with chem- istry researchers, involves constructing novel tasks of interest to chemical researchers and encom- passes the primary focal points of chemical research.
We conducted a fine-grained evaluation of LLMs in chemical tasks, using various prompt settings assess both general and specialized LLMs. This revealed significant differences between dif- ferent types of LLMS and identified challenging tasks with potential for optimization. This work offers critical insights to guide researchers in the optimization and application of LLMs. thereby enhancing their effectiveness in chemical research.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Large Language Models for Chemistry. The emergence of Large Language Models (LLMs) has revolutionized Natural Language Processing, with cutting-edge proprietary models like GPT-4o (Hurst et al. 2024) and open-source alternatives such as LlaMA (Touvron al. 2023) and Qwen Yang et al.) 2024) demonstrating exceptional capabilities across linguistic tasks. However, apply- ing these general models to chemistry reveals significant limitations in domain-specific knowledge. To bridge this gap. researchers have developed specialized approaches: Galactica (Taylor et al.,
2022) underwent pre-training on comprehensive scientific corpora, SciGLM (Zhang et al. 2024a)
employed strategic fine-tuning with scientific datasets, and ChemCrow (Bran al. 2023) enhanced performance by integrating expert-designed chemistry tools. Chemistry-focused models, including ChemDFM (Zhao et al., 2024). LlaSMol (Yu et al.) 2024), and C. (Zhang al.) (024b)
incorporate tailored training methodologies, while specialized applications such as Drugchat (Liang al. 2023) and Drugassist (Ye et al. 2023) specifically address molecular structures and chemical properties. Despite these advancements, achieving comprehensive chemical understanding through LLMs remains a promising frontier for further research and innovation.
Large Language Models Evaluations for Chemistry. The progress made in the field of LLMs is tightly linked to the establishment of robust evaluation frameworks. For general tasks, benchmarks such as MMLU (Hendrycks et al., 2020) and GLUE (Wang et al. 2018) have become standard tools for assessing model capabilities. In the scientific domain, recent initiatives like SciEval (Sun) et al. 2024), SceMQA (Liagg et al. 2024), and SciAssess (Cai et al. 2024) have been introduced to evaluate scientific reasoning and knowledge. In the chemistry domain, recent benchmarking initia- tives such as ChenLLMbench (Guo et al. (003). ChemBench (Mirza et al. 2024), and MaCBench (Alampara al.) 2025) have emerged, yet each presents significant limitations: Cheml.L.Mbench covers only eight task categories with unreviewed datasets; ChemBench offers 7,000 samples, but is limited by its reliance on multiple-choice questions, lack of open-ended tasks, and insufficient eval- uation metrics for chemical experiment design tasks such as synthesis pathway recommendations: while MaCBench introduces multimodal evaluation but exhibits similar constraints in task diversity and assessment metrics. The absence of precise benchmarking framework impedes LLM advance- ment in chemistry. a field with complex conceptual knowledge and computational challenges. To address this gap, introduce ChemEval, a precise evaluation framework designed to rigorously assess LLM capabilities across the multifaceted landscape of chemistry.
CHEMEVAL
To fill the absence of a holistic benchmark that encompasses the diverse range of tasks within the chemical domain, we introduce a refined benchmark named ChemEval specifically designed to eval- uate the comprehensive capabilities of LLMs within the chemical domain. It not only encompasses text tasks such as literature comprehension and experimental planning, but also incorporates multi- modal tasks, including molecular formula recognition and spectroscopic data analysis. As illustrated in Figure 3.1.I it contains four levels in the field of chemistry, each of which includes several dif- ferent chemical dimensions, ensuring a fine-grained evaluation of LLMs. This framework measures the models' ability to understand and infer chemical knowledge from a broad range of dimensions through series of meticulously designed tasks. In the following subsections, we will provide detailed introduction to the task content and data construction process of ChemEval.
3.1 TASK DESCRIPTION
3. ADVANCED KNOWLEDGE QUESTION ANSWERING
This segment is pivotal in assessing the models' proficiency in understanding and applying funda- mental chemical concepts, which include Objective Question dimension and Subjective Question dimension, total of 15 different tasks. Through a blend of objective and subjective tasks, the Advanced Knowledge Question Answering challenges the models to demonstrate their integrated capabilities in areas of chemical terminology, quantitative analysis and cross- modal reasoning. The tasks within this section are designed to be both comprehensive and diagnostic, providing a clear measure of the models' readiness to tackle more advanced chemical inquiries.
3

---

Published as a conference paper at ICLR 2026
intomation Extaniion
Orjectiva Corstions
Query:
starting from ethyl acatoacetate as the raw material synthesize the product 2.4 dinxopentano and ethyl estar, can you provide stepi-stpppaee synthesis route?
tassehiv
deneration
Advanced cnneitio
Liberatra
Underntanding
Question Ansseerin
sutpectia Queceer
Answer:
Ethyl aceloacetate undergues biomination
basioconditions to form a-bromoethyl acetoacetate.
to
The product with base form 2.4 dioooppennaiii aci ethyl ester.
ChemEval
Goncrall
Scientifi Knovtedge Deduction
Relwoay
Foatyse
Please cuiput the molecular name SMILES format corresponding this image <tragelferres
Query:
Images:
Ma Ralenstrr Phopurty
Underraoon Inderrlanden
Molecular
Reaetoo
Inedicieon
Anagon
Enervtiationl
Narmia
Moteouad
Conattionn ReGORITMEN
Answer: NCCeLco2XXeJCC)-ONN
MusacASS Devirrption
Readiin
O Dcacmmnn
Predtcen
Figure 1: The overview of CremEval. It includes 4 progressive levels, evaluating 13 dimensions of LLMs' capabilities and featuring 62 distinct chemical tasks that cover wide range of chemical knowledge, from foundational concepts to advanced topics suitable for graduate-level research.
3.1.2 LITERATURE UNDERSTANDING
Advanced Knowledge Question Answering is designed to assess the model's comprehension and mastery of chemical knowledge, while Literature Understanding evaluates the model's capacity to interpret and assimilate information from chemical literature, which is foundational for subsequent inductive generation tasks. Literature Understanding, which includes the Inductive Generation, In- formation Extraction, and Molecular Name Recognition dimensions, comprising total of 19 tasks, delves into tasks crucial for understanding and extracting meaningful information from the chemi- cal literature. The primary focus is assessing the LLMs' ability comprehend and extract key information from both textual content and image data chemical literature, enabling the execution of more complex information-intensivee tasks.
3.1.3 MOLECULAR UNDERSTANDING
This section builds upon the previous foundation assess the model's understanding and gener- ative capabilities at the molecular level. It includes 4 dimensions: Molecular Name Generation, Molecular Name Translation, Molecular Property Prediction, and Molecular Description, to- tal of 15 tasks. Molecular Understanding focuses on core tasks in molecular cognition, aiming to evaluate LLMs in molecular formula conversion, structural diagram interpretation, and the de- scription/prediction of molecular properties based on structural and spectroscopic data. These tasks assess the models' proficiency in interpreting and generating chemical information accurately.
3.1.4 SCIENTIFIC KNOWLEDGE DEDUCTION
Having established a solid grasp of basic chemical knowledge, the skill to interpret scientific liter- ature, and the capacity understand molecular structures, we expect that the model will proceed to conduct deeper chemical reasoning and deduction. The Scientific Knowledge Deduction level covers Retrosynthetic Analysis, Reaction Condition Recommendation, Reaction Outcome Predic- tion and Reaction Mechanism Analysis, total of 13 tasks, which are essential for effective chemical synthesis. This part evaluates the LLMs capabilities retrosynthetic analysis, recommending re- action conditions, predicting reaction outcomes, and analyzing reaction mechanisms. These tasks are essential for efficient chemical synthesis, requiring the model to accurately recognize chemical structures from images and perform complex reasoning and analysis using specific knowledge.
3.2
BENCHMARK GENERATION PIPELINE
3.2.
DATA COLLECTION
The overall process of benchmark construction is illustrated in Figure 2 Given the indispensable role of data in the realm of LLMs (Yin et al. (024), our data collection strategy consists of two primary components: open-source data and dexain-expert data.
4

---

Published as a conference paper at ICLR 2026
Data Crawling
Download Official Datasets
b)
Acadomic website
Relevant
Manual Screening
Expert
Construct G&A Pairs
Expert Manually
Raw Data
Professional Textbooks and Experimental Data
Relevant Knowledge and Data
Deduplication and Removal of Irrelevant Items
Advanced Knowledge Question Answering
Instruction
Manually Construct D&A Pairs
Literature Underslanding
Creation
Molecular Understanding
Q&A Dataset
Filtered Data
Scientific Knowledge Deduction
Figure 2: Data collection steps of ChemEval. The process is divided into three main steps: a). Data Collection: Raw data is collected from academic websites via web crawling, and experts manually gather data from professional textbooks and experimental data. b). Data Filtering: The raw data undergoes deduplication and removal of irrelevant items to produce filtered data. c). Q&A Pair Construction: Experts manually construct Q&A pairs related chemistry and create prompt structions, resulting in four instruction test sets.
For the open-source component, we retrieved relevant publications on chemical language models from academic repositories using keywords such as "chemistry", "large language models", "knowl- edge question answering". and "information extraction". From these papers, systematically extracted and codified downstream tasks and their associated datasets to develop our chemical eval-
uation framework Yu al. 2024 Guo al. 2023 Edwards et al.. 2022 Chen al. 2023 Guo et al. 2021 Zhou et al., 2023 Fang et al. 2023). We then downloaded the official datasets for these tasks, prioritizing those that included official test set.
However, relying solely on open-source data provides inadequate coverage. To enhance the eval- uation's rigor and breadth, compiled extensive corpus of domain-expert materials. These raw materials were sourced from: (1) approximately 500 university- chemistry textbooks, ex- ercise books, and examinations: and (2) around 9,000 real-world experimental records provided by collaborating laboratories. We utilized these resources to manually construct question-anse -aswer pairs tailored to specific task types. Crucially, to prevent potential data leakage, materials such as textbook exercises were not directly copied. Instead, chemistry experts used them references to author novel questions aligned with the target knowledge dimensions and task formats.
3.2.2 DATA PROCESSING
Through our data collection efforts, we curated a massive corpus of raw data across the chemical domain. To transform this raw material into robust benchmarking resource, we executed rigorous pipeline encompassing selection, filtration, task-specific formatting, and quality assurance.
Data Filtration and Formatting. Initially, domain experts conducted a manual filtration process to climinate unsuitable samples. Data was systematically removed if it fell into any of the following categories: (a) fask-mismatch: data misaligned with our predefined task parameters; (b) Ambiguity: questions susceptible to multiple valid interpretations; (c) Answer multiplicity: tasks permitting sev- eral reasonable answers yet lacking a comprehensive standard in the source material; (d) Outdated content relying on obsolete or revised chemical facts; and (e) Redundancy: dupli- cated or highly similar items. This strict protocol resulted the exclusion of approximately 200 items (roughly 2% of the initial pool). Upon passing this stage, the data processed across three distinct dimensions:
information:
(1) Advanced knowledge question answering. We formulated question-answer pairs using collegiate and graduate-leve textbooks. This set spans seven core disciplines (organic, inorganic, materials, analytical, biochemistry, physical, and polymer chemistry).
(2) Literature understanding. We extracted pertinent excerpts from scientific publications, pairing them with meticulously designed questions and reference answers.
5

---

Published as a conference paper at ICLR 2026
Elimination reactions
Ottee reactions 5%
Addition reactions 5%
Oxidationrrduccion reactions 10%
ChemEval 3360
Subssitution reactions 18%
Coupling reactions 62%
(a) Distribution of evaluation tasks in ChemEval.
(b) Percentage distribution of reaction types in the ChemEval.
Figure 3: Distribution of the CvemEval benchmark dataset. (a) A hierarchical breakdown of evalua- tion tasks categorized by four core capability levels and modalities. (b) The percentage distribution of major chemical reaction types evaluated within the dataset.
(3) Molecular understanding and scientific reasoning. By merging open-source datasets with pro- prietary empirical records from partner laboratories, built evaluation that rigorously simulate domain-specific scientific scenarios.
Quality Assurance and De-contamination. To uphold the integrity of this dataset, enforced strict three-tier quality assurance pipeline: Annotation, Review, and Final Audit. First, undergradu- chemistry students, guided by standardized SOP, conducted the primary annotations. Second, graduate student team cross-examined these annotations to verify logical consistency. Finally, chem- istry faculty members executed a conclusive audit, ensuring all ground-truth answers were strictly fact-based. Crucially, to preclude any risk of data leakage, we rigorously cross-referencd our down- stream test sets against the training corpora existing open-source chemical models, systematically purging any overlapping instances.
3.2.3 DATA STATISTICS
Following the comprehensive data processing and rigorous deduplication pipeline described above, we successfully finalized the ChemEval benchmark dataset. As illustrated in Figure (a), this ex- tensive curation effort yielded total of 3, 3,160 high-quality evaluation instances. To ensure that our benchmark captures both the breadth and depth required to assess state-of-the-arl large language models, strategically combined established evaluation paradigms with substantial volume of novel, challenging tasks. total. ChemEval encompasses 62 distinct task categories. Among these, 25 tasks were carefully adapted from highly regarded open-source repositories to maintain alignment with existing community standards. The remaining 37 proprietary tasks were engineered entirely in- house domain experts specifically address complex chemical reasoning capabilities that have been previously overlooked by existing benchmarks.
To comprehensively probe the multifaceted capabilities of modern AI models, these evaluation in- stances are systematically structured into distinct text and multimodal subsets. The text-only subset contains 1,960 meticulously annotated examples, comprising 18 open-source tasks and 24 in-house designed tasks. The multimodal subset consists of 1,200 test examples, which integrate 12 open- source tasks with in-house designed visual-textual tasks. Notably, several core tasks are intention- ally formulated in both text and multimodal formats, establishing robust framework to explicitly evaluate and compare the cross-modal alignment capabilities of current vision-language models.
Furthermore, to provide rigorous and chemically meaningful testing ground for scientific reason- ing, the dataset was designed to offer extensive coverage major chemical reaction mechanisms frequently encountered in real-world research and industrial applications. As summarized in Fig- ure 3 b), coupling reactions account for the largest proportion of the data. This is followed by fundamental reaction classes including substitution, oxidation/reduuctinn addition, and elimination reactions. The remaining instances are grouped into an "other" category that covers distinct and highly specialized mechanisms, such rearrangement, hydrolysis, and cyclization, which often test

---

Published as a conference paper at ICLR 2026
model's grasp of complex reaction pathways and structural transformations. This diverse distribu- tion ensures that LLMs are evaluated not merely on rote memorization, but on their genuine ability to generalize across wide spectrum of chemical transformations. Complete statistics regarding the precise data distribution and task provenance are detailed in Appendix
3.2.4 INSTRUCTION CREATION
To evaluate the effectiveness of the model, constructed task task-specifie prompts and 3-shot task- specific prompts for text downstream tasks (Wei et al., (022). For downstream tasks with open- source datasets, to facilitate evaluation, the evaluation system in this paper strengthens the format of the output data based on its instructions. For the domain expert-built part, the evaluation system in this paper will design instructions for task introduction and formatted output according to the task type, and continuously adjust the instructions based on the return results of GPT-4o, thereby strengthening the instructions for different self-constructed downstream tasks.
3.2.5 METRICS
In this study, we utilize a range of evaluation metrics for a fine-grained assessment of LLMs' per- formance across diverse tasks. For the majority of tasks, we utilize the Fl score and Accuracy. In addition, we utilize BLEU (Papineni et al. 2002), Exact Match, Normalized Root Mean Square Error, Valid Output Ratio, LLMs Score, L2 Score, and Overlap evaluation metrics for different tasks to accommodate various task requirements. A detailed introduction to the metrics is provided in the appendix C.2
EXPERIMENT
4.1 SETUP
To conduct a fine-grained diagnostic evaluation of LLMs' chemical capabilities, our framework assesses both general and specialized models. For general LLMs, we include OpenAl-ol/o3-mini
(Jaech al. 2024), GPT-4o (Hurst et al.) 2024), Claude-3.7-Sonnee (Anthropic 2225),, Gemini-2.5-
pro (Team al. 2023), , Qwen2.5-7B// 2Bt.5557//14B/2BB/// BB/14BB2BB/2B (Yang et al. (024)) LLaMA3.3-8P [Touvron]
cct al. 2023). Grok3 (xAl, (2025), and DeepSeek- V3/RI (Liu et al. [2024). For chemistry-speciffc
T.I.Ms. evaluate ChemDHM (Zhao et al. 2024), LlaSMol (Yu al. (2024), ChemLLMM (Zhang
et al. 2024b) and ChemSpark 2 For multimodal chemical tasks, we evaluated mainstream MLLMs,
Satude-3.7-Sonne (Anthropic) 2025). Qwen-VL Max (Bai et al. 2023), Phi-Vision-3.5 (Abdin et al. 2024). across four levels of multimodal chemistry tasks. We used the official APIs of general models for evaluation and ran the chemistry-specifie models on two A40 48GB GPUs We employed greedy decoding for all LLM inference in our experiments.
including GPT-40 (Hurst et al. 2024),
To illustrate the capability of LLMs in various chemical tasks, we present their average zero-shot performance across four levels, with detailed results shown in the [able[[ To assess their adaptability and in in-context learning abilities, we also report three-shot performance across the same levels. Some tasks, such as Chemical Paper Abstract Generation, are not included in our three-shot evaluation due to context length limitations.
4.2 PERFORMANCE RESULTS
We evaluate the models' performance for each task across four assessment Levels. Evaluation re- sults for text tasks summarized in Table and those for multimodal tasks are summarized in Table Certain models are unable to address specific tasks entirely. For example. LLaMA3.3-8B demonstrates poor instruction-followint capabilities TempRee task in the O-shot setting, which significantly impairs its ability to generate responses based on task prompts. Consequently, we are unable provide numerical results for the tasks affected by this limitation. We further discuss the key findings of our benchmark and analyze how different LLM configurations influence perfor- mance, offering practical insights for the development of chemistry-specific benchmarks.
https:// www. model wtt.secccccccmmlle///ooo

---

Published as a conference paper at ICLR 2026
Table 1: Performance overview of representative multi-level 0-Shot text tasks on ChemEval. Claude3.71 7T denotes Claude 3.7-Sonnet- Thinking, whereas Claude3. N.N denotes Claude 3.7- 3.7-Sonnet. For the complete experimental results, please refer the appendix D.
Upwatea
14:01
MCTB
2409
60.00
12.3
20.3
$2.30
6:30
$4.00
34/00
28.13
18.00
5,400A
333
14.0
64.33
61:30
16.33
$1.50
2013
3138
12.00
13300
14.50
17.80
13.33
1614
23.33
21.6
35.19
11:8
21.44
2147
11
20.3
0.34
1.5
18.31
22.55
44
3-4.5
1260
0330
2210
$3.00
1800
8.00
0880
0000
1100
11
1249
11.0
1360
13.01
7509
LAS 2L
2008
31.00
1653
21.0
210
2121
300
11
0530
1038
DESTIENI
TATTOR
Dad VOAR
LO
12
L30
UO
1400
30
ACO
DUPT
145
1M
100
113m
11.00
201
103
2610
A22000001 BI50 100%) CO0TRAD0 EONCIEPP
03TO 200
-20
LUM SIO
ADC 100)
192
2004
405
2113
2548
470
1M
3m
4.2.1 THE MODELS' PERFORMANCE ACROSS FOUR LEVELS.
The performance comparison of LLMs across four levels reveals distinct strengths and weaknesses:
Basic Knowledge. Within the level of Advanced Knowledge Question Answering, the results re- veal that OpenAl-ol exhibits superior performance in objective questions, and Gemini outperforms other models in subjective questions. which indicates the importance of reasoning ability in Q&A questions. Additionally, general LLMs like GPT-4o and Qwen2.5-72B also perform well in lit- erature understanding. However, chemistry-speciiaiied models (except ChemSpark) struggle with general tasks, highlighting instruction fine-tuning challenges, which suggests that general LLMs succeed stem from superior document comprehension and reasoning abilities.
Chemical Expertise. As for Molecular Understanding, ChemSpark stands out in these tasks de- manding an in-depth grasp of chemical molecules. Most models perform poorly in molecular name translation due a lack of formatting constraints in their outputs, owing the complexity of molec- ular expressions. ChemSpark's s advantage be attributed to training on diverse chemical literature with various molecular formula formats. Besides, observed that when confronted with complex tasks requiring quantitative calculations, models tend to provide overly cautious responses, such as "quan- tification software (Gaussian, ORCA, etc.) is needed" or "cannot determine from 2D structure", which significantly reduces the practical value of their answers.
Chemistry-specialine LLMs. Compared to general LLMs, specialized chemistry models show distinct patterns: 1). Drawbacks: Chemical LLMs notably lower in advanced knowledge answer- ing and literature comprehension, suggesting catastrophic forgetting during fine-tuning compro- mises their foundational language processing capabilities. 2). Advantages: Chemical models ex- cel tasks requiring specialized terminology and molecular properties. General models perform adequately on simpler tasks but struggle with complex chemical knowledge processing and infer- ence. 3). Instrrattionfoolloiing ability: Chemistry-specific LLMs demonstrate significantly lower instruction-followin capability than general LLMs likely due to limited exposure to diverse tasks during training. Without output format restrictions, these models default to patterns matching their fine-tuning data, sometimes producing interpretable results where format-constrained prompts removed, though with uncertain accuracy. This instruction inslluction-folooiingg deficiency could significantly impact the practical utility of these specialized models despite their domain expertise.
8

---

Published as a conference paper at ICLR 2026
Table 2: 3-shot performance changes text tasks relative O-shot in ChemEval. The symbols and accompanying values show performance changes compared to 0-shot, where "I" indicates an increase. 'I' a decrease, and '-' no change. The three values in the last column (t. () represent the number of tasks that show a significant increase, remain unchanged, and significantly decrease.
Tark
SATakk
CaleTrok Score
SubFF
TempF:
ProdF
ReaeTTR
MolPC
LRtcc
PathRec LLM
RaseProe Ovetlap
Change
Merre
LLM
F1
OpeAAFal
68.50 14.00
78:50 10.50
76.01
14:30
75:00
15.00
91.48
11:23
60:00 [35.00
71.60 16.10
18,00 115.00
40,63
110.01
14.41 16.67
(5.0. 1)
GPT-4
61.00 10.20
59.10
12.70
65:03
10:39
73:00
16.00
$6.88
10:70
71 70:00 135.00
68.55
15.98
15.60 12:40
25:00
12:13
20:27
16.47
(7.0. 3)
Gemini-2.00 S-Pm
20.00 [2.00
81
10.80
76.29
14:34
77.00
78.00
93.75
10:91
59:00
728.00
67.62
15.09
0.00
43.00
J0.75
20.08
12.06
(6. 1,3)
Deepseek v3
70:40 11.30
77.40
11
15.78
114.51
80.00
18,00
91.75
14.23
36.00 11..00
55.79
77.06
11.50 14,00
24,00
113.38
13.45
14.26
(6.0.5)
Qwen2.5 -20
60,80 [2.30
61.61
10.29
70.10
17.56
$0.00
15.00
$4.05
10.51
61.00
135,00
56.87
16:74
16.40 112.00
33.38
17.75
15,82
15.10
0.0
Mlmma333-88
29.00
(9.40
19.70 88.30
57.71
16.31
69.00
17.00)
73.26
11.28
39.00
113.00
53.20
15.85
2,40 10:27
17.58
(3.00)
14.29
17.38
(5.0.5) 5)
ChemDEM
30.50
11.70
16.40 11.70
20.04
10.05
41.00
[33,33
8,83
125.90
26.00
113.00
56.65
(4.70
12.49 113.51
28.75
14.63
17.46
113.67
(4.0.0)
ChemLLM
11.50
11.70
35.46
19.55
0..00
1.53
11.70
0.00
0.00
0.00
0.00
6.75
[4.13
0.00
(1
LlaSMol
23.50 19.00
68.37
160.87
0.000
0.00
0,00
0.00 [5.00
40.00
(6.50
0.00
17.50
17.50
0.00 13.68
(3.435)
ChenSpuk
31.60
[2.00
15.80 (2.70
72.86 (1.52
$0.00
13.00
96.40
14.00
14,00
32.00
115.00
82.88
[2.68
16,90 (20.80
27.00
111.75
11.03 18.13
(.0.7)
4.2.2 FACTORS AFFECTING MODEL PERFORMANCE IN CHEMISTRY TASKS
The influence of few-shot. Our experiment results of ICL are shown in Table 2 Few-shot prompt- ing influence model performance across different tasks. General LLMs tended to benefit from few- shot examples, especially in subjective question answering and literature understanding. In contrast, specialized chemistry models often show performance decreases with few-shot prompting. which may be attributed to catastrophic forgetting of ICL capabilities during task-specific fine-tuning. For complex chemistry-specifff tasks, performance variations remain minimal across all models, reflect- ing the inherent difficulty of these tasks and limitations in capturing expert-level chemical reasoning.
The impact of model scaling. We conducted experiments on Qwen2.5 models of different sizes. The results, as shown in Table 3 indicate a trend that increasing model size correlates with im- proves performance in most tasks, with notable gains in advanced knowledge Q&A and literature understanding. However, molecular understanding and scientific knowledge deduction tasks show minimal improvement the model scales. Tasks requiring specialized chemical knowledge (e.g., IUPAC2SMILES, CatRec) remain challenging despite parameter increases, with some tasks like MolPC even showing performance declines. This suggests that model scaling alone is insufficient for complex chemical tasks without specialized training data.
The impact of thinking models. While intuitively it may seem that thinking models possess stronger reasoning capabilities and might benefit in complex chemical tasks, our experimental com- parison of OpenAI-ol versus GPT-4o and DeepSeek-RI R1 versus DeepSeek-V3 V3 reveals more nu- anced reality. Although thinking models occasionally excel in specific tasks such as reaction prod- uet prediction, they demonstrate comparable performance to general models across most chemical tasks, with each architecture exhibiting distinct strengths in different tasks. Additionally, when prompted to employ chain-of-thoughh reasoning, some models declined to respond to certain tasks, citing insufficient information to formulate complete answers. We consider that the primary limita- tion in addressing sophisticated chemical challenges lies not in long reasoning ability but rather in insufficient domain -specff knowledge.
Stability analysis. As illustrated in the Table 12 we conducted robustness testing on multiple models and analyzed the stability of metrics across various tasks in the benchmark. The results demonstrate that the standard deviation for the vast majority of metrics does not exceed 5.0. indicat- ing consistent performance across evaluations. These results collectively indicate that our evaluation framework is robust, providing consistent and reliable assessments of system performance.
4.2.3 MULTIMODAL CHEMISTRY TASKS
The Table 9 illustrates the performance of mainstream multimodal large language models on ChemEval's multimodal tasks. Entries marked as -_' indicate instances where models failed to generate meaningful responses. Examining results across both Advanced Knowledge QA and Lit- erature Understanding levels reveals that while most models demonstrate satisfactory performance on elementary tasks such as molecular formula identification, they exhibit notable limitations when confronted with more sophisticated challenges involving chemical reaction pathways or molecular properties, evidenced in Pathway Parsing and Multiple Choice tasks. The performance degra-

---

Published as a conference paper at ICLR 2026
Table 3: The impact of model scaling on task performance.
Task
MCTask
SATask I.LM Score
CalcTask
CharME
CatTE
MolPC
CatRec
PPred
YPred
Metric
Accuracy
I.IM Score
F1
F1
Accuracy
F1
Fl
Accuracy
Qwen2.5-77
59.60
50.80
43.60
43.00
64.00
64.04
0.00
0.00
67.00
Qwen2.5-144
64.80
57.20
50.80
67.92
75.00
64.22
0.00
0,00
33.50
Qwen2.5-32B
67.20
58.10
57.40
79.42
100,00
67.70
0.00
0.53
85,00
Qwen2.5-72B
67.20
58.50
61.90
74.57
100.00
48.13
0.00
1.73
26.00
dation becomes even more pronounced in Molecular Understanding and Scientific Knowledge De- duction tasks, where models demonstrate considerable difficulty. These advanced tasks present a multifaceted challenge, requiring models to accurately recognize molecular structures and reaction equations from visual inputs while leveraging comprehensive chemical domain knowledge to formu- late correct responses. This combination presents considerable challenge to the models' integrated capabilities. It is worth noting that our evaluation exclusively assessed general-purpont multimodal large language models, without including specialized multimodal models designed specifically for chemical applications. Given that multimodal capabilities are increasingly crucial in chemical re- search, we think of this critical area that warrants further investigation and development.
LIMITATIONS AND FUTURE WORKS
Although ChemEval addresses a critical gap in evaluating LLMs in the chemistry domain by pro- viding comprehensive reference for model capability assessment, several limitations within the benchmark design remain. First, to achieve broad task diversity within finite resource constraints, the number of evaluation instances per individual task is relatively limited. While this design allows ChemEval to serve as a highly effective diagnostic tool for probing multifaceted capabilities, it may not suffice for exhaustive, large-scale stress testing of specific narrow tasks. Second, the current benchmark predominantly focuses on static, single-turn evaluation formats. does not yet incorpo- rate dynamic, interactive environments or systematically assess models' abilities to utilize external professional chemical tools and execute complex agentic workflows, which is essential for evaluat- ing weal-world scientific workflows. For the future development of ChemEval, we aim expand the scale of individual task datasets and introduce dynamic, agent-based evaluation protocols to assess end-to-end tool-use capabilities in advanced chemical research. Additionally, we plan involve chemical experts in manually evaluating complex LLM outputs to improve evaluation reliability, safety oversight, and alignment with human judgment.
CONCLUSION
In this paper, we presented ChemEval, a fine-grained benchmark for evaluating LLMS on chemical tasks across four assessment levels. Our experiments show that while general-purpose LLMs ex- cel in literature understanding and benefit from scaling and few-shot prompting, they struggle with molecular understanding and scientific knowledge inference. Chemistry-speciilizee models exhibit advantages in terminology and molecular property tasks but face challenges such as catastrophic for- getting and weaker Instruction-folloiin ability. These findings highlight that improvements in pa- rameter scaling or reasoning depth alone are insufficient to address complex chemical tasks. Instead, progress requires tighter integration of LLMSs with domain-specific knowledge, chemical simulation tools, and multimodal data. We hope ChemEval provides both a rigorous evaluation framework and foundation to inspire the development of chemistry-aware LLMs. ultimately driving advances in chemical research and accelerating the integration of artificial intelligence into the natural sciences.
ACKNOWLEDGMENTS
This work was supported by the Major Project of the National Social Science Fund of China: "Key Enabling Technologies for Agentic Document and Information Services in the Context of Digital and Intelligent Transformation" (Project No. 23&ZD228), the National Natural Science Foundation of China (Nos. 62441239, U23A20319, 62441227, and 62472394), and the Anhui Province Science and Technology Innovation Project (Nos. 202423k09020010 and 202423k09020011) We gratefully acknowledge the USTC Supercomputing Center for providing the computational resources essential to this research.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behi, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dong- dong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam Ade Jacobs, Mojan Javaheripi. Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng L.i, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu, Mengchen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh,, David Majercak, Matt Mazzola, Caio César Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez- Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilae, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy. Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang, Guanhua Wang. Lijuan Wang, Shuohang Wang, Xin Wang. Yu Wang. Rachel Ward. Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu. Ji- long Xue, Sonali Yadav, Fan Yang, Jianwei Yang, Yifan Yang. Ziyi Yang. Donghan Yu, Lu Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL https:: /ttpaii//aaa.v..e abs/ 2404 14219
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint
2023.
Nawaf Alampara. Mara Schilling- Wilhelmi, Martiño Khetarpal, Hargun Singh Grover, N. M. Anoop Krishnan, and Kevin Maik Jablonka. Probing the limitations of multimodal language models for chemistry and materials research, 2025. URI.
Indrajeet Mandal, Pranav
tattps://arxiv. wwwpxxv..r///... abs 2411.
Anthropic. Claude 3.7 sonnet. https: www.
2025.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond. arXiv preprint arXiv:2308.1240 2023.
Iz Beltagy, Kyle Lo, and Arman Cohan. Scibert: pretrained language model for scientific text. arXiv preprint arXiv: 10676, 2019.
Bishwaranjan Bhattacharjee, Aashka Trivedi, Masayasu Muraoka, Muthukumaran Ramasubrama- nian, Takuma Udagawa, Iksha Gurung, Rong Zhang, Bharath Dandala, Rahul Ramachandran, Manil Maskey, et al. Indus: Effective and efficient language models for scientific applications. arXiv preprint arXiv:2405.1072 2024.
Andres M Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. Chemcrow: Augmenting large-language models with chemistry tools. arXiv preprint arXiv:2304.05376, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Hengxing Cai, Xiaochen Cai, Junhan Chang, Sihang Li, Lin Yao, Changxin Wang, Zhifeng Gao, Yongge Li, Mujie Lin, Shuwen Yang, et al. Sciassess: Benchmarking llm proficiency in scientific literature analysis. arXiv preprint arXiv:2403.01977 2024.
Linqing Chen, Weilei Wang. Zilong Bai, Peng Xu, Yan Fang, Jie Fang, Wentao Wu, Lizhi Zhou, Ruiji Zhang, Yubin Xia, et al. Pharmgpt: Domain-specific large language models for bio- pharmaceutical and chemistry. arXiv preprint arXiv:2406. 18045, 2024.
11

---

Published as a conference paper at ICLR 2026
Ziqi Chen, Oluwatosin R Ayinde, James R Fuchs, Huan Sun, and Xia Ning. G 2 retro two- step graph generative models for retrosynthesis prediction. Communications Chemistry, 6(1): 102. 2023.
Carl Edwards, Tuan Lai, Kevin Ros, Garrett Honke, Kyunghyun Cho, and Heng Ji. Translation between molecules and natural language. arXiv preprint arXiv:2204. 11817, 2022.
Yin Fang, Xiaozhuan Liang, Ningyu Zhang, Kangwei Liu, Rui Huang, Zhuo Chen, Xiaohui Fan, and Huajun Chen. Mol-instructions: A large-scale biomolecular instruction dataset for large language models. arXiv preprint arXiv:2306.080 08,00,,,,,,,, 2023.
Hongchao Gu, Dexun Li, Kuicai Dong, Hao Zhang, Hang Lv, Hao Wang, Defu Lian, Yong Liu, and Enhong Chen. Rapid: Efficient parieval-augmented long text generation with writing planning and information discovery. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 16742-16763, 2025.
Zhouhong Gu, Xiaoxuan Zhu, Haoning Ye, Lin Zhang, Jianchen Wang, Yixin Zhu, Sihang Jiang, Zhuozhi Xiong, Zihan Li, Weijie Wu, et al. Xiezhi: An ever-updating benchmark for holistic domain knowledge evaluation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18099-18107, 2024.
Jiang Guo, A Santiago Ibanez-Lopez, Hanyu Gao, Victor Quach, Connor W Coley, Klavs F Jensen, and Regina Barzilay. Automated chemical reaction extraction from scientific literature. Journal of chemical information and modeling, 62(9):2035 2021.
Taicheng Guo, Bozhao Nan, Zhenwen Liang, Zhichun Guo, Nitesh Chawla, Olaf Wiest, Xiangliang Zhang, et al. What can large language models do in chemistry? a comprehensive benchmark on eight tasks. Advances in Neural Information Processing Systems, 36:59662-5999 -9668, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
Zhi Hong, Aswathy Ajith, Gregory Pauloski, Eamon Duede, Kyle Chard, and Ian Foster. The diminishing returns of masked language models to science. arXiv preprint arXiv:2205. 1342, 2022.
Yuging Huang, Rongyang Zhang, Qimeng Wang. Chengqiang Lu, Yan Gao, Yi Wu, Yao Hu, Xuyang Zhi, Guiquan Liu, Xin Li, Selfaug: Mitigating catastrophic forgetting in retrieval- auggenel-augmented generation via distribution self-alignment. arXiv preprint arXi::2509.039:44 2025.
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, al. C-eval: multi-level multi-discipline chinese eval- uation suite for foundation models. Advances in Neural Information Processing Systems, 36, 2024.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.2127.. 2024.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Cameey, et al. Openai ol system card. arXiv preprint arXiv:2412.167220 2024.
Sheng Liang, Hang Lv. Zhihao Wen, Yaxiong Wu, Yongyue Zhang, Hao Wang, and Yong Liu. Adaptive schema-aware event extraction with retrieval- -aumentl- generation. Findings of the Association for Computational Linguistics: EMNLP. 2025.
Youwei Liang, Ruiyi Zhang, Li Zhang, and Pengtao Xie. Drugchat: towards enabling chatgpt- like capabilities drug molecule graphs. arXiv preprint wwXiv:339.03900 2023.
Zhenwen Liang, Kehan Guo, Gang Liu, Taicheng Guo, Yujun Zhou, Tianyu Yang, Jiajun Jiao, Renjie Pi, Jipeng Zhang, and Xiangliang Zhang. Scemqa: A scientific college entrance level multimodal question answering benchmark. arXiv preprint 452 138, 2024.
12

---

Published as a conference paper at ICLR 2026
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chenggi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXfv:2412.19437. 2024.
Shengchao Liu, Jiongxiao Wang, Yijin Yang. Chengpeng Wang, Ling Liu, Hongyu Guo, and Chaowei Xiao. Chatgpt-powered conversational drug editing using retrieval and domain feed- back. arXiv preprint 23KKv::230..80000 2023.
Adrian Mirza, Nawaf Alampara, Sreekanth Kunchapu, Martiño Ríos-García, Benedict Emoek- abu, Aswanth Krishman, Tanya Gupta, Mara Schilling-Wilhelmi, Macjonathan Okereke, Anagha Aneesh, Amir Mohammad Elahi, Mehrdad Asgari, Juliane Eberhardt, Hani M. Elbeheiry, María Victoria Gil, Maximilian Greiner, Caroline T. Holick, Christina Glaubitz, Tim Hoffmann, Abdelrahman Ibrahim, Lea C. Klepsch, Yannik Köster, Fabian Alexander Kreth, Jakob Meyer, Santiago Miret, Jan Matthias Peschel, Michael Ringleb, Nicole Roesner, Johanna Schreiber, UIrich S. Schubert, Leanne M. Stafast, Dinga Wonanke, Michael Pieler. Philippe Schwaller, and Kevin Maik Jablonka. Are large language models superhuman chemists?, 2024. URL https://arxiv: org/ hwtp/244..014...
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Tingjia Shen, Hao Wang, Jiaqing Zhang, Sirui Zhao, Liangyue Li, Zulong Chen, Defu Lian, and Enhong Chen. Exploring user retrieval integration towards large language models for cross- domain sequential recommendation. arXiv preprint arXiv:2406..0m 03085, 2024.
Liangtai Sun, Yang Han, Zihan Zhao, Da Ma, Zhennan Shen, Baocai Chen, Lu Chen, and Kai Yu. Scieval: A multi-level large language model evaluation benchmark for scientific research. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19053-19061, 2024.
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085, 2022.
Gemini Team, Rohan Anil, Sebastian Borgeaud, nan--Batistte Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312./1800 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.099: 200.09988, 2023.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.0746) 2018
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 2424874--4887, 2022.
David Weininger. Smiles, chemical language and information system. 1. introduction to method- ology and encoding rules. Journal of chemical information and computer sciences, 28(1):31-23) 1988.
Andrew D White, Glen M Hocky, Heta A Gandhi, Mehrad Ansari, Sam Cox, Geemi P Wellawatte, Subarna Sasmal, Ziyue Yang, Kangxin Liu, Yuvraj Singh, et al. Assessment of chemistry knowl- edge in large language models that generate code. Digital Discovery, 2(2):368-377 2023.
13

---

Published as a conference paper at ICLR 2026
Team XAI Grok. https://x.att 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024.
Geyan Ye, Xibao Cai, Houtim Lai, Xing Wang, Junhong Huang, Longyue Wang, Wei Liu, and Xiangxiang Zeng. Drugassist: A large language model for molecule optimization. arXiv preprint WrXiv:2401.13333 2023.
Mingjia Yin. Chuhan Wu, Yufei Wang, Hao Wang, Wei Guo, Yasheng Wang, Yong Liu, Ruiming Tang, Defu Lian, and Enhong Chen. Entropy law: The story behind data compression and Ilm performance. arXiv preprint arXiv:2407.06645, 2024.
Botao Yu, Frazier N Baker, Ziqi Chen, Xia Ning, and Huan Sun. Llasmol: Advancing large language models for chemistry with a large-scale, comprehensive, high-quality instruction tuning dataset. arXiv preprint ar (xXx:2402.00399) 2024.
Dan Zhang. Ziniu Hu, Sining Zhoubian, Zhengxiao Du, Kaiyu Yang, Zihan Wang. Yisong Yue, Yuxiao Dong, and Jie Tang. Sciglm: Training scientific language models with self-reflective instruction annotation and tuning. arXiv preprint arXiv:2401.07955 0.74000, 2024a.
Di Zhang, Wei Liu, Qian Tan, Jingdan Chen, Hang Yan, Yuliang Yan, Jiatong Li, Weiran Huang, Xiangyu Yue, Dongzhan Zhou, al. Chemllm: chemical large language model. arXiv preprini arXiv:2402.06852, 2024b.
Zihan Zhao, Da Ma, Lu Chen, Liangtai Sun, Zihao Li, Hongshen Xu, Zichen Zhu, Su Zhu, Shuai Fan, Guodong Shen, et al. Chemdfm: Dialogue foundation model for chemistry. arXiv preprint arXiv:2401.14818, 2024.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint wrX1v:2304.06344, 2023.
Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: A universal 3d molecular representation learning framework. 2023.
A CHEMEVAL TASKS
In order systematically evaluate the multifaceted capabilities of large language models in the domain of chemistry, we propose multi-level and fine tine-grinedd evaluation framework that encom- passes broad spectrum of chemical knowledge and reasoning tasks. This framework is delineated into four primary categories: Advanced Knowledge Question Answering, Literature Understanding, Molecular Understanding, and Scientific Knowledge Deduction. Each of these categories represents progressively sophisticated level of chemical problem-solving. ranging from the assessment of fundamental chemical concepts and literature comprehension to molccular-level reasoning and high- level scientific deduction. The constituent tasks within each category are meticulously designed to interrogate specific competencies, such as objective and subjective answering. information extrac- tion, inductive generation, molecular property prediction, and retrosynthetic analysis. Collectively, this comprehensive benchmark offers a granular and holistic evaluation of LLMs' proficiency in both the understanding and application of chemical knowledge, thereby illuminating their potential utility and limitations in diverse chemical informatics applications.
A.1 ADVANCED KNOWLEDGE QUESTION ANSWERING
This segment is pivotal in assessing the models' proficiency in understanding and applying funda- mental chemical concepts, which include Objective Question dimension and Subjective Question dimension, total 15 different tasks. Through blend of objective and subjective tasks, the Advanced Knowledge Question Answering component challenges the models to demonstrate their insight in
14

---

Published as a conference paper at ICLR 2026
areas ranging from chemical terminology and quantitative analysis to the recognition and interpre- tation of chemical structures and diagrams. The tasks within this section are designed to be both comprchensive and diagnostie, providing clear measure of the models' readiness to tackle more advanced chemical inquiries.
A.1.1 OBJECTIVE QUESTIONS (OBIQA)
The first dimension is objective question answering, which primarily assesses the model's grasp of fundamental chemical knowledge and its capability to apply this knowledge in straightforward scenarios. Objective question answering encompasses the following tasks: Multiple Choice Task, Task, and True/False Task. By incorporating these tasks, ChemEval can more effectively gauge the model's overall proficiency in understanding and applying chemical knowledge across various contexts and formats. It should be noted that the True/False Task is exclusive to the text tasks and is not incorporated within the multimodal task set.
A.1.2 SUBJECTIVE QUESTIONS (SUBIQA)
The second dimension is subjective question answering, which includes Short Answer Task and Calculation Task, both aiming to evaluate the depth of the model's s comprehension and its ability to apply chemical knowledge effectively. Because on the basis of the previous task, the model also requires providing a detailed solution or reason, which involves the understanding of the chemical principles and concepts in the question, and applying these principles and concepts to construct logically clear and organized answers, which intuitively reflects the model's understanding of basic chemical knowledge.
Multimodal tasks further build upon these foundations, covering Statistical Chart QA, Statistical Ta- ble QA, Reaction Profile Diagram QA, Theoretical Potential Energy Surface QA, Infrared Spectrum QA, Raman Spectrum QA, UV-Vis Spectrum QA, Diffraction Pattern QA Kinetic Behavior Chart QA and Mass Spectrum TheThese tasks comprehensively evaluate the model's ability to interpret and reason using chemical graphics and experimental data.
A.2 LITERATURE UNDERSTANDING
Advanced Knowledge Question Answering is designed assess the model's comprehension and mastery of chemical knowledge. In contrast, Literature Understanding evaluates the model's abil- ity to interpret and assimilate information from chemical literature, which forms the foundation for downstream inductive generation tasks. Literature Understanding includes three dimensions: In- ductive Generation, Information Extraction, and Molecular Name Recognition, comprising a total of 19 tasks. These tasks are crucial for understanding and extracting meaningful information from chemical literature. The primary focus is on assessing LLMs' ability to accurately extract and in- terpret chemical data from text, and to subsequently generate new, contextually relevant content. Importantly, such tasks are not covered by other chemical benchmarks. The following subsections detail the specific tasks.
A.2.1 INFORMATION EXTRACTION (INFOE)
This is the first step to read a paper and also the foundation for the next inductive generation task. It involves the extraction of various elements related to chemistry, such as named entities, reaction substrates, and catalyst types, encompassing total of 11 tasks. These tasks aim to decompose and organize chemical information found in text, covering entities, relationships, and various aspects of chemical reactions.
A.2.2 INDUCTIVE GENERATION (INDUCGEN)
Based on Information Extraction, Inductive Generation involves creating new, coherent, and contex- tually relevant content based on existing data and knowledge. This process incorporates Chemical Paper Abstract Generation, Research Outline Generation. Chemical Literature Topic Classification, and Reaction Type Recognition and Induction, all focused on synthesizing and organizing chemical information in coherent and meaningful manner.
15

---

Published as a conference paper at ICLR 2026
A.2.3 MOLECULAR NAME RECOGNITION(MIN)
Molecular Name Recognition is a foundational step in the extraction and organization of chemical information, focusing on the accurate identification of molecular names and related entities from scientific literature and data sources. This task goes beyond simple text extraction and leverages multimodal techniques to integrate information from textual, structural, and graphical data alike. Its subtasks encompass Molecular Formula Recognition, Chemical Reaction Equation Recognition, 2D Molecular Structure Recognition, and Synthetic Pathway Analysis. Collectively, these subtasks enable comprehensive understanding and representation of chemical compounds and their trans- formations, serving as crucial underpinning for downstream knowledge discovery and advanced reasoning in chemical informatics.
A.3 MOLECULAR UNDERSTANDING
This section builds upon the previous foundation to assess the model's understanding and gener- ative capabilities the molecular level. It includes 4 dimensions: Molecular Name Generation, Molecular Name Translation, Molecular Property Prediction, and Molecular Description, total of 15 tasks. Molecular Understanding explores tasks essential for molecular understanding, evaluating the LLMs' ability to generate, translate, and describe molecular names and properties. These tasks assess the models' proficiency in interpreting and generating chemical information accurately. The following subsections detail various specific tasks within this objective.
A.3.1 MOLECULAR NAME GENERATION (MNGEN)
Molecular Name Generation is the basis of Molecular Understanding and only contains one task, Molecular Name Generation from Text Description. This task is purposed to evaluate the capacity of L.LMs to generate valid chemical structure representations. It necessitates that the models, based on intricate textual descriptions encompassing molecular structures, properties, and classifications, synthesize SMILES molecular formulas effectively.
A.3.2 MOLECULAR NAME TRANSLATION (MNTRANS)
Furthermore, Molecular Name Translation aims to enable deep understanding of molecular struc- tures and representations, which should serve as the fundamental knowledge for chemistry LLMss It focuses on converting molecular names between different formats, requiring L.MS to output specified alternative format based on given molecular representation. It involves the conversion between representations of molecules such IUPAC names and SMILES (Weininger 1988) molec- ular formulas, encompassing a total of five tasks, each focusing on distinct aspects of molecular notation conversion.
A.3.3 MOLECULAR PROPERTY PREDICTION (MPP)
Apart from molecular name understanding, the ability to predict molecular properties is also impor- tant. Molecular Property Prediction targets the forecast of wide range of physical, chemical, and biological attributes of molecules, encapsulated sification, which predicts categories of properties such as ClinTox, HIV inhibition, and polarity; and Molecule Property Regression, focusing on estimating numerical values such as Lipophilicity, polarity, and boiling point.
two core objectives: Molecule Property Clas-
A.3.4 MOLECULAR DESCRIPTION (MOLDESC)
To facilitate a deeper assessment of molecular understanding, the Molecular Description task has been developed to comprehensively evaluate LLMs' capabilities in interpreting and describing molecular structures and their properties. This task consists series of subtasks, each requiring the prediction of physicochemical properties of molecules based diverse input modalities. Be- sides the classic subtask of predicting physicochemical properties directly from molecular structures, this multimodal extension incorporates additional challenges: Physicochemical Property Predic- tion from Infrared Spectrum, Physicochemical Property Prediction from Raman Spectrum. Physic- ochemical Property Prediction from UV-Vis Spectrum, Physicochemical Property Prediction from
16

---

Published as a conference paper at ICLR 2026
Diffraction Pattern, Physicochemical Property Prediction from Mass Spectrum, and Physicochemi- cal Property Prediction from NMR Spectrum. Collectively, these tasks aim to assess LLMs' ability to interpret various molecular representations- spaaiinn textual, graphical, and spectral data for comprehensive property annotation and molecular understanding.
A.4 SCIENTIFIC KNOWLEDGE DEDUCTION
Having established solid grasp of basic chemical knowledge, the skill to interpret scientific litera- ture, and the capacity understand molecular structures, we expect that the model will proceed to conduct deeper chemical reasoning and deduction. So the part of Scientific Knowledge Deduction encompasses four key dimensions: Retrosynthetic Analysis, Reaction Condition Recommendation, Reaction Outcome Prediction and Reaction Mechanism Analysis, total of 13 tasks, which are es- sential for effective chemical synthesis. This part evaluates the LLMs' capabilities retrosynthetic analysis, recommending reaction conditions, predicting reaction outcomes, and analyzing reaction mechanisms. These tasks provide a comprehensive assessment of the models' performance in these eritical areas of chemical synthesis.
A.4.1 RETROSYNTHETIC ANALYSIS (RESSYN)
Retrosynthetic Analysis is crucial technique in the field of chemical synthesis, particularly in organic synthesis. The process begins with the target product and then examines potential synthesis pathways and reactant substrates. This approach highlights the reverse reasoning capabilities of LLMs in the field of chemical synthesis. It comprises Subsirate Recommendatioo, Synthetic Pathway Recommendation and Synthetic Difficulty Evaluation.
A.4.2 REACTION CONDITION RECOMMENDATION (RCREC)
Based on the results the Retrosynthetic Analysis, LLMs can recommend suitable reaction condi- tions. Reaction condition recommendation is key task in chemical synthesis. involving selecting the most suitable conditions for specific chemical reactions to ensure maximum efficiency, selec- tivity, and yield. This task integrates recommendations for conditions such ligands, reagents, and catalysts, encompassing total of six tasks, each targeting specific component of the reaction condition optimization.
A.4.3 REACTION OUTCOME PREDICTION (ROP)
After determining the reaction pathway and reaction conditions, the large model can predict possible reaction outcomes. Reaction outcome prediction is core technology in chemical synthesis aimed at predicting possible results of a reaction before is actually carried out. This encompasses Reaction Product Prediction, Product Yield Prediction, Reaction Rate Prediction.
A.4.4 REACTION MECHANISM ANALYSIS (RMA)
Reaction Mechanism Analysis critical area the study of chemical reactions, aiming to explain the detailed steps involved in the transformation from reactants to products. This is the final step the field chemical synthesis, including identifying various intermediates, and transition states, as well as the kinetic and thermodynamic parameters of each step in the reaction. Intermediate Derivation is the sole subtask in this phase.
B DETAILED DATASET STATISTICS
To comprehensively evaluate the chemical reasoning and knowledge capabilities of large language models, we constructed the ChemEval benchmark by integrating tasks from multiple sources. For the advanced knowledge question-answering level, our chemistry experts curated datasets based on textbooks and supplementary educational resources. For other levels, some tasks were adapted from existing datasets, such ChemRxnExtractor (Guo et al. 2021) Mol-Instruction (Fang et al. (023), ChemLLMBench (Guo et al. (023). and SMolInstruct Yu et al. 2024), while additional tasks were independently developed by chemistry experts.
17

---

Published as a conference paper at ICLR 2026
Table 4: Overview of text tasks in ChemEval.
Tasks
Source
Number
Multiple Tisk
Chemistry expert (Ours)
Filliisthe-Rlank
Chemistyy expert (Ours)
50
Thae/False Tisk
Chemistty expert (Ours)
50
Short Anwwer Task
Chemistry (Ours)
50
Calculation Tisk
Chemisty expert (Curs)
50
Clemical Fnilry Recogaition
Mol-Instmattioss
50
Chemical Eatity Relationchip Classification
Mol-Instractions
50
Symhetic Reaction Substrate Extraction
ChemRonEEttrattrr
50
Symhetic Reaction Additive Extraction
Chemistry expert (Ours)
20
Symthetic Reaction Solvent Extraction
Chemistry expert (Ours)
Reaction Temperature Extraction
Chemistry expert (Ours)
20
Reaction Time Extraction
Chemistry expert (Ours)
20
Reaction Product Extraction
ChemRonEEttzadoo
50
Charaterization
Chemistry expert (Ours)
20
Catalysi Type Extraction
Chemistry expert (Ours)
20
Yield Extraction
Clemisstyy eapert (Ours)
20
Chemical Paper Abstract Generation
Chemistry (Ours)
20
Research Outline Generation
Chemisty expert (Ours)
20
Chemical
Topic
Chemistry expert (Ours)
Reaction Type Recognition and Induction
Chemisty expert (Ours)
20
Molecular Name Generation from Text Description
Cheml. MBBnnh,, Mul-Instractions SMollnstruet Cheml J MRRnch, SMelllsstract MMench,, SMollnstract
50
IU Molecular Fonmala
50
SMILES Molecular
50
IUPAC to SMILES Conversion
Chenh.LMeench, SMollistract
50
SMILES to JUPAC Conversion
ChenLLLBennch, SMollastract
50
SMILES SELPIES and SELFIES SMILES Translation
Chemistry expert (Ours)
50
Property Classification
Chemistry expert (Ours), LMBnn.. Chemistry expert (Ours), ChemLLMBeuchh Mel- Instructions, SMollastract
Molecular Property Regression
220
Plyssiencerracal Property Prediction from Molecular Structure
CleniLLMeench, Mol- Mol-Instructions, SMellintruct
50
Substratic Recommenaation
ChemLL Mul-Instructions SMollnstruet
50
Symihetic Pallway Recemmendaaion
Chemistry expert
Symthetic Difficulty Evaluation
Chemistry expert (Ours)
20
Ligand Recommendatiin
ClentLLMBench
50
Reugent Recommctaaiion
50
Salvent Recommendation
Cheml J. M.Rennch
50
Catalyst Recommendazion
Chemistry expest (Ours)
Reaction Temperature Recommendation
Chemistry expest (Ours)
20
Reaction Time Resimmendttion
Chemistry (Ours)
20
Reaction Product Prediction
ChenLL.MBench, Mol-Instructions SMolinstmet
50
Yield Prediction
ChemLLMMeena
Reaction Rate Prediction
Chemistry expert (Ours)
20
Intermediate Derivation
Chemistry expest (Ours)
20
Total
1960
Tables and provide overview of the sources and sizes of text and multimodal tasks included ChemEval. The text subset contains 1,960 test examples, comprising 18 tasks sourced from other open datasets and 24 tasks designed in-house. The multimodal subset contains 1,200 test examples, integrating 12 tasks from other open datasets and 30 tasks independently designed by our team. Some tasks exist in both text and multimodal versions.
Table [6] summarizes the distribution of reaction types in our dataset. Coupling reactions account for the largest proportion, followed by substitution, oxidation/reduccion, addition, climination, and other reaction types. The "other" category covers 9 reaction types, including rearrangement, hy- drolysis, cyclization, and so on. This dataset encompasses most major reaction types, providing comprehensive coverage for evaluating chemical reactions.
Overall, ChemEval integrates 25 tasks sourced from other open datasets and 37 custom-designed tasks developed in-house, with duplicates removed to prevent double counting. This design ensures the benchmark's 's diversity and comprchensiveness, allowing large language models to be evaluated across multiple dimensions of chemical knowledge and reasoning. By rigorously cross-checking against existing model training sets and leveraging previously unpublished laboratory data, we min- imized potential risks of data leakage.
C DETAILED EXPERIMENTAL SETUPS
In this section, we introduce the details of our experimental setups, including the detailed description of the evaluated models and explanations of the metrics used in Section 3.2.5
18

---

Published as a conference paper at ICLR 2026
Table 5: Overview of multimodal tasks in ChemEval.
Multiple Choice Task
Chemistry expert (Ours)
30
Fill in the Blank Task
Chemisty expert (Ours)
20
Statitticll Chart
Chemisrry expert (Ours)
30
Table
Chemistry expert (Ours)
Reaction Profile Diagram QA
Chemistry expert (Ours)
20
Theoretical Potential Energy Surface QA
Chemisry expert (Ours)
20
Infrrred Spectram QA
Chemistry expert (Ours)
20
Raman Spectrum Q2A
Chemistry expert (Ours)
20
UV-Vis Spectrum QA
Chemisary expert (Ours)
20
Diffraction Partem QA
Chemistry expert (Ours)
20
Kineric Belavior Chart
Chemistry expert (Ours)
Spectram QA
Chemistry expert (Ours)
Short Answer Task
Chemisty expert (Ours)
20
Calculation Task
Chemistr expert (Ours)
30
Molocular Fomaala
Chemistry expest
30
Chemical Equation Recognition
Chemistry expert (Ours)
2D Molocular Structure Revognitio
Chenisity expet (Ours)
60
Synthetic Pathaay Analysis
Chemisrry expert (Ours)
30
Molecular Formala
Chenl M.Rench, SMollostmet
SMILES to Molocular Formula
MM SMollnstruet
20
IUPAC SMILES ES Conversion
ChenLLMBench, SMollnstruet
20
SMII ES to IU/AC Conversion
(heml M.Rench, SMolInstret
50
Property Classiscation
Chemistty eapett (Ours), ChenhLLMBucch SMollnstruct
Molecular Property Regression
Chenistry eapet (Ours), MBenh, Mol- Instructions SMollnstrue
Physicnchemin Property Prediction from Inframed Spectrum
Chemistry expert (Ours)
20
Physicechamical Property Prediction from Raman Spectrum
Chemistry expett (Ours)
20
Plyscchche Property Prediction
Spectrun
Chemistty expett Chemistry expert (Ours)
Physicachemical Property Prediction from Diffraction Patiern
20
sieschecceii Property Prediction from Mass Spectrum
Chenistty eapert (Ours)
20
Phy Physicnchemical Property Prediction from NMR Spoctrum
Chemistry expert (Ours)
20
Substrate Recommendation
ChennIMBennch Mol-Instruciooss SMolInstruct
Synthetic Pathway Recommendaton
Chemistry expert (Ours)
40
Synthetic Difliculty Evaluation
Chenistty eapet (Ours)
20
Ligand Recommendation
Cheml MBBnch
20
Reagent Recommennation
CMennc
Salvent Recommecsstioon
MMench
20
Catalyst Recommendation
Chemistry expert (Ours)
20
Reaction Temperature Recommendation
Chemistry expert (Ours)
20
Reaction Tine Revammendaion
expet
Reaction Product Prediction
Cheml LMBench, Mol-Instructioss SMolInstruct
20
Product Yield Prediction
MBBenhh
20
Intermediate Derivation
Chemistry expert (Ours)
20
C.I MODELS
In order comprehensively assess the scientific capabilities of Large Language Models (LLMs), evaluate several high-performing LLMs that are widely accessible, including general and specialized models. These models are selected represent diverse range of organizations and vary in size.
GPT-4o: GPT-4o is OpenAl's latest flagship multimodal large language model, capable of pro- cessing and generating text, audio, and images through a unified architecture for seamless cross- modal reasoning and interaction. It sets new benchmarks in multilingual, speech, and visual understanding, exhibiting advanced performance with significantly improved speed and efficiency compared to previous models.
OpenAI-ol/o3-minne OpenAl ol and o3-mini are lightweight, cost-effective reasoning models that deliver strong performance in science, mathematics, and programming tasks while offering significantly improved response speed and reliability compared to their predecessors, making them well-suited for rapid, real-world applications.
Table 6: Distribution of reaction types in ChemEval
Reaction Type
Counts
Percentage
Coupling reactions
321
62.4%
Substitution reactions
15.8%
Oxidation//eeductoon reactions
51
9.9%
Addition reactions
24
4.7%
Elimination reactions
13
2.5%
Other reactions
24
4.7%
19

---

Published as a conference paper at ICLR 2026
Table 7: Performance overview of multi-level 0-shot text tasks ChemEval (part 1). Claude3.7T denotes Claude 3.7-Sonnet- Thinking, whereas Claude3. M.7N denotes Claude 3.7-Sonnet.
Litrrdden
Matri
OpmAl-al
LIFE-6
ClasdeA71
ClasseLTNN
Deeperk-13
Qven25-722
Qwen253328
ORjAN 0NRA
72:181
50
01.0
11:01
25.08
07:34
LLM
1P7
08
40. 00
51.90
$3.50
58,00
75:20
67.20
58.40
40.25
SAYOAA
L.M Scine
50
$1.33
2V
51
31.31
18.90
Litratary Underssaading
CNER
FI
30
66.56
65.76
00.21
54.55
04.16
63.85
6161
36.33
F1
labal
73.71
05.76
48.05
1sbat
A:4l
76
81. 87
15.01
74.3 11
81 10
67
83.67
51.00
SelvE
16.01
54.30
90.20
$3.50
85000
70.00
Tine
Fi
05.00
15.00
95.00
85.00
05.00
93.00
95.00
1003
95.89
Phofi:
Annrncy
16.00
bik
CxTEE
F1
05.00
05.50
9600
$2.00
72.00
100.00
190.00
598.00
lafe
MickE
F1
65.10
85.00 60
7000
01.00
50.00
71
65.00
1500
18.69
GLGrs
M Sone
77.00
19:11
Tipe
Acdrocy
91.0
55.00
1999
56.01
51.00
57
52/81
54.01
15:00
baladai
ReaTR
20.00
25.00
33,00
23.00
36.00
21.00
23.00
Molcsler Underisnnnng
MNG
MoNG
$1.58 (785)
49.80 (725))
(899)
33 (30)
43.28 (785)
06050075)
51 19/968)
33.5 (756)
11 (6SS)
MNTiss
TURACTEE
122
D6714
6.7222
0.506
13333
0.539
D.976
04176
3407
XXiiss
BUSAISMILES
Tasiesto
20,55 (429)
(5:72
34.11 (83%)
S1.91185)
3612 (725)
46 (38T)
1550 (759)
50.55 (395)
A0Vinaa
Best Macc
6.00
0.00
6.00
CUM
1.20
0.00
0.00
UNTiss
SMI ESSUIAAC
000
501
75
90
13.01
$25
9.76 (309)
0.724429)
13 (82%)
9.57 (405))
10.5 (G45))
16:4 (110
(6.210028)
11.47 (555))
6.95 CPYR)
MIP
Accensy
72.85
67.50
64.57
$5.00
45.75
48.15
NIMSE
Millew
LIM Soote
1M 90
19.00
7.00
9.50
15.00
90
13 90
30.90
5.90
Jabya
Publlec
LLM Sons
49.33
20.83
UL3
41A8
52.75
37.35
41.13
20.55
aadrr
Sondlll:
NIMSE (oild))
4445 (229)
(150)
-075]
(0FS)
19854
130%)
-0983
0.2000 (3009))
-19%
32
2564
27.43
21.55
37.75
34.56
Sabiile
P1
16:00
10.00
20440
14:00
17.50
22
24J81
$1.00
Calles
R1 NRMSE (ulk))
1200
0000
0,250 (1095)
3276
(1095)
(1006)
02263
(1919)
0.5998
(1095)
0.2078
(1009)
00206-
STA2
(1006)
(1OYR)
RCRA)
TineRee
NRMSE (ulk))
0,2155 (1095)
0.2755 (1005)
24666 (1009)
0 36655 (1009)
0.000
(100S)
0,2291
(103%)
0.35T9 (00%))
0.3822
(1006)
6,2577 (100))
ROP
Fired
10.00
21-33
12.27
11.97
0492
L.2
ROP
Rasched
Chellp
16.76
21
13.01
9.06
7.21
17.12
13.71
1671
3.455
RMA
TMDer
L1M Sooee
20
20 00
$1.90
$1.90
$1.00
$0.90
77.25
79:00
Claude-3.7-Sonnet: Claude 3.7 Sonnet is Anthropic's most advanced hybrid reasoning language model to date, integrating rapid response with deep, stepwise analytical capabilities and offering flexible dual modes for both instant answers and complex multi-stage problem-solving across range of scientific and coding tasks.
Gemini-2.5-pro: Gemini 2.5 Pro is Google DeepMind's latest multimodal large language model that integrates advanced "thinking" mechanisms and hybrid attention architectures, enabling state- of-the-art reasoning, code generation, and long-context understanding across text, image, audio, and video inputs, with support for up to one million tokens in a single context window.
new generation of large language model developed by xAI. It has achieved breakthroughs in key benchmark tests such as mathematical reasoning, scientific logical reason- ing, and code writing. In addition, it supports multimodal interaction and can also access real-time information through the X platform to enhance the timeliness and accuracy of its responses.
Grok3: Grok
DeepSeek-V's V3: DeepSeek-V3 V3 is a powerful 67 I--iiliion-pareee parameetrr -oxture-of-Expert (MoE) lan- guage model developed by DeepSeek, trained on 14.8 trillion tokens with innovations like Multi- head Latent Attention (MLA) and Mokti Prediction (MTP) to achieve state-of-the-ar per- formance in mathematics, coding, and multilingual tasks. It features a 128K context window and efficient inference, with future versions expected to include mudti-modal capabilities.
model based on the DeepSeek- -3--eek--V-Bass architecture. It is trained with reinforcement learning and human feedback to enhance its perfor- mance in complex reasoning tasks such as logical deduction and mathematical groblem-solving while maintaining high safety and reliability.
DeepSeek-RR R1: DeepSeek-RR RI is a
Dwen225-77B144B/22B/22 Qwen 2.5 is a series of advanced large language models developed by Alibaba Cloud, featuring models with parameter sizes ranging from 0.5B to 72B. These models have significantly improved capabilities in areas such as coding, mathematics, and multilingual support, and they are trained on large-scale dataset of up to 18 trillion tokens
LLaMA3.3-8B Meta Llama 3 8B is a powerful large language model with 8 billion parameters, optimized for dialogue and text generation. It is trained on over 15 trillion tokens and features a 1 128K token vocabulary and Grouped-Query Attention for enhanced performance.
20

---

Published as a conference paper at ICLR 2026
Table 8: Performance overview of multi-level 0-shot text tasks ChemEval (Part 2).
Denemske
Qwed5-144
Quee25/78 1J
Grbk
Grnin-15PP
CheslEM
1as8S4
11M Soer
15
14:11
11 11
51.33
43.85
3136
5:40
13:33
31.17
41.14
Accancy
77.89
SATea
Soure
53.20
50300
73.59
12.33
13.30
14.30
33.00
43,30
SiP04
Sove
$380
11:00
10
$2.41
1330
158)
1.90
08
45.90
LAbIViSV Unvlorsadiyy
1458
ChER
F1
02:33
62.33
04.34
03.33
08.30
11:37
6.16
11.83
71.41
$3.66
183.
35.43
032
13.37
12.23
0.80
1031
14,30
3091
1458:
05.00
0.00
1691
13.31
$2.50
15.00
1838
12400
10
12.00
13.00
07.00
16.33
322
1000
$3.80
60.00
Proff
Accuvry
53.41
00
515
03
3773
0.00
0000
01.00
71 1
1858
73.11
0.80
12.05
ChiT
73.00
54.00
5.00
8300
165
T048
8010
05.90
5100
5000
0LCcs
LANE Soore
33.75
23335
53
39.30
040
375
31.35
31.50
52.50
Axcecyy
15.00
00
25.00
31.30
13.00
5.00
5.0
$7.00
3.00
Malectaar Undernesdyyy
MrNG
(508)
(27(075)
(01/975)
(156/9151
571 (T1269)
518 (689)
1000 SWP:
MXTha
L2
0.3126
011856
0.285
60710
63392
RATIS
D1424
6,000
MNTi
INNACTIVIERS RS
(581/905
41 (5155)
56 (35))
4.0 11/9653
3T.6 (0005)
250 (609)
MNTra
Ban Mtat.
022
000
123
CX0
1001
34.8
SME
BLZL
0.22
1338
0.35
SVI ENCESSRRA PAC
5.66
3.2
X71
50.93
100
410
320
66.36
350
MIP
MERC MARPE
Accency
6422
54.5
47.5
$5.00
45.5
01.35
0.00
40.55
85,57
66,00
Killoe
LLM Soee
7,30
$4.00
2.10
25.00
0/00
0.30
DUX
45.98
21.00
Answinder Dedaviion
Redyy
1.40
0177
0027
D10
D00
12.33
3.0
R:din
Pusske
LLNE Soee
12.03
27:13
23.88
33.13
43.75
36.15
HUSS
23.00
36.15
48.75
1880
6:93
2.01
215
15.00
810
3690
000
37.68
15:00
2X0
15.00
25.00
345
24.00
D30
12:00
Caileo
0:30
0.30
0000
010
D10
6,40
3.0
11000
0,222
BERc
Tiniile
NRMSE (old)
11202 (110%)
(104))
(091
(160R)
02425
(2005)
D4T33 (205)
1700 (8%)
04351 (8XS)
(DDR)
0,520 (72))
R3F
Yhhll
31.00
15.50
5/0
17.30
2/00
25.00
72.00
24.00
DCP
Onerlp
33324
0.00
8.77
27:81
3.79
5.0
3
2:00
Sovee
63.75
28:25
VL Max: Qwen-VL-Max is the most capable large visual language model in the Qwen-VL. LL series, offering optimal performance on a broad range of complex tasks. It has significantly en- hanced visual reasoning and mnslluction-oolloiing abilities, and can handle high-definition images with resolutions above one million pixels.
Phi-3.5-vision is lightweight, late-of-the-aatt open multimodal model developed by Microsoft, with 4.2B parameters and a 128K context length. It excels in handling both text and visual inputs, offering capabilities in general image understanding, optical character recognition, chart interpretation, and video summarization.
Phi- Vision-3.5;
ChemDFM: ChemDFM is a pioneering large language model (LLM) specifically designed for chemistry, trained 34 billion tokens from chemical literature and textbooks and fine-tuned using 2.7 million instructions. It demonstrates superior performance in various chemical tasks such as molecule recognition, molecular property prediction, and reaction analysis, significantly outperforming most representative open-source LLMs.
LlaSMol: LlaSMol is series of large language models fine-tuned on a large-scale, comprehen- sive, and high-quality instruction tuning dataset named SMolInstruct for chemistry tasks. These models, based on open-source LLMs like Galactica, Llama 2, Code Llama, and Mistral, demon- strate strong performance on various chemistry tasks, significantly outperforming previous LLMs and approaching the performance of state-of- oottte--fthheeeee of-thh-art task-specift models. We select the Mistral- based version for experiments due to its superior performance.
ChemLLM: ChemLLM is the first specialized large language model dedicated to chemistry, trained on unique dataset ChemData, and evaluated comprehensive benchmark ChemBench. This model shows remarkable capabilities in handling various chemistry tasks and exhibits strong general language skills.
ChemSpark: ChemSpark, formally known as Spark- Chemhhtrh-XX -1-ittyy-XX-1BBB,, LLM specialized for chemistry developed by iFLYTEK and released on the ModelScope platform. It was created by fine-tuning the iFLYTEK Spark-XI base model on various chemical task datasets.
21

---

Published as a conference paper at ICLR 2026
Table 9: Performance overview of multimodal tasks ChemEval.
Dimension
Task
Metric
GLM-4V
GPT-4o
Claude3.77
Qwen-vl-max
Phi-vision-35
Gemini-225-PP
Advanced Knowledge Question Answering
MCTTak
32.22
86
7.78
41.33
35.56
45.55
ObiQA
FBTask
Accuracy
52.41
17.77
48.12
15.02
58.80
SubjOA
SCOA
LLM Score
65.33
68.67
30.22
$2.00
44.44
$0.89
SahjQA
STQA
LTM Score
64.22
54.22
32.67
72.22
32.67
76.22
SubjQA
RPDQA
Score
50.67
20,00
70.67
37.67
70.00
SubjQA
TPESQA
LLM Score
62.33
69.33
21.67
76.33
45.67
70.67
SahjQA
IRSQA
LLM Score
53.33
59.00
35.33
62.33
42,00
66.33
SubjQA
RSQA
Score
70.00
35,67
71.33
51.33
76.00
SuhjQA
UVSQQ
LLM Score
62.67
62.67
33.33
66.00
48,00
69.31
SubjQA
SubjQA
KBCQA
LLM Score
68.33
77.00
48.67
81.67
51.00
79.33
SuhjQA
MSQA
LLM Score
66.33
74.40
22.00
83.67
46.33
72.00
SubjQA
SATask
46.67
46.33
SubjQA
CaleTask
LLM Score
49.11
60.67
51.78
62.00
36.89
79.78
Literature Understanding
MNR
MHR
Accurncy
100.00
95. 56
2.22
100.00
83.55
84.45
MNR
CRER
Accuracy
95,56
93.34
3.33
93.33
15.56
42.22
Tanimoto
MNR
PaihA
F1
0.00
0.00
0.00
0.00
0.00
Molecular Lnnderstanding
MNTrans
IUPAC2MP
L2
0,3048
0,5653
0.2106
0.1175
0.1690
0,5892
MNTras
SMILES2ME
1.2
0.1251
0.2144
0.0468
0.1367
0.1018
0.4951
MNTrans
IUPACZSMILES
Tanimoto
44.43
11,90
24.63
4.37
77.19
MNTrans
SMILESZIPPAC
Exact
0.00
0.00
0.00
0.00
0.00
2,00
MNTrans
PAC
BLEU
23.15
19.04
22.81
24.44
26.19
18.47
MNTrans
SMILESZUPPAC
Tanimoto
1.73
2.09
8,88
0.74
1,22
MPP
MclPC
Accuracy
50.51
49.70
54.67
58.32
53.75
62. 08
MPP
MclPR
NRMSE (valid)
2.3782 (57%))
1.0268. (71%)
0.3491
(29%)
179879
(100%)
3,0580
(435))
16. 1085 (100%)
Molless
IRS2PC
Score
58.00
66.33
45 55.00
60:67
MolDese
RS2PC
LLM Scvee
44.00
$1.67
63,00
$7.67
38.33
$5.53
MolDesc
UV2PC
LLM Score
$4.67
59.67
65.67
63.00
40.67
67.00
MolDese
DP2PC
LLM Score
58.33
65.00
74,00
69.00
41.33
69.33
MolDesc
MS2PC
LLM Score
54.33
61.67
75.33
67.00
38.67
69.00
MollJese
NMR2PC
37.67
Scientific Knowledge Deduction
RcSyn
ReSyn
PathRee
LLM Score
45.00
$7.00
67.00
54,67
31.67
61.67
Rc5yn
SyaDE
NRMSE
0-1220
0,3199
0,5575
0.2234
0,5437
RCRec
RRee
F1
0.00)
5.00
5.00
6.67
6.67
8.33
RCRee
SolvRee
FI
15.00
23.33
21,67
30,00
18.33
28.33
RCRec
TempRee
NRMSE
0.1220
0.4845
0.3913
0,5346
0.1777
RCRee
TimeRee
NRMSE
0.4378
1.67
ROP
YPred
Accurncy
43.33
20.00
25.00
78.33
31.67
RMA
IMPied
LLM Score
67.67
71.33
76.67
62.33
35.00
77.67
C.2 METRICS
In this study, employ variety of evaluation metrics fine-grained assess model performance across different tasks. The main metrics include:
F1 Score and Accuracy: These are the primary metrics used for most tasks. The F1 score combines precision and recall evaluate classification performance, while accuracy measures the proportion of correct predictions.
BLEU: Calculated by comparing the n n-gram overlap between the gedeerted text and the reference answer, incorporating brevity penalty to penalize overly short outputs. This metric is mainly used to assess the similarity between generated results and reference answers.
Exact Match: This metric checks whether the model output exactly matches the ground truth.
Normalized Root Mean Square Error (NRMSE); Used to evaluate the prediction emror in nu- merical or regression tasks, and lower values indicate better model performance.
Valid Output Ratio: The proportion of valid outputs provided by the model.
LLM Score:
Subjective evaluation other large language models, focusing on the reasonable-
ness and completeness the answers.
L2 Score (L2): An indicator for evaluating the similarity between molecular formulas. Specifi- cally, 1.2 Score is calculated 1/(1 1.2 distance), where the 1.2 distance refers the 1.2 norm
22

---

Published as a conference paper at ICLR 2026
Table 10: Performance overview of multi-level 3-shot text tasks on ChemEval (part 1). Claude3.7T denotes Claude 3.7-Sonnet- Thinking, whereas Claude3.71 7N denotes Claude 3.7-Sonnet.
Literadion
Mastrcc
UpenSl-nSentii
Upenttel
GPI-hh
ClasdeLTV
Clandod75
Deepush-Rt
Hhupoedk-13
Qpm225-200
Qven2
Arcrmay
63:181
10
60:00
LLM
45.99
ONjAA
TPlnk
Acczzoo
76.00
$6.00
$7.60
12.43
$8.00
72.50
40.40
59.59
LLM Scite
$3.00
$5.50
10
53000
71.40
13.00
001.0 50
55.90
Litrntr Undervsaneiigg
lant
CAER
F1
00.33
10.39
71.14
06.21
14.13
1025
03.35
05.32
15.43
23.11
78.1
F1
85.33
03.67
14117
0761
68.57
98.87
88 8380
$1.81
Sahit:
61101
61101
18100
10
3167
61103
75.00
K1:00
hik
TinsE
F1
95.00
95.00
95.00
05.00
95.09
95.00
95.00
05.80
95.00
PoalE
13.00
01.45
05.55
51334
01.35
01
06-55
11.38
bbbll
HAR
YIME
F1
00.00
00.00
12.00
57.00
5600
50.00
00.00
56.00
19,80
Ascerao
Unfertueiing
MNG
MiNG
1E (28)))
11.57 (W7S)
(1.491775)
(M06)
15:99905
25 (776))
(T96))
KNinas
SMILESIME
L2
IL5833
0.5942
52558
0,223
0.355
BJ456
DJ653
0.8983
0.228
30iina
BUPACSSMLLSS
Taaasso hasd:
$1.51 (525))
5853-5291
31.71 009)
25:35 (509)
4000 135%)
534905393
40.50 (509)
15.73 (609))
00.88 (SA5)
MNTos
AMB ESTURPA
3.44
4.59
419
4:31
7.53
1000
0.11
SINTOOS
SMI PSTTIPA
Tasirven
0.30
8.0
17.60
1763
2101
34.76
17.96
1304
7.07
18.24
12.06 (225)
1885)
(359)
21.36 (628)
18.00
33.05
71.00
$9.71
MPP
MUPP
0.55E (1/00)
(77566(0009)
0. 4115 (895)
1964 (010)
5.001 (RER)
0.7807
(100N)
0.795 (995)
0 11779 (ORR)
O. 7900 (1OOS)
Malline
MaPPY
TIM Scare
90
11.30
8.10
C.00
0.30
Scovnge Enewiidgr Dedovraa
Suckec
FI
2,7
300
216
2033
1.36
CJAT
Patelee
LLM Scare
$1.50
40A5
LO
20.25
28:75
21.15
2400
41.13
Smill
NAMSE (eald)
IL3836 (100%)
(000%)
04456 (107))
(100S)
(300%)
(000%)
0.0227
(199))
(BCOR))
1560
5.50
RCRA)
F1
45.00
41:07
30:33
33,55
38.51
37 26
25.17
Salvlles
45.01
2530
45000
10.00
$1.28
MLHa:
Templac
NRMSE (oald)
IL4551 (120%)
04127 (022R)
(1075))
0.5045 (JAMS))
CL4425 (S00P))
IL4341 (000%)
0.31%6
(1035)
(IMPS)
0.2261 (200%))
RERA
Tiasfee
NRMSE
0,2071 (1909)
19900
(009))
0,2168
(100S)
1913 (1005)
(1005)
(990010009)
0,3055
(107))
0.087 (1O05)
(O0%)
ROP
Yind
Acczray
34.01
34101
50.0
T5,00
$2.50
40.50
40300
91.50
18:00
ROP
kacheed
Ovecke
15.34
1441
21
12:37
15.82
29.24
15:45
15:42
15:40
between the predicted and reference molecular formulas. A higher value indicates greater similar- ity between formulas.
Used to assess the proximity between the predicted range and the reference range. is calculated the length of the intersection divided by the length of the union of the predicted and reference ranges.
Overlap:
D FULL PERFORMANCE RESULTS
D. PERFORMANCE RESULT OF O-SHOT SETTINGS
The table 7 and the table S show the complete experiment results of all models under the zero- shot setting. We tested all the aforementioned models under zero-shot settings ChemEval, as analyzed in Section 4.2.1 The results demonstrate that general-purpose models perform relatively well on knowledge question answering and literature comprehension tasks, while specialized models excel in more complex chemical tasks such as molecular property prediction. For certain tasks like CatRec, most models struggled to generate valid outputs, resulting in scores of zero.
D.2 PERFORMANCE RESULT OF MULTIMODAL TASKS
The 9 the performance of mainstream multimodal large language models on ChemEval's multimodal tasks, with *_' indicating meaningless responses. While most models handle basic tasks like molecular formula identification adequately, they struggle significantly with more com- plex challenges involving chemical reaction pathways and molecular properties. This performance gap widens further in Molecular Understanding and Scientific Reasoning tasks, which require both accurate molecular structure recognition from visual inputs and comprehensive chemical knowl- edge application. Our evaluation focused solely on general-purpost multimodal models, excluding tnemmistyy-specine ones. As multimodal capabilities become increasingly essential in chemical re- search, this represents critical area requiring urgent development.
23

---

Published as a conference paper at ICLR 2026
Table 11: Performance overview of multi-level 3-shot text tasks on ChemEval (part 2).
Adhatcal Anys indgr (nnattion Anrwering
ERIDA
MC T0:TH4
01:30
$5.60
35.50 (B
30.92
W1.50
01:30
11:00
1.93
13.01
FS7akk
Sone
55:40
SATak
LLM Sone
$2.20
48.70
27
13,00
T0.50
30.50
11.50
23. 50
31,60
S45-04
L.M Sone
$1.10
40.80
15.20
15.30
$1.60
16.40
15:46
05.53
57
$5.90
Licrobere Enleritaniing
51.35
lafeE
CBRC
FI
26.99
15.42
15.56
18.05
20.50
0.33
0100
35.18
lake
Suss
02.00
08.21
53.71
70.43
76.30
21144
01.00
000
71.90
lifd:
Solvt
11.00
$1.00
56.57
0.85
85.23
Tregfi
FI
10.00
79400
09.00
75/00
7730
1.55
0.03
$0.00
Thnd:
19:00
15.00
95.00
78.88
01.08
0100
65.00
bikE
CluME
70.25
62.55
32
75.35
50.09
12.53
0.50
0.0
39:12
lake
CATE
82.00
78.00
71.00
100.00
40.00
44.00
0.00
0.00
25.00
lafe
0.00
0.00
25.00
IadacSex
RaxfTR
4500
40.00
31.00
79:00
59.00
25.00
0.00
0.00
32,00
Melennter Unferwwiing
30NGss
Tastett (ruld))
(558)
4.33 (36%)
(24%)
40.36 192%)
72.3 (128)
34.290074) (074)
(96)
000.009)
61.35 (35%)
IUBYCIME AMILESTMP
0.1719
0,00C
MNNTow
D3333
0.190
0,1631
0.7581
066
B/5555
0.072
IL7134
IUPAESSII RX
Tieinren (wlld)
7.07 (085)
(908)
(145))
(915)
59 (175)
18.04 (983)
0.00 (95)
00030751
K3 (S95)
AINTruu
SMIL ESSHIPPTT
BLEU
0.15
D13
0.06
45.96
SINT
SWIl DISHLIPPT
Thimen
7.00
7:M
141
7KA1
SEAT
1.07
0100
0
$1J0N
Tasnecto
12,19 (114))
3.51 (4795)
27.56 (879))
2011
(256))
(96)
0001250
T966
50.37
0.00
33,08
MPP
M4PP
NRMSE (nil)))
1,757 (600%))
0 0.5915 (1009)
50.655 (81R)
0.2556 (1009)
0.2213 (100%)
L6438 (679))
8.2422
10,0340 (8PR)
1.1534 (L00%))
MelDew
MoZPC
LLM Sone
2.40
1.00
0.45
24.40
2.50
0000
0.00
9.50
66.20
Saisigle Knew fndge Dellation
0.20
0.00
Rclnn
Pabiles
LLM Sone
28.25
23.59
17.88
15.25
43.00
28.55
6.75
17.50
27,00
Relan
SaaDE
NRMSE (silll
0.3233 (200%)
0.7744 (1009)
0.7969-1099)
0.2716
0,2716
0.4244 (100))
0LEDAS (519))
0.5266
(.526110099)
0.467 (55%)
0,59G5 (06R)
BrTie:
41.04
70.78
9100
5.0
DV3
$7.45
RC3t
F1
1E:00
11:00
30.90
0.01
24.9
0100
33.01
BT3tc
Cattee
1887
13
0.34
1.1
3.90
D.00
RCRks
Tindlle
NEMGE
0,2053 (1004))
0
(1009)
0,9478
(1939)
0,1944
(1009)
0,3090
(1095)
C,193
0.4404
(1009)
0,3055
(100R)
044221 (1 (1.0%))
ROP
Phid
0.00
0.40
0,00
10.97
30,00
11.8
0.00
0.00
$3
0.00
Rachhll
14.29
RMA
LLM Sone
74.25
25.25
07.50
$0.50
$3.00
42.25
4.75
3.75
73.25
Table 12: Standard deviation across five trials for different models on ChemEval. Claude3.77 7T de- notes Claude 3.7-Sonnet- Thinking, whereas Claude3.71 7N denotes Claude 3.7-Sonnet.
SATask
CaldTask I.I.M Soore
UNER F1
CERC
S2S Tanimoto
LRoc
PPred
Metric
Accuncy
Accuracy
GPT-40
61.20
65:76
1.58
25.66
1.45
13.41
64.57
1.23
13,20
2.59
1.52
cladk3.77
56.70 1: 1.81
55.74 1 2.2
60.21
1 2.02
25.19
1.91
82
39
1 2.53
9.37
1 0.78
58,90
:: 1.96
2.00 - 1.26
12.27
: 4,71
daadde3.7N
55.10 2.18
53.60
+ 2:15
$4.55
4.02
24.77
1.18
85.04
: 1.88
10.58
1.14
$4.37
+ 3.24
4,40 = 1.50
16.16-
: 1.89
Deepsoek-Ri
68.50
27.18
0.35
6.80
V3
71.70 + 1.91
79.20
2.4
60.85
+ 1.13
24.94
1.12
87.52
+ 2.56
16.27
+ 1.44
48.73
1.33
7.60
1:00=2.0
0.93
1.14
Owen2.5-72B
58.50 1: 2.24
61:90 1: 2.08
61.61
1: 0.81
26.05
0.84
84.56
2 1.15
11.47
4
1.17
48.13
1 0.65
4,40
1.50
1.73 1.50
1Lama3.3-88
38.40
28.00
55.34
3,85
17.31
74.54
1.56
1.74
+ 0.65
47.26
1.86
2.13
1,29
0.00
Grok3
73.59
1.16
81.20
1.60
60.75
0,34
26.04
0.61
91.04
0.28
17.56
1:56::1..5
56.62
36.00
11.33 1.54
Gemini-2,5.4 P.5-Pr
72.00
1.41
82.40
+ 0.7
68.30
0.99
25.43
1.63
0282
+ 1.92
1:01 13.15
63.63 + 1.10
0.00
- 0.00
79:20
6.01
ChemDFM
32.20
1,57
14.70
1 1.17
41.17
1 2.25
8.74
2 2.52
31.73
1 2.9
2.12
0.31
61.35
2
26.00
1..00::: 3.79
18.50
2.29
ChemLLM
0.16
0.12
0.00
0.00
0.00
0.00
ChemSpark
33.60 0.97
18.50
1 2.02
71.44
4:
1.13
39.27
2.59
94.40
1 0.23
$7.36
1.46
85.57
2: 2.19
37.60
0.80
56,40
1: 3.44
D.3 PERFORMANCE RESULT OF 3-SHOT SETTING
As shown in the table 10 and the table II we evaluated all the aforementioned models under 3- shot settings on ChemEval. The results indicate that, similar the zero-shot scenario, general- purpose models perform relatively well on advanced knowledge question answering and literature understanding tasks, while struggling with more complex molecular understanding and scientific knowledge deduction tasks. Specialized models such as ChemLLM and LlaSMol, due to their poor eestuction-followie capabilities, failed return meaningful responses for most tasks, resulting in anomalous scores. These findings corroborate our previous analysis.
E RESULTS OF ANALYSIS EXPERIMENTS
We conducted experimental analyses in two key areas. First, to establish the reliability of ChemEval metrics and demonstrate our evaluation framework's 's robustness, we conducted three repeated trials across identical task categories and calculated the standard deviation of results. Due to computa- tional resource limitations, we were unable to conduct comprehensive experiments on all models and tasks. Therefore, we selected representative models and tasks for evaluation. Second, we investi-
24

---

Published as a conference paper at ICLR 2026
Table 13: Experimental Results on COT and Format Constraints.
Dimension
Tisk
Metric
ChemDEM--Nooorat
ChemDFM-CuT
ChemLLmm-NoFurmal Llaemo-NoFerrmat
Qwen2.5-7B-Col
Advanesd Knowledge Question Ascraszigg
OHJQA
MCTba
Accurncy
36,00 [5 30
32,00 [9 0.200
18.00 13.60
24.00 00
50,00 (9
O9IOA
197ck
LLM Scoce
24,00 10.16
25.38 11:23
31.58 13.39
20.88 16.96
27.64 [11.88
O9IOA
TPissk
46,00
32,00 114
16.00 13.20
56.00 12,00
70,00 114.30
Sub/UA
SATask
Seoce
44,80 112.60
44. 112.20
32.40 179.20
30,00
SobiQ4
CaleTack
50,00 117.30
30.40
32:40 116.50
22.00
Literatire Understonding
InfoR
CNER
FI
41.44 12:27
$7.95 [3.19
47.61 147.45
100. J0.62
67,02 15.75
Infol
CLRC
Fl
11.53 12:79
9.69 10.95
16.81 116.57
4.13 72.89
22,89 13:21
IuloE
SILE:
Accouncy
0.00 220.07
0.00 (20.07
0.00
01.00
0.00 LSX 43
InfoF
AddE
F1
1113611166
46.67 11.67
66.67 166.67
36.67 -36.67
65,33 13.66
IntoE
SolvE
65.00 115.50
60,00
76.50 174.83
0.00
78.33 14.17
InfoE
TompF
60,00 114.33
[4.33
70.00 166-77
40.00 -40000
00
112.00
50.00 '25.00
InteE.
Accuracy
0.80
0.61 134.12
74.13
26.51 150.49
InfoF
ChaMM
74,96 147.70
64.52
65.00
165.00
96 -44.06
65,
Infol
CallE
F1
35,00 114.00
40,00 19.00
45,00 145.00
0.00 5,00
55,00 19.00
Into
YiekiE
FI
(4.000 115.00
60.00 115.00
55,00
155.00
55. 00
-50.00
50,00 117.00
Induclian
AbsGien
LLM Score
20.00 120.50
20.00 120.00
20:00
134.50
11.00 115.25
73.00 118:25
Induiii
Ol Gen
1LM Soor
19.00 19.00119.00
18.00 118.00
40.00
136.25
25.00 16.25
58.00 133001300000
IndurGen
TopC
Accuracy
30,00 121.00
15.00166.00
35.00
[35,00
20.00 220.00
45,00 14.00
InduxGian
ReaeTR
FI
25,00 112.01122.0
15,00 12.00
30,00
130.00
0.00 55.00
20,00 111.00
Molecular Understanding
MNGGen
Taninoto (vaidd)
(945))
(92%)
(2%)
(3.7)
MNTrans
IUPAC2M
21. [400
6:99 12:45
1.00
9.93 [8.63
MNTrans
SMILES2MP
61.27 12.72
17.14 156.85
4.23 10.18
0.00
3,96 15.84
MNTrans
IL ILPACSMIIES
Taninato (valk))
50.37 (1668) 13.66
44.77 (849%) 11.94
(0%) (4.70
3.23 (299)) J0.23
MNTrans
S2S
Tanimote (valel)
0.14 (50%) 11:98
3.53 (46%) 11.41
2 (49)) 12.00
(09) 10.60
(25%) [0.20
MPP
MelPC
Accuracy
63,68 12:33
57.12 [4.23
45,36 145.36
54.92 18.42
45.60 118:45
MPP
MclPR
NRMSE
11.88 1363.07
240.91 1154.05
0.56 1178.178.80
12.19-17.7
46 96 [58 39
MolDesc
Mel2PC
LLM Score
28,80 [25
28.00 30.00 124
20.40 120.10
25.60 225.60
30,40 115.90
Seieoughe Knowerdge Dedocctoo
ReSyn
SubRee
13.09
13.00
0.00
-1,33
0.00
ReSin
Pathieec
LLM Score
48,00 123.98
40.50 116.38
24,00 113.13
30.50 "20.50
47,00 119.88
RCRc
L.Ree
Fl
4,00 00
4,80 (21 20
0.00
0.000
6.80 13.20
RCRec
RRtc
Fl
8.00 15.15
9.33 13.80
22.00 122,00
0.00
44.00 727
RCRRc
SalvRec
Fl
6,00 [4 53
14.00 13-47
8:00 18.00
2.00 11.50
20,00 15.60
RCRe:
Templte
NRMSE (valid)
0,421 (85%%) 10.04
0,2681 (8555) 10.11
0,9821 (4595) 10.14
7.9004 (1555) 17.03
0,3177 (555g)
RCRc:
Timektee
NRMSE (valid)
0,5357 (7095) 10.06
0,6022 (5588) 10.13
1.306 (259g) 10.49
(095)
0.4396 (10099) [0.12
ROP
PPred
F1
4.00
14,00 14.30
0,00
8.00 _8.00
0.00
RUP
YPoed
52.00 (509)
(509) 164.80
70.00 (509))
(50961
(50%) 113,00
RDP
RatePood
Dverlap
RMA
IMDer
Score
57.00.119:00
55,00 121.00
37.00 132.25
32.00 730.50
56,00 (22.75
gated the differential impact of reasoning-oriented and format-constraint instructions in prompts, amining how reasoning capabilities and instruction-following ability influence model performance on complex chemical tasks.
E. BENCHMARK STABILITY ASSESSMENT
The table 12 shows the result of our repeated experiments. The results reveal that standard devia- tions across most metrics remain below 5.0, demonstrating consistent performance across multiple evaluations. This statistical stability confirms the robustness of our evaluation framework, ensuring reliable and reproducible assessments of system performance.
E.2 ANALYSIS OF COT AND FORMAT CONSTRAINTS
As illustrated in Table K we evaluate four models: ChemDFM, ChemLLM, LlasMol, and Qwen2.5-7B under varied prompt configurations. When format restrictions were removed from the prompts, ChemDFM and LlasMol showed improved performance on simpler chemical tasks but declined on more complex ones. In contrast, ChemLLM achieved substantial performance gains across most tasks after the removal of format restrictions. This finding highlights that the loss of instruction- instruction-follooing ability can critically undermine the practical usability of domain-specific models. With respect to ceasoning-orientee instructions, CoT prompting produced inconsistent out- comes for ChemDFM, enhancing performance in certain tasks while reducing it in others. Notably, Qwen2.5-7B consistently exhibited performance deterioration under CoT conditions, suggesting that explicit reasoning mechanisms contribute little to performance improvements chemical tasks.
E.3 ANALYSIS OF CHEMISTRY-SPECIRET LLMS AND GENERAL DENERALL-PURPOS LLMS
Chemistry-speeifs language models are typically developed by fine-tuning open-source founda- tion models with domain-specifie corpora. For instance, CM is derived from InternI.22-
25

---

Published as a conference paper at ICLR 2026
Base-7B, while ChemDFM builds upon LLAMA-13B. Although these models incorporate exten- sive chemistry datasets and supplement them with general-domain data to mitigate catastrophic forgetting, their relatively small parameter scales impose fundamental limitations, particularly in instruction-following ability and task generalization. In contrast, closed-source state-of-the sys- tems such Gemini-2.0 and GPT-4o benefit from both larger parameter counts and massive pre- training corpora that include substantial amounts of open-source chemistry data, enabling them to preserve general reasoning capacity while delivering strong performance on chemistry tasks.
To further investigate the trade-off's between domain specialization and general-purpose robustness, we conducted comparative study on Llasmol and its base model Mistral-7B. As shown in Table [4] the results reveal mixed outcomes: Llasmol achieves modest improvements on classification-bar based tasks such as multiple choice and true/false questions, but performs poorly on information extrac- tion and molecular representation tasks where the base model significantly outperforms it. A closer analysis of the model outputs suggests that Llasmol suffers from weak instruction-followinn followinn, of- ten failing to generate answers in the specified format, as well as tendency to produce irrelevant responses that are misaligned with the posed questions. These findings highlight the limitations of smaller
models and underscore the advantages of large-scale general- purposs LLMs achieving both reliability and relevance in specialized scientific applications.
F CASE STUDY
We conducted a detailed analysis of the models' outputs, systematically categorizing the most com- types of errors they make, and provided two illustrative examples to highlight typical fail- ure cases. These examples demonstrate the models' difficulties in accurately adhering to chemical nomenclature rules, predicting reaction substrates, and correctly interpreting molecular structures, thereby offering concrete insights into their limitations and areas for potential improvement.
In the Advanced Knowledge Question Answering and Literature Understanding levels, the model demonstrates strong proficiency in fundamental chemistry knowledge. Covering the four major branches of chemistry, the model shows some minor inaccuracies in understanding basic conceptual definitions in fill-in-the-llank tasks, resulting in occasional incorrect responses, though the overall error rate remains low. In short-answer tasks, the model provides detailed and accurate responses, reflecting its strengths in foundational chemistry question answering.
In the Molecular Understanding level, which primarily involves organic chemistry, the model ex- hibits errors related to unfamiliarity with nomenclature rules for natural products, heterocycles, and macrocycles, leading to name confusion; insufficient knowledge of special functional groups and substituents; and incomplete understanding of basic organic and stereochemical nomenclature rules, resulting in incorrect compound naming and misattribution of physicochemical properties.
In the Scientific Knowledge Deduction level, the model's 's errors include lack of organic synthesis knowledge, insufficient understanding of organic reaction mechanisms, unfamiliarity with specific named reactions, limited grasp of reaction selectivity and reaction rules in organic compounds, and inaccurate prediction of target product structures.
These findings indicate that while the model performs well on foundational knowledge and basic molecular reasoning, it still has limitations in advanced organic synthesis reasoning, reaction condi- tion recommendation, and product prediction, highlighting areas for further improvement.
Table 14: Comparison of LlaSMol and Mistral-7B performance on partial chemistry tasks.
Tasks
Metric
Llasmol
Mistral-7B
Multiple Choice Task
Accuracy
0.24
0.20
True/False Task
Accuracy
0.58
0.34
Reaction Time Extraction
Fl
0.25
0.90
Reaction Temperature Extraction
Fl
0.00
0.60
IUPAC to Molecular Formula
L2
0.00
0.1142
SMILES Molecular Formula
L22
0.00
0.1076
Intermediate Derivation
LLM score
0.02
0.26
26

---

Published as a conference paper at ICLR 2026
Error case of Substrate Recommendation
Query: Chemical reaction equations are typically expressed in the following form:
reactant1. react 2. reactant........................ .*product........
In this form, each substance (reactant/product) represented using the SMILES notation. Now we will provide with incomplete chemical reaction equation, where the missing part will be represented by _.... The missing parts could one or more substances. Based on the remaining portions of the reaction equation, please infer what the missing part could be. Please only provide the missing part in your response, without any additional content. The Incomplete equation
follows:
>>CCOC (=0) C1 C1c2=C (CCNIC (=0) c1=c0=c0 (C (F) (F) =C1C1) (C1=CC -CC-N1) C-N2.
You must output your prediction. i.c valid SMILES, and follow the output format exactly as follows: "answer": [[ ["valid SMILES 1", "valid SMILES 2"..... ...... don't 't need any explanation, you just need to output your judgment in format.
Groundtruth answer:
C'CC (=0) CuNCCc2-e/NN--02211-----C-0 'O=C (0) C1=CC=CC (C (F) (C1C))
Model answer:
("Mnssee":""CCCC(0=01) (CCNIC (=0) c1=co=cc=C1) (C1=CC=CC CN2", "CIC1=CC=C FF) c=C1"])
The model- reaction substrate lacks the requisite reactive sites, preventing the two reactants from undergoing the reaction to yield the target product. Addi- tionally, the model demonstrates inaccurate understanding of the target product's structural characteristics, and the reaction substrate contains extraneous functional groups.
Error Analysis:
Error case of WAILLES-tO-IPAA Translation
Query: please give the IUPAC name of the molecule given by SMILES:
CC (C) OC1==00C=CCCCFF /c (=C\2/ [сеен] (N (C (=O)) C2=0) CCN3CCNCC3) C4=
CC=C (C=C4) C
You must output your prediction, i.c. valid IUPAC, and follow the output format exactly as follows: "answer": "The answer you judge". don't need any explanation, you just need output your judgment information.
Groundtruth answer:
(4E, 5S) -5 (4-chhoooppheyl)) 4- [hydroxy- (4- 2-tp:ppoooaa--oopp--aaaa 2-ycoyphhyyyll hrterh/yhhh..n.reeeeeeeeaaaarrir 3-di:: 3-dione
Model answer:
("answer": "CCO::/
=N1), "c1C1=CC=C (C
Error analysis: The molecular structure does not contain 4-hydroxypiperidinyl or 2- propenyl moieties, and the designation 1-one is also erroneous. The model exhibits an imadequate understanding IUPAC nomenclature rules for heterocyclic compounds; the heterocycles actually present in this structure are pyrrole and piperazine rings. Furthermore, additional nomenclature errors include: incorrect enumeration of principal functional groups (the structure contains two ketone carbonyls), improper substituent naming, and incomplete stereochemical specification while the structure possesses two stereocenters, the nomen- clature designates only one of them.
27

---

Published as a conference paper at ICLR 2026
G LLM USAGE
In this work, large language models were employed as a general assistive tool to improve the clarity and readability of the paper. Specifically, the models were used polish grammar, punctuation, and phrasing in the text. No LLMs were used to generate original scientific ideas, analyze data, or draw conclusions; all scientific content, experimental design, analysis, and interpretation were entirely performed by the authors.
28


Reviews:
[{"review_id": "KOV3fsUPXm", "reviewer": "Reviewer_DKh7", "summary": "the paper introduces ChemEval, a chemistry-specific evaluation suite designed with 4 progressive levels across 13 dimensions of LLM capabilities. The authors claim to innovatively introduce test sets related to information extraction. multimodal tasks are also included. Finally, tasks have been sourced from the web in openly accessible sites, as well as domain experts in chemistry.", "questions": "Please provide more details and ablations on the LLM used for evaluation for some of the tasks. See weaknesses section for more details and elaborate on how do you plan to move about this. If this is to be truly open and sustainable over time, the evaluations should rely on a specific snapshot of an open-weights model.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and goes in depth into details of analyzing the data, evaluation metrics, and organizing the test-set into tasks, dimensions and levels, which makes the paper and the benchmark very valuable for the field.\nThe paper also gives a very detailed evaluation of several relevant llms such as closed-source state of the art models, as well as chemistry-specific models. The analysis is also in depth and discusses differences between e.g. reasoning and non-reasoning models, open and closed, and chemistry specific and non-specific.", "weaknesses": "Some of the tasks, admittedly due to their complexity, require evaluation with an LLM Score. This score in itself was not evaluated, benchmarked against human baselines, or revised thoroughly. Additionally the LLM used to score was not disclosed, which further hinders transparency and reproducibility.\nGiven these points, it is not clear how the authors plan to make this benchmark endure the test of time. If the model is an open-weights model it would be good to mention it and release the weights as an asset together with the benchmark, so that the community can run these evaluations. If the model is a proprietary one, it is even less clear how to do this, as the continued availability of api-based models is not guaranteed.\nFurthermore, the paper is missing a discussion on why the specific llm used was used. For instance, what is the variance of the results if the LLM is changed? are there any specific differences in behaviour patterns between LLMs as evaluators?\n\nIt would also be great to provide a human baseline, to assess how good are expert chemists, and average humans, at the tasks in this evaluation set.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kl5yjGUSEC", "reviewer": "Reviewer_wAna", "summary": "The paper proposes ChemEval, a hierarchical benchmark to evaluate LLMs’ chemical capabilities. It organizes 62 tasks into four progressive levels (from basic concepts and literature-style QA → extraction and structured understanding → molecular-level reasoning → high-level synthesis/mechanism tasks). The benchmark is mixed-source: some tasks are adapted from existing open datasets (Mol-Instructions, ChemLLM-Bench, SMolInstruct, Llasmol), and 37+ tasks are newly authored/curated by chemistry experts from collaborating universities, including proprietary laboratory records and multimodal materials (tables, spectra, reaction schemes).", "questions": "Since many chemistry LLM applications are tool-augmented, can you report results for at least one LLM+tool pipeline (e.g., LLM → RDKit/OPSIN → answer) on a subset of tasks (name↔structure, condition extraction, reaction-type identification)?\n\nid you consider evaluating a standard chemistry agent (e.g., ChemCrow-style tool-calling) as a reference system?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Compared with prior suites (ChemLLM-Bench, ChemCrow eval, ChemBench-MM, Mol-Instructions), this work (i) unifies text-only and multimodal chemistry tasks in one place, (ii) explicitly stages difficulty into four levels, and (iii) evaluates many models under a comparable protocol. \n\n2. Good task–metric alignment. Tasks that are structure-like or chemistry-like use appropriate metrics (Tanimoto, L2, exact-format scoring), while the knowledge / reading tasks use accuracy-style metrics.", "weaknesses": "1. The core test size is 3,120 items across 62 tasks → that’s ~50 items per task on average, sometimes less. For high-variance LLM outputs (esp. with many format-sensitive IE tasks), this is thin, and you can’t draw very strong conclusions about model scaling, few-shot gains, or thinking-vs-non-thinking from 30–80 examples. Yet the result section does exactly that. ICLR will push back: “you are positioning this as a comprehensive benchmark but the statistical mass is closer to a curated diagnostic suite.” The paper acknowledges the cost of expert construction, but the conclusion section still speaks as if we have a definitive benchmark. That’s overstated relative to the data volume\n\n2. The paper does not adequately describe the annotation pipeline, especially for the scientific knowledge deduction tasks, which appear to be the most challenging and high-value part of ChemEval. In contrast to the relatively straightforward literature-understanding and molecule-level tasks, deduction requires multi-step reasoning, selection of relevant premises, and often domain-specific normalization. Right now, the description of how these instances were constructed, validated, and standardized is too brief to assess reliability. The authors are encouraged to spell out: (i) the source of the raw material (textbook, lab record, expert-written), (ii) how the gold answer was derived from it, (iii) whether multiple annotators or chemistry experts were involved, and (iv) how disagreements were resolved. Without this, it is hard to judge the difficulty and reproducibility of the most important part of the benchmark.\n\n Section 3.2.2–3.2.4 describes a 3-step pipeline (collect → filter → construct Q/A) but when we get to retrosynthesis, condition recommendation, reaction outcome prediction, mechanism analysis (the 13 tasks in Level 4), the paper stays at the narrative level. \n\n3. Although the paper frames ChemEval as addressing scenarios where models must read heterogeneous chemical inputs (text, tables, schemes) and then reason, the experiments only evaluate plain LLM inference (mostly greedy decoding) without any agent-style or chemistry-tool–augmented setups. For this domain, that’s a real gap: in practice, chemistry LLMs are often run with RDKit/OPSIN/reaction-database calls or ChemCrow-style tool chains to normalize structures, check validity, and retrieve reaction conditions. Without at least a small ablation (e.g. LLM-only vs. LLM + RDKit name-to-structure vs. LLM + reaction lookup) on 2–3 tasks, it is hard to tell whether ChemEval can distinguish different modeling paradigms (language-only vs. tool-augmented) or whether it is mainly measuring prompt-following in a chemistry setting. As written, the results mostly re-confirm a known pattern (“frontier general models > small chemistry-tuned models”) but do not show that the benchmark is sensitive to the way models are used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6zwlZEAlTU", "reviewer": "Reviewer_9BLh", "summary": "The authers introduce ChemEval, a hierarchical benchmark for evaluating LLMs on chemical tasks. The framework comprises four progressive levels, 13 capability dimensions and 62 tasks cover a comprehensive chemistry domain task.\nEvaluation of 16 general LLMs (GPT-4o, Claude-3.7, Gemini-2.5, Qwen, LLaMA, DeepSeek) and 4 chemistry-specific LLMs (ChemDFM, ChemLLM, LlaSMol, ChemSpark) reveals that general models excel at literature understanding and instruction-following but struggle with molecular reasoning, while specialized models show the opposite pattern, exhibiting catastrophic forgetting in general capabilities but superior performance in domain-specific tasks.", "questions": "What percentage of samples were rejected during filtering? What constituted \"irrelevant\" data (Figure 2 step b)? For LLM-as-judge metrics (LLM Score), which model was used—GPT-4o? This introduces potential bias where GPT-4o evaluates its own outputs.\nWhy does scaling fail? Table 3 shows Qwen2.5-7B→72B increases MCTask accuracy 59.6%→67.2% (+7.6pp) but decreases MolPC 64.04%→48.13% (-15.9pp). This is bizarre and unexplained. Is this overfitting? Requires error analysis.\nWhy do specialized models collapse on general tasks? The catastrophic forgetting observation (ChemLLM 0.00 on many tasks) is reported but not investigated.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "ChemEval's four-tier progression across 62 tasks with 37 novel dimensions (CharME, CatTE, multimodal spectroscopy/structure recognition) addresses gaps in prior benchmarks while covering real chemical workflows (retrosynthesis, reaction prediction, property estimation).\n\nA comprehensive coverage of data: three-stage pipeline combining 25 open-source datasets with expert-curated materials, explicit deduplication against model training sets (ChemDFM, ChemLLM), and professional annotation validation ensures quality and reduces data leakage risks.\n\nIn this work, 20 models with domain-appropriate metrics (Tanimoto, NRMSE, validity ratios) was evaluated, quantifying the specialization-generalization tradeoff and providing actionable guidance for model development while open-source release enhances reproducibility.", "weaknesses": "Insufficient Justification for Task Selection and Missing Critical Tasks: The paper claims comprehensive coverage but still missing several core chemical AI tasks: Spectroscopy-to-structure elucidation, Safety/toxicity prediction. The 37 custom tasks lack clear selection criteria.\n\nSingle-domain focus: Evaluation is confined to chemistry; claims about LLM capabilities don't transfer to biology, materials science, or physics without cross-domain validation.\n\nMinor:\nSection 3.2.4 mentions \"continuously adjusted instructions based on GPT-4o feedback\" GPT-4o-generated prompts may bias evaluation toward GPT-4o's strengths. Were prompts validated by human experts?\n\nThe paper claims deduplication against ChemDFM/ChemLLM training sets, but what about GPT-4o's training data (cutoff Oct 2023)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qlLEc9bk2O", "reviewer": "Reviewer_Rxp7", "summary": "This work proposes a comprehensive benchmark to evaluate the chemical capabilities of LLMs from multiple dimensions. Their benchmark, ChemEval, includes 4 progressive levels, evaluates 13 dimensions of LLMs’ capabilities, and features 62 distinct chemical tasks. The paper provides a thorough description of the data collection process and the task construction methodology. Finally, multiple advanced LLMs, including general LLMs and Chemical LLMs, are evaluated using ChemEval, and detailed analyses are conducted.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "The paper demonstrates substantial effort in data collection and task construction. I highly recognize its important contribution to establishing a standardized benchmark for evaluating LLMs’ chemical capabilities and to enriching the diversity of evaluation tasks in this area.", "weaknesses": "1. The position of the figures should be organized more properly.\n2. After carefully examining the authors’ anonymously released dataset, I found that the quality of some tasks is concerning. Certain tasks appear to lack chemical validity and, therefore, fail to reflect the model’s true chemical capability accurately. For example:\n    * In the reaction time recommendation task (TimeRec), the reaction time should depend on factors such as catalyst type, reaction temperature, reactant concentration, and target yield. However, the task only provides the reactants and products, making the question essentially unanswerable.\n    * The Synthetic Difficulty Evaluation task (SynDE) asks the model to assign a score to represent the synthetic difficulty of a molecule. The concept of “synthetic difficulty score” itself is human-defined, but the task neither specifies the range or meaning of the score nor clarifies whether a higher score indicates easier or harder synthesis. As a result, the question again becomes ill-posed and unanswerable.\n\n    Issues like these raise serious concerns about the overall quality and validity of the dataset’s tasks. While expanding and diversifying the evaluation of chemical capabilities is indeed very important, the added tasks must at least ensure that they meaningfully and correctly assess a model’s chemical understanding. (In addition, the released dataset is difficult to inspect: the task categorizations are inconsistent with those described in the paper, many entries contain Chinese characters, and the number of samples differs from what is reported. These inconsistencies made it quite challenging to locate and examine data for specific tasks. I hope the authors will address these issues in future updates, as doing so would greatly improve the usability of the dataset.)\n3. It seems that the number of entries in ChemEval is relatively small. For most individual tasks, the dataset contains fewer than 50 samples. It is unclear how such a limited number of instances can provide sufficient representativeness to reliably assess the model’s performance and capabilities on each task.\n4. The conclusion regarding catastrophic forgetting in the paper is not properly reached. To properly evaluate whether catastrophic forgetting occurs, the comparison should at least be made between each chemistry LLM and its corresponding general LLM base model, such as ChemDFM vs. LLaMA, ChemLLM vs. InternLM, and LLasMol vs. Mistral. The fact that these chemical LLMs exhibit weaker general language performance than more advanced general LLMs does not provide valid evidence for the presence or absence of catastrophic forgetting during training.\n5. A completely new model named ChemSpark appears in the paper, but there is no corresponding citation, paper, or webpage reference. I was unable to find any information about this model, so it is unclear how it was trained, whether there was any data leakage, or whether dedicated multi-task finetuning was involved. Without further clarification, the conclusions drawn from comparisons involving this model seem unreliable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)