CatalystBench: A Comprehensive Multi-Task Benchmark for Advancing Language Models in Catalysis Science
Authors: Xueqing Chen
Jian Xu
Ludi Wang
Yang Gao
Huihan Zhu
Yuanchun Zhou
Yi Du
Cheng-Lin Liu
Venue: ICLR 2026 Poster (2026)
Keywords: Scientific Benchmark
AI for Science
Catalyst Design
Large Language Models
Multi-task Learning
Domain Adaptation
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: The discovery of novel catalytic materials is a cornerstone of chemical engineering and sustainable energy, yet it remains a complex, knowledge-intensive process. While Large Language Models (LLMs) have demonstrated remarkable potential in various scientific domains, their application to catalysis is hindered by the lack of specialized, multi-dimensional benchmarks to guide their development and evaluation. To bridge the critical gap, we introduce CatalystBench, a comprehensive and challenging benchmark meticulously constructed from scientific literature and public datasets, specifically designed to assess the capabilities of LLMs in the nuanced domain of catalyst design. The tasks covered by this benchmark dataset encompass the entire closed-loop process of catalyst development, including reading comprehension, experimental analysis and scheme reasoning. Based on this benchmark, we propose a Multi-head Full-task (MFT) domain-specific fine-tuning method that employs coupling task-specific output heads. We systematically compare with other three distinct fine-tuning strategies: Single-Task (ST), Full-Task (FT) and Multi-head Single-Task (MST). The extensive experiments demonstrate that the MFT strategy consistently achieves the most substantial performance improvements across all tasks, underscoring the effectiveness of explicit multi-task architectures in complex scientific reasoning. The resulting CatalystLLM significantly outperforms a wide array of state-of-the-art open-source and closed-source models on CatalystBench. We will publicly release both the CatalystBench benchmark and the CatalystLLM model, providing the community with a robust evaluation framework and a powerful new tool to accelerate AI-driven research in catalytic materials science.

Published as a conference paper at ICLR 2026
CATALYSTBENCH: A COMPREHENSIVE MULTI-TASK BENCHMARK FOR ADVANCING LANGUAGE MODELS IN CATALYSIS SCIENCE
Xueqing Chen 123, Jian Xu Ludi Wang', Yang Gao Huihan Zhu Yuanchun Zhou 1.2, Yi Du 1.2.6.", and Cheng-Lin 2i42.3.4
Computer Network Information Center, Chinese Academy of Sciences, Beijing, China University of Chinese Academy of Sciences, Beijing, China
$Zhongguancun Acadamy, Beijing, China
MMAIS, Institute of Automation, Chinese Academy of Sciences, Beijing, China $CAS Key Laboratory of Nanosystem and Hierarchical Fabrication, National Center for Nanoscience and Technology (NCNST), Beijing, China Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou, China
xqchen&cnic.cm
ABSTRACT
The discovery of novel catalytic materials is cornerstone of chemical engineer- ing and sustainable energy. yet it remains a complex, knowledge-intende integge-ntteniive pro- cess. While Large Language Models (LLMs) have demonstrated remarkable po- tential in various scientific domains, their application to catalysis is hindered by the lack of specialized, multi-dimensionn benchmarks to guide their development and evaluation. To bridge the critical gap, we introduce CatalystBench, a compre- hensive and challenging benchmark meticulously constructed from scientific liter- ature and public datasets, specifically designed to assess the capabilities of LLMs in the nuanced domain of catalyst design. The tasks covered by this benchmark dataset encompass the entire closed-loop process of catalyst development, includ- ing reading comprehension, experimental analysis and scheme reasoning. Based on this benchmark, we propose a Multi-head Full-task (MFT) domain-specific fine-tuning method that employs coupling task-specific output heads. We sys- tematically compare with other three distinct fine-tuning strategies: Single-Task (ST), Full-Task (FT) and Multi-head Single-Task, (MST). The extensive experi- ments demonstrate that the MFT strategy consistently achieves the most substan- tial performance improvements across all tasks, underscoring the effectiveness of explicit multi-task architectures in complex scientific reasoning. The result- ing LMtMlystLLM significantly outperforms a wide array of state-of-the-art open- source and closed-source models CatalystBench. We will publicly release both the CatalystBench benchmark and the CatalystLLM model, providing the com- munity with robust evaluation framework and powerful new tool to accelerate Al-driven research in catalytic materials science.
INTRODUCTION
The advancement of catalysis comerstone of modern science and industry, pivotal achiev- ing sustainable future Swanson et al.) (2025): Fu et al.) (2025); Zhang et al.) (2025b); Song et al. (2025); Ma et al. (2025). In general, the property of catalysts depends on the complex interplay of composition, crystal structure, surface active sites and regulation strategies Zhu et al. (2017); Chen ct al. (200)). Designing new catalysts therefore formidable challenge: 1) Vast candidate space. The combination of multiple chemical elements, possible crystal phases and surface terminations
Yi Du and Cheng-Lin Lin Liu
the corresponding authors.

---

Published as a conference paper at ICLR 2026
leads to astronomical search space, far beyond the reach of exhaustive experimental screening; 2) Separated knowledge sources. High-fidelity theoretical datasets derived from density functional theory (DFT) calculations capture key descriptors like adsorption energies and electronic properties Chanussot et al. (201)) WWinther et al. (211)), while experimental literature documents synthesis conditions, stability and measured catalytic activities Chen al. (2024). These two streams often siloed, with no unified framework linking them for systematic analysis Zhang et al. (2025a). 3) Lack of realistic evaluation frameworks. Although Al methods have been applied materials science, catalysis still lacks a benchmark that reflects the stepwise workflow of catalyst design. The model development remains fragmented and benchmarking across approaches is inconsistent.
Large Language Models (LLMs) have triggered a paradigm shift in "AI for Science" Abramson jtt al. 2024 y; Merchant et al. (2023) |Szymanski et al. (2023), achieving notable success domains such as bioengineering Luo et al. ); Edwards et al. (202 Waisherg et al. (2024); Lamb et al.
Kristiadi et al. (2024) and industrial process opti- mization Yang et al. (2023) Saka et al. (2024). However, their deployment in catalytic materials sci- ence reveals a critical gap Wang et al. 2025b): existing benchmarks rarely reflect the multi-modal multi-stage workflows that characterize real catalyst R&D, where precise numerical regression, cat- egorical decision making and open-ended mechanistic reasoning coexist within a single process. We compare the limitations of current benchmarks in the material field across multiple dimensions and conduct a detailed analysis in Appenddx A.3] Models trained on homogeneous tasks or unified output formats struggle preserve accuracy across this spectrum, often suffering loss-landscape interference between qualitatively different objectives.
(2024), materials discovery Zhang et al. (2024b); Kristaiii
Traditional workflow
Relevant Data
Automated workflow
Relevant Tasks
Knowledge Organization
Information Extraction
Read Scientific Literatures
Scientific Texts
Knowledge Summary
Method Classification
Simulation Experiment
Structured Datasets
Prediction validation
Property Prediction
Analyze Experimental Principles
Experimental Science Reports
Reading Comprehension
Science Comprehension
manual
Multi-task
Experimental Design Scheme
CatalystBench
automatic
Figure 1: Data- model for scientific research processes in the materials field.
To bridge this knowledge gap, we present CatalystBench, the first multi-task benchmark tailored for catalysis science that explicitly unifies high-quality theoretical datasets and curated experimental literature into a structured Q&A format. Beyond aggregating and filtering data, we perform domain- specific curation and transformation that each evaluation task directly corresponds to stage in the catalyst design lifecycle, as shown in Fig T Given that these tasks span text-based reasoning and nu- merical property prediction, we propose a Multi-hese Full-task Fine-tuning (MFT) strategy. It's an architecture adaptation where classification heads, regression heads and the language modeling head are trained in parallel but decoupled from one another. This architectural decoupling is motivated by the distinctive composition of catalyst design tasks, which uniquely combine high-precision numer- ical prediction, categorical judgement and open-ended scientific reasoning within a single workflow. While multi-head architectures have been explored in other areas of multi-task learning, this work constitutes the first systematic empirical validation and ablation in the catalysis science domain. We then evaluate domain domain-adapted CatalystLLM against strong general and scientific- LLM baselines on CatalystBench, finding that it achieves oote-----hee-tt performance across most task categories. This not only validates the effectiveness of benchmark and adaptation strategy, but
2

---

Published as a conference paper at ICLR 2026
also yields insights into which aspects of catalyst-related reasoning remain challenging for current LLMs. We highlight the primary contributions this paper as follows:
We introduce CatalystBench, the first multi-task benchmark for catalysis that integrates theoretical simulation data and experimental literature into a unified and task-oriented for- reflecting the actual catalyst design workflow.
We develop domain-specifie adaptation strategy. The Multi-head Full-task Fine-tuming (MFT) approach addresses the heterogeneous nature of catalyst-related tasks by decoupling numerical prediction and language generation within the model architecture. We system- atically compare four different fine-tuning strategies and demonstrate the superiority of the MFT strategy.
We conduct a comprehensive evaluation of CatalystI LLM against multiple strong baseline models, demonstrating its SOTA performance our proposed benchmark. Our experi- ments and detailed analysis not only validate the effectiveness of our tuning strategies but also yield valuable insights into the current strengths and weaknesses LLMs in catalysis.
2 RELATED WORKS
2.1 BENCHMARKS IN CHEMISTRY AND MATERIAL SCIENCE
The increasing application of LLMs in scientific discovery has led to the emergence of numerous benchmarks in chemistry and materials science. Initial efforts focused on assessing domain knowledge. For instance, ChemBench evaluate the ability of LLMs on a wide range of chemical knowledge by constructing multiple-choice questions derived from textbooks and expert knowl- edg&Zhang et al.] (2024a); Mirza et al.] (2024). Similarly, MaSeQA Zaki et al. (2024) assesses the understanding of core materials science concepts using graduate-level exam questions, while SCIBENCH Wang et al. (2033) tests college-level scientific problem-solving abilities.
More recently, benchmarks have evolved to probe more specialized and complex abilities. Chem- Li (202)) was specifically designed to evaluate the step-by-step chemical reasoning of LLMs. moving beyond simple factual recall. For predictive tasks, LLM4Mat- Bench Rubungo et al. (2025) provides a comprehensive suite for materials property prediction. Furthermore, Mat- Tools Liu et al. (2025) uniquely evaluates the ability of LLMs to interact with and utilize materials science software libraries. However, existing themistry-focuser LLM benchmarks either empha- size theoretical molecular-level understanding or constrained problem solving and rarely integrate the complementary knowledge from experimental scientific literature Guo et al. (2023); Li et al.) 2024): Zhang et al. (2024a); Xic et al. (2024): Chen et al. (2025). This omiission is critical for catalysis, where real catalyst surfaces often deviate substantially from the idcalized models assumed in theoretical calculations Wang et al. (2023a) and design decisions depend heavily on synthesis conditions, stability data and structure-an activityy trends reported experimentally. Table 4 compares representative existing chemistry benchmarks with CatalystBench. CatalystBench addresses these gaps by combining high-fidelity theoretical datasets with curated experimental literature in unified, task-oriented evaluation framework.
2.2 DOMAIN-SPECIIA LLMS FOR CHEMISTRY AND MATERIALS SCIENCE
Beyond evaluating general models, a significant research direction involves creating specialized lan- guage models for chemistry and materials science through domain-specift fine-tuning or continued pre-training. Early efforts in this area often involved BERT-style encoder models (Chithrananda et al. (2020): : Ock et al. (2023); Trewartha et al. (2022): Zhao et al. (2024). More recent studies have shifted toward adaptive adjustments to large-scale generative L.Mss. For example, ChemI_LM Zhang et al. (2024a) is instruction-tumed on large set of templated Q&A pairs to handle diverse chemical tasks conversationally and in the materials domain, DARWIN 1.5 Xie et al. (2024) adopts multi-stage training strategy combining Q&A fine-tuning with multi-task learning internalize complex materials knowledge. Other generative models include ChemFormerr Irwin et al. (2022) for reaction prediction and CrystaLLM [oyttaLLMAAntuees Antunes al. (2024) for generating novel crystal structures.
Recently, several high-impact works further expand the landscape of chemistry LLMs. Llasmol Yu al. (2024) leverages high-quality instruction tuning dataset to significantly enhance chemical
3

---

Published as a conference paper at ICLR 2026
reasoning capabilities. Translation between Molecules and Natural Language Edwards et al. (2022) explores bidirectional mapping between molecular structures and textual descriptions, highlighting LLMs' potential for molecular communication. Instructmol [ao et al. (2023a) integrates multimodal data sources to build versatile, reliable molecular assistant for drug discovery.
However, current domain-specing LLMs typically optimized for single-task formats unified output styles, focusing on either symbolic reasoning or purely structural prediction. In contrast, complex scientific scenarios such as catalyst design require the combined ability to comprehend domain literature, reason mechanistically and perform high-precision numerical prediction within single workflow. The comparison results in Table 5 reveal that most domain-specifie large mod- els show limited improvement in numerical prediction tasks, further highlighting the challenges of current multi-task fine-tuning approaches. CatalystI tackles this challenge through a multi-head architecture explicitly adapted to the mixed-task nature of catalysis, enabling parallel handling of qualitative and quantitative tasks without mutual interference.
THE CATALYSTBENCH BENCHMARK
3.1 OVERVIEW OF CATALYSTBENCI
In order to explore the abilities of LLMs in the field of materials science, concentrate three fundamental capabilities: Understanding, Reasoning and Explaining. Fig I illustrates the data- driven paradigm shift in the research process within the field of catalytic materials. In response to this trend, the CatalystBench dataset combines theoretical simulations with scientific experimental data to construct series of tasks covering the entire process.
3.2 CANN CONSTRUCTION
Under complex conditions
Experimental Data
cutulytin scheme, regulation method and consens. Please questions owrers according to the following coles:
Method Classification
Science Comprehension
ACS Cornity
Clarivate
doocption ieporation
IC' adecthely prectt
clontoulage you banades adlac
Springer
Seeettie
Text mining
1. Key issues catalytic field:
pairs generating
medicuse Compuaic
discasio
teleed Irarter
compared io call--uuryoa
Co-hh with bey for Aile
inarge acs ti-
Callll unilo copran aoadadddy wcthe tki thit
Selentific Livernares
2. Related knowlddge
3. Example:
Oupes Sativaiaa andiaanies
Hupes Sumerec 12 (18))
contyec caivvty
Catalyst
Design
Multi-task
CatalystBench
Adsorption Prediction
Synthesis Extraction
Propety Prediction
Theoretical Data
nfeee Ciner praffocl sected
Iestrakke Prodics Prrscal
Catalyst Hul
Chanical
ou prapace
the cit gaee colayat
Data filtering
Language inter facerfacing
lapas Sianciprall staccalls AcDill
Adcoptioe
(2
site
Eustrcciie: Flocke cnidee
strigy
Public
Adseption
-1.005
PoTT belley (71 PMT PNT
cacur
High-volume data
Figure 2: The process for CatalystBench construction and examples demonstration.
3.2. 1
DATA COLLECTION AND ORGANIZATION
Fig shows the construction process and spe- cific examples of CatalystBench. For theoret-
Material (ME,
Regulation Clssificatio@@MM 2364]
ical simulation data, we select and integrate 8 publicly available catalytic datasets (Appendix Aito quantitatively evaluate the Understand- ing and Reasoning capabilities of LLMs. We filter out key features from descriptors re- lated to catalytic properties in public datasets, such as SMILES strings. We design a set prompt templates to convert these features and
Synthesia Extruction (SE, 6612)
6.2%
24 67%
Advorption Prodiction (AP, 3000)
Temorre
Regulation Scheme Comprehension (RSC, 4307)
d-band Prediction|d-cr 3000)
Formation Prediction (FP, 200%)
Furadaic Efficiency Prediction (FEP, 2148)
4
Figure 3: The statistics of CatalystBench Multi- Task Data.

---

Published as a conference paper at ICLR 2026
attributes into task-specified natural language sentences. This leads to instructions that com- bine inputs and expected outputs. For exam- ple, the instruction of property prediction like: Prrddict the adsorption energy for CO of given SMILES catalyst.' with input 'TissPds4' and our model should give the output -1.60033 eV", which can be converted a numeric value.
Beyond basic understanding and reasoning, we further explore the model's higher-order Explain- ing capabilities, including deep understanding of catalytic regulation schemes documented in the literature and accurate grasp of macro-level trends in specific catalytic fields. In this step, we ex- tract catalyst regulation and synthesis schemes from scientific literature based on existing catalytic knowledge frameworks. We use SciQAG Wan et al. (2024) to guide GPT-4o in interpreting large volume of regulation scheme texts, converting them into high-quality Q&A pairs. Then domain experts perform annotation and filtering. The method details are described in Appendix A.4
3.2.2 CATALYST TASKS
The proportion of these three major categories is illustrated in the Fig 3 We examine these capa- bilities by constructing 8 diverse and broadly acknowledged practical catalytic material tasks. The dataset size for each task in CatalystBench is smaller than that of general benchmarks, reflecting a common constraint in catalytic research: high-fidelin data acquisition is domain-speciff and re- quires substantial resources. To avoid data imbalance across tasks, construct the entire dataset based on the quantity experimental data, prioritizing task breadth to ensure comprehensive cov- erage of the entire process of the domain of catalyst design.
3.3 QUALITY ASSURANCE
To ensure the reliability and reproducibility of CatalystBench,, adopt multi-tiered quality assur- ance workflow across both theoretically simulated datasets and Q&A pairs generated from experi- mental literature. Automated validation identifies and removes invalid or duplicate entries, followed by expert review of a representative subset. For literature-derive cess eliminates context questions, while multi-stage generation improves semantic accuracy and diversity. The complete verification procedures and error samples are provided in Appendix A.5)
Q&A, a rule- based filtering pro-
4 METHODOLOGY
4.1 BASE LANGUAGE MODEL
We compare the performance the current open-source models on 3 representative tasks of Catalys(Bench-full. From Table 5 ChemLLM Zhang et al. (204a) has bridged this gap through domain-specific instruction fine-tuning on ChemData, infusing the model with a solid chemical foundation. Therefore, choose ChemLLM-7B as the core base model for multi-task fine-tuning.
4.2 FINE-TUNING STRATEGIES
Our goal is to design fine-tuning architecture that simultaneously harnesses cross-task synergies and avoids performance degradation due to task heterogeneity. Fig 44| illustrates the 4 paradigms investigated, with the Multi-heal (ull-task(MFT) strategy as proposed solution.
1) Single-task: We fine-tune the LLMs using a training set of each specific task, which allows us to assess the models' ability adapt to individual tasks and establish baseline for fair comparison. We obtain a fine-tuned model for each task in this setup, named "ChemLLM-STT.
2) Multi-head Single-task: Considering the inherent differences in the attributes of different tasks, we adopt a shared LLM backbone and three task-specific prediction head structures. We obtain a fine-tuned model for each task in this setup, named CCemLLLMMMST.
3) Full-task: we fine-tune the LLMs using a mixture of all training datasets from 8 tasks, as shown in Fig 4b The single instruction-followiin generative head is trained jointly all 8 CatalystBench
5

---

Published as a conference paper at ICLR 2026
Appac Lape
"Ineincction + Question"
bypulape
"Instruction + Question [Task_ [Task_type]
Mi-Chot
Mals-Camn ChanDiaa
Cheml.1M Backbone
ChanlLL
Chem!
Mutioss Capletsttas
Exger
in load
Oaper Cerr
OuperAgw
The Veloo
CataeLLLM
Text
Accuracy
Higher
Regrasoss
CloFLEM-MFF
ChenLLM-FT:: Full-aok Enc-tuning matled
ChemLLMMMPT Malti head Full-takk fine tining method
(a) Overview of Cata-
(b) Comparison of model architectures ChenLLM-MFT method.
between ChemLLM-FT and
lystLLM.
Figure 4: Overview fine-tuning strategies and final CatalystLLM
tasks. The single instruction follows the generation header (original (m_head) during joint training across all 8 CatalystBench tasks, with all task outputs routed to unified generation decoder. We obtain one fine-tuned model to perform all the tasks in this setup. named 'ChemLLL-FFT.
4) Multi-head Full-task: The MFT method extends FT method by decoupling the output space. As shown in Fig 4b] the task type token 7 appended to the end of the input sequence, where t directly controls which dedicated prediction head is activated after the final hidden representation h bn--)]- Here, f--1 denotes the last non-padding token position in the encoded input, while h(6,-1) represents the corresponding context embedding generated by the shared ChemLLM backbone Transformer network. For classification tasks, h/m--] is fed into task-specific MLP classification head, produc- ing a probability distribution is processed by a regression head optimized for mean squared error, generating a single continuous value. In the original generation task, decoding proceeds directly via the original Im head without architectural modifications, enabling the model to generate domain-specific explanatory text.
predefined categories. For regression tasks, the same h hi,-1] input
By training the shared backbone jointly across all tasks while isolating output modalities, MFT method alleviates the interference between loss landscapes of qualitatively different tasks. In prac- tice, we balance their contributions via weighted composite loss and compare the impact of loss- function in Appendix B.6) This approach leverages cross-domain feature learning that is critical in catalytic material research, while preserving the precision of task specific inference.
EXPERIMENTAL SETUP
5.1 BASELINE MODELS
For the sake of fairness, We select 15% of the data for each task in CatalystBench as the test set, according to catalyst material type. Then we compare the performance of different LLMs on CatalystBench-test, including open source models of comparable scale, such as LLaMA-2 Touvron al. (2023). Mistral Jiang et al.] (202)). ChatGLM GL.M et al. (2024) and Qwen3-8B Bai et al. (2023). , as well as closed-source models with strong instruction-followiii capabilities, such as GPT- 3.5 Ye et al. (2023), GPT-4 Achiam et al. (2023) and deepseek. Additionally, for regression tasks involving the prediction of material properties, we introduce comparisons with competitive ML algorithms. We select some SOTA baseline models, such as CatBERTa Ock et al. (2023) and GAP- CatBERTa Ock et al. (2024) from OC20 dataset and ML algorithm such as GPTchem Tttchhmmalloo... Jablonka et al.
2024). For certain tasks, we directly use the ML results from the original dataset papers Gao et al.
2023
We provide detailed introduction of these baseline models in Appendix

---

Published as a conference paper at ICLR 2026
5.2 FINE-TUNING SETUP
To achieve efficient fine-tuning, we adopt Low-Rank Adaptation (LoRA) to reduce the computa- tional cost with a rank of 8, a scale factor of 16.0 and dropout rate of 0.1. We fine-tune all linear modules. The training uses the AdamW optimizer with a learning rate of 5e-5, combined with a linear decay scheduler with warm-up. Additionally, we further enhance training speed and model robustness through techniques such as 6 mixed precision and Flash Attention-2.
5.3 EVALUATION PROTOCOL
We employ a suite of tailored evaluation metries. For the Text Classification and Information Ex- traction tasks, we report both Accuracy and the balanced F1-score to account for potential class imbalances. For the Regression tasks, we use MAE and R² score to measure predictive accuracy.
Given the non-factual features of the semantic Q&A task, we develop multi-dimensional evaluation protocol. This protocol begins by calculating the STS score, which measures the sentence-level semantic similarity between the generated answer and the reference answer. To assess the domain expertise of the answers, we employ a model- -based evaluation framework M. Bran et al. (2024) where both got-4o and deepseck-rl are prompted to score the generated answers across three criteria: reasonableness, accuracy and Usability. Finally, we conduct human evaluation random subset of the test dataset, with domain experts providing the final judgment the model's practical applicability and domain-specift correctnes(Dettils in in[ppeedixx
scale from 1-10
RESULTS AND ANALYSIS
6.1 COMPARISON OF FINE-TUNING STRATEGIES
After fine-tuning the ChemLLM-7B model, we evaluate their performance on CatalystBench- test for each task. In Fig 5 task performance to identify how different fine- tuning strategies influence the results. The results show that the MFT strategy achieves the best performance, with the highest av- erage performance improvement rate relative to the ST baseline, reaching 12.44%. This indicates that combining multi-task learning with sask-specific output heads more effec- tive than applying either technique in isola- tion. Among these, MT fine-tuning resulted an average improvement of 9. 9.24%, while task- specific output head design achieve an aver- age improvement of only 5.13%. This high- lights that coupling architectures help tailor the model's outputs for specific task format, but the impact is limited without multi-task train- ing. When all tasks share underlying semantic features, the model can leverage these features to learn shared and transferable representations. We name the final ChemLLM-MFT model as Cata- LystLLM, an open-source foundational LLM specifically designed for catalytic science. We provide a detailed analysis of the suitability of MFT architecture for catalytic process tasks in in/Apenddxx B.
Performance Change for Different Fine-tuning Strategies
Basn
Mubishead
compare
Hulishood Arg
Bose F1
20-
Muti-head ST
Percentage Change
Muti-head FT
15-
20
ME
SE
RMC
FEP
AP
6.CP
RSC
Figure 5: Comparison of the impact of different fine-tuning strategies on task performance. Base- ST serves as the baseline model and the bar chart shows the performance differences between differ- ent models on specific tasks.
6.2 MAIN BENCHMARK RESULTS
We evaluate the performance of LLMs on catalytic tasks on Catahyst and report the results in Table and Table 2 Table compares the performance of CatalystLLM with other LLMs on tasks with factual answers. The results show that CatalystLLM achieve the best performance across all 7 tasks and 12 metrics, significantly outperforming general LLMs of similar scale. Compared to ChemLLM, CatalystLLM demonstrates a significant improvement in its capabilities to understand and apply catalytic knowledge, highlighting the effectiveness of multi-task catalytic data fine-tuning.
7

---

Published as a conference paper at ICLR 2026
ME
SE
RMC
FEP
AP
d-CP
FP
Model
ACC
ACC
FT
ACC
FT
R2
MAE
R2
MAL
R2
MAE
R2
MAE
Closed-souree LLMs
claude-3
0,93
0.95
0.84
0.90
0.58
0.72
0.30
6.73
0.37
3.58
0.33
0.39
5.24
Gemini-2.5
0.97
0.98
0.87
0.91
0.66
0.80
0.29
3.19
0.34
4.15
0.36
5.78
0.40
4.98
gpt-3.5
0.89
0.93
0.78
0.86
0.49
0.62
0.28
5.25
0.35
4.20
0.34
4.50
0.38
5.41
gpt-40
0,96
0.92
0.81
3.10
0.43
3.05
0.41
3.35
0.46
gpt-4,1
0.98
0.99
0.91
0.95
0.75
0.85
0.56
2.51
0.61
2.40
0.59
2.15
0.65
2.68
Open-source LLMS
deepseck-v3
0.92
0.95
0.83
0.89
0.61
0.74
0.33
4.20
0.40
3.55
0.38
3.85
0.43
4.45
LLaMA2-7B
0.83
0.86
0.73
0.79
0.50
4.47
0.24
4.63
0.34
3.79
0.35
4.12
Qwen3-88
0.86
0.91
0.74
0.83
0.42
0.54
0.26
4.59
0.31
3.85
0.32
3.49
0.36
4.26
Mistral-7B
0.88
0.92
0.76
0.84
0.44
0.57
0.29
3.09
0.37
3,49
0.36
4.14
0.39
5.15
CharGLM3-6B
0.84
0.90
0.74
0.83
0.45
0.57
0.33
4.87
0.39
3.07
0.37
3.48
0.41
4.38
ChemLLM
0.93
0.95
0.79
0.88
0.52
0.66
0.45
2,80
0.63
2.05
0.54
2.36
0.64
2.75
Darwinl.5 .5
0,91
0.94
0.79
0.89
0.50
0.64
0.44
2.81
0.59
3,13
0.54
2,42
0.68
2.01
CatalystLLM
0.98
0.99
0.89
0.94
0.81
0.89
0.73
1.72
0.81
1.24
0.73
1.49
0.80
1.34
Table 1: The performance of Cat with other LLMs on tasks with factual answers. The best model is bold font and the second-best is underlined.
ChemLLM and Darwin1.5 outperform top general models on most metrics. This indicates that specialized models can learn more precise chemical named entities, reaction conditions and critical features in complex structure-activity relationships through fine-tuning in specific domains.
Notably, general I.LMs perform exceptionally well tasks such as ME, SE and RMC tasks. For instance, gpt-4.1 achieves higher accuracy and F1-scores than CatalystI.LM in the SE task. However, in numerical regression prediction tasks, there is a significant gap in capability between general models and specialized models, indicating that general LLMs currently can not accurately perform prediction tasks which require deep scientific reasoning and quantitative calculations. For numerical regression tasks, also compare the results of CatalystLLM and traditional single-task competitive ML algorithms in Appendix B.5
Table 2 compares the performance of Cata- lystLLM with other I.LMs in semantic un- derstanding tasks, including multi-dimensional evaluation metrics. In all three evaluation di- mensions, CatalystLLM achieves the optimal or suboptimal results, demonstrating the poten- tial of domain-spectie -seecff fine-tuning to enhance answer quality. Additionally, the STS scores of general LLMS are typically higher than those of open-source models, reflecting the superior answer generation capabilitics. General LLMs typically have higher LLM Scores, while spe- cialized models including CatalystLLM have relatively higher expert scores. This reflects two phenomenons: 1) Top general models gen- erate answers that are fluent, logically clear and comprehensive, which aligns with the prefer- ences of gpt-4o and deepseck-r1; 2) General models are prone to scientific hallucinations in spe- cialized fields like catalytic science Xu et al. (2024). These answers may contain incorrect catalyst performance analyses or explanations that conflict with physical chemistry principles and such er- rors can only be identified by domain experts. We provide an error example of L.MM-score and expert scores in Appendix C.2, Additionally, we provide case studies to highlight the impact of CatalystLLM's domain expertise on catalytic materials scientists in Appendix C.3
Model
Scores deepseck-ri
Experts dF Score
Scores
APC-4o
Closed-source LLMs
claude-3
0.73
0,74
0,73
0.46
Gemini-2.5
0.74
0.75
0.76
0.47
gpt-3.5
0.71
0.69
0,71
0.41
gpt-4o
0.72
0.46
gpt-4.1
0.72
0.85
0,88
0.56
Open-source
deepsook-v33
0.73
0.84
0,86
0.57
LLMMA2-7B
0.57
0.52
Qwen3-8B 8B
0.62
0.70
0.72
0.47
Mistral-BB
0.66
0.42
CharGLM3-6B
0.61
0.58
0.60
0.38
C.
0.68
0.64
0.68
0.54
Darwinl.5
0.67
0.59
0.61
0.49
CatalysiLLLM
0.79
0.82
0.86
0.75
Table The performance of CatalystLLM with other LLMs in semantic understanding tasks. The best model in bold font and the second-best is underlined.
6.3 ABLATION STUDY
We compare the impact of different dataset settings and experimental settings model capabilities to explore the key factors that determine LLM domain capabilities. the experimental setup, all models are based on the optimal MFT architecture.
8

---

Published as a conference paper at ICLR 2026
6.3.1
COLLABORATIVE EFFECTS OF TASK COMBINATIONS
MFT Regression Tasks
MFT w/o Classification Tasks
MFT Info Extraction Tasks
191 (uaidre))
Aut 101 Dusslied
10 Bereel
Perforiaa Score
4.0
MF
109
855
47
A3
4.33
8P
05
RMC
RGC
05
AG Chemistry-auta
-423
MF
001
8.5
02-
3.7
-2533
82
031
Figure 6: The experiments on collaborative effects of task combinations.
We investigate the potential collaborative relationship among different types of tasks under multi- task learning framework. We remove regression tasks, classification tasks and information extraction tasks from the training data separately and then evaluate the corresponding performance. As shown in Fig 6 compared to the Full-MFT baseline, the performance of all ablation models decrease on the corresponding tasks. Additionally, we divide these tasks into two groups based on the type of input data: Text data and Chemical data and compare the collaborative effects of tasks within and between groups.
Intra-group tasks exhibit strong synergistic effects. For example, when RMC data is removed, the performance of other tasks within the same group decreases significantly. This indicates that learning "how to regulate" (RMC) provides critical contextual knowledge for LLM to understand "how effective the regulation is" (FEP) and "the significance of the regulation scheme" (RSC).
Inter-group tasks exist certain degree of synergistic effects, For example, in the "MFT w/o Regression" setting, the performance of the RMC and RSC tasks is also affected to some extent. This reveals a deeper phenomenon of domain-specinf fine-tuning model: it does not merely memorize input-output patterns from training data but construct overall knowledge model of the catalytic domain. CatalystI.IM can span different data modalities, such as chemical formulas and natural language, thereby aiding meaksig in other tasks.
6.3.2 THE IMPACT OF INPUT FORMAT MODEL PERFORMANCE
To investigate the advantages of Catalyst- Bench, we investigate the impact of input data format on the performance of LLMs. The ex- periments aim to compare two input strate- gies: one provided only unstructured core in- formation, while the other provided structured and complete input. The two input strategies and related generated prompts are shown in the Appendix E.2 As shown in Fig 7 given the same amount of information, the quality and format of input data are key factors de- termining the upper limit of LLMs' perfor-
1.0
Ginarrl Aug (0.24)
Performance Score (Higher Better)
MiT--rrmated fug (c. 79)
MFT Gonaral Text
MF WFT-Formabedd Tuo:
0.8L
0.8
0.75
1L.55
0:75
0.7
0.6
mance in professional tasks, especially chemical representation of catalysts. The sin- gle SMILES string sequence can only provide macro-level representation, while structured knowledge allows the model to learn the quan- tity and connection methods of internal atoms, thereby learning potential structure-actiinn acttvvttt re- lationships.
the
0.5
RMC (F1)
d-CP
FP
(R2)
(R2)
(R2)
(R2)
Figure 7: The impact of input format on model performance.
9

---

Published as a conference paper at ICLR 2026
7 CONCLUSION AND DISCUSSION
This work presents CatalystBench, a comprehensive benchmark dataset covering crucial processes in catalytic science and systematically demonstrates the effectiveness of multi-head full-task fine- tuning in adapting to complex scientific tasks. Building upon this foundation, we develop Cata- LystLLM, high-perie perfrfrrffmmeee specialized language model tailored for the field of catalysis. Our experiments demonstrate the superiority of Cat over existing LLM models and highlight the pivotal role of CatalystBench enhancing performance. Further analytical experiments also provide crucial insights for developing LLM models for catalytic materials.
LL attains SOTA performance, yet certain limitations remain. Its knowledge scope is con- strained by the coverage of the CatalystBench dataset, potentially reducing predictive accuracy for catalytic materials outside this range. While the 7B-parameter scale balances efficiency and capabil- ity, larger models may exhibit superior reasoning and learning capacity. Furthermore, the inclusion of data from existing literature introduces potential bias related to source selection. Future work will aim to broaden CatalystBench encompass wider variety of catalytic systems and tasks, enhance CatalystLLLM's downstream inference and embed the model within a closed-loop catalyst discov- ery framework, enabling Al predictions directly inform synthesis and characterization, thereby accelerating the development of novel materials.
ETHICS COMPLIANCE
This paper adheres the ICLR Code of Ethics. We acknowledge that our research follows ethical guidelines, ensuring compliance with all relevant regulations. No human subjects were involved in this research. All data used in the study are publicly available datasets or generated synthetically. We have taken care to ensure that all data used complies with relevant data protection laws. Datasets used this research do not contain any personal, confidential, or sensitive information. We confirm that the research presented is original and has not been plagiarized. All sources and data properly cited and documented. We confirm that the research complics with all applicable laws and ethical standards, including data usage and research practices. This statement is meant address potential concerns in accordance with the ICLR Code of Ethics.
REPRODUCIBILITY STATEMENT
We ensure reproducibility by providing all necessary resources for others replicate our experi- ments. Specifically;
1) Benchmark Data and Experiment Setup: All datasets used experiments and detailed data pro- cessing steps are publicly accessible. The experimental setup is also provided, including hyperpa- rameters, evaluation metrics and the benchmarking environment.
2) Source code: The code used for data processing and benchmarking experiments is provided in the supplementary materials, containing all scripts required to reproduce the experiments described in the paper.
By providing these resources, we aim achieve full reproducibility of our research findings and facilitate subsequent studies based on this work. Detailed information required for reproduction can be found in the supplementary materials.
ACKNOWLEDGEMENTS
This work are supported by Zhongguancun Academy Project No. No.00202501, in part by the Natural Science Foundation of China under Grant No. T2322027 and 62442204, in part by the Key Research Program of the Chinese Academy of Sciences under Grant No.RCJJ-145 145-2--45----200 partially by the Nat- ural Science Foundation of China(No.9247022)) and partially by Young Scientists Fund Category C (No.6550335)).
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew J Ballard, Joshua Bambrick, et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, 630(8016):493-00,) 2024.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303. 08774, 2023.
Luis M Antunes, Keith T Butler, Ricardo Grau- Trau-Crespo, Ricardo Grau Grau-Crespo Ricardo Brau-Crespo, and Ricardo Grau-Crespo. Crystal structure generation with autoregressive large language mod- eling. Nature Communications, 1501), 2024.
Viraj Bagal, Rishal Aggarwal, PK Vinod, Deva Priyakumar, PK Vinod, Deva Priyakumar, PK Vinod, U Deva Priyakumar, PK Vinod, and U Deva Priyakumar. Molgpt: molecular gener- ation using transorrmer-decoder model. Journal of chemical information and modeling, 62(9): 2064-2076, 2021.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, al. Qwen technical report. arXiv preprint UrXiv:2309.16600 2023.
Suryanarayanan Balaji, Rishikesh Magar, Yayati Jadhav, and Amir Barati Farimani. Gpt-molberta: Gpt molecular features language model for molecular property prediction. arXiv preprint 0rXiv:2310.03030, 2023.
He Cao, Zijing Liu, Xingyu Lu, Yuan Yao, and Yu Li. Instructmol: Multi-modal integration for building a versatile and reliable molecular assistant in drug discovery. arXiv preprint arXiv:2311.16208. 2023a.
Zhonglin Cao, Rishikesh Magar, Yuyang Wang, and Amir Barati Farimani. Moformer: self- supervised transformer model for metal-organic framework property prediction. Journal of the American Chemical Society, 145(5):2958-299 2023b.
Jinho Chang, Jong Chul Ye, Jong Chul Ye, Jong Chul Ye. and Jong Chul Ye. Bidirectional generation of structure and properties through a single molecular foundation model. Nature Communications, 15(1):2323, 2024.
Lowik Chanussot, Abhishek Das, Siddharth Goyal, Thibaut Lavril, Muhammed Shuaibi, Morgane Riviere, Kevin Tran, Javier Heras- Dooinno, Caleb Ho, Weihua Hu, et al. Open catalyst 2020 (oc20) dataset and community challenges. Acs Catalysis, 11(10)6659-60722 2021.
Hui Chen, Xiao Liang, Yipu Liu, Xuan Ai, Tewodros Asefa, and Xiaoxin Zou. Active site engineer- ing in porous electrocatalysts. Advanced Materials, 32(44):0002435, 2020.
Xueqing Chen, Yang Gao, Ludi Wang, Wenjuan Cui, Jiamin Huang, Yi Du, and Bin Wang. Large language model enhanced corpus co2 reduction electrocatalysts and synthesis procedures. Sci- entific Data, 11(1):347, 2024.
Yi Chen, Jian Xu, Xu- Yao Zhang, Wen-Zhuo Liu, Yang- Yang Liu, and Cheng-Lin Liu. Recoverable compression: A multimodal vision token recovery mechanism guided by text information. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 2293-2301, 2025.
Seyone Chithrananda, Gabriel Grand, Bharath Ramsundar, Bharath Ramsundar, Bharath Ramsun- dar, and Bharath Ramsundar. Chemberta: large-scale self-supervised pretraining for molecular property prediction. arXiv preprint wwwp:rrinaaaa:222000ma arXiv:2010.09885 2020.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.0626/ 2025.

---

Published as a conference paper at ICLR 2026
Yi Du, Ludi Wang, Mengyi Huang, Dongze Song, Wenjuan Cui, and Yuanchun Zhou. Autodive: An integrated onsite scientific literature annotation tool. In Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 76-85, 2023.
Carl Edwards, ChengXiang Zhai, Heng Ji, Heng Ji, and Heng Ji. Text2mol: Cross-modal molecule retrieval with natural language queries. In Proceedings of the 202/ Conference Empirical Methods in Natural Language Processing, pp. 595-607, 2021.
Carl Edwards, Tuan Lai, Kevin Ros, Garrett Honke, Kyunghyun Cho, and Heng Ji. Translation between molecules and natural language. arXiv preprint 2001v:220..11817 2022.
Fei Fu, Qing-Qing Li, Fangrong Wang, Jie Hu, Tian-Tian Wang. Yun-Pei Liu, Weihong Xu, Zhili Lin, Fu-Qiang Gong, Yu- Fan, et al. Synergizing a knowledge graph and large language model for relay catalysis pathway recommendation. National Science Review, 12(S): 2025.
Yang Gao, Ludi Wang, Xueqing Chen, Yi Du, and Bin Wang. Revisiting electrocatalyst design by knowledge graph of cu-based catalysts for co2 reduction. ACS Catalysis, 13(13))8525-833, 2023.
Johannes Gasteiger, Shankari Giri, Johannes Margraf, and Stephan Günnemann. Fast and uncertainty-aware directional message passing for non-equilibrium molecules. arXiv preprint arXiv:2011.141155 2020a.
Johannes Gasteiger, Janek GroB, and Stephan Günnemann. Directional message passing for molec- ular graphs. arXiv preprint arXiv:2003.03 /233, 2020b.
Johannes Gasteiger, Muhammed Shuaibi, Anuroop Sriram, Stephan Günnemann, Zachary Ulissi, C Lawrence Zitnick, and Abhishek Das. Gemnet-oc: developing graph neural networks for large and diverse molecular simulation datasets. arXiv preprint arXiv:2204.085 02782, 2022.
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, al. Chatglm: family of large language models from glm- to glm-4 all tools. arXiv preprint arXiv:2406. 12793, 2024.
Taicheng Guo, Bozhao Nan, Zhenwen Liang, Zhichun Guo, Nitesh Chawla, Olaf Wiest, Xiangliang Zhang, et al. What can large language models do in chemistry? a comprehensive benchmark on cight tasks. Advances in Neural Information Processing Systems, 36:59662-59688, 2023.
Chaitanya B Hiragond, Niket S Powar, Junho Lee, and Su-Il In. Single-atom catalysts (sacs) for pho- tocatalytic co2 reduction with h2o: Activity, product selectivity, stability, and surface chemistry. Small, (8(29):2201428 2022.
Ross Irwin, Spyridon Dimitriadis, Jiazhen He, and Esben Jannik Bjerrum. Chemformer: a pre- trained transformer for computational chemistry. Machine Learning: Science and Technology, (1):015022, 2022.
Kevin Maik Jablonka, Philippe Schwaller, Andres Ortega- Grega-Grrrrer,,,, and Berend Smit. Leverag- ing large language models for predictive chemistry. Nature Machine Intelligence, 6(2):161-169, 2024.
Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, et al. Commentary: The ma- terials project: A materials genome approach to accelerating materials innovation. APL materials, 1(1), 2013.
Albert Q Jiang, Alexandre Sablayrolles. Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04mm 04088, 2024.
12

---

Published as a conference paper at ICLR 2026
Gueyoung Jung, Matti A Hiltunen, Kaustubh R Joshi, Richard D Schlichting, and Calton Pu. Mis- tral: Dynamically managing power, performance, and adaptation cost in cloud infrastructures. In 2010 IEEE 30th International Conference Distributed Computing Systems, pp. 62-73. IEEE, 2010.
Olga Kononova, Haoyan Huo, Tanjin He, Ziqin Rong, Tiago Botari, Wenhao Sun, Vahe Tshitoyan, and Gerbrand Ceder. Text-mined dataset of inorganic materials synthesis recipes. Scientific data, 6(1):203, 2019.
Agustinus Kristiadi, Felix Atriet-Kaathhof Marta Skreta, Pascal Poupart, Alán Aspuru-Guzik, and Geoff Pleiss. sober look at Ilms for material discovery: Are they actually good for bayesian optimization molecules? arXiv preprint arXiv:2402.05015. 2024.
Bernadette Lamb, Jonathan Herskovitz, Marta Jonson, Harlan Sayles, and Faruq Pradhan. $2185 use of large language model (Ilm) chatbots for the generation of specialized gastrointestinal diet meal plans: A pilot study. Official journal of the American College of Gastroenterology-Aee ACG, 119(105):1166, 2024.
Hao Li, He Cao, Bin Feng, Yanjun Shao, Xiangru Tang, Zhiyuan Yan, Li Yuan, Yonghong Tian, and Yu Li. Beyond chemical qa: Evaluating Im''s chemical reasoning with modular chemical operations. arXiv preprint arXiv:2505.2/4 233.,, 2025.
Jiatong Li. Junxian Li, Yunqing Liu, Dongzhan Zhou, and Qing Li. Tomg-bench: Evaluating 1lms on text-based open molecule generation. arXiv preprint arXiv:2412. 14642, 2024.
Yi-Lun Liao and Tess Smidt. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs. arXiv preprint [Tt.::2006.199990 2022.
Yi-Lun Liao, Brandon Wood, Abhishek Das. and Tess Smidt. Equiformerv2: Improved equivariant transformer for scaling to higher-degree representations. arXiv preprint arXiv:2306. 2023.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024a.
Pengfei Liu, Yiming Ren, Jun Tao, and Zhixiang Ren. Git-mol: multi-modal large language model for molecular science with graph, image, and text. Computers in biology and medicine, 171:108073, 2024b.
Siyu Liu, Jiamin Xu, Beilin Ye, Bo Hu, David Srolovitz, and Tongqi Wen. Mattools: Bench- marking large language models for materials science tools. arXiv preprint arXiv:2505.1083 10852, 2025.
Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie- Yan Liu. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings bioin- formatics, bbbac44, 2022.
Andres M. Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. Augmenting large language models with chemistry tools. Nature Machine Intelli- gence, 6(5):525-535, 2024.
Yujie Ma, Yang Gao, Ludi Wang, Ming Chen, Wenjuan Cui, Bin Wang, and Yi Du. Accelerating ma- terials discovery through active learning: Methods, challenges and opportunities. The Innovation Informatics, I (11::10013: 2025.
Amil Merchant, Simon Batzner, Samuel S Schoenholz, Muratahan Aykol, Gowoon Cheon, and Ekin Dogus Cubuk. Scaling deep learning for materials discovery. Nature, 624(7990):80-85, 2023.
13

---

Published as a conference paper at ICLR 2026
Adrian Mirza, Nawaf Alampara, Sreekanth Kunchapu, Martiño Ríos-Garcia, Benedict Emoek- abu, Aswanth Krishnan, Tanya Gupta, Mara Schilling- Wilhelmi, Macjonathan Okereke, Anagha Aneesh, et al. Are large language models superhuman chemists? arXiv preprint arXiv:2404.0147. 2024.
Janghoon Ock, Chakradhar Guntuboina, Amir Barati Farimani, Amir Barati Farimani, Amir Barati Farimani, and Amir Barati Farimani. Catalyst energy prediction with catberta: unveil- ing feature exploration strategies through large language models. ACS Catalysis, 13(24):16032- 16044, 2023.
Janghoon Ock, Srivathsan Badrinarayanan, Rishikesh Magar, Akshay Antony, and Amir Barati Fa- rimani. Multimodal language and graph learning of adsorption configuration in catalysis. Nature Machine Intelligence, 6(12):1501-1511, 2024.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Andre Niyongabo Rubungo, Kangming Li, Jason Hattrick-Simpers and Adji Bousso Dieng. Lim4mat-bench: benchmarking large language models for materials property prediction. Ma- chine Learning: Science and Technology, 6(2):020501, 2025.
Abdullahi Saka, Ridwan Taiwo, Nurudeen Saka, Babatunde Abiodun Salami, Saheed Ajayi, Kabiru Akande, and Hadi Kazemi. Gpt models in construction industry: Opportunities, limitations, and a use case validation. Developments the Built Environment, 17:100300, 2024.
Kristof T Schütt, Huziel Sauceda, P-J Kindermans, Alexandre Tkatchenko, and K-R Müller. Schnet-a deep learning architecture for molecules and materials. The Journal of chemical physics, 148(24), 2018
Zhilong Song, Shuaihua Lu, Minggang Ju, Qionghua Zhou, and Jinlan Wang. Accurate prediction of synthesizability and precursors of 3d crystal structures via large language models. Nature Communications, 16(1))6530, 2025.
Kyle Swanson, Wesley Wu, Nash I. Bulaong, John E Pak, and James Zou, The virtual lab of ai agents designs new sars-cov-2 nanobodies. Nature, pp. 1 1-3, 2025.
Nathan Szymanski, Bernardus Rendy, Yuxing Fei, Rishi E Kumar, Tanjin He, David Milsted, Matthew MeDermott, Max Gallant, Ekin Dogus Cubuk, Amil Merchant, et al. An autonomous laboratory for the accelerated synthesis of novel materials. Nature, 624(7990):86-91, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288. 2023.
Amalie Trewartha, Nicholas Walker, Haoyan Huo, Sanghoon Lee, Kevin Cruse, John Dagdelen, Alexander Dunn, Kristin Persson, Gerbrand Ceder, and Anubhav Jain. Quantifying the ad- vantage of domain-specific pre-training named entity recognition tasks in materials science. Patterns, 3(4), 2022.
Ethan Waisberg, Joshua Ong, Mouayad Masalkhi, and Andrew G Lee. Large language model (Ilm)- driven chatbots for neuro-ophthalmic medical education. Eye, 38(4):639-664 641, 2024.
Yuwei Wan, Aswathy Ajith, Yixuan Liu, Ke Lu, Clara Grazian, Bram Hocx, Wenjie Zhang, Chunyu Kit, Tong Xie, and Ian Foster. Sciqag: a framework for auto-generated scientific question answering dataset with fine- fine-grained evaluation. CoRR, 2024.
Ludi Wang, Yang Gao, Xueqing Chen, Wenjuan Cui, Yuanchun Zhou, Xinying Luo, Shuaishuai Xu, Yi Du, and Bin Wang. A corpus of co2 electrocatalytic reduction process extracted from the scientific literature. Scientific Data, 10(1):175, 2023a.
Ludi Wang, Dongze Song, Qiang Cui, Xueqing Chen, Yuanchun Zhou, Wenjuan Cui, and Yi Du. Autodive+: An adaptive model enhanced multimodal online annotation tool. In Companion Pro- ceedings of the ACM on Web Conference 2025, pp. 2919-2922, 2025a.
14

---

Published as a conference paper at ICLR 2026
Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level sci- entific problem-solving abilities of large language models. arXiv preprint arXiv:2307.10335, 2023b.
Xinming Wang, Jian Xu, Aslan H Feng, Yi Chen, Haiyang Guo, Fei Zhu, Yuanqi Shao, Minsi Ren, Hongzhu Yi, Sheng Lian, et al. The hitchhiker's guide to autonomous research: survey of scientific agents. Authorea Preprints, 2025b.
Kirsten T Winther, Max J Hoffmann, Jacob R Boes, Osman Mamun, Michal Bajdich, and Thomas Bligaard. Catalysis-hub. org, an open electronic structure database for surface reactions. Scientific data, 6(1):75, 2019.
Tian Xie and Jeffrey C Grossman. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Physical review letters, 120(14): 14(141, 2018.
Tong Xie, Yuwei Wan, Yixuan Liu, Yuchen Zeng, Shaozhou Wang, Wenjie Zhang, Clara Grazian, Chunyu Kit, Wanli Ouyang, Dongzhan Zhou, et al. Darwin 1.5: Large language models as materials science adapted learners. arXiv preprint vrXiv:2412.11970 2024.
Changwen Xu, Yuyang Wang, and Amir Barati Farimani. Transpolymer: a transformer-based lan- guage model for polymer property predictions. npj Computational Materials, 9(1):64, 2023.
Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. Hallucination is inevitable: An innate limitation of large language models. arXiv preprint arXiv:2401.11817, 2024.
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09338, 2025.
Fangkai Yang. Pu Zhao, Zezhong Wang, Lu Wang, Jue Zhang, Mohit Garg, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. Empower large language model perform better industrial domain-specific question answering. arXiv preprint arXiv:2305. 1541, 2023.
Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, al. A comprehensive capability analysis of gpt-3 and gpt-3.5 series models. arXiv preprint arXiv:2303. 10420, 2023.
Botao Yu, Frazier N Baker, Ziqi Chen, Xia Ning, and Huan Sun. Llasmol: Advancing large language models for chemistry with large-scale, comprehensive, high-quality instruction tuning dataset. arXiv preprint (ii::2402.0939) 2024.
Mohd Zaki, NM Anoop Krishnan, et al. Mascqa: investigating materials science knowledge of large language models. Digital Discovery, 3(2):3 2024.
Baicheng Zhang, Zhuoying Zhu, Huirong Li, Jiaqi Cao, and Jun Jiang. Revolutionizing chemistry and material innovation: An iterative eeoortical-experimeentl paradigm leveraged by robotic ai chemists. CCS Chemistry, 7(2):345-360, 2025a.
Di Zhang, Wei Liu, Qian Tan, Jingdan Chen, Hang Yan, Yuliang Yan, Jiatong Li, Weiran Huang, Xiangyu Yue, Wanli Ouyang, et al. Chemllm: chemical large language model. arXiv preprint arXiv:2402.06852. 2024a.
Huan Zhang, Yu Song, Ziyu Hou, Santiago Miret, and Bang Liu. Honeycomb: A flexible llm-based agent system for materials science. arXiv preprint arXiv:2409.001 00035, 2024b.
Yu Zhang, Yang Han, Shuai Chen, Ruijie Yu, Xin Zhao, Xianbin Liu, Kaipeng Zeng, Mengdi Yu, Jidong Tian, Feng Zhu, al. Large language models accelerate organic chemistry synthesis. Nature Machine Intelligence, pp. 1-13, 2025b.
Zihan Zhao, Da Ma, Lu Chen, Liangtai Sun, Zihao Li, Yi Xia, Bo Chen, Hongshen Xu. Zichen Zhu, Su Zhu, et al. Chemdfm: a large language foundation model for chemistry. arXiv preprint arXiv:2401. 14818, 2024.
15

---

Published as a conference paper at ICLR 2026
Yun Pei Zhu, Chunxian Guo, Yao Zheng, and Shi-Zhang Qiao. Surface and interface engineering of noble-metal- -feee electrocatalysts for efficient energy conversion processes. Accounts of chemical research, 50(4):915-925 923, 2017.
16

---

Published as a conference paper at ICLR 2026
TABLE OF CONTENTS IN APPENDIX
Data Construction
18
A.
Data Source
18
Task Formulation
19
A.3
The comparison of Chemical Benchmark
19
Q&A Generation
21
A.5
Quality Assurance Details
21
A.5.1
Theoretical Q&A Data Calibration
.ii.............................
A.5.2
Open-ended Q&A Data Calibration
21
A.6
Error Samples
22
A.7
Data Distribution and Diversity Analysis
23
Experimental Details
25
B.1
Domain-specifie Challenges in CatalystBench Multi-tas Learning
25
Baseline LLMS
25
B33
Baseline traditional ML. methods
26
B.4
Evaluation among open-sourced models
27
B
Evaluation with traditional ML methods
28
B.6
Additional Ablation study.
29
B.6.1
The impact of weighted loss-function model performance
29
B.6.2
OOD Leave-One: Task-Out Evaluation
30
Evaluation Details
LLM-based evaluation
31
Expert evaluation
Case Study: Unique Correct Reasoning in Complex Catalytic Context
32
D Question Templates
33
D.I Label Definition
33
D.2 Material Extraction Templates
34
D.3
Synthesis Extraction Templates
35
D.4
Regulation Method Classification Templates
35
D.5
Property Prediction Cemplates-Genetee text
36
D.6
Property Prediction Templates-Cheminn text
36
D.7
Regulation Scheme Comprehension
36
E
Prompts
38
Prompts for QA Generating
38
Prompts for chemical-text generation
38
17

---

Published as a conference paper at ICLR 2026
E.3 Prompts for LLM-based evaluation
40
Statement on the Use of Large Language Models
40
A
DATA CONSTRUCTION
We introduce the various sources of CatalystBench data, describe the specific tasks within each category and clarify the complete process used to compile this dataset.
Evaluation Metries
Ability
Task
Task Type
Datasets
Dataset Type
#count
Material Extraction (ME)
ElectroCaatlyyti
Generation
Experimental Data
2,480
Reduction( I & 2)
Accuracy, Fl score
Understanding
ElsctroCattayyn atalytic
Synthesis Extraction (SE)
Reduction( & 2).
Generation
Experimental Data
6,612
Solution-base based
synthesis dataset
Regulation Method Classification(NNN (RCC)
ElectroC atallyiic
Classification
Experimental Data
2,364
Reduction( I & 2)
Faradaic Efficiency Prediction(FEP) Adsorption Prediction (AP)
ClectroCatalytic
Prediction
Experimental Data
2,148
Reduction( 1 & 2)
R². MAE
Reasoning
OC20-Dense,
Prediction
Theoretical Simulation
3,000
Dataset
SACs Dataset, Catalysis-Hub,
d-band Center Prediction(d-CPP
Prediction
Catalytic Material
Theoretical Simulation
3,000
Database(CMD)
SACs Dataset,
Formation Prediction
Material Project,
(FP)
Prediction
Catalytic Material Database(CMD)
Theoretical Simulation
3,000
Regulation Scheme Comprchennson((SCC)
ElectroCatalytic Reduction( 2)
STorr
Explaining
Generation
Experimental Data
4,307
LLM Score
Table 3: The statistics of all tasks, datasets, dataset type and evaluation metrics.
A. 1
DATA SOURCE
We extract data from open-source databases. The name and description of each dataset are listed below.
ElectroCatalytic Reduction /aan al. (2023a) An open-source corpus of electro- catalytic CO2 reduction extracted from science literatures Du et al. (023).
ElectroCatalytic Reduction 2 Chen et al. 2024 A text-mining dataset describing the CO2 reduction process catalyzed by copper- electrocatalysts [Wang et al. (2025a), which specifically includes material, regulation method, product, Faradaic efficiency and relevant synthesis conditions.
Solution- synthesis dataset Kononova et al. 2211 A dataset of 35,675 solution- based synthesis procedures extracted from the scientific literature. Each procedure contains essential synthesis information including the precursors and target materials, their quanti- ties and the synthesis actions and corresponding attributes.
OC20-Dense Chanussot al. $2021 The largest dataset of catalyst-adsorbate inter- actions to date, designed to accelerate catalyst discovery through machine learning. The dataset contains over million DFT calculations covering an extremely wide range of cat- alyst materials (including bulk, surfaces and nanoparticles) and adsorbates.
Catalytic Material Database(CMD) CMD contains material composition, properties, reactions, products and other information.
SACs Hiragond et al. | A dataset catalyst samples constructed for electrocat- alytic reactions such as OER ORR. SACs contains dozens of transition metals, combined
18

---

Published as a conference paper at ICLR 2026
with several different coordination environments and substrates, resulting in hundreds of catalyst samples.
Catalyst Hub Winther et al. (2019) A featured database for surface reactions contains more than 100,000 chemisorption and reaction energies obtained from electronic structure calculations and is continuously being updated with new datasets.
Materials Project Jain et al. (2013) The Materials Project provides computed infor- mation on known and predicted materials as well as powerful analysis tools to inspire and design novel materials.
A.2 TASK FORMULATION
We evaluate the understanding, reasoning and explanation capabilities of LLMs in catalyst design by constructing 8 widely recognized practical catalytic material tasks. Sumberiiaamizzze these tasks, including their task types from machine learning perspective, the datasets used for evaluation and the evaluation metrics.
Converting structured chemical data into reasoning- adjusted data suitable for training LLMs presents two main challenges: 1) the creation of diverse templates and 2) the integration of chemical logic and reasoning into QA pairs. The information extraction tasks focus on identifying composition and operational parameters related to material design from scientific literatures. The classification tasks predict discrete categories or labels, while the regression tasks focus on continuous numerical property values. For template diversity, we initially developed foundational seed template meet the requirements of specific tasks. Using gpt-4o, We generate a series of prompt templates that express different meanings while maintaining semantic consistency. These diverse templates enhance the model's 's ability to interpret and respond to different instruction formats. For each structured data entry, we randomly select one of these templates to create a single-round dialogue sample.
Simple Instruction Data Synthesis Method:
The source datasets typically use various representations of catalytic materials to characterize the corresponding catalytic properties [Co et al. (2023b); [Balji et al. (2023); Xu et al. (2023); OOk et al. (2023). We propose key hypothesis: When the model is only provided with the material common name and its SMILES representation, it is difficult for LLM to learn the complex structure- property relationships implicit in the data. To validate this hypothesis and explore the impact of input information richness on model performance, we create prompts based on the source dataset to generate different types of catalytic characterization input data. The prompts included in Appendix E.2
Complex Scenario Data Synthesis Method: For information-rich complex texts such synthesis schemes, we enriched the data by constructing multi-round dialogues to provide contextual depth and logical consistency. By simulating the deep thinking and step-by-step reasoning involved in catalytic material design processes, we created highly specialized multi-round dialogue dataset, reducing the need for dialogue rounds and human intervention. The above template examples are included in Appendix D
A.3 THE COMPARISON CHEMICAL BENCHMARK
Catalysis research presents unique challenges that distinguish it from broader chemistry mate- rials science applications, particularly in the combination of heterogeneous knowledge modalities and the multi-stage, interdependent workflow of real catalyst design. While prior LLM benchmarks in chemistry and materials science have explored domain-specific reasoning or property prediction,
07106
http:s//aa..cco/0055577////cccca 13290
fighhare. 16583387.v
httaa///geennaallysttroooo.....
http: htte://dmdd.s..e...p///taaaatd
app/
ntta://ww.catallvstni
19

---

Published as a conference paper at ICLR 2026
Search
APS
Prompt: Given the provided scientific texts, please complete the following steps: Keyword Generation.
GPT-40
Scientifie papers
Knowledge paragraphs
ScIQAG
Evaluate
Keyword
Mothod_ type
Method Description
Sclence Contexts
[CugeCuxO nanoparticles, [Defective VE-Arl, [Carbon defects), (Hydrogen evolution suppression). [Excessive oxidation Impeding]
the well dispersed CuggCuxO na nanopaticles deposited on the defective VG-Ar
The improved CaZ recuction performance of CuVV-Ar species depos ted on VG
QA Generation
Composite
Qu: How does the presence well-disperee CufCCuxD nanoparticles defective VG-Ar contribute improved reduction performance?
Experts
The well-dispersed CurCuurD naneporticles defective VG- enhance
engineering material Integration.
Scientific papers
Figure 8: SciQAG.
The process of converting regulation method text into high-quality Q&A pairs using
none have simultaneously aligned tasks to the full, practical catalyst R&D pipeline - from extrac- tion of synthesis parameters in literature, through structure-properyy prediction, to the mechanistic interpretation of regulation strategies.
Workflow Alignment to Real Catalyst
Domain Focus
Data Modalities
Benchmark
Task Type
Design
General
ChemBench
chemistry
Multipl-chhoice factual recall
Text only
x Fragmented
knowledge
Materials
MaScQA
science
Q&A from graduate-level examinations
Text only
Fragmented
x
concepts
College-level
scientific
SCIBENCH
Multi-domain Q&A
Text only
problem
Fragmented
solving
Chemical
Step-wise chain-of-thoughh
Text symholic expressions Structured material descriptors
ChecCoTBench
reasoning
Fragmented
LLM4Mat-Bench
property
Material
Regressiondclasiffcatioo on numerical properties
x Partial
prediction
Text structured chemical
8 tasks: information extraction, method classification, numerical property prediction. regulation scheme comprehension
Cutalysis-
CatalystBench
specific, combining theory experiment
/ Full workflow coverage
(ours)
descriptors
numerical
values
Table 4: The comparison of CatalystBench and the current Benchmark across dimensions such as domain categories, data modalities and task types.
Table 4 compares representative existing chemical benchmarks with CatalystBench in terms of task categories, data modalities and alignment to real-world catalyst design workflows. The compari- illustrates that CatalystBench uniquely integrates both high- high-fidelity theoretical simulations and curated experimental literature into task suite directly reflecting the sequential steps of modern catalytic materials discovery.
20

---

Published as a conference paper at ICLR 2026
A.4 Q&A GENERATION
We use SciQAG Wan et al. (024) to guide GPT-4o in interpreting large volume of regulation scheme texts, converting them into high-quality comprehension Q&A pairs. The pre-defined task is defined as follows: Given a seed input catalytic design scheme text T, for each input text 1, the generator should first generate keywords that capture the most important terms and concepts in the text and then generate a set S = {(qi,:)) J== focusing on the generated keywords k, where Vi E (1),. qi is the question and a; is the answer to qi- We use the 1,198 catalytic design scheme texts from the ElectroCatalytic Reduction dataset as input to generate 5,990 seed Q&A pairs by prompting GPT-4o. The generating prompts are included in Appendix E. To reduce the occurrence of non-knowledge-based questions that can only be answered using information from the given article, we develop a sule-based method to remove all pairs containing "this paper" or "this study." As shown in Fig 8 we ultimately generate 4,307 semantic understanding text Q&A pairs in this step.
A.5 QUALITY ASSURANCE DETAILS
In order to guarantee the scientific credibility of CatalystBench and to ensure reproducibility of results, a multi-layered quality assurance framework was implemented during dataset development. This framework covers both theoretically simulated catalytic property data and open-ended Q&A A content extracted and generated from experimental literature.
A.5.1 THEORETICAL Q&A DATA CALIBRATION
For task data derived from theoretical simulations, we first perform foundational consistency checks before feeding it into the conversion interface and generating corresponding QA-formatted con- tent. This process removes samples with obvious errors, such as invalid SMILES representations, missing chemical formulas, or attribute values falling outside the original data's statistical range. Si- multaneously, samples with duplicate structures or identical text are cleaned using hash fingerprint comparison and field uniqueness checks. To maintain scientific rigor while preserving large-scale data volume, we randomly selected 20% of samples from the cleaned theoretical dataset and invited two independent experts in catalytic materials for manual verification. The review process employs a bhiule cross-evaluation approach to ensure conclusions are not influenced by individual reviewer preferences and to achieve explicit consensus on answer consistency assessments.
A.5.2 OPEN-ENDED Q& DATA CALIBRATION
For open-ended comprehension Q&A data sourced from experimental literature and automatically generated by SciQAG-guided GPT-4o, we implement an additional rule-based filtering system to remove potentially noisy article-internal reference questions. This includes questions relying solely on non-generic knowledge cues like "this paper" or "this study." This filtering effectively mitigates the risk of model over-reliance specific textual contexts during training. Furthermore, during the generation phase, we avoid a single-round direct instruction-to- -ouuuut approach. Instead, we design multi-round generation process: first extracting keywords and key mechanisms, then guiding the model to generate Q&& focused on core scientific concepts, followed by structured rewriting and semantic consistency review. This multi-round strategy significantly mitigates GPT-4o's tendency toward fixed stylistic expressions and non-scientific rhetoric in lengthy responses. Consequently, the retained data exhibits greater expressive diversity, enhanced scientific rigor and high consistency with actual catalytic mechanisms.
We further introduce verifiable knowledge attribution mechanism to reduce and identify poten- tial ballucination content. During the extraction of contextual keywords, we concurrently build an internal retrieval database comprising three elements: 1) We retain the unique DOI identifiers of corresponding literature to ensure each Q&A pair can be traced back to its original publication; 2) We preserve the complete contextual paragraphs underlying generated questions for offline review and revalidation; 3) We associate extracted core scientific keywords with their corresponding text fragments, establishing a seyword-context-De DOI mapping. This structured database not only pro- vides additional quality control measure beyond expert review for open-ended Q&A data but also enables rapid programmatic verification of answer consistency with original sources. This maintains the high standards for scientific verifiability and reproducibility of datasets.
21

---

Published as a conference paper at ICLR 2026
We also randomly sample 50% of the dataset and conduct manual verification by experts in the field of catalysis. We invite several experts from the National Nanotechnology Center's catalysis field to participate, including associate researcher and 6 doctoral student. The review process covers three central aspects: 1) The reasonableness and scientific value of the open-ended Q&A task questions, whether they reflect the development trends and focal points of the catalysis field; 2) The accuracy and completeness of the answers, ensuring they include all key conditions; and 3) The relevance and contextual fidelity of the Q&A tasks, ensuring there are no hallucinations generated by the model itself. After an initial filtering, approximately 90.8% of the Q&A tasks are identified accurate and scientifically meaningful. To further validate the rigor of our filtering process, conduct IAA analysis on 500 randomly selected samples. Two domain experts independently annotate each sample as accept or reject achieving a Cohen's 's Kappa coefficient of 0.75. This further confirms that the dataset reflects domain knowledge rather than model bias.
A.6 ERROR SAMPLES
The errors in automatically generated open-ended Q&A responses primarily stem from models in- troducing inferences inconsistent with mainstream consensus the catalysis field, mismatched ex- perimental conditions, or unfounded explanations of reaction mechanisms during answer generation. To clarify the feature of this issue, we present several representative error samples, all annotated and verified by two independent domain experts.
Error Sample 1
Question:
What is the specific role of Zn sites in the CO electro-reduction process within this Cu-Zn bimetallic catalytic system?
Reference Answer:
The Zn site serves as an efficient active center for CO generation, selectively reducing CO the #CO intermediate and synergistically promoting subsequent C-C coupling with the Cu site to form ethylene.
The answer generated by LIM:
Zn sites directly participate C-O bond cleavage, promoting the reduction CO to CO while enhancing the selectivity of Cu sites toward ethylene.
The error causes:
The model crroneously interprets the Zn site's action as directly participating in bond cleavage, which contradicts the explicitly documented mechanism in the literature of "indirect action via intermediate generation." This constitutes a mechanistic error.
22

---

Published as a conference paper at ICLR 2026
Error Sample 2
Question:
What the primary contribution of nitrogen doping when employing an electrocatalytic system supported by nitrogen-doped graphene for Fe single-atom active sites?
Reference Answer:
Nitrogen doping alters the local electronic structure, increases the electron density at active sites and improves reaction kinetics by enhancing adsorption capacity.
The answer generated by LLM: Nitrogen doping enhances the carrier conductivity, enabling to maintain structural stability under high current densities.
The error causes:
The generated answer unduly emphasizes enhanced conductivity while overlooking the dominant effects verified the literature: regulation of active site electronic structures and improvement in adsorption capacity. This error constitutes critical omission of key factors.
Error Sample 3
Question:
What is the primary purpose of introducing nanoporous structures in Au-Cu alloys for CO electrolytic reduction?
Reference Answer:
The porous structure increases the specific surface area and the number of exposed active sites while improving the mass transfer efficiency of reactants.
The answer generated by LLM:
Naphthalene-based structures are primarily employed to suppress the formation of methane byproducts, enhancing selectivity through altered electron cloud distribution.
The error causes:
The response focuses solely on selective enhancement and methane suppression as its sole objectives, lacking descriptions of surface structure and mass transfer effects. This results insufficient scientific rationale and inconsistency with the original context keywords.
A.7 DATA DISTRIBUTION AND DIVERSITY ANALYSIS
To comprehensively demonstrate the composition of CatalystBench, this section conducts detailed statistical and diversity analysis of the dataset. We aim to reveal its coverage breadth and data distribution characteristics across dimensions such as catalyst types, regulation method types and reaction types through quantitative data, thereby validating its rationale and challenge as a com- prehensive benchmark. Fig illustrates the distribution characteristies of the catalysts and reaction types involved in CatalystBench.
Distribution of Key Metal Elements: We compile the key metal elements in catalysts across the dataset, particularly in tasks derived from experimental literature such as ME, SE and RMC tasks. As shown in Fig 9a| Cu is the most frequently occurring element, accounting for 43.7%. This aligns closely with our data sources" focus the current state of research in CO2 electrocatalytic sopper-based based catalysts represent a major research focus. Simultaneously. the dataset extensively includes transition metals such Fe, Co and Ni, which play crucial roles in electrocatalysis, along with various other metallic elements. This ensures the model can learn diverse elemental knowledge.
(CanettionnC,RRRR where copper-
23

---

Published as a conference paper at ICLR 2026
Others adsorption,
etc)
F.3
Caraal
oxygen evolution reaction(OER) and orygen reduction
NAT
Noa Nooiie
electrocatalytic reduction(co,an)
reaction(ORR)
(a) The statisics of cat- alyst diversity data.
(b) The statisics of catalyst regulation methods.
(c) The statisics of electrocatalytic reaction type.
Figure 9: The distribution characteristics of the catalysts and reaction types involved in Catalyst- Bench.
Distribution of Catalyst Regulation Methods: CatalystBench incorporates variety of advanced catalyst design and control strategies. We statistically analyze 2,364 samples from the RMC task. As shown in Fig 9b Structure control and Composite materials are the two most prevalent approaches, accounting for 46.5% and 22.3% respectively. This reflects the current mainstream trend of en- hancing catalytic performance through constructing heterojunctions and multi-component synergies. Additionally, Surface modification, Alloying, Defect engineering and Atomic-level dispersion also account for significant proportions, comprehensively covering key technologies from macroscopic morphology to atomic-scale regulation.
Distribution of Reaction Type: CatalystBench encompasses several core reaction types in the clec- trocatalysis field. As shown in Fig 9c CO2 electrocatalytic reduction(CO;Rk) constitute the largest proportion of the dataset, accounting for 64.3%, directly attributable to our integration of multiple literature sources and datasets focused on CO2RR. Simultaneously. by incorporating datasets like SACs and OC20, we have also included substantial amount of theoretical calculation data related to oxygen evolution reaction(OER) and oxygen reduction reaction(ORR) combined as 22.6%. Addi- tionally, small number of other surface adsorption and catalytic reactions included, collectively forming diverse set of reaction scenarios.
24

---

Published as a conference paper at ICLR 2026
B EXPERIMENTAL DETAILS
B.I DOMAIN-SPECIFIC CHALLENGES IN CATALYSTBENCH MULTI-TASK LEARNING
CatalystBench presents a unique configuration of tasks that diverges substantially from typical multi-task learning settings, primarily due to the heterogeneous nature of inputs, outputs and the rich domain-specific dependencies intrinsic to catalytic science. Unlike general-purpose multi-task benchmarks where tasks may share uniform modality and output format, the catalytic design work- flow necessitates the simultancous handling of mixed input modalities comprising structured chem- ical descriptors, unstructured scientific text and numerical parameters. These inputs range from SMILES strings, atomic coordination tables, or facet specifications to free-form descriptions of syn- thesis methods and regulation schemes extracted from literature, creating a complex representational space in which feature alignment across modalities is non-trivial.
The heterogeneity further extends to output modalities. Regression tasks such as Adsorption Predic- tion, d-band Center Prediction and Faradaic Efficiency Prediction demand high-precision numerical values. Classification tasks, exemplified by Regulation Method Classification, require discrete cate- gorical decisions grounded in chemical taxonomy. Generative tasks, including Material Extraction and Regulation Scheme Comprehension, produce long-form textual responses that must balance flu- ency and factual correctness while integrating domain knowledge. This diverse set of output spaces imposes fundamentally different optimization landscapes; mean squared error (MSE) objectives for regression tasks often operate on much larger numerical scale than cross-entropy losses used in classification, or token-level negative log-likelihoods in generation. Without explicit control mech- anisms, the disproportionatelyy high gradients from regression losses can dominate joint training, suppressing learning in classification and generative objectives, a phenomenon that is particularly detrimental in scientific settings where each capability is equally critical.
Moreover, tasks within CatalystBench are not independent in their knowledge requirements. The Regulation Method Classification task, for instance, provides crucial mechanistic context that di- rectly influences downstream predictive accuracy in Faradaic Efficiency estimation. Similarly, un- derstanding textual regulation schemes is often prerequisite to correctly interpreting property trends. These upstream downssream dependencies mean that multi-task learning in this domain must not only avoid interference between tasks, but also preserve and exploit beneficial cross-task syner- gies. Naively unifying outputs in shared generative head, as in the FT setting, disregards both the modality-spede sppecifcc optimization needs and the asymmetric informational flow between tasks, often leading to diluted performance gains. Conversely, the MST approach, while introducing modality- specific heads, treats each task in isolation during training, forfeiting the representational transfer that arises from shared backbone exposure the full diversity of inputs.
Empirical evidence from FinJJunderscores these points. The FT configuration suffers notable degra- dation in numerical prediction tasks, reflecting loss-scale interference and insufficient modality dif- ferentiation, whereas MST exhibits limited average improvement due to the absence of multi-task semantic alignment. By contrast, the MFT strategy achieves the highest mean improvement rate of 12.44% over the ST baseline, leveraging joint backbone training to capture shared domain features, while decoupled output heads mitigate loss conflict across heterogeneous modalities. These findings substantiate that the challenges in CatalystBench, such as input heterogeneity, output space diver- gence, loss imbalance and domain-specifie task coupling necessitate an architecture that is simulta- neously modality-awan and synergy-preserving requirement uniquely fulfilled by the proposed MFT tunint framework.
B.2 BASELINE LLMS
The following sections will introduce the specific open-source LLMs and general closed-souree LLMs we evaluate in the main text. The comparison results are presented in the main text.
LLaMA-2 Touvron al. (023)) is set of large language models with parameter ranges from 7 billion to 70 billion. The model architecture remains largely unchanged from LLaMA-I, but over 40% of the data is used to train the base model. Specifically, LlaMA-2 includes pre- tre-trained and fine- tuned models optimized for conversational applications and we have chosen to use LLaMA-2-7B as the model for comparative evaluation.
25

---

Published as a conference paper at ICLR 2026
Mistral Jung et al. (2010) is open-source large language model with 7 billion parameters, de- signed to offer high efficiency and performance on a wide range of downstream tasks. The model features transformer architecture with various optimizations tokenization and training data di- versity. In this study, we select Mistral-7B for comparative evaluation.
ChatGLM GLM al. (2024) is bilingual large language model containing 6 billion parame- ters. It adopts the General Language Model (GLM) architecture and is optimized for dialogue and question-answeringg scenarios. ChatGLM-6B is pre-trained on extensive Chinese and English cor- pora, enabling strong cross-lingual generation capabilities.
Qwen3 Yang et al. 22255) are trained on diverse multilingual datasets improve performance across languages and domains, The model is optimized for multilingual and multitask settings, benefiting from large-scale pre- pre-training. For evaluation purposes, Qwen3-8B is selected representative of mid-sized models in the Qwen3 series.
CLemLLM Zhang et al. (2024a) is an open-source language model deeply optimized for the chemi- cal field, with 7B parameters. ChemLLM has converted vast amounts of structured chemical knowl- edge sources into over 40 million high-quality single-round Q&A pairs, known as ChemData. By fine-tuning on this large-scale instruction dataset, ChemLLM can learn a wealth of chemical facts and understand and execute diverse chemical tasks through seamless conversational interactions.
Darwin1.5 Xie et al. (2024) is open-source foundational LLM tailored for material science and chemistry. The model is designed with a two-stage training strategy, namely question-answering (QA) fine-tuning and multi-task learning, to enable LLM to proficiently perform chemical and ma- terials tasks.
Claude-3-haike is one member of the Claude-3 series of commercial large language models devel- oped by Anthropic. It is designed to provide fast, safe and helpful conversational abilitics, partic- ularly for business and enterprise applications. Being closed-source model, interactions with Claude-3-haiku are conducted via API, employing few-shot inference to ensure fair and efficient evaluation.
Deeniii225-fflsh--rrevvv Comanici et al. (2025) is a pre-release version of Google's Gemini-2.5 series, offering improvements in dialogue understanding and response speed. As closed-source model. we access Gemini-2.5--1 2.5-flasb--reviii through its API, enabling batch evaluation for compar- ative studies while maintaining low overhead.
Deepseek-v3 Liu et al. (2024)) is the third-gencration model from the Deepseek series, developed enhance code understanding, generation and natural language interactions. With advanced pre- training strategies on code and text data, Deepseek-v3 demonstrates strong capabilities across mul- tiple tasks.
GPT-3.5-turbo and GPT-4. For closed models such as OpenAl GPT Family GPT-3.5-turbe Ye et al. (2023) and GPT-4/Achiam et al. (2023), we employ batch inference via APIs for conducting few-shot prompt inference. This approach significantly enhances evaluation efficiency and reduces overhead.
B.3 BASELINE TRADITIONAL ML METHODS
CatBERT Ock et al. 2023) is a Transformer model based on the RoBERTa architecture, specif- ically designed for predicting adsorption energies in catalytic systems. It can replace traditional graph neural network(GNN) methods, which rely precise three- firre-dimensional atomic coordinates, by using human- readable text inputs that describe the "adsorbate- cadsorbate-catalsst" system. By processing text containing key information such adsorbate types, catalyst host material compositions and interacting atoms, CatBERTa achieves accurate predictions of catalyst adsorption energies.
GAP-CatBERT Ock et al. (2024) improves the accuracy of catalyst property prediction through graph-assisted pre-training strategy. During the pre-training phase, the model utilizes a powerful graph neural network, EquiformeVV2 Liao et al. (2023); Liao & Smidi (2022), to generate high- quality graph embeddings from precise atomic structures. While the CatBERTa model generates text embeddings from corresponding textual descriptions. Subsequently, this method aligns the embeddings from these two different modalities through contrastive learning, thereby transferring and injecting the rich structural knowledge contained in the graph model into the text model. The
26

---

Published as a conference paper at ICLR 2026
resulting GAP-Catee CatBRaT achieves respectable accuracy in adsorption energy prediction tasks using only text input.
GPTchem Jablonka al. (2024) is an effective prediction method which combines the power- ful representation capabilities of LLM with Gaussian Process Regression(GPR) strategics. In this framework, the model uses LLM fine-tuned by massive amounts of chemical text to convert discrete chemical entity information into feature embedding vectors, which are then fed into a GPR model. GPR can not only predict specific chemical properties of molecules such as formation energy, but also provide a quantified confidence interval for the prediction results through Bayesian inference.
CGCNN Xie & Grossman (2088) (Crystal Graph Convolutional Neural Network) is a graph convolutional neural network (GCN) model specifically designed for crystalline materials. Within CGCNN, the crystal structure is abstracted as a crystal graph, wherein atoms serve the graph's nodes and chemical bonds between atoms represent the graph's 's edges. Node features typically rep- resent invariant properties characterising atomic species. By stacking graph convolutional layers and pooling layers, this model learns both local and global features within the crystal structure, ulti- mately enabling accurate prediction of various physicochemical properties such as formation energy and bandgap.
SchNet Schüit et al. (2018) is a deep learning model specifically designed for molecular and pe- riodic structures, aiming to learn potential energy surfaces and quantum chemical properties. At its core, SchNet employs continuous filtering convolutional layers to process atomic environments, with these convolutional operations being rotationally and translationally invariant. It progressively refines atomic features through interaction blocks that utilise interatomic distance information. Cru- cially, SchNet introduces Gaussian Radial Basis Functions (RBFs) to represent interatomic dis- tances, enabling the model to capture long-range interactions that vary continuously with distance. The model efficiently and accurately predicts atomic energies, forces and other quantum chemical properties.
DimeNet++ Gasteiger al (2020a) geometric deep learning model that builds upon the DimeNet (aanteiiGastegeerr et al. (2020b) framework, focusing on capturing angular information and tripar- tite interactions within molecules. Unlike traditional GNNs that solely consider atoms and bonds, DimeNet++ introduces Directional Message Passing mechanism. It accounts not only for the connection from atom to j. but also for the message transmitted from atom k to i via j. thereby explicitly encoding the angular information formed by atoms i, j and k. This explicit handling of the three-bodi term enables DimeNet++ to achieve higher accuracy and rotation invariance when predicting molecular properties, while simultancously enhancing computational efficiency through optimised architecture.
GemNet-OC Gasteigger et al. (2022) is a sigh--prrfomance graph neural network model, specifically designed as a variant for catalyst adsorption system architectures. Building upon DimeNet and SchNet, it further emphasises the utilisation of geometric information. By integrating explicit three- body terms and higher-order angular information, this model accurately characterises the intricate interactions between adsorbates and surface atoms catalysts, thereby enabling high-precision predictions of key catalytic properties such as adsorption energies. It stands as one of the state-of- the-art models in the field of catalysis that relies on atomic coordinate inputs.
Faradaic efficiency prediction is a relatively new property prediction task in the field of catalytic materials. Gao al. Gao al., (2023) construct a knowledge graph of electrocatalysts based on sci- entific literature and propose a deep learning- based prediction model, which integrates the semantic information from the scientific literature (word embedding) with the correlation of knowledge triples (graph embedding) and realizes the prediction of the Faradaic efficiency for targeted case.
B.4 EVALUATION AMONG OPEN-SOURCED MODELS
Previous studies have shown that general LLMs often perform poorly when handling tasks requiring deep understanding of chemical structures, such as SMILES strings and complex chemical reason- ing, as they lack internalized, domain-specinn
knowledge systems Bagal et al. (2021): Liu et al. 2024b); (Cang al. (2024). To select an appropriate base model, systematically evaluated multiple existing open-souree LLMs with 7B/8B parameters on representative tasks from Catalyst- Bench. The evaluation tasks encompass 3 core capabilities: structured numerical prediction, text
27

---

Published as a conference paper at ICLR 2026
classification and text generation. This approach aims to comprchensively characterize the models' reasoning and generalization abilities across theoretical and experimental data conditions. The Re- sults are shown in Table which demonstrate that ChemLLM-7B maintains stable and outstanding performance across most tasks, exhibiting significant advantages over other candidate models partic- ularly in reaction mechanism analysis and catalyst performance prediction. Therefore, designate ChemLLM as the base model for fine-tuning to ensure subsequent experiments build upon a robust foundation of comprehensive performance.
The decision to limit the model size to small parameters stems from a balanced considera- tion of computational resources and applica- tion scenarios. On one hand, a 7B/8B-scale model can undergo multi-task fine-tuning on typical GPU clusters with relatively low train- ing overhead, ensuring reproducibility and gen- cralizability throughout the research process, On the other hand, this parameter scale al- ready achieves satisfactory task performance chemistry and materials domains while main- taining high operational feasibility for practical deployment.
Model
IE
RMC
AP
ChemLLM-77
0.86
0.52
0.63
Daswin1.5
0.84
0.48
0.59
LLaMA2
0.76
0.38
0.24
Qwen3-8B
0.80
0.41
0.31
Table 5: The performance of four open-source LLMs of similar scale on representative tasks in CatalystBench.
B.5 EVALUATION WITH TRADITIONAL ML METHODS
We compare LM with traditional ma- chine learning methods on property predie- tion regression tasks in CatalystBench. The ML results for each task are obtained using specific single-task data. Taking the adsorp- tion energy prediction task as an example, both the atomic
Model
AP
d-CP
FP
CGCNN
0.79
0.66
0.78
SchNet
0.72
0.65
0.77
DimeNet++
0.84
0.69
0.80
GemNet-OC
0.85
0.72
0.84
CatBERTa and GAP-CatBERT: text information of the catalyst, which is also the generic text in the template Appendix D.5 as input to predict the adsorption energy value of the catalyst. These ML algorithms typ- ically only accept one input format and do not have results for all tasks. For the AP, d-CP and FP tasks, we introduce mainstream graph neu- ral network benchmark models currently preva- lent in materials science, including CGCNN, SchNet, DimeNet++ and GemNet-OC. These models widely regarded as standard bench- marks in adsorption energy prediction benchmarks such as OC20. We compare CatalystLLM against publicly available checkpoints of these algorithms, all of which are trained the same data set size as CatalystBench. is particularly noteworthy that these graph neural network models rely on com- plete atomic 3D geometric coordinates and lattice parameters as inputs. They explicitly encode 3D spatial structural information through message-passing mechanisms to achieve high-precision pre- dictions. In contrast, LatalystLLM is limited textual descriptions extracted from experimental literature or databases, along with simplified structured text sequences such as chemical formulas, crystal plane information and adsorption site descriptions. Additionally, due to differences input representation formats, not all tasks have MI baselines, such as text classification and semantic un- derstanding tasks. For information extraction tasks, L.MS like the GPT series are more accurate than traditional entity extraction methods. The detail results of evaluation is shown in Table 6 and Table 7
CatBERTa
0.82
/
GAP-CaIBERTa
0.86
/
/
GPTchem
/
0.69
0.85
CatalystLLM (ours)
0.81
0.73
0.80
Table 6: The results of CatalystLLM compared with machine learning baselines for 3 prediction tasks. The best model is in bold font and the second-best is underlined.
As can be seen from Table 6 the advantage of CatalystLLM over traditional ML methods lies in its exceptional versatility. It can provide effective predictions across a wide range of task types, whereas other models typically focus one two specific tasks. In terms of specific performance, CatalystLLM achieve the best score of 0.73 in the d-CP task, outperforming GPTchem. In other tasks, CatalystI. LLM also demonstrate strong competitiveness. For example, its adsorption energy
28

---

Published as a conference paper at ICLR 2026
prediction score is 0.81 1 and close to SOTA model GAP- Cat-EaaBR In summary, while highly op- timized specialized models may perform better in certain single tasks, CatalystLLM demonstrates its immense value as efficient catalyst design LLM through its balanced and competitive perfor- mance across a wide range of catalytic prediction tasks.
Notably, when compared against graph- structured prediction models, CatalystLLLM's predictive accuracy falls slightly below that of top-tier graph models like GemNet-OC and DimeNet++, which rely on precise geometric structures. Moreover, CatalystLLM outper- forms earlier SchNet models. This indicates LMMs ability to effectively capture chemical knowledge and structural features implicit in textual descriptions through multi- task fine-tuning. Crucially, widely-used GNNs require 3D coordinates derived from costly DFT relaxations. CatalystLLM achieves com- parable accuracy using only text information. This enables instantaneous inference on vast chemical spaces where 3D structures are unknown. For the FEP prediction task, we introduce traditional text prediction methods from the current scientific domain. As shown in Table 7 although it falls short of specially trained state-of-the-an models for this task, CatalystLLM sigmificantly outperforms other ML approaches, demonstrating considerable potential in this domain.
Model
MSF
MAE
R-
multi- layer perceptron
0.03
0.15
0.48
support vector regression
0.06
0.20
0.11
linear regression
0.05
0.17
0.20
BRR
0.05
0.18
0.23
GPR
0.04
0.15
0.34
original paper
0.01
0.08
0.84
CatalystLLM
0.02
0.13
0.73
(ours)
Table 7: The results of CatalystI compared with machine learning baselines for FEP task. The best model in bold font and the second-best is underlined.
B.6 ADDITIONAL ABLATION STUDY
In addition to the ablation experiments described in the main text, we also compare the impact of different experimental settings model capabilities to explore the crucial elements of LLM fine- tuning experiments.
B.6.1 THE IMPACT OF WEIGHTED LOSS-FUNCTION ON MODEL PERFORMANCE
This experiment aims verify the critical role of weighted loss-function calculation in multi- head Full-task fine-tuning setting. In the ex-
Performance Score (Higher is Better)
1.0
0.98
MFT:Weighted Aeg (0.83)
NFT.Jnwwigtiad
MET:Weighted
perimental setup described in the main text, we manually adjust the weights based on the performance of CheMLLM-MST model the validation set for each task to balance the con-
0.9
0.09
0.51
0.8-
tribution of different tasks to the model gradi-
0.75
21
5/0.3%
ent update. In addition, we use an unweighted loss-function the Multi-head architecture, in which the weight coefficients A for all tasks are set to 1.
0.7
Fig 10 shows the comparison of the results of the two of experiments. MFFT-wiihhted demonstrate better performance in most tasks, proving that loss weighting is necessary step to achieve -egforerform multi-task learn- ing. Among them, MFT-Unweighted shows a significant decline in performance in tasks such as classification and information extraction. This may be because in multi-task learning, the numer- ical scales of the loss functions of different tasks often differ by orders of magnitude. For example, the mean squared error in regression tasks has a larger scale than the cross-entropy loss in classi- fication tasks. leading the model to prioritize regression tasks during back pro-ppropggttion Therefore, we need to set different weight parameters to balance the contributions of different tasks to the to- tal gradient, thereby maintaining the multi-task capability of CatalystI L.M. In addition, possible
ME
SE
RMC (F1)
FEP
AP
d-CP
FP RSC
(F1)
(F1)
(R2)
(R2)
(R2)
(R2)
(STS)
Figure 10: The impact of weighted loss-function
of MFT on model performance.
29

---

Published as a conference paper at ICLR 2026
future optimization direction is set the weight parameters as hyperparameters and dynamically adjust them during model training.
B.6.2 OOD SEEVE-ONE-TASK-OOE EVALUATION
To systematically evaluate CatalystI.LMM generalization capabilities on unseen tasks, we design a Seave-One-TTak---e
experimental strategy. Building upon [Section 6.3] of the main text, we further conduct out-of-domain performance analysis for specific tasks. Considering the prior rela- tionships among the 8 CatalystBench tasks pasks-lartieularly the strong correlations imput modalities and knowledge dimensions for certain tasks, we select the Faradaic Efficiency Prediction (FEP) task as the holdout set. This task is excluded from model training and only evaluated during the inference phase. This task belongs to the numerical regression category, with inputs derived from structured catalyst characterization and performance data in experimental literature. It exhibits po- tential eestreamm-donssreean
logical connections with tasks like Regulation Method Classification (RMC) and Regulation Scheme Comprehension (RSC). FEP prediction requires comprehensive un- derstanding and inference regarding the regulation methods, structural features and corresponding intermediate formation processes of the catalyst system.
This experiment aims to validate whether LatalystLLM can retain certain level of performance through cross-task transfer learning and shared representation modeling, even under training condi- tions with complete absence of task-specific data. We compare CatalystLLLM with ChemL-LM-FT and representative seneral-purpon open-source models LLaMA2-7B and Mistral-7B under iden- tical data partitioning and inference settings. All models are fine-tuned without FEP task training data, using the remaining seven tasks as training sets. The results are shown in Table g
Experimental results indicate that under the FEP-withheld setting, CatalystLLM still signif- icantly outperforms baseline models on both
Model
R²
MAE
ChemLLM-FT
0.58
3.21
R² and MAE metrics, representing an improve- ment of approximately 0.11 over ChemL.LM- FT and exceeding 0.20 compared to general- purpose models. The outcome indicates that FEP task performance largely benefits from knowledge transfer and sharing within the multi-task architecture, particularly contribu- tions from structural regulation tasks in terms of semantics and patterns. Furthermore, de- spite performance declines compared the original setting with FEP training data, Catalyst still generates predictions with physicochemical plausibility. This demonstrates that its generaliza- tion capability relies not only on task matching but also on the internal organization and expression of domain knowledge.
LLaMA2-7B
0.49
3.87
Mistral-77
0.51
3.75
CatalystLLM 0.69
2.84
Table 8: The experiments on OOD Leave-One- Task-Out evaluation on FEP task
30

---

Published as a conference paper at ICLR 2026
C EVALUATION DETAILS
We conduct a multi-dimensional evaluation of open-ended Q&A tasks focused on understanding and explaining catalytic regulation schemes.
C.1 LLM-BASED EVALUATION
The RSC task aims to evaluate the model's deep understanding of catalytic regulation schemes and its text generation capabilities. The output is segment of natural language, rather than fixed labels or numerical values. Traditional automated evaluation metrics, such as BLEU Papineni Papinnnn et al. (2002) or Li 2004) based on phrase overlap rates, cannot effectively measure the factual accuracy of answers. For example, an answer may use different words from the standard answer but still be semantically correct and conversely. Therefore, we use gpt-4o and deepseek-rl as evaluation models M. Bran al. (2024). They prompted to assume the role of catalytic field expert, but has access to external tools such as internet browsing. We collaborate with domain experts to design a set of evaluation prompts so that LLM would imitate scientists in their thinking patterns and give scores in different dimensions: 1) Reasonableness assesses whether the content generated by the model follows basic scientific principles and causal relationships; 2) Accuracy assesses whether the technical terms used in the model output are consistent with recognized scientific knowledge; 3) Usability assesses whether the answer generated by the model is practical the question and has certain degree of reality and operability. The LLM-based evaluation prompt we designed is shown in Appendix E3
C.2 EXPERT EVALUATION
Reasonableness 0.87
Accuracy 0.75
Expert
LLM Usability r= 0.92
LLM Score Weighted Total 0.89
10
LLM Score
LLM Score
Figure 11: The sub-metrics and the overall Pearson score between LLM Score and Expert Score.
Although general LLMs have vast knowledge reserves and powerful reasoning capabilities, they have prevalent issue: the generation of hallucinations such incorrect or fabricated information
31

---

Published as a conference paper at ICLR 2026
presented as factual Xu et al. (2024). Therefore, invite domain experts evaluate the model's s performance on a test set of 100 samples. We observe a clear trend of consistency between LLM scores and expert ratings across the overall distribution of these samples. The Pearson correlation coefficient for the average scores of the 100 samples is 0.89, indicating a high degree of linear cor- relation in the numerical variations between the two types of scoring. We further analyze the sub- metrics and the overall score. As shown in Fig [I] samples rate highly by the LLM also tended to receive relatively high scores from experts, with a similarly pronounced consistency observed for low-scoring samples. This outcome demonstrates that despite divergences in assessing the accuracy of specific procedural details, both evaluation systems exhibit close alignment in judging overall answer quality trends. This consistency provides practical support for leveraging prompt-optimized LLM scoring during initial data screening and offers statistical justification for subsequent integra- tion with expert review to achieve efficient quality control.
Regarding the variability in human scoring, Fig 12 shows the differences between LLM evalu- ation and human expert evaluation. Although human experts prefer CatalystLLM'ss responses based on the accuracy of catalytic materials and corresponding mechanisms, LLMS prefer GPT- 4.1's 's answers, typically basing their evaluation on the fluency and apparent completeness of GPT-4.1 responses. Therefore, we focus on ex- cluding GPT 4.1's answers that are complete and fluent enough to cover the question but contain incorrect information. Fig [3 shows an example of an illusion in LLM-based score and expert score. GPT4. 's answer achieves higher LLM-based score, but it makes inferences that violated the principles of catalysis, which is "K* directly participates in C-O bond cleav- age". In the field of electrocatalysis, there is non-covalent interaction mechanism whereby alkali metal ions stabilize intermediates through electrostatic field effects rather than directly partici- pating in bond formation or bond breaking. In general, LLMs tend to favor answers that are formally rigorous and logically coherent. However, in scientific evaluation, the correctness of the answer is more valuable than complete and detailed explanation. For scientific tasks requiring real-world knowledge, LLM--baeed evaluation methods cannot replace expert human assessment for now.
Aggregate evaluation scores
GPT44.1
6
Score
CatalystLLM
GPT-4.3
Reasonaaleness
Accuracy
Usability
Expert senne Export Score
LLM-based Score
Figure 12: Average scores for gpt-4o and deepseek-rl across three metrics and aggregated results from expert evaluators (n=100) across all tasks. Error bars indicate confidence intervals (95%).
C.3 CASE STUDY: UNIQUE CORRECT REASONING IN COMPLEX CATALYTIC CONTEXT
To more intuitively demonstrate CatalystLLLM's s domain adaptation capability, we select a test case that only LM could answer completely and correctly among all comparison models, which is shown in Appendix D.7 This task belongs to the Regulation Scheme Comprehensson((SCC category, involving high-order coupled multi-metal catalytic system whose underlying mechanism was not directly present in the training data.
Among all comparative model responses, CatalystLLM uniquely link the inferred product shift ward methane and hydrogen to the synergistic mechanism involving *CO production at the Zn site. accurately identifies that losing the Zn site reduces intermediate concentration and alters the reac- tion pathway. thereby disrupting the essential conditions for ethylene formation. This response not only aligns with established physicochemical principles but also resonates with mainstream under- standing of electrocatalytic mechanisms at the mechanistic level, demonstrating the model's ''s capacity for abstract reasoning when inferring complex sequential mechanisms. This case demonstrates that CatalystLLM can leverage cross-task learned catalytic reaction knowledge to make domain-logine inferences even without direct case training. This holds significant reference value for chemical en- gineers and catalytic scientists exploring novel multifunctional catalyst designs. More importantly, this case reflects the unique potential of nomain-adapter LLMs in supporting high-level scientific problem-solving. Their accuracy stems only from data coverage but also from the learned rep- resentation's 's ability to capture and structurally express catalytic principles during fine-tuning.
32

---

Published as a conference paper at ICLR 2026
Intraction: f-fe-Mk- crtastic system, how asecticaal ce petmuaum lors (K), enhence the dedroctallyicc reduction co, con antrrgen dogeed grophree supsort peffemence, potardium ccoppoata
Truth QA pair
lopant catllys sigle monoodde (P0).
promotes
desrersddctionn
A_ Aawweer Potasdum len (C) sgnifiaatly enhance the citalyic converson of COp through severa syreggate mechansims mu:
aaaswen ti caalsit, potasiimm orhance
directly nicipote diemical bord
insteed accumatite the viretty active cites during the reartinn This loca lir righ- soncentrution cataant fie effectivdl subiiles boy resction imme mediioes portinuadly neger only charged spedis
singe stam
Pantidpation
Cit, mulccull forming trrnsknnt Fo C. 6. autive voz mesiitt This evong electrottte sttrect on dramraticlly weakens the c bord, sigrificanlly lower og the encrgy burrier for bond dearage thus directty prometing the nor mation co. accumdaation
loes directly coerdiaae with
aetivarinn
mection pattnaay, which monttetts
opdimiaatton
maprosopiaall lightr reuction rite and groate to towards the CD product.
charged key reection intermadiins and impraaee the local monpenttation of at the wantion invebire Medulation of Blettroiic Structure: The prosenne of indirectly influennes electics di dene of the sunounding autive slss, making then more favorabre achorpe
Reosinatieeess Frosh Total Scores
summany potassium lans mutifnational promotes
GPT-4o
with their core ae be ng direct part piriaiio in and catalpis of the O bond broaking process.
CatalystLLM (ours)
Mehhoggh Moder asseer u conctss, capturec sneetiff patrre the porectium laaruca "oon direct ohctrosiats, foll affest" and the dtatubation of the gegpeellyy charped "cock intereedloos, whish & Jully conoistnt with she macstivam sclentate sorrrmuus paid. ouser descestrates high to comples mechathima"
comessly
Accursoy: & boo 10 Noou Store: B-d
"whtt Madel resporse structaretly cleer and opperes romporeencis, Ie now exgumeen contatrs e fotal suicttll erie to soucrice thot "notoro ure ioes dosetty pentopere C-d beot deawagge severety concoulttss folfff eletrocattlsis meter rend cnteradbeter phrough elecroooel fiest offers rother than direclly porthopsting bond formatio clecrage Ih's Aunderernn conneptuct reeder toe entiee asswer not volaokss sto hightr
GPT-4o
CatalystLLM
Answer integrity
Innovative insights
Major hallucinations
Reasonable explanations
Inaccurate explanations
Occasional flawed answers
and
profecalonal hameaarr, seith
contuiig for specialtss
Figure 13: For the example truth Q&A pair of the same regulatory scheme semantic understanding task, the LLLM-based scores and expert scores of answers given by GPT-4.1 1 and CatalystLL.MM The yellow highlights indicate positive evaluations and the green highlights indicate negative evaluations.
D QUESTION TEMPLATES
In recent years, there has been a surge in the application of machine learning to chemistry, resulting in a wealth of datasets and benchmarks in chemistry and material field. However, few of these benchmarks focus on assessing LLMs for tasks specific to Catalytic Science and standardized evaluation technique has not yet been established. This section provides concrete examples for each catalytic science task category, demonstrating how to formulate inputs and showcasing the model's expected outputs for representative problems.
D.1 LABEL DEFINTTION
<Contexts>>: The text describing catalytic material control schemes synthesis schemes in scientific literature. e.g. "The synthesized Cu catalysts show relatively high crystallinity dominated by (111) facets, promoting the generation of CH4."
<Material representation>>: The actual representation of catalytic material typically shows its elemental composition or structural details, including SMILES strings, chemical formulas, CIF and other forms of existence. e.g. Cu2Pd.
<Operations>: The common operations in catalytic material synthesis schemes include ADDING, HEATING, CURING, ELECTROCHEMICAL ANODIZATION, FILTERING.
33

---

Published as a conference paper at ICLR 2026
DRYING, DIPPING and REACITON. e.g. The reaction mixture was then heated at 85 C with constant stirring for h.
<Operation conditions>>: The relevant attributes of catalytic material synthesis operations, including heating temperature and duration. c.g. 85 OC and h
<Property>: The characteristies or properties of catalytic material describe its chemical characteristics and crucial performance, including adsorption energy, d-band center, formation energy and actual catalytic Faradaic efficiency. e.g. 90%.
D.2 MATERIAL EXTRACTION TEMPLATES
Material Extraction Templates
Instructions:
What specific materials are mentioned as catalysts in the text?
List all keywords that refer to catalyst materials their types discussed in the article.
Which terms in the document are associated with or describe catalyst materi- als?
Input: <Contexts>
Output: <Material <MMateiial.eppreeentaae representation>
34

---

Published as a conference paper at ICLR 2026
D.3 SYNTHESIS EXTRACTION TEMPLATES
Synthesis Extraction Templates
Instructions:
Round 1: What is the target material synthesized in the given text?
Input: <CContext>>
Output: The target material synthesized in the text is <Materiillepprretttttioon>>>>>>>
Round List the main experimental operations described in the synthesis process.
Input: <Contexts>>
Output: The experimental operations mentioned are <Operations>>>>>
Round 3: For <Operations>> specify the relevant <Operation conditions>.
Input: The reaction mixture then heated 85 CC with constant stirring for 3 h.
Output: The <Operation conditions>> is 800CC.
D.4 REGULATION METHOD CLASSIFCATION TEMPLATES
Regulation Method Classification Templates
Instructions:
What the type of regulation method applied to the catalytic material in the described scheme?
Which regulation approach technique is used modify the catalytic material as detailed in the text?
Identify the specific method of regulation implemented in this catalytic mate- rial scheme.
The options: A. Alloy B. Defect C. Atomic level dispersion D. Surface/interface modifica- tion Composite Structure control
Input:
Cu/Bi bi-metal compounds derived from MOFs.
Output: E. Composite.
35

---

Published as a conference paper at ICLR 2026
D.5 PROPERTY PREDICTION TEMPLATES-GENERAL TEXT
Property Prediction Templates
Instructions:
What is the <Property> of the catalytic material represented by this [Material.representtn
Can you provide the <Property>> associated with this catalytic material <Material representation> for reduction
reductioon of CO2 to produce CO?
What is the <Property> for the catalytic material described by <Material representation
Input: CHTi (2 1 0) [C Ti Pd Pd hollow (Ti Pd Pd Pd...
Output: -1.6033 eV.
D.6 PROPERTY PREDICTION TEMPLATES-CHEMIE
Property Prediction Templates
Instructions:
Calculate the adsorption energy of CH species on the 71,8Pd;4(210) surface using the <Material representation>:
input:
Adsorbate: CH
Catalyst: 7i,8
Facet: (2 1 0)
Active site: hollow site coordinated by C, Ti, Pd, Pd
Output: -1.6033 eV.
D.7 REGULATION SCHEME COMPREHENSION
User
Instruction:
How does the core-shell architecture of CoP@NiFe-LDH specifically address the common limitations of using NiFe-LDH alone for the Oxygen Evolution Reaction (OER)?
Method Description:
To construct highly efficient catalyst for the Oxygen Evolution Reaction (OER), hierar- chical core-shell structure was designed. The core consists conductive cobalt phosphide (CoP) nanowires, which hydroxide (NiFe-LDH) nanosheets were then grown directly on the surface of the CoP nanowires. The resulting composite material, denoted CoP@ NiFe-IDDH, leverages the high intrinsic activity of the NiFe-IDH shell and the metallic properties of the CoP core.
scaffold. Vertically aligned nickel-iron layered double
36

---

Published as a conference paper at ICLR 2026
CatalystLLM
[Answer]
The CoP@ NiFe-LDH core-shell architecture strategically overcomes two primary limita- tions of using standalone NiFe-L.H as an OER catalyst: its poor intrinsic electrical conduc- tivity and the tendency for nanosheets to aggregate, which reduces the number of available active sites.
User
Instruction:
Based on the described tandem catalysis mechanism, the zinc (Zn) sites in this catalyst were be completely removed deactivated, leaving only the functional copper (Cu) sites, what change is expected for the catalyst's 's major product? Please explain the reasoning.
Method Description:
To efficiently convert carbon dioxide (CO2) into high-value ethylene (CzH44, researchers designed dual-function catalyst. This catalyst consists of atomically adjacent copper (Cu) and zine (Zn) sites dispersed on nitrogen-doped carbon (NC) support, denoted Zn-Cu/NC. The act as efficient "CO factories," selectively reducing CO2 to the carbon monoxide (#CO) intermediate. Subsequently, due spatial proximity, the generated *CO intermediate rapidly migrates to the adjacent Cu sites, where the critical C-C coupling reaction occurs to ultimately produce ethylene (C2H4).
mechanism is "tandem catalysis" process: first, the Zn sites
CatalystLLM
[Answer]
Removing the zinc (Zn) sites would shift the catalyst's selectivity from ethylene (C2H4) to methane (CHa) and hydrogen (H2). This is because eliminating the Zn "*CO factory" breaks the synergistic tandem mechanism, depriving the copper (Cu) sites of the high concentration of #CO intermediates needed for C-C coupling. Consequently, the catalyst would behave as less efficient, single-function copper system, where the reaction favors deep reduction methane or hydrogen evolution.
37

---

Published as a conference paper at ICLR 2026
E PROMPTS
Here the prompt instructions we build during the benchmark construction and evaluation process.
E.1 PROMPTS FOR QA GENERATING
<Q& Generation Prompt
Please complete the following tasks based on the text of the catalytic regulation scheme provided below and its context:
1. Task requirements:
Extract core keywords related to the catalytic scheme [Method description] from the text and context of the regulation scheme (e-g., reaction type, material system, innova- tive mechanisms, performance enhancement points, etc.).
Based on the extracted keywords, identify 3-5 key questions regarding the character- isties, advantages and improvements this regulation scheme compared to traditional catalytic schemes and provide answers based on the scheme and its context.
Questions should focus on: regulation schemes, material innovation, performance ad- vantages and practical application prospects. Avoid asking simple definitional ques- tions.
Assume that the does not have access to the original paper any external sources, so ensure that the questions and answers are self-contained. Answers should be concise and specific, based on textual and contextual [Contexts] facts.
2. Example:
Input:
.Method description: (example regulation method description}
Coontext:: {example [example_contexts) contexts]
3. output format:
Pleasse present the generated question-answer pairs in the following format:
Q1: [Question 1]
A1: [Answer 1]
QS: [Question 5]
AS: [Answer 51
Use of multiple sentence structures. Questions need to be phrased in a way that is easy understand.
Input:
Method description: {regulation method description]
Contexts: {contexts}
E.2
PROMPTS FOR CHEMICAL-TEXT GENERATION
In
cheminformatics, SMILES (Simpli-
Task Type
Text
Formatied Text
fied Molecular Input Line Entry System) is widely used method for representing molecules linear strings. It encodes molecule's 's topo- logical structure, atom types, bond types and partial stereochemical information using a sequence of ASCII characters. This compact format facilitates storage and transmission. However, this encoding relies on a character sequence governed by syntactic rules, with structural information implicit within the string pattern. For language models lacking specialized chemical parsing components, extracting relationships between atoms and bonds requires symbolic parsing, resulting in relatively low information explicitness.
"Mutrial: Cu,
"Namo-porous
RMC
Scheme: AuşCu,
AusCa alloy
Structure: Nano-porous"
"Nano-porous
"Material: Cu.
Au,Ca alloy
Scheme: AuzCu,
material catalyzes
Structure: Nano-porous,
CO2 reduction
Reactant: CO).
to produce CO"
Product: CO"
"Aclive_center: Fe,
"FeN4 graphene OH
Coordination: No.
AP
Support: graphene, Adsurbate: =O1!"
Table The examples of different input format for CatalystBench tasks.
38

---

Published as a conference paper at ICLR 2026
employ only plain SMILES strings but also introduced structured molecular representations, such as explicit bond lists or atomic property ta- bles. Table shows information two input strategies from the same original catalyst dataset. These representations are theoretically equivalent to SMILES in information content, covering all connectivity information of the molecular structure. However, the structured format directly presents atomic indices, bond types and connectivity relationships, encoding the molecular graph topology as explicit data structures in the input. Since this representation aligns with LLMs parsing patterns for tabular and graph-structured data, it reduces the model's 's need to infer relationships from character sequences, making easier for the model to correctly understand and utilize the information.
In the comparative experiments Section 6.3
<Catalytic chemical-text g Prompp
You are an expert in materials science and chemistry. Your goal is parse the un- structured, natural language text describing catalytic material and convert it into a structured JSON object.
1. Task requirements:
Analyze the [Material general text]: Carefully read the description of the catalytic material.
Identify Components: Extract the distinct chemical and structural components of the catalyst system, including:
The primary catalytically Active Component (the core material driving the reaction). The Support or Substrate which the active component is loaded.
Any Dopants, Promoters, or Modifiers introduced to alter the properties.
Extract Key Features: Identify crucial characteristics for each component, such as material composition, morphology (e.g,, nanoparticle, nanosheet, single-atom), crystal facets, size and specific properties (e-g., conductive, porous).
Generate Structured Formula: Create a concise, standardized formula-like string that represents the overall catalyst architecture (e.g., Active@Support Dopant- Active/Support).
Format as JSON: Organize all extracted information into a clean, hierarchical JSON object according to the specified Output Format. If category is not mentioned in the text, use an empty list [].
2. Example:
Input:
Materiil general text: {example (example_material material general
3. output format:
Material chemicl text: (material_json].
The output must be single, valid JSON object with the following structure. Do not include any explanatory text outside of the JSON block.
Input:
Material general text: [material.general.
39

---

Published as a conference paper at ICLR 2026
E.3 PROMPTS FOR LLM-BASED EVALUATION
<LLM-based evaluationn > PProppt
Please complete the following tasks based on the truth QA pair and two answers by
model A and model B provided below and its context:
Task requirements:
Extract core keywords related to the context of the regulation scheme from [Truth answer](e.g, reaction type, material system, innovative mechanisms, performance enhancement points, etc.).
Extract the core content of the [Answer by model A] and [Answer by model BJ without changing the original content.
Based on the extracted keywords, Score the core content of the two answers according the following scoring criteria. Provide specific details and explanations for any deductions.
Scoring criteria:
Reasonableness: whether the answer is consistent with the description of the regula- tion method in the question. If the answer consistent, 10 points will be given and if is not, 0 points will be given.
Accuracy: whether the answer is consistent with the key elements of the correct answer. 3 points if one key element is included, 6 points if two key elements included and 10 points if all are consistent.
Usability: whether the answer actually answers the question. If the response in unreliable which means it does not have any correct facts, 0 points will be given, If the response is reliable in its own right but does match the question, 5 points will be awarded and if the response actually answers the question, 10 points will be awarded. Example:
Input:
Truth question: (example wtwp:lettrttttttttttooeeeeoee
Truth answer: fexample.trutee )
Answer by model A: {example answer]
Answer by model B: {example answer)
3. output format:
Please rate the [Answer by model A] and [Answer by model B] out of 10 in terms of reasonableness, accuracy and usability. Reasonableness is 20 %, accuracy is 50% and usability is 30%.
Reasonableness: [Score reaaerraaaoallesss
Explanation for reasonableness: Texplanatiorrraaoobblennnss
Accuracy: [Score accuracy]
Explanation for accuracy: [Explanation
papllnationacccracy) accuracy]
Usability: [Scori_usabilit]]
Explanation for usability: Exaplanation.usabilic
Total score: [Total score]
Analysis: [Analysis]
F STATEMENT ON THE USE OF LARGE LANGUAGE MODELS
During the preparation of this manuscript, large language models (LLMs) were employed in lim- ited capacity to assist with linguistic refinement, grammar correction and improvement of clarity in the English prose. The scientific content, experimental design, data analysis and interpretation of results are entirely the work the authors and no LLM was used for the generation novel sci- entific claims or data fabrication. All factual statements, numerical results and interpretations have been manually verified by the authors to ensure accuracy and integrity. The assistance of LLMs was restricted improving readability and cohesion and its complies with prevailing academic publishing policies concerning Al-assisted writing.
40


Reviews:
[{"review_id": "mfmO6ACfVx", "reviewer": "Reviewer_UAyg", "summary": "This paper presents CatalystBench, a large-scale multi-task benchmark for evaluating language models in catalysis science. It covers seven task categories, from catalyst knowledge to mechanism reasoning and molecular design, built through automated data extraction and expert validation. Experiments across general and chemistry-specific LLMs reveal that while models handle basic factual tasks reasonably well, they still perform poorly on mechanism-level reasoning and condition prediction, highlighting major gaps in domain understanding.", "questions": "Please refer to the cons", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Evaluating LLMs in catalysis is both scientifically important and practically valuable. The benchmark spans multiple levels of reasoning, from basic recognition to scientific inference.\n2. The combination of automated extraction and expert validation ensures data reliability. Figures and examples make the complex tasks easy to follow.", "weaknesses": "1. Although the results highlight model weaknesses, the paper does not analyze why models fail (e.g., due to missing domain knowledge, poor reasoning ability, or lack of grounding). Adding this discussion would make the findings more actionable.\n2. The paper could include more analysis of data quality and diversity, such as the balance of catalyst types, reaction categories, and distribution of task difficulties.\n3. While the benchmark is comprehensive, the work is mainly a dataset contribution, with relatively limited technical novelty in terms of modeling or learning methodology.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HiOnE2uUWX", "reviewer": "Reviewer_SpDV", "summary": "The paper proposes CatalystBench, a catalysis-focused benchmark that tries to cover the full catalyst R&D workflow in one place by assembling eight tasks: extracting catalyst materials and synthesis steps from literature, classifying the regulation strategy used (e.g., alloying, defect, interface modulation), predicting key catalytic properties from public datasets via natural-language instructions (adsorption energy, d-band center, formation/efficiency values), and understanding long “regulation-scheme” texts that explain why a given catalyst modification improves performance.", "questions": "How much is “real” experimental/literature text vs LLM-generated Q&A? \n\nAre the 8 public catalytic datasets actually licensed / stable?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper correctly observes that catalysis papers mix narrative experimental text, semi-structured synthesis details, and simulation/numerical descriptors — and real catalyst design jumps between them. Existing benchmarks like ChemBench, LLM4Mat, SciBench are more fragmented and not tied to the full catalyst R&D loop. Their comparison table makes this point clearly.\n\n2. The 8 tasks do cover the four capability buckets the authors call Understanding → Reasoning → Explaining (their Fig. 1 / Sec. 3.1) and they really do span text, structured catalyst descriptors, and numbers.\n\n3.  Many “LLM-for-science” benchmarks just dump all tasks into SFT and hope; here they at least propose an architectural separation (MFT) and run ablations against other fine-tuning strategies (claimed in Sec. 3.2 / 3.3 and Appendix). That’s a reasonable, domain-motivated engineering idea.", "weaknesses": "1. A lot of the “hard” data is LLM-assisted before human filtering. For the regulation-scheme comprehension, the pipeline is: literature → GPT-4o (with SciQAG) → Q&A → human filtering. That is not the same as “fully human authored”, and it raises the usual questions: how much of the style/format is baked in from GPT-4o, how diverse the questions really are, and whether future GPT-4.x will get an unfair advantage because it’s closer to the generation distribution. The paper says experts “perform annotation and filtering,” but does not give inter-annotator agreement or rejection rates in the main text. I would ask to quantify that.  \n\n2. The biggest task, SE, has 6,612 instances; several key tasks (FEP 2,148; RMC 2,364; RSC 4,307) are in the low-thousands. The authors do mention that this “reflects a common constraint in catalytic research” and that they “prioritize task breadth”, but it does mean that (i) leaderboard variance could be high, and (ii) large multi-task LLMs may overfit the linguistic shell rather than the chemistry. This should be made more explicit.  \n\n3. MFT novelty is incremental. Multi-head, task-decoupled training for mixed classification/regression/generation is not new in ML; here it is justified by domain heterogeneity, but methodologically it’s a small step. For ICLR, I would like to see either (i) a clearer theoretical/optimization story (e.g., loss interference curves between numerical vs generative heads) or (ii) a stronger empirical claim like “without MFT, regression collapses by X%, with MFT, text tasks do not degrade.” Right now the paper mostly states superiority.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vpGCP7zrDH", "reviewer": "Reviewer_dAYs", "summary": "The paper introduces CatalystBench, the first comprehensive multi-task benchmark tailored for LLMs in catalysis science. It covers the entire catalyst research and development (R&D) workflow, integrating high-fidelity theoretical simulation and curated experimental data.\nTo address the inherent heterogeneity of tasks (including regression, classification, and language generation), the authors propose the Multi-head Full-task (MFT) fine-tuning strategy. MFT achieves this by decoupling output heads, which successfully alleviates interference between loss landscapes and results in the most substantial performance improvements across all tasks.\nThe resulting specialized model, CatalystLLM, achieves state-of-the-art performance and exhibits superior domain expertise compared to general LLMs.", "questions": "1. is it possible to add the non-LLM based SOTA of each task for a more comprehensive comparison?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Unified, workflow-aligned benchmark\n\nCatalystBench unifies theoretical data and experimental literature, covering the entire catalyst development process (reading comprehension, analysis, reasoning).\n\n2. Effective MFT fine-uuning\n\nThe MFT strategy decouples output spaces for diverse tasks while sharing a backbone, achieving a 12.44% performance improvement over single-task baselines.\n\n3. SOTA domain performance\n\nCatalystLLM significantly outperforms leading LLMs on the benchmark, demonstrating superior domain expertise and fewer scientific hallucinations.", "weaknesses": "1. Limited knowledge scope and bias\n\nThe predictive accuracy of CatalystLLM is constrained by the coverage of the CatalystBench dataset, potentially limiting utility for materials outside this range, and introducing potential bias from the selection of scientific literature sources.\n\n2. Performance comparison can be enhanced further\n\nWhile versatile, CatalystLLM is sometimes outperformed by highly optimized traditional ML baselines in specific single-task numerical predictions. It also could be better if more traditional machine learning baselines could be included to further demonstrate the effectiveness of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0hJalHMoVh", "reviewer": "Reviewer_2hjb", "summary": "This paper introduces CatalystBench, a multi-task benchmark designed to evaluate LLMs in catalysis science, comprising 8 tasks (26,911 samples total) spanning the entire catalyst development workflow: material extraction (ME), synthesis extraction (SE), regulation method classification (RMC), Faradaic efficiency prediction (FEP), adsorption prediction (AP), d-band center prediction (d-CP), formation energy prediction (FP), and regulation scheme comprehension (RSC). The authors propose a Multi-head Full-Task (MFT) fine-tuning strategy that decouples output heads for classification, regression, and text generation tasks,", "questions": "Despite claiming \"interpretable reasoning,\" the paper provides minimal qualitative analysis of where and why CatalystLLM fails. e.g. Table 1 shows CatalystLLM achieves 0.81 R² on AP prediction—what about the remaining 19% variance?\n\nFor regression tasks, the paper compares only against 3 traditional ML baselines (CatBERTa, GAP-CatBERTa, GPR in Appendix B.5), missing state-of-the-art graph neural networks (GNNs) for materials: (1) SchNet, DimeNet++, GemNet-OC (message-passing NNs explicitly encoding 3D geometry)—these are standard baselines for OC20/OMAT24 adsorption energy prediction. (2) Equivariant transformers \n\n[1] Schütt, Kristof T., et al. \"Schnet–a deep learning architecture for molecules and materials.\" The Journal of chemical physics 148.24 (2018).\n[2] Gasteiger, Johannes, et al. \"Fast and uncertainty-aware directional message passing for non-equilibrium molecules.\" arXiv preprint arXiv:2011.14115 (2020).\n[3] Gasteiger, Johannes, et al. \"GemNet-OC: developing graph neural networks for large and diverse molecular simulation datasets.\" arXiv preprint arXiv:2204.02782 (2022).", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper targets catalysis science, a cornerstone of sustainable energy and chemical engineering with direct industrial relevance (CO₂ reduction, hydrogen evolution, ammonia synthesis). Unlike generic molecular benchmarks, CatalystBench explicitly mirrors the real-world catalyst R&D workflow.\nThe MFT strategy is a thoughtful strategy to the heterogeneous task composition inherent in catalyst design.\nCatalystLLM shows great improvements, which achieves best performance across 19/20 metrics, with particularly dramatic improvements in regression tasks: +18% R² (AP: 0.81 vs. 0.63 for ChemLLM), +16% R² (d-CP: 0.73 vs. 0.54), +38% MAE reduction (FEP: 1.72 vs. 2.80). For classification, improvements are more modest but consistent (+29% accuracy for RMC: 0.81 vs. 0.52).", "weaknesses": "CatalystBench focuses on CO₂ reduction and surface catalysis, with tasks heavily biased toward electrochemical systems (FEP: Faradaic efficiency for CO₂RR) and metal/alloy catalysts (AP: transition metal surfaces). Some other important catalysis systems can be included such as: (1) Organocatalysis non-metallic catalysts like proline, BINOL which is important for asymmetric synthesis. (2) Enzymatic catalysis (biocatalysts), critical for pharmaceutical manufacturing. (3) Photocatalysis. The current benchmark is relatively narrow.\n\nThe authors provide no theoretical analysis of why MFT works e.g., gradient conflict mitigation, task relatedness measures, relying solely on empirical validation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)