Towards Fully-Automated Materials Discovery via Large-Scale Synthesis
  Dataset and Expert-Level LLM-as-a-Judge
Authors: name: Heegyu Kim, name: Taeyang Jeon, name: Seungtaek Choi, name: Jihoon Hong, name: Dongwon Jeon, name: Sungbum Cho, name: Ga-Yeon Baek, name: Kyung-Won Kwak, name: Dong-Hee Lee, name: Sun-Jin Choi, name: Jisu Bae, name: Chihoon Lee, name: Yunseo Kim, name: Jinsung Park, name: Hyunsouk Cho

Materials synthesis is vital for innovations such as energy storage,
catalysis, electronics, and biomedical devices. Yet, the process relies heavily
on empirical, trial-and-error methods guided by expert intuition. Our work aims
to support the materials science community by providing a practical,
data-driven resource. We have curated a comprehensive dataset of 17K
expert-verified synthesis recipes from open-access literature, which forms the
basis of our newly developed benchmark, AlchemyBench. AlchemyBench offers an
end-to-end framework that supports research in large language models applied to
synthesis prediction. It encompasses key tasks, including raw materials and
equipment prediction, synthesis procedure generation, and characterization
outcome forecasting. We propose an LLM-as-a-Judge framework that leverages
large language models for automated evaluation, demonstrating strong
statistical agreement with expert assessments. Overall, our contributions offer
a supportive foundation for exploring the capabilities of LLMs in predicting
and guiding materials synthesis, ultimately paving the way for more efficient
experimental design and accelerated innovation in materials science.