CMPhysBench: A Benchmark for Evaluating Large Language Models in
  Condensed Matter Physics
Authors: name: Weida Wang, name: Dongchen Huang, name: Jiatong Li, name: Tengchao Yang, name: Ziyang Zheng, name: Di Zhang, name: Dong Han, name: Benteng Chen, name: Binzhao Luo, name: Zhiyu Liu, name: Kunling Liu, name: Zhiyuan Gao, name: Shiqi Geng, name: Wei Ma, name: Jiaming Su, name: Xin Li, name: Shuchen Pu, name: Yuhan Shui, name: Qianjia Cheng, name: Zhihao Dou, name: Dongfei Cui, name: Changyong He, name: Jin Zeng, name: Zeke Xie, name: Mao Su, name: Dongzhan Zhou, name: Yuqiang Li, name: Wanli Ouyang, name: Yunqi Cai, name: Xi Dai, name: Shufei Zhang, name: Lei Bai, name: Jinguang Cheng, name: Zhong Fang, name: Hongming Weng

We introduce CMPhysBench, designed to assess the proficiency of Large
Language Models (LLMs) in Condensed Matter Physics, as a novel Benchmark.
CMPhysBench is composed of more than 520 graduate-level meticulously curated
questions covering both representative subfields and foundational theoretical
frameworks of condensed matter physics, such as magnetism, superconductivity,
strongly correlated systems, etc. To ensure a deep understanding of the
problem-solving process,we focus exclusively on calculation problems, requiring
LLMs to independently generate comprehensive solutions. Meanwhile, leveraging
tree-based representations of expressions, we introduce the Scalable Expression
Edit Distance (SEED) score, which provides fine-grained (non-binary) partial
credit and yields a more accurate assessment of similarity between prediction
and ground-truth. Our results show that even the best models, Grok-4, reach
only 36 average SEED score and 28% accuracy on CMPhysBench, underscoring a
significant capability gap, especially for this practical and frontier domain
relative to traditional physics. The code anddataset are publicly available at
https://github.com/CMPhysBench/CMPhysBench.