PHYBench: Holistic Evaluation of Physical Perception and Reasoning in
  Large Language Models
Authors: name: Shi Qiu, name: Shaoyang Guo, name: Zhuo-Yang Song, name: Yunbo Sun, name: Zeyu Cai, name: Jiashen Wei, name: Tianyu Luo, name: Yixuan Yin, name: Haoxu Zhang, name: Yi Hu, name: Chenyang Wang, name: Chencheng Tang, name: Haoling Chang, name: Qi Liu, name: Ziheng Zhou, name: Tianyu Zhang, name: Jingtian Zhang, name: Zhangyi Liu, name: Minghao Li, name: Yuku Zhang, name: Boxuan Jing, name: Xianqi Yin, name: Yutong Ren, name: Zizhuo Fu, name: Weike Wang, name: Xudong Tian, name: Anqi Lv, name: Laifu Man, name: Jianxiang Li, name: Feiyu Tao, name: Qihua Sun, name: Zhou Liang, name: Yushu Mu, name: Zhongxuan Li, name: Jing-Jun Zhang, name: Shutao Zhang, name: Xiaotian Li, name: Xingqi Xia, name: Jiawei Lin, name: Zheyu Shen, name: Jiahang Chen, name: Qiuhao Xiong, name: Binran Wang, name: Fengyuan Wang, name: Ziyang Ni, name: Bohan Zhang, name: Fan Cui, name: Changkun Shao, name: Qing-Hong Cao, name: Ming-xing Luo, name: Muhan Zhang, name: Hua Xing Zhu

We introduce PHYBench, a novel, high-quality benchmark designed for
evaluating reasoning capabilities of large language models (LLMs) in physical
contexts. PHYBench consists of 500 meticulously curated physics problems based
on real-world physical scenarios, designed to assess the ability of models to
understand and reason about realistic physical processes. Covering mechanics,
electromagnetism, thermodynamics, optics, modern physics, and advanced physics,
the benchmark spans difficulty levels from high school exercises to
undergraduate problems and Physics Olympiad challenges. Additionally, we
propose the Expression Edit Distance (EED) Score, a novel evaluation metric
based on the edit distance between mathematical expressions, which effectively
captures differences in model reasoning processes and results beyond
traditional binary scoring methods. We evaluate various LLMs on PHYBench and
compare their performance with human experts. Our results reveal that even
state-of-the-art reasoning models significantly lag behind human experts,
highlighting their limitations and the need for improvement in complex physical
reasoning scenarios. Our benchmark results and dataset are publicly available
at https://phybench-official.github.io/phybench-demo/.