PhyX: Does Your Model Have the "Wits" for Physical Reasoning?
Authors: name: Hui Shen; name: Taiqiang Wu; name: Qi Han; name: Yunta Hsieh; name: Jizhou Wang; name: Yuyue Zhang; name: Yuxin Cheng; name: Zijian Hao; name: Yuansheng Ni; name: Xin Wang; name: Zhongwei Wan; name: Kai Zhang; name: Wendong Xu; name: Jing Xiong; name: Ping Luo; name: Wenhu Chen; name: Chaofan Tao; name: Zhuoqing Mao; name: Ngai Wong
Published: 2025-05-21 18:33:50

Abstract: Existing benchmarks fail to capture a crucial aspect of intelligence:
physical reasoning, the integrated ability to combine domain knowledge,
symbolic reasoning, and understanding of real-world constraints. To address
this gap, we introduce PhyX: the first large-scale benchmark designed to assess
models capacity for physics-grounded reasoning in visual scenarios. PhyX
includes 3K meticulously curated multimodal questions spanning 6 reasoning
types across 25 sub-domains and 6 core physics domains: thermodynamics,
electromagnetism, mechanics, modern physics, optics, and wave\&acoustics. In
our comprehensive evaluation, even state-of-the-art models struggle
significantly with physical reasoning. GPT-4o, Claude3.7-Sonnet, and
GPT-o4-mini achieve only 32.5\%, 42.2\%, and 45.8\% accuracy
respectively-performance gaps exceeding 29\% compared to human experts. Our
analysis exposes critical limitations in current models: over-reliance on
memorized disciplinary knowledge, excessive dependence on mathematical
formulations, and surface-level visual pattern matching rather than genuine
physical understanding. We provide in-depth analysis through fine-grained
statistics, detailed case studies, and multiple evaluation paradigms to
thoroughly examine physical reasoning capabilities. To ensure reproducibility,
we implement a compatible evaluation protocol based on widely-used toolkits
such as VLMEvalKit, enabling one-click evaluation.