DeepPHY: Benchmarking Agentic VLMs on Physical Reasoning
Authors: name: Xinrun Xu, name: Pi Bu, name: Ye Wang, name: Börje F. Karlsson, name: Ziming Wang, name: Tengtao Song, name: Qi Zhu, name: Jun Song, name: Zhiming Ding, name: Bo Zheng

Although Vision Language Models (VLMs) exhibit strong perceptual abilities
and impressive visual reasoning, they struggle with attention to detail and
precise action planning in complex, dynamic environments, leading to subpar
performance. Real-world tasks typically require complex interactions, advanced
spatial reasoning, long-term planning, and continuous strategy refinement,
usually necessitating understanding the physics rules of the target scenario.
However, evaluating these capabilities in real-world scenarios is often
prohibitively expensive. To bridge this gap, we introduce DeepPHY, a novel
benchmark framework designed to systematically evaluate VLMs' understanding and
reasoning about fundamental physical principles through a series of challenging
simulated environments. DeepPHY integrates multiple physical reasoning
environments of varying difficulty levels and incorporates fine-grained
evaluation metrics. Our evaluation finds that even state-of-the-art VLMs
struggle to translate descriptive physical knowledge into precise, predictive
control.