PhysBench: Benchmarking and Enhancing Vision-Language Models for
  Physical World Understanding
Authors: name: Wei Chow; name: Jiageng Mao; name: Boyi Li; name: Daniel Seita; name: Vitor Guizilini; name: Yue Wang
Published: 2025-01-27 18:59:58

Abstract: Understanding the physical world is a fundamental challenge in embodied AI,
critical for enabling agents to perform complex tasks and operate safely in
real-world environments. While Vision-Language Models (VLMs) have shown great
promise in reasoning and task planning for embodied agents, their ability to
comprehend physical phenomena remains extremely limited. To close this gap, we
introduce PhysBench, a comprehensive benchmark designed to evaluate VLMs'
physical world understanding capability across a diverse set of tasks.
PhysBench contains 10,002 entries of interleaved video-image-text data,
categorized into four major domains: physical object properties, physical
object relationships, physical scene understanding, and physics-based dynamics,
further divided into 19 subclasses and 8 distinct capability dimensions. Our
extensive experiments, conducted on 75 representative VLMs, reveal that while
these models excel in common-sense reasoning, they struggle with understanding
the physical world -- likely due to the absence of physical knowledge in their
training data and the lack of embedded physical priors. To tackle the
shortfall, we introduce PhysAgent, a novel framework that combines the
generalization strengths of VLMs with the specialized expertise of vision
models, significantly enhancing VLMs' physical understanding across a variety
of tasks, including an 18.4\% improvement on GPT-4o. Furthermore, our results
demonstrate that enhancing VLMs' physical world understanding capabilities can
help embodied agents such as MOKA. We believe that PhysBench and PhysAgent
offer valuable insights and contribute to bridging the gap between VLMs and
physical world understanding.