Do Vision-Language Models Have Internal World Models? Towards an Atomic
  Evaluation
Authors: name: Qiyue Gao; name: Xinyu Pi; name: Kevin Liu; name: Junrong Chen; name: Ruolan Yang; name: Xinqi Huang; name: Xinyu Fang; name: Lu Sun; name: Gautham Kishore; name: Bo Ai; name: Stone Tao; name: Mengyang Liu; name: Jiaxi Yang; name: Chao-Jung Lai; name: Chuanyang Jin; name: Jiannan Xiang; name: Benhao Huang; name: Zeming Chen; name: David Danks; name: Hao Su; name: Tianmin Shu; name: Ziqiao Ma; name: Lianhui Qin; name: Zhiting Hu
Published: 2025-06-27 03:24:29

Abstract: Internal world models (WMs) enable agents to understand the world's state and
predict transitions, serving as the basis for advanced deliberative reasoning.
Recent large Vision-Language Models (VLMs), such as OpenAI o3, GPT-4o and
Gemini, exhibit potential as general-purpose WMs. While the latest studies have
evaluated and shown limitations in specific capabilities such as visual
understanding, a systematic evaluation of VLMs' fundamental WM abilities
remains absent. Drawing on comparative psychology and cognitive science, we
propose a two-stage framework that assesses Perception (visual, spatial,
temporal, quantitative, and motion) and Prediction (mechanistic simulation,
transitive inference, compositional inference) to provide an atomic evaluation
of VLMs as WMs. Guided by this framework, we introduce WM-ABench, a large-scale
benchmark comprising 23 fine-grained evaluation dimensions across 6 diverse
simulated environments with controlled counterfactual simulations. Through 660
experiments on 15 latest commercial and open-source VLMs, we find that these
models exhibit striking limitations in basic world modeling abilities. For
instance, almost all models perform at near-random accuracy when distinguishing
motion trajectories. Additionally, they lack disentangled understanding --
e.g., some models tend to believe blue objects move faster than green ones.
More rich results and analyses reveal significant gaps between VLMs and
human-level world modeling.