VideoGameBench: Can Vision-Language Models complete popular video games?
Authors: name: Alex L. Zhang; name: Thomas L. Griffiths; name: Karthik R. Narasimhan; name: Ofir Press
Published: 2025-05-23 17:43:27

Abstract: Vision-language models (VLMs) have achieved strong results on coding and math
benchmarks that are challenging for humans, yet their ability to perform tasks
that come naturally to humans--such as perception, spatial navigation, and
memory management--remains understudied. Real video games are crafted to be
intuitive for humans to learn and master by leveraging innate inductive biases,
making them an ideal testbed for evaluating such capabilities in VLMs. To this
end, we introduce VideoGameBench, a benchmark consisting of 10 popular video
games from the 1990s that VLMs directly interact with in real-time.
VideoGameBench challenges models to complete entire games with access to only
raw visual inputs and a high-level description of objectives and controls, a
significant departure from existing setups that rely on game-specific
scaffolding and auxiliary information. We keep three of the games secret to
encourage solutions that generalize to unseen environments. Our experiments
show that frontier vision-language models struggle to progress beyond the
beginning of each game. We find inference latency to be a major limitation of
frontier models in the real-time setting; therefore, we introduce
VideoGameBench Lite, a setting where the game pauses while waiting for the LM's
next action. The best performing model, Gemini 2.5 Pro, completes only 0.48% of
VideoGameBench and 1.6% of VideoGameBench Lite. We hope that the formalization
of the human skills mentioned above into this benchmark motivates progress in
these research directions.