V-MAGE: A Game Evaluation Framework for Assessing Visual-Centric
  Capabilities in Multimodal Large Language Models
Authors: name: Xiangxi Zheng, name: Linjie Li, name: Zhengyuan Yang, name: Ping Yu, name: Alex Jinpeng Wang, name: Rui Yan, name: Yuan Yao, name: Lijuan Wang

Recent advancements in Multimodal Large Language Models (MLLMs) have led to
significant improvements across various multimodal benchmarks. However, as
evaluations shift from static datasets to open-world, dynamic environments,
current game-based benchmarks remain inadequate because they lack
visual-centric tasks and fail to assess the diverse reasoning skills required
for real-world decision-making. To address this, we introduce Visual-centric
Multiple Abilities Game Evaluation (V-MAGE), a game-based evaluation framework
designed to assess visual reasoning capabilities of MLLMs. V-MAGE features five
diverse games with 30+ handcrafted levels, testing models on core visual skills
such as positioning, trajectory tracking, timing, and visual memory, alongside
higher-level reasoning like long-term planning and deliberation. We use V-MAGE
to evaluate leading MLLMs, revealing significant challenges in their visual
perception and reasoning. In all game environments, the top-performing MLLMs,
as determined by Elo rating comparisons, exhibit a substantial performance gap
compared to humans. Our findings highlight critical limitations, including
various types of perceptual errors made by the models, and suggest potential
avenues for improvement from an agent-centric perspective, such as refining
agent strategies and addressing perceptual inaccuracies. Code is available at
https://github.com/CSU-JPG/V-MAGE.