ING-VP: MLLMs cannot Play Easy Vision-based Games Yet
Authors: name: Haoran Zhang; name: Hangyu Guo; name: Shuyue Guo; name: Meng Cao; name: Wenhao Huang; name: Jiaheng Liu; name: Ge Zhang
Published: 2024-10-09 05:17:38

Abstract: As multimodal large language models (MLLMs) continue to demonstrate
increasingly competitive performance across a broad spectrum of tasks, more
intricate and comprehensive benchmarks have been developed to assess these
cutting-edge models. These benchmarks introduce new challenges to core
capabilities such as perception, reasoning, and planning. However, existing
multimodal benchmarks fall short in providing a focused evaluation of
multi-step planning based on spatial relationships in images. To bridge this
gap, we present ING-VP, the first INteractive Game-based Vision Planning
benchmark, specifically designed to evaluate the spatial imagination and
multi-step reasoning abilities of MLLMs. ING-VP features 6 distinct games,
encompassing 300 levels, each with 6 unique configurations. A single model
engages in over 60,000 rounds of interaction. The benchmark framework allows
for multiple comparison settings, including image-text vs. text-only inputs,
single-step vs. multi-step reasoning, and with-history vs. without-history
conditions, offering valuable insights into the model's capabilities. We
evaluated numerous state-of-the-art MLLMs, with the highest-performing model,
Claude-3.5 Sonnet, achieving an average accuracy of only 3.37%, far below the
anticipated standard. This work aims to provide a specialized evaluation
framework to drive advancements in MLLMs' capacity for complex spatial
reasoning and planning. The code is publicly available at
https://github.com/Thisisus7/ING-VP.git.