MARBLE: A Hard Benchmark for Multimodal Spatial Reasoning and Planning
Authors: name: Yulun Jiang, name: Yekun Chai, name: Maria Brbić, name: Michael Moor

The ability to process information from multiple modalities and to reason
through it step-by-step remains a critical challenge in advancing artificial
intelligence. However, existing reasoning benchmarks focus on text-only
reasoning, or employ multimodal questions that can be answered by directly
retrieving information from a non-text modality. Thus, complex reasoning
remains poorly understood in multimodal domains. Here, we present MARBLE, a
challenging multimodal reasoning benchmark that is designed to scrutinize
multimodal language models (MLLMs) in their ability to carefully reason
step-by-step through complex multimodal problems and environments. MARBLE is
composed of two highly challenging tasks, M-Portal and M-Cube, that require the
crafting and understanding of multistep plans under spatial, visual, and
physical constraints. We find that current MLLMs perform poorly on MARBLE --
all the 12 advanced models obtain near-random performance on M-Portal and 0%
accuracy on M-Cube. Only in simplified subtasks some models outperform the
random baseline, indicating that complex reasoning is still a challenge for
existing MLLMs. Moreover, we show that perception remains a bottleneck, where
MLLMs occasionally fail to extract information from the visual inputs. By
shedding a light on the limitations of MLLMs, we hope that MARBLE will spur the
development of the next generation of models with the ability to reason and
plan across many, multimodal reasoning steps.