MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation
  Models
Authors: name: Mahir Labib Dihan; name: Md Tanvir Hassan; name: Md Tanvir Parvez; name: Md Hasebul Hasan; name: Md Almash Alam; name: Muhammad Aamir Cheema; name: Mohammed Eunus Ali; name: Md Rizwan Parvez
Published: 2024-12-31 07:20:32

Abstract: Recent advancements in foundation models have enhanced AI systems'
capabilities in autonomous tool usage and reasoning. However, their ability in
location or map-based reasoning - which improves daily life by optimizing
navigation, facilitating resource discovery, and streamlining logistics - has
not been systematically studied. To bridge this gap, we introduce MapEval, a
benchmark designed to assess diverse and complex map-based user queries with
geo-spatial reasoning. MapEval features three task types (textual, API-based,
and visual) that require collecting world information via map tools, processing
heterogeneous geo-spatial contexts (e.g., named entities, travel distances,
user reviews or ratings, images), and compositional reasoning, which all
state-of-the-art foundation models find challenging. Comprising 700 unique
multiple-choice questions about locations across 180 cities and 54 countries,
MapEval evaluates foundation models' ability to handle spatial relationships,
map infographics, travel planning, and navigation challenges. Using MapEval, we
conducted a comprehensive evaluation of 28 prominent foundation models. While
no single model excelled across all tasks, Claude-3.5-Sonnet, GPT-4o, and
Gemini-1.5-Pro achieved competitive performance overall. However, substantial
performance gaps emerged, particularly in MapEval, where agents with
Claude-3.5-Sonnet outperformed GPT-4o and Gemini-1.5-Pro by 16% and 21%,
respectively, and the gaps became even more amplified when compared to
open-source LLMs. Our detailed analyses provide insights into the strengths and
weaknesses of current models, though all models still fall short of human
performance by more than 20% on average, struggling with complex map images and
rigorous geo-spatial reasoning. This gap highlights MapEval's critical role in
advancing general-purpose foundation models with stronger geo-spatial
understanding.