MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
Authors: Mahir Labib Dihan
MD Tanvir Hassan
MD TANVIR PARVEZ
Md Hasebul Hasan
Md Almash Alam
Muhammad Aamir Cheema
Mohammed Eunus Ali
Md Rizwan Parvez
Venue: ICML 2025 spotlightposter (2025)
Keywords: MapLLM
Map Query
Geo-Spatial Question Answering
Geo-Spatial Reasoning
Location Based Service
Google Maps
Primary area: applications->everything_else
TL;DR: MapEval is a benchmark evaluating foundation models' geo-spatial reasoning across textual, API-based, and visual tasks using diverse map-based queries.

Abstract: Recent advancements in foundation models have improved autonomous tool usage and reasoning, but their capabilities in map-based reasoning remain underexplored. To address this, we introduce MapEval, a benchmark designed to assess foundation models across three distinct tasks—textual, API-based, and visual reasoning—through 700 multiple-choice questions spanning 180 cities and 54 countries, covering spatial relationships, navigation, travel planning, and real-world map interactions. Unlike prior benchmarks that focus on simple location queries, MapEval requires models to handle long-context reasoning, API interactions and visual map analysis, making it the most comprehensive evaluation framework for geospatial AI.  On evaluation of 30 foundation models, including Claude-3.5-Sonnet, GPT-4o, Gemini-1.5-Pro, none surpasses 67% accuracy, with open-source models performing significantly worse and all models lagging over 20% behind human performance. These results expose critical gaps in spatial inference, as models struggle with distances, directions, route planning, and place-specific reasoning, highlighting the need for better geospatial AI to bridge the gap between foundation models and real-world navigation.

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
Mahir Labib Dihan' Md Tanvir Hassan 1*
Md Tanvir Parvez? Md Hasebul Hasan 1 Md Almash Alam
Muhammad Aamir Cheema
Mohammed Eunus Ali Md Rizwan Parvez²
Abstract
1. Introduction
Recent advancements in foundation models, particularly large language models (LLMs) and lasion-language models (VLMs), are significantly enhancing the capabilities of Al systems autonomous tool usage (Qin et al.; Yao al.. 2023) and reasoning (Lu et al.; Wei et al., 2022). These developments facilitate the automation of everyday tasks through natural language instructions, especially in domains that require interaction with specialized tools like map ser- vices.
Recent advancements in foundation models have improved autonomous tool usage and reason- ing, but their capabílities in map-based reason- ing remain underexplored. To address this, we introduce MapEval, benchmark designed to assess foundation models across three distinct tasks-textual API-based, and visual reason- ing ing-through 700 multiple-choice questions span- ning 180 cities and 54 countries, covering spa- tial relationships, navigation, travel planning, and real-world map interactions. Unlike prior benchmarks that focus on simple location queries, MapEval requires models to handle long-context reasoning, API interactions and visual map analy- sis, making it the most comprehensive evaluation framework for geospatial AI. On evaluation of 30 foundation models, including Claude-3.5-Sonnet, GPT-4o, Semini-1.5-PPro none surpasses 67% ac- curacy, with open-source models performing sig- nificantly worse and all models lagging over 20% behind human performance. These results expose critical gaps in spatial inference, as models strug- gle with distances, directions, route planning, and place-specific reasoning, highlighting the need for better geospatial AI to bridge the gap between foundation models and real-world navigation. All the resources are available on the project website.
As platforms such as Google Maps or Apple Maps have become ubiquitous for accessing various location-based services (a.k.a tools/APls) -ranging from finding nearby restaurants to determining the fastest routes between ori- gins and destinations there has been a growing interest in integrating maps with foundation models (Xie et al.; Zheng et al., 2024). A couple of recent initiatives, such as WebArena (Zhou et al.) and VisualWebArena (Koh et al., 2024), have introduced new tasks that involve map usage in practical scenarios.
However, despite the widespread adoption of map services and the promising potential of interactions between founda- tion models (e.g.. LLMs and VLMs) and these services, no studies have rigorously tested the capabilities foundation models in location or geo-spatial reasoning. This gap is critical, effective map-hased reasoning can optimize navi- gation, facilitate resource discovery, and streamline logistics in everyday life. Addressing this gap is essential for advanc- ing the practical utility of AI in real-world applications.
We introduce MapEval, a novel benchmark designed to evaluate the geo-spatial reasoning capabilities of founda- tion models and Al agents in complex map-based scenarios. MapEval addresses critical gap in existing benchmarks by evaluating models' ability to process heterogeneous geo- spatial contexts, perform compositional reasoning, and inter- act with real-world map tools. It features three task types- API, Visual, and Textual- that require models to collect world information via map tools, a deep visual understand- ing, and reason over diverse geo-spatial data (e.g., named entities, coordinates, operational hours, distances, routes, user reviews/ratings, map images), all of which remain chal-
Equual contribution 'Department of Computer Science and En- gineering Bangladesh University of Engineering and Technology (BUET) Statisties, Islamic University, Bangladesh 'Bangladesh Computer Council (BCC) *Faculty Information Technol- ogy, Monash University, Melbourne, Australia 'Qatar Com- puting Research Institute (QCRI). Correspondence to: Mahir Labib Dihan mmahirlabibdihane @ comi..com>. Mohammed Eu- nus Ali <mohammed. cunus. ali gomi, Md Rizwan Parvez <mparvez hbw..cdu.qu.........obbbbu....k.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
MapEval Annotation
NapEval-Textual
MapEval-APT
MapEval- Visual
Visual Context
Textual Context
Neadby Restaurants
Textual Context
Visual Context
McDloaat''
Ruting ratinga)
Question
Question
Question
Level: ineepensive.
Rating:
Dubei
ratings).
Choices
Choices
Choices
Level
Bikaneeii
Rating:
Expert
Level:
Annotator
ratinge)
Level:
API
LLM
Cached
Agent
VLM
Response
Response
Response
Question
Choices
Compare with ground truth
Compare with
Compara with ground
1. McDonald's
Which nearby restaurant Dubai Mall has the highest rating?
2. The GrillShack of
3. Nando's Dubai Mall
Score
Score
Score
4. Bikanervala Dubai Mall
Figure 1. Overview of MapEval. On the left, show the annotation process, where expert gathers either visual snapshots or textual data from Google Maps to create multiple-choice questions with ground truth labels. On right, we depict the evaluation process and input/output for the three benchmark tasks in MapEval.
lenging for state-of-the-ane foundation models. Comprising 700 unique multiple-choice questions across 180 cities and 54 countries, MapEval reflects real-world user interactions with map services while pushing state-of-ther -ftt-- models to understand spatial relationships, map infographics, travel planning, POI search, and navigation. MapEval ensures geographic diversity, realistic query patterns, and evaluation across multiple modalities. By integrating long contexts, visual complexity. API interactions, and questions requiring commonsense reasoning recognition of insufficient infor- mation (i.e., unanswerability), it offers a rigorous framework for advancing geo-spatial Al capabilities. In Fig 1, we depict an overview of MapEval.
2. Related Work
Geo-spatial question answering presents significant chal- lenges for foundation models (Mai al., 2024). Early re- search in GeoQA (Mai al., 2021) has focused on template- based methods (Zelle & Mooney, 1996; Chen et al., 2013; Chen, 2014; Punjani et al., 2018; Kefalidis et al., 2023), where predefined templates classify queries and retrieve in- formation from structured databases like OpenStreetMap or DBpedia (Auer et al., 2007). While effective in certain scenarios, these methods constrained by the static nature of the databases and the predefined templates, limiting their flexibility in handling complex dynamic queries. There has been limited effort to assess (Roberts et al.) and im- prove (Balsebre et al., 2024) LLMSs' capabilities in geospa- tial reasoning. Recent benchmarks such as Travel Plan- ner (Xie et al.), ToolBench (Qin et al.), and API-Bank (Li et al., 2023) integrate map tools and APIs for location-basse queries. While these benchmarks handle real-world tasks like itinerary planning or querying map data, the use of map APIs is limited to more straightforward use cases, such as calculating distances or identifying nearby points of interest. In addition, remote sensing research (Bastani et al., 2023; Yuan al., 2024; Zhang et al., 2024; Lobry et al., 2020) has focused on extracting physical features from satellite imagery. While valuable for environmental monitoring and urban planning, this approach differs significantly from the task of reasoning over interactive digital map views, which
With MapEval, evaluated 30 prominent foundation mod- els, where Claude-3.5-Sonnet GPT-4o, and Gemini-L.5-Pro showed competitive performance overall. However, sig- nificant gaps emerged in MapEval-API, with Claude-3.5- Sonnet agents outperforming GPT-4o and Gemini--155PP by 16% and 21%, respectively, and even larger disparities compared to open-source models. Our detailed analyses revealed further insights into model strengths and weak- nesses. Despite these advances, all models still fall short of human performance by over 20%, especially in handling complex map images and rigorous reasoning, underscoring MapEval's role in advancing geo-spatial understanding.
2

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
involve understanding spatial relationships, map symbols, and navigation elements in dynamic, user-interactive con- text. For detailed discussion of template-based approaches, benchmarks involving map APIs, and comparisons with re- mote sensing methodologies, see Appendix A.
an open-ended variant of MapEval to assess its flexibility beyond MCQ format.
Cost-effective and Focused. To maintain cost-effective yet comprehensive evaluation framework, MapEval com- prises 700 instances, deliberately designed to cover diverse geo-spatial reasoning tasks, question types, and global loca- tions. This focused approach ensures a manageable testbed while providing sufficient diversity for robust and reliable assessment, avoiding redundancy without compromising depth or breadth.
3. The MapEval Dataset
3.1. Design Principles
Reasoning. Geo-spatial reasoning in map-based tasks presents distinct challenges for foundation models, includ- ing: (a) understanding complex problem descriptions in natural language, (b) collecting relevant world information using map tools or APIs, (c) performing compositional and spatio-temporal reasoning, (d) interpreting map visuals, and (e) synthesizing information from heterogeneous geo-spatial contexts (e.g., named entities, distances, and temporal val- ues). These tasks test the limits of state-of-the-and art models, which struggle to fully grasp geo-spatial relationships, navi- gation complexities, and POIs.
3.2. Tasks
Textual. The objective of MapEval-Textual is to answer MCQs by decomposing complex queries and extracting rele- vant information from long textual contexts. These contexts describe map locations, POIs, routes, navigation details, and travel distances/times, often including user ratings or reviews. Unlike typical reading comprehension tasks, these texts combine structured data (e.g., coordinates, distances) with unstructured narratives and subjective content. The model must reason over this heterogeneous information to select the correct answer. This task evaluates the model's ability to analyze fine-grained map-related information pre- sented in text.
Realistic. MapEval reflects real-world map usage by cap- turing typical user interactions with map services, such as: (a) varied usage patterns like location-based baaeed searches and travel planning, and (b) informal, often fragmented queries, without relying perfect grammar structure.
API. In the MapEval- task, an Al agent interacts with map-based APIS retrieve data (e.g., nearby POIs, distance caleulations). The task involves generating API queries based on user questions, interpreting the returned structured data, and integrating it into reasoning processes to answer MCQs. This task evaluates the model's 's ability to handle data retrieval, API interactions, and the synthesis of structured information in real-world, map-driven scenarios.
Diversity. MapEval ensures geographic diversity and broad evaluation across models and tasks: (a) capturing locations across cities and countries globally, and (b) offering a wide variety of question types and contexts, which test founda- tion models' spatial, temporal, data retrieval, and visual reasoning abilities.
Long Contexts, Multi-modality, API Interac- tions. MapEval challenges models with: (a) long geo-spatial descriptions, including POIs and navigational data, (b) complex map-specific images with location markers, and (c) API interactions, testing models' abilities as language agents in real-world map- based tasks.
Visual. VapEval-Visual task requires the model to interpret and analyze map snapshots, specifically digital map views from services like Google Maps. These snapshots repre- sent complex spatial relationships, routes, landmarks, OCR texts (e.g., rating), and symbolic elements (e.g., logos or traffic signs), which differ from typical image recognition tasks. The model must extract relevant information from the visuals, integrate it with spatial reasoning, and use it to answer MCQs. This task assesses the model's ability to tackle map-specific visual contents and perform spatial reasoning.
Unanswerability, Commonsense. MapEval includes ques- tions where context is insufficient provide an answer, testing models' ability to identify missing or incomplete information, rather than making incorrect guesses. It also assesses commonsense reasoning and handling uncertainty, essential for reliable decision makion in real-world applica- tions.
3.3. Dataset Construction
Multiple Choice Questions (MCQs), We employ MCQs in MapEval, similar to MMLU (Hendrycks et al.), rather than open-ended queries. This approach circumvents the evaluation challenges associated with generated responses (Sai et al., 2022), allowing for a more straightforward and reliable accuracy-based assessment of map- based reasoning capabilities. As discussed in Appendix G.2, we also evaluate
Data Annotation.
To create a high-quality benchmark dataset for MapEval, we utilized Google Maps, a widely adopted map service. The process of constructing the textual context presented significant challenges, particularly in ensuring accuracy and efficiency. For example question like "What are the open-
3

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
Type
Task
Question Example
Count
Textual/API
What is the direction of Victoria Falls from Harare?
64
Place Info
Visual
Is there any Hospital marked with a star symbol on the tourist map of Rome?
121
Textual/API
Find restaurants nearby Louvre Museum above 4.0 rating. stayed at SpringHill Suites by Marriott Portland Hillsboro. Can you recommend the nearest restaurant to my location?
83
Nearby
Visual
91
am driving Brassica in Bexley Via E Whittier St. After reaching Lockbourne Rd, where should go next?
Textual/API
66
Routing
Visual
What is the fastest route from Times Square to Central Park by walking?
80
Which road should I follow from Wola to Mokotów to avoid flooded roads in heavy rains?
Textual/API
20
Unanswerable
Which way should be efficient while visit from Abis bus station to
Visual
20
KONO so that Victoria park is on the way
have an afternoon free in New York and plan to visit The Metropolitan Museum of Art for hours, followed by a 30-minute coffee break at a nearby cafe, and then spend 1 hour in Central Park. Plan a schedule to ensure I have enough time for everything.
Trip
Textual/API
67
Counting
Visual
How many hospitals are there in the left side of the river?
88
Table 1. Examples of different question categories. HopEval-TTxtus and MapEval-Vissee Visual questions are accompanied by both textual and visual context (Sec appendix I for full qualitative example queries, contexts and evaluation model outputs during evaluations.)
ing hours of the British Museum?" requires precise data to provide valid options and correct answer. Manually search- ing for the "British Museum" Google Maps and looking for its opening hours can be both time-consuming and prone to errors, making this method inefficient. To address these challenges, we used MapQaTor (Dihan et al., 2024), a web interface built on Google Maps APIs, designed to streamline the collection of textual map data. MapQa'Tor automates data retrieval from map APIs, collecting key information like opening hours and location details to build the textual dataset (Details in Appendix B.1). For each user query, we first fetch the necessary context data using MapQa'Tor. Ques- tions were then paired with their corresponding contexts, and multiple-choice options were carefully curated based on this information. The ground truth answers were derived from the same context.
information. To maintain traceability, we save the Google Maps URL for each snapshot. Additionally, to examine model capabilities at different zoom levels, capture snap- shots at varying zoom depths'.
We create the following question types for MapEval: (a) Place Info: detects POIs and asks about specific details related to a place (c.g., location, rating, reviews); (b) Nearby: identifies nearby places or POIs; (c) Routing: navi- gates between locations, considering routes and landmarks; (d) Unanswerable: when the map information (c.g., from google map) or the textual and visual context is insufficient to answer the question. Note that, in each category we for- mulate a few questions that requires general knowledge or reasoning about locations and navigation (c.g., there are 52 commonsense QAs in MapEval- Visual).
Moreover, - and MapEval-APT exclusively feature Trip questions, which involve planning multi-stop journeys across various POIs. Due to the complexity and details of trip planning, these questions are difficult to repre- sent in a single visual snapshot. Conversely, Counting tasks are unique to MapEval-Visse Visual, where models count spe- cific items or locations on a map a challenge specifically tailored to visual contexts.
For MapEval-APII the same questions MapEval-Textual but without textual contexts, requiring the language agents to interact with tools directly. To address consistency issues with real-time data updates, we created a controlled evaluation environment. This involves caching place information and simulating API interactions. Details of the geudo-Google Maps setup are provided in Appendix C.
used in
Quality Control and Human Performance To ensure qual- ity, each QA pair is annotated by multiple members of our
For the visual context, we capture map snapshots from Google Maps, covering random locations across various cities and countries worldwide. Based on each snapshot, we formulate relevant questions with multiple-choice options, where the correct labels are derived directly from the map
'Zoom levels found in map URLs indicate depth (c.g., url has zoom level 16.71), with higher values (c.g.. 16 and above) showing more detail, compared to level (world map)- See Appendix F.I
4

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
Statistics
Number
Total unique question instances
700
Questions with api textual-context
300 (42.86%)
Questions with visual-context
400 (57.14%)
Nearby 13.00%
Total unique countries
54
Counting 12.57%
Total unique cities
180
Maximum textual-context length
1500
Place Info 17.29%
Maximum question length
107
Visual
Average textual- context length Average question length
435.63
Routing 11.43%
21.41
Unanswereblee 96%
Unique number of textual-context Unique number of visual-context
215
Textual/AP)
270
Unnerbwerrce
2. 655%
Place Info 9.1 9.14% %
Min, Max, Avg questions from a country
6, 132, 12.96
Min, Max, Avg questions from city
1, 44, 3.89
Nearby 11.86% %
Min, Max, Avg Choices
2, 7, 4.004
Routing 9.43%
Max zoom visual-context
21.0
9.57%
Min zoom of visual-context
8.0
Average zoom of visual-context
15.26
Table 2. Key statistics of MapEval. Lengths are in words. Visual- context means Map snapshots/images. Some questions are yes/no and some have additional complexity with 4+ choices.
Figure 2. MapEval category statistics.
team, achieving initial 76% mutual agreement. At least two team members then manually verify and resolve any dis- putes on the remaining pairs; if consensus cannot be reached (i.e., ambiguous), that pair is filtered out. To compute hu- man scores, two team members who did not participate in annotation process attempt to answer the questions, and their highest-scoring attempts are reported the human per- formance benchmark. For MapEval-API, the questions are identical to NapEval-Texxuul we report the same human performance for both.
ing varying levels of complexity and detail. Furthermore, in Appendix F.I, we illustrate the zoom level distribution in HappEvl-Visuall adding another dimension to the dataset's diversity and evaluation challenges.
4. Experiments
4.1. Experimental Protocol and Setup
We evaluate all tasks using the accuracy metric, defined as the percentage of correct choices selected by the model. We prompt models with the respective context, question, tool usage documentations (only for MapEval-APD, an- swer format guidelines, and choices. We assess L.MSs for MapEval-Textual VLMs for MapEval- Visuull, and ReACT agents (Yao al., 2023) (known for effective tool interaction (Zhuang al., 2023)) built on various LLMs for MapEval- API, aligning each task with appropriate model types. Ap- pendix I presents example prompts for all tasks. Our LLMs and VLMs spans both open and closed-source mod- els. Closed-source models include Claude-3.5-Sonnd. GPT- 4o, GPT-4-Turbo. GPT-3.5-Turbo Gemini-1.5 (Pro, Flash), with all except GPT-335-Turbb being multi-modal founda- tion models used in all tasks, while GPT-3.5-Turbe which is text-only, is utilized solely in the MapEval- Textual and MapEval-APi tasks. Open-source LLMs include instruct versions of Gemma-2.0 (9B, 27B), Llama-3.2 (3B, 90B), Llama-3.1 (8B, 70B) Mistral-Nemo-751 Mixtral-8x7B,
3.4. Dataset Statistics and Analysis
The main statistics of MapEval are presented in Table 2 and Figure 2. Examples of each question type and their numbers are presented in Table 15. We visualize the global distribution of locations our dataset using coordinates (Fig. 7 in appendix). Table 12 (Appendix) lists all countries and their frequencies in MapEval. We use OpenStreetMap''s Nominatim API for reverse geocoding to determine coun- tries from coordinates. Textual context includes the coordi- nates of places in it. In case of visual context, we can find the coordinates from the associated Map URL with each snapshot. For example, coordinate of an example url, is 35.7048455, 119. 139.763263. We visualize the distribution of question and textual context lengths the Appendix (Fig- ures and 6). Overall, beyond their diversity in types, ques- tions and contexts also vary significantly in length, reflect-
5

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
Overall
Place Info
Nearby (#83)
Routing (#66)
Trip (#67)
Unanswerable
Model
(#300)
(#64)
(#20)
Close-Source
(Proprictary)
LLMs
Claude-3.5-Sonnel (Anthropic, 2024)
66.33
73.44
73.49
75.76
49.25
40.00
Gemini-1.5-PP (Team et al., 2024a)
66.33
65.63
74.70
69.70
47.76
85.00
GPT-4o (OpenAI, 2024)
63.33
64.06
74.70
69.70
49.25
40.00
GPr-4-Turbo (OpenAI, 2023)
62.33
67.19
71.08
71.21
47.76
30.00
Gemini-1.5-Fllsh (Team et al., 2024a)
58.67
62.50
67.47
66.67
38.81
50.00
GPT-4o-mini (OpenAl, 2024)
51.00
46.88
63.86
57.58
40.30
25.00
GPT-3.5-Turbo (OpenAI, 2022)
2022)
37.67
26.56
53.01
48.48
28.36
5.00
Open- Open-Source LLMs
Llama-3.1-70B Meta,eta, 2024)
61.00
70.31
67.47
69.70
40.30
45.00
Llama-3.2-900) (AI@ Meta, 2024)
58.33
68.75
66.27
66.67
38.81
30.00
Qwen2.5-72B (Team, 2024)
57.00
62.50
71.08
63.64
41.79
10.00
Qwen2.5-14B (Team, 2024)
53.67
57.81
71.08
59.09
32.84
20.00
Gemma-2.0-2777 (Team et al., 2024b)
49.00
39.06
71.08
59.09
31.34
15.00
Gemma-2.0-9B (Team et al., 2024b)
47.33
50.00
50.60
59.09
34.33
30.00
Llama-3.1-8B (AI@Meta, 2024)
44.00
53.13
57.83
45.45
23.88
20.00
Qwen2.5-7B (Team, 2024)
43.33
48.44
49.40
42.42
38.81
20.00
Mistral-Nemo (AI, 2024)
43.33
46.88
50.60
50.00
32.84
15,00
Mixtral-8x7B (Jiang et al., 2024)
43.00
53.13
54.22
45.45
26.87
10.00
Phi-3.5-mini (Abdin et al., 2024)
37.00
40.63
48.19
46.97
20.90
0.00
Llama-3.2-3B (AI@ Meta, 2024)
33.00
31.25
49.40
31.82
25.37
0.00
Human Performance
Human
86.67
92.19
90.36
81.81
88.06
65.00
Table 3. MapEval-Textual performances. Figure 16 visualizes the categorical accuracy.
Qwen2.5 (7B, 14B, 72B) Phi-3.5-mini. For MapEval-
spatial reasoning tasks. Models perform well in "Place Info". "Nearby"" and "Routing" tasks (best performance ~75% due to the comprehensive context extracted by MapEval- Textual, such as textual descriptions, opening hours, and dis- tances. In contrast, performance "Trip" planning scenar- ios remains poor (best ~49%), reflecting difficulties in han- dling multi-step reasoning and aggregating spatio-temporal constraints. Performance "Unanswerable" queries is mixed, with Gemini-1.5 achieving 85% accuracy while most models fall between 0-45%. This disparity highlights the importance of recognizing insufficient information for real-world applications. Consistent underperformance in "Trip" tasks and varied accuracy in unanswerable queries point to fundamental limitations in geo-spatial reasoning across architectures The results also show that larger mod- els generally outperform smaller ones. However, the gap between open and closed-source models indicates room for advancement in open-source systems. Additionally, Fig. 17 reveals challenges faced by open-source models handling longer contexts.
Visual, we considered the open-source VLMss: Qwen2.5-
VL-72B-Instruct, Dwwen2-VL-7B-Inttteee Llama-3.2-90B- Vision, MiniCPM-Llama3- V-2.5, Llama-3VVLA1.5--0 glm-4v-9h, Interml.n-xcompposeer, paligemma-3b-mix-22
DocOw11.5, hawaavl..-mmittra--bb-h- and 1lava-1.5-7b-hf.
In MapEval-API task, concentrate our exploration high-capacity open- source LLMs, specifically Llama-3.2- 90B, Llama-3.1-70B. Mixtral-8x755 and Gemma-2.0-99 We limit our evaluation of open-source models in AI agents due to the task's s complexity and resource demands, the lower performance of smaller models, and the excessive number of calls for both LLMs and map APIs.
4.2. Results and Analysis
4.2.1. MAPEVAL-TEXTUM.
We present a summary of MapEval-Texxe -exttull results in Ta- ble 3, highlighting insights into the current state of geo- spatial reasoning language models. Closed-source models generally outperform open-source ones, with Claude-3.5- Sonnet achieving 66.33% accuracy, while the best open- source model, ..amm-33.-7000 reaches 61.00%. However, the substantial gap between top-performing models and hu- man accuracy (86.67%) underscores the challenges in geo-
4.2.2. MAPEVAL-ARE
We present the MapEval-ANl API results in Table 4, highlight- ing key insights into the geo-spatial reasoning abilities of
6

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
Overall
Place Info
Nearby (#83)
Routing (#66)
Trip (#67)
Unanswerable
Model
(#300)
(#64)
(#20)
Close-Source
(Proprictary)
LLMs
Claude-3.5-Soneel (Anthropic, 2024)
64.00
68.75
55.42
65.15
71.64
55.00
GPT-4-Turbo (OpenAI, 2023)
53.67
62.50
50.60
60.61
50.75
25.00
GPT-4o (OpenAI, 2024)
48.67
59.38
40.96
50.00
56.72
15.00
Gemini-1.5-PP (Team al., 2024a)
43.33
65.63
30.12
40.91
34.33
65.00
Gemini-1.5-Flash (Team et al., 2024a)
41.67
51.56
38.55
46.97
34.33
30.00
GPT-3.5-Turbo (OpenAI, 2022)
27.33
39.06
22.89
33.33
19.40
15.00
GPT-4o-mini (OpenAL, 2024)
23.00
28.13
14.46
13.64
43.28
5.00
Open-Source LLMs
Llama-3.2-90B (AI@ Meta, 2024)
39.67
54.69
37.35
39.39
35.82
15.00
Llama-3. 3.70- (AI@ Meta, 2024)
37.67
53.13
32.53
42.42
31.34
15.00
Mixtral-8x -x (Jiang et al., 2024)
27.67
32.81
18.07
27.27
38.81
15.00
Gemma-2.0-99 (Team al., 2024b)
27.00
35.94
14.46
28.79
26.87
45.00
Table 4. MapEval-APl evaluation performance (See Figure visualize categorical accuracy)
language models with map APIs. MapEval-API generally underperforms compared to -extual-Texttal with signifi- cant drops in Nearby tasks (from 74.70% to 55.42%) and Routing tasks (from 75.76% to 65.15%). Figure 3 visualizes these differences. While Claude-3.5-Somnc performed con- sistently, other models showed declines due to lack of con- text and tool complexity, emphasizing the need for advanced agents surpassing ReAct's capabilities. In the Trip category, MapEval-API improved by approximately 22% compared to HopEval-TTexurr demonstrating its effectiveness in step-by- step reasoning for complex problems. Claude-3.5-Sonnee led with 64.00% overall accuracy, excelling as a tool agent and in generic graph reasoning. However, a significant gap exists between closed-source and open-source models, with the best open-source model, Llama-3.2 90B, achieving 39.67%. Performance "Unanswerable" queries varied widely (5% to 65%), highlighting the need for better han- dling of insufficient information.
4.2.3. MAPEVAL- VISUAL
We evaluate models the MapEval- Visual task in Table 5. Closed-source models outperform open-source ones, with Claude-3.5-Soman leading at 61.65%, followed by GPT- 4o (58.90%) and Gemini-l.5-Pa (56. 14%). Among open- source models, Qwen2.5-VLL-22B achieves the highest ac- curacy (60.35%). While models excel in Place Info tasks (82.64%). they struggle with complex tasks like Counting. Nearby, and Routing, highlighting the need for improve- ment. Models trained generic images underperform map-specific tasks, likely due to limited exposure to de- tailed map data. Fig 21 (Appendix) shows accuracy drop- ping significantly at higher zoom levels (e.g., streets, sym- bols, demarcations) beyond level 14, where map complexity increases. Our benchmark reveals substantial gap be- tween Al and human performance, particularly nuanced tasks. For example, humans achieve 85.18% on Routing tasks versus 50% for the best model, and 78.41 % on Count- ing tasks versus 47. 47.73%. While closed-source models like Claude-3.5-Sonnee and Gemini-1.5-Pry excel in identify- ing unanswerable questions (90% and 80%, respectively), open-source models struggle significantly.
80
Accuracy (%)
60
40
4.3. Qualitative Error Analysis
20
0
LLMs struggle with spatial, temporal, and commonsense reasoning in location-based queries. spatial reasoning, they often fail with straight-line distances (Listing 1), car- dinal directions (e.g.. East, West, North, South; Listing 2), and step-by--step route planning, including tasks that involve mathematical computations or counting (e.g., nearby restau- rant counts; Listing 3). Temporal reasoning challenges include inefficient trip planning and errors in travel times or visit durations (Listing 4). Commonsense reasoning failures arise when models hallucinate miss simple contextual
Sonnet
G-Pro
GPT-40
GPT-4
L
G-Flash
LIB.2
4o-mini
Gemma2
Mixtral
GPT-3.5
MapEval-Teaae Textual [o MapEval-API
Figure Comparison between MapEval- Textual MapEval-
API.

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
Overall
Place Info
Nearby
Routing (#80)
Counting (#88)
Unanswerable
Model
(#400)
(#121)
(#91)
(#20)
Close-Source
(Proprietary)
VLMs
Claude-3-5-Somme (Anthropic, 2024)
61.65
82.64
53.56
45.00
47.73
90.00
GPT-4o (OpenAI, 2024)
58.90
76.86
57.78
50.00
47.73
40.00
Gemini-1.5-Pro (Team et al., 2024a)
14
76.86
56.67
43.75
32.95
80.00
GPT-4-Turbo (OpenAI, 2023)
55.89
75.21
56.67
42.50
44.32
40.00
Gemini-1,5-Flash (Team et al., 2024a)
51.94
70.25
56.47
38,36
32.95
55.00
GPT-4o-mini (OpenAI, 2024)
50.13
77.69
47.78
41.25
28.41
25.00
Open Source VLMs
Qwen2.5-VI-72B (Bai et al., 2025)
60.35
76,86
54.44
43.04
52.33
90.00
Qwen2-VL- VL-7B (Wang et al., 2024)
51.63
71.07
48.89
40.00
40.91
40.00
Llama3. 2ataaa.-9----isss Vision (AI @ Meta, 2024)
50.38
73.55
46.67
41.25
36.36
25.00
Glm-4v-9b (GLM et al., 2024)
48.12
73.55
42.22
41.25
34.09
10.00
IntemLm-Xccoppose22 (Dong et al., 2024)
43.11
50,41
48.89
43.75
34,09
10,00
MiniCPM-Llama3--2255 (Yao et al., 2024)
40.60
60.33
32.22
32.50
31.82
30.00
Llama-3-VILA115--88 (Lin et al., 2023)
32.99
46.90
32.22
28.75
26.14
5.00
Blava-vl.6-MMistraa/a (Liu et al., 2024b)
31.33
42.15
28.89
32.50
21.59
15.00
DocOwl1.5 (Hu et al., 20244
31.08
43.80
23.33
32.50
27.27
0.00
Paligemma-3B-minn (Beyer al., 2024)
30.58
37.19
25.56
38.75
23.86
10.00
Llava-1.5-7B-hff (Liu al., 2024a)
20.05
22.31
18.89
13.75
28.41
0.00
Human Performance
Human
82.23
81.67
82.42
85.18
78.41
65.00
Table 5. MapEval- Visual evaluation performance. (Fig 20 visualizes categorical accuracy).
deductions (Listing 5). LLM-based agents also struggle with map tools and APIs, especially in Nearby and Rout- ing queries. Incorrect parameter usage results in failures, such missing key parameters, using incompatible values, or entering infinite request loops when no valid results are found. Visual tasks reveal further issues, with VLMs strug- gling to maintain spatial awareness, misidentifying closely located POIs, or incorrectly counting POIs in map images (e.g., malls/stores). These limitations highlight the need for better spatial awareness, temporal reasoning, and robust tool usage (details Appendix E).
Straight-Line
Cardinal Direction (#24)
Distance (#47)
Model
L.M++
LLM+
LLM
LLM
Calculator
Calculator
Close-Souree (Proprictary)
LMMs
Claude-3.5-Somee
51.06
85.11
91.67
95.83
GPT-4o
46,81
70.21
62.50
87.50
GPT-4-Turbo
40.43
76.59
58.33
91.67
Gemini-1.5-PP
38.29
72.34
62.50
91.67
Gemini-1.5-4 Flash
46.81
63.83
58.33
87.50
GPT-4o-mini
34.04
78.72
29.17
91.67
GPT-3.5-Turbo
19.15
55.32
20.83
62.50
Open-Souree Source LLMs
Llama-3.2-90B
42.55
68.90
66.67
87.50
Llama-3 1.ma--31177BB
48.94
61.7
66.67
95.83
5. Addressing Failures and Enhancing Geospatial Reasoning in LLMs
Mixtral-888 8x7B
38.29
59.57
33.33
79.17
Gemma Gemma-2.0-9B
29.79
68.09
37.50
75.00
Failures in MapEval-TTxxuall To further analyze LLMs performance in -Txxuull we examined subset of questions requiring (i) straight-line distance calculations (47 questions), (ii) determining cardinal directions (24 ques- tions), and (iii) counting-reline related queries (23 questions). The results revealed significant variability model performance across these tasks (Fig. 11, Fig. 12, Fig. 13): (i) Claude-3.5- Sonnet achieved the highest accuracy (91 ()) in identifying cardinal directions, while Gemma-2.0 2.0B-BB scored the low- est (16.67%). (ii) Straight-line distance calculations posed challenges for all models, with the best accuracy at only 51.06%. (iii) Counting tasks also proved difficult, with
Table 6. Performance Improvement of LLMs (MapEval Textual) in Straight- Distance and Cardinal Directions Analysis (Fig. 14 and 15 visualizes the improvement).
even Claude-3.5-Sonee underperforming compared the open-source Gemma-2.0-277 (60.87% accuracy).
Calculator Integration for Complex Spatial Computa- tions: These results suggest that LLMs struggle with spatial reasoning tasks that involve precise numerical computa- tions. To mitigate these limitations, we extended model capabilities by integrating external tools (c.g., calculators) for tasks like calculating distances and determining cardi-
8

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
nal directions (details in Appendix G). This led to dramatic improvements (Table 6): (i) Claude-3.5-Sonnet' accuracy in calculating straight-line distances increased from 51.06% to 85. (ii) GPT-4o-mini, which initially struggled with cardinal directions, improved from 29.17% to 91.67%. (iii) Even open-source models like Gemma-2.0-9B saw accuracy boosts in straight-line distance tasks, rising from 29.79% to 68.90%. These results highlight the challenges LLMs face in spatial reasoning without external support. Integrating tools allows models to offload computationally intensive tasks, yielding more precise and reliable results. However, spatial reasoning is just one aspect of location-based tasks where models underperform. Temporal reasoning tasks, such as accounting for travel times or determining optimal visiting hours, could similarly benefit from tool integration. Expanding tool support could enhance performance across multiple domains but would also increase architectural com- plexity, requiring effective management of diverse tools.
significantly boosting accuracy. Another promising alter- native is developing an ensemble system that combines query classifier with type-specific LLM deployment. This system would first classify incoming queries and then route them to the best-performing L.M for that particular query type, potentially achieving superior performance.
6. Conclusion
In this paper, introduce MapEval, benchmark dataset designed to assess foundation models in geo-spatial rea- soning through textual, API-based, and visual evaluation modes. MapEval incorporates diverse real-world scenar- ios to evaluate model capabilities on geo-spatial reasoning tasks. Our findings reveal that while leading models like Claude-3.5-Sonnee GPT-4o, and Gemini-1.5-PPy excel in certain areas, they still underperform compared to human accuracy, especially when using open-source foundation models. This highlights critical areas for improvement, par- ticularly in managing complex map-based queries requiring multi-step apatio-tempora reasoning, efficient tool utiliza- tion, and domain-spedne focus on developing specialized geospatial models, integrat- ing LLMs with external tools like map APIs, and enhancing VLMs' visual understanding of map images. We anticipate that MapEval will catalyze ongoing research in geospatial reasoning and broader QA domains.
Failures in MapEval-API: In ReAct-based systems, a significant challenge arises from the heavy responsibility placed on a single agent to extract relevant parameters from a question, call APIs with those parameters, and then pro- vide the final answer based on API responses. This complex process often leads to issues such parameter extraction er- rors, incorrect API calls, or dead loops (e.g.. GPT-3.5-Tuure encountering 16 infinite iterations; see Fig. 19). These prob- lems become more pronounced when the agent struggles with reasoning through the task, reducing task completion rates. Moreover, processing large volumes of API data, even in plain text form (i.e., long contexts in MapEval- Textual task), poses significant challenge for LLMs, as discussed in Section 1 I and reflected in the low perfor- mances in Table 3.
knowledge. Future work could
Impact Statement
We ensure the reproducibility of our results by providing the complete dataset and evaluation codes used in our exper- iments² 3 These resources include the inference process for LLMs, detailing parameters such as temperature, top-k, and top-p. Any updates or bug fixes will be made available in the repository to maintain long-term usability.
Adaptive Routing of Tools and Models: limitations, the Chameleon Framework (Lu et al., 2024) of- fer a robust solution by adaptively breaking tasks into mul- tiple tool-usage modules .., multi-agent systems). The
To address these
Despite these efforts, our study has several limitations. The dataset focuses on five Google Maps APIs from the Places and Routes categories- -exxt Search, Place Details, Nearby Search, Directions, and Distance Matrix-exclinnn categories like Maps and Environment. This limits the di- versity of queries and geospatial insights evaluated. Addi- tionally, updates to these APIs may render the dataset less relevant for real-time applications, making it more suitable for archival purposes.
Category
ReAct
Chameleon
other
Place Info
39.06
54.69
Nearby
22.89
54.21
Routing
33.33
51.51
Trip
19.40
43.28
Unanswerable
15.00
25.00
Overall
27.33
49.33
The generalizability of our findings to other domains tools remains untested, and variations in prompt formulations, which may influence results, were not explored. Future
Table 7. Performance improvement of GPT-3.5-Turbo in MapEval- API
integration of Chameleon into MapEval-API has already demonstrated a notable improvement in GPT-3.5-Turbo performance (Table 7). Moreover, Chameleon's 's ability to decompose tasks and handle errors more efficiently results in fewer parameter extraction errors and prevents dead loops,
MapEval-Textine
http:////ggithbbbccm/aaaaaalal/a http:///ggchuub.com/aaaa//////// MapEval-Visual/

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
research could address these gaps by incorporating a broader range of APIs, examining the impact of prompt design, and testing the methods across diverse domains.
Koppula, S., Liu, F., Gryener, A., Gritsenko, A., Houlsby, N., Kumar, M., Rong, K., Eisenschlos, J., Kabra, R., Bauer, M., Bošnjak, M., Chen, X., Minderer, M., Voigt- laender, P., Bica, I., Balazevic, I., Puigcerver, J., Papalam- pidi, P., Henaff, O., Xiong, X., Soricut, R., Harmsen, J., and Zhai, X. Paligemma: versatile 3b vlm for trans- fer, 2024. URL https: http///axvvvvo//// htrg/abs/2407 07726.
By acknowledging these limitations and sharing our re- sources, we aim to support ongoing research and encour- age further exploration of geospatial reasoning tasks using LLMs.
Acknowledgment
Bollacker, K., Cook, R., and Tufts, P. Freebase: A shared database of structured general human knowledge. In AAAI, volume 7, pp. 1962-1963, 2007.
We gratefully acknowledge the support of Qatar Computing Research Institute (QCRI) for providing access to APIs and computational resources, including GPU support, which were instrumental conducting this research.
Chen, W. Parameterized spatial sql translation for geo- graphic question answering. In 2014 IEEE international conference on semantic computing, pp. 23-27. IEEE, 2014.
References
Abdin, M., Jacobs, S. A., Awan, A. A., Aneja, J., Awadallah, A. Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H., al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404. 142/9, 2024.
Chen, W., Fosler-Lussier, E., Xiao, N., Raje, S., Ramnath, R., and Sui, D. A synergistic framework for geographic question answering. In 2013 IEEE seventh international conference on semantic computing, pp. 94-99. IEEE, 2013.
AI. M. Mistral and nvidia collaboration, 2024. URL
Dihan, M. L., Ali, M. E., and Parvez, M. R. Mapqa- tor: A system for efficient annotation of map query datasets, 2024. URL https: http:///xxvvvvrooee.o 2412.21014
mistral-
Al@Meta Llama 3 model card. 2024. URL 1laaaallaaaaaaa
https: //ithhbb wwt.ub..co/mnie com/metaa
Dong, X., Zhang, P., Zang, Y., Cao, Y., Wang, B., Ouyang, L.., Wei, X., Zhang, S., Duan, H., Cao, M., Zhang, W., Li, Y., Yan, H., Gao, Y., Zhang, X., Li, W., Li, J., Chen, K., He, C., Zhang, X., Qiao, Y., Lin, D., and Wang, J. Internin-xcomposer2. Mastering free-form text-image composition and comprehension in vision- language large model, 2024. URL httpt:://aaaa org/ abs /400 16420.
blob/main/ /MMDEL CARD.me
Anthropic. Claude 3.5 sonnet. https://wwi
anthropic. www.co/////oooo /tom//wws////ooo 2024.
Auer, S., Bizer, C., Kobilarov, G. Lehmann, J., Cyganiak, R., and Ives, Z. Dbpedia: A nucleus for a web of open data. In international semantic web conference, pp. 722- 735. Springer, 2007.
Fang, B., Yang. Z., Wang, S., and Di, X. Travellm: Could you plan my new public transit route in face of network disruption? arXiv preprint arXiv:2407. 14926, 2024.
Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang. S., Tang, J., et al. Qwen2.5-vl technical report. arXiv preprint 13X2 2025.
GLM, T., Zeng, A., Xu, B., Wang. B., Zhang, C., Yin, D., Zhang, D., Rojas, D., Feng, G., Zhao, H., Lai, H., Yu, H., Wang, H., Sun, J., Zhang, J., Cheng, J., Gui, J., Tang, J., Zhang, J., Sun, J., Li, J., Zhao L., Wu, L., Zhong, L., Liu, M., Huang, M., Zhang, P., Zheng, Q. Lu, R., Duan, S., Zhang, S., Cao, S., Yang, S., Tam, W. L.. Zhao. W. Liu, X., Xia, X., Zhang, X., Gu, X.. Lv, X., Liu, X., Liu, X., Yang, X., Song, X., Zhang, X., An, Y., Xu, Y., Niu, Y., Yang. Y., Li, Y., Bai, Y., Dong, Y.. Qi, Z., Wang, Z. Yang. Z., Du, Z., Hou, Z., and Wang, Z. Chatglm: A family of large language models from 13m-130b to glm-4 all tools, 2024. URL https://aryy /ttp:///xxvvvv wwciiioorrg/bbb. abs 240. 12793.
Balsebre, P., Huang, W., and Cong, G. Lamp: A language model on the map. CoRR, 2024.
Bastani, F,, Wolters, P., Gupta, R., Ferdinando, J., and Kem- bhavi, A. Satlaspretrain: A large-scale dataset for re- mote sensing image understanding. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 16772-16782, 2023.
Beyer, L., Steiner, A., Pinto, A. S., Kolesnikov, A., Wang, X., Salz, D... Neumann, M., Alabdulmohsin, I., Tschan- nen, M., Bugliarello, E., Unterthiner, T., Keysers, D.,
10

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference Learning Representations.
Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reason- ing, and world knowledge, January 2024b. URL https://llava-va wttpraaaii.iiitt........ github.thhh..... citlub..o/bll/// 2024-01-20 -aaaa-0-laa---et--.-
Hu, A., Xu, H., Ye, J., Yan, M., Zhang, L., Zhang, B., Li, C., Zhang, J., Jin, Q.. Huang, F., and Zhou, J. mplug- docowl 1.5: Unified structure learning for ocr-free doc- ument understanding, 2024. URL htg/abbs/240..12880
Lobry, S., Marcos, D., Murray, J., and Tuia, D. Rsvqa: Visual question answering for remote sensing data. IEEE Transactions on Geoscience and Remote Sensing, 58(12):
8555-8566, 2020.
Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. Math- vista: Evaluating mathematical reasoning of foundation models in visual contexts. In The Twelfth International Conference on Learning Representations.
Jiang, A. Q.. Sablaynolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint 240::24000000888, 2024.
Karalis, N., Mandilaras, G., and Koubarakis, M. Extend- ing the yago2 knowledge graph with precise geospatial knowledge. In The Semantic Web-ISWC 2019: 18th In- ternational Semantic Web Conference, Auckland, New Zealand, October 26-30, 2019, Proceedings, Part 18, pp. 181-197. Springer, 2019.
Lu, P., Peng, B., Cheng, H., Galley, M., Chang, K.-W., Wu, Y. N., Zhu, S.-C., and Gao, J. Chameleon: Plug-and- play compositional reasoning with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Mai, G., Janowicz, K., Zhu, R., Cai, I.., and Lao, N. Ge- ographic question answering: challenges, uniqueness, classification, and future directions. AGILE: GIScience series, 2:8, 2021.
Kefalidis, S.-A., Punjani, D., Tsalapati, E., Plas, K., Pollali, M., Mitsios, M., Tsokanaridou, M., Koubarakis, M., and Maret, P. Benchmarking geospatial question answering engines using the dataset geoquestions 1089. In Interna- tional Semantic Web Conference, pp. 266-284, 284. Springer, 2023.
Mai, G., Huang, W., Sun, J., Song, S., Mishra, D., Liu, N., Gao, S., Liu, T., Cong, G., Hu, Y., et al. On the oppor- tunities and challenges of foundation models for geoai (vision paper). ACM Transactions on Spatial Algorithmss and Systems, 10(2):1-46, 2024.
Koh, J. Y., Lo, R., Jang, L.,, Duvvur, V., Lim, M., Huang, P.-Y,, Neubig, G., Zhou, S., Salakhutdinov, R., and Fried, D. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 881-905, 2024.
Open Geospatial Consortium. Oge geosparql geographic query language for rdf data. http: / //ww. www..opengeesppt..... org/ htg///tandadds..
requests/ 80, 2011. Document 11-05213.
Li, M.. Zhao, Y., Yu, B., Song. F., Li, H., Yu, H., Li, Z., Huang, F., and Li, Y. API-bank: A comprehensive bench- mark for augneneddd LLMs. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 3102-31 16, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18653w1/2023. URL https: bett:ss:/aclaateee htt///aaaaaaaoooocccccaao org/2023. emnlp main.187/.
OpenAI. Chatgpt, 2022. 2022. URL https: http://ppennaiiccc blog/ /catgpt......
OpenAL. Gpt-4 technical report. arXiv:2303.08874.00 2023.
arXiv preprint
OpenAL. http::/oppeaa.cccmm//eeeeelllllp
2024.
Punjani, D., Singh, K., Both, A., Koubarakis, M., Angelidis, I., Bereta, K., Beris, T., Bilidas, D., loannidis, T., Karalis, N., et al. Template-based question answering over linked geospatial data. In Proceedings of the 12th workshop on geographic information retrieval, pp. 1-10, 2018.
Lin, J., Yin, H., Ping, W., Lu, Y., Molchanov, P., Tao, A., Mao, H., Kautz, J., Shoeybi, M., and Han, S. Vila: On pre-training for visual language models. 2023.
Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L.., Lu, Y., Lin, Y., Cong, X., Tang, X., Qian, B., et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. The Twelfth International Conference on Learning Representations.
Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296-26306, 2024a.

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
Roberts, J., Lüddecke, T., Das, S., Han, K., and Albanie, S. Gpt4geo: How language model sees the world's geography. In NeurlPS 2023 Foundation Models for Decision Making Workshop.
Yuan, Z., Xiong, Z., Mou, L., and Zhu, X. X. Chatearthnet: A global-scale image-text dataset empowering vision- language geo-foundation models. Earth System Science Data Discussions, 2024:1-22 2024.
Sai, A. B., Mohankumar, A. K., and Khapra, M. M. A survey of evaluation metrics used for nlg systems. ACM Computing Surveys (CSUR), 55(2):1-39, 2022.
Zelle, J. M. and Mooney, R. J. Learning parse database queries using inductive logic programming. In Proceed- ings of the national conference artificial intelligence, pp. 1050-100000 1055, 1996.
Suchanek, F, M., Kasneci, G., and Weikum, G. Yago: a core of semantic knowledge. In Proceedings of the 16th international conference World Wide Web, pp. 697- 706, 2007.
Zhang, Z., Zhao, T., Guo, Y., and Yin, Rs5m and georsclip: A large scale vision-language dataset and large vision- language model for remote sensing. IEEE Transactions on Geoscience and Remote Sensing, 2024.
Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, I.., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXiv:2403.0553 05530, 2024a.
Zheng, H. S., Mishra, S., Zhang, H., Chen, X., Chen, M..
Nova, A., Hou, L., Cheng, H.-T., Le, Q. V., Chi, E. H., et al. Natural plan: Benchmarking Ilms natural lan- guage planning. CoRR, 2024.
Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, I.., Mesnard, T., Shahri- ari, B., Ramé, A., et al. Gemma 2: Improving open language models at practical size. arXiv preprint 0rXiv22408.001188 2024b.
Zhou, S., Xu, F., Zhu, H., Zhou, X., Lo. R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., al. Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learn- ing Representations.
Team, Q. Qwen2.5: A party of foundation models, Septem- ber 2024. URL https: htwpllwe.mmmmmmmmm////.....mm... github. io/ blog/ blog/qwen2. qwwn2.5/.
Zhuang, Y.. Yu, Y.. Wang, K., Sun, H. and Zhang, C. Toolqa: A dataset for Ilm question answering with exter- nal tools. Advances in Neural Information Processing 2023.
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W. Fan, Y., Dang. K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. Qwen2-vl: Enhancing viniun-language model's perception of the world at any resolution, 2024. URL https:/ / / arxiv. orgg abs 2209 12191.
Systems,
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F. Chi, E., Le. Q V., Zhou, D., et al. Chain-of-thougle prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Xie, J., Zhang, K., Chen, J., Zhu, T., Lou, R., Tian, Y., Xiao, Y., and Su, Y. Travelplanner: A benchmark for real-world planning with language agents. In Forty-first International Conference on Machine Learning.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. React: Synergizing reasoning and act- ing in language models. In International Conference on Learning Representations (ICLR), 2023.
Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Li, H., Zhao, W., He, Z., Chen, Q. Zhou, H., Zou, Z.,, Zhang, H., Hu, S., Zheng, Z., Zhou, J., Cai, J., Han, X., Zeng, G., Li, D., Liu, Z., and Sun, M. Minicpm-v: gpt- 4y level mllm on your phone. arXiv preprint 2408.01800, 2024.
12

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
A. Detailed Related Work
A.1. MapEval-Textual
Template-based GeoQA models (Zelle & Mooney, 1996; Chen et al., 2013; Chen, 2014; Punjani et al., 2018; Kefalidis et al., 2023) have predominantly followed a two-step strategy for answering geographic questions: (1) classifying a natural language query into predefined templates and (2) using these templates to query structured geographic knowledge sources such as PostGIS, DBpedia (Auer et al., 2007), YAGO (Suchanek et al., 2007), Freebase (Bollacker et al., 2007), and OpenStreetMap. While these approaches are effective for structured queries, they are limited by the predefined question templates and their reliance on static databases. They typically convert natural language questions into structured query language scripts. For instance, GeoQuestions 1089 (Kefalidis et al., 2023) contains 1089 questions with corresponding GeoSPARQL (Open Geospatial Consortium, 2011) queries over the YAGO2geo (Karalis et al., 2019) geospatial knowledge graph.
In contrast, our -aptval-Textual approach shifts the focus from database querying to assessing geospatial reasoning in Large Language Models (LLMs). Annotators collect factual map services data using MapQaTor, which then provided as context to LLMs. This setup isolates and evaluates the model's ability to reason over geospatial relationships, addressing the challenge of frre-form, complex map-related queries in dynamic environment. This approach allows for a more holistic evaluation of LLMss reflecting real-world usage where users interact with map tools using natural language queries. Thus, in MapEval, the responsibility lies with LLMs to answer the questions, whereas in previous works, the models were tasked with generating queries (e.g., Geoquery, GeoSPARQL), which are used to query external knowledge bases.
GPT4GEO (Roberts et al.) explored GPT-4's factual geographic knowledge by characterizing what it "knows" about the world without plugins or Internet access. Their evaluation focused on analyzing a single model using templated queries about generic location and direction-orienter facts, such as routing, navigation, and planning for well-known cities and places. However, this approach is inherently constrained by the training data of GPT-4, making it incapable of answering questions about knswn places. While the findings suggest that GPT-4 shows promising geo-spatial knowledge, this approach neither establishes a benchmark for geo-spatial reasoning nor incorporates real-life user queries or map services (e.g., Google Maps) as a geospatial information base.
Our approach employs fundamentally different evaluation and design principles. We establish a benchmarking of deeper geo-spatio-temporal reasoning capabilities across multiple foundation models using real user queries rather than templates. Uniquely, our evaluation encompasses multimodal understanding, tool interactions, and answerability determination. Additionally, we provide foundation models with fine-grained map services data through both context and API access, enabling a more comprehensive benchmarking of their geospatial question-answering abilities.
A.2. MapEval-API
The MapEval-API task adopts practical approach by leveraging map APIs to answer location-basse questions directly, providing a more real-world scenario for evaluating the capabilities of Large Language Models (LLMs) in map- map-based reasoning. Recent advancements in LLMS have led to growing interest in planning tasks (Xie et al.; Balsebre et al., 2024; Zheng et al., 2024; Fang et al., 2024) that involve map data. For instance, the Travel Planner (Xie et al.) benchmark assessed multi-day itinerary planning using Google Maps API to determine distances, travel times, and details of nearby attractions. This task demonstrated the utility of map data in real-world planning scenarios, highlighting the potential for LLMs to integrate real-time geospatial information into decision-making.
Additionally, tool-calling benchmarks such as ToolBench (Qin et al.) and API-Bank (Li et al., 2023) have included location- based queries as subtask, testing the ability of LLMs to interact with APIs in structured ways. These benchmarks typically focus on simpler query types, such as retrieving distances or nearby points of interest (POIs), but they do not fully address the complexity and diversity of real-world map-based questions.
In contrast, MapEval-APP pushes the boundaries by evaluating LLMs on a wide variety of complex geospatial tasks that require not only querying map APIs but also integrating multiple pieces of information, such travel itineraries, nearby services, and spatio-temporal reasoning. This more comprehensive evaluation of API-based reasoning challenges the models to process complex, multi-faceted questions, highlighting their ability to handle nuanced map interactions and effectively
13

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
synthesize data retrieved from APIs.
A.3. MapEval-Visura
Prior works in geospatial analysis and map-based question answering have predominantly focused on remote sensing images (Bastani et al., 2023; Yuan et al., 2024; Zhang et al., 2024), which involve satellite aerial imagery. These images often contain complex data about the Earth's 's surface, including land cover, vegetation, urban infrastructure, and other environmental features. Models designed for interpreting remote sensing images (Lobry et al., 2020) typically rely on convolutional neural networks (CNNs) and other computer vision techniques for object detection, segmentation, and classification tasks. These methods often focus on identifying physical entities like roads, buildings, and natural features from high-resolution imagery.
In contrast, our MapEval- SopEval-Visual approach focuses on digital map view snapshots, which are 2D representations of map services (such Google Maps). Unlike remote sensing images, which represent physical realities captured from a top-down perspective, these digital maps show geospatial information in structured, interactive format. The focus of MapEval- Visual is to evaluate a model's s ability to interpret and reason about these structured map views, which include not just physical features, but also symbolic and navigational information such as traffic signs, routes, landmarks, and visual cues from the map interface itself.
While remote sensing image analysis typically involves extracting physical data from raw image pixels, MapEval- Visual re- quires models to engage with spatial reasoning and map-based symbols, demanding a different set of computational skills, In this task, the model must not only understand the spatial relationships between map features but also reason about the context provided by digital map interfaces, which include additional elements such zoom levels, icons, and navigation markers. This distinction sets MapEval-Visun apart from traditional remote sensing tasks and presents new challenges in the field of geospatial reasoning and map-based visual question answering.
B. Data Collection Details
B.1. MapQaTor: Annotator Interface
For the creation of the textual contexts and design MCQs based on that, we employed a custom-built web interface named MapQaTor. As illustrated in Figure 4, this interface was central to the dataset development process, offering an intuitive, user-friendly environment that simplifies complex tasks, such as API interaction and context generation.
The annotator interface is designed to reduce technical complexity for users, allowing them to concentrate on the core aspects of dataset annotation, such as selecting relevant locations, providing information on distances, durations, and directions between places, as well as identifying nearby points of interest. Its streamlined workflow facilitates efficient dataset creation by automating repetitive tasks, which not only minimizes errors but also significantly accelerates the annotation process.
MapQaTor uses five key Google Maps APIs: Text Search, Place Details, Distance Matrix, Directions, and Nearby Search, based on their relevance common map-based tasks and their ability to provide comprehensive location data.
MapQaTor caches all API call responses, creating static database for evaluation purposes. This ensures consistent responses when evaluating MapEval-API. Specifically, when an API call is made, the cached response is returned instead of a real-time query, maintaining a controlled and static evaluation environment.
Once the dataset is generated, it can be easily exported in JSON format, making readily usable for further analysis and evaluation in downstream tasks, such model training and benchmarking.
B.2. Filtering via LLMs:
To ensure the challenge and quality of our dataset, we evaluated a range of LLMs. We filtered out samples where the majority of the LLMs could easily provide the correct answer, considering these samples "too easy" and removing them from the dataset. Additionally, we identified samples where most LLMs failed to answer the questions based on the given context. In such cases, we re-examined the questions, correcting any inconsistencies to improve clarity and relevance.
14

---

ansaggh MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
Context Generator
Map Dataset Creator
given
question appropriate
Add places in the Context google
Create new query
Contect
Search
SEARCH (9)
CLEAR
Show Contact
Places found In the Databasse
Places fetched from Google Map
exactly context)
Matto Espresso 1144 Loain gton Ave, New
Todlay Monday. have afternoon New York plan visit Metropofilan
Museum of Art for hours, followed by 30-minute coffee break at nearty cafe, and then (Neoz
speedd everything
Plan acdouuule
Control Park cafe 1070 5th Ave, New York, USA
York,
Oplions (Mark correct option)
NY 10020, USA
No places found
3.00PPM-M 6.00PM)) Visit The Matropp
of 6..02PM--- 3.32PM)
Petrie Court Café 1000 5th New York, NY 10022), USA
coffee at the Great Ballony Care 28.200---- 7.30-MM) Spend time in Central Park.
The Great Hall Belcomy Cafe 1000 5th Ave USA
(3.00PM 6.00PM) Vist The Metropo itan Musoum of Art 00.09PMM 6,38PM) Have 42FM
Opalon
Optin
6.00PM) Vistt Mctropo
(6.05PM 6.35MM)
7.39PMI Spard Central Park
PREVIQUS
(2 5.00PM) Metropo Museum
6.35PM)
cofine al Malli Fapresse > (5 47PM 47PM) Rpaind lime Dentral Park
Places Information
Cangery +bb
List of placas in the contaxt
(3)
B SAVE
Figure Screenshot of our Annotator Interface: MapQaTor
C. Evaluation Details
C.1. Psogle-Google Maps Environment
To ensure consistency between annotation and evaluation, pseudo-Google Maps environment was developed with the following features:
Caching: Information for over 13,000 locations cached using Google Maps place_ids ids during both annotation and evaluation stages, ensuring consistency across updates. Table 8 presents the number of data entries for each API tool in our database
API Simulation: A proxy interface mimics actual API interactions, enabling controlled testing while maintaining dynamic map-like attributes (e.g., travel times and place lists).
Key-Query Mapping: Discrepancies between user queries and database keys were handled by storing all data using standardized place ids obtained via a real API call.
This method maintains a static evaluation environment to preserve answer validity while simulating real-world API interactions by controlling dynamic variables like travel times, place attributes, and nearby location lists, which often change in live settings.
C.2. NapEval-Textune Textual Evaluation
In this evaluation setting, we provide the LLM with pre-fetched context containing detailed information about specific locations, such as opening hours, distances between points of interest, and nearby amenities. The context designed to simulate a real-world scenario.
15

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
Listing 4 demonstrates example of this evaluation process. The context includes details about The Metropolitan Museum of Art, including its location, opening hours, and nearby cafes. The query asks for time-optimized schedule that includes 3-hour visit to the museum, followed by 30-minute coffee break at a nearby cafe, and I hour spent in Central Park.
The available options offer different schedules, and the models are tasked with selecting the most appropriate one based on the provided context. As illustrated, models like Blaude-3.5-Sonee
Tool
Entries (#)
Gemini-1.5-Pro, and GPT-4o correctly identify Option 3 the best fit, considering the opening hours of each location and the feasible travel times between them. In contrast, Gemma-2.0-99 selects an incorrect option, indicating misunderstandingg of the cafe's s closing hours.
PlaceDetailkToon
13,354
TravelTime Tool
1,142
Directions
317
NearbySearchTool
481
This pre-fetched context evaluation allows us to test the model's ability to reason over structured information and make contextually informed decisions. It highlights the importance of understanding spatial relationships, operating hours, and timing constraints, all which are crucial in real-world trip planning tasks.
Table 8. Number of data entries in the datahase for each API tool
C.3. MapEval-API Evaluation
In this evaluation approach, we leverage a Zero Shot React Agent, which utilizes a dynamic tool-based framework to enhance the model's ability to respond to user queries effectively. Listing 6 illustrates the structured system prompt guiding the agent in employing various available tools. This framework allows the agent access range of functionalities, including retrieving place IDs, obtaining detailed information about locations, and estimating travel times between points of interest.
The Zero Shot React Agent's dynamic capabilities enable it to interact with tools in systematic manner, ensuring accurate and contextually relevant responses. For example, the agent can utilize the PlaceId tool to obtain the unique identifier for a specified location, which can then be employed in subsequent actions, such as fetching detailed information with PlaceDetails or finding nearby places using NearbyPlaces. This modular approach not only simplifies complex queries but also grounds responses in real-time data.
The prompt structure encourages the agent think critically about each step, starting with the user's question and leading to carefully considered action. It determines the appropriate tool use, specifies the necessary input, and provides well-structured JSON blob for the action. The observation of the tool's output informs the next steps, allowing for iterative refinement of the response.
By employing this Zero Shot React Agent framework, we can assess the model's proficiency in utilizing external tools to generate accurate, contextually aware responses, ultimately enhancing its effectiveness in real-world applications.
C.4. MapEval-Vissee Visual Evaluation
In this scenario, we provide Large Language Models (LLMs) with a map snapshot that offers critical geospatial context necessary for answering the query. This snapshot includes a default map view with clearly labeled locations and roads, aiding the understanding of spatial relationships.
The evaluation process is demonstrated in Listing 9. corresponding query with multiple answer options. In the example provided, the visual context includes map displaying several golf clubs and a complex roadway network. The options represent possible answers that the evaluated models can choose from. The listing also shows the responses of various models, along with their explanations.
The input consists of two parts: image visual context and
Models like Gemini-L.5-Prro GPT-4o-mini, and Claude-3.5 -.ndde--.--Sonee 5-nnnntt successfully answered the query by correctly interpreting the geospatial information. On the other hand, Swenn22VL-CC selected an incorrect option, highlighting its difficulty in understanding spatial distances, which led to an erroneous answer.
This evaluation underscores the importance of providing visual context when testing LLMs' geospatial reasoning capabilities. By leveraging such visual aids, we can better assess how well these models understand and process spatial relationships in real-world scenarios
D. Foundation Models' Details
Tables 10 and provide comprehensive details of the open-source models utilized for dataset evaluation.
16

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
Tool Name
Parameters
Description
PlaceSearch
placeName, placeAddress
Given a place name with address our tool calls Text Search API to get a list of places. Then choose the top place among them and returns its place id.
PlaceDetails
placeld
Given a place id our tool first searches in our database not found, then uses Place Details API to fetch the details of the place.
TraeelTime
originld, destinationld, travelMode
Given the place id of origin and destination, and travel mode tool first searches in database the duration (+distance) to go from origin to destination by preferred travel mode. If not found then queries Distance Matrix API.
Directions
originld, destinationId, travelMode
Given the place id of origin and destination, and travel mode our tool first searches in our database the available routes to go from origin to destination by preferred travel mode. If not found then queries Directions API.
NearbySearch
location, type, rankby, radius
This tool requires the place id of the place around which to retrieve place information. Additionally, the type of places, the order in which results are listed and distance within which to return place results. It then searches the database for stored Nearby Places. If absent, it queries Nearby Search API.
Table 9, Summary of the tools used in evaluation.
E. Fine-grained Qualitative Error Analysis
E.1. MapEval-Texxual
Commonsense Reasoning: (i) Consider scenario where the context states, "(Place A} serves dinner, lunch, vegetarian food." When asked, "Does {Place_A} serve breakfast?" many LLMs respond, "There is not enough information in the context to answer," instead of simply saying "No." A human would deduce that since breakfast is not listed, (Place A} does not serve it. (ii) Another challenge arises in planning questions. Even when opening hours are included in the context, LLMs may plan schedules that overlook constraints, such as visiting during closed hours while satisfying other conditions. For instance, although {Place_A A} is open from 9:00AM to 3:00PM, the model might schedule a visit at 5:00PM, possibly due to inadequate training for this scenario.
(i) LLMs particularly struggle with queries requiring the calculation of spatial relationships, such as cardinal directions, straight-line distances, nearest points of interest (POIs), or step-by-step route planning. For example, in Place Info, Nearby, and Routing questions, examining 50 random questions that required such computations we observed a 10% decreased accuracy than others. This decline highlights the limitations of even dominant models like Gemini, which struggle with straight-line distance and direction calculations from geo-spatial data. (ii) LLMs also encounter difficulty with our domain specific questions that involve maths even counting, especially when the count large. For instance, in a query like "How many nearby restaurants have at least a 4.5 rating?", LLMs often fail to provide an accurate count.
Spatial Reasoning:
Temporal Reasoning: LLMs struggle with temporal reasoning, which affects their performance on tasks like trip planning that require time manipulation. For example, when asked, "I want to visit A, B, and C. What is the most efficient order to visit?" the model must calculate travel times and determine the optimal route but often fails. Similarly, in a query like, "I want to visit A for hour. What is the latest time I can leave home?" the model needs to subtract the visit duration and travel time from A's closing time, yet frequently makes errors in these simple time calculations.
E.2. MapEval-API
Incorrect Tool Usage by Agents: LLM- based agents often exhibit varying degrees of errors when utilizing map tools/APIs, particularly impacting Nearby queries. This task requires a complex set of arguments, and misinterpretatton or improper use of these parameters frequently leads to failures in retrieving accurate results.
17

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
Context Window
Context Window
Model
Parameters
Model
Parameters
Phi-3.5-ma 1....55mmni-instee instruct
3.8B
128K
Qwen2.5-VL VL-72B-Instruc Instruct
72B
32k
Histral-Nemo-Instruce
7B
128k
Llama3.2-90B--V Vision
90B
128k
Hixtral-8x7B-IIstmnee
7B
MiniCPM-LJama3-VV
7B
8.2k
Swve2..5-7BIInstmun
7B
128K
Qwen2-V1--7--nnsee
8B
32K
Qwen2.5 1aee2.5--4B-Inttteee IIsttrcctt
14B
128K
Liama-3-VILAI.544
8B
8.2k
72B
128K
glm-4y-9b
4.9B
100k
Llama-3. 1-tia--.--B---sttuue
8B
128k
IntemLan-xcomposer2
7B
96K
Llama-3. 1lama-3.1-70B
70B
128k
aaligemma-3b-mix-
3B
Llama-3.2 -2--3B-Innttuct
3B
128k
DocOw11.5
8B
Llama-3.2 3.0---...-0----- 11-2--0--txxt-preee
-texx-preview
90B
128k
Ilava-v1.6
7B
gemma-2-27b-it
27B
8.2k
Ilava-1.5-7b-ff
7B
gemma-2-9ba.
9B
8.2k
Table 10. LLM model scales.
Table 11. VLM model scales.
Agents Stuck in Infinite Loops: Invalid actions and repetitive loops contribute significantly to errors, especially in Routing queries. When there are no valid routes between an origin and destination, agents often fail to reconsider their approach or stop the process. Instead, they repeatedly attempt the same query with the same parameters, resulting deadlock and preventing progress.
E.3. MapEval- VapEval-Visual Visual
Spatial Reasoning: In the Nearby category. models often exhibit confusion when multiple POIs are visually close together, leading to incorrect location selections. This indicates a struggle with fine-grained spatial analysis, affecting their ability to provide reliable responses and emphasizing the need for improved spatial awareness mechanisms.
Temporal Reasoning: In Routing queries, determining the fastest route requires detailed analysis of the source and destination, as well as transportation paths. VL.Ms often struggle with these calculations, resulting in noticeable decline in performance and underscoring the difficulties in processing geographical information effectively.
Detecting and Counting: Models often struggle to accurately identify and count POIs in map images. For instance, when asked, "How many shopping stores or malls are there?" many proprictary VLMs may count incorrectly, with Claude-Sonnet listing an ATM as store, leading overcounting. Conversely, they sometimes undercount, (e.g., detecting only 6 to out of actual 12 malls).
F. Dataset Statistics and Analysis
Country
Count
Country
Count
Country
Count
Bangladesh
United States
57
United Arab Emirates
India
33
Canada
31
United Kingdom
27
Japan
24
Australia
19
Pakistan
16
Qatar
15
Saudi Arabia
12
China
12
Germany
Argentina
10
Luxembourg
Italy
Spain
8
Brazil
South Africa
Poland
7
New Zealand
France
Denmark
7
Bhutan
Hungary
Czechia
7
Chile
Sierra Leone
Malaysia
Sweden
Norway
Peru
6
Colombia
Zimbabwe
Ireland
6
Mexico
Egypt
Greece
6
Austria
Indonesia
Nepal
6
Netherlands
Vietnam
Belgium
6
South Korea
Portugal
Morocco
6
Finland
Thailand
South Sudan
6
Russia
Switzerland
Turkey
6
Singapore
Table 12. Distribution of Questions Across Countries.
18

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
70
300
60
Number of Queries
Number of Queries
250
40
200
30
150
20
100
10
50
300
600
900
1,200
1,500
20
100
120
Number of words
Number of words
Figure 6. Distribution of Question Lengths (700 samples)
Figure Distribution TTxtual-Cooter Lengths (300 samples)
Figure (a)
Figure 6. (b)
Figure 7. Geographical Distribution Textual and Visual Contexts. The left heatmap (a) represents the locations of places mentioned in textual contexts, while the right hcatmap (b) shows the locations derived from map snapshots in visual contexts.
F.1. Zoom Details
In our dataset, zoom levels range from 8.0 to 21.0, shown in 10.Each visual context is paired with Google Maps URL, such as https:/www.goooge wwogll..co//map/eet.. 7048455, 3399766263,,, (1..66,,,...122222222 1e..1...11.. 16.712?entry=ttu, where the value before the "z" (e.g.. 16.71) represents the zoom level. This allows us to easily extract zoom information directly from the URL, ensuring that each visual context can be accurately mapped to its respective level of detail.
G. Additional Experiment Results
G.1. Addressing Failures and Enhancing Geospatial Reasoning in LLMs
For additional experiments, filtered questions from textual/API dataset into three subcategories:
1. Straight-Line Distance (47 questions): These questions require the computation of straight-line distances, such "What is the straight-line distance between the Atomium in Brussels and the Belfry of Bruges?"
2. Cardinal Direction (24 questions): These involve determining cardinal directions e.g., "What is the direction of the Little Mermaid statue from Copenhagen Central Station?"
3. Counting (23 questions): Questions involving counting entities, such as "How many convenience stores are there within 400 m radius of the Tokyo Tower?"
Shttps://en wikipedia .rg.
wttp:/wwwwwiiiiii.oo/ii//ii////e /aa.//iiii//aaaiaaaa rection
19

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
Yustima
Sacchm
Vingyda
YUSHIMA
CRCUE
.0pa
Biver
North Dceen
ye Watewwrr wores
Figure 8. Zoom level
Figure 9. Zoom level 16
150
Number of Visual Contexts
120
90
60
30
15
18
Zoom Level
Figure 10. Distribution of Zoom Levels (400 Visual Samples)
We visualized the accuracy of various LLMs on these subcategories under the HoppEva--Textur setting, with the following findings:
1. Straight- Line Distance: Figure 11, illustrates the accuracy on straight-ling distance related questions. We can see that models struggled, with the best accuracy being only 51 .06%.
2. Cardinal Direction: Figure 12, illustrates the accuracy on pardinal-direction related questions. Here LLMs showed significant variability. While -andee-35-Sonne achieved 91 % aceuracy, Gemma-2.0-277 scored only 16.67%.
3. Counting: Figure 13, illustrates the accuracy on counting related questions. In this case, Claude-3.5 Bauude-3,5-Some 3.5-Sonnet underper- formed compared to the open-source Gemma-22.027B (60.87% accuracy).
We identified scope for improvement in these areas and enhanced the models' capabilities by integrating external tools (e.g., calculator) specifically designed for calculating straight-line distances and cardinal directions. For straight-line distances, employed the Haversine formula to compute the great-circle distance. To determine cardinal directions, calculated the bearing" between two geographic coordinates. Figures 14 and 15 demonstrate significant improvement in model performance with these tools.
For straight-line Distance-related questions, the best accuracy jumped from 51.06% to 85.11%. Similarly, for cardinal- direction questions, the top model achieved an accuracy 95. 95,83%, compared to the previous maximum 91.67%. In the case of GPT-4o-mini, these enhancements led even further progress, with the model demonstrating leap both straight-line distance and cardinal direction accuracy, surpassing previous models. In the case of GPT-4o-mini, these enhancements led to even further progress, with the model demonstrating remarkable leap in both straight- distance and cardinal direction accuracy. Specifically, the straight-line distance accuracy improved from 34.04% to 78.72%, while cardinal-directto accuracy increased from 29.179. to 91.67%.
"https: httee::/ee..wiiiiieeee httpr///wwikaavesiiiiiiee htw./aavveesiiee formula Iormil 'https://ent http://wwwwiiieiia...o.o.ee htt.:////.iiieeeeainnnnneeeeeaee (navigation
20

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
100
Accuracy (%)
80
60
40
20
0
Qwen2.5- 14B
Claude 3.5 S-onneet
Alama-3.1-70B
Gemini- 1.5 1.5-lash
GPT-40
Llame-3 3.1 3.1-8B
Qwen2.5- 72B
Llama-3.2 90B
GPT-4-Turbo
Qwen2.5-7B
Mixtral-8xx -x78B
Gemini-L.5-PPro
Mistral- -emoo
GPT-4o-mini
Phi-3.5-Mini
Llama-3.2 3.2--3..-3BB
Gemma-2.0- 2.00-27B
Gemma-2.0-99
GPT-3,5-turbo
Figure 11. Accuracy of LLMs on questions which needs Straight-Line Distance computation
100
Accuracy (%)
80
60
40
20
0
Claude S-Sonnet 3.5-onnett
Llama- Llama-3.1-700 3.1-70B
Llama-3.22 3.2 3.2-90B
Qwen2.5-722
GPT-4o
Gemini- 1iminni--.-PPmm ..5Prr
GPT-4-Turho
Gemini-1.5 -5-Flash
Llama-3. 3.18-B31--BB
1-8B
Mistral Nemo
Qwen2.5-77 5-Bn2.5-7BB
Gemma-2.0-98
Mixtral- -x
Qwen2.5-14B
GPT-4o-mini
Phi-3.5-Mini
GPT-3.5-turbo
Llama-3.2-3B
2.0-2a-2.0-2BB
Gemma- 2.0-27B
Figure 12, Accuracy LLMs on questions which needs Cardinal Direction computation
These results highlight the limitations of current LLMs handling fine-grained geospatial queries independently and emphasize the value of augmenting LLM capabilities with external computational tools. Future work explore the integration of more robust external services to address the nuances of spatial reasoning comprehensively.
G.2. Open-Ended Evaluation
To demonstrate the flexibility of MapEval beyond the MCQ format, conducted additional experiment where removed the answer choices from both textual and visual variants of the benchmark and evaluated open-ended responses from Claude-3.5-Somee The outputs were automatically graded using the o3-mini model.
Tables 13 present the performance results across all categories. While the performance in the open-ended setting is generally lower compared to MCQ, it shows that the queries remain valid and evaluable without answer options. Notably, Claude-3.5- S-nnett still performs competitively in several categories such as Routing and Trip.
We also observed limitations of automatic evaluation. For example, in the Unanswerable category of MapEval-TTetuall 03- mini initially assigned an accuracy of 55%, but manual inspection revealed that approximately 80% of Claude-3.5-Somee responses were actually correct. This illustrates the challenge of reliably assessing open-ended spatial reasoning with current tools.
Moreover, open-ended evaluation effectively doubles API costs: each query requires two calls-one one to generate a response and another score Since longer, free-form responses use more tokens, cost scales significantly compared the MCQ
21

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
100
Accuracy (%)
80
60
40
20
0
Gemma- 1amma--..02211 2.0-27B
Claude- 5-Sonnet
Qwen2.3-725 3-72B
GPT-4-Turbo
Mistral-Nent -emoo
Qwen2.5- 14B
1-1B
Phi-3.5-Mini
Gemini-1 1..--ro
1.ma-3.1170p.
Gemini- 1.5 -5--lash
GPT-40
Llama-3 3. 1--8-
Llama-3.2 2-B
GPT-4o-mini
Qwen2.5-7B
GPT-3.5-turbo
Llama- Llama-3.2-30 3,2-3B
Gemma-2.0-0
Mixtral 8xx7B
Figure 13. Accuracy of LLMs on Counting related questions
100
Accuracy (%)
80
60
40
20
Claude-3.5 3.5 3.uude-3,5-onne S-Soneet
Llama- 1.ee-..1-71000
Gemini- 15-Flash
GPT-4o
1lama-3,2-900
GPT-4-urbo
Mixtral 8x7B
Gemini-1.5-Pr
GPT-4o-mini
Gemma-2.0-99
GPT-3.5-turbo
In LLM Dr LLM+ Calculator
Figure 14. Improved accuracy of L.LMs after integrating calculator to compute Straight-L.in Distance
setting.
Thus, while MapEval fully supports open-ended evaluation, MCQ format remains more cost-effective and scalable solution for benchmarking LLMss.
MapEval-Textue Textual
Evaluation
Overall
Place Info
Nearby
Routing
Trip
Unanswerable
Open-ended
55.33
43.75
43.37
69.70
67.16
55.00
MCQ
66.33
73.44
73.49
75.76
49.25
40.00
Map Visual
Evaluation
Overall
Place Info
Nearby
Routing
Counting
Unanswerable
Open-ended
51.88
67.70
51.11
35.00
43.18
65.00
MCQ
61.65
82.64
55.56
45.00
47.73
90.00
Table 13. Performance Claude-3.5-Somee under open-ended MCQ settings.
22

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
100
Accuracy (%)
80
60
40
20
0
Claude-3.5 S--Snnet 3.5-onet
Llama-3.1- 70B
Llama-3.2 2-.---0B
GPT-40
Gemini-1 Demini-1.5-Prr 1.5-Pro
GPr-o-Turo
Gemini- 1.5-Flash
Gemma-2.000 -.0-9B
Mixtral-8x7BB
GPT-4o-mini
GPT-3.5-turbo
LLM Dr LLM +
Calculator
Figure 15. Improved accuracy of LLMs after integrating calculator compute Cardinal Direction
G.3. Fine-Tuning on MapEval
To assess whether the challenges posed by MapEval stem merely from lack of training exposure, we conducted additional experiments fine-tune several open-source models subset of -eptyallTTextul.
We randomly split the 300 MCQ questions into a training set of 97 ques- tions and test set of 203 questions. We fine-tuned five representative models-Phi-3.5- Phi-3.5-mini, LLama-3.2 3.2-33-.---3, Qwen-2.5-7B, LLama-3. and Gemma-2.0-99 on the training portion using standard instruction tuning.
Model
Pretrained
Finetuned
Phi-3.5-mini
39.90
34.48
LLama-3.2-2
34.98
35.96
Qwen-2.5-7B
41.87
43.35
As shown Table 14, performance gains from fine-tuning were generally small (< 5%), and in some cases, performance slightly declined. Even after fine-tuning, all models perform significantly worse than leading LLMs such as GPT-4o, Saaude-3.5-Soneee or Gemini 1.5 Pro.
LLama-3.1-8B
46.31
44.33
Gemma-2.00999
46.80
51.23
Table 14. Performance comparison of open- source models on MapEval-Textur (test set) be- fore and after fine-tuning 97 MCQs.
These results suggest that the difficulty of MapEval is not solely attributable to lack of task-specific examples. Instead, the benchmark reveals deeper limitations current LLMs' ability to handle fine-gredt grained, geospatial rea- soning tasks. We believe this highlights the need for more advanced training strategies and dedicated geospatial reasoning capabilities in future LLMs.
H. Evaluation Results Visualization
In this section, we present the results our evaluations through series of charts that summarize the performance of different models across various categories. These visualizations provide a clear and concise comparison of model effectiveness in addressing textual, API-based, and visual geospatial queries. The charts are designed to highlight key trends, strengths, and limitations the evaluated approaches.
H.I. HappEval-Textur
Figure illustrates the performance of models on MapEval-TTextall
Figure 17 illustrates how accuracy of models in MapEval-Textual changes with different context length.
H.2. MapEval-API
Figure 18 illustrates the performance of models on MapEval-API.
23

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
100
Accuracy (%)
80
60
40
20
0
Place Info
Nearby
Routing
Trip
Unanswerable
Claude-3.5-Sonne
In Gemini-1.5-Pra
GPT-4o
GPT-4-turbo
Ir 1--70B
Gemini-L.5-Flass
Llama-3.2-90B
Qwen2.5-72B
II
Qwen2.5-14B
Dn GPT-4o-mini
Gemma-2.0-277B
Gemma-2.0-99
Llama-3.1
Du
Qwen2.5-7B
11 Mistral-Nemo
Mixtral-8x7B
GPT-3.5-turbo
D Phi-3.5-mini
Llama-3.2-31
Figure 16. Hoppvaa--Txttur categorical accuracy
100
Accuracy (%)
80
60
40
20
0
0-300
301-600 (26.67%)
601-900 (9.67%)
901-1200
1201-1500
(52.33%)
(5.67%)
(5.67%)
Context Length (data percentage)
II Claude-3.5-Some
Dn
Gemini-1.5-Pra
GPT-4o
GPT-4-turbo
Llama-3.1-700 1-000B
Du Gemini-1.5-Flash
Llama-3,2-901
Qwen-2.5-72B
Qwen-2.5-148
Du
GPT-4o-mini
Gemma-2.00277
Gemma-2.0 2.0--9
Llama-3.1-8B
Qwen2.5-7B
11 Mistral-Nemo
Mixtral-8x77 -xtrall-xxBBB
[[ GPT-3.5-turbo
0| Phi-3.5-mini
Llama-3.2-3B
Figure 17. Accuracy vs. Context Length MapEval-Texxuu
100
80
Accuracy (%)
60
40
20
Place Info
Nearby
Routing
Trip
Unanswerable
D] Haude-3.5-Somee
In GPT-4-Turbo [] GPT-4o
On Gemini-I.5-Flash
D Gemini-1.5-. 1,5-rr
lu Llama-3.1-70B Iı LJama-3.2-90B
Iu GPT-3.5-Turbo
0
Mixtral-8x777
lu Gemma-2.0-99
Оu GPT-4o-mini
Figure 18. MapEval-API categorical accuracy
24

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
Additionally in Figure 19, can visualize the number of times agent stopped due iteration limit. This happens when agents repeatedly calls the same api with same parameters and gets the same response. While GPT-o.5-Turbo encounters 16 infinite iterations, Soude-3..-Soomee doesn't I face this issue.
20
15
Count
10
5
0
Sonnet
G-Pro
GPT-4o
GPT-4
LI3.1
G-Flash
L13.2
4o-mini
Gemma2
Mixtral
GPT-3.5
Figure 19. Number of times agent stopped due iteration limit
H.3. MapEval-Vin Visuual
Figure 20 illustrates the performance of models on MapEval- Visual.
100
80
Accuracy (%)
60
40
20
0
Place info
Nearby
Routing
Counting
Unanswerable
Claude-3-5-Sonnet
1 Gpt-4o
NI Gpt-4-turbo
I Gemini-1.55
1 ] Gpt-4o-Mini
Il
Gemini- 1.5-Flash
D] Qwen2.5-VL-72B
lo Qwen2-VL-7B
] Llama-3.2-90B
] MiniCPM-V-2 5
D| Llama-3-VILAI.5-.
ln Glm-4v-9b
1 CheemLm-Xxcompssee
Paligemma-3B
Ia DocOwl1.5
I|
1] Llava-1.5-7B
Figure 20. MapEval-Visse Visual categorical accuracy
80
Accuracy (%)
60
40
20
8-11 (2.5%)
11-14 (28.5%)
14-17 (50.0%)
17-20 (17.2%)
20-23 (1.8%)
Zoom Level (data percentage)
[| Claude-3-5-Sonnet
In Gpt-4o
[] Gpt-4-turbo
I Gemini-1.5-PP
D Gpt-4o-Mini
Iu Gemini-L.5-Fles
[ Owen2.5-VL-72B
Iu Qwen2-VL-7B
D[ Llama-3.2-90B
Do MiniCPM-V-2.5
D Llama--VVILA1.5-8B
II Glm-4v-9b
D[ Interl.m-Xcooppseed
1n Paligemma-3B
I DocOw11.5
[] Llava-vi 1.66 6-Mistral-7B
D[ Llava-1.5-7B
Figure 21. Accuracy by Zoom Level.
25

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
I. Qualitative Examples
Type
Task
Question Example
Which coffee shop is situated between Louvre Museum and Eiffel Tower?
Textual/API
What is the direction and straight-line distance from Victoria Falls to Hwange
Place Info
National Park?
I'm at Baridhara K Block. feeling unwell, and need some medicine. What is a nearby pharmacy with good rating that is open?
Visual
How many shopping malls are there within a 500 m radius Berlin Cathedral? am at Toronto Zoo. Today Sunday and it's currently 8:30 PM. How many nearby ATMs are open now?
Textual/API
Nearby
I'm currently staying at Hörselberg-Hainidee
while my friend is staying at Tüngeda. After meet up. I want visit amusement park nearby. Can you suggest one that's close to us?
Visual
I want to walk from D03 Flame Tree Ridge Aster Cedars Hospital, Jebel Ali. Which walking route involves taking the pedestrian overpass?
TextualAAPI
On the driving route from Hassan II Mosque to Koutoubia via A3, how many roundabouts will encounter in total?
Routing
Which restaurant on the left side of the route from Metro Golf to Metro
Visual
Tobalaca LI?
How many food stalls are there north of the overbridge at Hakaniemi?
Textual/API
Find good coffee shop the left side of my driving path from my home near Petaling Jaya my office in Kuala Lumpur.
Unanswerable
How much time it would take to go to Igreja Nossa Senhora Da Conceição do Coroadinho Matriz?
Visual
live in Indira Road. At tomorrow 2 pm will leave my house. need to go to Military Museum to visit with friends for hours and Multiplan Center to buy keyboard (which will take 20 minutes) and Sonali Bank, BUET to receive my check book (which will take 30 minutes). In which order I should visit the places that I reach there time and come back home early possible. will use public transport.
Trip
Textual/API
Counting
Visual
How many restaurants or clubs are on the bottom side of Linnakatu road?
Table 15. Additional Complex Examples
26

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
Listing 1. Example evaluation of TennEval-Txxtull Green
Correct Answer. Red
Wrong Answer.
Context:
Information of ELffel Tower:
Location: ky. Gustave Biffel, 75007 Paris, Prance (18.8584, 2.2945).
Information of Most Saint Michel: Location: 50170 Nont Saint-Wiched France (18.6361, -1.5115).
Query:
What is the streight straight-line line distance between the Biffel Tower in Paris, France, and the Mont Saint -iichel in Normendy, France?
Frompt:
Please respond in the following JSON format:
"optiin_naa"s soption index>, // "optioon on_ refers to the number corresponding to the chosen answer from the 11st options. It should be between and 4.
"expl anat on?: "creason>"
example prompt:
Question: the capital Francs?
Optionl: Berlin
Option2: Paris
Option3: Madrid
Option4: Rone
Example Response:
"option_no"" 2,
"axpl anation": "Paris is the capital of France."
Previde your answer format. using external knowledge or assumptions.
question strictly based on given context, without
Opt tons:
1. 230 km
3. 280 km
Calude 3.5 S..S-Sennet:
To answer this question, I'll need to calculate the straight-line distance between the two points using their given cocrdinates. 1'12 use the llaversine formula for this calculation. Biffel Tower: (48.85E2 8584, 2.2945) Mont Saint Michel: (48.6361, -1.5125) after performing the eslculation, the approximate distence is 234 Based this result, the clesest option is 280 km.
Comini-1.5-Pros
cannot ca culate the straight- distance between the Riffal Towey and Kont Saint-Michel without vaing external This salculation sequices geographic
knowledge
machems:ical Iormulas.
CPT-40:
straight-line distance between the coordinates of the Riffal Tower (48.8584, 2.2945) and Mont saint=Xichel 5361, -1,5115) is approxima:elyy 280 km.
Comma-2 2.ma--2.09881
To determing the straight-line distonce, we'd need to use the coordinstes provided and apply a distance formula. question based given context...
Unfortunately, can't perform calculatioons **hherefore,
27

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
Listing 2. Example evaluation of MapEval-Textus Green Correct Answer. Red
Wrong Answer.
Context:
Information Lon of Beurasaari Open Air Museun:
Location:
Seurassers, 00250 Helsinki, Pinland 160., 24.8948)
Information of Helsinki Central Station:
Location: Knivoketu 00100 Helsinki, vinlendi60.1113 24.9414).
Query:
What is the direction of the Seurassari Open-Aiz Museum frem Helsinki Central Station?
Frompt:
Please respend in the follewing J90N format:
"option_na": coption indax>, // "aption_na" refers to the number corresponding to the chosen answar from the 1ist options. should
and
"explanation"! "creasen>"
Example Prompt:
Question: what is the capital of France?
Optiont: Berlin
Option2: Paris
Option3: Madrid
Option4: Rone
Example Response:
"option_no": 2,
"explanation"; "Pazie the capital
Previde your in this fermat. Remember the question strictly based the context, without uning any external knowledge or assumptions
Opt lonso
East
2. Weat
3. No:th
South
Calude 3.5 Sonnet:
To determine the direction of Seuraseari Open-Air Maseum from Helsinki Central Station, I'll compace their
cocrdinates: Seurasaari Opez-Air Kuseum: 60.1856, 24.8846 Helsinki Central Station: 60.1719, 24.9414 Analyzing the
cocrdinates: 1. Latitude: Seuraseari (65E) is slightly north of Helsinki Central Statier. (60.1719) 2. Lengitude: Station (24.9414) primary
seurasaari (24.8846)
west, with slight
northward componert.
Comini-1.5-Pre
Seurasaari Open-Air Museum (60. 1856, 24.8846) is located at higher laticude and almest Longitude compared Helsinki Central Station (50.1719, 24.9414). lligher latitude corsesponds so North.
CPT-401
Seurassari Open-Air Museum located to the north Feleinki Central Station based en their coordinates.
Gemma-2 0.0
Saurasoari Open-Air Museum locatad at 60.1856, 24.8846 Helainki Cantral Station located 60.1719, 24.9414 9414, The museun's coordinates are further north.
28

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
Listing Example evaluation MapEval- Textual Green Correct Answer.
Red
Wrone Answer.
Nearby Restaurants St. Lavrence Mazket
1. A&W Canada
Open: Morday: Oper 24 hours
Tuesday: upen 24 hours, wednesday: Open 24 hours, Thursday: Open 24 hours, Friday:
open 24 hours, Saturday: Open 24 houre, Sunday: Open 24 hours.
7. quik Sushi
Open: Konday: 9:00AM 7:00PM
Tuesday: 11:308M
Wednesday: 9:00AM 7:03PM, Thursday: 9:00AM Sunday: 9:00AM-7:00PM.
Friday: 9:00AM-7:C0MM, Saturday:
9: 9:00--77:00M
9. Big Pita
Open: Konday: 10:30AM-9:00PM
Tuasday: 30AM- :0PMM, Wednasday: 10:30AM-9:00FM, Thuraday: 10: 30:M-990PPMM
Friday:
10:30AN 10:30AN-3:00PM, Esturday:
10:00M49:
Sunday:
14. Pizza pizza
Open: Konday: 11:00AM-1 100AY
Tuasday: 11:02AM-1 10GAM, Wednaadays 11:00AM-1;
Thuraday: 11:0004-1:C0AM
Fridays :000MM-:00AMM Saturday: 1000M-2:COMM, Sunday: 11:000-1::00MM
Market sereet Catch
Open:
Konday: 0::000
luesday: 9: Wednesday: 11:00AM-9: 9:00M, Thursday: 0::00000::::::0
Friday: 11:00AM-260 11:00M-LC::0PM, Saturday: 11:00M--1::0PPM Sunday: 11:00AM-8:00PM.
20. Churrasce of The St Lawrence
Open: Monday: Closed, Tuesday: 7:00AK-7: :00EM, Kednaaday: 7:00AM-7:00PM, Thursday: 7:C0AM-7:30PM, Friday: 7:00AM
-7:00PM, Saturday: 7:0GMM-7:00PM, Sunday: 10:00AN-::00MM.
Query:
low many nearby restaurents of St. Lawrence Market are on Monday?
Prompt:
please respond following J8ON format:
"option_na": coption index>, // "option_na" refers to the number corresponding to the chesen answer from the list optiona. should
4,
"explanation"; "creasons*
ixample Prompt:
Quession: the capital France?
Optionl: serlin, option2: paris, option3: Madrid, Uptiond: Rome
Example Response:
"option_ao": 2,
"explanation": "Paris capital Prance."
Provide your answer thta format. Remember arswer the question strietly based on the given context, without using any external knowledge assumptions.
Options:
1. 2
2.
3.4
Calude 3.5
Based the information, nearby restaurants open Konday. These A&ll Canada 24 hours), Quik Sushi (open 9:00 AM 7:00 P8), and Big sita (open 10:30 AM 9:00 FM). All other listed restauranta are either clesed Monday their Monday hours specified in provided information.
Com-ni-1.5-Pras
provided information, these restauranta open Monday: Canada, Quik Suaat, Pisza Plesa, Market Street Catch.
CPT-401
Only AdM Canada and Quik Sushi are open on Monday,
Gemma-2, 2.0ma-2.0099: 0-98:
Only A&R Canada is open 24 hours on Monday.
29

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
Listing Example evaluation MapEval-Texttul Green Correct Answer. Red
Wrong Answer.
context:
Information of The Metropalitan itan Musaum
Location: 1000 5ta Ave, New York, NY 10028, USA,
Tuesday: 10:00AM-5:30PM, Madnesday: Closad, Thuradoy: 10:00AM-5:COFM, Friday: AX-9:00FM, Saturdays 10:00M-99:05-MM Sunday: 50:00MM5::00PM
open: Monday: :C:00AM-5:00PPM
Travel
Metropol itan Museum of Art American Wing Cafe foot minss (0.3 kmt.
Travel
Netropolitan
Balcony Cafe foot km)
Nearby Cafea of The lat Kuseum of Art are (sorted by distance in zacanding order): York! 2. The Great Bal: Balcony Cafe (1000 Sth Avenue 2nd Flouz, New York) Cours (1000 Sth Avenue, New York)
American Wing Cate (1000 Avenue,
Starbucks (1142 Madison ävenue, New York)
Query:
Today Monday, have efternoen
plan to visit The Mesropolitan Museun for hours, followed by 30-minute coffee break at nearby cafe, and then spend 1 hour in Central zark (New York, NY, USA) plan a schedule ensure have enough time everything. will walk between planes.
P PrompL:
Please respand in the following JSON format:
"option_na": coption index>, "option_ne" refers of optiona. It should be between and 4.
corresponding to answer from
"explanation"; "creasons*
example Prompt:
Quession: what is the capital of France? Optionl: Berlin, Option2: Paris, option:: Macrid, Option4; Rome Example Response:
"option_no"; 2,
"explanation": "Paris is the capital France."
Provide your
in this format. Remember using any external knowledge or assumptions.
the questioo strietly based the context, without
Opsiens:
1. 13. (3.00PM DCPM - DOFM) Visit The Metropol itan Museum of Art + (5,02PM 6.32FM) Have coffee at The Great Hall Balcony
(5.38FM 7.382M: Spend Lime in Central
2. (3.0CPM 6. (0PM) Visit The Netropolitan Museum of Art + (6,08PM
6.38))) Have coffee at Central Park cafe +
46.4
7.422M) Spend Line in Central Pack.
(3.00FM 6.00PM) Visit The Metropolitaa Museum of Ast + (6.05PX Pazk.
6. 3.35MM) llave coffee at Starbucks + (6.39
FX
7.392M1
Spend time
4.
12.00PM
5. D0FM) Visit The Metropolitan Muaeum of Art (5. (9,09PM 5.39FX) Hlve coffee at Matto Espresao
15.4 6.4728) Spend time in Central Pack.
Calude SonneL:
Option is the nost suitable schedule based the given information. The Ketropolitan Museum of Art is open Mondays from 10:00 An to 5:00 PM, so visiting from 3:00 24 to 6:00 PM is possible. Starbucks is open until 7:30 eu Mondays, making
viable option for coffee break after the museun visit. Central Park in open until 1:00 AN, sc spending time there from 6:39 RM to 7:39 PM is feasible.
Gemini-1.5-Pros
Option3 is the only option that has sime slets within the opening hours of all the places menticned. The ropalitan Musevm of until 1:00AX.
open until 5:00PM Monday, Starbucks la open until 7:30PM Central Park open
CPT-40:
Option is feas"ble as Starbucks is open Monday and is rearby. The schedule allows for a 3-hour visit to The Metrooolitan Museum of Art, a 30-minute coffee break, and hour in Central Park.
Gomma-2.0-98t
Option provides the most feasible schedule. It allows 3-hour visis to the musous, a 33-minate coffee break at The Great Bal. Balcony Cafe (which is within walking diatance), and 1-hour visit to Central Park, all while ensuring enough time for travel between locations.
30

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
Listing 5. Example evaluation of MapEval- Textual
Green Correct Answer.
Red
Wrong Answer.
Context:
There are routes from Petaling Jaya to Rederal Territory Ruala Lumpur by ear, They a2e:
1. Via Damansera Link/Lebuhraya SPKINT/Siatem Penyuraian Trafik Kuala Lumpur Barat/a23 21 mins
15.2 km
Read cast on Forsiaran Tropicana toword Persieran Tropicara
3. Via Damaneaza Liak/Lebuhzaya SPRINT/SLatem Penyuzalan Trafik Kuala Iumpur DazaL/B23 and Jalan Maarof 25 mins
leaad eaat on Persiaran Tropicana toward Persiaran Tropicana
Nearby Cafes of Petaling Jaya (sorted distance in ascending order):
1. Dareia Caffee
Pa2-13 Tropicana, Persiaran Tropicana, Tropicana, Petaling Jaya.
Rating: 5. (7 zatinge).
Nearby Cafes of Pedezal Territory of Kuala Lumpur (sorted distance in sacending ordez):
1. Sweku Coffee
Address: Labby Nenara 1 sentral, Jalan Stesen Sentral Kuala sumpur Sentral, Kuala Sumpur. Rating: 4.2, (62 ratings!.
Que
Find good cotfee shop the left aide of driving path From home near Petaling Jaya to office in Kuala Lumpur.
Prompt:
respond follewing
"option_ne": coption index>, "option_no" refers to the number corresponeing to the chosen answer fcom the 1ist options. It should be between and 4.
"explanation": "creason>*
Example Prompt:
Questtons What is the capital I. of France? Option0: Unanswerable, Optionl: Berlin, Option2: Faris, Option3: Kadrid, Optioad: Rome Example Respense:
"option_na"""
"explanation": "Parle is the capital of France."
Provide your answer in this Tormat, Remember to answer the question strie.ly based on the given context, without using any external knowledge or assumptions.
Options:
Cnanswerable
1. Dazela Coffee
Coffee
3. The Lemon Tree
4. Coffeeloca
Calude 3.5
The Lenon Tree is good coffee option the left side of the driving path from Fetaling to Kuala lumpur. it is located at the Ground zloor of Kenara Lien lee, which is slong Persiaran iropicana. This cafe is on the initial of the journay and has rating of 4 out of basad 836 ratings, indicating it'a popular and well- liked spot. It's also described as having a Moderrate price lavel, making it suitable choice for coffee stop on way so office.
comini-1.5-Prs
There 15 no informatior on aide the driving path the cafes are located.
co1-40:
Located
Stasen Sentral
on driving path Petaling Jaya Lumpue.
Genma 2.0 9B:
Located lumpur Sentral, which driving peth Petaling Jaya Kuala Lumpur.
31

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
Listing MapEval-API System System Prompt
Respond to the human as helpfully and accuzately as possible. You have access to the following tools:
PlaceSearch; Get place ID for given location., args: ("placeNamme' ('descriptien'; "Name of the place', 'title'; Placename', 'type': "string", "placsaddress': d'description': "Address of the place', 'title'; "Flaresddress', 'type (: (strring 1)]
PlaceDetails:
given place ID., args: ("placeId's ('description' 'Place title': 'Placeid', 'type': 'string")]
location',
JearbySearch: Get nearby places eround location., args: ('placeld'; : t'description'; 'Tee id of the place around which to retrievs nearby places.', '=itle'; 'Placeid', 'type': : 'string"), 'type': ('description': "Type of place (e. reataurant, hospital, stc). Restricts the reaults places matching the specified type.', 'title's 'Type', type'; ; 'string" b 'ranbby': ('defaul?'; 'distance', 'description's 'Specifies the order in which places are llated. Possible values are: (1. promineace (default): is specified, the radius paramazer is required. 2. distance: This option sorts places in ascending order by their distance from the specified lecation. When distance is specified, radius disellowed. In case you are not rankby distance.)', 'title': "Kankby', 'type': 'string'), ), 'radius'; ('anyof'; [t'
option sorts results
their importance.
prominence
concerned about the zadius, type'; "intager'i; in l'type': "mull"],, 'éafault'; None, 'dascription'; 'Dafines the distancs (in metars) within which return place results.', 'tiple": (Radiua")]
between two places., args: l'originid': ('description': 'place of Origin', title': : 'originid', "type': "string'), ), 'destinationid'; l'description's 'elace Id of Destination', 'titls'; Destinationid', "type'; rstring' 'szavelMode' ('description's "Mode of transpertation Idriving, walking, bicycling, transit)', 'title': 'Travelmode', 'sype': 'string'
Travellime:
Estinate
Directiona: direetions/zouter between places., arge: l'originId': ("deacription' 'Flace Id of Origin', title': : "Originid', "type': "string'), ), 'destinationId': l'description's 'Place Id of Destination', 'title': Destinationid' "type': "string"), 'ssaveltode'; d'description': 'Mode transportation ldriving, walking, bicycling, trensit)', 'title': 'Travelmode', "sype': "string']] ]]
jaon blob specify taol providing action key (tool name) and action_iapur input key (taol input).
Valid "action" "Final Answer" PlaceSearch, FlaceDetails, NearhySearch, TravelTime, Directions
Provide only ONE per BJS08 as
"action"; STOOT_ NAME, "sction_inputt;: SINPUT
Follow this Format:
Question: input question to answer Thought: consider previous subsequent steps Action:
$JBON_BLOB
Observation: action result
(zepeat Thought/Action
& tines)
Thought: know respond Action:
"sction"; "Pinal Answer",
"action_input"; "Final responae
Begin! Remincer ALMAYS respond with a valid 1son blob of single action. Use tools necessary, Respond directly if appropriste. Fornat is Action: 13 M1S9ON__LOD 'thenn Observation:
Thought:
32

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
Listing 7. Example evaluation of MapEval-API Green
Correct Answer. Red
WWrong Answer.
Oue
Can visit The Great Fyramid of Gisa at 5:00 PM Saturday?
Opticns:
1. Yes
No
Collude-3.5-Somgn
To answer this question accurately, I'11 need to check the opening hours The Creat Pyramid of Giza. Lot me start by getting TD for locatton.
action:
"action"; "FlaceSearch"" "action_inpu:"
"glaceName"; "The Pyramid Glza",
"glacedddress"; "Al laram, Bazles B1 Serman, Al Giza Desect, Giza Governorate, Bgypt"
Observation:
Thought: the ID Pyramid Giza, I'll it to get details about the location, including its opening hours.
lotion:
"sction"; "FlaceUetails",
"action_input"? "placeid"; "Ga.ugmeerddneeeussssie...
Observation:
Governorate 3512201, Egypt 129-9791705, 31.1342046:- open: Monday: 8:00A8-4:30PM, Tresday: 8:00Ax-4:3002 Hednesday: 8:00AM-4430PM, Thuraday: 8:00AM-4:302M, Friday: 8:00AM-4:30PM, Saturday: 8:00AM-4:30PM, Sunday: 8:03AM-4:30PM
Location: Haram, Nazlet E1-Semman,
Racing: 4.6, (27048 ratings).
thought: the information provided the Flazevetails tool, question asks visited
questien.
Pyremid of open Saturdays 5:00 PX Saturday. Since 5:00 PK is ofter the closing time of 4:30 PK, the answer in No.
to
Therefore, the correct option is Option 2: No.
"action"; "Pinal Answer", iaction_input":: "2"
33

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
Listing Example evaluation of MapEval-AP1 Green Correct Answer. Red
Wrong Answer.
Quezy:
visit Senao-11 Temple for hour, Shibuya Crossing for 1.5 hours, Veno Pack Ioc 2 hours, and the Tokyo Skytzee for 1.5 hours. have 3.5 hours places.
currently staying Khaosan Tokyo Origami Asakusa, Tokyo.
availeble. will leave my hostel am. many places can visit? will
Options:
1. Pour
Calude-9.S-Sennett
To answer this questien accurately, 1'11 need to celculate the :zavel times between these locations and the time spent each place. Let's break this down step step.
Pirst,
get the place for all locations mentioned.
Action:
"action": "Placesearch",
"action_inpur""
"placeNlame" "Khacsan Tokyo Origami", "glacekdrress"; "Azakuse, tokyo*
Observation: chL/BBedddoosawzzvvweeeeee
Thought: rhhank you for providing the place Tokyo skytree. New that have place TDS, letfa the travel times between these locations using the walking mode of transportation.
Letra with travel time Khaosan Tokyo Origami to Senso-?1 Temple, Action:
"action"; "TravelTime",
"action_inpu=":
"originte"; *chiJJeenN780000GRyyyy j5 "destinationed:; *ChIJ8T1CC *C1J0011ess00eaa00300000000 "travelKode": "valking"
Response: :[l'dlelance'; (ftexofs km/, "valuef: 267), 'duration's ("Lextr: mins", 'value': 227), "status": (10CAL 11]
Observations Travel Time foos is mins (3.5 km).
Thought: :New that we have all the travel times and the time to spent at each lecation, let's celculate how many places can visited within 3.5 available, starting at
Khansan Tokyo Origami Senso-ji Temple: minutes
2. Time Senao-j1 Temple: 1. hour
3. Total time so fer: hour minutes
At this point, it's 8:04 an, and we've used hour 4 minutes of the available 3.5 hours.
The next destination, shibuya crossing, is 2 hours 52 minutes away foot, which would exceed the remaining time (2 hours m² nutea) ever before spending any time there.
Therefore, the traveler can only visit one place (Senso-ji Temple) within the given time censtraints.
Accion:
"sction"; "Pinal Answer", "action_inpus": "Optionl*
34

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
Listing 9. Example evaluation of MapEval- Visual Visuuual-
Green
Correct Answer.
Red
Wrong Answer.
Aftnn Brookiake Councz ClUb
PILLADE
Turle 2
Cased Courn
Coyulc ange
git Partrog
Coopermen Domasba
Madison
Dearge
Monic
Arter
Srosiaee
Ess Ura SolloSuure
South
Last Ohange KUTDeT
Reservaton
South Orange
Selon
Popests
DAK 31062
maw www.esclooe wepleeeiiiieeooo comu bannetees
LONGE
Chathar
VAlLS
myyming
IVe HILL
The Gravit Voul
Chatiaam.N
Harsshmm 1n DUS..
AID umm
Mazlowood
Toomalip
Milbuum
Fidet
Invintton
UEFTR CLINTOR
XAUEMALE
Houdalls Jublry
Go
Web
Baluuii carcaa cue
PARK
Sprigield
Kemy
@ Ayms
Comt Aun Providence
Google
Union
A
Coptiiy Windsood
Query:
Shich GolE Club is located at the insersection of SpringEield and Boudaille Querey?
Frompt:
Chcose the correct option from the image also show the answer with visible quotation and give proper explanation.
lons:
1.Brooklske Country Club
Baltuarol Golf Club
3.Bast Orange Course
Reeevvs-Beee Asboretum
Gamini-1 S-Pros
The golf club located the insersection Springtield and Houdaille Quarry is "Baltuarol Golf Club", You can clearly see the label for Baltusrol Golf Club positioned right above Springfield Ave, with the golf flag icen situated directly the point where springfield intersects with Houdaille quarsy.
CPT-6o-min1:
The correct optior is "2. Beltusrol Golf Club." Explanation:
Tn the image, the location of Boudai 1le quarry and Springfield can be found near the center at the jurction of major roads. The Beltusrol Golf Club is nazked nearby, iadicating it situated that intersection The other optiens, such Bzocklake Country Club, Bas: Crange Self Course, and Reeves- Reed Arboretun, do appear located the intersection of springfield and Houdnille Quarry.
1ade=33.--Sonett
Based mag image, Explanation:
Club*
located near intersection Springfield avenue Houdaille Quarry. can see "Soltusrol Galf club" marked the just south of where Route 24 meets I-78, which is very close to where Springfield Avenue and Roudaille quarry (both visible on the map) would intersect. Therefore, Baltusrol Solf club 18 the golf club located et the intersection of Springfield Avenue Moudaille Quarry.
map saitusrol
1amma-3v-LAJ) 5-832
The ansker 25 3.East Drange Golf Course. The location of East Orange Golf Course is at the intersection of Springfiald and Houdaill1 Quarry, shown in image. image screenshot of map, and location Dast Orange Golf Course is macked with a zed pin.
35

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
Listing 10. Example evaluation of MapEval-Visuull
Green
Correct Answer.
Red
Wrong Answer.
Context:
Rut Unirondy
Proe
Foase Commune- Free
McToitor Reservol
Grand Sta SCRCE Moan
Emillo Gravestone
+
Rue feel Stepe
Pesrit
Paniggi
Melinyra Mistical
Science Bunding...
Poste da conolens du police
Mo Royal Phrking
unilers Mesicine
Selcen
Caves Chapel
MOL'E nepalPap
Galdheel Bollingrar
Conselate Ger the fluss an Peceration
Cimen Notre Dame des neiges
Google
Quezy:
presently ly visiting
Mount Royal Park
you please about nearby hisstorical
Promp::
Choose the correct opt fon from the image also show the answer with vistble quotation and give proper expl a
Options:
1. pircle Stone
2.secret pool
3.Moison William Coldwell Cottingham
4.Poste de cavalerie di: Service de police de la Vi de Montreal
Gemini-1.5-Pro
The correct answer is "Maison william caldwell cottinghan".
map shows losation "Mount Royal Park". Just to the east that marker "Nai.son Colewel) Cottinghan", The other options located further away from Mount Royal Park.
GPT-4o 19T-4o-miniz mini:
The correct option is:"3, Maison william Coldwell Cottingham
Explanation
Maison William Caldwell Costinghen is historical landmark lozated near Nount Royal Park. It reflects the area, providing insights into past. other options, while interesting do not hold the same historical significaren foaaree as Maison Wi 1 am Ga caldwell cottingham.
architectural heritage history
calude-1.S-Sonme
Raged on the map image provided, tha correct answer ta:"Cirale Stona"
Explanation:
The question asks about nearby hissorical landnark to Mourt Royal Pork. Looking at the map, we can see Mount Reyal clearly labeled. Among the given options, "Cirole Store" is closest historical landnark viaible the map nest Xouat Roys: Pazk, It's marked with pin icon just north of the park acea.
away "secret poo1* is vioibla but seems to be farther to the northezat.
options are either
visible on this parsicular map view:
"Matson llla Caldwell intting visible but located the southeast the park.
"Foste de cavalerie Service police de la vills de Nontresl* visibla but located to the of the park.
Llama- LA1.
The ansker 25 S.nnison William Caldwell Sottingham. Maison Wi am Caldwell Cantinghar la historia house locatad Montr Quebac, Canada. is located
the Mount Royal Park, which is a popular touriat attraction in Montreal, The house was ballt in 1851 and is a fine archisecture.
example Gothic
showcases history
36

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
Listing 11. Example evaluation of MapEval-VVsiae Visual Green
Correct Answer.
Red
Wrong Answer.
Context:
Santagg
VILLA TALLERES
8ivd Estre
SPURR
Ave Santannaagggg Dassio
de deliiioo
Gullermo Tones
TGS
Planta Galván
SALADERO
Unipar Indupa SAIC
CLUB ATLETICO
PUERTO
Recently viewed
ESTACIÓN WHITE
Compania Mega
18 duloo
Ingeniero
Central Piedra Buena SA Una Empresa de.
Puerto Bahla Blanca
White
253
Paseo Portuario
Ingeniero White
Google
Oue
which school is closed to club Atletica Puerto Comercial?
F Prompt:
Chcose the option from the image also shew the with visible quotation and proper explanation.
Options:
Cnanswerable
Secondacy school No. 25
PROVINCIAL SCHOOL No. TOMAS ESPORA Province of Buenes
SCHOOL 507
scHoor, No.5 Province of Ruenos rea
Gemini 1...i-11.... 1-50-po:
The answer is *0] Unanswerable".
map provided does not show any schools. Therefore, imponsible to deternine
cloaed
Atlezico Puerto comercial based solely the provided image.
GPT to mini:
To determing which school is closest to Glub Acletico Fuerto Comercial, we need to analyze the locations of the club's position
llated
in
map.
Anslysis:
club Atletico Pserto
located near center
map-
The other schools must found or deduced based on their general proximity to this loeation
From the visible landnarks and overall arrangement on the map, PRCVINCIAL SCHOOL No. TOMAS ESFORA appears to be the closest to club Atletico Puerto Conercial.
Colude-3.cc 3.0--3n----nnn
Rased information provided in image, Explanation:
Unanswerable
map
any schools educational institutions neer
Puesto. Shile
landmarks and businesses, including "Club Atletico Puerto", there is no Information about any schoola visible this map. The options provided mention spesific schools, but none of these are shown er lsbeled on the map image. school locations, the question cannot be answered
Given the limited information the image and the absence based solely on what's a shown hare, making "fnanswerabll* the most appropriate choice.
Llama-3-v 1-1--1111111111
Secondary school 25.
The image showa a map of Puerto Comerctal, Argentina, with varioua locations marked, One of the locations marked is comercial, answer in 1) secondery school
secondary schoal, is No. 25.
school to club
37

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
Listing 12. Example evaluation of MapEval- Visual
Green
Correct Answer.
Red
Wrong Answer.
Narayana Hospital Chunavati
P
A
Horghy
METIABRUZ oft
SANKRAIL STRe
Ward
Number
R
Kalighat Man
Sonartari Events
Maheshtala मटशग
hr 8 min
24.1 km
23 km
JADANPUR
Tarc
Kapi
Reerless H Hespital
SARSUNA
BANGDRONI
PANCHPOTA
THAKURPUKUR
KHURIGACHI
ulta
Indian Institute of Management Calcutta ar Bntieefbe
Rajpur Sonarpur
Quezy:
How far 1a Radpur Sonarpur in West Rengal, Tndia, from Narayana Fospital, Chunavati, Andul Rd, Howrsh, West Rangal, ir drive?
Frompt:
Chcose the correct option from the image also show the enswer with visible quotation and give proper explanation.
options:
23 km
2. km
3. 24, km
4. 6km
Comini-1.5-Pro
The image clearly shows the total diatance as "23 km". Therefore, correct ansver *1) km".
GPT-4o-minis
Based on the provided information. and the inage, the distance Trom. Narayana Hoapital, Chinavatl, Bowrah, to Ra Rajpur Sonargur is approximately 23 kn.
The answer option 1) *23 km"
Explanation:
parta, the distance calculated displayed in mapt The total time shown on the map indicates travel duration of around hour and minutes for the distance of 23
The consista
Colude-3. 1atie-3.5-00an 5-Soonect
Rased on the Information provided in the image, the correct answer 1a:*23 km
Explanation:
frem Narayana Hospissl, Chunavati (which is the Hooghly River in the northwest of the map ) to Sajpuz šonarpur (in the southeast the map). The route is highlighred in blue. in the bottom left cornar of time
The map shows
this blue zoute, theze's
clearly visible:") min
blama V-LA1, 83:
The answer la 27.6km.
Narayana kospital, Chunavati, The rouse you described irvolves travel ing along Andvl Rd, NH 12, xlipare Rd, caris Main Rd/Netaji. Subhash phendra Bose Re, and passing by Jharna Ghee And Honey on the right. The total diatance covered is 27.6 killometea.
distance
Howrsh, West Bengal, Bajpur Scnarpur kilometers
38

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
Listing 13. Example evaluation of MAPEVAL-VISUAL Green
Correct Answer.
Red
Wrong Answer.
Context:
Bengal Bo Dhanmondi
Road
Hotel Nandini catrbi Top rated
No
Square Hospital
apars aureermr
Road No
Road Road No-15A
Ginl School Rd
Lake Giicus Rd
Sina
Panthapath
Specialized Hospità
Bangl bangladesh.oo Medical
College Hospital
Rd 12/4
Satmasii Road
Abahani Club Field
Rd No 94
Dhanmond
Women's Complex
sraninil witerr
Pizzaburg Dhanmondi Aresrnd
Road No.
Kalabegan 1st
Rd 121
Kacchi Bhai Dhanmondi wit 4mme
Green Coll
Villa
Query:
I'm staying at Hotel Nandini and heading to Dhanmondi Abshani Field to meet friend for gane. I plan to check blood pressure playing. you suggest
hospital this?
P PrompL:
Choose the correct option from the image also shew the answer with visible quatation and give proper explanation.
ponions:
1.square Hospits'
2. 2.Racchi Bhai Dhanmondi
Hangladesh Medical college hospital
4. IBN Sina specialized Rospital
Gemini 1.5 Pro:
The best option is "1. Square Hospital", Here's why: Familiarity: staying
healthcare provider you're re familiar generally a good idea, especislly somechiigg ng 1' ke checking blood pressure.
Location: While not the absolute clasest, Square kospital is still relatively convenient to both Hotel Nandini and Abahant Field. d.
trip away both locations.
GPT-60 GPT-6o-ming
The most suitable option for checking your blood pressure after playing at Dhanmendi Abshani Field las "IBN Sins Specislized Hospital". Explenation:
1. Proximity: IBN Sina spacialized Hospital is located clase to Abahani club Field, making it convenient to access wight after your game.
2. Alternatives: The other options may Tarther away may not be specialized for such trealments.
calude-3.5 S-5n-nnttt
The most suitable option for checking your blood pressure after playing az Abaheni club sield would be: "square Hospital*. ixplanatioo:
1. You mentioned that you usually ge: your treatment at 8quare Rospiial, which is visible on the map in the upper
sight corner.
2. Square Hospital is relatively clase both Hotel kandini (where you're re staying) and Abahani club rield (there
you' 11 be playing).
3. while dangladesh Medical college Hospitsl and 18N sina specialized Hospital also visible the map and volid redical facilities, Square #cspital in your usual treatment locstion, making it the most auitable choica for continuity
1lama-3-9-1A1.5--
The anaker 3 4. TRN Sina Special Hospital. Explanstion: The image shows map the area around kotal Nandini phanmondi Abshani Field. Special 1sed Roap' tal is lonated near Hotel Nandini and is aui tabla optioon for chacking blond pressure. Tt. is also close to Dhanmondi Aoahani Field.
39

---

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
Listing 14. Example evaluation MAPEVAL- VISUAL
Green
Correct Answer.
Red
Wrong Answer.
Context:
miral ss
All cha
Hampton Hilton Doha Town
Cator Plagrounde
4 (511)
Nakilgt
Playground
Jean Miche lothonel
Fourtains
Draw ineo ConsuR
Cor Kussemm
Oatar Floopole
Sateraa
AY Sattina
National Museum Ratar
Sheekh Abdulla bin Daa
Google
Qpery:
What the neazest any Museum from Qatar Plagpole
Frompt:
Chcose the correct option from the image also show the answer with visible quotation and give proper explanation.
1. Car Museum
2.Sheikh Abdulls bin Jassim Al-Thani Palace 9.National Museum of Gatar Playgrounds NaNtional Museum of Qatar
Gemini-1.cm
The correct option "1) Caz Museun", Frem the image, you zan see the "Cataz Flagpole" location merked. The "a* Museum"x is located just above the Qotar Flagpale, naking it the nearess
GPT-co-mints
The neareat museum from the Cataz Flagpole is "National Muaeum of Qatar". Explanation:
when examining the the National Museum of Qatar is directly adjacent to the Qater Flagpole, while the other options are farther away, The Caz Museum and Sheikh Abdulla bin Jassim Al Thani Palace are located nearby, but the Natiooal Museum
is the closest, indicated by preximity the map.
Calude 3.5 Sonnet:
Based on the map image provided, the correst answer is: "Car Museum" Explonation:
The Datar F' l'agpo' e is vistble on the map, and amorg the museum opt l1sted the Car Miseum appears to be the Jatar elagpole marker map. The Nationa". Museum of gatar Playgrounds and the Natioral Museum of Ostar are both further away to the north. Shefkh Abdulla bin Jassim Al-Thani Palace located further south of the flagpole.
elcsest it.
Kuseum
located just slightly northwest
-lama-3---1A1 5-88:
The nearest museum from Qatar Flagpole is the National Museum of Catar, The anawer is opt 4. The National Museum of Qatar is located on the Qatar Flagpole, as indicated on the map. The National Museum of Qatar is museun. in Doha country
history culture of largest in the Middle East. The museum is located on the Cozniche, waterfron: pcomenade in Doha, and is easily accessible by public treasportazion. historical documents, and contemporary art. The museum alse offers aducational. programs and workshops for visitors ages.
Dacar, dedicated
the
museum features variety exhibits, including archseological artifacts,
40


Reviews:
[{"review_id": "yNMMTFKNkV", "reviewer": "Reviewer_syVz", "summary": "The authors introduce a benchmark designed to assess the map-based reasoning capabilities of foundation models. This benchmark consists of 700 multiple-choice questions covering locations, including 180 cities and 54 countries across tasks such as processing spatial relationships, navigation, travel planning, etc. The study evaluates 28 foundation models and finds that there still exists a significant performance gap compared to human capabilities, especially in complex map-based reasoning tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of the applicability, effectiveness of our proposed benchmarking task, and comprehensive experiments in our work!\n> W1: One potential limitation is that the benchmark focuses on multiple-choice questions, which may not fully capture the open-ended nature of real-world spatial reasoning tasks.\n\nIn Section 3.1 (line 145 onwards) we discuss the motivations for MCQ based evaluation choice over open-ended ones.\n \nBesides, our dataset is designed to be flexible: the MCQ format was chosen for evaluation purposes, but the questions themselves are structured so that open-ended evaluation can be done simply by removing the answer choices. To further support this, we conducted an experiment where we removed answer choices from MapEval-Textual and MapEval-Visual questions and evaluated open-ended responses from **Claude-3.5-Sonnet** against ground truth using **O3-mini**. The results in Table 1 and 2 show Claude-3.5-Sonnet’s performance in the open-ended setting, demonstrating that our queries remain valid for such evaluations.\n\n|Evaluation|Overall|POI|Nearby|Routing|Trip|Unanswerable|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|\n|Open-ended|55.33|43.75|43.37|69.70|67.16|55.00|\n|MCQ|66.33|73.44|73.49|75.76|49.25|40.00|\n\n*Table 1: Performance of Claude-3.5-Sonnet in MapEval-Textual*\n\n|Evaluation|Overall|PlaceInfo|Nearby|Routing|Counting|Unanswerable|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|\n|Open-ended|51.88|67.7|51.11|35.00|43.18|65.00|\n|MCQ|61.65|82.64|55.56|45.00|47.73|90.00|\n\n*Table 2: Performance of Claude-3.5-Sonnet in MapEval-Visual*\n\nAs discussed in Section 3.1 that while open-ended assessments are more natural they introduces additional challenges, particularly in automated grading. For example, O3-mini initially assessed Claude-3.5-Sonnet’s accuracy in the \"Unanswerable\" category as **55%** (Table 1), but manual inspection revealed an actual accuracy of **80%**--showing the limitations in an automated evaluation for open-ended setting.\n\nAdditionally, open-ended evaluation doubles API costs since it requires two calls per query—one for generating responses and another for evaluation—unlike MCQ, which allows direct comparison. This makes large-scale assessments far more expensive. Moreover, as LLM API costs scale with token usage, long-form responses further amplify expenses.\n\nThus, while open-ended evaluation is possible with MapEval if intended, our MCQ-based approach remains the more cost-effective and reliable method for benchmarking.\n\n> Q1: Can authors provide a more detailed discussion/insight on why different models perform differently on the datasets?\n\nIn this work, we provide a detailed evaluation of various models on our datasets, highlighting their strengths and weaknesses in different categories. Our results show clear differences in model performance across fine-grained categories, with some models excelling in certain tasks and others facing challenges in specific areas. These insights, particularly discussed in Section 4.3 and Section 5, suggest that models may be better trained for certain reasoning aspects while struggling with others, which could be due to factors such as the nature of their training data or the types of tasks they were explicitly trained to handle.\n\nHowever, a more thorough causal analysis is challenging, as the full training procedures and datasets for these open-source foundation models are rarely disclosed. This lack of transparency limits our ability to directly analyze the root causes of the performance differences we observe. While we acknowledge this limitation, we believe that conducting such causal analysis falls outside the scope of this paper and will be explored in future work.\n\n> Q2: Is there any geo-specialized foundation models that the authors consider evaluating and would potentially outperform the current ones?\n\nAt the time of our evaluation, we did not find any geo-specialized foundation models that met the specific requirements of our task. Most existing models in this domain are Vision-Language Foundation Models designed primarily for remote sensing images, which are not directly applicable to our evaluation. We have also addressed this in Appendix A.3.\n\nHowever, we evaluated **K2** [1], a model specifically designed for geoscience-related tasks and built on LLaMA-7B. However, as shown in Table 3, its performance on our benchmark was extremely poor, achieving an overall accuracy of only 20.33%, which is close to random guessing. Given its limited effectiveness across all categories, we decided not to include it in our main evaluations.\n\n|Overall|Place Info|Nearby|Routing|Trip|Unanswerable|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|20.33|25.00|20.48|15.15|20.90|20.00|\n\n*Table 3: Performance of K2 in MapEval-Textual*\n\nNonetheless, if you refer us any specific model, we will report that.\n\n**References:**\n\n[1] Deng, C., et al. \"K2: A foundation language model for geoscience.\" WSDM (2024).", "claims_and_evidence": "The major claims are supported by their evaluation using different foundation models and their benchmark.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria generally make sense for assessing geospatial reasoning in foundation models. The authors make the benchmark relevant to real-world applications (e.g., navigation) and compare across different models, locations and tasks. The authors provided an error analysis, which offers insight into failure modes.", "theoretical_claims": "This is an application and benchmark work.", "experimental_designs_or_analyses": "The focus of this work is the proposal of a new benchmark to evaluate the foundational model's geo-spatial reasoning capacities.", "relation_to_broader_scientific_literature": "This work complements existing benchmarks in natural language processing by focusing on this specific domain (geo-spatial reasoning).", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "The authors introduced a novel and well-structured benchmark for an interesting problem. The authors evaluate a wide range of models, providing insights into current limitations, and also discuss how to address failures and improve the reasoning ability in the future. One potential limitation is that the benchmark focuses on multiple-choice questions, which may not fully capture the open-ended nature of real-world spatial reasoning tasks.", "other_comments_or_suggestions": "Se questions.", "questions_for_authors": "Can authors provide a more detailed discussion/insight on why different models perform differently on the datasets?\nIs there any geo-specialized foundation models that the authors consider evaluating and would potentially outperform the current ones?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "veNtAfzedX", "reviewer": "Reviewer_Z5ay", "summary": "The paper introduces MapEval, a benchmark designed to evaluate the geospatial reasoning capabilities of foundation models across textual, API-based, and visual tasks. It comprises 700 multiple-choice questions covering spatial relationships, navigation, travel planning, and map interactions across 180 cities and 54 countries. This paper evaluates various foundation models, including both closed models, including GPT-4o, Claude-3.5-Sonnet, and Gemini-1.5-Pro, and open-sourced models, revealing significant performance gaps. The results demonstrate critical weaknesses for existing LLMs in spatial inference, including difficulty in handling distances, directions, route planning, and location-specific reasoning. The paper highlights the need for better geospatial AI models that integrate improved reasoning and API interactions to bridge the gap between foundation models and real-world navigation applications.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of the applicability, effectiveness of our proposed benchmarking task, and comprehensive experiments in our work!\n\n> W1: Despite the proposed benchmark evaluation dataset, its size (700 questions) remains relatively small compared to other LLM benchmark evaluation dataset.\n\nDue to the high cost of both foundation models as well as tools/APIs, recent language models often tend to evaluate on a small number of sub-sampled datasets. For example ReACT [1] uses only 500 random samples from AlfWorld dataset, similarly Reflexion [2] uses only 100 examples from HotpotQA. Therefore, many recently proposed tool oriented or intense reasoning benchmark datasets are found to be reasonable in size in order to be cost-effective: API-Bank: [3] (400 instances), Logical-reasoning benchmark LogiQA: [4] (641 examples), the most popular problem solving (code generation benchmarks) HumanEval: [5] (164 instances only), CodeContests [6] (156 problems), Tau-bench [7] (165 problems), OS World [8] (369 problems), App world [9] (750 problems), TravelPlanner [10] (1.2K problems). Consequently, we carefully construct our problem instances balanced in size and covering different challenges.\n\n> W2: The paper does not explore how fine-tuning models specifically on MapEval would impact performance. Simple fine-tuning on part of the instances may greatly enhance the model performance. If that’s the case, the capability of answering these questions wouldn’t be an issue for LLM.\n\nWe conducted additional experiments to assess the impact of fine-tuning on the performance of smaller models using the MapEval-Textual dataset. Specifically, we have split the dataset of 300 MCQs into train (97 questions) and test (203 questions) set and fine-tuned a selection of models on the train set and evaluated their performance on the test set. However, the results, as shown in Table 1, reveal that fine-tuning on MapEval does not lead to significant performance improvements (<5%) and remains remarkably lower than large capable models such as GPT-4o, Claude-3.5-Sonnet or Gemini 1.5 Pro which we already included in the paper. Rather, we believe our evaluation benchmark MapEval will promote future developments of new geo-spatial model with sophisticated fine-tuning and other learning methods. \n\nThese findings suggest that the challenges in MapEval are not merely due to a lack of training exposure but reflect deeper limitations in LLMs' geospatial reasoning.\n\n\n| Model\t\t\t| Pretrained | Finetuned |\n|---------------|:-------------:|:--------------:|\n| Phi-3.5-mini \t\t|   39.90      |    34.48      |\n| Llama-3.2-3B \t|   34.98      |    35.96      |\n| Qwen-2.5-7B  \t|   41.87      |    43.35      |\n| Llama-3.1-8B \t|   46.31      |    44.33      |\n| Gemma-2.0-9B \t|   46.80      |    51.23      |\n\n\n*Table 1: Performance Comparison of Open-Source Models on MapEval-Textual (Test set), before and after fine-tuning.*\n\n> W3: Some questions, as observed in the appendix, appear to be relatively simple, such as estimating travel time, which may not fully test deep geospatial reasoning.\n\nWhile some questions (e.g., travel time estimation) appear simple, our dataset prioritizes real-world map usage scenarios over exclusively testing deep geospatial reasoning. It includes practical queries (multi-stop routing, proximity-based decisions) and complex tasks (route optimization, accessibility constraints) requiring spatio-temporal reasoning. Simple examples in the appendix aim to clarify concepts, but the full dataset contains advanced reasoning challenges as well. As we discussed, even for the humanly simple cases, the advanced foundation models capable of doing more complex reasoning in other tasks fall significantly behind in MapEval. However, we will expand on these in the final version of the paper with more illustrated examples. \n\n**References:**\n\n[1] Yao, S., et al. \"ReAct: Synergizing Reasoning and Acting in Language Models.\" ICLR 2023.\\\n[2] Shinn, N., et al. \"Reflexion: Language agents with verbal reinforcement learning.\" NeurIPS 2024.\\\n[3] Li, M., et al. \"API-Bank: A Benchmark for Tool-Augmented LLMs.\" EMNLP 2023.\\\n[4] Liu, J., et al. \"LogiQA: A challenge dataset for machine reading comprehension with logical reasoning.\" IJCAI 2021.\\\n[5]Chen, M., et al. \"Evaluating large language models trained on code.\" arXiv:2107.03374, 2021.\\\n[6] Li, Y., et al. \"Competition-level code generation with alphacode.\" Science 378.6624 (2022).\\\n[7] Yao, S., et al. \"tau-bench: A Benchmark for Tool-Agent-User Interaction.\" arXiv:2406.12045, 2024.\\\n[8] Xie, T., et al. \"Osworld: Benchmarking multimodal agents for open-ended tasks.\" arXiv:2404.07972, 2024.\\\n[9] Trivedi, H., et al. \"AppWorld: A Controllable World of Apps and People.\" ACL 2024.\\\n[10] Xie, J., et al. \"TravelPlanner: A Benchmark for Real-World Planning.\" ICML 2024.", "claims_and_evidence": "The evaluation is comprehensive to test the effectiveness of LLMs.", "methods_and_evaluation_criteria": "There are several concerns regarding this evaluation benchmark. First, while the benchmark introduces a novel evaluation approach, the number of questions remains relatively small in scale. Second, the three task types could see significant performance improvements when integrated with appropriate tools. The difficulty of these questions seem to be limited. Third, some questions, as illustrated in the appendix, appear to be relatively superficial—for example, identifying travel time for a given route. Fourth, while certain LLMs do not perform well on this benchmark, it remains unclear how fine-tuning on such a small dataset would impact their performance on these tasks.", "theoretical_claims": "No mathematical proofs are provided in the paper.", "experimental_designs_or_analyses": "The experiments to judge the accuracy of the answer look reasonable to me.", "relation_to_broader_scientific_literature": "The paper propose to establish a benchmark evaluation dataset for assessing the performance of geospatial reasoning capabilities for LLMs. Such a benchmark dataset is not available in previous studies.", "essential_references_not_discussed": "The references generally contain the representative studies in this field.", "other_strengths_and_weaknesses": "Strengths:\n\nS1. The paper introduces MapEval, a well-structured benchmark that evaluates geospatial reasoning across textual, API-based, and visual tasks, covering 180 cities and 54 countries. The benchmark includes 700 multiple-choice questions that span a range of real-world map interactions, such as navigation, travel planning, and spatial relationships.\n\nS2. The study highlights significant gaps in the geospatial reasoning abilities of both closed and open-source models, showing their struggles with certain types of tasks.\n\nWeaknesses:\n\nW1. Despite the proposed benchmark evaluation dataset, its size (700 questions) remains relatively small compared to other LLM benchmark evaluation dataset.\n\nW2. The paper does not explore how fine-tuning models specifically on MapEval would impact performance. Simple fine-tuning on part of the instances may greatly enhance the model performance. If that’s the case, the capability of answering these questions wouldn’t be an issue for LLM.\n\nW3. Some questions, as observed in the appendix, appear to be relatively simple, such as estimating travel time, which may not fully test deep geospatial reasoning.", "other_comments_or_suggestions": "None", "questions_for_authors": "Please response to the comments in W1-W3.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QF7LCNI4HQ", "reviewer": "Reviewer_irQQ", "summary": "This paper introduces a geospatial benchmark called MapEval. It covers textual, visual and API-related tasks, and evaluates a set of close-source and open-source LLMs and VLMs. The results highlight the gap between close-source and open-source models and between current foundation models and humans, suggesting potential improvement for map-related abilities in foundation models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: The benchmark uses Google Map as the source. Can it be extended to other map services, or other modalities (e.g. Satellite images in Google Map)?\n\nYes, the benchmark can be extended to other map services. The latest MapQaTor update integrates OpenStreetMap, Mapbox, and TomTom, broadening applicability. While satellite images can be used, we focus on digital map snapshots as they better reflect everyday interactions like navigation and place searches.\n\n> Q2: How can we prevent LLMs/VLMs from using pretrained knowledge to answer queries? A baseline with no textual/API/visual context could demonstrate that external context is necessary.\n\nWe evaluated the top-performing model Claude-3.5-Sonnet on 300 MCQs used in MapEval-Textual and MapEval-API without textual/API context. The overall accuracy is 6.67%, demonstrating the necessity of external context.\n\n> W1: The evaluation includes large LLMs, but only smaller VLMs (<10B parameters) are considered. Larger models (e.g. Qwen2.5-VL 72B, InternVL2.5-78B) might narrow the gap with closed-source models.\n\nMapEval is agnostic to the model families or sizes and any foundation models of corresponding modalities can be studied. Based on your suggestion, we performed additional experiments with larger Qwen2.5-VL-72B, and Llama3.2-90B-Vision, and the results (Table 1) confirm a reduced gap between closed (61.65%) and open-source (60.35%) models.\n\n|Model|Overall|POI|Nearby|Routing|Counting|Unanswerable|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|\n|Qwen2.5-VL-72B|60.35|76.86|54.44|43.04|52.33|90.00|\n|Llama3.2-90B-Vision|50.38|73.55|46.67|41.25|36.36|25.00|\n\n*Table 1: Performance of larger VLMs in MapEval-Visual.*\n> W2: For Listing 1, models fail to correctly convert geospatial coordinates to distances and require math capabilities, as shown in the calculator experiments. These abilities slightly deviate from the \"map-based reasoning\" focus of the proposed benchmark.\n\nNumerical and mathematical capabilities are integral to map-based reasoning, as geospatial tasks inherently involve calculations such as distance estimation, travel time computation, and spatial relationships. Effective map-based reasoning requires models to first identify and understand different place types, recognize origin and destination locations, and interpret spatial context. Only then can they apply mathematical reasoning to solve tasks like route optimization, distance-based decisions, and travel cost estimations. Thus, rather than being separate from map-based reasoning, these numerical skills are essential for accurately processing and answering real-world geospatial queries.\n> W3: For Listing 12, it seems that the intermediate points in the query do not appear on the map.\n\nWe have ensured that each question in our dataset can be answered using the given question text and the accompanying map snapshot. However, not all questions necessarily require the map snapshot for a correct response—some can be answered based on the textual information alone. Importantly, these types of queries constitute a small fraction of the dataset. These cases still align with real-world map usage scenarios where users may ask about locations or routes that are not fully displayed but can be inferred through reasoning.\n> W4: For Listing 13, the answer should be 19.4/5.4=3.59 hours, while the options are all in minutes. Besides, the visual context seems to be unnecessary to answer this question.\n\nWe acknowledge the issue (options should be in hours, not minutes) and will correct this example in the final paper.\n> W5: The benchmark's sample size is limited, especially for tasks like \"place info,\" where some models in Table 3 show only a 1-2% performance difference. With just 64 samples in the textual place info task, the actual difference is only one or two samples, making it insufficient for statistically significant model comparisons.\n\nIt is true that some models in Table 3 show only a 1-2% difference in performance for the Place Info category. However, this is expected when benchmarking 19 LLMs, as minor variations naturally occur between models. That said, we carefully designed the benchmark to cover all practical variations within the 64 questions in this category. The Place Info task primarily focuses on factual attributes and spatial relationships, such as cardinal directions or straight-line distances, which inherently limit the number of distinct question types that can be meaningfully introduced. Simply increasing the number of questions would likely lead to redundancy rather than new insights into model performance. However, if the reviewers can suggest additional meaningful variations that we may have overlooked, we are open to expanding this category in the final version of the dataset.\n> W6: The questions are simple MCQs, which are effective for evaluation, but open-ended questions or more complex responses (e.g., TravelPlanner) may be better suited for tasks like trip planning.\n\nSee W1 in Reviewer syVz’s rebuttal.", "claims_and_evidence": "The proposed benchmark covers textual, API and visual tasks and is close to real-world scenarios, and the gap for improvement is validated through extensive experimental results. The appendix provides detailed comparison regarding different models from many perspectives.", "methods_and_evaluation_criteria": "1. The number of samples in the benchmark is somewhat insufficient, especially considering each single task. For instance, some models in Table 3 exhibit closer performance in \"place info\" with only 1-2% difference. Considering there are only 64 samples in the textual place info task, the actual difference is only one or two samples. This data size is not sufficient to compare different models with statistical significance.\n2. How to prevent LLMs/VLMs from using their pretrained knowledge (e.g. Knowledge of POIs in some cities) to answer the queries? The sentence \"without using any external knowledge or assumptions\" in the prompt may not necessarily be effective. Although the authors use LLMs to filter out questions that can be answered easily in Appendix B.2, this seems to manual process. I suggest providing a baseline with no textual/API/visual context to demonstrate that external context is necessary for answering these questions.\n3. The questions are all designed as simple MCQs. While this is supposed to be an effective evaluation approach, open-ended questions or more complex responses (e.g. TravelPlanner) may be more suitable for certain tasks like trip planning.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "1. It is good to see large LLMs with 70 and 90B parameters are included in the experiments. However, for VLMs, current evaluation only considers models with less than 10B parameters. Larger VLMs (e.g. Qwen2.5-VL 72B, InternVL2.5-78B), since they have stronger OCR and reasoning abilities and are supposed to narrow the gap between close-source models.\n2. I check the examples in the appendix and find some questions:\n\n  - For Listing 1, models fail to answer correctly due to inability to convert geospatial coordinates to actual distances. It also requires math capabilities, as validated by the experiments with calculators. These abilities somewhat deviate from \"map-based reasoning\", which is the focus of the proposed benchmark.\n\n  - For Listing 12, it seems that the intermediate points in the query do not appear on the map.\n\n  - For Listing 13, if I understand correctly, the answer should be 19.4/5.4=3.59 hours, while the options are all in minutes. Besides, the visual context seems to be unnecessary to answer this question.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "I suggest including more discussion on benchmarking and improving general 2D spatial reasoning (i.e. not specially for maps or geospatial tasks) in LLMs and VLMs, such as [1-4]. I wonder whether they have similar conclusions or can be helpful in the proposed geospatial benchmark.\n\n[1] Yang, Jianwei, et al. \"Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v.\"\n\n[2] Ramakrishnan, Santhosh Kumar, et al. \"Does Spatial Cognition Emerge in Frontier Models?.\"\n\n[3] Tang, Yihong, et al. \"Sparkle: Mastering basic spatial capabilities in vision language models elicits generalization to composite spatial reasoning.\" \n\n[4] Li, Chengzu, et al. \"Topviewrs: Vision-language models as top-view spatial reasoners.\"", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "Many figures and examples are provided in the appendix but frequently referred in the main paper, affecting the coherence and integrity of the paper. It is recommended that the discussion text related to the figures in the appendix should also be placed in the appendix, with only a brief mention in the main text.", "questions_for_authors": "1. The benchmark uses Google Map as the source. Can it be extended to other map services, or other modalities (e.g. Satellite images in Google Map)? Discussion is encouraged.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eflVvR5H0X", "reviewer": "Reviewer_V6Fj", "summary": "This paper presents a novel benchmark dataset called MapEval constructed based on Google Maps for various map-based geospatial reasoning question answering. The dataset consists of three components: MapEval-Textual, MapEval-Visual, and MapEval-API which correspond to different types of geospatial questions that involves text, map images, and API calls.\n\nStrength:\n1. The MapEval presents a very unique and important dataset for the whole AI community, especially for the geospatial artificial intelligence community in general. The dataset is very useful to evaluate LLMs' performance on different geospatial reasoning capabilities.\n2. The paper conducts a systematic evaluation across 28 different LLMs and provides a very comprehensive view of the presented GeoQA challenge.\n3. The limitations of the current LLMs are highlighted and explored in great details which facilitates the future LLM and Geospatial LLM development.\n\nSuggestion:\n1. Can you describe the way how you select the geographic questions? Many QA benchmark works start at collecting important question sets such as HotpotQA and Web Questions. Where do you collect these 700 questions? How do you define the overall question types?\n2. Any reason why the performance of Claude-3-5-Sonnet (90%) can outperform human (65%) in such a large margin on the Unanswerable type in Table 5? The same can be seen in Table 3. \n3. If I understand correctly, the authors use MapQaTor to cache all API call responses and create a static database for model evaluation. If the generated structured API calls are a little bit different from the golden API calls, then they will get no results from the cached database but they might be able to get the correct answers if using the real Google APIs. Will this systematically penalize the model performance?", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of the applicability, effectiveness of our proposed benchmarking task, and comprehensive experiments in our work!\n> Q1: Can you describe the way how you select the geographic questions? Many QA benchmark works start at collecting important question sets such as HotpotQA and Web Questions. Where do you collect these 700 questions? How do you define the overall question types?\n\nFor defining question types, we conducted an extensive literature review on geospatial and geographic question answering [1-6]. However, since our benchmark differs from prior works by focusing specifically on map-based user queries, we established our own question types based on the kinds of questions users typically ask on map services. To ensure coverage and relevance, we manually annotated 700 questions, drawing from both the literature review and our own experiences with daily map usage.\n> Q2: Any reason why the performance of Claude-3-5-Sonnet (90%) can outperform human (65%) in such a large margin on the Unanswerable type in Table 5? The same can be seen in Table 3.\n\nClaude-3.5-Sonnet systematically marks a question as Unanswerable when the necessary information is not available in the provided context. In contrast, human participants tend to rely on intuition or external knowledge to select the most plausible option from the given choices, even when the correct answer is not explicitly present. This difference in approach leads to a significant performance gap, as the model strictly adheres to the available context while humans may introduce subjective reasoning.\n> Q3: If I understand correctly, the authors use MapQaTor to cache all API call responses and create a static database for model evaluation. If the generated structured API calls are a little bit different from the golden API calls, then they will get no results from the cached database but they might be able to get the correct answers if using the real Google APIs. Will this systematically penalize the model performance?\n\nIn MapEval-API, agents can query details of specific places, list of nearby places around a location, route between places, travel duration and distance between places. Here you have a misunderstanding that agents generate structured API calls. Rather agents have access to simplified functions. For example, to get travel duration between places agents need to call TravelTimeTool(origin, destination, travelMode). This function then generates the actual structured API calls, which mitigates small variations in API calls. So, if an agent needs the driving duration from place A to B, it needs to call TravelTimeTool(placeId_A, placeid_B, ‘drive’). No other function call is valid in this scenario. In table 8 (in the main paper), you can see we have cached enough API calls in our database. So, the model performance is not penalized due to absence of cached data. This claim can be justified by comparing Claude-3.5-Sonnet’s performance in MapEval-Textual (66.33%) and MapEval-API (64%). If the model were penalized, the performance of MapEval-API would have been much lower.\n\n**References:**\n\n[1] Kefalidis, Sergios-Anestis, et al. \"Benchmarking geospatial question answering engines using the dataset GeoQuestions1089.\" International semantic web conference. Cham: Springer Nature Switzerland, 2023.\\\n[2] Hamzei, Ehsan, et al. \"Place questions and human-generated answers: A data analysis approach.\" Geospatial Technologies for Local and Regional Development: Proceedings of the 22nd AGILE Conference on Geographic Information Science 22. Springer International Publishing, 2020.\\\n[3] Mai, Gengchen, et al. \"Geographic question answering: challenges, uniqueness, classification, and future directions.\" AGILE: GIScience series 2 (2021): 8.\\\n[4] Punjani, Dharmen, et al. \"Template-based question answering over linked geospatial data.\" Proceedings of the 12th workshop on geographic information retrieval. 2018.\\\n[5] Chen, Wei, et al. \"A synergistic framework for geographic question answering.\" 2013 IEEE seventh international conference on semantic computing. IEEE, 2013.\\\n[6] Mai, Gengchen, et al. \"On the opportunities and challenges of foundation models for geospatial artificial intelligence.\" arXiv preprint arXiv:2304.06798 (2023).", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No theoretical claim.", "experimental_designs_or_analyses": "See above", "relation_to_broader_scientific_literature": "This paper has a significant contribution to the GeoAI community as well as the LLM research. It will also benefit the general public since this dataset can be used for evaluating LLMs for various geospatial tasks that are conducted daily by the general public.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "See above.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents MapEval, a novel and comprehensive benchmark for evaluating geospatial reasoning in foundation models across textual, visual, and API-based tasks. The benchmark is well-structured, covers a diverse set of cities and countries, and reveals critical limitations of current models, making a strong contribution to both the LLM and geospatial AI communities. While some reviewers noted limitations in benchmark scale and question complexity, the overall consensus highlights the benchmark's originality, relevance, and potential for future impact. I believe it is a clear accept.
Decision: Accept (spotlight poster)