FRIEDA: Benchmarking Multi-Step Cartographic Reasoning in Vision-Language Models
Authors: Jiyoon Pyo
Yuankun Jiao
Dongwon Jung
Zekun Li
Leeje Jang
Sofia Kirsanova
Jina Kim
Yijun Lin
Qin Liu
Junyi Xie
Hadi Askari
Nan Xu
Muhao Chen
Yao-Yi Chiang
Venue: ICLR 2026 Poster (2026)
Keywords: large vision-language model
multi-step reasoning
multi-image reasoning
mapvqa
Primary area: datasets and benchmarks
TL;DR: We introduce FRIEDA, a benchmark that tests VLMs' multi-step cartographic reasoning and cross-image inference, highlighting critical gaps in current performance.

Abstract: Cartographic reasoning is the skill of interpreting geographic relationships by aligning legends, map scales, compass directions, map texts, and geometries across one or more map images. Although essential as a concrete cognitive capability and for critical tasks such as disaster response and urban planning, it remains largely unevaluated. Building on progress in chart and infographic understanding, recent large vision language model (LVLM) works on map visual question-answering (VQA) often simplify maps as a special case of charts. In contrast, map VQA demands comprehension of layered symbology (e.g., symbols, geometries, and text labels) as well as spatial relations tied to orientation and distance that often span multiple maps and are not captured by chart-style evaluations. To address this gap, we introduce FRIEDA, a benchmark for testing complex open-ended cartographic reasoning in LVLMs. FRIEDA sources real map images from documents and reports in various domains (e.g., geology, urban planning, and environmental assessment) and geographical areas. Following classifications in Geographic Information System (GIS) literature, FRIEDA targets all three categories of spatial relations: topological (border, equal, intersect, within), metric (distance), and directional (orientation). All questions require multi-step inference, and many require cross-map grounding and reasoning. We evaluate eleven state-of-the-art LVLMs under two settings: (1) the direct setting, where we provide the maps relevant to the question, and (2) the contextual setting, where the model may have to identify the maps relevant to the question before reasoning. Even the strongest models, Gemini-2.5-Pro and GPT-5-Think, achieve only 38.20\% and 37.20\% accuracy, respectively, far below human performance of 84.87\%. These results reveal a persistent gap in multi-step cartographic reasoning, positioning FRIEDA as a rigorous benchmark to drive progress on spatial intelligence in LVLMs.

Published as a conference paper at ICLR 2026
FRIEDA: BENCHMARKING MULTI-STEP CARTO- GRAPHIC REASONING IN VISION-LANGUAGE MODELS
Jiyoon Pyo'
Yuankun Jiao'
Dongwon Jung²
Zekun Li'
Leeje Jang', Sofia Kirsanova'
Jina Kim'
Yijun Lin'
Qin Liu?
Junyi Xie
Hadi Askari
Nan Xu
Muhao Chen²
Yao-Yi Chiang'
University of Minnesota- Cities
University of California, Davis
University Southern California
ABSTRACT
Cartographic reasoning is the skill of interpreting geographic relationships by aligning legends, map scales, compass directions, map texts, and geometries across one or more map images. Although essential as a concrete cognitive ca- pability and for critical tasks such disaster response and urban planning, it remains largely unevaluated. Building on progress in chart and infographic un- derstanding, recent large vision language model (LVLM) studies map visual question-answe answering (VQA) often simplify maps as a special case of charts. contrast, map VQA demands comprehension of layered symbology (e.g., sym- bols, geometries, and text labels) as well as spatial relations tied to orientation and distance that often span multiple maps and are not captured by chart-style evaluations. To address this gap, we introduce FRIEDA, benchmark for test- ing complex open-ended cartographic reasoning in LVLMs. FRIEDA sources real map images from documents and reports in various domains (e.g.. geology. urban planning, and environmental assessment) and geographical areas. Follow- ing classifications in Geographic Information System (GIS) literature, FRIEDA targets all three categories of spatial relations: topological (border, equal, inter- sect, within), metric (distance), and directional (orientation). All questions re- quire multi-step inference, and many require cross-map grounding and reasoning. We evaluate eleven state-of-the-ar LVLMs under settings: (1) the direct set- ting, where we provide the maps relevant to the question, and (2) the contextual setting, where the model may have to identify the maps relevant the question before reasoning, Even the strongest models, Gemini-2.5-Pro and GPT-5-Think, achieve only 38. 20% and 37. 37.20% accuracy, respectively, far below human perfor- mance of 84.87%. These results reveal persistent gap in multi-step cartographic reasoning, positioning FRIEDA as rigorous benchmark drive progress on spatial intelligence in L.V.Ms..
INTRODUCTION
Recent advances large vision-languee lisoon-aagguage models (LVLMs) have markedly improved multimodal reasoning, with strong results across diverse visual -anssirnn (VQA) tasks (Dong et al.) 2024 Souibgui et al. 2025). Education and cognitive science research characterizes reasoning as broad capability that spans numeric reasoning. logical deduction (Holyoak & Morrison| 2012). and
textual interpretation (Whatton Kintsch, 1991). as well as interpreting pictures (Mayer 2020), spatial data (Li et al.) 2025), and map images (Goodchild 2012). Extensive LVLM benchmarks cover many of these facets: visual numeracy in chart and infographics (Lin et al. 2025 Mathew
et al. 2022 Masry et al. 2022), document and layout reasoning (Duan et al. 2025 Mathew al.
2021, multi-image inference (Kazemi et al., 2025 Xia et al.) (025), and even spatial relations
in natural images (Shiri et al. 2024). However, reasoning over maps, also core human compe-

---

Published as a conference paper at ICLR 2026
Q: What is the name of the Potentially Eligible Resources area that shares border with the MD Priority Funding Areas?
A; Kinsinger Farm
Dacaing Bedioo CStRRe
85
Chart offieeeeeee
60
Gtadtedo claSS
CINY
Columbia Tranamission Line
Priorty Funding
Potentially Eligible Resources
Sarage River State Forost
Tominsson Inn and Little Meadows (Stone House)
Gomrett County Employmmnt Centers
National Register Historic Site
Figure 1: Example of FRIEDA question requiring multi-map, multi-step cartographic reasoning.
To solve the question, the model must (1) use each legend to locate the two referenced regions, (2) evaluate the border spatial relation between them, and (3) read the map label of the qualifying feature to answer "Kinsinger Farm."
tence (Tversky 2003 Kastens & Ishikawa, 2006 Ishikawa & Newcombe, 2021). which we refer to cartographic reasoning, remains under-examined LVLMs.
Unlike natural images, maps encode information with an abstract, symbolic visual grammar (e.g.. map scales, compass/north arrows. and thematic symbology) (Buckley) (0066) which demands deeper interpretation than simple pattern recognition. Mastery of these elements must be coupled with the comprehension of spatial relations that are commonly grouped into topological reasoning (e.g., detecting shared boundaries), metric inference (e.g., converting map lengths to real-world dis- tances through the map scale), and directional reasoning (using a compass or north arrow) (Clemen- tini et al. 1993; Cohn & Renz 20077. In addition, human map-reading competencies (Liben et al.) 2010 Muir| 1985) frequently require these inferences across multiple maps within a single doc- ument. Correctly answering map question, therefore, draws map-specific skills (Hegarty & Waller 2005) such as interpreting map elements, reasoning over spatial relations, and integrating evidence across multiple maps, well broader capabilities emphasized in LVLM research that include text grounding (Singh et al., 2019 9
(Sidorov et al.) 2020), numeric and logical inference (Lu al. 2024a; Hu al. 2023). multi-image integration (Wang al.) [2024a] Xia al. (025). and retrieval (Wang et al. 2025a 2024b). A cartographic reasoning benchmark can therefore probe comprehensive reasoning and provide clear understanding of the spatial intelligence of LVLMs.
A growing line of work began to evaluate LVLMs on map VQA, yet these benchmarks do not fully assess cartographic reasoning. Earlier datasets pose chart-style questions that can answered with- out interpreting spatial relations, which bypasses the topological. metric, and directional inferences that are central to map comprehension (Koukouraki forts cover only a subset of relations as they target specific tasks such as navigation (Feng et al.) 2025 Kazemi et al. 2025) or entity identification (Dihan et al. 2025). While suitable for those objectives, such coverage is insufficient for evaluating human-like map understanding (Liben al.) 2010). Many benchmarks also restrict the stylistic variability of maps. Some focuses on choro- pleths (Koukouraki et al. 2025 Chang al. 2022 Mukhopadhyay et al. 2025). others rely maps created with map- map-coloring tools (Srivastava et al. 2025) or common web basemaps (Kazemi et al.)
al.) 2025 Chang et al., 2022). Other
2025 Dihan al. 2025). Several others focus limited thematic domains (e.g., geology (Huang
2

---

Published as a conference paper at ICLR 2026
restricted geographic coverage (Chang et al. 2022 Srivastava et al. 2025 These constraints overlook the heterogeneity in toponyms. label ing conventions, projections, and symbol- ogy that real-world cartography demands (Slocum et al.] 2022, Robinson et al. 1978). Multi-map reasoning is rarely evaluated, with limited exceptions ( (Kazemi et al. 2025). even though practi- cal cases often require integrating evidence across multiple maps (e.g., reconciling transit maps with future land-use maps for urban planning) and aligning overlapping information (Luiee & Moreland 1987). Moreover, although document-level multimodal understanding is emphasized in other LVLM benchmarks, existing map VQA benchmarks seldom require selecting the correct map among many images in long reports, despite government documents and technical documents containing numerous, visually similar, context-dependent maps (Federal Emergency Management Agency| 2025 U.S. Environmental Protection Agency 2025 SEDAR+ 2025). As a result, current map VQA settings underestimate the demands of comprehensive map understanding, leaving it un- clear whether LVLMs possess human-like map-reading competencies. Full cartographic reasoning remains beyond the scope of what existing VQA benchmarks assess.
al. 2025)
We introduce FRIEDA, a benchmark designed for evaluating multi-map, multi-step, comprehen- sive cartographic reasoning in LVLMs. We curate maps from public documents of various the- matic domains (e.g.. geological surveys. planning reports, environmental studies) to develop ques- tions that require models interpret maps as they appear in reports, mirroring practical scenarios in which a reader must synthesize evidence from maps embedded in a document (see Figure [ The collection encompasses diverse range of styles, projections, and scales. We create each question such that it requires (1) reasoning over topological, metric, and directional relations, (2) interpreting map elements and their semantics (e.g., legends, map scales, and north arrows), and, when appli- cable, (3) integrating information across multiple maps, and (4) selecting the appropriate map(s) from document to answer the query. To probe genuine reasoning rather than random guessing, the answers are in free-form (not multiple-choice) format. The benchmark evaluation comprises two settings: a direct setting, in which the model is given the relevant map images along with the question to evaluate map comprehension, and contextual setting, in which the model must first retrieve the correct maps from a broader within-document collection before answering. We score outputs using unified, task-aware protocol aligned to the three spatial-relation categories. We evaluate textual responses (topological and semantic labels) with LLM-assJudge (Gu et al.) 2025), distance responses (numeric values with units) with unit-aware parsing and mean absolute percent- age error (MAPE), and directional responses (cardinal directions for relative position) with angular tolerance over the eight directions. We compare the result against the human upper bound derived from multi- annotator agreement to contextualize LVLM performance. By aligning our tasks with the competencies expected of human map-readers (Goodchild) [012) Liben et al.) (2110) and explic- itly targeting compositional cross-image inference that is largely absent from prior map VQA work, FRIEDA fills a crucial gap in state-of-the-art LVLM evaluation.
Across 11 LVLMs (both proprietary and open source), we find that even tfate-of-the-art models struggle with multi-step cartographic reasoning. With FRIEDA-direct, where the relevant maps are provided, the best-performing model (Gemini-2.5-Prr) correctly answers fewer than 40% of the questions, far below human performance (> 80%). Overall accuracy remains essentially un- changed in the contextual setting, indicating that retrieval and disambiguation are not the primary bottlenecks; the core difficulty lies in cartographic reasoning itself. Our error analysis highlights re- curring failures, such as misreading legends (confusing symbol shapes and colors) and misaligning information across maps when map styles, projections, or map scales differ. We also observe hetero- geneous strengths across models (e.g., GPT-5-Think on multi-map questions and Claude-Sonnet-4 on distance queries). However, overall accuracy remains low, highlighting the gap between current LVLMs and the multi-step, cross -mage cartographic reasoning skills required.
We organize the remainder of the paper as follows. Section 2 formalizes the tasks and core skills of cartographic reasoning, Section 3 describes the benchmark design and dataset statistics; Section details the models, experimental setup, and evaluation protocol, and reports the results; Section presents ablations and error analyses.
3

---

Published as a conference paper at ICLR 2026
2
TASK DEFINITION
Cartographic reasoning is the ability to interpret maps and draw justified inferences from them. In FRIEDA, we design questions to assess core map-reading competence while mirroring realistic document use, where reader may need to navigate a document to locate the relevant map(s). All questions require (1) reasoning over spatial relations, (2) interpreting heterogeneous map elements, and (3) integrating evidence across multiple maps when necessary. We also include a (4) contextual setting in which additional maps are provided, requiring the model to identify relevant map(s) before performing the reasoning. We detail these categories and the accompanying taxonomy below.
Spatial Relation Spatial relations describe how geographic features relate in space (Carlson & Logan 2001), how they are positioned in space (Majic et al. 2021). and how their geometries interact Reezhhngg 1998). In geographic information systems (GIS) and spatial cogmition, these relations are often grouped into three categories: topological, metric, and directional (Cohn & Renz, 2007 Clementini et al. 1993). To make these abilities measurable and comparable, FRIEDA sep- arates questions by spatial relation type and grounds the topologiaal portion in the 9-intersection model (Clementini et al. 1993). We consolidate finer-grained subtypes into their broader categories cross classified as intersect, and contain classified as within), yielding four topological classes: border (shared boundary between regions), equal (coincident geometries), intersect (crossing or overlap of features), and within (containment or inclusion of one area inside another). We comple- ment these with one metric primitive, distance, and one directional primitive, orientation. Together, these six relations maintain the expressiveness of spatial queries while aligning with users' intuitive spatial reasoning.
cross
Map Elements Maps are symbolic representations that encode spatial information through ab- stract conventions (Slocum et al., 2022). Therefore, interpreting map elements is a distinct skill cen- tral to cartographic reasoning. The key elements we target are map text (place and feature names), legends (mappings from color, icons, and patterns semantic classes), map scaley (measurements that convert the map distance to the real-world distance), and the compass (ESRI, 2021). The styles of these components vary widely across maps: map texts may use different typography or placement rules (Monmonier] 2015), legends may use continuous color ramps or discrete pictograms (Slocum ct al. 2022), map scales may appear as bars or frames around the map (Robinson 1995). and the compass may be a compass rose or north arrow (Slocum et al., 2022). Practical map interpreta- tion requires grasping the concepts of map elements rather than simply recognizing their shapes. Consequently, our design includes questions that require reading map texts, decoding legends, using the map scale, and applying orientation to demonstrate true map literacy by linking abstract visual encodings to their underlying semantics.
Multi-Map Reasoning Beyond interpreting spatial relations and map elements, practitioners reg- ularly perform cross-map comparison and fusion to synthesize multiple map editions or thematic layers (Lupien & Moreland) 1987). Our multi-map setting reflects this practice: we curate questions that present two more maps together and require the model to integrate evidence by aligning shared symbols, reconciling differences in labels, map scales, and orientation, and identifying co- referent regions or features (Foody (0077). Extracting distributions and patterns is widely recognized as a core capability (Ishikawa 2016, Rexigel et al. 2024)) MMrita & Fukuyal (025). By testing this setting, we move beyond isolated spatial computation evaluate deeper cartographic reasoning across varied depictions of the same space.
Contextual Setting To mirror practical workflows (Mathew al., 2021 Tanaka et al. 2023), we evaluate a contextual setting map before answering a question. In this scenario, we provide the model with multiple maps from the same source (i.e., document), and the model must perform within-document retrieval using cues in the maps, such as titles, legends, or labels. By evaluating model performance on FRIEDA- contextual, capture core aspect of real map use: the model must understand how map elements encode meaning and leverage that understanding to select the required map from thematically related alternatives that vary in data layers. geographic extent, or purpose (Ishikawa 2016).
in which a model must identify the relevant
4

---

Published as a conference paper at ICLR 2026
Statisties
Number
Single
(41)
Malú (30)
Total questions
500
Total number of documents
210
Stagle (49)
Within (114)
Border (71)
Single (42)
Total number images
17,030
Map text
366 (73.2%)
Distance
Legend
417 (83,4%)
Orientation (89)
(91)
Mulíi (49)
Multi (57)
Compass
137 (27.4%)
Equal ($4)
Scale
46 (9.2%)
Interseet
(80)
Single-map
202 (40,4%)
Single
Mali
(12)
(54)
Multi-map map
298 (59.6%)
Mold (42)
(88)
Avg maps contextual
Inleeean
1:5.71
Figure Question distribution by spatial relation (in- ner) and map count (outer). Sizes are proportional to the number of questions each category.
Table 1: Key statistics.
FRIEDA
We present FRIEDA, benchmark for assessing LVLM's comprehensive cartographic reasoning. with an emphasis on cross-map (i.e., multi-image) scenarios. This section summarizes the bench- mark statistics and details the dataset curation procedure.
3.1 BENCHMARK STATISTICS
Table that FRIEDA comprises 17,030 map images drawn from 210 documents. To capture real-world variability, these maps span diverse geographies (32 countries) and six thematic domains, exhibiting heterogeneous styles, including varied color palettes, legends, and symbol conventions. The benchmark comprises 500 questions, of which 202 are single-map and 298 are multi-map. The multi-map subset consists primarily two-map comparisons (295 questions), with few cases requiring reasoning across three (2 questions) four (1 question) maps.
Figure reports the question distribution for each spatial relation, stratified by single- and multi- map questions The distribution is roughly balanced across relations and settings. We provide the detailed counts by spatial relation and setting in Appendix C.I and include a representative example of each in Appendix
For FRIEDA-contextull, we provide between 2 and irrelevant maps in addition to the relevant map(s) required to answer the question. The total number of input images averages 9.54 maps (a = 1.27) per question across both the single- and multi-map settings, with an average relevant-to- irrelevant maps ratio of 1:5.71.
3.2 BENCHMARK CONSTRUCTION
This section describes the construction of FRIEDA, which proceeds in four stages: map image collection, question generation, are-annotation curation, and validity verification.
Map Image Collection To capture stylistic and geographic diversity, we curate maps from pub-
licly available government and multilateral reports across domains including geology (SEDAR+
2025), national park management (National Park Service 2025), environmental assessments (U.S.
Environmental Protection Agency| 20255] Environmental Protection Agency 2025 Ministry Agri-
culture Climate Change and Environment (025), disaster response (Federal Emergency Manage-
ment Agency| 2025), urban planning (Seattle Planning and Community Development (024] De-
As equal denotes identical geometry (i.c., identical location and size), the benchmark contains single- map equal questions.

---

Published as a conference paper at ICLR 2026
partment of Community Development 2025 Urban Redevelopment Authority 2025
City of Cape Town 2025). and infrastructure investment (AllB) 2025). We limit the sources to documents writ- ten in Latin characters to focus cartographic reasoning rather than translation. We extract images using Idefics3-8B (I (Laurençon et al. 2024) with a custom prompt (Appendix [B.1) and manually ver- ify that each extracted contains only cartographic maps (examples of excluded non-maps appear in Appendix B.I.I. To support
we retain only documents with at least four maps. We select contextual maps based on their page proximity the target map; this ensures they are thematically and stylistically related to the target maps. We then shuffle the map order to prevent LVLM from using positional cues to identify the target maps.
Question Generation For each collected map, we use GPT-4 and GPT-o3 (see Appendix B.6 to propose candidate questions, their corresponding targeted spatial relations, and reference answers. Rather than enforcing fixed templates, we accept any phrasing that unambiguously expresses the intended relation to reflect various forms of paraphrases (e.g., "Is A within B? vs. "Does con- tain A?"). To ensure the benchmark tests visual cartographic reasoning over memorization or web retrieval, we screen each question for searchability using GPT with web search enabled. In addi- tion, we discard questions that can be answered without inspecting the map image. We label each question's 's answer type as textual (short span or categorical label), distance (numeric with units), or direction to enable analysis by response modality.
Pre-Annotation Curation All LLM-LLMMOOOE proposed candidate questions undergo a pre-annotation cu- ration stage. The two question curators (one with years of GIS experience and another with 2 years of experience in geospatial data) manually verify gold answers against source maps and rewrite or discard ambiguous questions. This step ensures that FRIEDA consists solely of clear, high-quality questions prior to the broader validation phase with the annotator.
Annotation Pipeline We validate each question using annotations from 11 Ph.D, researchers (eight with map expertise), collected over four weeks. Annotators confirm that each question is answerable from the provided map(s) and, for multi-map questions, verify that all maps are required to answer the question. To prevent bias, curators do not validate their own edits. We retain the ques- tion only if the majority (2 2/3) agrees with the gold answer. In a rare case (currently two questions in FRIEDA) where all three annotators agree an answer that contradicts the gold answer, we conduct a secondary review to update the gold answer consensus is reached. In total, we remove 61 questions that do not meet the agreement threshold of > 2/3, Appendix [B.3 de.3dis the instruction prompt provided to the annotators, and Appendix B.5 shows the annotation interface.
4 EXPERIMENTS
This section details the experimental setup, baselines, and evaluation metries, and then presents the main result, showing that FRIEDA is challenging benchmark even for the strongest LVLMs.
4.1 EXPERIMENTAL SETUP
Models We evaluate 11 LVLMs with multi-image support on FRIEDA. For proprictary models, we test three models: Gemmiii-2.5-P (Gemini Team 2025). GPT-5-Think (OpenAl 2025). and
- (Anthropic, 2025). For open source models, we consider eight model families and evaluate the largest available model from each family: LLaVA-NeXT-I 10B (Li et al. 2024b),
(Chen et al. 2024b). LLa VV---A-VVi Vision- 72B (Li et al. 2024a). Sive2.5VV--22. (Bai et al. 2025). InternVL3.35333 (Wang et al. 2025b
GLM4.5V- 108B (Team et al.) 2025), VnternVL.331
Ovis2-34B (Lu et al. 2024b), and Ovis2.5-9B (Lu et al.) 2025)..
To enforce determinism in open-source models, temperature-0. For proprietary models, we use the default settings for each model with maximum reasoning enabled (e.g., reason reasoning=highh for GPT-5-Think) and append the instruc- tion "Do not use search" to disable external retrieval. All models receive the same set of instructions that human annotators receive (Appendix B.44.
set do.sample-False and
We evaluate the 38B variant rather than the 241BA28B variant the latter activates only 28B parameters during inference. We report the results for the 241BA28B setting in Appendix F22

---

Published as a conference paper at ICLR 2026
Evaluation metries Answers in FRIEDA fall into three categories: textual, distance, and direc- tion. For textual answers, we employ an Small 3.1 (Mistral AI, 2024)) as the evaluator The full judge prompt appears in Appendix E.1 This setup handles minor variation (e.g.. "Cypress Creek" vs. "Cypress') by matching semantics rather than identifying exact string equality. For distance-based answer, report mean absolute percent- age еттоr (MAPE) and consider predictions within 20% error as correct, following Lewis] (1822). For directional answers, we mark response correct if matches the target cardinal direction within one adjacent label (e.g., if the gold answer is North, accept North, North West, and North East), reflecting the perceptual nature of the labels. We validate the reliability of the evaluation method against manual annotations, achieving Cohen's k of 0.9028 across all judged questions, which supports suitability for evaluation.
(Gu et al. 2025) method, utilizing Mistral
4.2 EVALUATION RESULTS
Figure summarizes the overall performance, and Table 2 accuracy by spatial relation. As FRIEDA retains questions with at least 2/3 annotator agreeing on the gold answer, we report ac- curacy for two subsets: All-Agree, where all three annotators agreed, and Partial-Agree, where 2/3 annotators agreed. All-Agree items serve as an indirect indicator of questions that are easier and less ambiguous for the annotators under our task and instructions, whereas Partial-Agree items may be considered as intrinsically more difficult or ambiguous to answer correctly. We also report the Overall Accuracy, which aggregates both subsets. Even the strongest LVLM (Gemini-2.5-PP) re- mains below 40% overall accuracy, well behind human performance at 84%. The best open-source result (Ovis2.5-9B-Thinkk achieves 24% overall accuracy, underperforming proprietary systems and falling far below human performance. We find no clear relationship between model size and perfor- mance, suggesting that training data, training objectives, and explicit reasoning mechanisms matter more than scale for cartographic reasoning. We present further analyses in the following section and in Appendix F22
FRIEDA-Direct
100
Proprictary
> 100B
-70B
-35B
<10B
All Agres Purce
80
Accuracy (%)
60
40
20
Human
Gemini--255-k 2.5-Pro
GPT--- Por-5-Tlink Think
Clande- Senmat
GLM- GLA--43V-1088 1V---00
Inter IntarVL33-70 3-7-88
L.L. One Vision 72B
Qwren2 Qwron255VV 77--
Inturn 1.mmlLL3...500 5.338
Ovis2 24B
Ovish S.9-- Think
Models
ALaVA-NeXT-IIO 1108
Figure 3: Overall accuracy of different models the FRIBDA-diredt benchmark.
ANALYSIS
Error analysis on Gemini Pro To pinpoint where LVLMs fail, we analyze Gemini-2.5-Pro on the All-Agree subset (167 questions in total). This ensures that our analysis targets distinct model
Mistral is not the language backbone of any tested LVL.M, thereby reducing potential bias Panickssery 2024

---

Published as a conference paper at ICLR 2026
Overall (500)
Border (71)
Distance (91)
Equal (54)
Intersect (80)
Orientation (89)
Within (115)
Human Average
84.87
89.00
78.28
89.10
85.53
91.80
88.08
Proprietary LVLMS
Semini-2.5-PPro
38.20
32.39
25.27
33.33
28.75
71.59
35.34
GPT-5-Think
37.20
25.35
27.47
44.44
31.25
69.32
28.45
Claude-Sonnet-
31.60
33.80
23.08
37.04
22.50
56.82
21.55
Open Source LVLMS
LLaVA-NeXT- -I0B
8.60
4.23
10.99
11.11
16.25
0.00
9.48
GLM-4.5V-IOBB
5,41
2.15
21.57
6.17
1.16
7.83
Intern VL3-78B
11.00
1.41
4.40
12.96
5.00
34.09
7.76
LLaVA-One Vision-72
13.00
9.86
10.99
5.56
8.75
29.55
10.34
Qwen2.5-VL-72
25.60
11.27
14.29
25.93
17.50
55.68
25.86
Intem
14.20
11.27
8.79
14.81
2.50
36.36
11.21
Ovis2-34B
17.80
25.35
13.19
25.93
26.25
2.27
18.97
Ovis2.5-9B-Think 25.80
12.68
20.88
24.07
22.50
51.14
21.55
Table 2: Overall and per spatial relation accuracy of human and LVLMs on FRIEDA-direct.
failures on questions that humans find straightforward. We assign each incorrect answer to a single primary error category. When multiple issues co-occur, we prioritize errors that occur earlier in the reasoning pipeline that propagate to downstream steps. The largest source of error is the misinter- pretation of legends (25.61%) %): cases in which the model assigns colors or symbols class. The remaining 23.78% is due to cross-map interpretation failures, which reflect difficulties in aligning the map scales and shared features across maps, and 16.46% due to spatial-relation se- mantics error, which arises when the model mixes up spatial relations (e.g., labeling region B within when it only touches at the boundary). Map-element misunderstandings include mistakes with the map scale (9. 9.76%; unit or ratio errors), map text (8.93%; selecting the wrong place feature from labels), geometry or shape reference (3.66%; pointing to the wrong area on the map), and ori- entation (3.05%; ignoring tilted compass). Finally, observe generic VQA errors not specific to cartography, such as miscounting (6.71% %),, subject-object confusion (1.82%; referring 'A relative to B' as 'B relative A'), and hallucination (1.20%). For the top three error categories, we provide examples and rationales returned by the three proprietary models in Appendix F.I
the wrong
Performance by spatial relation Figure 4 reports per-spatial relation accuracy for hu- man annotators and the three proprietary mod- els. LVLM performance broadly tracks the hu- man baseline: both most accurate ori- entation and struggle most with distance. On questions where an annotator answers incor- rectly. LVLMs are also incorrect 84.53% of the time. While GPT-5-Think and Gemini- 2.5-Pro achieve comparable overall accuracy, GPT-5-Think is stronger on tasks that require multi-map reasoning (Table[[) indicating bet- ter integration of evidence across maps. This is most evident in the equal relation ques- tions, a multi-map exclusive task, where GPT- 5-Think's accuracy is nearly 13% higher com- pared to Gemini-2.5-Pru. Sonnet-4 is the strongest distance questions, particularly those that require interpreting the map scale compute exact distances.
Equal
100
60
40
Intersect
Börder
Human
Gemini 2.55ii22PPPP
GPT-S-Thiik
Cloude Sonnet
Orientation
Within
Figure 4: Per spatial relation accuracy (%) of human annotators and three proprietary LVLMS (Gemini-2.5-Pp) GPT-5-Think, and Claude- Sonnet-4) on FRIEDA-direct.
Notably, Claude-
8

---

Published as a conference paper at ICLR 2026
Performance on contextual setting We observe minimal difference in accuracy between the FRIEDA-direct (Figure 3 and Table 2 and PRIEDA--ontextur (Appendix E.3 To verify that this is not an artifact of the accuracy metric, directly compare the per-question perfor- mance of the eight open-source models under deterministic settings (i.e., do sampl e=Fa sse and Camperatured
We observe 88.03% agreement in per-question performance between the di- rect and contextual settings, indicating that contextual images (maps from the same document that are not required to answer the question) rarely affect the model's predictions.
Impact of reasoning (think) on cartographic ques- tion types Despite being the smallest model tested with FRIEDA, Ovis2.5-9B-Thing
Model
Accuracy (%)
attains strong results (4th overall and 1st among open source models). To iden- tify what drives this performance,
Ovis2.5-9B
19.00
further evaluate Ovis2.5-9B with explicit reasoning (i.e., Think) disabled (Table 3 The overall accuracy of Ovis2.5-9B remains above the open-source average, indicating that model characteristics (e.g., architecture, training data) contribute to its strong performance. Enabling Think adds an addi- tional 5% performance gain. To identify which question types benefit from explicit reasoning and whether it improves cartographic performance, we manually analyze the 60 questions that only the Think variant answers correctly. Reasoning helps mostly with cardinal-direction questions, where north faces the top of the image (48.33%) followed by multi-map map alignment (23.33%). Additional improvements arise from correctly reading map text (15%), interpreting the map scale (5%), as- sociating the legend with the symbol (5%), and counting (3%). Together, these patterns suggest that explicit reasoning primarily strengthens orientation-related and multi-map questions, which are central to cartographic reasoning, whereas yielding smaller gains in symbol and map-scale interpre- tation
Ovis2.5-9B-TII. 24.80
Table 3: Performance of Ovis2.5 model on FRIEDA-dired
RELATED WORK
Document& & Infographic/Chart VQA Recent benehmarks established baselines for LVLM rea- soning over documents and designed graphics. In the document domain, DocVQA (Mathew et al.,
202T) and DocV (Souibgui al., 2025) introduce a large-scale guestion-answering (QA) dataset over real forms and reports, while DocoPilot (Duan et al., (025) extends evaluation to sci- entific articles, which involve embedded figures. For graphics, Infographics VQA (Mathew et al.) 2022) and InfoChartQA (Lin et al., (025) test reasoning over rich layouts and charts. In general, frontier LVLMs reveal competence at high-level patterns, such as trends and extrema, but struggle with precise value extraction and robustness. FRIEDA evaluates these shortcomings in a carto- graphic setting where layout, symbols, legends, scales, and compass orientation interact tightly to measure how well LVLMs integrate these signals to answer map-based questions.
Map VQA and Spatial Reasoning While recent map VQA benchmarks have expanded the scope of evaluation, they remain constrained to single-map tasks or specific domains. MapQA (Chang lt al. 2022) evaluates question answering on choropleth maps and shows that general VQA and ChartVQA systems underperform on maps. MapWise (Mukhopadhyay et al. 2025) broadens the geographic coverage, while MaplQ (Srivastava et al.) 2025) extends the map type coverage to in- clude cartograms and proportional-synee -yymoll maps. MapEval (Dihan et al., (025) assesses geospatial reasoning across various cities, and it reports consistent human-LVLMs performance gaps. Domain- specific efforts include PEACE (Huang et al. 2025) for geology, CartoMark (Zhou et al. 2024) for text extraction and recognition, and MapBench (Xing et al., 2025) and ReasonMap (Feng et al.) 2023) for navigation. However, these benchmarks rarely test cross-image reasoning on heteroge- neous sources and often rely limited set of spatial relations. While ReMI (Kazemi et al. 2025) explores the cross-image setting, the questions lack cartographic focus. We detail key differences between prior map VQA benchmarks and FRIEDA in Appendix
G
Spatial reasoning benchmarks, such as Spatial VLM (Chen et al. 2024a) and SpatialRGPT (Cheng al. 2024), have advanced spatial perception and reasoning on natural images. In the geospatial
"We further evaluate the association between performance and model size in Appendix F2
9

---

map el- Published as a conference paper at ICLR 2026
domain, GeoChain Yerramilli al. 2025) enhances tasks like geolocalization by inducing step-by- step geographic reasoning. However, these works do not engage with symbolic conventions unique to maps (i.c., legends, scales, compasses, and map texts). In contrast, our benchmark closes this gap by evaluating multi-step cartographic reasoning over heterogeneous, real-document maps, which re- quires models integrate evidence across multiple figures and align legends, scales, and orientation to infer key spatial relations (i.e., border, distance, equal, intersect, orientation, and within).
CONCLUSION
We present FRIEDA, a benchmark for evaluating multi-step cartographic reasoning over real-world maps across six core spatial relations, where answering often requíres aligning evidence across mul- tiple maps. Our evaluation of state-of-the-arr LVL.Ms reveals a substantial gap between current performance and the proficiency required for reliable map understanding. Models frequently fail to ground symbologic map entities, to integrate evidence distributed across multiple maps, and to cor- rectly interpret spatial topology. Our analysis further shows that these failures fied versions of errors observed in prior VQA benchmarks, but also expose additional map-specific challenges, including combol-to-sseantice mapping. adherence geometry-aware reasoning. These findings highlight the need for novel architectures and effective training strategies that incorporate cartographic priors and enable explicit reasoning ements and spatial structure. To catalyze progress, we release FRIEDA along with the question taxonomy, baseline results, and evaluation scripts to enable graingd diagnosis and reproducible comparisons. We encourage the community to build on FRIEDA with methods that explicitly in- tegrate text, symbology, and geospatial structure, moving toward LVLMs that reason robustly and reliably over real-world maps.
merely ampli-
cartographic conventions, and
ETHICS STATEMENT
We introduce benchmark for evaluating cartographic reasoning in large vision-language models. We curate maps from publicly available documents (e.g., government reports, planning, and envi- ronmental studies) and retain only the figures necessary for research purposes. To the best of our knowledge, we use all materials under terms that permit research and non-commercial distribution.
annotators provided informed consent. We collected no personal data about annotators beyond task performance. Our institution's 's IRB reviewed the annotation protocol and determined that the project does not constitute human subjects research; therefore, no further IRB review was required.
The benchmark inevitably reflects the patterns in the source documents and may exhibit represen- tation bias, including uneven geographic coverage and map types, English-languse focus, and un- equal representation across regions and themes. We document these limitations and their potential impact in the dataset card (Appendix A to aid transparency and interpretation.
REPRODUCIBILITY STATEMENT
We released the benchmark dataset (images. QA JSON, and taxonomically classified questions) on HuggingFace along with all prompt and configuration files used for annotation and inference. We also provide the full codebase for data loading, inference, and evaluation on GitHub well code to replicate the annotation interface. The results obtained during testing are released resuut branch.
ACKNOWLEDGEMENTS
This material is based upon work supported in part by the National Science Foundation under Grant No. BCS-2419334 A subset of the authors received support from NSF Grants OAC-2531 126 and ITE-2333736, as well as from the Amazon Nova Trusted Al Prize.
REFERENCES
AIIB. Our projects, 2025, URL http://pps/./w.aaa....oo//////pp html
10

---

Published as a conference paper at ICLR 2026
Anthropic. System card: Claude opus 4 & claude sonnet 4, 2025.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang. Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint ar 2rv2 13923, 2025.
Emily M. Bender and Batya Friedman. Data statements for natural language processing: To- ward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 1:587-604, 2018. doi: 10.1162/tac...00041... URL https:
Aileen Buckley. Make maps people want to look at: five primary design principles for cartography.
ArcNews Online, 2006.
I. A Carlson and G D Logan. Using spatial terms to select an object. Mem. Cognit., 29(6):883-892, September 2001.
Shuaichen Chang, David Palzer, Jialin Li, Eric Losler-Lussier, and Ningchuan Xiao. MapQA: A dataset for question answering choropleth maps. In NeurIPS 2022 First Table Representation Workshop, 2022. URL httpt::tt///tt.. /ttt::::///peereieww net. wttp://wwww.ettttooddddee----- F
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing rision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14455- 14465, June 2024a.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185 24185-24198, 2024b.
An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang. Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision- lsioualanguage models. In NeurlPS, 2024.
City of Cape Town. Document - centre
Document centre, 2025. URL
za/
Eliseo Clementini, Paolino Di Felice, and Peter van Oosterom. A small set of formal topological relationships suitable for end-user interaction. David Abel and Beng Chin Ooi (eds.), Advances in Spatial Databases, pp. 277-295, Berlin, Heidelberg, 1993. Springer Berlin Heidelberg. ISBN 978-3-540-44765-5.
Anthony G. Cohn and Jochen Renz. Handbook of Knowledge Representation. Elsevier Science San Diego, CA, USA, 2007. ISBN 0444522115.
Department of Community Development. Media centre, 2025. URL htttp:///ddde /atp::/.ddddd.go.... gov. ae/
Mahir Labib Dihan, MD Tanvir Hassan, MD TANVIR PARVEZ, Md Hasebul Hasan, Md Almash Alam, Muhammad Aamir Cheema, Mohammed Eunus Ali, and Md Rizwan Parvez. Mapeval: A map-based evaluation of geo-spatial reasoning in foundation models. In Forty-second Interaa tional Conference on Machine Learning. 2025. URRLbttts:///peneeeee net.ee.wweeeeeeeee forum?
id 1d=hS2Ed5XYRG
Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang. Winston Hu, Yongming Rao, and Ziwei Liu. Insight-v: Exploring long-chain visual reasoning with multimodal large language models. arXiv preprint 2Xi::2411.14433 2024.
11

---

Published as a conference paper at ICLR 2026
Yuchen Duan, Zhe Chen, Yusong Hu, Weiyun Wang. Shenglong Ye, Botian Shi, Lewei Lu, Qibin Hou, Tong Lu, Hongsheng Li, et al. Docopilot: Improving multimodal models for document-level understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp- 4026-4037, 2025.
Environmental Protection Agency.
Publications, 2025.
URL https:
publications/
ESRL Map elements, 2021. wttptttss:///esttoppoee maapgee-_ayyuutttapp-eeeeee hwwp://tnaapeeeeettsssssssss
Federal Emergency Management Agency. Fema, 2025. URL https: fema. çov/
Sicheng Feng, Song Wang, Shuyi Ouyang, Lingdong Kong, Zikai Song, Jianke Zhu, Huan Wang, and Xinchao Wang. Can milms guide me home? a benchmark study on fine-grained visual reasoning from transit maps. arXiv preprint WrXiv:2505.88677 2025.
Giles M Foody. Map comparison in GIS. Prog. Phys. Geogr, 31(4):439 445, August 2007.
Timnit Gebru, Jamie Morgenstem, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. Datasheets for datasets. Commun. ACM, 64 64(12):86-9)) 92, November 2021. ISSN 0001-0782. doi: 10.1145/3458723. URL htllhhtts://dee htttps://doi.crgg///
1145/3458723
Google Gemini Team, Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025.
Michael F. Goodchild. The fourth r? rethinking gis education. ArcUser Fall, pp. 46-51, 2012.
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. A survey on Ilm-as-a-judge, 2025. httpt://///txx org/abs/
2411. 15594
Mary Hegarty and David A. Waller. Individual Differences Spatial Abilities, pp. 121-169. Cam- bridge Handbooks in Psychology. Cambridge University Press, 2005.
Keith J. Holyoak and Robert G. Morrison. The Oxford Handbook of Thinking and Reasoning, Ox- ford University Press, 03 2012. ISBN 9780199734689, doi: 10. 10.10999/o4f00ahhh77779999994499 001.0001.
URL http:://.ooo...rrr0............0
0001
Pengbo Hu, Jingxian Qi, Xingyu Li, Hong Li, Xinqi Wang, Bing Quan, Ruiyu Wang, and Yi Zhou. Tree-of-mixed-thoe Combining fast and slow thinking for multi-hop visual rea- soning. ArXiv, abs/2308 006658, 2023. URL https:/ sateps://api... /api...... seeanticsccool. org/ Corpus 11:
Yangyu Huang. Tianyi Gao, Haoran Xu, Qihao Zhao. Yang Song, Zhipeng Gui, Tengchao Lv, Hao Chen. Lei Cui, Scarlett Li, al. Peace: Empowering geologic map holistic understanding with mllms. arXiv preprint arXiv:2501.06/88 2025.
Toru Ishikawa. Spatial thinking in geographic information science: Students' geospatial concep- tions, map-based reasoning, and spatial visualization ability. Ann. Am. Assoc. Geogr, 106(1): 76-95, January 2016.
Toru Ishikawa and Nora S Newcombe. Why spatial is special in education, learning, and everyday activities. Cogn. Res. Princ. Implic., 6(1):00, March 2021.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray. Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL TtIlhttps:/// wtt./viivvv...///// org/ abs 2001. 08361
Kim A Kastens and Toru Ishikawa. Spatial thinking in the geosciences and cognitive sciences: A cross-disciss discippisyyy look at the intersection of the two fields. In Earth and Mind: How Geologists Think and Learn about the Earth, pp. 53-76. Geological Society of America, 2006.
12

---

Published as a conference paper at ICLR 2026
Mchran Kazemi, Nishanth Dikkala, Ankit Anand, Petar Devic, Ishita Dasgupta, Fangyu Liu, Bahare Fatemi, Pranjal Awasthi, Dee Guo, Sreenivas Gollapudi, and Ahmed Qureshi. Remi: a dataset for reasoning with multiple images. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS '24, Red Hook, NY, USA, 2025. Curran Associates Inc. ISBN 9798331314385.
Eftychia Koukouraki, Auriol Degbelo, and Christian Kray, Assessing Map Reproducibility with Visual Peestion-Answering. An Empirical Evaluation. In Katarzyna Sila-Nowicka, Antoni Moore, David O' O'ulliiaa,, Benjamin Adams, and Mark Gahegan (eds.), 13th International Con- ference on Geographic Information Science (GiScience 2025), volume 346 of Leibniz Interna- tional Proceedings in Informatics (LIPlcs), pp. 13:1-13:: 12, Dagstuhl, Germany, 2025. Schloss Dagstuhl Leeibniz-Zentrumn für Informatik. ISBN 978-3-95977-378-2. doi: 10.4230/LIPics. GIScience. 2025. 13. URL penttts://droo http://s....apa....eeeeeeeeeeaaa
[2200 2tw.eec2innnee222..... 2025.13
Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better under- standing vision-language models: insights and future directions., 2024.
C.D. Lewis. Industrial and Business Forecasting Methods: A Practical Guide to Exponential Smoothing and Curve Fitting Butterworth scientific. Butterworth Scientific, 1982. ISBN 9780408005593. URL attps: httpss://boooss wooooko..ooogee...oooooooooooooo google. 7id-
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.00 033266 2024a.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895 2024b.
Zekun Li, Malcolm Grossman, Mihir Kulkarni, Muhao Chen, Yao Yi Chiang, et al. Mapqa: Open- domain geospatial question answering on map data. arXiv preprint arXiv:2503.0787,, 2025.
Lynn S. Liben, Lauren J. Myers, and Adam E. Christensen. Identifying locations and direc- tions on field and representational mapping tasks: Predictors of success. Spatial Cognition & Computation, 1012-3):105-134, 2010.
doi: 10.1080/13885586000555533
URL https:
Minzhi Lin, Tianchi Xie, Mengehen Liu, Yilin Ye, Changjian Chen, and Shixia Liu. Infochartqa: A benchmark for multimodal question answering on infographic charts, 2025. URL https:
//arxi
19028
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference Learning Representations (ICLR), 2024a.
Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural embedding alignment for multimodal large language model. 2rXiv:2405.20797, 2024b.
Shiyin Lu, Yang Li, Yu Xia, Yuwei Hu, Shanshan Zhao, Yanqing Ma, Zhichao Wei, Yinglun Li, Lunhao Duan, Jianshan Zhao, Yuxuan Han, Haijun Li, Wanying Chen, Junke Tang, Chengkun Hou, Zhixing Du, Tianli Zhou, Wenjie Zhang, Huping Ding, Jiahe Li. Wen Li, Gui Hu, Yiliang Gu, Siran Yang. Jiamang Wang, Hailong Sun, Yibo Wang, Hui Sun, Jinlong Huang, Yuping He, Shengze Shi, Weihong Zhang, Guodong Zheng, Junpeng Jiang, Sensen Gao, Yi-Feng Wu, Sijia Chen, Yuhui Chen, Qing-Guo Chen, Zhao Xu, Weihua Luo, and Kaifu Zhang, Ovis2.5 technical report. arXiv:2508. 11737, 2025.
Anthony E Lupien and William H Moreland. A general approach to map conflation. In Proceedings of 8th International Symposium on Computer Assisted Cartography (AutoCarto 8), pp. 630-639, 1987.
13

---

Published as a conference paper at ICLR 2026
Ivan Majic, Elham Naghizade, Stephan Winter, and Martin Tomko. RIM: ray intersection model for the analysis of the between relationship of spatial objects in a 2D plane. Geogr. Inf. Syst., 35 (5):893-915 May 2021.
Ahmedand Masry, Doand Long, Jia Qingand Tan, Shafiqand Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACI. 2022, pp. 2263-2279, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.186/3/v1/0222 www.:o....innnnaal..... URL
attps://aclann 200. 2022. wtt.:22.....ninee. findings-acl. 177
Minesh Mathew, Dimosthenis Karatzas, and Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2200-2209, January 2021.
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and C. V. Jawa- har. Infographievga. In 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2582-2591, 2022. doi: 00.100//AACV1145.2200..22
Richard E Mayer. Multimedia learning. Cambridge University Press, Cambridge, England, edi- tion, July 2020.
McNEMAR. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153-157, June 1947.
Ministry of Agriculture Climate Change and Environment. Resources, 2025. URL https://
macce. wtt.ggpcc.....re.eaa..
Mistral AL. Mistal-smallliisttnee--0999
htttss://hugginggaacecoo//
Mistral- Small Insarltnsttuctt-.. -2409 2024. Hugging Face model card. License: Mistral AI
Research License (MRL).
M. Monmonier. The History of Cartography, Volume Six: Cartography the Twentieth Century. University of Chicago Press, 2015, ISBN 9780226152127. URL httht:t//ooskk:: google.
cdlbookssidd Bidoossid-B BARGEEAAREAAABA
Aiko Morita and Izumi Fukuya. Integrative processing of text and multiple maps in multimedia learning: an eye-tracking study. Front. Psychol., 16(1487439)11487433, August 2025.
Sharon Pray Muir. Understanding and improving students' map reading skills. Elem. Sch. J., 86(2):
207-216, November 1985.
Srija Mukhopadhyay, Abhishek Rajgaria, Prerana Khatiwada, Manish Shrivastava, Dan Roth, and Vivek Gupta. MAPWise: Evaluating vision- lssionllangagg models for advanced map queries. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume 1: Long Papers), pp. 9348-9378, Albuquerque, New Mexico, April 2025. Association for Computational Linguisties. ISBN 979-8-89 11----9977679--- doi: 10.18653/v 1/2025.
naacl-long.473 473. URL
National Park Service. publications.c .hm
Publications, 2025.
URL
OpenAl. GPT-5 system card, 2025.
Arjun Panickssery, Samuel R. Bowman, and Shi Feng. Lim evaluators recognize and favor their own generations. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NeurlPS '24, Red Hook, NY, USA, 2024. Curran Associates Inc. ISBN 9798331314385.
Guo Renzhong, SPATIAL OBJECTS AND SPATIAL RELATIONSHIPS. Geo Spat. Inf. Sci., 1(1): 38-42, October 1998.
14

---

Published as a conference paper at ICLR 2026
Eva Rexigel, Jochen Kuhn, Sebastian Becker, and Sarah Malone. The more the better? systematic review and meta-analysis of the benefits of more than two external representations in STEM education. Educ. Psychol. Rev., 36(4), December 2024.
A.H. Robinson. Elements of Cartography. Wiley, 1995. ISBN 9780471555797, URL https:
7/books. goog wwwpogle..c/oooooooooooooooooooo http:///ooosiiimmyyAAAAAaaa
A.H. Robinson, R.D. Sale, and J.L. Morrison. Elements of Cartography. Wiley, 1978. ISBN
9780471017813. URL httpss://bbbtto https////ooooooooolllllo www.loooooooooooooo com/ www/books/////ooooooooooooooosso www.ooksssiddoknnttueeeee
Seattle Planning and Community Development. Current projects, 2024. URL https: //www.
seattle, gtwp:ppdcoooorpeeeeccae jeote
SEDAR+.
Sedar archive, 2025.
URL
$ SeeneeetwysSsiillessseeeeeerrrrn
Fatemeh Shiri, Xiao- Yu Guo, Mona Golestan Far, Xin Yu, Reza Haf, and Yuan-Fang Li. An empir- ical analysis on spatial reasoning capabilities of large multimodal models. In Yaser AI-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 21440-21455, Miami, Florida, USA, Novem- ber 2024. Association for Computational Linguistics. doi: 10. 10.18863/v1/2224.cmmo 1e122224.mmlpmaai.... 1195.1195. 111-mmiiin.n11111
URL aattps::/aclanthooogg.oo org. emnlp-main. 1195/]
Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. 2020.
Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pp. 8317-8326. 2019.
Terry A Slocum, Robert McMaster, Fritz C Kessler, and Hugh Howard. Thematic cartography and geovisualization, fourth edition. CRC Press, London, England, 4 edition, August 2022.
Mohamed Ali Souibgui, Changkyu Choi, Andrey Barsky, Kangsoo Jung, Ernest Valveny, and Dimosthenis Karatzas. DocA Context-aware visual explanations for document ques- tion answering. In Forty-second International Conference Machine Learning, 2025. URL
Varun Srivastava, Fan Lei, Srija Mukhopadhyay, Vivek Gupta, and Ross Maciejewski. MapIQ: Eval- uating multimodal large language models for map question answering. In Second Conference on Language Modeling, 2025, Utthttps: openreview net/ forum ? VdVVSRRttBBnn
Ryota Tanaka, Kyosuke Nishida, Kosuke Nishida, Taku Hasegawa, Itsumi Saito, and Kuniko Saito. Slidevqa: a dataset for document visual question answering on multiple images. In Proceedings of the Thirty-Sevente AAAI Conference on Artificial Intelligence and Thirty- Fiffy Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium Educational Ad- vances in Artificial Intelligence,
AAAI Press, 2023. ISBN 978-1- 57735-880-0. doi: 10. Na.16609aai.vv331/ 26598.. URL https: // org 0 609/ aaai. 017111.26598
Team, Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi. Junhui Ji, Lihang Pan, Shuaiqi Duan, Weihan Wang, Yan Wang, Yean Cheng, Zehai He, Zhe Su. Zhen Yang. Ziyang Pan, Aohan Zeng, Baoxu Wang. Bin Chen, Boyan Shi, Changyu Pang, Chenhui Zhang, Da Yin, Fan Yang. Guoqing Chen, Jiazheng Xu, Jiale Zhu, Jiali Chen, Jing Chen, Jinhao Chen, Jinghao Lin, Jinjiang Wang, Junjie Chen, Leqi Lei, Letian Gong, Leyi Pan, Mingdao Liu, Mingde Xu, Mingzhi Zhang, Qinkai Zheng, Sheng Yang, Shi Zhong, Shiyu Huang, Shuyuan Zhao. Siyan Xue. Shangqin Tu, Shengbiao Meng, Tianshu Zhang, Tianwei Luo, Tianxiang Hao, Tianyu Tong, Wenkai Li, Wei Jia, Xiao Liu, Xiaohan Zhang, Xin Lyu, Xinyue Fan, Xuancheng Huang, Yanling Wang. Yadong Xue, Yanfeng Wang. Yanzi Wang, Yifan An, Yifan Du Yiming Shi, Yiheng Huang, Yilin Niu, Yuan Wang, Yuanchang Yue, Yuchen Li, Yutao Zhang, Yuting Wang, Yu Wang, Yuxuan Zhang, Zhao Xue, Zhenyu Hou, Zhengxiao Du,
15

---

Published as a conference paper at ICLR 2026
Zihan Wang, Peng Zhang, Debing Liu, Bin Xu, Juanzi Li, Minlie Huang, Yuxiao Dong, and Jie Tang. Glm-4.5v and Iv-thnkkig Towards versatile multimodal reasoning with scalable reinforcement learning, 2025.
abs /bt.////s///22 01006
Barbara Tversky. Navigating by mind and by body. In Christian Freksa, Wilfried Brauer, Christopher Habel, and Karl F. Wender (eds.), Spatial Cognition III, pp. 1-10, Berlin, Heidelberg, 2003. Springer Berlin Heidelberg. ISBN 978-3-540-45004-7
Urban Redevelopment Authority. Master plan, 2025. URL https:/ hopps://www... www. ura.
Corporatee/Plaannnn//aatttt Plan
U.S. Environmental Protection Protectionnngeeeeeeeen Agency. Envirommental impact statement (eis) database, 2025. URL wwws//iittttiii..eie html
Fei Wang, Xingyu Fu, James Y Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, et al. Muirbench: A comprehensive benchmark for robust multi-image understanding. arXiv preprint arXiv:2406. 09411, 2024a.
Hengyi Wang. Haizhou Shi, Shiwei Tan, Weiyi Qin, Wenyuan Wang, Tunyu Zhang, Akshay Nambi, Tanuja Ganu, and Hao Wang. Multimodal needle in a haystack: Benchmarking long- context capability of multimodal large language models. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 3221-3242 -3241,, Albuquerque, New Mexico, April 2025a. Association for Com- putational Linguistics. ISBN
189-6. doi: 10. wogy 0r0/ nwaclaaaa-aamm -amaaclllogg.. 1667
URL
Weiyun Wang, Shuibo Zhang, Yiming Ren, Yuchen Duan, Tiantong Li, Shuo Liu, Mengkang Hu, Zhe Chen, Kaipeng Zhang, Lewei Lu, et al. Needle in a multimodal haystack. arXiv preprint arXiv:2406.07220,, 2024b.
Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internv13.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint a5Xiv: 18265, 2025b.
Cathleen Wharton and Walter Kintsch. An overview of construction-integratin model: a theory of comprehension as a foundation for a new cognitive architecture. SIGART Bull., 244):169-173 July 1991 ISSN 0163-5719. doi: 10.1145/1224441122399 URL
1145/122344 122379
Peng Xia, Siwei Han, Shi Qiu, Yiyang Zhou, Zhaoyang Wang. Wenhao Zheng, Zhaorun Chen, Chenhang Cui, Mingyu Ding, Linjie Li, Lijuan Wang. and Huaxiu Yao. MMIE: Massive multi- modal interleaved comprehension benchmark for large vision-language models. In The Thirteenth International Conference on Learning Representations, 2025. URL htthhttst///opeeeeeess
net/ www.mmm2.ee.hhhhhe.....
Shuo Xing, Zezhou Sun, Shuangyu Xie, Kaiyuan Chen, Yanjia Huang, Yuping Wang, Jiachen Li, Dezhen Song, and Zhengzhong Tu. Can large vision language models read maps like a human?, 2025. URL httttttt///aaaktt
orq/ abs 2503. 14607
Sahiti Yerramilli, Nilay Pande, Rynaa Grover, and Jayant Sravan Tamarapalli. GeoChain: Mul- timodal chain-of-thought for geographic reasoning. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Findings of the Association for Computational
Linguistics: EMNLP 2025, pp. 23624 23639, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN
doi: 10. 10.18653/v 1/2025. findings-emni
1284. URL htt.hhtsss/cccaatttllooooo
findings emn pp 1284/
Xiran Zhou, Yi Wen, Zhenfeng Shao, Wenwen Li, Kaiyuan Li, Honghao Li, Xiao Xie, and Zhigang Yan. Cartomark: a benchmark dataset for map pattern recognition and map content retrieval with machine intelligence. Scientific Data, 11(1), November 2024. ISSN 2052-4463, doi: 10310 $415977-02-044057-
UtLbttp: http:///w.........oooo..../////.
16

---

Published as a conference paper at ICLR 2026
A DATACARD
We adopt the data statement framework of Bender & Friedman (2018) and integrate complementary fields from Datasheets for Datasets (Gebru et al. 2021) to centralize key information for the analysis, reuse, and deployment of FRIEDA.
A.1 CURATION RATIONALE
We design FRIEDA evaluate carographic reasoning (the ability interpret map-specific symbols, comprehend spatial relations, and integrate evidence across one or more maps). We source questions from public documents reflect map-reading tasks encountered in practice (e-g.. planning, hazard assessment, and geology). High-level goals, task definitions, and design choices appear in the main text (Section Fand and Section Bp.
We further expand the benchmark curation process in Appendix
A.2 BENCHMARK COMPOSITION
Total size: 500 validated questions; each question with gold answer
Agreement:
Each question annotated by three annotators; we record the problem-level agreement and mark items with unanimous agreement on the gold answer as All-Agree, and those with 2/3 agreement as Partial-Agree.
Modalities:
Every question involves one or multiple map image(s) and associated question
text.
Spatial relations (6): Border, Equal, Intersect, Within, Distance, Orientation
Answer types (3): Textual (short text), Distance, and Direction
Public documents from 32 countries across six continents. Documents are from six domains (urban planning, environmental assessment, national park manage- ment, geologic reports, disaster and hazard reports, infrastructure and investment reports).
Provenance:
Sources are detailed further in Section 3 and Appendix C.3
Languages: Questions and instructions are in English (en-US); source maps primarily use English labels but may include other languages written in the Latin script.
A.3 DATA COLLECTION PROCESS
Acquisition: We collected maps from public reports, then filtered for reading map elements and task suitability.
Question creation: Curators wrote questions that required reading the legend, scale, and compass, and reasoning over one or more spatial relations; questions were rejected if (1) they were solvable without using any maps or (2) question ambiguity could not be re- solved by manual editing.
A.4 ANNOTATOR DEMOGRAPHIC
We share the annotator demographics contextualize potential biases while preventing identification.
Count: 11 Annotators in total (2 also served question curator)
Academic background: Ph.D. Researchers [100%]
GIS/cartography background: <1 year: [27%]; years: [27%]] 3-5 years: [18%]; 5+
years: [27%].
Language: All authoring and communication used American English (en-US), As the task focuses on cartographic symbols and spatial relations (not dialect), we do not report in- dividual annotator nationalities. Non-native participation may introduce minor phrasing variance. We standardized qustion phrasing during review and removed questions flagged ambiguous by 2 2/3 annotators.
17

---

Published as a conference paper at ICLR 2026
A.5
EVALUATION & METRICS
Primary metric: Accuracy
Textual (LLM-as-Judge) After attempting exact string match, LLM-as-Judge to compare model outputs to gold answers. Appendix E.1 provides the judging prompt for reproducibility.
Distance (MAPE): We apply mean absolute error (MAPE) and unit-aware parsing and consider all distance answers with in 20% as correct.
Direction: We canonicalize directional answers to the eight cardinal directions and con- sider all cardinal direction within one adjacent unit as correct.
A.6 KNOWN LIMITATIONS & BIASES
Regional representation bias: As FRIEDA uses only English-language documents, re- gions where English is dominant language are overrepresented, and non-English conven- tions and locales are not covered.
Domain skew: The corpus emphasizes planning, environmental, and government reports with less coverage on other types of maps, such nautical or military charts.
B DETAILED BENCHMARK CONSTRUCTION
B. MAP IMAGE FILTERING
We use Idefics3-8l 8B Laurençon al. 2024) to filter map images from the document. To produce strict Yes/No decision, we prompt the model:
Is this cartographic map? Answer only y with Yes or
We consider any image for which the model responds Yes as a candidate map.
B.1.1 NON-MAP EXAMPLES
We manually verify all map candidate images and remove those that we do not consider maps. For example, although Figure 5 shows silhouette of city with subdivision, consider it stylized graphic rather than cartographic map. The image lacks essential map elements (i.e., map texts, legend, scale, and compass), which are needed support cartographic reasoning. Without these components, we cannot reason about locations, distances, or spatial relationships; therefore, exclude such images from dataset and do not treat them maps for FRIEDA.
B.2 DEFINITION OF SPATIAL RELATION
Figure 6 visualizes the four topological spatial relations evaluated in FRIEDA: border, equal, inter- sect, and within.
B.3 ANNOTATOR PROMPT
To standardize responses and minimize ambiguity, we supply annotators with fixed instruction set (Figure 7). We introduce these guidelines during task onboarding and repeat them at the start of every question to promote consistent answer format.
B.4 LVLM SYSTEM PROMPT
To ensure consistency, use the same instruction provided to human annotators the prompt for the LVLM system. As some LVLMs produce intermediate reasoning, we append final line standardize the output: Give the final answer in 'Final answer: (your answer). r). ,. For the proprietary models, we additionally include the clause Do not use online search to prevent external browsing
'ee add this clause as precautionary measure; during the dataset construction phase, we verify that ques- tions not directly answerable through web search.
18

---

Published as a conference paper at ICLR 2026
CITY CAPE TOWN ISIKEKO SASEKAPA
Figure 5: An example non-map image flagged by Idefics3-8B as a candidate map. The image is graphic from the cover page of the document. We exclude it from the benchmark after manual verification, as we consider graphical image rather than a cartographic map.
Border
Equal
Within
Intersect
A
A
B
A
B
A
B
B
Figure 6: Illustrations of the spatial relations evaluated in the benchmark. Columns show Border, Equal, Within, and Intersect; rows provide representative cases across geometry types-areas areas, lines, and points.
19

---

Published as a conference paper at ICLR 2026
For cach one, please verify whether it can be answered (Q# Validation) using the provided map(s). If an image appears too small, click the image. For question with multiple images, please mark whether all images required correctly answer the question (Q# M). You may tools like ruler or calculator,
use online search.
For each questions:
General:
If question can answered, write answer short answer box answer a text from map, copy as it appears
Numerical Answers:
Include units indicated the map (Don't I convert 1200m 1.2km)
both map frame and ruler scale available, use ruler scale If question asks for an area, use {unit) 2
Use numerical values (e.g, 4 instead of four)
Directional Answers:
cardinal directions only: North, North East, East, South East, South, South West, West, North West
Write 'North' or 'South' before 'East' 'West'
Notice that the north arrow compass do not always point upward
Multi-Part Answers:
Separate with semicolon (:) (e.g. Zone A; Zone B)
Figure 7: Instruction provided annotators at the beginning of every question.
Answer the questions
following oriteria:
General:
question answered, answer
answez
If answer is text from the map, copy it as it appears
Numerical Answers:
Tnalude units as indi on the map (Don't convert 1200m to .2km1 available, If question asks for an aree, use [unit]
both map
values (e.g.r instead four)
Directional Answera:
Use cardinal directions only: North, North Bast, East, South East, South, South West, Weat, Norta Kest
'North'
'Soutl. before
'Reat'
Nctice that the north srrow compess do not always point upward
M:lti-Part Answers:
Separate with semicolon. (s) (e.g., Zone Ar Zone B) Sive the finel answer in 'Finel answer: (your answez)'
(Do no: use online seazch.
Figure 8: System prompt used for LVLM inference. For readability in the figure, newline characters ((n) are shown as actual line breaks.
20

---

Published as a conference paper at ICLR 2026
B.5 ANNOTATION PLATFORM
We built web interface (Figure to collect annotator responses. We provide the annotator instruc- tion (Figure 7 at the top of every question similar to how LVLMs receives the system instruction for cach question. For each question, annotators then (1) enter a short free-text answer it is considered answerable, (2) mark answerability by se- lecting either "Can be answered" ''Ma doesn't contain information answer the question" (the latter requires a brief justification), and (3) for multi-map questions, indicate whether all images are necessary to precisely answer the question without guessing.
the question and its associated map image(s),
B.6 LLM TO GENERATE QUESTIONS
We use GPT-4 and GPT-o with tailored prompt (Figure 10) to draft candidate questions for FRIEDA. In addition to the prompt, we supply 10 randomly selected map images for question generation. Two of the authors then manually review each candidate question, editing or discarding questions as needed to ensure correctness, clarity, and coverage of targeted spatial relations before adding them the benchmark.
C EXTENDED BENCHMARK DETAILS
This section provides expanded details on the distribution of questions within FRIEDA. To visualize the hierarchical nature of the task dimensions formalized in Section we present Sankey diagram (Figure ITI. Additionally, we provide granular breakdown counts for other dataset attributes, in- cluding question frequency per spatial relation, national representation, and domain diversity.
C.I QUESTION COUNT PER SPATIAL RELATION
In Table report the number of questions in FRIEDA by spatial relation, including totals as well as the counts split into single-map and multi-map questions. The distribution is roughly balanced: Within the largest class (23.0%), while Equal is the smallest (10.8%).
Spatial Relation
Total Count
Single-map Q Count
Multi-map Q Count
Border
71 (14.2%)
41 (8.2%)
30 (6.0%)
Distance
91 (18.2%)
42 (8.4%)
49 (9.8%)
Equal
(10.8%)
(0.0%)
54 (10.8%)
Intersect
80 (16%)
38 (7.6%)
42 (8.4%)
Orientation
89 (17.8%)
32 (6.4%)
57 (11.4%)
Within
115 (23.0%)
49 (9.8%)
65 (13.0%)
Table Question statistics in FRIEDA across six spatial relations. The table reports the total num- ber of questions per relation, along with their breakdown into multi-map and single-map settings.
C.2 EXAMPLE QUESTION PER SPATIAL RELATION
In Table we present one sample question for each spatial relation, split by map count (single-map vs. multi-map).
C.3 NATION AND DOMAIN COVERAGES
Nation Coverage FRIEDA includes maps from government documents and multilateral reports
from 32 countries across six continents (Figure 12 T)) We also report the ten most-represented countries by question count in Figure 13
Questions generated before the relcase of GPT-5.
21

---

Published as a conference paper at ICLR 2026
Map Question
Survey (Pields
Instructions
General
question can answered, write answer in short answer answer is text from the map, copy it as it appears
Numerical
Include units as indicated on the map (Don't convert 1200m to 1.2km) nuler scale
both map frame and ruler scale is available,
question asks for an area, use (unit}²
Use numerical values (e.g. 4 instead of four)
Directional Answers
Use 8 cardinal directions only. North, North East, East, South East, South, South West, West, North West
Write 'North' before
Notice that the north arrow compass do not always point upward
Multi-Part Answers
Separate with semicolon () (eg. Zone A; Zone B)
Q24/32: Each 'Dllinetted Wetland is labeled with unit code the form Ws. Which delineated wetland does the boundary of the 'Green Island Coastal Barrier Resource System' cross? Provide only the number.
Answer
Type your short answer here...
Validation
Can be answered
Map dosan't contain information to answer the question
Please briefly explain what information was missing (required when selecting this option):
e.g. The map shows roads only; no distances names to identify the feature.
This required only selecting "Map contain
question".
Are all images necessary to answer precisely without guessing?
Submit
Figure 9: Annotation interface for validating questions of FRIEDA.
22

---

Published as a conference paper at ICLR 2026
I'm trying create benchmark dataset test out generative Al's ability complex cartographical reasoning maps. The hard questions should provide in this benchmark should leverage information from one or a the given maps above, and should involve some reasoning. Also, questions should follow these criteria:
Answer should be self-contained, non-binary, and not-multiple choice questions. Question should not be solved by searching online We assume that the image refer known when answering question. We assume that the image to refer to is not known when answering the question.
Give of questions, the maps to refer and the answer the question. Target spatial relation is (Spatial Relation].
Figure 10: GPT-o3.
prompt used to prompt for candidate questions to either GPT-4 or
Within
Equal
Legend
Intersect
Mulh Map
Distance
Map Texi
Single Map
Orientation
Seale
Barder
Compass
Figure 11: Sankey diagram illustrating the hierarchical structure of FRIEDA's 's question taxonomy. Each question is annotated with its count maps (left), the spatial relation that defines the core reasoning objective (middle), and the specific map elements that must be interpreted to answer the question (right). The flow demonstrates how the dimensions interact practice, highlighting that FRIEDA's 's questions typically require reasoning across multiple cartographic primitives.
23

---

Published as a conference paper at ICLR 2026
Spatial Relation
Map
Question Example
Count
Single
Which DMMUs along the Inner Harbor Navigation Canal share boundary with 'DMMU 4'? Answer in the form DMMU #.
Border
Multi
Identify the 'National Road Imap1]' that crosses the 'Ou Ta Paong' area. Which two 'Irrigation Schemes [map2]" does this road serve as border for? Provide the names without the word 'Area'.
Single
What is the approximate straight-line distance between the SLC-6 Launch Site and the 2 psf contour of the Falcon Heavy Launch line?
Distance
Multi
In Tinian, each 'Heritiera longipetiolata" species observation area is associated with a name. Which "observation area[mapl] is located closest to the Noise Sensitive Receptor labeled mm15m1?
Equal
Multi
Which [map]]] on the infrastructure map corresponds to 'Existing Component 60 map2]" of the Santander project?
Single
How many Asanko tenement blocks does the Haul Road intersect?
Intersect
Multi
Which "claim block [mpp1] of the UEX Christie Lake Project are crossed by the 'power line 2
Single
What is the name of the northermmost 'National Air Monitoring Site' as recorded by Ordnance Survey Ireland?
Orientation
Multi
In the Lumberton Loop Project Arca, what is the orientation of the 'Crosswal
Stripping [map1]' in relation the "Walnut Street Componentmmap21"
Single
Along Pine Street and Pike Street, how many 'Future Redevelopment & Renovation Project' arcas overlap with the 'West Focus Arca'?
Within
Multi
Identify the area of Nighthawk Gold Property located North of the 'Winter Road[map1]] How many 'Gold meposits(map2] are located within this area?
Table 5: Example questions by spatial relation and map count. For multi-map questions, entities are annotated with [map]][[map) only for illustration, indicating different source maps; these tags are not part of the actual questions. We underline the word/phrase that denotes the target spatial relation.
FRIEDA Coveruge: 32 Countries
Global country coverage of FRIEDA. Countries included in the dataset are shown in pur- ple; countries where English is a primary or official working language but not covered by FRIEDA are shaded light gray. Coverage spans six continents (32 countries).
Figure 12:
Domain Coverage We source maps from domains where spatial reasoning is essential: geologic reports SEDAR+ 2025), national park management reports (National Park Service 2025). invest- ment and infrastructure reports (AЛl (025). disaster and hazard assessments (Federal Emergency Management Agency 2025), city and regional planning documents (Seattle Planning and Commu-
24

---

Published as a conference paper at ICLR 2026
Country
Count
Country
Count
United States
251
Mexico
18
Canada
82
Portugal
2
South Africa
32
New Zealand
1
Peru
9
Chile
4
Burkina Faso
1
Brazil
2
Guyana
2
Guinea
3
Ireland
24
Colombia
2
Seychelles
14
Ecuador
1
Singapore
9
Cuba
1
Kazakhstan
6
Argentina
3
Cambodia
Bolivia
2
India
Spain
Bangladesh
6
Sweden
Sri Lanka
3
Australia
United Arab Emirates
3
Namibia
2
Ghana
Nicaragua
Table 6: Country coverage in FRIEDA. Count reflects the number of questions whose maps origi- nate from each country.
Top-10 Countries by Question Count
United States
25
Canada
South Africa
-2
Ireland
Country
Mcxico
Scychelles
14
Singapore
Peru
India
Bangladesh
50
100
150
200
250
Questions
Figure 13: Top 10 countries by question Count
nity Development 2024 City of Cape Town 2025 Department of Community Development 2025 Urban Redevelopment Authority 2025), and environmental reviews (U.S. Environmental Protection A0en Ministry of Agriculture Climate Change and Environment 2025 Environmental Pro- tection Agency| 2025). Several of these are umbrella categories that can be further subdivided. For example, environmental assessments may target facilities, hydrology. land use/land cover, habi- tat. For consistency, we retain the top-level labels used by the source repositories. Across these domains, maps employ varied symbol conventions (legends, scale bars, north arrows) and hetero- geneous geometry types (areas, lines, points), encouraging generalization beyond any single map style. Figure 14/ summariis the domain coverage.
25

---

Published as a conference paper at ICLR 2026
FRIEDA Map Domain Distribution
Env invirrmmenall review
(100)
Geology report
(66
Planning (citg/region) (112)
Pack & trail map
Investmege infrastrcctur
(27)
Disaster/hazard
(83)
Figure 14: Domain distribution in FRIEDA by document category (e.g., geology, planning). Slices indicate categories, and parentheses denote question counts.
D EXAMPLES OF FRIEDA
We store each example as a JSON record containing the question, the gold answer, the required map image(s), any contextual image(s), and metadata such the number of maps, target spatial relation, and answer type. Figure iillustratss an example of a single-map question, and Figure shows an example of multi-map question.
"question.red":"e_1
; "What is the orientation of 'eryan Folmer & Harry Maust! in relation to Blocher! within Mayersdale Study Area?", " "imago.urls"
"ET5/VOL-3-FETSAPPP 20
]r
*spatial_rolattonchhipp'
anaver..cype"4"aardnnn
"contextusi_urrss:1
1055:000-9-PP5555ppmm0--------20 pmp.""
*BI8/Ve1 #EISAppendisa X 2025 202552meeeemmmeee11l...ggg
*BIS/Vol FEISAşpendixs May 2025/1mage10.1..0m
921s/Vo1-3-EE1AApeee 2xxxxy-2002//mmsgee22....p.
"Rorii" ewv,
"map.elements":
*map tex:",
*compass"
Figure 15: Example question single map
E DETAILED BENCHMARK RESULT AND ANALYSIS
E. LLM-AS-JUDGE PROMPT
To evaluate free-form textual answers, employ LLM-as-Judge (Gu al. 2025) using Mistral- Small-3.1 (Mistral Al 2024). Since not all models follow our requested output format ("Final answer: (your answer)") and minor wording differences may occur (e.g.. '15.0%% vs. '15'),
26

---

Published as a conference paper at ICLR 2026
"question_text"" "The Aberdeen-Roque Roquian North Shore Tevee is classiried into calegories. In which calegury is the 'llequiam Police Station' located?", "expected answer"; "Nrrth Shore Levee [Keat)", "image.urls"
}.
"map.count"; :
"ppatial.ee.etiinship.... "Intersectt,
"anawer.type"" Sxsswer.cyppe":MMexttaaa
"contextusi..rrss:::
11t.:ee.ecciaaaaadd2a2aaaaaaaa.. 1.ng""
0302.N8
*2 10-10-EX--2220---1100 http::/ww......aaaa///aaaaaaa-a.
"EEX//BICC EMS 2020 BR 102 10:000000..
202411.26//maeee331...gg
1.pg""
DEA 000041126///0
"domain": ""isaater/hazaare
mespallements": *legend"
Figure 16: Example question multi map
first require the LLM to extract the answer span based on the question and then compare the extracted portion to the gold answer with tolerance for minor variants (Figure 17
You be given a triple consisting question, an expected answer, and given response. Your task is to output either 'yes' or 'no'. Given the question and response, extract only the exact portion of the text that serves as the answer from the given response. Then output "yes" if the user response conveys the same meaning as the expected answer in relation to the question. Output 'no' if it does not. For questions with multiple correct answers, the expected answers separated by semicolons. The user response is correct if matches all required answers, regardless order. When provides more items than required, the response is treated as incorrect. If the user lists fewer items than expected, mark the response as incorrect. Differences in plurality, extra details such as acronyms counts, minor typographical errors, and differences in wording style do not affect correctness. Focus only whether the meaning matches.
user
Question: (Question}
Expected answer: (Expected Answer)
Given response: (User Response]
Does the response correctly answer the question based on the expected answer? Answer strictly 'yes' or "no"
Figure 17: The input prompt generate questions.
E.2 STATISTICAL SIGNIFICANCE OF FRIEDA-DIRECT RESULTS
As FRIEDA partitions questions into large number of fine-grained categories, some subsets con- tain relatively few examples (fewer than 100). In such cases, raw accuracy comparisons can be unreliable due limited sample size. To more rigorously assess whether observed performance differences within these smaller subcategories are statistically meaningful, we apply McNemar's test McNEMAR (1947) the top three proprietary models. We the exact binomial version of the test when the number of disagreements is small (< 50), and the s-suureed version with correc- tion when disagreements are larger (2 50). Table 7 reports the resulting p-values.
27

---

Published as a conference paper at ICLR 2026
Category
Gemini 2.5 vs. GPT-5
Gemini 2.5 Sonnet-4
GPT-5 Sonnet-4
Single-Map
0.03
0.01
1.00
Multi-Map
0.05
0.02
<0.01
Border
0.17
1.00"
0.10
Distance
0.80
0.84
0.54
Equal
0.10'
0.63'
0.21
Intersect
0,50
0.13
0.03
Orientation
0.82
0.02
0.08
within
0.11
<0.01
0.09
Table 7: p-values from pairwise McNemar's tests across key subcategories. Bold values indicate statistical significance at 0.05. An asterisk (") indicates that for each pair (A vs. B), model B
achieved a higher accuracy.
E.3 PERFORMANCE ON FRIEDA-CONTEXTUAL
Table reports overall and per-spatial relation performance for FRIEDA As noted in Section models show little difference between the FRIEDA-dires and FRIEDA-contextull settings. Figure 18 summarizes overall accuracy across models on
FRIEDA-Conttxtual
100
Proprictary
100B
-70B
-35B
=10B
All Agres
80
Agrce
Accuracy (%)
60
40
20
Gemimi-2255PP
GPT--- Think--Tiink Think
Clande- Chade---Soome Sennet
ALaVA-NeXT-I
GLMI GLAM-43V--1 1.31-108B
Inter 1Lmn11337788 3-7788
1.Ia Vaee--meeeeii Oneiion Vision- 72B
Qwren 5-V1-22
Inter Muum/L33.. 3.3.5:33333 5.338
Ovis2- 34B
Ovist Ovic2 S.9- Think
Models
Figure 18: Overall accuracy across models in the FRIEDA-contextull setting.
E.4 PERFORMANCE ALL-AGREE SUBSET
To validate that the performance gap reported in Section 5 is not an artifact of annotation noise, we evaluate models not only on the full dataset but also on the All-agree subset, where all three annotators unanimously agreed on the gold answer. Table 9 presents the results for both the full dataset and the All-agree subset, for both the direct and contextual setting of FRIEDA.
E.5 PER MAP COUNT & ANSWER TYPE RESULT BREAKDOWN
We also report performance by map count and answer type for FRIEDA (Table 10 and FRIEDA-contextual (Table II] In the FRIEDA-direct setting, GPT-5-Think leads multimmpp questions, outperforming the best-best model (Gemini-2.5 2.5-Pro by roughly 5%. Staude-Sonnel performs best Distance answers but underperforms directional (i.e., Orientation) questions.
28

---

Published as a conference paper at ICLR 2026
Overall (500)
Border (71)
Distance (91)
Equal (54)
Intersect (80)
Orientation (89)
Within (115)
Proprietary LVLMS
Gemini 2.5 Pro
36.60
28.17
29.67
50.00
21.25
64.77
30.17
GPT-5-Think
37.00
28.17
27.47
40.74
36.25
61.36
30.17
Claude Sonnet 4
28.40
19.72
19.78
27.78
23.75
55.68
23.28
Open Source LVLMs
LLaVA-NeXT-I10F
3.80
2.86
3.30
3.64
7.50
0.00
5.17
GLM-4,5V-108B
7.40
9.46
0.00
6.00
13.41
1.12
12.82
IntemV113-788
9.20
2.82
5.49
5.56
3.75
30.68
5.17
JJaVA-OneVisioon722
9.20
7.04
5.49
3.7
7.5
17.05
11.21
Qwen2.5-VI.-2BB
26.40
12.68
16.48
29.63
16.25
55.68
25.86
IntemVL3-78B
2.82
5.49
5.56
3.75
30.68
5.17
Intern 3
12.00
8,45
4.40
7.41
7.50
34.09
8.62
Ovis2-34B
16.00
21.13
14.29
18.52
18.75
2.27
21.55
Ovis2.5-9B-Think
24.80
18.31
9.89
27.78
21.25
56.82
17.24
Table 8: Performance of the 11 LVLMs across 6 spatial relationships on FRIEDA-contextual set-
ting. Values represent performance scores (in %) for each spatial relationship and the overall accu- racy.
FRIEDA Frrect-direct
FRIEDA-
Full (500)
All-agree (297)
Full (500)
All-agree (297)
Human Average
84.87
93.93'
Proprietary LVLMs
Gemini 2.5 Pro
38.20
46.13
33.06
15.56
GPT-S-Think
37.20
44.11
30.65
26.67
Claude Sonnet
31.60
24.26
25.81
28.89
Open Source LVLMs
LLaVA-NeXT-IIU 10BB
8.60
9.43
10.48
8.89
GL.M-.5V--08B
6.40
8.67
8.33
0.00
InterVVL.3-27
11.00
13.80
6.18
4.44
ULaVA-OOneVision-222
13.00
14.48
9.41
11.11
Swen2.5-VL-72
25.60
28,28
21.24
8.89
Intern VL3.5-38B
14.20
14.81
9.68
6.67
Ovis2-34B
17.80
20.54
22.58
11.11
Ovis2.5-9B-Thiim
25.80
29.97
20.43
20.00
NNoe: Altioough All-agree subser complate homan consensus ground truth, haman average 93. 33% than our evalaaion pipeline relies on as L-M-s--udge In other words, the 93.39% accurcc reflects the LLM Jadge's assessment of the haman--ronided answer on the All-agree semss not human disagreement.
Table 9: Performance of humans and 11 LVLMs on the All-agree subset for FRIEDA and
29

---

Published as a conference paper at ICLR 2026
Map Count
Answer Types
Overall
Single (202)
Multi
Textual (372)
Distance (45)
Direction
(500)
(298)
(83)
Human Average
84.87
84.91
88.08
87.93
67.18
92.15
Proprietary LVLMS
Gemini 2.5 Pro
38.20
32.67
41.95
33.06
15.56
73.49
GPT-5-Think
37.20
23.76
46.31
30.65
26.67
72.29
Claude Sonnet
31.60
24.26
36.58
25.81
28.89
59.04
Open Source LVLMs
ALaVA-NeXT-IIU
8.60
7.43
9.40
10.48
8.89
0.00
GLM-4.5V-108B
6.40
4.81
7.53
8,33
0.00
1.23
Intern VL3-78B
11.00
6.93
13.76
6.18
4.44
36.14
LLaVVA-OneVision-222
13.00
15.35
11.41
9.41
11.11
30.12
Qwen2.5-VL-72B
25.60
21.78
28.19
21.24
8.89
54.22
Intern
14.20
11.88
15.77
9.68
6.67
38.55
Ovis2-34B
17.80
17.33
18.12
22.58
11.11
0.00
Ovis2.5 Thinhkk
25.80
22.28
28.19
20.43
20.00
53.01
Table 10: Performance of humans and 11 LVLMs the map count types and three answer types on FRIEDA-direct.
Map Count
Answer Types
Overall (500)
Single (202)
Multi (298)
Textual (372)
Distance (45)
Direction (83)
Proprietary LVLMS
Gemini 2.5 Pro
36.60
25.25
44.30
31.99
17.78
67.47
GPT-5-Think
37.00
26.24
44.30
31.72
28.89
65.06
Claude Sonnet
28.40
20.30
33.89
23.12
17.78
57.83
Open Source LVLMS
LLaVA-NeXT-I108
3.80
1.99
5.03
4.85
2.22
0.00
GLM-4.5V-I0B
7.40
6.19
9.95
0.00
InternV13-77
9.20
6.93
10.74
4.57
4.44
32.53
LLaVA-Onee
9.20
7.43
10.40
7.53
6.67
18.07
Qwen2.5-VL-72B
26.40
18.32
31.88
21.24
11.11
57.83
Intern VL3.53L33
12.00
8.42
7.53
4.44
36.14
Ovis2-34p Ovis2.5-9B-Think
16.00
14.36
17.11
19.62
15.56
0.00
24.80
20.79
27.52
19.62
6.67
57.83
Table 11: Performance of the 11 LVLMs across the map count types and three answer types on
FRIEDA -ootte--ont.tuull.
30

---

Published as a conference paper at ICLR 2026
E.6 PER MAP ELEMENT & MAP ELEMENT COUNT RESULT BREAKDOWN
We analyze performance based the specific map elements required answer each question, as well as the number distinct element types involved, for both FRIEDD--direct (Table (2) and FRIEDA--ontexxn (Table As map elements require interpreting multiple elements simultaneously to produce correct answer.
mutually exclusive, single question may
Humans outperform every model by large margin across all four map elements. Accuracy is high- est when only one or two elements are required, but drops substantially when four elements must be combined, indicating that even expert map readers experience increased difficulty as compositional complexity grows. On the other hand, the best proprietary model performance occurs at three ele- ments; this may be because questions involving multiple components compel the model to search the image to identify relevant elements.
Map Element Type
Map Element Count
Map text (366)
Legend (417)
Compass
Scale (46)
2
(137)
(132)
(279)
(80)
(9)
Human Average
80.97
83.61
75.91
63.78
84.09
81.84
80.00
51.85
Proprietary LVLMs
Gemini @ei-5-Think
38,80
37.41
56.20
35.61
35.13
55.00
22.22
38,52
34.05
53.28
28.26
36,36
34.41
48.75
33.33
Claude Sonnet
31.69
31.41
51.83
30.43
24.24
29.75
47.50
55.56
Open Source LVLMs
LLaVA-NeXT-I100
7.38
0.73
8.70
14.39
8.24
0.00
11.11
GLM-4.5V-08B
5.72
5.74
3.62
0.00
12.12
5.00
2.47
0.00
Intern VIe-7VL
9.84
10.31
23.36
4.35
9.85
9.32
20.00
0.00
LAAVA-Oneiision7222
13.39
11.27
20.44
10.87
10.61
13.62
16.25
0.00
Qwen2.5-VL-72B
26.23
24.46
40.88
10.87
21.97
24.37
37.50
11.11
IntermVL3.5-38B
14.48
12.23
25.55
6.52
12.12
14.34
17.50
11.11
Ovis2-34B Ovis2.5-9B-Thinn
16.12
18.47
4.38
10.87
27.27
17.20
6.25
0.00
26.78
23.26
38.69
21.74
23.48
24.73
33.75
22.22
Table 12: Performance of humans and 11 LVLMs across the map element types and count of map elements on FRIEDA-direct.
Map Element Type
Map Element Count
Map text (366)
Legend (417)
Compass (137)
Scale (46)
2
3
4
(132)
(279)
(80)
Proprietary LVLMs
Gemini 2.5 Pro GPT-5-Think
37.43
35.49
50.36
19.57
34.85
34.77
46.25
33.33
38.25
34.53
51.09
30.43
34,09
35.48
48.75
22.22
Claude Sonnet 4
29.78
26.38
43.07
19.57
25.00
27.24
37.50
33.33
Open Source LVLMs
LLaVA-NeXT-1108
3.01
4.32
0.00
2.17
6.06
3.94
0.00
0.00
GLM-4.V-1088
5.93
6.75
4.67
0.00
10.95
6.13
3.33
0.00
Intern
8.47
8.39
20.44
4.35
6.06
9.32
15.00
0.00
LLaVA-OneVision-22
7.65
13.87
6.52
9.85
8.96
8.75
11.11
Dwen2.5-VL-72.
27.32
24.46
41.61
13.04
21.97
26.88
32.50
22.22
Intern VL3.5-3
13.39
10.31
23.36
4.35
6.06
13.62
17.50
0.00
Ovis2-34B Ovis2.5-9B-Think 26.23
13.93
16.31
5.11
15.22
25,00
14.70
7.50
0.00
22.30
40.15
8.70
21.97
24.01
33.75
11.11
Table 13: Performance of the 11 LVLMs across the map element types and count of map elements on conteetalll...
31

---

Published as a conference paper at ICLR 2026
E.7 PER DOMAIN RESULT BREAKDOWN
In addition, report performance by domain for FRIEDA-direct (Table 14 and FRIEDA- contextual (Table (5). The domain can serve as an indicator of map style heterogeneity. For ex- ample, reports from park maps (labeled "Parks" in the table) and disaster reports typically follow the same formalized format because they are produced by the same source (usually the government). In contrast, reports from the investment and infrastructure domain (labeled "Investment") and the geology domain (labeled "Geology") originate from various sources, as they are usually authored by different companies, resulting in more diverse map styles.
Overall
Planning (112)
Investment
Environment (100)
Disaster (83)
Parks
Geology
(500)
(27)
(22)
(166)
Human Average
84.87
86.60
88.89
82.33
83.13
75.76
76.91
Proprietary LVLMS
Gemini 2.5 Pro
38.20
37.25
33.33
43.00
49.40
45.45
30.12
GPT-5-Think
37.20
35.29
25.93
40.00
54.22
68.18
25.90
Claude Sonnet
31.60
33.33
22.22
28.00
42.17
50.00
26.51
Open Source LVLMs
-LaVA-NeXT-110F
8.60
9.80
18.52
6.00
8,43
13.64
7.23
GLM-4.5V-108P
6.40
3.92
0.00
5.00
8.43
9.09
8.33
Intern VL3-7713378B
11.00
12.75
7.41
12.00
16.87
22.73
5.42
13.00
16.67
11.11
11.00
7.23
22.73
13.86
Qwen2.5-VL-72B
25.60
29.41
18.52
21.00
34.94
22.73
22.89
InternVL3.5-38B
14.20
13.73
22.22
15.00
18.07
18.18
10.24
Ovis2-34B
17.80
18.63
14.81
19.00
21.69
22.73
14.46
Ovis2.5-9B-Thing
25.80
21.57
22.22
23.00
33.73
40.91
24.70
Table 14: Performance of humans and 11 LVLMs across the seven domain types on FRIEDA-direct.
Overall (500)
Planning (112)
Investment (27)
Environment (100)
Disaster (83)
Parks (22)
Geology (166)
Proprietary LVLMS
Gemini 2.5 Pro -hT-5--Thikk
36.60
39.22
40.74
34.00
50.60
50.00
27.11
37.00
36.27
25.93
34.00
49.40
72.73
30.12
Claude Sonnet 4
28.40
30.39
25.93
24.00
42.17
45.45
21.08
Open Source LVLMs
LLaVA-NeXT-110F
3.80
14.81
3.00
7.23
4.55
0.60
GLM--.5V-108B
7.40
5.61
0.00
5.66
10.23
0.00
8.29
Intern Intern VL3-78B
9.20
8.82
7.41
12.00
13.25
18.18
4.82
LLaVA-OneVisioon OneVision-72B
9.20
12.75
14.81
9.00
10.84
18.18
4.22
Twen2.5-VL-722
26.40
25.49
37.04
20.00
44.58
36.36
18.67
Intern VL.3.5-888
12.00
10.78
14.81
13.00
15.66
27.27
7.83
Ovis2-34B
16.00
16.67
25.93
12.00
22.89
18.18
12.65
Ovis2.5-9B-TThe -hin-
24.80
20.59
18.52
25.00
36.14
36.36
21.08
Table 15: Performance of the 11 LVLMs across the seven domain types on FRIEDA-contextull
F EXTENDED ANALYSES
EXAMPLES OF EACH ERROR CATEGORY
We illustrate the three most frequent error categories for Gemini-2.5-Pra and show each example alongside answers and reasoning from Gemini-2.5-Pro, GPT-5-Think, and --auee-Sonnet-4.
32

---

Published as a conference paper at ICLR 2026
InternVL3.5 Overall Accuracy vs. Model Size
25
23.2
Overall Accuracy (%)
Model Size
Accuracy (%)
23.0
20
IB
9.40
20.0
2B
12.80
4B
20.00
15
8B
23.20
14.2
14B
23.00
10-
30BA3B
24.20
38B
14.20
BS11
3088
241
241BA28B
11.40
Parameters (B)
Table 16: IntemVV33.5 perfor- mance by size
Figure 19: Performance of Intern VL3.5 by model parameter size
Misinterpretation of legend Listing | presents case where the model fails to map a legend symbol or color to its intended semantic class, leading to the selection of the wrong feature despite the correct evidence being present.
Cross-map interpretation failure Listing shows failure that arises when reasoning requires aligning information across multiple maps or overlays; the model identifies the wrong subject when the maps must be cross-referenced.
Spatial-relation semantics error Listing 3 illustrates a case where the model misinterprets the key spatial relation, yielding incorrect answer.
F.2 ANALYSES ON MODEL SIZR
In the main evaluation (Figure 3) the results deviate from the usual scaling law (Kaplan et al. 2020), which states that the performance of the model improves with size. Among open-source models, LLa despite having the most parameters, ranks near the bottom, whereas Ovis-2.5-9BB the smallest model, ranks near the top. We, therefore, hypothesize that cartographic reasoning is not an emergent ability (i.e., capability absent in smaller models but present larger ones). To test this, we evaluate the InternVL3. family (Wang et al. 20255) on FRIEDA: IB, 2B, 4B, 8B, 14B, 30BA3B, 38B, 241BA28B where 'A' denotes parameters active at inference. The trend (Figure Table 16) shows modest gains up to roughly 30B parameters, followed by degradation thereafter.
G EXTENDED RELATED WORKS
To provide a comprehensive context for FRIEDA, we detail the scope related benchmarks across three arcas: general document and infographics understanding, map visual question answering, and broader spatial reasoning.
G.1 DOCUMENT AND INFOGRAPHICS VQA
Benchmarks this domain have established baselines for LVLM reasoning over structured text and graphical images, including charts. In the document domain, Doc (Mathew et al.) 202T) intro- duces large-scale question-answering dataset over real forms and reports. DocV XQA (Souibgui et al. 2025) builds upon the benchmark to design self- self-explanatory framework that produces in- terpretable rationales for LVLMs. Docopilot (Duan et al. 2025) evaluates LVLMs on scientific articles, which not only test text understanding but also the interpretation of embedded figures such as charts. For graphics, Infographics VQA (Mathew et al. 2022) tests joint reasoning over text, lay- out, and pictorial elements in visually rich infographics. InfoChartQA (Lin al. 2025) extends this by pairing plain charts and infographics and identifying design elements that degrade LVLM per- formance. In general, VQA evaluation on frontier LVLMS reveals consistent trend: competence
33

---

Published as a conference paper at ICLR 2026
high-level patterns, such identifying trends in the data or the extrema, but struggles with precise value extraction and robustness. FRIEDA evaluates these shortcomings in a cartographic setting, where layout, symbols, legends, scales, and compass orientation interact tightly to measure how well LVLMs integrate these signals to answer map-based questions.
G.2 MAP VQA
Research in map understanding can be categorized into general map VQA, navigation-centered rea- soning, and domain-specific question-answering
General Map VQA MapQA (Chang et al. 2022) establishes a baseline for choropleth map un- derstanding by creating question-answer pairs targeting value retrieval and region identification. However, the dataset is limited to a single map type (i.e., choropleth maps) and geographically re- stricted the United States, thereby limiting the diversity of cartographic styles and toponyms. MapWise (Mukhopadhyay et al., (025) broadens geographic coverage to the United States, India, and China and introduces diverse question templates for probing relative spatial relationships; yet. still relies solely on choropleth maps and remains constrained to single-map reasoning, which lim- its its ability to model real-world cartographic complexity. MapIQ (Srivastava et al. (025) further advances visualization literacy by introducing cartograms and proportional-symool maps, which are commonly used in analytical tasks. While the expanded map diversity valuable, MapIQ's s maps generated using map-coloring tools rather than drawn from heterogeneous, noisy wolldw doc- uments. In contrast, FRIEDA explicitly captures this real-world variability by sourcing map images directly from government and scientific reports.
Navigation--entereed Reasoning Benchmarks centered on navigation often require more complex reasoning than simple semantic retrieval, yet they tend to remain domain-narrow, narrow. MapEval (Dihan et al. 2025) evaluates LVLMs' geospatial reasoning through multiple-choice travel-planning ques- tions spanning 180 cities. Still, it relies on standard web basemaps (e.g., Google Maps) whose clean, uniform designs lack the layered, domain-specific symbology (e.g., variable legends, irregular pro- jections, and customized north arrows) often found in professional cartography. ReasonMap (Feng iet al. 2025) moves beyond basemaps by using high-resolution transit maps and designing naviga- tion tasks that closely simulate real-world subway routing, though its scope is restricted to transit systems. MapBench (Xing et al., (025) evaluates LVLMs' spatial reasoning and chain-of-thouu inference by testing outdoor navigation performance on diverse map types, such as park and trail maps. Despite their contributions, all of these benchmarks remain focused on navigation-centric tasks. In contrast, our benchmark generalizes spatial reasoning across six distinct spatial relations that extend well beyond navigation, capturing the broader landscape of cartographic reasoning re- quired in professional and scientific contexts.
Domain- and Task-specific QA Specialized benchmarks address domain-specific needs or par- ticular visual modalities, but they tend to trade breadth for depth. PEACE (Huang al., 2025) introduces a benchmark focused on geologic map understanding and develops a framework for an- swering domain-specific questions, such as identifying lithologic units, fault lines, and structural patterns. While the benchmark and the approach is highly effective for geology-speciff evalua- tion, the scope is limited single scientific domain, and it lacks the thematic diversity required for broader cartographic reasoning. CartoMark (Zhou et al. (024) provides wide range of maps across various styles, but its core task centers on simple pattern recognition, such scene classification and text annotation. These tasks primarily test perceptual recognition and, in many cases, do not require reasoning at all. ReMI (Kazemi et al.) (025)) offers framework for multi-image reasoning that eval- uates how models integrate and compare information across visual inputs. However, ReMI operates on natural images and uses simple web-based maps. Therefore, it does not assess the specialized challenges of multi-map cartographic reasoning, such as aligning heterogeneous legends, reconcil- ing differing spatial scales, and interpreting mismatched orientations across maps. These capabilities form the core of FRIEDA's 's multi-map setting, which reflects real-world analytical scenarios where experts must synthesize information from multiple, heterogeneous cartographic sources.
To situate FRIEDA within the broader landscape of MapVQA benchmarks, we provide compara- tive summary of existing works in Table [7] The table evaluates each dataset along four key dimen- sions: (1) the types of spatial abilities evaluated, (2) diversity of map elements (measured through
34

---

Published as a conference paper at ICLR 2026
country and domain coverage), (3) whether multi-map reasoning is supported, and (4) whether contextual setting is included to emulate real-world map-use scenarios.
We use orange checkmarks (V) to indicate partial limited coverage within category. For ample, in the topological relation category, we treat questions such as "how many points lie along the route to location A? partially covering topological relation such tasks contain the notion of intersect while it does not examine the relation with the depth or rigor as required FRIEDA. Overall, the comparison highlights that prior Map VaA benchmarks tend emphasize narrow task settings, limited spatial relations, or constrained map vide comprehensive, cross-domain evaluation that reflects the complexity of real-world cartographic reasoning.
styles, whereas FRIEDA is designed to pro-
Spotial Relation
Heterogoncity
Topological
Directionl
Country
Contextual
Chane
2022
X
Wap My Masiopplyya LY
2025
X
Mapiq
Saystig 24
200
X
X
MapBence
4025
UNK
Mapkyal
RcMi
Kazein al
4015
V
1007
PEACE Tiiin 21
2005
V
(Feig
FRIEDA
/
Note: RaMi Kazemi et
205
reports counts by city, not by country: consqueetlyy the commspondnng country total is less Chan 100.
Table 17: A comparison of FRIEDA with prior map VQA benchmarks. FRIEDA covers broader set of map-reading abilities and exhibits greater geographic and thematic diversity.
G.3 SPATIAL REASONING
Spatial reasoning benchmarks have advanced model capabilities in perception and localization, though often outside the cartographic domain. Benchmarks such SpatialVI.M (Chen et al. 2024a) and SpatialRGPT (Cheng et al., 2024) focus on natural images, testing model's ability to reason about 2D and 3D spatial relationships, relative positions, and object dimensions in photographic scenes. In the geospatial domain, GeoChain (Yerramilli et al. 20225) enhances tasks such as ge- olocation by inducing step-by-step geographic reasoning link visual cues to geographic entities. However, these works primarily rely on natural scene understanding or semantic knowledge re- trieval and not engage with the abstract symbolic conventions unique to maps. FRIEDA closes this gap by evaluating multi-step cartographic reasoning, in which models must not only perceive space also decode specific symbolic rules infer topological, metric, and directional relations across heterogeneous real-world maps.
H THE USE OF LARGE LANGUAGE MODELS
We acknowledge the use of large language models (LLMs) for benchmark question curation, vision, and polishing of this paper. The details of usage, the exact prompt used, and all related information provided in the main paper appendices. All questions created with the assis- tance large language model have been verified and modified by the authors. The paper's 's main contribution remains with the authors.
35

---

Published as a conference paper at ICLR 2026
Listing 1:
Legend misinterpretation example Gemini-2.5-Par on FRIEDA. Other models are
shown for reference.
Orange Task Instruction.
Green Correct Answer.
Red Incorrect Answer.
eystem:
Anawer the questiona based the following criterla:
General:
answered, write in short box If answer is a text from the map, copy it as it appears
if question
Acal Anawers:
Tnclude units as indicsted the map (Don't convert 1200m 1.2km] If both Trame and ruler scale ls available, the ruler acale
IE question asks for an ares, use (unit)"2
numerical values (e.g.r 4 instead of four)
Doretiooaal Answers:
Use 8 cardinal directions only: North, North East, East, South Dast, South, South West, West, North West
Nrite 'oorth' or "South" before 'East' West
Notice that the north srzew compass da not always point upward
Mo Multi-Part Answers:
Separale semicolon (F) (e.g.,
Give the final answer in 'Final answer: syour answers!
Do not online search.
imagen:
Teemen legh
Daby
(riff))
Questiion: The 23rd Ave On St Rezone (Draft) claast rles into four oi inot Rezone With Bezone
'Rezone 4' its vesternmest boundary. the name of the area in form Rezona Area #.
Gold Answer: Kezone Area
6
Gemini 2.5 Pros
of "Rezone analyzed mop decenatrucsed the task. "Rezone Area the bluc-hatched area. Its western border is zlong 22nd Ave. I've identified the adjacent pink the legend, the pink/magenta area corsesponds to "Rezone Area 5*, Finsl answer: Bezone Area
desermined user's objective:
"Bezone
according to
GPT-S-Think: The smage shows soning map with areas labeled 4-7, and the title mentions "23 (DzafL)
about Rezone Area 4'a westerrmost boundary. I'll need to find Area 4 on the map and check which srea lies dizectly to the west it, likely along labeled in light blue, ia near the left margin, adjacent to *NC1-30* and *MC1-49." It seema that Area a westernnost boundary touches Area 5, which is dizectly to the left of Area 4, So Rezane Area
Cherry
question
ave. The map shows colored boxes
Areas
4,
suspect 5 shares this boundary. Final
claude-4-Sonned Looking the map, 1 can identify sezone Area by its blue diagonal lins pattern. This area is located in the wastern portion of the map. Examining the magenta/pink colored azea, which according to the legend cozresponds to Bezone Area 6. Pinal answer: Bezone
boundary of Rezone
cirectly borders
36

---

Published as a conference paper at ICLR 2026
Listing Cross-map interpretation failure example of Gemini-2.5-Pro on FRIEDA, Other models
are shown for reference. Orange Answer.
Task Instruction. Green Correct Answer.
Red
Incorrect
Systimm:
Answer the questions based following criteria:
Genersl:
quastion can answered, write answer short answer box if answer is from the map, copy
appears
Numer Answera:
Include unils as indicated on the map (Don't convert 1200m to 1. 2kmi is available, the If quastion zaka for an area, use (unit]"2
map
and
Use numerical values (e.g.r instead four)
50 rectioral Answars:
cardinal direct Lons only: North,
East, East, South East, South, South
Wess, North West
'North' 'south' before 'east'
Notica that the north arrow compass do not always point upward
Multi-Part Answers:
Separate with semt colon (t). (e.g., zone A Zone 5)
Give the final answer in 'inaal answer: eyour answers!
not use online search.
Tmages:
Mankooing LOCK
Localonn
COMS
COMS
Question: Identify the Quality Monitoring location closest to the Noise Level Monitoring Location 'NL 5", What the crientation of this point in relation 'Nu.
Gold Answer: North East
Comini-2, 22.5- S-Pro: I've determined that the closast point NL 5, Now, analyzing their relative orientation. It oppears that from NL 5, AQ liad in mest-northwest: direction Final answer: AQ 6; West
Locking a: maps, seems nearest to NI5, based on their locatiens along the river. AgS seems be slightly northeast of NLS, just bis
of it. answer the quastion, it makes sense to say AQS is closent to 8L5. As for the orientation, think Ag5 The triangle
rive", south of the ziver bend, while NLS is to the sorth, I'll go with: AQS lies southeast of NLS. This seems the most aceurate ccncluston. Pinal answer: Ag 5; sout heast.
liea slightly southeast
located
clande-4-sonnet- Looking both maps, need locate Locations Map and find the closest AQ location on on the Ambient Air Quality Monitoring tocations Map. NL la located near the Rarnatali River area. Examining the AQ locationa on the firat map:
the Noise Level Mcritoring
AQ 5 is positioned very close to the same Kernateli siver area slightly further
- 2, AQ 3, AQ d are further south
nerch
Ag 5 is clearly the closest Ambient Air Qua ity Nonitoring Location NL. 5.
Compasing the relative pesitiens AQ 5 and 5, AQ appears to be positioned to the east of NL 5, Final
AQ 5; Bast*)
37

---

Published as a conference paper at ICLR 2026
Listing 3: Spatial-relation semantics error example of Gemini-2.5-Pro on FRIEDA. Other models are shown for reference. Orange Answer.
Task Instruction. Green
Correct Answer.
Red
Incorrect
System:
Anawer the questions based on the tollowing criteria:
if question can be answered, write answer in short answer box enswer is a Lext from the may, copy it as it appears
Numerical Answers:
(Din't convert 1200m 1.2km] If both map Trame and rulez scale is available, use the zuler acale area, use (unit) Use numerical values (e.g-r 4 instead of four)
Tncllue
indicated
question
Directional Answers:
Use cardinal directions only: North, North East, East, South East, South, South West, West, North
Nrite 'Horth' or 'Bouth' before 'East' or West'
arzew compass always point upward
Multi-Part Answers:
Separate with semicolon (1) (e.g.,
'Final answer: eyour Do not use online search.
Imagas:
BERK
Question: Which Natiional Registe: Districts' overlap with both the "Ballard' and "Itterbay Dravus' Industrial Lands Subareas within the Boliard-interray MIC?
Gold Answer: Washington Ship Canal
currently focused dissecting usar's complex quary. My goal is determine the exact reqiens where the yellow "National Register Districts" overlap with both the purple "Bcllard" and orange "Interbay Dravus* subareas, all contained within the "Sallard= map, T've broken down request irto map elementa and used the legend to visualize the features, I've concluded that the only district Washington Ship Canal." Pinal answer:
Gamini-2 -0mint-2.5-Pro S-Pros
Tnterbay MTC." T'm visualizing
overlay
meeting criteria Chittenden Locks and Lake washingten Ship Canal
"Chittenden
GPT 11-5-Thinkk
National Register Interoay Draves" subaress within the Ballard-Interbay MTC. I've got the map, and think the overlapping yellow polygens trepresenting districts) fall within both the Ballard (pink?) and Interbay Dravus (green?) subereas. I'll check for yellow dots (propertiesi and orange polygons area,
need figure
everlap
'sallard'
Miittori Dist:ict. The "Mashington Ship Cansi" Nazional Register District seems to be the most likely answer, as follows canal overlaps
idistricts), Locusing
Ballard
subareas.
answer: Washington ship tanal
map, need 1dartify National Register Matriats (shown orange) that everlap with both the sallard industrial Lands Subarea (shown purplel and the Interbay Dravus Industrial Lands Subarea (shown in orangs/peach). After careful examiratior map,
claude-4-Sonneti Tooking
Register 3istricts that overlap with and Interbay Dravus Industrisl Lands Subareas within the Ballard Inte:bay Final anewe::
Ballard
38


Reviews:
[{"review_id": "N4HYzgH9mm", "reviewer": "Reviewer_8fK6", "summary": "This paper introduces FRIEDA, a benchmark designed to evaluate multi-step cartographic reasoning in vision-language models (VLMs). FRIEDA focuses on reasoning over real-world maps and includes both single-map and multi-map questions.\nThe benchmark contains 500 questions and supports two evaluation modes: (1) the direct setting, where the relevant map(s) are provided, and (2) the contextual setting, where models must first identify the relevant map(s) from a document set before reasoning.\n\nThe authors evaluated 11 VLMs, finding that even the strongest models achieve less than 40% accuracy, far below the human baseline of 84.9%.", "questions": "Besides the points listed under Weaknesses, I have a few clarification questions.\n\n1. For the multi-map questions, how many maps are provided on average in the direct and contextual settings?\n\n2. In the contextual setting, do the VLMs see the entire document or only the map images when answering a question?\n\n3. How does the contextual setting compare in difficulty for humans? Are these questions also easy for human annotators?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.The paper addresses the need for a benchmark that evaluates VLMs’ reasoning abilities over diverse maps. In addition to direct map-based question answering, it includes a retrieval-like setting, where the model must first identify useful maps before answering.\n\n2.All questions are verified by human annotators, ensuring high dataset quality. The appendix provides clear documentation of the annotation and curation processes, which helps readers understand how the dataset was built.\n\n3.The paper is well-written and easy to follow.", "weaknesses": "1. With only 500 questions, the dataset may be too small for robust model evaluation, especially when analyzing performance across subcategories (e.g., Table 2 and Figure 4). Can you conduct significance tests on the small categories?\n\n2. The paper asserts that all questions require visual reasoning, but no text-only baseline (question without the map) is reported. This makes it unclear whether VLMs rely on internal knowledge to answer some questions.\n\n3. The questions are synthesized by LLMs given maps, which may make them too artificial and not well aligned with how humans naturally seek information from maps. The textual context of the original documents could better reflect real human interest in these maps but is not utilized. Beyond verifying the validity of these questions, human annotators should also analyze whether the questions resemble those naturally encountered in real-world settings.\n\n4. It is unclear whether the questions contain explicit spatial references (e.g., “the red dots in the image”) or rely only on abstract factual references such as legend words. Prior work [1] suggests that VLMs struggle with understanding visual features, which is particularly important in map interpretation.\n\n5. The contextual setting experiment setup is not clearly explained.\n\nIn general, I believe this paper is well motivated and well executed, though it still has a few concerns that need to be addressed. I would be happy to adjust my scores after seeing the authors’ response.\n\n[1] Rahmanzadehgervi, Pooyan, et al. \"Vision language models are blind.\" Proceedings of the Asian Conference on Computer Vision. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "doUjt0h2b6", "reviewer": "Reviewer_VsEU", "summary": "This paper proposes a multi-step cartographic reasoning benchmark dataset for cartographic reasoning in 3 dimensions: topological, metric, and directional spatial relationships. The maps are sourced from publicly available dataset, 500 questions are drafted by propriety models. Evaluation was performed across 11 LVLMs, providing 5 findings.", "questions": "I am skeptical towards accepting adjacent labels. Also, isn’t it ambiguous to include partially-agreed questions in the benchmark dataset? Does the findings change if we do not consider those partial-agree items?\n\nHow did the authors analyze the errors and misinterpretations of Gemini-Pro? Are those manually checked following the reasoning trace?\n\nIn the “Performance by spatial relation” paragraph, are there any reasons why the models are ranked as in Fig 4? Does the amount or ratio of spatial understanding data in each train split matters?\n\nHow do RL-trained models perform on this benchmark? Are there any inductive bias observed from visual encoders? Are there any underlying biases or trends from the training set? These points are revealed that these two points matter [1,2].\n\nWill there be an explanation for the distribution of questions for multi-map querying i.e how many maps does one question have? \n\nReferences\n[1] Wang et al. VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Models. arxiv:2406.16338\n[2] Li et al. Vidhalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understanding. CVPR 2025.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The study does well to highlight the gap in research across all the other Map VQA studies. For example, it highlights that multi-map VQA is an area not evaluated, and if it is evaluated, it’s very limited (Kazemi et al, 2025).\n\nThe study does a good job linking the human cognition aspect of map querying and exploring how it would be interesting to see how LVLMs reason over maps. A strong real world foundation is provided in this study and it drives the need for the benchmark presented. \n\nEvaluation metrics are clear and fitting for each category of questions tested. LLM as a judge is used for textual questions while distance based answers are evaluated on MAPE. Directional answers are mapped to the relevant locations labeled with North, South, East, West metrics and all directions in between. \n\nAlthough the queries are drafted by proprietary models, they are manually reviewed by humans to maintain the quality. Provides logical findings through extensive evaluations and analyses, found a huge gap between humans and models. Also, the overall writing is good and easy to follow.", "weaknesses": "The motivation of this paper is weak. I am not convinced that cartographic QA is different from the other MapQAs, even after reading the whole body of Section 2. Further comparison between related works should be highlighted to demonstrate the novelty of this research direction.\n\nThe scale of the benchmark set is moderately small (500 questions), compared to the other LVLM benchmarks. A very limited novelty is there. Doesn't compare with vast literature of prior work on MapQA[1], MapWise[2], MapIQ [3], MapBench[4], CartoMark[5], MapQA (GQA)[6] and many more. A proper comparison should be there with prior works to show how this is different.\n\nThe task definition introduces too many dimensions for each category. In Spatial Reasoning, many dimensions to the category are formalized, giving the impression that the benchmark would extensively test against it. Figure 2 and Table 1 don’t align with the in-depth explanation for the tasks formalized in section 2, so there is a disconnect with how each dimension is tested. \n\nIn the benchmark section, 500 questions are evaluated against 17,030 maps from 210 documents. Because it’s a multi-map task, the map-per-question distribution may be uneven; while the authors note that each question uses “two or more maps,” they don’t state a maximum. That matters: LVLM performance can differ wildly—two-map queries are manageable for state-of-the-art models, but queries involving hundreds of maps will strain context windows (and humans).\n\nLimiting the scope to Latin-script text is concerning, and demonstration in Figure 1 does not look like it is from Latin-script text. Also, does it mean that all components including legends, region descriptions are translated or retrieved from references?\n\nReferences [1] https://arxiv.org/abs/2211.08545 [2] https://arxiv.org/pdf/2409.00255v1 [3] https://arxiv.org/abs/2507.11625 [4] https://arxiv.org/pdf/2503.14607 [5] https://www.nature.com/articles/s41597-024-04057-7 [6] https://arxiv.org/pdf/2503.07871", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QE8y2zA5jj", "reviewer": "Reviewer_RzLv", "summary": "The paper introduces FRIEDA, a new benchmark designed to test how well large vision-language models (LVLMs) perform multi-step reasoning over real maps. The benchmark spans 500 questions drawn from 17k maps across 210 documents, covering topological, metric, and directional spatial relations. Tasks require understanding legends, scales, and compass orientation, often across multiple maps. The study evaluates 11 LVLMs (Gemini-2.5-Pro, GPT-5-Think, Claude-Sonnet-4, and several open-source models). Even the best systems achieve under 40% accuracy compared to 85% human performance, revealing a major gap in cartographic reasoning.", "questions": "1. How reproducible are results across different LLM-judge configurations?\n2. Did you evaluate whether models can explain their reasoning steps (not just final answers)?\n3. Could GeoChain-style supervised reasoning traces help improve performance on FRIEDA?\n4. Are some relation types (e.g., metric vs. topological) more prone to hallucination?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: The paper targets a truly underexplored domain. Most prior map-VQA datasets treat maps as static charts. FRIEDA is the first to combine multi-map reasoning with explicit tests of spatial relations, bridging cognitive geography and LVLM evaluation.\n\nQuality: The benchmark is carefully built with validated questions, human agreement checks, and diverse sources. The experimental setup is solid, and the error taxonomy is informative.\n\nClarity: The writing is clear, structured, and well-motivated. The examples and figures make the task definition easy to grasp.\n\nSignificance: The benchmark fills an important gap between general multimodal reasoning and geospatial intelligence. The authors’ detailed error analysis provides actionable insight into how LVLMs misinterpret legends, scales, and orientation.", "weaknesses": "1. The dataset size (500 questions) may limit statistical robustness and fine-grained analysis.\n2. While the benchmark claims multi-step reasoning, most reasoning is still visual-symbolic.\n3. More discussion on relation to existing reasoning datasets like GeoChain or MapIQ would clarify complementarity and citation positioning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ekNMrgjXuu", "reviewer": "Reviewer_Y3WZ", "summary": "This paper introduces FRIEDA, a new benchmark designed to evaluate multi-step cartographic reasoning in LVLMs. FRIEDA targets the full spectrum of spatial reasoning, topological (border, equal, intersect, within), metric (distance), and directional (orientation), using real-world maps drawn from public documents such as geological surveys, planning reports, and environmental studies. The benchmark includes 500 questions and 17,030 map images, testing models under two settings: a direct mode, where relevant maps are provided, and a contextual mode, where the model must identify the correct maps before reasoning. Results across 11 state-of-the-art LVLMs show that even the strongest systems (Gemini-2.5-Pro and GPT-5-Think) achieve less than 40% accuracy, far below human performance. The study highlights persistent challenges in spatial and multi-map reasoning, offering FRIEDA as a resource to advance geographic intelligence in LVLMs.", "questions": "1. Could the authors clarify in more detail how the ground-truth answers were finalized during dataset construction? Specifically, when GPT-generated “gold answers” disagreed with human annotators, how were such conflicts resolved, and were any systematic patterns of disagreement observed?\n\n2. For the contextual setting, could the authors elaborate on the retrieval challenge? It would be helpful to understand how many candidate maps were typically presented per question and how similar or visually confounding these maps were. This clarification would help assess whether the contextual setup truly captures document-level reasoning complexity.\n\n3. The paper mentions that the benchmark draws from multiple thematic domains (geology, environment, urban planning, etc.). Did the authors observe domain-specific variations in model performance? If not, would they consider including such an analysis to better understand where models struggle the most?\n\n4. Could the authors comment on how they plan to ensure community adoption of the benchmark—e.g., through public leaderboards, evaluation APIs,\n\nor standardized prompts—once anonymity is lifted", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper presents a well-motivated and timely contribution that fills an underexplored gap in multimodal reasoning research. Its originality lies in extending the evaluation of vision-language models to the domain of cartographic reasoning, which requires understanding spatial relations, map symbology, scales, and orientation—skills that differ fundamentally from those assessed in standard visual question answering or chart-understanding benchmarks. The formulation of three spatial-relation categories (topological, metric, and directional) is conceptually strong and well grounded in GIS literature, offering a structured way to test spatial intelligence in LVLMs.\n\nIn terms of quality, the benchmark construction appears rigorous, combining LLM-assisted question generation with extensive human verification by multiple annotators, ensuring reliability of the ground truth. The inclusion of a human performance baseline and evaluation across both direct and contextual settings enhances the depth of analysis and provides a meaningful comparison point for model performance. The paper is also clearly written and logically organized, guiding readers from the motivation through dataset design, evaluation protocol, and results. The argumentation in the introduction—especially the rationale for curating maps from real public documents—is convincing and helps establish the benchmark’s practical relevance.\n\nFinally, the work is significant because it pushes LVLM evaluation toward a domain that mirrors real-world map interpretation tasks relevant to areas like urban planning and environmental monitoring. By revealing a substantial human-model gap in spatial reasoning, FRIEDA sets the stage for a new line of research on geospatial and multi-map understanding in multimodal AI.", "weaknesses": "A few minor aspects could be clarified or expanded to further strengthen the work.\n\nFirst, although the authors describe a careful validation process, the groundtruth generation pipeline (Section 3.2) partly relies on LLMs (GPT-4 and GPT-o3) to propose initial questions and reference answers before human verification. While this hybrid approach is efficient, it may introduce subtle mismatches between the automatically generated “gold answers” and human interpretations. Providing more detail on how such discrepancies were resolved or examples of discarded cases would improve confidence in the dataset’s reliability.\n\nSecond, the evaluation analysis (Section 5)—focused mainly on accuracy and error frequency—could be enriched with more nuanced insights. For example, reporting domain-wise breakdowns (e.g., geology vs. urban planning) or qualitative examples of borderline cases could illuminate specific reasoning challenges faced by LVLMs.\n\nFinally, the contextual setting (Appendix E), although well designed, yielded similar results to the direct setting, raising questions about whether the retrieval aspect was sufficiently demanding. Clarifying how the contextual map sets were selected and randomized, or adding a more difficult retrieval scenario, could make this part of the benchmark more informative.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)