MapIQ: Evaluating Multimodal Large Language Models for Map Question Answering
Authors: Varun Srivastava
Fan Lei
Srija Mukhopadhyay
Vivek Gupta
Ross Maciejewski
Venue: COLM 2025 (2025)
Keywords: Large Language Models
Visual Question Answering
Maps
Geospatial Analysis
Visual Analytics
Benchmark Dataset
TL;DR: We introduce MapIQ, a benchmark for evaluating multimodal large language models (MLLMs) on map-based visual question answering (Map-VQA) across three map types, also assessing their robustness to map desing variations.

Abstract: Recent advancements in multimodal large language models (MLLMs) have driven researchers to explore how well these models read data visualizations, e.g., bar charts, scatter plots. More recently, attention has shifted to visual question answering with maps (Map-VQA). However, Map-VQA research has primarily focused on choropleth maps, which cover only a limited range of thematic categories and visual analytical tasks. To address these gaps, we introduce MapIQ, a benchmark dataset comprising 14,706 question-answer pairs across three map types—choropleth maps, cartograms, and proportional symbol maps spanning topics from six distinct themes (e.g., housing, crime). We evaluate multiple MLLMs using six visual analytical tasks, comparing their performance against one another and a human baseline. An additional experiment examining the impact of map design changes (e.g., altered color schemes, modified legend designs, and removal of map elements) provides insights into the robustness and sensitivity of MLLMs, their reliance on internal geographic knowledge, and potential avenues for improving Map-VQA performance.

Published as a conference paper at COLM 2025
MapiQ: Benchmarking Multimodal Large Language Models for Map Question Answering
Varun Srivastava Fan Lei
School of Computing and Augmented Intelligence Arizona State University
Univer
Tempe, AZ 85281, USA
[vsriva11, flei5) @asu. edu
Srija Mukhopadhyay
Department of Computer Science
International Institute of Information Technology Hyderabad, India
jatjaaakkhopaahyayy@iiii a...aaaiiiit.t...ii......... in
Vivek Gupta & Ross Maciejewski School of Computing and Augmented Intelligence Arizona State University
Tempe, AZ 85281, USA
(vgupt140, rmacieje eesiiiieee@euu.... @asu. edu
Abstract
Recent advancements multimodal large language models (MLLMs) have driven researchers to explore how well these models read data visualiza- tions, e.g,, bar charts, scatter plots. More recently, attention has shifted to visual question answering with maps (Map-VQA). However, Map-VQU -QA research has primarily focused on choropleth maps, which cover only a limited range of thematic categories and visual analytical tasks. To ad- dress these gaps, we introduce MaplQ, a benchmark dataset comprising 14,706 question-answer pairs across three map chpes-choroppethh maps, cartograms, and proportional symbol maps spanning topics from six dis- tinct themes (e.g., housing, crime). We evaluate multiple MLLMs using six visual analytical tasks, comparing their performance against one another and a human baseline. An additional experiment examining the impact map design changes (e.g., altered color schemes, modified legend designs, and removal of map elements) provides insights into the robustness and sensitivity of MLLMs, their reliance on internal geographic knowledge, and potential avenues for improving Map-VQA performance.
1 Introduction
Multimodal Large Language Models (MLLMs) can process and reason across multiple modalities, including text, images, and structured data, enabling applications in various do- mains, from general visual understanding to specialized fields like medical imaging (Zhou et al., 2023; Zhang et al., 2023). Recently, researchers have begun exploring how effectively these models interpret data visualizations, specifically evaluating their ability to understand and reason about scatter plots, bar graphs, line charts, etc (Kafle al., 2018; Masry et al., 2022). Building upon this interest, Map Question Answering (Map-VQA) has emerged to assess the capabilities of MLLMs reading geospatial visualizations. However, current Map-VQA research predominantly focuses choropleth maps (Mukhopadhyay et al., 2024b; Chang et al., 2022; Slocum et al., 2022) and typically evaluates simple visual analytic (VA) tasks. Moreover, these simple VA tasks often overlook the broader range of analytical

---

Published as a conference paper at COLM 2025
tasks described in Information Visualization and Geographic Information Systems liter- ature (Munzner, 2014; Amar et al., 2005; MacEachren, 2004; Brewer, 2016). Additionally, cognitive science literature indicates thematic content can influence human map-reading accuracy due to prior biases (Herrmann & Pickle, 1996), yet it remains unclear whether MI.LMs exhibit similar thematic biases.
Motivated by these limitations, we introduce MapIQ, a benchmark dataset comprising 14,706 question-answer pairs designed to gauge the map-reading capabilities of state-of- the-art MLLMs. MapIQ introduces two previously unexplored map types-cartograms (specifically hexbin maps, which use uniform shapes to reduce visual bias from varying geographic unit sizes (Fan et al., 2024)) and proportional symbol maps (which encode data through varying symbol sizes instead of color gradients (Slocum et al., 2022))-in addition to traditional choropleth maps. MapIQ incorporates six VA tasks across local (tasks involv- ing individual states or small regions) and global spatial scales (tasks requiring synthesis across entire maps), aligning with varying analytical complexity critical in geospatial anal- ysis (MahEachren, 2004). MaplQ also encompasses metadata sourced from six thematic categories, allowing exploration of how thematic content affects MLLM performance.
Using the MaplQ benchmark, we evaluate seven MLLMs, both closed-source and open- source, to address: Are MLLMs biased toward choropleth maps due to their prevalent use in training datasets? How does MILM performance vary across different VA tasks? Does thematic content significantly impact map-reading accuracy? What performance gaps exist between open-source and closed-source models in Map-VQA? Additionally, we establish human performance baseline, examining the alignment between model- generated answers and expert human readers. Finally, aligned with recent research (Wu et al., 2024a; Mukhopadhyay al., 2024a), we investigate the robustness and sensitivity of MLLMs to variations in visual elements, such as map legends and color schemes.
2 Related Work
VA Tasks and Chart-VQA: VA tasks are broadly categorized into low-level and high-level tasks (Brehmer & Munzner, 2013). Low-level tasks involve visual queries, such retrieving values or comparing data points, and rely solely on visual information without requiring broader contextual knowledge. High-level tasks demand deeper engagement, such as identifying trends and patterns, and often require domain expertise and nuanced interpre- tation (Brehmer Munzner, 2013). Low-level VA tasks have been widely used to assess visualization literacy in humans (Lee et al., 2016; Pandey & Ottley, 2023). Building on this, Chart-VQA studies emerged to evaluate MLLMs' ability answer low-level analytical ques- tions about charts Masry et al. (2022). Recent progress in MLLMs, e.g., ChatGPT (OpenAI, 2024) and Claude (Anthropic, 2024), has renewed interest in this area (Bendeck Stasko, 2024; Xu & Wall, 2024), and recent works Wu et al. (2024a); Mukhopadhyay et al. (2024a) as- sess MLLM proficiency across various low-level VA tasks, probing their robustness to design variations. Our study extends these established tasks to geospatial contexts (MacFachren, 2004; Slocum et al., 2022) to evaluate the capabilities and robustness of MLLMs in map- reading tasks.
et
Map-VQA: Studies in Map have emerged with approaches spanning both high-level and low-level map-reading tasks. High-level studies have evaluated MLLMs' capabilities in complex spatial reasoning tasks such as pathfinding and geolocation detection (Xing et al., 2025; Roberts et al., 2024; Hochmair et al., 2024). These efforts rely on specialized maps, including remote sensing imagery and navigation maps, and require domain-specific knowledge and advanced spatial interpretation. In contrast, research on low-level VA tasks involving thematic maps cartographic representations that use simple visual elements like colors and symbols to depict spatial distributions of specific themes (Slocum et al., 2022)-has been relatively limited. P'oonerring work from Chang et al. (2022) introduced dataset of choropleth maps representing data from a single theme (healthcare), evaluating MLLMs on three core tasks: basic map-reading literacy, value retrieval, and identifying spatial extremes. More recently, WapWise (Mukhopadhyay et al., 2024b) expanded on this by benchmarking state-of-the-ar MLLMs, incorporating counterfactual testing and
2

---

Published as a conference paper at COLM 2025
(3) Avenage Tesperatue ("F) 2024
(b) Officses per 100,000 People,
(e) Houscholds gas utility Beating (N),,
Figure 1: Three baseline maps. (a) Choropleth, (b) Proportional Symbol, and (c) Cartogram
comparing model performance with human readers, though still limited to choropleth maps. Building on these foundations, our work presents MapIQ, a comprehensive dataset featuring multiple thematic map types across diverse themes.
3
MapIQ Dataset
In this section, we detail the process followed to create the MaplQ dataset, including the metadata, map selection and generation, question generation, and data validation.
3.1 Metadata
In our study, one of the goals was to assess whether the thematic content of maps influenced the performance of MLLMs in Map- VQA. We identified six representative themes com- monly visualized using thematic maps: social, economic, health, crime, environment, and housing (Dent, 1999; Slocum et al., 2022). After theme selection, we sourced datasets for map generation, focusing only the USA. Data related to social, economic, and housing themes were obtained from the U.S. Census Bureau (U.S. Census Bureau, 2024b); environmental data were sourced from the Environmental Protection Agency (EPA) (U.S. Environmental Protection Agency, 2024) and weather.gov (National Weather Service, 2024); crime data were collected from the FBI Crime Data Explorer (Federal Bureau of Investigation, 2024), and health-related data were gathered from the Centers for Disease Control and Prevention (CDC) (Centers for Disease Control and Prevention, 2024). Our metadata collection process resulted in 258 datasets across six distinct themes, and full dataset details are provided in Appendix A.1.1.
3.2 Baseline Map Design
MaplQ explores three map types: choropleth, cartogram, and proportional symbol (Fig 1). For all map renderings, we adhered to established cartographic best practices (Slocum et al., 2022). We chose discrete classification, which is the most commonly used method in the- matic mapping (Slocum et al., 2022), categorizing data into five classes using Fisher-Jenks classification (jenks Caspall, 1971). For the choropleth and cartogram maps, we employed sequential blue color scheme from ColorBrewer (Harrower & Brewer, 2003). For propor- tional symbol maps, we selected circles with a range-graded size variation (Slocum et al., 2022). All maps were annotated with the official two-letter U.S. state abbreviations (Fed- eral Aviation Administration, 2024), and all maps are rendered utilizing the Albers USA projection (Snyder, 1982)
white background. Following best practices, data were nor- malized across maps, and leader lines were used for clear labeling in cases of small spatial units (Slocum et al., 2022). These design considerations resulted in 258 (datasets) 3 (map types) = 774 774 unique map images. We plan to open-source our map generation pipeline to facilitate the development of new Map-VQA benchmarks and enable MLLMs conduct further research into map comprehension.
3.3 Task Selection, Question Generation, and Ground Truth Extraction
To assess how effectively MLLMs extract geospatial information at local (specific points or regions) and global (entire map) scales (MacFachren, 2004), we selected six VA tasks from
3

---

Published as a conference paper at COLM 2025
established literature: Retrieve Value, Pairwise Point Comparisons, Spatial Extremes, Spatial Clusters, Determine Range, and Regional Comparisons (Lee et al., 2016; Amar et al., 2005; Munzner, 2014; MacEachren, 2004). For tasks requiring local-level analysis on regional scale, we divided the 49 contiguous U.S. states into four zones: West, Midwest, Northeast, and South (U.S. Census Bureau, 2022). Detailed definitions of each task type are provided in Appendix A.1.2. We generated questions in four formats-binary (True/Falsse multiple- al.,
choice (MCQ), single value, and list-aligned with prior literature (Mukhopadhyay 2024b; Lu et al., 2022). Binary and MCQ formats were used for all tasks, while inherent differences among tasks required selective use of single-value (two tasks) and list formats (five tasks). Each question was carefully created using manually validated templates, resulting in 19 unique questions per map. This process yielded 774 maps 19 questions per map = 14,706 question-answer pairs. Ground truth answers to all questions were extracted programmatically using Python. A human expert carefully reviewed the final question-answer dataset to ensure consistency and accuracy. The complete MaplQ dataset is provided in the supplementary materials, while the question templates, example questions, and additional details the ground truth extraction process are provided in Appendices A.1.3 and A.1.4, respectively. A comparative overview with existing Map-VQA benchmarks appears in Appendix A.1.5.
4 Benchmarking MLLMs with MapIQ
For our experiments, we considered both closed- and open-source models. Closed-source models, benefiting from a higher parameter count, typically achieve superior VQA per- formance, and we investigate whether this advantage persists in the MapIQ dataset. We selected ChatGPT-40 (OpenAI, 2024), Gemini 1.5 Pro (Gemini, 2024), and Claude 3.5 Son- net (Anthropic, 2024), given their impressive multimodal capabilities. For open-source models, we focused on recently released (2024) models with robust documentation hosted on HuggingFace (Hugging Face, 2024). To ensure fairness and manage computational re- sources, we constrained our selection to models within the 7B-SB parameter range, choosing Qwen2-VL (Wang al., 2024), Molmo (Deitke et al., 2024), IntemVL2.5-MPO (Chen al., 2024), and Idefics3 (Laurençon et al., 2024). DeepSeek (Wu et al., 2024b) and MiniCPM-V 2.6 (Yao al., 2024) were initially tested but excluded due to incoherent or missing outputs.
Prompting Strategy The prompts for benchmarking MLLMs were designed for this experiment in a zero-shot setting (Radford et al., 2019), and each prompt was tailored according to task type and question type. Each prompt comprised two primary components. The first component included general instructions, providing a brief overview of the map- reading task that the model was expected to undertake. Additionally, this component specified instructions for formatting the model's responses according to the question type. For the Spatial Clusters task, these general instructions were further supplemented with clear definition of spatial clusters, details about the spatial adjacency rule, and the minimum cluster size. The second component of the prompt was designed for tasks requiring zoning information (e.g., spatial extremes, regional comparisons). For these tasks, details about geographic zones (U.S. Census Bureau, 2024a) and their constituent states were provided first, followed by the previously described general instructions. Prompt details for all task types are available in Appendix A.2.1.
Test Dataset- To manage computational resources, we used representative subset of the full MapIQ dataset as our test set. We selected 35% of the entire dataset, resulting in 5130 QA pairs. To ensure this sample was representative and balanced across the experimental variables, we employed stratified random sampling, considering map type, task type, question type, and theme. More information about the sampling process is included in Appendix A.2.2, and the complete test dataset is provided in supplementary materials.
Evaluation Metrics Given the diversity of question types in our benchmarking experiment, we tailored evaluation metrics specifically for each type. For binary questions, accuracy was selected, and scores of 100 (correct) or 0 (incorrect) were assigned and aggregated across the dataset. For MCQ-type questions, where tasks could have multiple correct options, we employed the F1 score due to its balanced treatment of precision and recall,
4

---

Published as a conference paper at COLM 2025
Task Type
Map Types
Models
Determine Range
Pairwise Point Comparisons
Regional Comparisons
Retrieve Value
Spatial Clusters
Spatial Extremes
Cartogram
Choropleth
Symbol
Human Baseline
94.03
95.39
85.13
96,91
86.19
97.25
92.52
92.62
93.23
MLLMs Overall
49.36
51.98
31.14
51.11
42.53
40.75
Open Source MLLMs
Qwen2-VL
36.05
54.29
49.75
61.89
28.99
54.01
47.39
52.21
45.16
Molmo
24.20
43,90
47.22
37.67
26.93
38.92
37.46
36.86
35.29
VieemVL2.5-MER
31.23
51,68
48.18
49.31
28.59
52.24
41.69
52,22
37.62
Idefies3
15.56
44.06
48.49
35.55
26.33
44.83
33.63
39.59
34.15
Closed Source MLLMs
Gemini 1.5
21.85
44.12
50.12
51.68
39.33
40.54
Clande 31.48
59.10
ChatGPT-40 16.30
48.93
54.29
45.68
33.15
54.93
42.48
43.16
41.55
Figure 2: Performance of Humans and 7 MLLMs (overall and individual) across 6 Task Types and 3 Map Types. Values represent average performance scores (in %)) calculated separately for each task and map type. Bolded values indicate the best-performing model for each experimental condition.
rewarding partial correctness and penalizing false positives and negatives (Van Rijsbergen, 1974; Derczynski, 2016). Similarly, list-type questions utilized F1 scores, effectively handling partial matches. Individual F1 scores were calculated per response and averaged for overall evaluation. For single-value questions, we followed the same scoring method as binary questions.
MLLM Response Extraction and Validation - The model responses were extracted using Python scripts on Linux-based system equipped with an NVIDIA A100 GPU and 128 GB of RAM. We manually validated outputs to ensure quality before evaluating them with previously described metrics. During validation, some models provided contradictory responses by selecting "None of the above" (NOTA) alongside other options (e.g., "My answer is [a, b, e)," where e is NOTA), which we marked incorrect to maintain evaluation integrity. Additionally, several models frequently included unsolicited explanatory text or reasoning in their responses. Removing these extraneous details was particularly time- consuming, especially for IntemVL2..--POO and Molmo (open-source models), and Gemini 1.5 Pro (closed-source model). The evaluated and validated datasets for each model are provided in the supplementary happlementryymaann materials, while Appendix A.2.3 contains additional details about the MILM response validation process.
Human Baseline Evaluation Along with comparing the performance of various MILLMs, we aimed to establish how these models compare to expert human performance in the Map-VQA context. We established a human baseline by uniformly sampling around 9% of the test set, resulting in 450 unique QA pairs. Special care was taken to ensure approximately balanced representation across all experimental variables guarantee fairness in the human evaluation. Using this sample, two expert human map readers independently answered all 450 questions. Subsequently, third independent expert validated their responses to ensure consistency and reliability. Humans received the same questions as those posed to the MLLMs, and identical instructions were provided across evaluations.
Benchmarking MLLMs with MapIQ Results and Discussions
Detailed performance results across the MaplQ dataset are provided in Fig 2 and Fig 3. In this section, we discuss key findings comparing models to humans and stratifying results across task types, map types, question types, and themes.
Comparing MLLMs Across Experimental Variables Overall, Claude 3.5 Sonnet consis- tently outperformed other models across all four experimental variables- Task Type, Map Type, Theme, and Question Type-cleate establishing itself as the most robust MLLM evaluated in this study. It achieved top rankings in most tasks and settings, demonstrating

---

Published as a conference paper at COLM 2025
Question Type
Themes
Models
Binary Acc.
MCQ
List
Single Value
Crime
Economic
Environment
Health
Housing
Social
Acc.
Human Baseline
95.72
92.69
90.59
90.20
90.47
91.03
94.65
92.69
93.98
93.87
MILLMs Overall
61,00
35.34
36.91
28.04
37.55
44.96
47.16
42,60
39.52
45.27
Open Source MLLMs
Qwen2-VL.
75.25
37.97
37.70
38.52
46.96
49.36
49.24
48.81
46.69
48.46
Molmo
62.16
25.11
29.81
10.74
24.20
43.90
47.22
37.67
26.93
38.92
Intem 5L2.5-PO
62.59
38.66
34.51
26.48
41.94
45,06
44.86
42.86
42.85
45,49
Idefies3
57.10
27.29
28.88
14.63
37.15
36,90
36.69
35.40
34.38
34,22
Closed Source MLLMs
Gemini 1.5 Pro
47.78
35.12
39.64
31.30
21.85
44.12
50.12
38.99
32.96
51.68
Claude 3.5 Sonnet
62.84
48.99
49.83
47.41
49.97
53.06
56.40
51.82
53.51
54.35
ChatGPT-4o
59.26
34,23
38.03
27.22
40.75
42.29
45.61
42.65
39.29
43.79
Figure 3: Performance of Humans and 7 MLLMs (overall and individual) across 4 Question Types and 6 Themes. Values represent average performance scores (in %) computed for each question type and theme. Bolded values indicate the best-performing model for each experimental condition.
strong adaptability to various map-reading challenges. Additionally, Qwen2-VL, an open- source model, also showed competitive results, emerging the second-best model overall, making it a compelling open-source alternative. Detailed analysis per experimental variable is given below:
Task Type: As shown in Fig 2, Claude 3.5 Sonnet emerged as the best- best-performing model in four out of six tasks, notably excelling in Regional Comparisons with an accuracy of 65.84%. Qwen2-VL, performed best in the Determine Range and Retrieve Value tasks, achieving particularly strong performance in Retrieve Value (61.89%). In contrast, Molmo and Idefics3 consistently showed the weakest performance, with Idefics3 recording the lowest overall accuracy (15.56%) in Determine Range. Overall, MLLMs performed best in Regional Comparisons, followed by Spatial Extremes, Pairwise Point Comparisons, Retrieve Value, Spatial Clusters, and Determine Range. Interestingly, despite its relative ieapliccty-rquurrnng only straightforward extraction of visual encoding information for single mlst ML.LMs encountered significant difficulties with the Retrieve Value task. Conversely, Regional Comparisons, which demand comprehension of broader spatial patterns and zoning information, appeared comparatively easier for MLLMs.
Map Type: Across all three map types, Claude Sonnet consistently emerged the highest- performing model, see details Fig 3. In contrast, Molmo and Idefics3 consistently ranked among the weakest performers, with Idefics3 notably struggling Cartograms, achieving an accuracy only 33.63%. Overall, MLLMs generally performed best on Choropleth maps, followed by Cartograms and Proportional Symbol maps. Notable exceptions include Claude 3.5 Sonnet and Molmo, which exhibited slightly higher accuracy on Cartograms, possibly due enhanced color differentiation capabilities facilitated by uniform spatial unit sizes. Another exception was Gemini 1.5 Pro, which showed its highest accuracy on Proportional Symbol maps, highlighting its relatively stronger capability in differentiating symbol sizes as opposed to color variations.
Theme: Similar the results by map type, Claude 3.5 Sonnet consistently emerged the top-performing model across all themes, while Molmo and Idefics3 generally ranked among the lowest performers (Fig 3). Notably, Gemini 1.5 Pro exhibited the weakest performance within the Crime theme, with an accuracy of 21.85%, marking the lowest result among all evaluated models and themes. Further analysis revealed that MLLMs achieved the highest accuracy on maps depicting environmental data, followed by social, economic, health, housing, and crime-related maps. Model performance varied substantially across these themes, ranging from 47.16% (environment) to 37.55% (crime). This significant variation indicates a possible bias toward certain thematic topics, potentially due to models leveraging internal, topic-specific knowledge.

---

Published as a conference paper at COLM 2025
(3) Mean performance difforence Task Types
(b) Moan performance difference ncross Theme (in
-3.55
Deromiro magg
-0.04fcrmee
Pornwise paint compation
2,06
Ecooomic
2.68
Regional comperison
8:34
Enviorment
6.211
Retreve valce
1881
3,30
Sputil dlstts
Housing
Spacial ocnemne
8.43
3.17
Open
Closed Source
Open Source
Closed
Source
Source
(c) Mean performance difference across Question Types (in %)
(4) Mean perfomaance difference Map Types
-7.3
Sinzy
MCQ
5.91
Canegram
9,42
Chocopien 10.08
Single valaz
11.94
Symbel
Open
Closed
Open
Closed
Source
-7
6
5
10
11
Source
Source
Source
Figure 4: Mean performance differences (in %) between open- and closed-soure models across four experimental variables: (a) Task Type, (b) Theme, (c) Question Type, and (d) Map Type. Bars extending to the right of zero indicate better performance by closed models, while bars to the left indicate better performance by open-source models.
Question Tyype:
In general, MLLMs performed best on binary questions, followed by list, MCQ, and single-value question types. An intriguing observation is that models performed slightly better on list questions than MCQs (36.91% vs. 35.34%), despite MCQs typically being considered easier by human standards, as demonstrated by our Human Baseline results. This indicates that while models can identify multiple relevant elements within a list, they may face difficulty selecting the single most appropriate option from multiple-choice scenarios. At the model level, Qwen2- notably excelled in binary questions, achieving an impressive accuracy of 75.25% (Fig 3). For all other question types- lists, MCQ, and single-value Claude 3.5 Sonnet consistently delivered the best performance, reaffirming its position the most robust model overall.
Comparing closed-sourced and open-sourced MLLMs Comparing differences in mean performance scores across experimental variables provided detailed view of how closed- source models fare against open-source ones. As shown in Fig 4(a), closed-source models generally performed better, though the average gap was modest at 4.18%. Notably, open- source models outperformed closed-soure ones in the Determine Range task, largely due to ChatGPT-4o's poor performance (16330%), the second-lowest overall (Fig 2). For map types (Fig 4(d)), the largest gap (6.29%) was in Proportional Symbol maps, indicating that closed-source models are better at interpreting symbol size as visual encoding. The small- est gap (0.08%) appeared in Choropleth maps, suggesting similar proficiency, likely due to their frequent inclusion in open-source training data. Furthermore, open-source models out- performed closed-source models on binary questions (Fig 4(c)), mainly due to Qwen2- VL's strong performance and Gemini 1.5 Pro's weaker showing (47.78%). Thematic differences were also notable Fig 4(b) shows: larger gaps appeared in social and environmental themes, while performance was similar in crime. This variation may reflect dlosed-soure models' broader internal knowledge, aiding the interpretation of certain topics. On average, the performance gap across all variables remained modest at around 4.4%. These results suggest that open- and closed-source models are becoming increasingly comparable in interpreting thematic maps. While closed-source models hold slight edge, the rapid progress of open-source collaborative advances in geospatial reasoning and map-based visual question answering
Qwen2-VL-
the strong potential for
Comparing MLLMs with Human Baseline While open and closed source models demon- strate somewhat comparable performance, our results show that MILMSs performed consid- erably worse than the human baseline, with humans outperforming MLLMs by an average margin of 50.35% across all four experimental variables. Upon analyzing specific task types, the largest performance gap (68.79%) appeared in the Determine Range task. Human read- executed this task effectively, whereas MLLMs struggled to interpret visual encodings within localized regional contexts. With respect to map types, the greatest performance difference was observed in Proportional Symbol Maps (52.48%) compared to Carlogram
7

---

Published as a conference paper at COLM 2025
per Task Type (%)
Effect per Map Type (%%
0.36
-1.06
-2.55
-2.45
+2.79
-1.50
-4.47
Large Lahels-
-0,93
0.19
-0.13
-1.97
-0.99
-2.55
-1.06
-0.71
-2.23
-0.39
Small Labels-
0.92
3.51
-0.90
-1.75
-0.72
-1.61
-0.19
-3.06
1.79
Largo Legend
1.31
0.65
Map Design Variation
Legend
0.37
-5.79
-2.25
0.35
-1.55
Legend Orientation
-3.25
1.91
1.06
-1.92
-1.14
-0.03
+0.78
-2.91
-1.85
Color Divergent
-0.47
-11.45
-1.15
-9.13
3.57
-11.33
4.99
-5.16
-9.02
7.09
Color Spectral
-0.47
12..55
-0.92
-10.87
2.25
-6.24
4.80
-6.71
-7.28
-7.0
Color Flipped
-1.86
-27.25
-4.85
-13.83
8.87
-18.63
9.59
-10.44
-12.95
11.70
Large Legend Font
-2.78
-0.37
-1.08
-4.06
-1.13
0.19
-1.5
0.37
-3.05
-2.33
-1.65
Small Legend Font-
0.00
-2.05
-0.16
-2.51
1.35
-2.57
0.99
1.26
-2.43
-2.04
-1.0
Legend Bottem Right-
1.23
1.22
-1.27
-6.63
0.25
-1.31
+109
1.27
-1228
-4.13
1338
Rotated X-axis
2.45
0..41
-3.85
Rotated Y-axis
-3.25
-3.62
4.45
4.55
Map
-3.57
-0.75
4.94
0.97
-0.58
-1.3
1.27
-3.11
-1.64
-1.5
Legend
-0.4
-0.50
-5.65
-0.89
-2.55
-1.97
-3.7
-9.09
4.92
Pairwise Point Comparisons
Detemmne Range
Regional Comparisons
Retrieve Value
Spatial Clasters
Spatial Extremes
Cartogram
Chotopleth
Proportional Symbol
Avg: -1.68
Avg: 22.96
Overall
Overall
Effect (%)
Effect (%)
(a) Task Type
(b) Map Type
Figure 5: Difference in performance of Qwen2-VL relative to the baseline across 15 map design variations, broken down by (a) Task Type and (b) Map Type. Each cell represents the percentage change in accuracy for specific task map type under given variation.
and Choropleth maps. Overall, this comparative analysis demonstrates that MLLMs are currently well below human levels of performance for Map VQA tasks.
Map Design Variations
We were also interested in determining how MLLM's robustness (i.e., performance degrada- tion relative baseline map design) and sensitivity (i.e., performance fluctuations across different design changes) are impacted by variations in map design.
6.1 Variations Tested
We selected 15 map design variations targeting visual elements essential to map-reading tasks, spanning five categories: label size (Large vs. Small), legend properties (size, font, orientation, placement), color schemes (divergent, spectral, flipped), map orientation (180a X and Y-xxis rotations), and removal of key elements (title, legend). Divergent and spectral schemes from ColorBrewer (Harrower & Brewer, 2003) were chosen for their effectiveness in class separation (Slocum et al., 2022), enabling evaluation of model behavior under non-default color encodings. The flipped scheme (lighter shades = higher classes) tested robustness against counterintuitive mappings. Title and legend removals assessed the models' ability to rely solely on visual encoding without supplementary textual context, while modifications label and legend properties examined sensitivity minor visual changes. Axis rotations were introduced to evaluate potential "north-up" bias and the effect of orientation on map-reading performance. Some variations were not uniformly applicable across all map types due to differences in visual encoding structures (e.g., color schema changes did not apply to proportional symbol maps). Additionally, for maps without titles or legends, minor prompt edits (e.g., "darker shades indicate higher class values") ensured fair evaluation. Further information on exceptions and an illustrative example is provided in Appendix A.3.1.
8

---

Published as a conference paper at COLM 2025
Kfffect per Task Type
Kfffet per Map Type
1,65
1.66
40.11
4008
-0.15
-2.25
-2541
-3.30
Large Labels
-3.73
-2.72
-1.02
3.36
-1.03
-1.13
Small
-1885
-2.41
-3.71
-2.16
0.76
2.54
-0.63
-3.92
-0.63
Large Legend
1.85
1.67
0.37
-1.10
0.35
-2.19
0.11
-2.53
-2.18
Map Design Variation
Small Logend
1.85
1.01
11.779
-1.61
-1.54
-1777
-5.84
-3.28
4.50
Legend Orientation-
-3.71
0.81
1.06
5.56
2.24
5.60
1.93
-1.23
0.93
-0.5
Color Divergent-
-2.32
-15.65
2.59
12.73
5.51
-2.54
0.05
-6.07
2.97
-1.55
Color Spectral
10.18
-5.10
0.27
26.24
13:94
8.01
8.92
3.12
12.01
7.57
Color Flipped
-7.87
-34.46
-16.44
-8.85
3.80
26.25
-5.00
11.93
-15.98
16:96
Large Legend Font-
1.85
0.20
-0.44
0.92
-2.59
2.95
0.488
0.27
0.22
1.01
0.50
Small Legend Font-
-3.71
-0.42
-2.75
2.22
-1.68
-4.18
1,75
-3.10
0.16
-1.69
-1.54
Legend Bottom Right-
0.00
4.31
1.57
-4.19
1.29
-2.23
0.113
-1.73
-0.66
2.08
-010
Rotated X-axis
1.83
0.91
8.47
40.13
5.89
-1.56
2.57
0.07
0.25
0.16
Rotated Y-axis-
00.33
2.26
5.09
3.35
-5.54
0.10
0.72
-5.18
2.36
1141
No Map Title-
-5.56
-5.65
1.01
-1.61
1.86
4.39
-2.99
-5.16
-1.65
-0.22
-234
No Logend-
-0.21
2,99
-6.62
-2.34
-1.99
-0.55
6.98
0.62
2.45
Determine Range
Pairwise Point Comparisons
Regiondl Comparisons
Retrieve Value
Spanial Chasters
Sperial Extremes
Cantgrra
Proporional Symbol
Chomopett
Jb
Avg: -0.65
Avg: -1.45
Overall
Overall
Effect (%)
Effect (%6
(a) Task Type
(b) Map Type
Figure 6: Difference in performance of Claude 3.5 Sonnet relative the baseline across 15 map design variations, divided by (a) Task Type and (b) Map Type. Each cell represents the percentage change in accuracy for specific task map type under a given variation.
systematically applied these variations to representative subset of 36 maps, selected by randomly choosing two topics from each of the six themes across all three map types in the MapIQ dataset. This resulted in 540 uniquely varied maps (36 maps 15 variations), accompanied by 684 QA pairs, with all maps provided in the supplementary materials. Baseline performance was first established using the unmodified maps, after which the test dataset was evaluated using Claude 3.5 Sonnet and Qwen2-Vi Lwen2-VL-ttee the top-performing closed- and open-source models from our benchmarking experiment.
To manage computational requirements,
6.2 Results and Analysis
We computed the performance difference for each variation relative to the baseline accuracy, separately by Task Type and Map Type. Overall, Claude 3.5 Sonnet exhibited greater robustness to design perturbations than its open-source counterpart, Qwen2-VL. As shown in Fig 5(a), Qwen2-VL experienced an average performance degradation of -1.68% across task types, with 11 out of 15 variations negatively impacting performance. At the map type level (Fig 5(b)), the model showed an even larger average decline of -2.96% where all 15 variations reduced accuracy. These results suggest general vulnerability of Qwen2-VL to visual design changes.
Further analysis revealed that variations related to color schemes produced the most pro- nounced negative effects on Qwen2-VI across both task and map crrpe--particllee Color Flipped variation, which resulted in the steepest decline in performance, At the map level (Fig 5(b)), Choropleth maps saw the greatest performance drop, possibly due to their over-representation in the model's training data. For Proportional Symbol maps, the most severe drop occurred under the No Legend condition (-9.09%), highlighting the model's reliance on legends for interpreting symbol sizes. Conversely, the Small Labels variation led to modest improvement (+1.79%), likely due to reduced visual clutter. Notably, altering legend font size also caused a performance drop across task and map types, suggesting difficulty in interpreting complex or non-standard legends. These patterns collectively point to insufficient visual grounding and heightened sensitivity to design variations in Qwen2-VI..
the

---

Published as a conference paper at COLM 2025
In comparison, Claude 3.5 Sonnet exhibited smaller average degradation of -0.65% across task types (Fig 6(a)), with only 7 out of 15 variations negatively impacting performance. Fig 6(a) further indicates that tasks requiring comprehension of class hierarchies, such as Spa- tial Extremes and Pairwise Point Comparisons, were disproportionately affected. In contrast, Spatial ceasters-whiche depend more spatial proximity than value -hoeringgshowedd improved performance. Across map types (Fig 6(b)), Claude showed an average drop of -1.45, again lower than Qwen2-VL. As with Qwen2-VL, the Color Flipped variation caused the most significant performance decline. This drop exceeded that of Qwen2-VL, indicating that Claude may be more sensitive to disruptions in sequential color schemes. However, Claude exhibited a strong performance boost under the Color Spectral scheme, suggesting superior ability interpret color encodings using distinct hues. Interestingly, under the No Map Legend condition, Claude's performance in Choropleth maps improved (+6.98%), possibly indicating better reliance on visual encoding when legend information is absent. Claude also showed sharper decline than Qwen2-VL under the No Title condition, perhaps reflecting a greater dependence on internal contextual knowledge poteetialllll due to its larger and more diverse training corpus.
Limitations and Future Work
While our study provides benchmark for evaluating the Map-VQA ability of an MLLM, there are several key limitations. We exclusively employed zero-shot prompting to evaluate MLLM performance, without exploring alternative strategies such role-play prompting or visual prompting. The geographic scope was limited to U.S. state-level resolution, restricting our ability examine how spatial granularity influences MLLM reasoning. Moreover, we did not evaluate model robustness against misleading or deceptive maps-a critical dimension for understanding visual comprehension with limited contextual knowledge. The study was also limited regarding map type diversity and visual complexity. We focused Cartogram, and Proportional Symbol
on three standard thematic map types- Sppes-Choropletiee maps, leaving out other important variants such as Isopleth maps and non-contiguous cartograms. Future research should expand on these aspects by incorporating a broader range of map types, exploring finer geographic resolutions (e.g., state-county city-district level maps), and investigating high-level tasks such as automated caption generation and insight-based map summarization.
Conclusions
In this study, we introduced MaplQ, diverse and thematically rich benchmark dataset designed to evaluate the low-level map-reading capabilities of state-of-the-art MLLMs. We observed notable variations in model performance across map themes, suggesting that prior knowledge about the topic influences effectiveness, with models underperforming on less familiar themes. Among the models evaluated, Claude 3.5 Sonnet stood out for its superior accuracy and greater robustness to design variations compared to other MLLMs; however, MI.LMs still lag behind human baselines. These findings highlight the need for more generalized and context-aware MLLMs laying the groundwork for more trustworthy and interpretable visual reasoning systems.
References
Robert Amar, James Eagan, and John Stasko. Low-level components of analytic activity in information visualization. In JEEE Symposium Information Visualization, 2005. INFOVIS 2005., pp. 111-117. IEEE, 2005.
Anthropic. Claude 3.5 sonnet. http:://ww..atrrppt..c.co/eeeeee S-se,, 2024.
Alexander Bendeck and John Stasko. An empirical evaluation of the gpt-4 multimodal language model on visualization literacy tasks. IEEE Transactions on Visualization and Computer Graphics, 2024.
10

---

Published as a conference paper at COLM 2025
Matthew Brehmer and Tamara Munzner. A multi-level typology of abstract visualization tasks. IEEE transactions on visualization and computer graphics, 19(12):2376-2333 2385, 2013.
Cynthia Brewer. Designing Better Maps: A Guide for GIS Users, 2nd Edition. ESRI press, 2016.
Centers for Disease Control and Prevention. Cdc public health data, 2024. URL https:
//www. cdc. gww.gpaace//ineeeeeeee html.
Shuaichen Chang, David Palzer, Jialin Li, Eric Fosler-Lussier, and Ningchuan Xiao. Mapqa: A dataset for question answering on choropleth maps. arxiv preprint ar arXiv:2211.0854 2022.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185-24198, 2024.
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for taate-of-thearre multimodal models. aiv preprint 17X1::2499.17146, 2024.
Borden D Dent. Cartography: Thematic Map Design. Ingram, 1999.
Leon Derczynski. Complementarity, f-score, and nlp evaluation. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pp. 261-266, 2016.
Arlen Fan, Fan Lei, Michelle Mancenido, Alan M Maceachren, and Ross Maciejewski. Understanding reader takeaways in thematic maps under varying text, detail, and spatial autocorrelation. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, pp. 1-17, 2024.
Federal Aviation Administration. Two-letter state and territory abbreviations, 2024. URL https://ww.. faa.
htt.fiiccpublccaatoo/////aa/aaa/ httpp//s////tt.tmaaaaaaaaaaaaaaa htmll.
Federal Bureau of Investigation. Crime data explorer, 2024. URL https://cde.ucr... cjis. gov/.
Google Gemini. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXio:2403.05530, 2024.
Mark Harrower and Cynthia A Brewer. Colorbrewer. org: an online tool for selecting colour schemes for maps. The Cartographic Journal, 40(1):27-37,20 2003.
Douglas Herrmann and Linda Williams Pickle. A cognitive subtask model of statistical map reading. Visual Cognition, 3(2) 1996.
Hartwig H Hochmair, Levente Juhász, and Takoda Kemp. Correctness comparison of chatgpt-4, gemini, claude-3, and copilot for spatial tasks. Transactions in GIS, 28(7): 2219 2231, 2024.
Hugging Face. Hugging face: The ai community building the future. https://huggingfaat co/,2024.
George Jenks and Fred C Caspall. Error on choroplethic maps: definition, measurement, reduction. Annals of the Association of American Geographers, 61 (1):2217722 1971.
Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. In Proceedings of the IEEÉ conference on computer vision and pattern recognition, pp. 5648-5656, 2018.
Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better understanding vision-language models: insights and future directions., 2024.
11

---

Published as a conference paper at COLM 2025
Sukwon Lee, Sung-Hee Kim, and Bum Chul Kwon. Vlat: Development of a visualization literacy assessment test. IEEE Transactions on Visualization and Computer Graphics 23(1): 551-560, 2016.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chur Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
Alan M MacEachren. How maps work: representation, visualization, and design. Guilford Press, 2004.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. ar Xiv preprint ar arXiv:2203. 10244, 2022.
Srija Mukhopadhyay, Adnan Qidwai, Aparna Garimella, Pritika Ramu, Vivek Gupta, and Dan Roth. Unraveling the truth: Do vlms really understand charts? a deep dive into consistency and robustness. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 16696-16717, 2024a.
Srija Mukhopadhyay, Abhishek Rajgaria, Prerana Khatiwada, Vivek Gupta, and Dan Roth. Mapwise: Evaluating vision-language models for advanced map queries. arxiv preprint arXie:2409.00255, 2024b.
Tamara Munzner. Visualization analysis and design. CRC press, 2014.
National Weather Service. Weather.gov national weather data, 2024. URL https://www. beather.gov/.
OpenAL. Hello gpt-4o. http://pppaaaccoo/mm/eeeee/lllop 2024.
Pandey and Alvitta Ottley. Mini-vlat: A short and effective measure of visualization literacy. In Computer Graphics Forum, volume 42, pp. 1-11. Wiley Online Library, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188:99, 2019.
Jonathan Roberts, Timo Lüddecke, Rehan Sheikh, Kai Han, and Samuel Albanie. Charting new territories: Exploring the geographic and geospatial capabilities multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp- 554-563, 2024.
Peter A Rogerson. Statistical methods for geography: a student's guide. SAGE Publications Ltd, 2019.
Terry A Slocum, Robert B McMaster, Fritz C Kessler, and Hugh H Howard. Thematic Cartography and Geovisualization CRC Press, 2022.
John Parr Snyder. Map projections used by the U.S. Geological Survey. Technical report, US Government Printing Office, 1982.
Judi Thomson, Elizabeth Hetzler, Alan MacEachren, Mark Gahegan, and Misha Pavel. A typology for visualizing uncertainty. In Visualization and Data Analysis, volume 5669, pp. 146-157, 2005.
U.S. Census Bureau. Geographic levels, 2022. URL
cwwsus.ceeu....mm gov/ /wwurreeyy/eooommmmeeee////////e weweiliinneegggggrapeee reographies/levels. html. Accessed: 2025-03-22.
U.S. Census Bureau. Census regions and divisions of the united states, 2024a. URL https: //www2, census. gov/ gov/geo/pdff//mppp httpa//a//////p/eeeeeeeeee///ss/
.pdf.
U.S. Census Bureau. American community survey (acs), 2024b. URL https:://// htwp://wwwwwwsw..////o//e//sss.. gov/programs
htv/pproraass-survesss/ssss surveys/acs.
12

---

Published as a conference paper at COLM 2025
U.S. Environmental Protection Agency. Epa environmental data, 2024. URL https://www.
epa. gov/aqs.
Cornelis Joost Van Rijsbergen. Foundation of evaluation. Journal of documentation, 30(4):
365-373, 1974.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhanc- ing vision-language model's perception of the world at any resolution. ar preprint ar arXiv:2409.12199 2024.
Yifan Wu, Lutao Yan, Leixian Shen, Yunhai Wang, Nan Tang, and Yuyu Luo. Chartinsights: Evaluating multimodal large language models for low-level chart question answering. Xiv preprint ar xrXi:2405.000011 2024a.
Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang, Liang Zhao, Yisong Wang, and Chong Ruan. Deepseek-v12. Mixture-of-experts vision-language models for advanced multimodal understanding, 2024b. URL httpps://axxiv http:/vw...rggb//////// 10302.
Shuo Xing, Zezhou Sun, Shuangyu Xie, Kaiyuan Chen, Yanjia Huang, Yuping Wang, Jiachen Li, Dezhen Song, and Zhengzhong Tu. Can large vision language models read maps like human? arviv preprint ar Xiv::2003.14600 2025.
Zhongzheng Xu and Emily Wall. Exploring the capability of Ilms in performing low-level visual analytic tasks on svg data visualizations. In 2024 IEEE P and Visual Analytics (VIS), pp. 126-130. TEEE, 2024.
Yuan Yao, Tianyu Yu, Ao Zhang. Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm on your phone. ar Xiv preprint ar Xiv:2408.00 000000 2024.
Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint ar Xiie:2306..11011 2023.
Hongjaan Zhou, Fenglin Liu, Boyang Gu, Xinyu Zou, Jinfa Huang, Jinge Wu, Yiru Li, Sam Chen, Peilin Zhou, Junling Liu, et al. A survey of large language models in medicine: Progress, application, and challenge. arXiv preprint (ii0:2311.05111 2023.
A Appendix
Supplementary Materials
All supplementary materials are available at OSF: https://osff.oo/k//j4/// 242441277000F00611222200e888
A.1 More on MapIQ Dataset
A.1.1 Metadata Classified by Theme
The metadata for MapIQ was sourced from reputable sources, which informed the topical focus of the maps. Map topic is fundamental element of thematic maps, as it helps readers contextualize the information being visualized and interpret the content more effectively (Herrmann & Pickle, 1996). However, due to data quality constraints, the number of datasets per map theme in the full MapIQ dataset was not balanced. To address this in our analysis, the test dataset was sampled to ensure equal number of question instances from each theme. Table summarizes the number of datasets associated with each metadata theme.
13

---

Published as a conference paper at COLM 2025
Theme
# Datasets
Example Topic
Economic
46
Percentage of Population in the Labor Force
Housing
68
Percentage of rental units with monthly rent under $500
Social
49
Percentage of cohabiting couple households
Health
40
Cognitive disability among adults (in percent)
Crime
23
Burglary Offenses per 100,000 People
Environment
32
Annual CO2 Emissions (Million Metric Tons)
Table 1: Distribution of datasets by Theme, along with example map topics corresponding to each category.
A.1.2 Task Type Definitions
MaplQ uses six distinct visual analytical tasks to evaluate the performance of MLLMs in map-reading contexts. Each task is clearly defined, emphasizing its geospatial scale (local or global), the complexity of the required visual analysis, and the specific cognitive skills involved.
1. Retrieve Value: This task involves identifying the attribute class of a specific state by referencing the map's legend. This fundamental map-reading task requires effectively linking legend classifications with the corresponding visual encodings on the map (Lee et al., 2016; MacEachren, 2004). The geospatial scale of this task is local, as it focuses on retrieving information about individual states rather than analyzing broader spatial patterns across the entire map.
2. Pairwise Point Comparisons: This task involves comparing the attribute classes of two specified states to determine whether one is greater or less than the other (Lee al., 2016; Amar et al., 2005). Pairwise Point Comparisons require interpreting and evaluating visual encodings for two distinct states simultaneously. Despite this added complexity, the geospatial scale remains local, as the analysis is limited solely to the two states in question and does not require an understanding of broader spatial patterns across the entire map.
3. Spatial Extremes: This task involves identifying states with either the highest or lowest attribute class values (Lee et al., 2016; Munzner, 2014). Spatial Extremes can be assessed at both local and global geospatial scales (Thomson et al., 2005). The local version of the task requires finding extreme values within a specific map region (e.g., the West Zone), thereby limiting the analysis to predefined regional subset. In contrast, the global version involves identifying extremes across the entire map, demanding a more comprehensive understanding of spatial patterns distributed throughout all states.
4. Spatial Clusters: This task involves identifying spatial clusters Slusters-groups groups of contiguous states sharing similar attribute values-which demands higher-level pattern recognition skills (Lee et al., 2016; MacFachren, 2004). Including this task required defining spatial contiguity, for which we adopted the queen adjacency rule, wherein states are considered neighbors if they share a common border or vertex (Rogerson, 2019). Additionally, we focused specifically on clusters of states exhibiting extreme attribute values-clusterts con- taining states with either consistently high or consistently low values-sine identifying such "hot spots" or "cold spots" is among the most frequently executed spatial analy tical tasks in practice (Rogerson, 2019). We evaluated both local and global versions of this task. The local variant required identifying clusters within predefined map regions, while the global variant involved recognizing clusters spanning the entire map.
This task involves determining the range of attribute class values present within specific map region Munzner, 2014; Lee et al., 2016. This task is strictly local, as it confines the analysis exclusively to predefined subsets of the map. Executing this task demands careful identification of the minimum and maximum attribute class values among a selected group of states within the given region, effectively interpreting the visual encodings of multiple spatial units simultaneously. The inherent complexity arises
Determine Range:
14

---

Published as a conference paper at COLM 2025
Question Type
Task Type
Geospatial Scale
Total
Binary
MCQ
List
Single Value
Retrieve Value
Local
Pairwise Point Comparisons
Local
/
Spatial Extremes
Local and Global
/
X
3
Spatial Clusters
Local and Global
/
3
Determine Range
Local
x
Regional Comparisons
Global
Table 2: Eligibility of each task type for different question types, along with the associated geospatial scale (local or global). The "Total" column indicates the number of question types applicable to each task.
from synthesizing information across the region rather than focusing on individual states, highlighting the task's 's reliance on precise legend interpretation and detailed visual analysis within localized contexts.
This task involves comparing the overall attribute patterns between two distinct regions on the map to determine whether one region generally exhibits higher or lower attribute class values than the other (MacEachren, 2004). Unlike tasks that focus solely on individual states or single-region analyses, Regional Comparisons require synthesizing and evaluating broader spatial patterns across multiple states within each of the two selected regions. Among the tasks examined, this is the most open-ended, demanding strong visual pattern recognition and correlation skills to effectively discern general trends and differences between regions. This task is global in scale, involving reasoning about spatial configuration and distribution patterns across the map rather than isolated units (MacEachren, 2004).
6. Regional Comparisons:
A.1.3 Question Templates
Due to the large dataset size, questions for MapIQ could not be generated manually. Instead, we developed set of manually crafted templates for each task and corresponding question type, illustrated in Figure 7. A total of 19 unique question templates were created, with placeholders such as [attribute class], [state name], and [range] randomly populated during generation. Additionally, the placeholder [USA/mo //SA/map&aa zone] was used to determine the spatial scale of tasks, allowing us to investigate both local and global spatial analysis. Example questions generated using these templates can be seen in Figure 8. Furthermore, due to the varying nature of the tasks, not every question type was eligible for task types; these exceptions summarized in Table 2.
A.1.4 Ground Truth Extraction and Validation
Ground truth answers were programmatically extracted from geospatial metadata (Geoj- SON files) using Python scripts, without any visual map inspection. Given access the original metadata and the objective, factual nature of our questions (e.g., "What is the attribute class of AL?"), programmatic extraction was both feasible and more accurate than manual annotation. Scripts were developed for each Task QupestQustoon Type combination to systematically extract the required information.
For example, for multiple-choice question under the Retrieve Value task type, the script would parse the GeoJSON file to identify the map topic and target state (e.g., Alabama), look up the ground truth value (e.g., 11% population mobility), match it to the correct option (e.g., "d. 10.0-11.2"), and return the corresponding ground truth answer (e.g., I'd']). We validated representative samples and refined the scripts iteratively based consistent error patterns to ensure accuracy. The ground truth responses are provided in the supplementary materials (OSF: Folder Name Ground Truth Responses).
15

---

Published as a conference paper at COLM 2025
Task Type
Question Type
Template
What is the attribute class of (state namej?
[attribute class]
b. [attribute class]
mcq
(attribute class
retrieve
(attribute class]
None of the above
binary
True or False: The attribute class of [state name] is [attribute class]
single value
What is the attribute class of (state name]?
list
List the states with the attribute class (attribute class].
Which state(s) have an attribute class [less/greater] than (state name 1)?
[state name]
(state name]
pairwise point comparisons
[state
(state name]
None of the above
binary
True False: The attribute class of [state name 1] is [less/greater] than (state name 2].
list
List the states with an attribute class [less/greater] than [state name 1].
the [USA/map zone]. which state(s) have the [lowesthighest] attribute class?
name]
[state name]
spatial extremes
(state name)
(state name]
None of the above
binary
True False: In the [USAlmap zone], [state name] has one of the [lowes/highest] attribute class.
list
the [USA/map zone], list the states with the [lowesthighest] attribute class.
Local: Which zone(s) on the have Class [1/5] cluster?
Northeast
Midwest
South
None of the above
moq
spatial clusters
Global: How many Class [1/5] clusters can you identify on the map?
More than
binary
True or False: There is a Class [1/5] cluster [on the maplin the map zone).
list
List all the spatial clusters you can identify [on the maplin the map zone)
What is the range of attribute value in the [map zone]? [Range]
[Range]
mcq
(Range)
determine
[Range]
range
None above
True or False: In the [map zone]. the range of attribute value between (Range Lower Limit] to
binary
|Rngee Upper Limit)
single value
What is the range of attribute value in the [map zone]?
Which of the following zone(s) generally have [lowerhigher] attribute value than the [map
zone?
[map zone]
mcq
[map
regional comparisons
[map zone]
None of the above
binary
True False: The [map zone 1] generally has [lower/higher] attribute value than the [map
2).
list
the zone(s) generally have [lowenhigher] attribute value than [map zone 1) zone.
Figure 7: Question templates for each Task Type, organized by corresponding Question Type.
A.1.5 Comparing MapIQ with Existing Map-VQA Datasets
Benchmark datasets focused on low-level map-reading tasks are still in their early stages, with limited coverage terms of map diversity, task types, and topical themes. MapIQ advances this space by offering more comprehensive benchmark that expands across three dimensions: a broader range of map types (including Choropleth, Cartogram, and Proportional Symbol maps), diverse set of well-defined task types grounded in visual m analytics, and thematically rich content drawn from real-world datasets. This breadth allows for more nuanced evaluation of model capabilities across visual interpretation
16

---

Published as a conference paper at COLM 2025
Task Type
Question Type
Example Question
Retrieve Value
Binary
True or False: The attribute class of CO is 46.4-52.7
Which state(s) have attribute class less than II.? a. ID b. NJ c. OR d. MT e. None the above
Pairwise Point Comparisons
MCQ
the West zome, list the states with the highest attribute class.
Spatial Extremes
List
Sputial Clusters
Binary
True or False: There Class I cluster in the South zone.
Determine Range
Single Value
What is the range of attribute value in the Northeust zone?
Regional Comparisons
List
List the zonels) s) that generully have higher attribute value than the West zone.
Figure 8: Example questions across different task types and formats in the MaplQ dataset
Benchmark
Maps
QA Pairs
Map Types
Question Types
Task Types
Themes
MapQA !
-6K
-800K
3
MapWise
300
3,000
1
4'
3
MapIQ (ours)
774
14,706
3
4
1 MapQA (Chang et al., 2022)* MapWise (Mukhopadhyay al., 2024b) 'In MapWISE, the Single
Word, Range, and Count question types all require open-text responses consisting of single value. Therefore, they have been consolidated into the Single Value category, resulting in four distinct question types instead of the six originally reported.
Figure 9: Comparison of MaplQ with prior Map-VQA benchmarks.
and contextual understanding. Figure summarizes how MaplQ compares to existing benchmarks along these key dimensions.
A.2
Benchmarking MLLMs
A.2.1 Prompt Details
Due to the specialized nature of our benchmark, we adopted a zero-shot prompting strategy tailored our experimental setup. Each prompt followed a standardized format consisting of a zoning instruction (where applicable) followed by general instruction. The general instructions used across tasks are illustrated in Figure 10. The zoning instruction introduced zone mapping for the contiguous U.S. states, which was stated as follows: "In the provided data, the states in the USA are classified into four zones: West, Midwest, Northeast, and South. Use this classification to answer question based on map."
In addition, for the Spatial Clusters task, we incorporated supplementary information in the General Instruction, to define the concept of spatial cluster, stated as follows: "A spatial cluster consists of geographically proximate locations sharing the same attribute class. You will be shown map with data categorized into five classes and asked to identify spatial clusters. Only Class (lowest) and Class 5 (highest) qualify as clusters, while Classes 2-4 do not. Clusters follow the Queen adjacency rule, meaning all states in the cluster must be connected without leaving the set. Each cluster must contain at least three states, with no upper limit, and multiple clusters can exist within the same class."
A.2.2 Test Set Sampling
For sampling the test set for the benchmarking experiment, we first identified all valid combinations of the experimental variables: map type, task type, question type, and theme. Since not all question types were compatible with every task type, we calculated the total number of valid combinations follows: (map types) 6 (task types) (themes) x
17

---

Published as a conference paper at COLM 2025
Task Type
Question Type
General Instruction
You will be shown map and asked to identity the attribute class of specific stale. Choose the correct aption from the given choices. There only one oo rect answer. Format your resporse as: My enswer Your Optonj'
maq
binary
You wil be shown map and ssked whether state's sttribute class metches a given value, Answer only with True' or False'
rotrovo
single
You will be clesa of
and ashed question your resporse "My
legand.
identiyy the attribute specific state. Your (uttributt class)
Hist
You will be shown map and asked to ist all statee that match a specific attribute class. Provide your answer isst of two letter state abbroviations. Format your resaonse as. Myy answvrr is (Ist of state abbrevictions)
You will be shown map and ssked to identily the coneat statees) based on the given c-fteria, Select al appiiable options from the provided choices. There may be more than one correct answer. Format your resporse ac: 'My anowor is (Your Optionjal"
maq
painwise paint comparisona
binary
You will be Falso
map
compere the attribute claaseo stotes. only
You will be shown a map and asked to ist all states with attribute class (esss/graaterj than given state. Format reeponse 'Myy answer jist state abbreviations
lil
You will be asked identify the statels) with the [ewestthighee] attribute class within a specified zone. Chaose the corect optiong)) from the given choices. There may be more than one cornect answer. Format your resporiee ss: Myy answer [Your Optionéll]
moq
opatial extromes
binary
You be asked whothor a speañi or False'
the [looot/highoet] attribute
given
zone. Answer only with
list
You will be asked list the statos with the [cwwes/highest] attribune class within specified zone. Format your resporis 'My answer [let
abtreviatione)
Supplemantary Instruction Choose the correct actionts from the givon choices. There may be more than one correct answer. Formal your reeponse ex 'My answer la (Your Optiont8l
maq
bintry
only Supplementary Instr.ction Format your response as followe:
Supplementary
'Felee
Cless 1:
spetial clusters
Cluster [Ist state abbrevistione)
Chaater [Ist sbbrrovisiiona)
... on
Clans 8:
Diuster [Est of state abbraviationa)
Cluster 2: [ist of state abbrevistione]
a.d so on
You saked idinttily the range of attrizule valuse within a specified zone. Chocae the connec option from the given choices. There is only one correct answer: Format your resporso as: 'My answer is [Your Option]
moq
cleteriin range
binary
You will be asked whather attribute range specified only with
given limits. Answver
You will bo asked identtfl the range of attriauta values within a specified zone. Formet your response as: My answer is [Range Lower Limit - Range Upper Liml''
single valuee
You will be asked identify which zone(s) generally have Jower/iighed attribute value than a given zone.
moq
Cheose the comect optionss) from the given choices. There may be more than ane cornect answer. Format your response as: 'My answer is [Your Optionéal]
regional comparitone
binary
You will be asked whether one zone generally [lowerhigher attribute value another, Anewer only with or 'False'
list
You will be asked list the zanejg) that generally have Jowennigher attribute value than given zone.
Formet your response MMy answer list
Figure 10: General instructions provided as part of the prompts for each Task Type and Question Type, shared with MLLMs and human participants.
2 (question types applicable to all tasks-binary and MCQ) = 216 combinations; 3 (map types) (task types: Retrieve Value and Determine Range) 6 (themes) x 1 (question type: single-value) = 36 combinations; 3 (map types) x 5 (all task types except Determine (question type: list) 90 combinations.
Range) x (themes)
This yielded a total of 342 unique combinations. We then selected 15 QA pairs per combi- nation to ensure uniform representation, resulting balanced sample of 5130 QA pairs. For instance, the combination "map type: Cartogram, task type: Retrieve Value, theme: Economic, question type: MCQ" included exactly 15 QA pairs the final dataset (see Fig 11 for an example QA pair from this combination). To ensure topical diversity, map topics within each theme were randomly selected without replacement. While the test set maintained perfect balance across map types and themes, minor imbalances remained across task and question types due to inherent compatibility limitations between certain tasks and question formats.
18

---

Published as a conference paper at COLM 2025
Percentage of Population Not in the Labor Force, 2023
ME
SD
OK
FN
IU
CT
OKL
W
CO
XE
MBO
CA
L4
SC
XX
GA
29.8-32.4
32.5-35.3
35.4-38.7
38.8-45.5
45.6
Question: Which state(s) have an attribute class less than AR? a. MS
b. LA
c. MD d. PA
e. None of the above
Ground Truth Answer: c, d
Figure 11: Sample test dataset map with the following variable
Type: Car-
togram, Task Type: Pairwise Point Comparisons, Question Type: MCQ, Theme: Economic. The map displays the question alongside multiple-choice options and the ground truth answer.
A,2.3 MLLM Response Validation
Manual postprocessing was performed across the entire test dataset for all model responses to ensure consistent and fair evaluation. Despite providing explicit formatting instructions (e.g., "Format your response as: 'My answer [Your Option|")), models frequently included extraneous content such as reasoning steps, explanations, or hallucinated information alongside their answers. Manual validation was essential extract the actual answers, enforce consistent formatting, and prevent formatting variations from artificially penalizing model performance during evaluation.
For example, for a Determine Range question, Molmo returned: "To answer this question, I need to: [reasoning steps...] After examining the map and legend, Washington's circle falls into the second largest category, which corresponds to the 99.9-163 range. My answer is 99.9-163." This was cleaned to [99.9, 163] for evaluation. Similarly, Idefics responded "No." " to Binary question and was cleaned to FALSE for consistency. This process was repeated systematically across the entire test set to ensure accurate and reproducible evaluations. All raw model responses and their cleaned versions provided in the Supplementary Materials (OSF: "MLLM Responses/ Baseline").
It important to distinguish this validation process from the human baseline evaluation. Ground truth extraction and MLLM response validation were carried out by single human, distinct from the two humans involved in human baseline evaluation.
A.2.4 Sensitivity Comparison with Human Baseline
We further analyzed the sensitivity of MLLM and human performance across the four primary experimental variables using the standard deviation of performance across vari- ables. Figure 12 shows that the standard deviation in MLLM performance consistently exceeded that of humans across all variables, indicating greater sensitivity to contextual
19

---

Published as a conference paper at COLM 2025
Standard Deviation
14.32
MLLLMs Overall
10-
3,89
Humans Baseline
253
0.3
1.7
Task Type
Map Type Theme Question Type
Experiment Variable
Figure 12: The standard deviation of human baseline and MLLMs across experiment variables.
changes. Notably, the highest variability was observed with respect to Task Type (11.46%) and Question Type (14.3%%). While such variations were expected due differences in task and question complexity, the significantly higher sensitivity of MLLMs suggests that these models are more affected by changes in difficulty than human readers. Additionally, whereas human performance remained relatively stable across different map types and themes, MLLMs showed more variability, suggesting potential biases toward specific map types and thematic content.
A.3 Map Design Variations
A.3.1 Exceptions
While we aimed apply all 15 map design variations uniformly across all three map types and task types, differences in visual encoding made this infeasible. For instance, color scheme variations were not applied to Proportional Symbol maps, which rely on symbol size rather than color. Similarly, the Determine Range and Retrieve Value tasks require a legend, making the "No Legend" variation incompatible. To ensure fair evaluation in maps without titles legends, minimal prompt edits (e.g., "darker shades indicate higher class values") preserved essential contextual information. Figure 13 presents illustrative examples of the tested design variations.
20

---

Published as a conference paper at COLM 2025
Pansaage coibiiyg oxgle bendetk 2003
Original Map
4
03/04
1149
R.4M
41455
(b) Spectral Color Scheme
(c) Legend Orientation
Zerenaage colcktika oogle locodalss 2013
Peseerage of cclatiikk booleloooowll.... 2013
1142
AIRZ
(d) Map Rotated X Axis
(e) Color Divergent Variant
cogle
copple
620.
2,020
3140
A103
Figure 13: Four examples of map design variations compared to the baseline design (a). (b) and (e) depict changes in color scheme, (c) illustrates a modification in legend orientation, and (d) shows a rotated map layout.
21


Reviews:
[{"review_id": "dm4p1hBAQ9", "reviewer": "Reviewer_Kk1H", "summary": "The paper introduces **MapIQ**, a new benchmark for evaluating multimodal large language models (MLLMs) on map-based visual question answering (Map-VQA). It expands beyond prior benchmarks by including three map types—choropleth, cartogram, and proportional symbol maps—and questions derived from six thematic domains like crime, housing, and environment. The benchmark consists of over 14,000 QA pairs targeting six different visual analytic tasks (e.g., value retrieval, spatial clusters, regional comparisons) and various question formats (binary, multiple choice, list, single value). The authors evaluate seven MLLMs, including closed-source (Claude 3.5, Gemini 1.5, GPT-4o) and open-source (Qwen2-VL, InternVL2.5-MPO, etc.) models, comparing their performance across tasks, map types, and question formats against a human baseline. They also study model robustness by introducing 15 map design variations (e.g., rotated maps, removed legends, color flips). Claude 3.5 performs best overall, but all models significantly underperform humans, especially in tasks like range detection and interpreting proportional symbol maps.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The benchmark fills a clear gap in current VQA datasets by extending evaluation from common charts (e.g., bar graphs) to geospatial maps, a domain with different cognitive demands and visual representations. Including three distinct map types and six task types allows for a much broader and more nuanced evaluation of MLLMs' spatial reasoning and visual grounding. The paper is also methodologically rigorous—it controls for thematic domains, question types, and task difficulty, and includes both closed- and open-source models, enabling fair comparisons. The added robustness study on map design variations is a thoughtful touch, revealing how fragile some models are to simple visual tweaks.\n\nAnother strength is the introduction of **thematic bias analysis**, probing whether MLLMs rely on real-world topic priors (e.g., assuming higher crime rates in certain regions) rather than actual map interpretation. This goes beyond accuracy and touches on model behavior, which is often underexplored. The dataset is also well-constructed: questions are grounded in geospatial literature, carefully templated, and validated, while the maps follow strong cartographic principles (e.g., consistent color schemes, spatial normalization), making the benchmark high-quality and reusable.", "reasons_to_reject": "While the benchmark introduces different map types, the task types themselves are still relatively low-level (e.g., identifying max/min values, comparing two states). The benchmark doesn't evaluate more complex or realistic tasks like navigation, multi-hop spatial reasoning, or causal inference across maps, which could better showcase the capabilities of MLLMs. There's also a lack of diversity in geographic resolution—it only includes U.S. state-level maps, missing finer-grained (e.g., county or city) or international data.\n\nThe analysis of model failure modes is somewhat shallow. Although Claude 3.5 performs best, the paper doesn't deeply explore *why* certain models fail—e.g., whether it’s due to poor legend parsing, weak visual grounding, or reliance on text priors. Some question types (like single-value answers) seem to trip up models significantly, but the authors don’t investigate how to fix that. \n\nVery Minor: While seven models are tested, many of the stronger open-source MLLMs (e.g., Qwen2.5VL, recent Gemini API variants) are missing, and the zero-shot-only evaluation setup may underrepresent models’ true potential.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a4vTHR4Css", "reviewer": "Reviewer_2KAj", "summary": "The topic of the paper is about visual question answering with maps (Map-VQA) especially extending the range of VQA to more map types than have been considered in the past which was basically focusing only on choropleth maps to now additionally cover  cartograms and proportional symbol maps. An important part if the paper is to create and provide a new Map-VQA benchmark called MapIQ which consists of 14,706 question-answer pairs. The corpus covers six different themes and is used to evaluate six different visual analysis tasks including human baseline. Seven different Multimodal LLMs (MLLMs), both closed-source and open-source, are exploited during the evaluation using the new MapIQ benchmark. The major outcomes (Fig. 2 and 3) display that humans are quite good in answering VQA related map questions across map types and task types, where basically all tested MLLMs are still far behind human performance. Although the closed-source perform in average better than the open-source models, some of the open-source are competitive viz Qwen2-VL.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- new interesting benchmark for Map-VQA \n\n- zero-shot evaluation of different MLLMs, both closed-source and open-source\n\n- evaluation across several themes and different map types", "reasons_to_reject": "- as far as I understand all questions are instances of a small set of question templates where for each task type, where in each case three to four questions types are considered and in each case only one question template is defined (see Figure 7). This means that there is a very small (actually) no range of question paraphrases. Since the question templates seem to be very task and domain specific, the current setting of the experiments do not consider robustness of the MLLMs wrt question paraphrases\n\n- Some details are missing in sec 3.3. wrt how many human experts were involved and inter-annotator agreement.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S2cACq31wh", "reviewer": "Reviewer_inaB", "summary": "This paper introduces MapIQ, a Visual QA benchmark focused on US maps. The benchmark contains QA pairs across three map types, generated using government data from six domains: economic, housing, social, health, crime, and environmental. The authors evaluate seven MLLMs - four open-source and three closed-source - on a subset of the benchmark. Their findings show that there still exists a large performance gap between these models and human annotators. They also provide analysis across different map types, tasks, question formats, and domains. Additionally, the authors test the robustness of MLLMs’ performances by modifying map design elements such as font sizes, legends, and color schemes to measure how these variations affect performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents a benchmark highly relevant to real-world applications and reveals significant performance variations among state-of-the-art multimodal models in visual analytics tasks.  The inclusion of the varying map design experiments is also quite interesting for testing MLLMs’ robustness - how different font sizes and color schemes can lead to performance changes.", "reasons_to_reject": "**Qualitative Analysis and Useful insights** While the paper presents many quantitative results in baselines and ablations, it’s still hard to understand what type of errors do models make given different map types/question types. Given the same question, how do different model answer it if map types/question formats are different? What are the confounding factors when model makes a mistake? It is difficult to further understand or extract useful insights for future model improvement without knowing the specifics about error types. \n\n**Use of Different Domain** From Figure 10 in the appendix, it seems that all types of questions are less related to the specific data domain on which the map is based. If this is the case, can the map be generated using synthetic data drawn from a random distribution? The current setup is useful as a visual analytics dataset without eliciting any further demographic knowledge. I think the dataset would be more interesting if it could somehow incorporate some domain-specific questions.", "questions_to_authors": "- In the Validation section on page 4, you mentioned that “Removing these extraneous details was particularly time-consuming”. Is this postprocessing manually done for all test datapoints for all models? Can you providen more details?\n- Please use consistent tense across the paper. Sometimes the paper frequently switches between the present and past tense. E.g. line 78 and 164\n- Any thoughts on why Gemini 1.5 Pro is performing worse than the random baseline for the binary question in Figure 3?", "extra_scores": {}}, {"review_id": "tlebK9ipSI", "reviewer": "Reviewer_5c5R", "summary": "This paper investigates  how well multimodal large language models (MLLMs) read data visualisation. To this end, a new benchmark data set is created for visual question answering with maps (Map-VQA). This data set, MapIQ, contains 14K question-answer pairs spanning three map types, and is used to evaluate various MLLMs. In addition, the impact of map design changes is evaluated (e.g., different color schemes).", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This research provides interesting insights into the robustness of MLLMs, and their reliance on internal geographic knowledge for the task of Map-VQA, by establishing a new benchmark data set that (1) introduces two less researched map types (cartograms and proportional symbol maps) in addition to the better researched choropleth maps, (2) six VA tasks, and also offers (3) metadata sourced from six thematic categories. A variety of open-source and closed-source models are evaluated by means of this benchmark, and compared with a human performance baseline.", "reasons_to_reject": "Some parts of the methodology and evaluation are difficult to follow.\nIt is, for instance, not completely clear to me how the gold standard was created. In order to reliably evaluate the system, the only trustworthy way to create the gold standard for the test set is to manually annotate the labels. In Section 3, it is stated that the answers of the question-answer pairs were reviewed by humans to ensure accuracy. In section 4, however, the authors state than only 9% of the test set was manually validated by humans. I don’t understand how systems can be evaluated automatically with great precision if the answers have not been validated manually to create a real gold standard for the rest of the test set?", "questions_to_authors": "See remark above about the creation of the gold standard test set.\n- Some information could be added to the explanation to clarify the details of the methodology, e.g.\n* line 107 what ar the five class labels?\n* line 172 why do you assign scores of 0 or 100 to calculate accuracy? (usually accuracy just calculates the number of correct answers divided by the size of the test set).", "extra_scores": {}}]

Author rebuttal:

Meta-review: This model proposes a new benchmark for testing multimodal large language models based on questions on different types of maps, going beyond past work, which tested less detailed visual setups. The authors analyze a large set of models and highlight their main errors. Reviewers agree that this benchmark fills a gap in the current literature and will be useful for practitioners.
Decision: Accept