Can MLLMs Guide Me Home? A Benchmark Study on Fine-grained Visual Reasoning from Transit Maps

## Abstract

Abstract Multimodal large language models (MLLMs) have recently achieved significant progress in visual tasks, including semantic scene understanding and text-image alignment, with reasoning variants enhancing performance on complex tasks involving mathematics and logic.
However, their capacity for reasoning tasks involving fine-grained visual understanding remains insufficiently evaluated. To address this gap, we introduce ReasonMap , a benchmark designed to assess the fine-grained visual understanding and spatial reasoning abilities of MLLMs. ReasonMap encompasses high-resolution transit maps from 30 30 30 30 cities across 13 13 13 13 countries and includes 1 , 008 1 008 1,008 1 , 008 question-answer pairs spanning two question types and three templates.
Furthermore, we design a two-level evaluation pipeline that properly assesses answer correctness and quality.
Comprehensive evaluations of 15 15 15 15 popular MLLMs, including both base and reasoning variants, reveal a counterintuitive pattern: among open-source models, base models outperform reasoning ones, while the opposite trend is observed in closed-source models.
Additionally, performance generally degrades when visual inputs are masked, indicating that while MLLMs can leverage prior knowledge to answer some questions, fine-grained visual reasoning tasks still require genuine visual perception for strong performance.
Our benchmark study offers new insights into visual reasoning and contributes to investigating the gap between open-source and closed-source models.

## 1 Introduction

Figure: Figure 1: Overview of ReasonMap. ReasonMap is a benchmark dataset designed to evaluate fine-grained visual reasoning abilities of MLLMs, encompassing $1,008$ question–answer pairs constructed over high-resolution transit maps from $30$ cities, spanning two question types and three templates.
Refer to caption: x1.png

Multimodal large language models (MLLMs) [1, 2, 3, 4, 5] have recently achieved notable advancements across a range of vision-language tasks, including visual grounding [6, 7], reasoning segmentation [8, 9, 10, 11], and text-image alignment [12, 13].
Building upon these developments, reasoning MLLMs [14, 15, 16, 17, 18, 19, 20] have further improved performance on complex visual reasoning tasks such as visual math problems [21, 22], visual question answering (VQA) [23, 24, 22], and spatial reasoning [23, 25, 26].
These capabilities are critical for a wide range of real-world applications, including embodied AI, autonomous agents, and decision-making systems such as autonomous driving [27, 28, 29].
As multimodal tasks grow in complexity and practical relevance, the need for rigorous benchmarks to assess fine-grained visual reasoning becomes increasingly essential.

To address the growing demand for robust evaluation of multimodal reasoning, several benchmarks have been proposed.
Datasets such as MathVQA [22] and MMMU [24] incorporate multimodal questions but often permit models to succeed via shallow heuristics, without requiring genuine visual grounding.
MathVerse [30] mitigates this limitation by introducing diverse problem variants that encourage reliance on visual input.
VisuLogic [31] further enforces visual reasoning by explicitly eliminating language-only shortcuts.
Other efforts, such as VisualPuzzles [32], VGRP-Bench [33], and R-Bench [34], target logical and structural reasoning, while CityBench [35] and DriveBench [36] focus on domain-specific applications like urban tasks and autonomous driving. V^∗Bench [37] emphasizes detailed visual understanding.
MapBench [38] addresses spatial reasoning by introducing structured scene graphs for map navigation.
Despite these advances, systematic evaluation of fine-grained visual reasoning remains limited, especially for structured and information-rich diagrams like high-resolution transit maps, leaving a critical gap in existing benchmarks.

In this paper, we introduce ReasonMap (Figure [1](https://arxiv.org/html/2505.18675v2#S1.F1)), a benchmark designed to evaluate the fine-grained visual understanding and spatial reasoning capabilities of MLLMs using high-resolution transit maps.
As structured and information-dense visual artifacts, transit maps inherently require precise spatial interpretation, making them well-suited for assessing detailed visual reasoning.
ReasonMap comprises $1,008$ human-verified question-answer pairs spanning $30$ cities across $13$ countries.
Each instance includes a map, two stops, two questions (short and long), multiple reference routes, and two difficulty labels (map and question difficulty).
The questions cover two types and three prompting templates—one for short and two for long questions—capturing both coarse and fine-grained spatial reasoning.
To ensure data quality, we perform manual route verification, promote question diversity, and balance difficulty distribution.
For evaluation, we propose a two-level framework that independently measures answer correctness (via accuracy) and quality (via a proposed map score), reflecting both feasibility and efficiency in route planning.

We conduct comprehensive experiments on $15$ widely-used MLLMs, encompassing base and reasoning models.
Our results reveal a counterintuitive finding:
among open-source models, base variants outperform their reasoning counterparts, whereas the opposite holds for closed-source models.
Moreover, when only textual inputs are provided, models can still answer some questions based on inner knowledge, but in most cases, their performance noticeably drops.
This highlights a critical limitation in the current model behavior. While some models can leverage prior knowledge and textual cues to solve certain tasks, the tasks (e.g., fine-grained visual reasoning tasks) requiring genuine visual understanding still necessitate effective integration of multimodal information for robust reasoning.

Our main contributions are summarized as follows:

- •
We develop an extensible, semi-automated pipeline for dataset construction, facilitating scalable expansion to additional maps and cities. Using this pipeline, ReasonMap is constructed to evaluate fine-grained visual reasoning capabilities in MLLMs.
- •
We propose a structured two-level evaluation framework that separately quantifies answer correctness and quality using accuracy and the proposed map score, respectively, enabling fine-grained answer assessment.
- •
A comprehensive benchmarking study is conducted across $15$ MLLMs, providing insights into model performance, robustness, and the interplay between visual and textual cues, thereby informing future research on multimodal reasoning.

## 2 Related Work

Reasoning in LLMs & MLLMs.
Recent advances in large language models (LLMs) have demonstrated significant improvements in reasoning capabilities through reinforcement fine-tuning paradigms [14, 15, 39, 40], which leverage GRPO [41] to unlock the reasoning potential of LLMs. This paradigm has also been extended to the multimodal domain, with increasing interest in applying reinforcement learning (RL) to visual reasoning [42, 43, 44, 45, 46].
Both open-source and closed-source communities have introduced advanced reasoning MLLMs built upon earlier systems [3, 2, 47, 48]. Notable open-source models include Kimi-VL [16], Skywork-R1V [17, 18], and Qwen-QVQ [20], while Doubao-1.5-Pro [19], OpenAI o3 [48], OpenAI 4o [49], and Gemini [50] represent leading closed-source efforts.
Despite recent progress, systematic evaluation of fine-grained visual reasoning in MLLMs remains limited, as existing benchmarks primarily target coarse-grained tasks and fail to capture model performance on complex real-world visual content.

**Table 1: Comparison between ReasonMap and existing multimodal reasoning datasets. For entries in the dataset size column with notation like ($\times n$), each base problem has multiple versions to enforce visual grounding. Specifically, VGRP-Bench is constructed by sampling over 20 core puzzles.**
| Name | Year | Dataset Size | Avg. Resolution | Training Set | Step Evaluation | Multilingual (Count) |
| --- | --- | --- | --- | --- | --- | --- |
| MMMU [24] | 2024 | $11.5$k | $684\times 246$ | ✗ | ✗ | ✓ ($2$) |
| MathVerse [30] | 2024 | $2,612$ ($\times 6$) | $577\times 487$ | ✗ | ✗ | ✗ |
| VisuLogic [31] | 2025 | $1,003$ | $601\times 331$ | ✓ | ✗ | ✗ |
| VisualPuzzles [32] | 2025 | $1,168$ | $767\times 464$ | ✗ | ✗ | ✗ |
| VGRP-Bench [33] | 2025 | $20$ ($\times 5$) | $790\times 790$ | ✗ | ✓ | ✗ |
| R-Bench [34] | 2025 | $665$ | $629\times 348$ | ✗ | ✗ | ✓ ($2$) |
| V^∗Bench [37] | 2023 | $191$ | $2,246\times 1,582$ | ✗ | ✗ | ✗ |
| ReasonMap | 2025 | $1,008$ ($\times 2$) | $5,839\times 5,449$ | ✓ | ✓ | ✓ ($4$) |

Multimodal Reasoning Datasets.
As multimodal reasoning has rapidly progressed, various benchmarks have emerged to evaluate MLLMs across different reasoning dimensions (see summary in Table [1](https://arxiv.org/html/2505.18675v2#S2.T1)). Datasets such as V^∗Bench [37], VisualPuzzles [32], VisuLogic [31], and VGRP-Bench [33] primarily examine abstract visual reasoning through synthetic tasks involving logic, structure, and pattern recognition. In parallel, CityBench [35] and DriveBench [36] shift focus to real-world spatial reasoning, assessing model performance on complex urban or autonomous driving scenarios.
For mathematical reasoning, benchmarks like MathVQA [22], MMMU [24], and MathVerse [30] integrate multimodal inputs, with MathVerse notably introducing varied problem formats to strengthen visual dependence.
Additionally, MapBench [38] employs structured scene graphs combined with CoT prompting to support navigation tasks based on manually curated and verified questions. Its image resolution, while relatively low, reflects a common characteristic shared by current datasets.
Unlike these works, we first introduce a benchmark for evaluating fine-grained visual reasoning capacities with high-resolution transit maps.

Map-based Spatial Reasoning.
Among the many directions of multimodal reasoning, map-based spatial reasoning has emerged as a crucial area, with broad applications in navigation, urban planning, and autonomous systems [51, 52, 53, 54]. Recent efforts have focused on enabling models to interpret and reason over various types of map data. CityBench [35] provides a dataset for evaluating urban scene understanding, while MapLM [55] introduces a benchmark for map and traffic scene understanding. PlanAgent [56] and PReP [57] explore embodied planning in environments that require interpreting map information.
MapEval [26] proposes a structured evaluation suite for map-related reasoning, and GeoNav [53] investigates geospatial navigation using LLMs.
Most existing methods [26, 35, 56] depend on external tools (e.g., map services or APIs) to complete spatial tasks, which often bypasses the need for genuine visual reasoning. However, spatial reasoning based solely on visual understanding remains essential. Our work targets to fill this gap by evaluating such capabilities without tool assistance.

## 3 ReasonMap Construction

Figure: Figure 2: The building pipeline of ReasonMap consists of three main stages: (1) data collection and preprocessing, (2) question–answer pair construction, and (3) quality control. Steps (2-4) in the figure correspond to the question–answer pair construction stage.
Refer to caption: x2.png

In this section, we first present the complete dataset building pipeline as shown in Figure [2](https://arxiv.org/html/2505.18675v2#S3.F2), which consists of three main stages: (1) data collection and preprocessing; (2) question–answer pair construction; and (3) quality control. We then report comprehensive statistics of the dataset.

### 3.1 ReasonMap Building Pipeline

#### 3.1.1 Data Collection and Preprocessing

We collect and manually select $30$ high-resolution transit maps covering $30$ cities across $13$ countries from publicly available online sources, in compliance with relevant licenses and regulations, ensuring diversity and a balanced range of map difficulty. We then leveraged MLLMs to extract the names of transit lines and their corresponding stops, followed by manual correction to ensure correctness. Special cases such as transfer stops and branch-starting stops were annotated in a standardized format appended to the respective stop entries. Finally, for subsequent usage, all route and stop information was saved in a unified JSON format, referred to as the Map Metadata.

#### 3.1.2 Question-Answer Pair Construction

The construction of question–answer pairs involves three key steps: (1) Question Generation, where we formulate questions based on predefined templates; (2) Reference Route Collection, where we obtain corresponding reference routes using Gaode Map(^1^11[https://console.amap.com/dev/index](https://console.amap.com/dev/index)) and Google Map(^2^22[https://developers.google.com/maps/apis-by-platform](https://developers.google.com/maps/apis-by-platform)); and (3) Label Annotation, where we assign difficulty levels for both the maps and the questions.

Question Generation.
We randomly select two stops (refer to stop1 and stop2) from the current high-resolution transit map. We then generate one short question and one long question based on predefined question templates and two stops (Figure [2](https://arxiv.org/html/2505.18675v2#S3.F2)). The short question has only one fixed template, while the long question is randomly assigned one of two available templates during generation. Additionally, the two long question templates differ in focus: one asks for the number of via stops, while the other requires identifying each via stop (see detailed templates in Appendix [A.1](https://arxiv.org/html/2505.18675v2#A1.SS1)).

Reference Route Collection.
For each question, we query all valid transit routes between stop1 and stop2 using APIs from map services (e.g., Gaode Map for Chinese cities and Google Map for other cities). The retrieved routes are stored in a unified format containing relevant metadata (e.g., route name, departure stop, arrival stop, via stops, and number of via stops). We discard routes that cannot be visually traced on the map, ensuring consistency with the visual content.

Label Annotation.
Two levels of difficulty labeling are included in this stage. For map difficulty, we manually assign each map to one of three difficulty levels (easy, medium, hard), ensuring a balanced split across $30$ maps, with $10$ maps per level. For question difficulty, we assign difficulty based on the number of transfers in the reference route: routes with no transfers are labeled as easy, those with one transfer as medium, and all others as hard. To ensure balance, we set a fixed difficulty distribution threshold of $20:15:5$ (easy:medium:hard) for each map, generating $40$ questions. Once the quota for a difficulty level is reached on a given map, no additional questions of that level are retained.

#### 3.1.3 Quality Control

To ensure the reliability and balance of the dataset, we perform quality control from three perspectives: correctness, diversity, and difficulty balance. Incorrect question–answer pairs are either manually corrected or discarded. We then involve both automatic checks and manual adjustments to ensure consistency and coverage across all difficulty levels. One reserved example is shown in Figure [1](https://arxiv.org/html/2505.18675v2#S1.F1).

### 3.2 Dataset Statistics

The ReasonMap consists of $30$ high-resolution transit map images (see map sources in Appendix [A.2](https://arxiv.org/html/2505.18675v2#A1.SS2)) with an average resolution of $5,839\times 5,449$ pixels. In total, it contains $1,008$ question–answer pairs, including stop names in four languages (e.g., English, Hungarian, Chinese, and Italian). The distribution of question difficulty is as follows: $57.7\%$ are labeled as easy, $34.4\%$ as medium, and $7.8\%$ as hard. Additionally, a subset of $312$ samples is manually selected as the test set for the benchmark experiments described in Section [5](https://arxiv.org/html/2505.18675v2#S5), while the remaining samples serve as a training set for future use.
To ensure diversity and difficulty balance, the test set includes $11$ cities with a $4:3:4$ map difficulty ratio and a question difficulty distribution ($181$ easy, $108$ medium, $23$ hard) that maintains consistency with the full dataset.

## 4 Evaluation Framework

This section systematically introduces a two-level evaluation framework for assessing model performance on the ReasonMap. This framework separately evaluates the correctness and quality of answers produced by models. Specifically, we quantify correctness using accuracy and design map score to measure the quality of answers, considering multiple factors (e.g., route efficiency and alignment with the reference routes from map services).

Preparation for Evaluation.
We first parse the model-generated answers according to the required format. Answers that do not comply with the specified format or cannot be parsed due to model hallucination [58] are marked as invalid. Invalid responses are excluded from subsequent evaluations, with accuracy and map score set to zero. For the correctness evaluation, we utilize the Map Metadata mentioned in Section [3.1.1](https://arxiv.org/html/2505.18675v2#S3.SS1.SSS1) as ground truth. For the quality evaluation, we adopt the collected reference routes as presented in Section [3.1.2](https://arxiv.org/html/2505.18675v2#S3.SS1.SSS2) as the ground truth.

### 4.1 Correctness Evaluation

We evaluate the correctness of the answer using Algorithm [1](https://arxiv.org/html/2505.18675v2#algorithm1) in Appendix [B](https://arxiv.org/html/2505.18675v2#A2). Specifically, the evaluation checks the correctness of the overall departure and arrival stops (stop1 and stop2), verifies if the route name of each segment exists in the Map Metadata, ensures the departure and arrival stops are valid for each segment, and confirms that transfer stops between consecutive segments are consistent. An answer is considered correct only if all the above checks are satisfied. Additionally, we apply the same correctness evaluation algorithm to the answers of short and long questions.

### 4.2 Quality Evaluation

To evaluate the quality of the answers, we introduce a unified scoring metric, referred to as the map score, which is applied to both short and long questions using the evaluation procedure (see Algorithm [2](https://arxiv.org/html/2505.18675v2#algorithm2) in Appendix [B](https://arxiv.org/html/2505.18675v2#A2)). The overall evaluation framework for route quality follows a structure similar to that used in Section [4.1](https://arxiv.org/html/2505.18675v2#S4.SS1). The following evaluation procedure assumes a single reference route for simplicity. In practice, if multiple reference routes are available, the answer is evaluated against each of them, and the highest score is taken as the final map score.

For short questions, the map score solely focuses on route-level and endpoint consistency, excluding all long-question-specific parts. We compute the score by comparing segment pairs in the answer and reference route. Specifically, correctly matching stop1 and stop2 contributes one point, matching the route name adds two points, and matching the departure and arrival stops within each route segment provides one point each. The final score is capped at $10$, and an additional bonus is awarded if the answer is judged correct based on the correctness evaluation procedure described in Section [4.1](https://arxiv.org/html/2505.18675v2#S4.SS1). This design ensures that a correct answer always receives a higher score than any incorrect one.

For long questions, the evaluation incorporates additional scoring components tailored to the two question templates introduced in Section [3.1.2](https://arxiv.org/html/2505.18675v2#S3.SS1.SSS2). These components are designed to capture the increased reasoning depth required in long-form responses. As with short questions, a bonus score is also added for correct answers. The two additional scoring components are detailed below.

Via Stop Count Evaluation.
For long questions that require models to predict the number of via stops for each segment, we introduce the num_via_stop_score. This score compares the via stop count of the answer and reference route by computing the absolute error and mapping it to a fixed score ($4$). A perfect match yields full points, while larger discrepancies receive proportionally lower scores. The score is then capped at $10$ for the full route.

Specific Via Stop Evaluation.
For long questions that require explicit enumeration of intermediate stops, we compute via_stop_score using a combination of two factors: the number of correctly matched via stops, and the intersection-over-union (IoU) between via stop sets of the answer and reference route. The final score for this component is obtained by averaging the IoU score (scaled to $10$) and the exact match count (capped at $10$), and then clipped to a maximum of $10$.

## 5 Experiments

**Table 2: Evaluations of various MLLMs on ReasonMap. $S.$ represents results for short questions, while $L.$ denotes results for long questions. The map score is capped at 20 for short questions, while for long questions, the maximum score is 40. Bold indicates the best results among open-source and closed-source models, respectively, while underline represents the second best.**
| Model | Type | Weighted Acc. ($S.$) | #Tokens ($S.$) | Weighted Acc. ($L.$) | #Tokens ($L.$) | Weighted Map Score ($S.$ / $L.$) |
| --- | --- | --- | --- | --- | --- | --- |
| Open-source Models |  |  |  |  |  |  |
| Qwen2.5-VL-3B-Instruct [2] | Base | $8.68\%$ | $42$ | $7.99\%$ | $151$ | $2.75$ / $3.70$ |
| Qwen2.5-VL-32B-Instruct [2] | Base | $16.49\%$ | $36$ | $15.71\%$ | $112$ | $3.88$ / $6.84$ |
| Qwen2.5-VL-72B-Instruct [2] | Base | $\mathbf{26.65\%}$ | $33$ | $\mathbf{24.22\%}$ | $104$ | $\mathbf{5.09}$ / $\mathbf{8.80}$ |
| InternVL3-38B [3] | Base | $14.84\%$ | $43$ | $13.45\%$ | $68$ | $3.48$ / $6.31$ |
| InternVL3-78B [3] | Base | $\underline{25.35\%}$ | $33$ | $\underline{19.62\%}$ | $62$ | $\underline{4.80}$ / $\underline{7.50}$ |
| Kimi-VL-A3B-Instruct [16] | Base | $12.76\%$ | $41$ | $12.33\%$ | $41$ | $3.30$ / $5.37$ |
| Kimi-VL-A3B-Thinking [16] | Reasoning | $5.47\%$ | $754$ | $5.47\%$ | $1,287$ | $2.44$ / $3.17$ |
| Skywork-R1V-38B [17, 18] | Reasoning | $6.86\%$ | $645$ | $3.21\%$ | $842$ | $2.11$ / $3.11$ |
| QvQ-72B-Preview [20] | Reasoning | $9.03\%$ | $1,279$ | $4.25\%$ | $1,619$ | $1.59$ / $1.55$ |
| Closed-source Models |  |  |  |  |  |  |
| Doubao-115 [19] | Base | $34.20\%$ | $32$ | $38.02\%$ | $118$ | $5.25$ / $11.96$ |
| OpenAI 4o [49] | Base | $41.15\%$ | $34$ | $42.80\%$ | $58$ | $6.84$ / $13.57$ |
| Doubao-415 [19] | Reasoning | $43.14\%$ | $536$ | $\underline{46.09\%}$ | $1,796$ | $7.33$ / $\underline{14.67}$ |
| Doubao-428 [19] | Reasoning | $37.15\%$ | $532$ | $37.85\%$ | $2,167$ | $5.52$ / $11.73$ |
| Gemini-2.5-Flash [50] | Reasoning | $\underline{46.09\%}$ | $806$ | $29.86\%$ | $1,419$ | $\underline{7.64}$ / $9.98$ |
| OpenAI o3 [48] | Reasoning | $\mathbf{63.02\%}$ | $1,236$ | $\mathbf{59.11\%}$ | $2,372$ | $\mathbf{9.53}$ / $\mathbf{17.96}$ |

### 5.1 Experimental Setups

We conduct extensive benchmark experiments on ReasonMap using $15$ popular MLLMs under different inference settings, analyzing their performance and comparing results. Several interesting insights emerge from this comparison. The detailed experimental settings are described below.

Evaluated Models.
We evaluate a diverse set of MLLMs categorized into two groups based on whether they are reasoning-oriented models with a long-thinking process. Reasoning models include:
Skywork-R1V-38B [17, 18], QvQ-72B-Preview [20], Kimi-VL-A3B-Thinking/Instruct [16], OpenAI o3 [48], Gemini-2.5-Flash [50], Doubao-1-5-thinking-vision-pro-250428 (Doubao-428), and Doubao-1.5-Thinking-Pro-M-250415 (Doubao-415) [19]. Base models include: Qwen2.5-VL series (3B, 32B, 72B) [2], InternVL3 series (38B, 78B) [3], OpenAI 4o [49], and Doubao-1.5-Vision-Pro-32k-250115 (Doubao-115) [19]. Additionally, the Doubao 1.5 Pro series has an activated parameter size of $20$B.

Inference Settings.
For open-source models, we set the max output token limit to $2,048$, while keeping other parameters consistent with the official HuggingFace configurations. All open-source models are deployed using PyTorch and the HuggingFace Transformers library on $8$ NVIDIA A100 GPUs. For closed-source models, we access their official APIs for evaluation and follow the default settings provided by each model’s official documentation. We further discuss the diverse image processing strategies when handling high-resolution visual inputs in Appendix [D](https://arxiv.org/html/2505.18675v2#A4).

Difficulty-Aware Weighting.
To better reflect the varying complexity of different samples, we adopt a difficulty-aware weighting strategy based on the combination of question difficulty and map difficulty. Specifically, each difficulty pair is assigned a predefined weight, with harder combinations receiving higher values. The complete weight matrix is provided in Appendix [B.2](https://arxiv.org/html/2505.18675v2#A2.SS2). Both accuracy and map score are evaluated using this weighting scheme, ensuring that models are more strongly rewarded for correctly solving more challenging examples.

### 5.2 Experimental Results

#### 5.2.1 Performance of MLLMs with Full Input

Figure: Figure 3: Accuracy across difficulty combinations for four representative MLLMs (Qwen2.5-VL-72B-I, InternVL3-78B, OpenAI o3, and Doubao-415). Each difficulty combination is denoted by a pair (e.g., easy-hard), where the first term indicates question difficulty and the second term represents map difficulty. The pair (hard-middle) contains only one sample, leading to an accuracy of 100%. We summarize the number of evaluation samples in each difficulty bucket: $55$ samples for easy-easy, $46$ for easy-middle, $28$ for middle-easy, $7$ for hard-easy, $23$ for middle-middle, $80$ for easy-hard, $1$ for hard-middle, $57$ for middle-hard, and $15$ for hard-hard.
Refer to caption: x3.png

The principal results are summarized in Table [2](https://arxiv.org/html/2505.18675v2#S5.T2). Notably, we observe a counterintuitive phenomenon: among open-source models, reasoning models consistently underperform their base counterparts, whereas the opposite holds in the closed-source setting(^3^33Although the comparison across closed-source models may not be fair due to lack of transparency in details, the reasoning variants exhibit stronger performance in this category.). Prior work suggests that RL may improve sample efficiency without introducing fundamentally new reasoning abilities [59, 60, 61], while RL-trained models tend to bias their output distributions toward high-reward trajectories, which helps produce more correct responses but may simultaneously constrain the model’s exploration capacity and reduce its ability to leverage broader foundational knowledge. In addition, recent studies indicate that multimodal models may sometimes rely on inner knowledge priors instead of truly attending to visual inputs [62, 63, 64, 30]. This tendency is further supported by the results in Section [5.2.2](https://arxiv.org/html/2505.18675v2#S5.SS2.SSS2), where open-source models still maintain part of their performance even without visual input, indicating limited visual grounding. In contrast, closed-source reasoning models outperform their base variants. One possible explanation lies in the broader knowledge coverage and better visual integration observed in these models [19, 48, 50].

We further analyze the effect of model size by examining performance within the same architecture series. Qwen2.5-VL and InternVL series show a consistent trend: larger models achieve better accuracy with fewer tokens, suggesting that the scaling law [65] continues to hold even in fine-grained visual reasoning tasks. Figure [3](https://arxiv.org/html/2505.18675v2#S5.F3) presents accuracy distributions across different combinations of question and map difficulty. As expected, performance degrades as task complexity increases.
Additionally, Figure [4](https://arxiv.org/html/2505.18675v2#S5.F4) illustrates accuracy variation across cities. We observe a negative correlation between map difficulty and accuracy. Moreover, model performance varies notably even among cities with comparable map difficulty levels. This disparity can be partially attributed to factors such as city prominence and the language used for stop names, both of which are closely tied to the model’s pretrained knowledge. For instance, OpenAI o3 performs significantly better on complex cities like Singapore compared to Hangzhou, likely due to Singapore’s higher international visibility and the use of English stop names, whereas Hangzhou is less prominent and its stop names are Chinese.

Figure: Figure 4: Accuracy across different cities for four representative MLLMs (Qwen2.5-VL-72B-I, InternVL3-78B, OpenAI o3, and Doubao-415). Each city is marked with the corresponding map difficulty and the country flag. Each city in the test set provides a specific number of samples per model: $32$ samples for Auckland, $34$ for Los Angeles, $7$ for Miami, $35$ for Lisboa, $18$ for Geneva, $40$ for Beijing, $39$ for Hangzhou, $17$ for Budapest, $39$ for Singapore, $40$ for Rome, and $11$ for Toronto.
Refer to caption: x5.png

**Table 3: Evaluations of various MLLMs on ReasonMap without visual inputs. $S.$ represents results for short questions, while $L.$ denotes results for long questions. The map score is capped at 20 for short questions, while for long questions, the maximum score is 40. Bold indicates the best results among open-source and closed-source models, respectively, while underline represents the second best. Green highlights improved results compared to the full input setting (Table [2](https://arxiv.org/html/2505.18675v2#S5.T2)), while red indicates performance drops.**
| Model | Type | Weighted Acc. ($S.$) | #Tokens ($S.$) | Weighted Acc. ($L.$) | #Tokens ($L.$) | Weighted Map Score ($S.$ / $L.$) |
| --- | --- | --- | --- | --- | --- | --- |
| Open-source Models |  |  |  |  |  |  |
| Qwen2.5-VL-3B-Instruct [2] | Base | $9.38\%$ $\!\uparrow\!$ $0.7\%$ | $47$ | $9.72\%$ $\!\uparrow\!$ $1.73\%$ | $147$ | $2.93$ $\!\uparrow\!$ $0.18$ / $4.51$ $\!\uparrow\!$ $0.81$ |
| Qwen2.5-VL-72B-Instruct [2] | Base | $\mathbf{16.41\%}$ $\!\downarrow\!$ $10.24\%$ | $28$ | $\mathbf{15.71\%}$ $\!\downarrow\!$ $8.51\%$ | $108$ | $\mathbf{4.03}$ $\!\downarrow\!$ $1.06$ / $\mathbf{6.49}$ $\!\downarrow\!$ $2.31$ |
| Kimi-VL-A3B-Instruct [16] | Base | $\underline{11.81\%}$ $\!\downarrow\!$ $0.95\%$ | $41$ | $\underline{9.81\%}$ $\!\downarrow\!$ $2.52\%$ | $49$ | $\underline{3.37}$ $\!\uparrow\!$ $0.07$ / $\underline{5.32}$ $\!\downarrow\!$ $0.05$ |
| Kimi-VL-A3B-Thinking [16] | Reasoning | $4.17\%$ $\!\downarrow\!$ $1.30\%$ | $1,039$ | $2.08\%$ $\!\downarrow\!$ $3.39\%$ | $1,755$ | $2.06$ $\!\downarrow\!$ $0.38$ / $1.64$ $\!\downarrow\!$ $1.53$ |
| Closed-source Models |  |  |  |  |  |  |
| Doubao-115 [19] | Base | $\underline{13.72\%}$ $\!\downarrow\!$ $20.48\%$ | $34$ | $\underline{13.98\%}$ $\!\downarrow\!$ $24.04\%$ | $99$ | $\underline{3.50}$ $\!\downarrow\!$ $1.75$ / $\underline{6.48}$ $\!\downarrow\!$ $5.48$ |
| Doubao-415 [19] | Reasoning | $\mathbf{21.53\%}$ $\!\downarrow\!$ $21.61\%$ | $352$ | $\mathbf{17.19\%}$ $\!\downarrow\!$ $28.90\%$ | $1,047$ | $\mathbf{4.85}$ $\!\downarrow\!$ $2.48$ / $\mathbf{7.68}$ $\!\downarrow\!$ $6.99$ |

#### 5.2.2 Performance of MLLMs without Visual Input

To further investigate the reliance of MLLMs on visual input, we selected representative open-source and closed-source models for additional experiments, where the visual input was masked. The results are reported in Table [3](https://arxiv.org/html/2505.18675v2#S5.T3). We observe that while most models can leverage internal knowledge to answer certain questions, their performance generally declines to varying degrees when visual input is removed, with the decline being more pronounced among closed-source models. Model performance is positively correlated with the performance drop after masking visual inputs, indicating effective use of visual information. In contrast, models like Qwen2.5-VL-3B-I show minimal or even improved performance, suggesting a reliance on inner knowledge rather than genuine visual reasoning.

### 5.3 Error Analysis

Figure: Figure 5: Error case analysis of various MLLMs using ReasonMap. For reasoning models, the reasoning process is explicitly marked with <think> and </think> tags. We highlight error contents in the answers with red and categorize them accordingly.
Refer to caption: x7.png

Figure [5](https://arxiv.org/html/2505.18675v2#S5.F5) presents representative failure cases from ReasonMap, revealing several recurring error types. A common issue is visual confusion, where the model misidentifies the transit line due to similar colors or adjacent layouts, for instance, mistaking Line $9$ for Line $16$ (OpenAI o3, left column; Doubao-428, right column). Another frequent problem is format errors, where responses deviate from the required structure, making them unprocessable despite containing correct route information (Doubao-115 and QvQ-72B-Preview, left column).
We also observe instances of hallucination [58], where the model repeats the correct answer (Kimi-VL-A3B-Thinking, middle column) or generates information that is not present in the input, such as mentioning image generation, as seen in Skywork-R1V-38B (right column).
Refusal cases are also present, where models explicitly decline to answer (Skywork-R1V-38B, middle and right column). Notably, these errors may occasionally co-occur within a single response (Skywork-R1V-38B, right column). These behaviors highlight the limitations in visual grounding and response robustness, especially when handling fine-grained visual details (see more case analyses in Appendix [C](https://arxiv.org/html/2505.18675v2#A3)).

## 6 Conclusion

In this work, we introduce ReasonMap, a benchmark designed to evaluate the fine-grained visual understanding and spatial reasoning capabilities of MLLMs using high-resolution transit maps.
Through a semi-automated and scalable data building pipeline, we curate a diverse set of human-verified question-answer pairs across $30$ cities.
Our two-level evaluation framework enables a nuanced assessment of both correctness and quality.
Experimental results on $15$ popular MLLMs reveal key insights into model behavior, highlighting performance gaps between base and reasoning models, as well as the crucial role of visual input.
These findings underscore the need for more rigorous evaluation and training approaches to advance visual reasoning in multimodal systems.

## References

- [1]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.
Gpt-4 technical report.
arXiv preprint arXiv:2303.08774, 2023.
- [2]
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al.
Qwen2. 5-vl technical report.
arXiv preprint arXiv:2502.13923, 2025.
- [3]
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al.
Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.
arXiv preprint arXiv:2504.10479, 2025.
- [4]
Yangliu Hu, Zikai Song, Na Feng, Yawei Luo, Junqing Yu, Yi-Ping Phoebe Chen, and Wei Yang.
Sf2t: Self-supervised fragment finetuning of video-llms for fine-grained understanding.
arXiv preprint arXiv:2504.07745, 2025.
- [5]
Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jie Qin, Jianke Zhu, and Lei Zhang.
Tokenpacker: Efficient visual projector for multimodal llm.
arXiv preprint arXiv:2407.02392, 2024.
- [6]
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei.
Kosmos-2: Grounding multimodal large language models to the world.
arXiv preprint arXiv:2306.14824, 2023.
- [7]
Jingru Yang, Huan Yu, Yang Jingxin, Chentianye Xu, Yin Biao, Yu Sun, and Shengfeng He.
Visual-linguistic agent: Towards collaborative contextual object reasoning.
arXiv preprint arXiv:2411.10252, 2024.
- [8]
Yi-Chia Chen, Wei-Hua Li, Cheng Sun, Yu-Chiang Frank Wang, and Chu-Song Chen.
Sam4mllm: Enhance multi-modal large language model for referring expression segmentation.
In ECCV, 2024.
- [9]
Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Chen Change Loy, and Shuicheng Yan.
Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding.
In NeurIPS, 2024.
- [10]
Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin.
Pixellm: Pixel reasoning with large multimodal model.
In CVPR, 2024.
- [11]
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia.
Lisa: Reasoning segmentation via large language model.
In CVPR, 2024.
- [12]
Xinli Yue, JianHui Sun, Junda Lu, Liangchao Yao, Fan Xia, Tianyi Wang, Fengyun Rao, Jing Lyu, and Yuetang Deng.
Instruction-augmented multimodal alignment for image-text and element matching.
arXiv preprint arXiv:2504.12018, 2025.
- [13]
Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roee Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Szpektor.
What you see is what you read? improving text-image alignment evaluation.
In NeurIPS, 2023.
- [14]
OpenAI.
OpenAI o1.
[https://openai.com/o1/](https://openai.com/o1/), 2024.
- [15]
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
arXiv preprint arXiv:2501.12948, 2025.
- [16]
Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, et al.
Kimi-vl technical report.
arXiv preprint arXiv:2504.07491, 2025.
- [17]
Yichen Wei, Yi Peng, Xiaokun Wang, Weijie Qiu, Wei Shen, Tianyidan Xie, Jiangbo Pei, Jianhao Zhang, Yunzhuo Hao, Xuchen Song, et al.
Skywork r1v2: Multimodal hybrid reinforcement learning for reasoning.
arXiv preprint arXiv:2504.16656, 2025.
- [18]
Yi Peng, Xiaokun Wang, Yichen Wei, Jiangbo Pei, Weijie Qiu, Ai Jian, Yunzhuo Hao, Jiachun Pan, Tianyidan Xie, Li Ge, et al.
Skywork r1v: pioneering multimodal reasoning with chain-of-thought.
arXiv preprint arXiv:2504.05599, 2025.
- [19]
ByteDance.
doubao-1.5-pro.
[https://seed.bytedance.com/en/special/doubao_1_5_pro](https://seed.bytedance.com/en/special/doubao_1_5_pro), 2025.
- [20]
Qwen Team.
Qvq: To see the world with wisdom.
[https://qwenlm.github.io/blog/qvq-72b-preview/](https://qwenlm.github.io/blog/qvq-72b-preview/), 2024.
- [21]
Zhen Yang, Jinhao Chen, Zhengxiao Du, Wenmeng Yu, Weihan Wang, Wenyi Hong, Zhihuan Jiang, Bin Xu, and Jie Tang.
Mathglm-vision: Solving mathematical problems with multi-modal large language model.
arXiv preprint arXiv:2409.13729, 2024.
- [22]
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li.
Measuring multimodal mathematical reasoning with math-vision dataset.
In NeurIPS, 2024.
- [23]
Fatemeh Shiri, Xiao-Yu Guo, Mona Golestan Far, Xin Yu, Gholamreza Haffari, and Yuan-Fang Li.
An empirical analysis on spatial reasoning capabilities of large multimodal models.
arXiv preprint arXiv:2411.06048, 2024.
- [24]
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al.
Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.
In CVPR, 2024.
- [25]
Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vulić, and Furu Wei.
Imagine while reasoning in space: Multimodal visualization-of-thought.
arXiv preprint arXiv:2501.07542, 2025.
- [26]
Mahir Labib Dihan, Md Tanvir Hassan, Md Tanvir Parvez, Md Hasebul Hasan, Md Almash Alam, Muhammad Aamir Cheema, Mohammed Eunus Ali, and Md Rizwan Parvez.
Mapeval: A map-based evaluation of geo-spatial reasoning in foundation models.
arXiv preprint arXiv:2501.00316, 2024.
- [27]
Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, and Cheston Tan.
A survey of embodied ai: From simulators to research tasks.
TETCI, 6(2):230–244, 2022.
- [28]
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al.
A survey on large language model based autonomous agents.
Frontiers of Computer Science, 18(6):186345, 2024.
- [29]
Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, et al.
A survey on multimodal large language models for autonomous driving.
In WACV, 2024.
- [30]
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al.
Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?
In ECCV, 2024.
- [31]
Weiye Xu, Jiahao Wang, Weiyun Wang, Zhe Chen, Wengang Zhou, Aijun Yang, Lewei Lu, Houqiang Li, Xiaohua Wang, Xizhou Zhu, et al.
Visulogic: A benchmark for evaluating visual reasoning in multi-modal large language models.
arXiv preprint arXiv:2504.15279, 2025.
- [32]
Yueqi Song, Tianyue Ou, Yibo Kong, Zecheng Li, Graham Neubig, and Xiang Yue.
Visualpuzzles: Decoupling multimodal reasoning evaluation from domain knowledge.
arXiv preprint arXiv:2504.10342, 2025.
- [33]
Yufan Ren, Konstantinos Tertikas, Shalini Maiti, Junlin Han, Tong Zhang, Sabine Süsstrunk, and Filippos Kokkinos.
Vgrp-bench: Visual grid reasoning puzzle benchmark for large vision-language models.
arXiv preprint arXiv:2503.23064, 2025.
- [34]
Meng-Hao Guo, Jiajun Xu, Yi Zhang, Jiaxi Song, Haoyang Peng, Yi-Xuan Deng, Xinzhi Dong, Kiyohiro Nakayama, Zhengyang Geng, Chen Wang, et al.
R-bench: Graduate-level multi-disciplinary benchmarks for llm & mllm complex reasoning evaluation.
arXiv preprint arXiv:2505.02018, 2025.
- [35]
Jie Feng, Jun Zhang, Junbo Yan, Xin Zhang, Tianjian Ouyang, Tianhui Liu, Yuwei Du, Siqi Guo, and Yong Li.
Citybench: Evaluating the capabilities of large language model as world model.
arXiv preprint arXiv:2406.13945, 2024.
- [36]
Shaoyuan Xie, Lingdong Kong, Yuhao Dong, Chonghao Sima, Wenwei Zhang, Qi Alfred Chen, Ziwei Liu, and Liang Pan.
Are vlms ready for autonomous driving? an empirical study from the reliability, data, and metric perspectives.
arXiv preprint arXiv:2501.04003, 2025.
- [37]
Penghao Wu and Saining Xie.
V*: Guided visual search as a core mechanism in multimodal llms.
In CVPR, 2024.
- [38]
Shuo Xing, Zezhou Sun, Shuangyu Xie, Kaiyuan Chen, Yanjia Huang, Yuping Wang, Jiachen Li, Dezhen Song, and Zhengzhong Tu.
Can large vision language models read maps like a human?
arXiv preprint arXiv:2503.14607, 2025.
- [39]
Sicheng Feng, Gongfan Fang, Xinyin Ma, and Xinchao Wang.
Efficient reasoning models: A survey.
arXiv preprint arXiv:2504.10903, 2025.
- [40]
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.
Measuring massive multitask language understanding.
In ICLR, 2021.
- [41]
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al.
Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
arXiv preprint arXiv:2402.03300, 2024.
- [42]
R1-V Team.
R1-V.
[https://github.com/Deep-Agent/R1-V?tab=readme-ov-file](https://github.com/Deep-Agent/R1-V?tab=readme-ov-file), 2025.
- [43]
EvolvingLMMs Lab.
Open R1 Multimodal.
[https://github.com/EvolvingLMMs-Lab/open-r1-multimodal](https://github.com/EvolvingLMMs-Lab/open-r1-multimodal), 2025.
- [44]
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang.
Visual-rft: Visual reinforcement fine-tuning.
arXiv preprint arXiv:2503.01785, 2025.
- [45]
Huajie Tan, Yuheng Ji, Xiaoshuai Hao, Minglan Lin, Pengwei Wang, Zhongyuan Wang, and Shanghang Zhang.
Reason-rft: Reinforcement fine-tuning for visual reasoning.
arXiv preprint arXiv:2503.20752, 2025.
- [46]
Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al.
Vlm-r1: A stable and generalizable r1-style large vision-language model.
arXiv preprint arXiv:2504.07615, 2025.
- [47]
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al.
Qwen2 technical report.
arXiv preprint arXiv:2407.10671, 2024.
- [48]
OpenAI.
OpenAI o3 and o4-mini System Card.
[https://cdn.openai.com/pdf/2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf](https://cdn.openai.com/pdf/2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf), 2025.
- [49]
OpenAI.
Hello gpt4-o.
[https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/), 2024.
- [50]
Gemini, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.
Gemini: a family of highly capable multimodal models.
arXiv preprint arXiv:2312.11805, 2023.
- [51]
Zhibin Bao, Sabir Hossain, Haoxiang Lang, and Xianke Lin.
A review of high-definition map creation methods for autonomous driving.
Engineering Applications of Artificial Intelligence, 122:106125, 2023.
- [52]
Ari Seff and Jianxiong Xiao.
Learning from maps: Visual common sense for autonomous driving.
arXiv preprint arXiv:1611.08583, 2016.
- [53]
Haotian Xu, Yue Hu, Chen Gao, Zhengqiu Zhu, Yong Zhao, Yong Li, and Quanjun Yin.
Geonav: Empowering mllms with explicit geospatial reasoning abilities for language-goal aerial navigation.
arXiv preprint arXiv:2504.09587, 2025.
- [54]
Song Wang, Wentong Li, Wenyu Liu, Xiaolu Liu, and Jianke Zhu.
Lidar2map: In defense of lidar-based semantic map construction using online camera distillation.
In CVPR, 2023.
- [55]
Xu Cao, Tong Zhou, Yunsheng Ma, Wenqian Ye, Can Cui, Kun Tang, Zhipeng Cao, Kaizhao Liang, Ziran Wang, James M Rehg, et al.
Maplm: A real-world large-scale vision-language benchmark for map and traffic scene understanding.
In CVPR, 2024.
- [56]
Yupeng Zheng, Zebin Xing, Qichao Zhang, Bu Jin, Pengfei Li, Yuhang Zheng, Zhongpu Xia, Kun Zhan, Xianpeng Lang, Yaran Chen, et al.
Planagent: A multi-modal large language agent for closed-loop vehicle motion planning.
arXiv preprint arXiv:2406.01587, 2024.
- [57]
Qingbin Zeng, Qinglong Yang, Shunan Dong, Heming Du, Liang Zheng, Fengli Xu, and Yong Li.
Perceive, reflect, and plan: Designing llm agent for goal-directed city navigation without instructions.
arXiv preprint arXiv:2408.04168, 2024.
- [58]
Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou.
Hallucination of multimodal large language models: A survey.
arXiv preprint arXiv:2404.18930, 2024.
- [59]
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang.
Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?
arXiv preprint arXiv:2504.13837, 2025.
- [60]
Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al.
Reinforcement learning for reasoning in large language models with one training example.
arXiv preprint arXiv:2504.20571, 2025.
- [61]
Sheng Zhang, Qianchu Liu, Guanghui Qin, Tristan Naumann, and Hoifung Poon.
Med-rlvr: Emerging medical reasoning from a 3b base model via reinforcement learning.
arXiv preprint arXiv:2502.19655, 2025.
- [62]
Botian Jiang, Lei Li, Xiaonan Li, Zhaowei Li, Xiachong Feng, Lingpeng Kong, Qi Liu, and Xipeng Qiu.
Understanding the role of llms in multimodal evaluation benchmarks.
arXiv preprint arXiv:2410.12329, 2024.
- [63]
Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, and Yu Cheng.
Can mllms reason in multimodality? emma: An enhanced multimodal reasoning benchmark.
arXiv preprint arXiv:2501.05444, 2025.
- [64]
Aarti Ghatkesar, Uddeshya Upadhyay, and Ganesh Venkatesh.
Looking beyond language priors: Enhancing visual comprehension and attention in multimodal models.
arXiv preprint arXiv:2505.05626, 2025.
- [65]
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.
Scaling laws for neural language models.
arXiv preprint arXiv:2001.08361, 2020.
- [66]
Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun.
Rotary position embedding for vision transformer.
In ECCV, 2024.
- [67]
Junqi Ge, Ziyi Chen, Jintao Lin, Jinguo Zhu, Xihui Liu, Jifeng Dai, and Xizhou Zhu.
V2pe: Improving multimodal long-context capability of vision-language models with variable visual position encoding.
arXiv preprint arXiv:2412.09616, 2024.

## Appendix

We provide a comprehensive overview in the Appendix, covering key components of our dataset, methodology, and analysis. Specifically, we include the question templates used during dataset construction, sources of transit maps from $30$ cities, detailed descriptions of the evaluation algorithm, experimental setup, and in-depth analyses of both correct and erroneous cases.

In addition, we further discuss diverse image processing strategies employed by various MLLMs when handling high-resolution visual inputs. We also address the stated limitations of our work, potential broader impacts, ethical considerations related to the dataset, and licensing information for the MLLMs used in our experiments.

[appendices]
\printcontents[appendices]l1

## Appendix A Dataset Construction Details

### A.1 Question Template Summary

We present one short question template and two long question templates as follows.

### A.2 Map Source

We provide the sources of all maps included in ReasonMap for further reference (Table [A1](https://arxiv.org/html/2505.18675v2#A1.T1)).

**Table A1: Source links for city transit maps used in ReasonMap. We present a total of $30$ cities sourced from $13$ countries.**
| City | Source | City | Source | City | Source |
| --- | --- | --- | --- | --- | --- |
| Budapest | [Link](https://dt.369.me/city/budapest/) | Oslo | [Link](https://transitmap.net/oslo-2016/) | Rome | [Link](https://www.mappametroroma.it/mappa-metro/mappa-metro-roma-2025.png) |
| Lisboa | [Link](https://www.metrolisboa.pt/en/travel/diagrams-and-maps/) | Geneva | [Link](https://dt.369.me/city/geneva/#google_vignette) | Dubai | [Link](https://www.rta.ae/links/rail/rail-network-map.pdf) |
| Auckland | [Link](https://at.govt.nz/media/xaqlzv4n/auckland-transport-train-and-rapid-bus-network-map.jpg) | Sydney | [Link](https://nsw-transport.net/network-maps/metro-train-maps/) | Singapore | [Link](https://www.lta.gov.sg/content/dam/ltagov/getting_around/public_transport/rail_network/pdf/SM_TEL4_Eng.pdf) |
| Kuala Lumpur | [Link](https://myrapid.com.my/bus-train/rapid-kl/rapid-kl-integrated-transit-map/) | Los Angeles | [Link](https://www.metro.net/riding/guide/system-maps/) | Miami | [Link](https://www.fozhoubest.com/?p=34928) |
| New York | [Link](https://www.mta.info/map/5256) | Toronto | [Link](https://www.ttc.ca/routes-and-schedules) | Washington | [Link](https://wmata.com/schedules/maps/upload/system-map-rail.pdf) |
| Guiyang | [Link](https://www.gyurt.com) | Shanghai | [Link](https://jtw.sh.gov.cn/csgdjt/index.html) | Huhehaote (Hohhot) | [Link](https://www.hhhtmetro.com/org/platform) |
| Nanchang | [Link](https://www.ncmtr.com) | Nanning | [Link](http://nngdjt.com/index.html) | Shenzhen | [Link](https://www.szmc.net/shentieyunying/yunyingfuwu/szsgdjtyyxlwlt/) |
| Hangzhou | [Link](https://www.hzmetro.com/EptionUload/image/dtxwt_big.jpg) | Dalian | [Link](https://www.dltransgrp.com/portal/indexShow.do) | Kunming | [Link](https://km.ynairport.com/zhjt/5749.jhtml) |
| Hefei | [Link](https://www.hfgdjt.com/) | Beijing | [Link](https://map.bjsubway.com/) | Changzhou | [Link](https://www.czmetro.net.cn/) |
| Jinan | [Link](https://www.jngdjt.cn/) | Xi’an | [Link](https://www.xianrail.com/#/index) | Changshang | [Link](http://www.hngdkg.com/) |

## Appendix B Evaluation Details

### B.1 Correctness and Quality Evaluation

We present the detailed algorithms for evaluating answer correctness and quality in Section [4](https://arxiv.org/html/2505.18675v2#S4) (Algorithm [1](https://arxiv.org/html/2505.18675v2#algorithm1) for correctness evaluation and Algorithm [2](https://arxiv.org/html/2505.18675v2#algorithm2) for quality evaluation).

Figure: Algorithm 1 Correctness Evaluation

Figure: Algorithm 2 Quality Evaluation

### B.2 Details about Difficulty-Aware Weighting.

Each difficulty pair is assigned a predefined weight that reflects its relative challenge level. The full weight matrix is shown below, where the first element in each pair denotes the question difficulty and the second denotes the map difficulty:

| ("easy", "easy"): $1.0$ | ("middle", "easy"): $1.5$ | ("hard", "easy"): $2.0$ |
| --- | --- | --- |
| ("easy", "middle"): $1.5$ | ("middle", "middle"): $2.0$ | ("hard", "middle"): $2.5$ |
| ("easy", "hard"): $2.0$ | ("middle", "hard"): $2.5$ | ("hard", "hard"): $3.0$ |

This weighting scheme rewards models more for correctly solving harder question–map combinations, reflecting the increased reasoning complexity they entail, while maintaining moderate differences between buckets to prevent excessive score variance and preserve evaluation stability.

## Appendix C Case Analysis

We provide additional case analyses covering both correct and incorrect predictions, along with detailed comparisons of their respective reasoning processes.

We compare Doubao-415 and Doubao-428 (Figure [A1](https://arxiv.org/html/2505.18675v2#A3.F1)), both of which reach the correct destination (from Augustins Station to Poterie Station) but via distinct reasoning paths. Doubao-415 correctly identifies early that both stations are on Line 18 and efficiently converges on the optimal, direct route without transfers. In contrast, Doubao-428 misclassifies Augustins as being on Line 12 and, assuming Poterie is on Line 18, proposes a transfer route via Plainpalais—functionally correct but suboptimal due to unnecessary complexity. Both models engage in extensive self-correction ($7270$ tokens for Doubao-428; $4474$ for Doubao-415), highlighting the significant downstream impact of early-stage misjudgments. Moreover, visual reasoning limitations persist: despite correctly recognizing Augustins on Line 12, Doubao-415 commits to a transfer path and fails to re-evaluate the possibility of a direct connection. This indicates room for improvement in both early visual grounding and global route optimality awareness.

We then analyze the observed pattern when comparing the full input and text-only variants in the case (in Figure [A2](https://arxiv.org/html/2505.18675v2#A3.F2)). The model with full visual access accurately identifies both stations on the Yellow Line and outputs the optimal direct route with the correct number of via stops. In contrast, the text-only variant makes an early misclassification, placing both stations on the Blue Line (Azul) and constructing a plausible but entirely incorrect sequence of intermediate stops. Although the final answer format appears coherent, the underlying logic is flawed due to the initial error in line recognition. This further illustrates the importance of visual input in spatial reasoning tasks, where even minor misinterpretations can lead to fundamentally incorrect conclusions. Additionally, some models, such as the InternVL3 series, default to rejection when visual input is absent.

We further present several error cases (in Figure [A3](https://arxiv.org/html/2505.18675v2#A3.F3)), where Doubao-415 still exhibits visual confusion. In contrast, Qwen2.5-VL-32B-I, when lacking visual input, behaves differently from the InternVL3 series: rather than rejecting the query outright, it attempts to reason over the available information without producing a final answer, while explicitly notifying the user of the missing visual input.

Figure: Figure A1: Case analysis of various MLLMs using ReasonMap (Case N1). For reasoning models, the reasoning process is explicitly marked with <think> and </think> tags. We highlight error contents in the answers with red and correct contents in green.
Refer to caption: x8.png

Figure: Figure A2: Case analysis of various MLLMs using ReasonMap (Case N2). For reasoning models, the reasoning process is explicitly marked with <think> and </think> tags. We highlight error contents in the answers with red and correct contents in green.
Refer to caption: x9.png

Figure: Figure A3: Case analysis of various MLLMs using ReasonMap (Case N3). For reasoning models, the reasoning process is explicitly marked with <think> and </think> tags. We highlight error contents in the answers with red and correct contents in green.
Refer to caption: x10.png

## Appendix D Further Discussions

### D.1 High-Resolution Image Preprocessing.

We compare how different Multimodal Large Language Models (MLLMs) handle high-resolution image inputs in Table [A2](https://arxiv.org/html/2505.18675v2#A4.T2). Specifically, we examine three key components in their preprocessing pipelines: dynamic resolution handling, positional encoding, and token compression.

**Table A2: Comparison of high-resolution image preprocessing strategies across different MLLMs. We use “$-$” to denote unspecified or unclear content.**
| Model | Dynamic Resolution Handling | Positional Encoding | Token Compression |
| --- | --- | --- | --- |
| Qwen2.5-VL series [2] | ✓ | 2D-RoPE | ✓ ($2\times 2$ Concat + MLP) |
| QVQ-72B-Preview [20] | ✓ | 2D-RoPE | ✓ ($2\times 2$ Concat + MLP) |
| InternVL3 series [3] | ✓ | V2PE | ✓ (Unshuffle + MLP) |
| Kimi-VL series [16] | ✓ | 2D-RoPE | ✓ (Shuffle + MLP) |
| Skywork-R1V-38B [17, 18] | ✓ | - | ✗ |
| Gemini [50] | ✗ (Tiling+Resize) | - | ✗ |
| Doubao-1.5-Pro series [19] | ✓ | 2D-RoPE | ✓ ($2\times 2$ Pooling + MLP) |

- 1.
Dynamic resolution handling refers to whether the model can directly accept images of arbitrary sizes without resizing or cropping. Most recent models support native resolution processing, enabling them to preserve fine-grained spatial information. In contrast, models like Gemini [50] rely on image tiling and resizing to fit fixed input constraints.
- 2.
Positional encoding helps the model retain spatial structure among visual tokens. Common strategies include 2D Rotary Positional Encoding (2D-RoPE) [66], as seen in Qwen2.5-VL [2] and Doubao [19], or flexible alternatives like V2PE [67] in InternVL3 [3]. Some models (e.g., Gemini, Skywork-R1V [17, 18]) do not explicitly disclose their positional encoding scheme, which we mark as “–” in the table.
- 3.
Token compression aims to reduce the number of visual tokens for more efficient processing. Different models adopt different strategies: Qwen2.5-VL and QVQ [20] compress tokens via $2\times 2$ patch concatenation followed by an MLP; InternVL3 [3] and Kimi-VL [16] utilize spatial transformations like pixel unshuffle or shuffle, also followed by MLPs; Doubao averages over $2\times 2$ patches before projection. Models without token compression may incur higher memory and computation costs when processing high-resolution inputs.

### D.2 Limitations and Future Work

While ReasonMap provides a carefully curated benchmark for evaluating fine-grained visual reasoning with high-resolution transit maps, we acknowledge that it represents only one type of structured visual diagram. As such, caution should be taken when generalizing observations to other domains that involve different types of visual content or reasoning styles. Additionally, although efforts were made to ensure diversity across cities and languages, the current version may not fully capture all geographic or linguistic variations. Future iterations could further expand coverage and explore additional forms of reasoning to enhance generality.

### D.3 Broader Impact

Advancing the capabilities of MLLMs in fine-grained visual reasoning has the potential to benefit a wide range of real-world applications, including navigation systems, urban planning tools, and assistive technologies for visually impaired individuals. By offering a structured and rigorous benchmark, ReasonMap encourages the development of MLLMs that can more effectively interpret complex visual artifacts and perform spatial reasoning. This could contribute to the long-term goal of building intelligent agents that interact more naturally and safely with human environments. Furthermore, the dataset’s emphasis on high-resolution, globally sourced transit maps promotes research that is inclusive of diverse visual formats and geographic contexts. We hope ReasonMap can serve as a step toward more transparent, robust, and generalizable multimodal systems.

## Appendix E License and Consent Information

### E.1 Ethical Considerations

All experiments are conducted on ReasonMap, which is built using publicly available transit maps collected in compliance with relevant licenses and usage terms. The maps are selected to ensure geographic diversity and legal validity. Upon code release, we provide the source of each map for further reference. ReasonMap is intended solely for academic research on fine-grained visual understanding and spatial reasoning in MLLMs. It does not redistribute any copyrighted map images. All annotations are based on public information, contain no personal data, and are created under academic oversight. The benchmark is not intended for safety-critical use. We take care to ensure fairness, legal compliance, and responsible data handling.

### E.2 Public Implementation

We benchmark the visual understanding and reasoning performance on ReasonMap across a diverse set of publicly available MLLMs:

- •
KimiVL [16](^4^44[https://github.com/MoonshotAI/Kimi-VL](https://github.com/MoonshotAI/Kimi-VL).) MIT License
- •
Skywork-R1V [17, 18](^5^55[https://huggingface.co/Skywork/Skywork-R1V2-38B](https://huggingface.co/Skywork/Skywork-R1V2-38B).) MIT License
- •
QVQ-72B-Preview [20](^6^66[https://huggingface.co/Qwen/QVQ-72B-Preview](https://huggingface.co/Qwen/QVQ-72B-Preview).) Qwen License
- •
Gemini-2.5-Flash [50](^7^77[https://deepmind.google/technologies/gemini](https://deepmind.google/technologies/gemini).) Closed-Source
- •
InternVL-3.0 [3](^8^88[https://github.com/OpenGVLab/InternVL](https://github.com/OpenGVLab/InternVL).) MIT License
- •
Qwen2.5-VL [2](^9^99[https://github.com/QwenLM/Qwen2.5-VL](https://github.com/QwenLM/Qwen2.5-VL).) Apache 2.0 License
- •
Doubao-Pro 1.5 [19](^10^1010[https://www.volcengine.com/product/doubao](https://www.volcengine.com/product/doubao).) Closed-Source
- •
OpenAI o3 [48](^11^1111[https://platform.openai.com/docs/models/o3](https://platform.openai.com/docs/models/o3).) Closed-Source
- •
OpenAI 4o [49](^12^1212[https://platform.openai.com/docs/models/gpt-4o](https://platform.openai.com/docs/models/gpt-4o).) Closed-Source

To ensure fair and reproducible evaluation, we implement all inference procedures by adhering closely to the official documentation and recommended practices of each model.
The code is released under the MIT License to support transparency and reproducibility. Additionally, we provide detailed usage instructions on the project website to ensure easy access and reproducibility for future users.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: 1. Have you checked if the map-score metric aligns with human judgments, for example through correlation on a small subset?
2. Could the weaker performance of reasoning models be due to longer outputs or token truncation?
3. Did you try cross-city generalization, such as training on some cities and testing on unseen ones?
4. Have you measured how much performance changes when using lower-resolution maps to test fine-grained perception?
soundness: 3
strengths: 1. Originality: Using transit maps as a testbed for multimodal reasoning is creative and well-motivated. Prior datasets such as MathVerse, VisuLogic, and CityBench rely on synthetic or narrow tasks, while this one focuses on structured, real-world visuals that naturally require spatial understanding.


2. Quality: The dataset pipeline is solid, combining automated extraction with human verification and balanced difficulty design. The two-level evaluation (accuracy and map score) provides a more nuanced picture of reasoning quality than simple accuracy metrics.


3. Significance: The findings reveal a real limitation in current MLLMs: their weak visual grounding even on structured and information-rich inputs. The contrast between open and closed models is also interesting.
summary: This paper presents REASONMAP, a benchmark for evaluating fine-grained spatial reasoning in multimodal large language models using high-resolution transit maps. The dataset includes about 1,000 human-verified QA pairs from 30 cities and introduces a two-level evaluation that measures both answer correctness and route quality (map score). The authors benchmark 15 MLLMs across open-source and closed-source variants and find that open-source reasoning models often perform worse than their base versions, while closed-source reasoning models perform better. Additional experiments with masked visual input show that most models rely heavily on textual priors instead of true visual grounding.
weaknesses: 1. The dataset is relatively small (around 1k QA pairs), which makes it hard to draw statistically strong conclusions. Reporting variation across random subsets or expanding the dataset would help.


2. The map-score metric feels somewhat heuristic. There is no validation showing that it correlates with human judgment of route plausibility or quality.


3. The explanation for why reasoning models underperform is not fully convincing. It could also be due to token length limits or strict output formatting rather than just RL-based training bias.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. For a given stop-pair, why was only one of the two long templates randomly assigned? Why not test both templates for the same route to measure the model's robustness to different formatting requirements?
2. How were multilingual maps handled during data creation and evaluation? If a map has stop names in two languages, which language was chosen for the ground truth, and how are answers in the other valid language scored by your strict evaluation?
3. How is manual annotation done for different languages?
soundness: 2
strengths: 1. The paper details a semi-automated data creation pipeline. Great detail is paid to ensure a high-quality dataset that is diverse in map style and language.
2. The inclusion of text-only and language-replacement experiments (in Appendix) provides valuable insights. These ablations effectively help disentangle the model's reliance on visual grounding versus its pre-existing world knowledge, strengthening the paper's conclusions.
3. The paper is generally well-written and easy to follow. The overview figures effectively illustrate the dataset's components and the data creation pipeline.
summary: The paper introduces REASONMAP, a benchmark dataset for evaluating the fine-grained visual understanding and spatial reasoning capabilities of MLLMs. The core task requires MLLMs to perform pathfinding by interpreting high-resolution transit maps. The dataset comprises 1,008 question-answer pairs derived from public-domain maps of 30 cities across 13 countries. The authors employ a semi-automated data creation pipeline: MLLMs are first used to extract transit line and stop names into a structured JSON format, which is then manually verified. Ground-truth routes between two randomly selected stops are generated using map service APIs (Google Maps and Gaode Maps). The tasks are structured into three templates: one short question (identifying the route) and two long question variants (additionally identifying either the number of intermediate stops or the names of intermediate stops). The authors manually annotated map difficulty and used the number of transfers in the ground-truth route to programmatically set question difficulty. The benchmark is evaluated on a sample test set using 15 MLLMs. The primary experimental finding is a significant performance divergence: closed-source reasoning models outperform their base counterparts, whereas the opposite is true for open-source models. Ablation studies confirm that while models use visual information, they also rely heavily on prior knowledge.
weaknesses: 1. The paper's core motivation is unconvincing and missed the "why" for this specific task. Given that navigation is a well-solved problem via APIs (which the authors use for ground truth), it is unclear why an MLLM must perform this end-to-end visual reasoning. The task could be more realistically framed as a test of tool-use. The paper does not justify why this capability is intrinsically valuable for a standalone MLLM.
2. The evaluation relies on strict string matching for correctness. This is overly brittle and penalizes models for trivial, semantically irrelevant formatting deviations (e.g., "Line 1" vs. "1"), stylistic variations, or different language choices (on multi-lingual maps). A more robust semantic evaluation like using an MLLM judge would be more appropriate for assessing true reasoning.
3. The paper contains several magic numbers without justification. The 20:15:5 (easy:medium:hard) question difficulty ratio is arbitrary. The scoring algorithm seems hand-tuned. These choices lack a principled foundation and affect the final reported scores.
4. The font size in all tables is very small. Additionally, Figure 3 should include data labels on the bars to convey the precise accuracy numbers.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: 1. Could you provide a specific, quantifiable heuristic or formula that was used to manually classify the maps into "easy," "medium," and "hard" difficulty levels?

2. What is the justification for the point values assigned in the "map score" (e.g., why is a correct route name worth two points while an endpoint is worth one)?

3. Could you elaborate on the methodology and reasoning behind the 3x3 "difficulty-aware" weighting matrix used in your evaluation?

4. Regarding the 2,048 maximum token limit for open-source models, could this have prematurely cut off the "chain-of-thought" reasoning process? Why was a similar constraint not reported for the closed-source models?

5. Given that the evaluation was conducted on a small test set of 312 examples, have you performed any statistical tests (e.g., bootstrapping) to confirm the stability and significance of your findings?
soundness: 2
strengths: 1. **Novel and Important Problem:** The paper addresses a critical and timely gap in MLLM evaluation. The task of reasoning over complex, information-dense transit maps pushes beyond simple OCR or captioning tasks, requiring genuine fine-grained visual understanding and multi-step spatial reasoning, which is a significant step forward for multimodal benchmarks. 

2. **Strong Benchmark Validation:** The inclusion of an experiment where visual inputs are masked is a major strength. This ablation effectively validates that the benchmark tests visual perception and reasoning rather than relying solely on the models' pre-existing parametric knowledge, confirming the dataset's value to the community.
summary: This paper introduces REASONMAP, a new benchmark dataset designed to evaluate the fine-grained visual understanding and spatial reasoning capabilities of Multimodal Large Language Models (MLLMs). The dataset consists of 1,008 question-answer pairs based on high-resolution transit maps from 30 cities. The authors also propose a two-level evaluation framework to assess both the correctness and the quality of the models' generated routes. A key finding from their experiments on 15 MLLMs is a counterintuitive performance gap, where open-source base models outperform their reasoning-focused counterparts, while the opposite is true for closed-source models.
weaknesses: The paper's primary weaknesses stem from the limited scale of its evaluation, a perceived arbitrariness in its evaluation framework, and a lack of deeply grounded analysis for some of its key findings.

---

### Small and Potentially Biased Evaluation Set
A significant limitation is the size of the test set. The entire dataset contains just over 1,000 question-answer pairs, and the evaluation is conducted on a small subset of only 312 examples. While the authors reserve the rest for future training use, this small evaluation set raises concerns about the statistical significance and generalizability of the findings. With such a limited number of data points, the results may be susceptible to noise and may not be representative of the models' true capabilities across a wider range of maps and questions. This small scale is a critical weakness for a paper introducing a new benchmark.

---

### Subjectivity and Lack of Rigor in the Evaluation Framework
Several core components of the evaluation methodology feel arbitrary and are not sufficiently justified, which undermines the reliability of the results. A more principled and transparent approach is needed.

**Map Difficulty:** The difficulty level of maps ("easy," "medium," "hard") is assigned manually without a clear, replicable metric. While the appendix mentions proxies like line and transfer counts, the paper would be much stronger if it defined a concrete heuristic or a quantitative formula used for this categorization. Without it, the classification appears subjective.

**Arbitrary Scoring and Weighting:** The point system for the "map score" and the "difficulty-aware" weighting scheme both seem ad-hoc. For instance, the rationale for assigning specific point values (e.g., two points for a route name vs. one for an endpoint) is not explained. Similarly, the 3x3 weight matrix for combining map and question difficulty adds a layer of complexity that feels convoluted and whimsical rather than methodologically sound. These arbitrary choices make the final performance scores difficult to interpret and trust.

---

### Speculative Analysis and Unanswered Questions
The paper presents interesting results but often stops short of providing a rigorous, evidence-based analysis, leaving key questions unanswered.

**Unsupported Claims on Model Performance:** The paper's explanation for why open-source "thinking" models underperform their base versions is presented as speculation rather than a conclusion drawn from rigorous analysis. The authors suggest this may be due to RL-induced biases, but this claim is not substantiated with targeted experiments. A deeper analysis, such as examining the models' reasoning traces or conducting specific ablations, is needed to provide a more grounded explanation for this surprising and important finding.

**Confounding Experimental Variables:** The decision to cap the maximum output tokens at 2,048 for open-source models is a potential confounder that is not adequately addressed. It is plausible that this limit prematurely terminated the "chain-of-thought" process for the reasoning-centric models, thereby artificially depressing their performance. The paper needs to clarify the rationale for this choice and explain why a similar constraint was not applied or reported for the closed-source models, as this is crucial for a fair comparison.

---

### Clarity and Readability
The paper is dense and can be difficult to follow at times. Simplifying the language and improving the narrative flow would help make the contributions more accessible and engaging for a broader audience.

---

confidence: 5
contribution: 3
initial_rating: 4
presentation: 2
questions: 1. What is the specific rationale for the 20:15:5 (easy:medium:hard) question ratio, and why is this "balanced"?
2. What exact logic and thresholds were used to manually categorize the 30 maps into difficulty levels?
3. What specific "automatic checks" and "manual adjustment" protocols were used for quality control?
4. How many of the 312 test samples are non-English, and why is this sufficient to evaluate multilingual performance?
soundness: 3
strengths: 1. The paper tackles a valuable and challenging problem: testing MLLM capabilities on dense, high-resolution, practical-use diagrams, which is a clear gap in current benchmarks.
2. The two-level evaluation framework, which separates route correctness (is it valid?) from route quality (is it optimal?), is a strong contribution. This is a nuanced approach that is well-suited for this pathfinding task.
summary: This paper introduces REASONMAP, a benchmark to evaluate the visual reasoning of MLLMs using high-resolution transit maps. The task requires models to find routes between stations. The authors propose a two-level evaluation metric to assess both the correctness and quality of the generated routes and present results from 15 MLLMs.
weaknesses: 1. The dataset is too small for objective evaluation. 30 cities and 1008 QA pairs are insufficient to draw general conclusions. Furthermore, these 30 cities may be popular, meaning their transit systems are likely part of the MLLMs' pre-training data. This contaminates the results, making it unclear if models are demonstrating visual reasoning or recalling prior knowledge.

2. Why was a 20:15:5 (easy:medium:hard) ratio chosen for question difficulty? The paper claims this is for "balance," but a 1:1:1 split would be balanced. The rationale is not provided.

3. How were maps split into easy, medium, and hard? The paper mentions this was done manually and lists proxies in the appendix, but the actual logic or thresholds used for this assignment are missing.

4. The paper mentions "automatic checks and manual adjustments" for quality control but provides no details on what these checks were or how they were performed.

6. The dataset is too small to be split. A test set of 312 samples across 11 cities is insufficient and likely biased. How were these 11 cities chosen? How many of them fall into the multilingual category to test robustness?

7. The difficulty categories (map difficulty x question difficulty) combined with a separate difficulty-weighting scheme make the results convoluted and hard to interpret. A single, holistic difficulty score for each QA pair, combining all factors, would be simpler and more effective.

8. The analysis section is shallow. The main takeaway from the difficulty breakdown (Fig. 3) is that "performance degrades as task complexity increases," which is an obvious conclusion. The paper observes trends (e.g., base vs. reasoning models) but fails to provide a deep diagnosis why these trends occur, beyond citing prior work.

9. The paper incorrectly uses "fine-grained visual understanding" and "spatial reasoning" interchangeably. This benchmark clearly tests reasoning (planning a path). However, it's less of a pure test of visual understanding, which would be more about perception (e.g., "find this station," "trace this line"). This benchmark has potential as a diagnostic tool, but its claim to be a definitive measure of fine-grained visual understanding is not fully supported.