RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learning
Authors: Sicheng Feng
Kaiwen Tuo
Song Wang
Lingdong Kong
Jianke Zhu
Huan Wang
Venue: ICLR 2026 Poster (2026)
Keywords: MLLMs
Fine-grained Visual Reasoning
Visual Understanding
Reinforcement Learning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Fine-grained visual reasoning remains a core challenge for multimodal large language models (MLLMs). The recently introduced ReasonMap highlights this gap by showing that even advanced MLLMs struggle with spatial reasoning in structured and information-rich settings such as transit maps, a task of clear practical and scientific importance. However, standard reinforcement learning (RL) on such tasks is impeded by sparse rewards and unstable optimization. To address this, we first construct ReasonMap-Plus, an extended dataset that introduces dense reward signals through Visual Question Answering (VQA) tasks, enabling effective cold-start training of fine-grained visual understanding skills. Next, we propose RewardMap, a multi-stage RL framework designed to improve both visual understanding and reasoning capabilities of MLLMs. RewardMap incorporates two key designs. First, we introduce a difficulty-aware reward design that incorporates detail rewards, directly tackling the sparse rewards while providing richer supervision. Second, we propose a multi-stage RL scheme that bootstraps training from simple perception to complex reasoning tasks, offering a more effective cold-start strategy than conventional Supervised Fine-Tuning (SFT). Experiments on ReasonMap and ReasonMap-Plus demonstrate that each component of RewardMap contributes to consistent performance gains, while their combination yields the best results. Moreover, models trained with RewardMap achieve an average improvement of 3.47% across 6 benchmarks spanning spatial reasoning, fine-grained visual reasoning, and general tasks beyond transit maps, underscoring enhanced visual understanding and reasoning capabilities.

Published as a conference paper at ICLR 2026
REWARDMAP: TACKLING SPARSE REWARDS IN FINE- GRAINED VISUAL REASONING VIA MULTI-STAGE RE- INFORCEMENT LEARNING
Sicheng Feng".1, Kaiwen Tuo Song Wang", Lingdong Kong', Jianke Zhu?,
Huan Wang" 1.*
Westlake University Tongji University
&Zhejaang University
"National University of Singapore
i Equal contributions.
*Conresponding author.
Dataset & Toolkit: https: //fpss://fcccd:/// git io io/RewardMap
ABSTRACT
Fine-grained visual reasoning remains core challenge for multimodal large lan- guage models (MLLMs). The recently introduced REASONMAP highlights this gap by showing that even advanced MLLMs struggle with spatial reasoning in structured and information-rien settings such as transit maps, a task of clear practi- cal and scientific importance. However, standard reinforcement learning (RL) on such tasks is impeded by sparse rewards and unstable optimization. To address this, we first construct REASONMAP-PLLC an extended dataset that introduces dense reward signals through Visual Question Answering (VQA) tasks, enabling effective cold-start training of fine-graine greeggaaied visual understanding skills. Next, we propose REWARDMAP, multi-stage RL framework designed to improve both visual un- derstanding and reasoning capabilities of MLLMs. REWARDMAP incorporates two key designs. First, we introduce difficulty- porates detail rewards, directly tackling the sparse rewards while providing richer supervision. Second, we propose multi-stage RL scheme that bootstraps train- ing from simple perception to complex reasoming tasks, offering a more effective cold-start strategy than conventional Supervised Fine-Tuning (SFT). Experiments on REASONMAP and MAAASONMAPPLLLAAE demonstrate that each component of REWARDMAP contributes to consistent performance gains, while their combination yields the best results. Moreover, models trained with REWARDMAP achieve an average improvement 3.47% across benchmarks spanning spatial reasoning. fine-grained visual reasoning, and general tasks beyond transit maps, underscoring enhanced visual understanding and reasoning capabilities.
reward design that incor-
INTRODUCTION
Fine-grained visual reasoning over structured visual inputs remains a significant challenge for multimodal large language models (MLLMs) (Bai et al., 2025b; OpenAI, 2025; Zhang et al., 2025e; Liang et al., 2025; Li et al., 2025c;d), Recently, REASONMAP (Feng et al., 2025b) was introduced as a benchmark on high-resolution transit maps in the real world, where tasks (e.g., route planning) combine visual understanding with spatial reasoning, jointly constituting the fine-grained visual reasoning challenge (Xie et al., 2025a; Hu et al., 2025b; Kong et al., 2025b;a). This task is not only of practical value for real-world navigation and transportation systems, but also of fundamental scientific interest as it exposes reasoning gaps in current MLLMs. Despite steady advances in lision-language pre-training (Liu et al., 2023; 2024; Bai et al., 2025b), existing models consistently struggle with the visual and spatial reasoning demands in REASONMAP. This gap motivates us to investigate how reinforcement learning (RL) (Zhang et al., 2025a; Shao et al., 2024; Guo et al., 2025a) can be adapted to enhance fine finengrained visual reasoning abilities in structured visual domains such transit maps.
However, directly applying standard RL methods (Rafailov et al., 2023; Shao et al., 2024) to complex tasks such as REASONMAP is highly challenging, supervision signals are inherently sparse, i.e.

---

Published as a conference paper at ICLR 2026
success is typically judged only at the final answer after long reasoning chain (Quadros et al., 2025; Cao et al., 2024; Chen et al., 2025). The difficulty of the tasks further amplifies this sparsity, which in turn destabilizes optimization and hinders effective exploration (Wang et al., 2025a). While classical approaches like Supervised Fine- Tunin (SFT) (Liu et al., 2023; Wei et al., 2025) offer dense supervision, they fall short in equipping models for the long-chain decision-making intrinsic to visual reasoning tasks. This mismatch between task complexity and supervision signal forms a critical bottleneck in leveraging RL for fine-grained visual reasoning.
To address this issue, we first construct REASONMAP-PLUS an extended dataset that introduces dense reward signals for further cold-start training. Tasks in REASONMAP- along a natural difficulty continuum, from simple Visual Question Answering (VQA) that enhances perception to progressively harder visual tasks reflecting the complexity of fine-grained visual reasoning queries. We further introduce REWARDMAP, multi-stage RL framework with detailed reward design. It consists of two key components: (1) a reward scheme that, beyond basic format and correctness rewards, incorporates detail rewards to mitigate sparsity in supervision for hard samples and adopts a difficulty-awwee departs from SFT-based initialization (Xu et al., 2024; Guo et al., 2024; Wei et al., 2025; Yan et al., 2025) by directly employing RL, ensuring alignment between reward signals and task objectives from the outset. Training data are organized from easy to hard across multiple stages, with dense and accessible rewards at lower levels supporting effective cold-start training. This staged strategy systematically bridges perception and reasoning within a unified RL framework.
are organized
design to account for task complexity; and (2) a cold-start strategy that
We conduct extensive experiments on REASONMAP and REASONMAP-PLUS to evaluate the ef- fectiveness of REWARDMAP. Results indicate that each component yields measurable gains, with their integration delivering the best overall performance. Moreover, beyond the targeted benchmarks, models trained with REWARDMAP achieve consistent improvements (3.47% average) across six benchmarks (Wu & Xie, 2024; Wang et al., 2024a; Li et al., 2024; Wang et al., 2025f; Masty et al., 2022; Chen et al., 2024a) covering spatial reasoning, fine-grained visual reasoning, and general tasks, suggesting enhanced general visual perception and reasoning capabilities.
In summary, this work makes the following contributions: (1) We introduce REASON an extended dataset organized from easy to hard, providing dense supervision for multi-stage RL training; (2) We propose REWARDMAP, a multi-stage RL framework that integrates cold-start curriculum data (i.e., easy + hard) with difficulty-awan that REWARDMAP not only improves performance on REASONMAP and REASONMAP AP-PLLS but also enhances performance across broader visual benchmarks beyond transit maps. Together, these contributions establish a principled approach to overcoming sparse reward challenges in visual reasoning, advancing the capabilities of MLLMs in structured visual tasks.
detail reward design; (3) Extensive experiments demonstrate
2 RELATED WORK
Visual Reasoning in MLLMs. The development of MLLMs has rapidly progressed from founda- tional models that bridge vision and language encoders, such as Flamingo (Alayrac et al., 2022), to those enhanced by visual instruction tuning like LLaVA (Liu al., 2023; 2024). To elicit more complex, step-by-step reasoning, researchers have adapted Chain-of-Thhugge (CoT) (Wei et al., 2022) prompting from the language domain to the multimodal context (MCoT) (Zhang et al., 2023). However, a key limitation of early MCoT is its reliance on purely textual rationales (Wang et al., 2025g), which can be a bottleneck for expressing fine-grained visual logic (Zhang et al., 2025c). More recent work has thus focused on developing iision-centric reasoning processes (Dong et al., 2025; Man al., 2025), such as generating intermediate visual representations as perception tokens to aid the reasoning chain (Bigverdi et al., 2025). Despite these advances, significant performance gap persists. Benchmarks specifically designed to test abstract, spatial, and logical reasoning, such as VisuLogic (Xu et al., 2025) and REASONMAP (Feng et al., 2025b), reveal that even state-of- the-art models struggle with tasks that require high-fidelity visual and topological understanding, underscoring the need for new methods tailored for structured visual domains, such as transit maps.
Reinforcement Learning for Reasoning. Reinforcement Learning (RL) (Wu et al., 2025a; Sarch et al., 2025; Feng et al., 2025a) has emerged as a powerful paradigm for improving the reasoning capabilities of multimodal and language-only models beyond the static nature of Supervised Fine- Tuning (SFT) (Liu et al., 2023; Wei et al., 2025). The evolution of RL for reasoning spans from
2

---

Published as a conference paper at ICLR 2026
Local Counting
True or False
M
Q
How many intermediate stops between [Clarke Guay) and (Farrer Park]?
Farrer Parw
True that [Somerset] station is the same line as [Bras Basah) station?
2. [Choby Ghaut Litle india).
Drchard
Urtte
No [Somerset/ / Bras Besah the NS/
Bendeme
("easy": 350: "middle": 350: "hard": 350)
Somerset
Rechor
C"aasy" t 34 "middle": 349: "hard" 346)
Local Counting 2
Havslock
Chaut
OTd Jalan
True or False
Q
How many lines pass through mn [Dhoby Ghautj?
MTC
true that [Chinatown]
Fort Canving
the North South Line (NS)?
PA32 R114
A No [Chinatown on the lines]
(NS CCJ.
Clarks Quay DE33 Chinatown
Bras
Teasy": "middle" "hard":
l' C'easy:: 278: "middle"" 5TI: "hard": 340)
Counting
NL
Hall
EASY
Mocwell
Map Difficulty
Questian Types
Telok Ager
Comttey
13 Countries
F
Global Counting
Cities
How many lines in the map?
5 Tyles
ReasonMap+
6 (for the map o' Singapare)
Guestions
Figure I: Overview of REASONMAP- REASONMAP-PE PLUS comprises 4,018 questions from 5
extended question types and maps from 30 cities across 13 countries.
traditional RLHF pipelines (Bai et al., 2022) to more stable and direct policy optimization objectives such as Direct Preference Optimization (DPO) (Rafailov et al., 2023) and Group Relative Policy Optimization (GRPO) (Shao et al., 2024). Recent work further demonstrates the effectiveness of structured or curriculum-baset RI. strategies for enhancing L.M reasoning. Curriculum RI. (Parashar et al., 2025) shows that progressing from easy to hard tasks can significantly strengthen reasoning robustness; Kimi K1.5 (Team et al., 2025) adopts multi-stage RL to improve long-chain and tool- integrated reasoning; and Logic-RL (Xie et al., 2025b) leverages logical structure to guide policy optimization. While SFT on CoT-rich datasets can activate latent reasoning abilities (Xu et al., 2024; Guo et al., 2024; Wei et al., 2025), it often leads to overfitting and cognitive rigidity (Chu et al., 2025; Shen et al., 2025; Wang et al., 2025b). RL provides dynamic alternative by allowing models to learn reasoning policy through exploration and reward (Guo al., 2025a; Zhang et al., 2025a). However, its direct application to complex visual reasoning remains challenging due to sparse reward signals (Quadros et al., 2025; Cao al., 2024), where supervision is only provided the end of long reasoning trajectories (Wang et al., 2025a). These limitations motivate the development of an RL framework tailored for structured visual tasks that alleviates sparse rewards from the outset.
Spatial Reasoning on Maps. Spatial reasoning on transit maps (Wu et al., 2020) has traditionally been approached with multi-stage computer vision systems. These methods first employ Optical Character Recognition (OCR) (Memon et al., 2020; Liu et al., 2024) to extract textual information such as station names, and then use specialized image processing and graph algorithms to identify key topological elements like stations (nodes) and the routes connecting them (edges) (Cherry al., 2006). Pathfinding is subsequently performed on this extracted graph representation of the transit network using classical search algorithms (Noto & Sato, 2000; Deng et al., 2012). While logical, these systems are often brittle, as errors in early stages can propagate and lead to incorrect final paths. MLLMs offer a promising end-to-end alternative, yet benchmarks consistently show they fail fundamental spatial tasks like judging relative positions and orientation (Xing et al., 2025). These failures are often attributed to architectural limitations in how vision encoders process positional information (Chen et al., 2024b). When applied directly to map-based planning, MLLMs struggle to comprehend environmental constraints and execute the multi-hop reasoning required (Feng et al., 2025b). Our work addresses above concerns by constructing cold-start data for fine- fine-grained understanding and through multi-stage reinforcement learning training.
REASONMAP-PIUSS CONSTRUCTION
In this section, we first introduce the construction pipeline of REASONMAP- PPLSS as shown in Figure 1, which builds REASONMAP (Feng et al., 2025b), and comprises three stages: (1) data collection and preprocessing, (2) design-guidee construction of question-ansse answer pairs, and (3) quality control. We report comprehensive statistical overview of REASONMAP-PI.E in Appendix A.1.
3

---

Published as a conference paper at ICLR 2026
Multi-Stage GRPO RL w/ RewardMap
Accarding the aubway map, I from [Bencoolen] to (Clarka Quay|?
PlasonNMap-Plus "o' Reasonflap
Belued
@wen2.5- VL -BBB
Name: Downioen
Bocker
Departure Skop: Bencoslen Station Arrival
Stop: Ciar te Duey Station
Visual Confusion
RewardMap (Ours):
Dosnntrnn Line,
Split
Split
Full
Departure Stop: Bencoplan Station Arriual Stop:
Chinsizen Starkon
Stage
Final Stage
St
Route Name: North East Line
Departure Ohi iitoonn
easy
hard
Arrival Stopp Clerke Guey Stadion
Training Data
0,
Reward Model
D2
Mup Difficutty
R2
Computation
Farmat
Group
Sampling
Guestion Difficulty.
+ Question
Chobb
Accuracy
Details
Policy
Os
Question Type
Model
Basic Reward
Reward by RewardMap
RewardMap
Figure 2: Overview of REWARDMAP. The framework enhances fine-graine -eeegraieed visual understanding and reasoning in MILMs through reinforcement learning with Group Relative Policy Optimization (GRPO). It consists two key components: (1) a difficulty-wware reward design (Section 4.2), which combines format, correctness, and detail rewards with difficulty-base weighting; and (2) multi-stage RL curriculum (Section 4.3), which schedules training data from simple perception tasks to complex reasoning tasks, ensuring effective optimization tackling sparse rewards.
3.1 CONSTRUCTION PIPELINE OF REASONMAP-PLUS
We follow the construction pipeline of REASONMAP on stages (1) and (3) for REASONMAP-PLS Specifically, for (1) data collection and preprocessing, we reuse the collected high-resolution transit maps and annotated line-stop information (refer to the Metro Data) in REASONMAP to drive the following question-answer answer generation; for (3) quality control, we manually review the automatically generated question -nnswer pairs to verify correctness and, when necessary. adjust the question distribution to maintain diversity and a balanced difficulty. We then present the details of stage (2).
Construction of Question-Answer Pairs in REASONMAP-PLE PLUU.. We extend the planning question in REASONMAP to 5 related categories covering counting and True or False questions (Figure 1), preserving consistency between REASONMAP-PLE
and REASONMAP, For each category, we construct questions based on predesigned question templates (see Appendix A.2) and automatically derive answers from the Metro Data to form question-answer pairs.
We consider the following question types: (1) Global Counting assesses global fine-grained visual understanding by asking for the number of lines in map. This type sparse, and each map yields only one question; (2) Local Counting evaluates local fine-grained visual understanding through two variants: counting the intermediate stops between two specified stops, and counting the number of lines that pass through a specified stop; (3) True or False probes
visual understanding through two variants: judging the spatial relation between two specified stops, and between one stop and one line. We balance yes and no answers to prevent models from exploiting label frequency.
Difficulty Annotation. For map difficulty, we follow the manual assignment in REASONMAP, which uniformly categorizes all maps into three levels (e.g., easy, middle, and hard). For question difficulty, these questions probe basic visual understanding of transit maps rather than the complex visual reasoning required by the planning questions in REASONMAP. Accordingly, define question difficulty by the corresponding map difficulty label.
METHODOLOGY
In this section, we propose REWARDMAP to enhance fine-grained visual understanding and reasoning in MLLMs. We begin by presenting the overview of our target tasks, baseline, and our proposed REWARDMAP. We then introduce the difficult--ware reward design and illustrate multi-stage reinforcement learning with the Group Relative Policy Optimization (GRPO) process.
4

---

Published as a conference paper at ICLR 2026
4.1 OVERVIEW
Target Tasks, We investigate two target tasks in this paper: (1) fine-grained visual understanding in REASONMAP-PL.US (Section 3) and (2) fine-grained visual reasoning in REASONMAP (Feng et al., 2025b). Both are cast as Visual Question Answering (VQA), given a high-resolution image I and an instructional question Q, the model must produce an answer Afomatted that conforms to the required output format and correctly addresses Q. Unlike conventional benchmarks, our tasks foreground fine-grained perception to assess a model's ability exploit high-resolution details. Additionally, the route planning task in REASONMAP further evaluates spatial reasoning.
Baseline. We conduct baseline experiments under a standard setup with training data from the original REASONMAP. We train the model with GRPO reinforcement learning with basic reward function consisting of format reward and a correctness reward. However, training on this setting exhibits sparse rewards under high task difficulty (see the results of Wiven.55VVL3/7BB Insttrcct in Table 1). In GRPO, given an input æ and a group G = { 1 of sampled outputs with returns fre the centered group advantage drives policy updates:
K
K Σ 1=1
A,
max (())
A log To at)
K
==1
With sparse rewards, most ry s so A, either collapses to near zero (all failures) or becomes highly skewed (rare positives), yielding low-signal or high-variance gradients and thus slowing convergence.
REWARDMAP. To mitigate the sparse reward issue observed in the baseline, we propose RE- WARDMAP, which consists of two components: (1) -rrinnedreented reward with difficulty weighting, and (2) a multi-stage RL regimen that exploits dense-reward questions from REASONMAP- PLUS for effective cold start. Implementation details are provided below.
4.2 DIFFICULTY-WWARE REWARD DESIGN
Our reward function comprises three terms: format reward, correctness reward, and detail reward, scaled by a difficulty factor and a weighting coefficient for the detail reward:
R W Waiffeuity/ (Riormat + Reorrectess
Rdeail)
where Rivemat, Reverectness and Raetail denote the format, correctness, and detail rewards, respectively; >> O controls the relative strength of the detail term (a > 0 is set to 0.5 for subsequent training); and W difficulty scales the overall reward according difficulty.
Format Reward. The format reward enforces compliance with task-specific output conventions: for REASONMAP- REASONMAP, the reward is computed using the benchmark's 's original formatting specification'.
answers elicited within "\boxed " to localize the output; for
For training data in REASONMAP-PLUS we use exact-match scoring to compute rewards, as these questions have a single deterministic ground truth (e.g., numerals or yes/no). For training data in our proposed REASONMAP, the correctness reward is computed using the benchmark's 's official evaluation algorithm².
Correctness Reward.
Detail Reward. To alleviate sparse rewards caused by the difficulty of planning tasks, we add a detail reward that grants partial credit for correct items of the answer. Specifically, we reward/penalize correctness of the origin and destination stops, route names, transfer stations, and the number of route segments (see the computation pipeline in Algorithm l). Additionally, modulate this influence of detail reward on training with a weighting coefficient a.
Difficulty- Aware Weighting. To incorporate problem difficulty, we scale the sum of the three rewards by a difficulty aware weight. We consider two fine-grained factors: (1) Map difficulty (for all training data from REASONMAP and REASONMAP- PLUS), with weights assigned by the three levels (e.g., g., easy, medium, hard, see Appendix A. 1); and (2) Question difficulty (for the training data in REASONMAP), with weights determined by the transfer count of the required route. The weighting
Pleasse refer to Appendix A.1 in TEASONMAP paper for more details. See Appendix B.I of the REASONMAP paper (Feng et al., 2025b).
5

---

Published as a conference paper at ICLR 2026
Table Evaluations of reference models and fine-tuned models on REASONMAP and REASONMAP-
PLUS.. "S." represents results for short questions, while "I." denotes results for long questions. Bold indicates the best results among fine-tuned models, while underline represents the second best.
RAASONMAP (S.)L.)
REASONMAP-PLUS
Model
Training Data
Weighted Acc.
Weighted Map Score Weighted Ace. Weighted Ace. (Count / TorF)
Reforence Models
Kini VL-ABB Thinking
5.47% 5.47%
2.44/3.17
33.95%
18.17% 50.39%
Kimi-VI SoniVVi-AAB-Instmeee
12.76%
12.33%
3.30/5.37
32.55%
14.75% 51.08%
Deen2.5-VL-3B-In-ee
8.68% / 77.99%
2.75/3.70
37.61%
22.68% 53. 16%
16.49% 15.71%
3.88/6.84
58.32%
46.96% 14%
Qwen2,5-VL-72B--1
26.65% 22%
5.09/8.80
53.21%
13 13/16% 06776/663.60 36%
34.20%
65.26% 82.23%
GPT-4o
41.15% 422 80%
6.84 13.57
64.42%
69. 08%2 69.77%
50.95% 62.500%
9.48
88.05%
80.40% 91.60%
Baxefine & REWARDMAP
Bowen2.--V--7--lmaee
13.28% 7.12%
4.01 5.774
44.21%
39% 51..2%
+ KL (GRPO)
Beram
26.22% 26, 26.017%
5.32 /9762
41.61%
37. 57.57%%77 3.2 01%
RL (REINFORCE++)
Reyocs
27.17% 27. 60%
5.68 10.12
44.61%
36. 88..2% / 5:2 79%
RL (ReMax)
Reram
26.22% 27. 26%
5.37 /9.99
40.39%
38.37% 52. 70%
SFT
RPhusteato
13.6639%
9.11%
4.09 6.25
37.933%
50.73% 63, 44%
SFT RL
R RPhustoii Recoin
28.82%
30.35%
5.88. / 10.62
60.53%
55.38% 65.90%
(baseline) REWARDMAP
RI.
RPlusgaan Havain
29.51%
229.51%
6.00 10.41
67.61%
68.37% 66.82%
RPhustram Revain
31.51%%
31.77%
61.1/11.22
74.25%
72.18% 76.42%
scheme is defined follows:
W difficulty
W map
+
W question:
Ye:
map difficulty easy
30:
transfer count
W map
7mx
map difficulty medium
W Qquestion
31
transfer count
map difficulty = hard
4.3 MULTI-STAGE GRPO-BASE BBSED REINFORCEMENT LEARNING
To effectively exploit REASONMAP-PLUO and alleviate the sparse- rpware-reward issue in complex tasks such as route planning in REASONMAP, we design a multi-stage curriculum built upon GRPO-based reinforcement learning. The core idea is to progressively schedule training data in principled manner, ensuring a smoother optimization process and more stable reward propagation. We adhere to two complementary principles:
(1) Global curriculum principle. We impose a coarse-to-fine learning schedule by partitioning tasks into distinct stages according to both question type (from binary judgment -> counting planning) and target task (from fine-graing -re-grained visual understanding - fine-grained visual reasoning). This global ordering ensures that the agent acquires fundamental perceptual skills before engaging in more abstract reasoning, as illustrated in Figure 2.
(2) Local stochasticity principle. Within each stage, avoid strict deterministic ordering by introducing randomness, i.e., shuffling training samples instead ranking them solely by heuristic difficulty metrics (e.g., map question complexity). This stochasticity prevents overfitting fixed curriculum trajectory and enhances robustness.
By jointly applying these principles, the proposed multi-stage RL scheme transforms curriculum training into structured combination of reward shaping and task scheduling, thereby enabling effective reinforcement learning on inherently sparse-reward visual reasoning problems.
EXPERIMENTS
5.1 EXPERIMENTAL SETUPS
Training Details. We conduct training experiments under various settings with the training data from REASONMAP (696 samples, Rtrain) (Feng et al., 2025b) and our proposed REASONMAP-PLUE (2, 570 samples, RPlustrain) on Qwen2.5-VV VL models (Bai et al., 2025b) with NVIDIA H800 GPUs. For GRPO (Shao et al., 2024) RL training, AdamW with initial learning rate of

---

Published as a conference paper at ICLR 2026
Table 2: Evaluation of reference models and fine-tuned models on various benchmarks. Bold indicates
the best results among fine-tuned models, while underline represents the second best. 8, 8, *, denote the results from the technical report or the official HuggingFace repository (see result sources in Appendix C.3), while all other results are obtained from our own experiments.
Spatial Reasoning
Fine-grained Visual Reasoning
General Task
Model
SEED-Bench- 2- SpatialEval
V
IRBeenc
ChurQQA
MMStur
Reference Models
Kimi-VL-ABB VL-A33-Innttru
58.49%
52: 02.6
59.16%
35. 35%
87.08%
61.70%
62.41%
Qwen2. 0-1-2-5--------smmmmmm
58.86%
56.04%
1256
66. 25%
84.00364
50.90%6
63.03%
Qwen2.5.VL S-----L20-----e--- Instrrctt
72.10%
57.99%
82.72%
63.50%
64.90%
69.50%
68.45%
Qwen2 1.222.5... 5.2... M-.--1172---niiii
70.00%
86.40%
80.00%
70.8036
$3.00%
87.400%
76.20%7
Museline REWARDMAP
Qwen2.5-VI. 1---2----V--7--nstnte
60.97%
57. 10%
78.01%
68. 75%
86.12%
61.67%
69.80%%
SFT + RL REWARRMAP
61 55% 10.02%
G9. 06%
01%
TO
100
86.92%
TO
62, 27%
T1. 18%
-2.6855
61.96% 1999
70.81% 11 80. 10%
19 0005
25%
1888138800 13.600
87. 11.19% 612. 222.27%
to 72.2773 13 47%
1.0 10 and a KL divergence coefficient of 1.0 x 10-3, The global batch size is 16, and we sample 8 responses per query. Besides RL training, we conduct baseline experiments using SFT with training data from REASONMAP PP.P.... We apply LoRA (Hu et al., 2022) to the language blocks and the text-vision projector with an initial learning rate of 1.0 10-4, For implementation, we adopt LLaMA-Factory (Zheng et al., 2024) and VeRL (Sheng et al., 2024). Additionally, we include two RL training baselines with REINFORCE+ (Hu et al., 2025a) and ReMax (Li et al., 2023).
Inference Details. All evaluations use greedy decoding (temperature=0), For open-source models, we cap the maximum output length 2, 048 tokens and retain all other settings from the official HuggingFace configurations; deployments use PyTorch with Transformers' on 8 NVIDIA H800 GPUs. For closed-source source models, we evaluate through official APIs using default settings. The evaluated models include: 3/7/32/72B-Instruc (Bai et al., 2025b), heed1.5-vi (Guo al., 2025b), GPT-5/40 (OpenAI, 2024).
(Team et al., 2025), Qwen2.5-VI.
Evaluation Datasets. We first evaluate on the test sets of REASONMAP and REASON PLuS, and we report the metrics adjusted by the difficulty-aware weighting scheme (see details in Appendix C.I). To further assess the capability gains brought by REWARDMAP, we next employ six widely-used benchmarks spanning three dimensions (e.g., spatial reasoning, fine- -reiggaaied visual reasoning, and general tasks): SEED-Bench-2.
(Li et al., 2024), SpatialEval (Wang et al., 2024a), V V*Bench Wu Xie (2024), HRBench (Wang et al., 2025f), ChartQ (Masry et al., 2022), and MMStar (Chen et al., 2024a) (see Appendix C.2 for details). All evaluations are conducted with VLMEvalKir'.
5.2 MAIN RESULTS
Results on REASONMAP and REASONMAP-PLUS We evaluate the proposed REWARDMAP on REASONMAP and REASONMAP-PIA PLUS,, both of which provide tne-grained difficulty annotations
to assess visual understanding, visual reasoning, and spatial reasoning. In addition to the baselines introduced in Section 1, we compare against two widely-used settings: (1) SFT - RL baseline, which applics SFT on REASONMAP-PAA -PLU followed by RL on REASONMAP, and (2) RL baseline, which performs RL the combined training data from REASONMAP and DEASONMAP-PLUS For both baselines, the reward design includes only format and correctness terms.
As shown in Table I, REWARDMAP consistently outperforms all baselines across different ques- tion templates in REASONMAP and question types of REASONMAP- substantially surpasses the best open- result (Qwen2.5-VL-722B VL-72B-Inntruct) and approaches the performance of the closed-source model (SeedI.5-VL.). On REASONMAP PP-US, REWARDMAP not only exceeds all open-source models but also outperforms Seed1.5-VL.
On REASONMAP,
Results on Other Benchmarks. We further evaluate the generalization ability of our method on six benchmarks spanning three task categories introduced in Section 5.1. Table 2 reports the results of reference models, the SFT + RL baseline, and our proposed REWARDMAP. Across all six benchmarks, REWARDMAP achieves consistent improvements, with the most substantial gain of
hhtpps://gt thub. www.b..coo/oocoooooooooeeeoeee/e www.///pp--aeeeo wwwpm///pecccopaas///oooooo VVMMva Ki
7

---

Published as a conference paper at ICLR 2026
Example
G: According to the map, how do get from 52 to ,
?h? + Question Template
@wen2.5-VL-7B-B Route Name: Line 1',n Departure Stop: A6 n Arrival Stop: #
Visual Confusion
-WV-n2.-VLL-7B-1- RewardMap: Route Name: Line 5
Departure Stop: e0 m
Arrival Stop: #
Example b
Baliina cxciria Menica
Herkege Sa
G: According to the map, how do I get from 7th St/ //Mtrtro Ctr Little Takyo/ Arts Dist? Question Template
Vernonn/Bewerty
Wered Au Beve
Chisatons
Kimi-VL-A3E Thinking: <hinke A thinle Route Name: Line A Departure Stop: 7th SistrDistrict Station' n- Route Name: Line A Departure Stop: 7th St.t.District Station Hallucination
DOWNTOWN LA
fso F31S)
Maceetrr Park
peeching
Th ScMarru
Liste nere that
Pleeitioon
Pasza
Qwan2.5-VL-22
RewardMap: Route Name: Line A
LAT
Prop
Departure Stop: 7th St/ Metro Ctr Station
Jefeeent
LATT
Arrival Stop: Little Takyo/ Arts District Station
Example
Q: According to the map, how do I get from Puhinui Station to Waitemată (Britomart) Station?+ Guestion Template
Qwen2.5- ---1-B Baseline RL: Route Name: Northern
(-10)
6-40
Busway I Departure Stop: Puhinui Station \ Arrival Stop:
Waitematä (Britomart) Station
Gremiare Dasc
Percce
Visual Canfusion
Procheete
Diven2.5-L-77B + RewardMap: Route Name: Eastarn Line
To Novp 2.19
nn Departure Stop: Puhinui Station \n Arrival Stop: Waitemată
Olick argc
Atroet
(Britomart) Station
Figure 3: Qualitative comparisons among reference models, baseline, and our proposed RE- WARDMAP. We crop and zoom in on the transit map for clearer presentation.
13.51% observed on SpatialEval. While the SFT RL baseline also yields stable improvements, its performance remains inferior to REWARDMAP. These results highlight the significant contributions of both REWARDMAP and the datasets (both REASONMAP and REASONMAP-PLUS) to enhancing model general capability. Beyond the aggregate results, we further report task-wise improvements. On SEED-Bench-22Piie REWARDMAP improves the map split. For SpatialEval, accuracy rises substantially on mazenav (19.60% + 57. (0%) and moderately on spatialreal (70.37% + 72.59%). HRBench, performance increases for both single-image reasoning 85.25% + 88.00%) and cross-image alignment (52.25% - 54.00%). For MMStar, REWARDMAY achieves steady gains across fine-grained perception (59.60% + 60.80%) instance reasoning (68.00% + 71.20%), and mathematical reasoning 60% + 63. 63...%). These results show that REWARDMAP consistently
strengthens visual grounding and robustness in diverse spatial and reasoning tasks.
5.3 QUALITATIVE RESULTS
Figure 3 presents qualitative comparison between reference models, the baseline RL model, and our proposed REWARDMAP. Across diverse maps, we observe that reference models and the baseline RL. model often suffer from visual confusion (e.g., mistaking the route or stop as shown by Example & с in Figure 3) or even hallucination (e.g., repeating the same route many times in Figure 3 (b)). In contrast, REWARDMAP consistently produces the correct target routes. These examples highlight the effectiveness of REWARDMAP in handling visually complex maps (e.g., Example & b in Figure 3) and reducing both visual confusions and hallucinations. Additionally, we present further comparison cases that provide additional evidence that REWARDMAP improves visual grounding and reduces the likelihood of visual confusion or hallucination. These examples are included in Appendix D.
5.4 DIAGNOSTIC EXPERIMENTS
We provide extensive ablation studies using Qwen2.5-VL. tiveness of REWARDMAP in addressing sparse-rewaid issues, across different model scales (e.g., Qwen22.5-VL--B--nnesee Instrrucc) and different model architectures (e.g., --------ssttuc)..
VL- -LL-7B-Instruct, and further verify the effec-
8

---

Published as a conference paper at ICLR 2026
Table 3: Ablation on reward design and multi-stage design of REWARDMAP. "S." represents results for short questions, while "I." denotes results for long questions.
REASONMAP (5./L.)
REASONMAP-PLLL
Model
Training Data
Weighted Acc.
Weighted Map Score Weighted Acc. Weighted Acc. (Count / TorF)
RL (baseline)
Rivain
26.22% / 26 06.04%
5.52/ 9.52
44.64%
37.57% / 52.001%
RL + Reward Design
Revamn
29.08% / 29.95%
5.88/ 10.53
15.16%
37.79% / 52.84%
RI. (huseline)
RP Mavain 29.51% 29.51%
6.00/ 10.41
67.61%
68.37% / 66.82%
RL + Reward Design
RPlustram Reoain 30. 30.36% / 30.38%
6.12/ 10.62
71.07%
67.61% / 771.6%%
Multi-Stage Stage Design RPlustran Becai 30.61% / 31.55%
6.08 10,88
73.12%
69.52% 76.88%
REWARDMAP
RPlusteat + Recais 31.51% 31.77%
6.21 11..22
74.25%
72.18% / 76.42%
Table 4: Ablation on REASONMAP-PLCS. "S." represents results for short questions, while "L." denotes results for long questions.
REASONMAP (S./L)
REASONMAP-
Model
Training Data
Weighted Acc.
Weighted Map Score Weighted Acc. Weighted Acc. (Count / TorF)
RI. (haseline) Rerain
26.22% / 26.04%
5.52 /9.52
44.64%
37.57% 52.01%
RL (baseline) RPlustraai + Beroin 29.51% / 29.51%
6.00 / 10.41
67.61%
68.37% 66. 82%
SFT RL RPluSerain Rerain 28.82% / 30.38%
5.88 / 10.62
60.53%
55.38% 65.90%
Ablation on Reward Design and Multi-Stage Design of REWARDMAP. We ablate the reward design and multi-stage design of REWARDMAP under two training data settings. For the configuration using only REASONMAP training data, where the multi-stage design cannot be applied, reward design alone. As shown in Table 3, enabling either component individually yields performance gains both REASONMAP and REASONMAP-PLAN confirming the effectiveness and complementarity of the two components.
ablate the
while combining them achieves the best results,
Ablation on REASONMAP-PLUS Next, assess the impact of REASONMAP- PLUS under two training strategies (e.g. SFT and RL). As shown in Table 4, conducting cold-start training with REASONMAP- consistently improves performance on both REASONMAP and REASONMAP- PLUS, regardless of the chosen strategy.
Ablation on Granularity of Multi-Stage Design. We further conduct an ablation study on the granularity of the multi-stage design by comparing a coarse-grained variant with our proposed REWARDMAP. As shown in Table 5, switching to the coarse strategy leads to performance degradation while still outperforming the baseline, thereby reinforcing the effectiveness of the multi-stage design.
Ablation on the Hyperparameter a. Based on the full REWARDMAP pipeline, we varied only a. The results are reported in Table 6. We observe slight performance drop when is small, while the performance becomes similar once a enters a higher range. These results suggest that the detailed reward is indeed beneficial (as a determines its contribution to the total reward), and they also indicate that a is not sensitive within a reasonable interval.
Ablation on the Difficulty-Aware Weighting Scheme. Using the REASONMAP training set, further conduct two ablations: one varying only the relative magnitudes of Ne/m/h and an- other varying only those of 90/1- As shown in Table 7, performance decreases slightly when
False
Counting
Planning
Mean Truining Reward
the weights become nearly uniform, but remains stable once clear difficulty distinctions are intro- duced. These results support the effectiveness of our difficulty-awar and B0/1 modulate map and question difficulty, respectively-ante and indicate that both weights are insensitive within a reasonable range.
weighting -weeii -where Te/mm/ hh
30
300
12s
150
175
200
Step
Effectiveness of Tackling Sparse Reward. We validate the effectiveness of REWARDMAP to ad- dress sparse rewards. As shown in Figure 4, we compare reward trajectories of REWARDMAP with a baseline RL trained on REASONMAP, aligned at the planning stage. The results show
Figure 4: Comparison of training rewards between baseline RL and REWARDMAP. The yellow curve denotes the reward trajectory of AEWARDMAP, while the blue curve corresponds to the baseline RL trained solely on REASONMAP.
that REWARD MRAMAAAA MAP alleviates reward sparsity, further confirming its effectiveness.
9

---

Published as a conference paper at ICLR 2026
Table 5: Ablation on granularity of multi-stage design. "S." represents results for short questions, while "L." denotes results for long questions.
MASNNMPPPPLUS
Model
Training Data
Weighted Acc.
Weighted Mup Score Weighted Acc. Weighted Acc. (Count / TorF)
(baselinc)
Relarniin + Rerutn 29. 19.51% 29.51%
6.007 10.41
67.61%
68.37% 06 82%
RI (cnarse- graineeg Multi-Stage) Stuge)
RPlusiaai Rirsin 29.60% 30.21%
5.98 10.83
70.30%
06.02% / 774. 77.76%
RL (Mugli REWARDMAP RiPuinn Ruvein 31.51% 71.777
6.21
74.25%
72.18% 76.42%%
Table 6: Ablation on the hyperparameter a. "S." represents results for short questions, while "L."
denotes results for long questions.
(SASONMAP(S./L)
KEASONMAP-PLUS
Model
Training Data
Weighted Ace.
Weighted Map Score Weighted Acc. Weighted Acc. (Count / /TorF)
REWARDMAP [n==0,3] RPlusteain + Ruvain 30.73% 3611
6.01 / 10.81
72.08% 71.25%
68. 41% / 75.91%
REWARDMAP [a=0..5] RPlustvain + Rarii 31.51 51%51% 731.77%
6.21 / 11.22
12.18% 76.42%%
REWARDMAP [n=0.77] RPlusteain + Rerain 32.03% /32 20%
6.23/ 11.20
72.81%
70.14% / 75.5%%
Table 7: Ablation on the difficulty-aware weights. "S." represents results for short questions, while
"L." denotes results for long questions.
MAASONMAP (S.7L)
REASON PLUS
Model
Training Data
Weighted Acc. Weighted Map Soore WeighWedt Ace. Weighted Acc. (Count TorF)
Qwen2 S-VL--BB-nnstuue
13.28% 12%
1.01/5.77
11.21%
37.39% 51.32%
Reward Design ['aja/h (1.0, 1.1, 1.2)]
Rerain
26.91% 28.47%%
5.64 10:24
45.34%
37.97% / //3.3329
Reward Design P1////h (1.0, 1.2, 1.5)]
Rrain
29.05% 29.95%
5.88 10.53
45.16%
37.79% 552.84%
Reward Design Peemm/h [1.0, 1.5, 2.0)]
Rerain
29.80% / 28.90%
5,93 10.31
45. 16%
37.31% / 63.30%
Reward Design Ld6/1 00.0,0.2]]
Rerain
28. 29..00% 2:133
5.68 10.12
41.00%
37.39% / 52, 552.10%
Reward Design [e, (0.0.0.5]
Rerain
29.08% / 29.9699 9699
5.88/ 10.53
45.16%
37.79% / 662.84%
Reward Design [06/1 (0.0.0.8)
Rerain
29.60 29.60% / 29.319
5.89 10.23
45.39%
38.68% / 55..88%
Table 8: Evaluation of REWARDMAP across model scales. "S." represents results for short questions,
while "L." denotes results for long questions.
REASONMAP (S./L.)
REASONMAP- A--MMMPPPPPSSSS
Model
Training Data
Weighted Acc.
Weighted Map Score Weighted Acc. Weighted Acc. (Count / TorF)
Qwen2.5 --L--------- (baseline)
8.68% / 7..99%
2.75 3,70
37 37.61%
22.68% / 53.16%
Recain
11.46% / 10.50%
3.81 6.09
38.29 29%
22.06% 55.19%
REWARDMAP
RPlusteain + Rysin 19.36% / 15.89%
4.79 / 7.53
65.91%
63.58% / 34%
Effectiveness of REWARDMAP across Model Scales, We evaluate the effectiveness of RE- WARDMAP across different model scales. Due to training cost constraints. we adopt Qwen2.5- VL-3B-Instrun as the base model and compare the baseline RL with REWARDMAP. As shown in Table 8, REWARDMAP achieves the promising results, demonstrating its robustness and effectiveness.
Table 9: Results on Kimi-VL (Kimi- VLL-AB--Instruch "S." represents results for short questions, while "L." denotes results for long questions.
REASONMAP (S.LL))
REASONMAP-PLCC
Model
Training Data
Weighted Acc. Weighted Map Score Weighted Acc. Weighted Acc. (Count / TorP)
Kimi-VL-A3B--nstree
12.76% / 12.33%
3.30/5.33
32.55%
14.75% / 51.08%
+ REWARDMAP
Recoin
18.58% / 17.36%
4.70/7.69
35.92%
15.20% / 57.50%
Generalization of REWARDMAP across Model Architectures. We further trained Kimi- VL (Kimi- VL-A3B- with the REWARDMAP pipeline using the training set from REASONMAP. As shown in Table 9, the model achieves substantial performance gains, demonstrating that our method generalizes beyond the Qwen series.
6 CONCLUSION
In this work, we address the challenge of applying reinforcement learning to fine- ane--grained visual reasoning, where sparse rewards and long reasoning horizons often hinder effective optimization. Building upon the REASONMAP benchmark, we introduce REASONMAP-PIUS an extended dataset that organizes tasks along a difficulty continuum, providing dense supervision to facilitate cold-start training. Furthermore, we propose REWARDMAP, a multi-stage reinforcement learning framework that combines curriculum-style task scheduling with difficulty-aware reward design. Our experiments demonstrate that each of these components contributes to stable and effective training, and that their integration yields the strongest improvements. REWARDMAP not only advances performance REASONMAP and REASONMAP-PE but also enhances robustness across broader visual reasoning benchmarks, indicating improved perceptual and reasoning capabilities of multimodal models.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This paper is supported by Young Scientists Fund of the National Natural Science Foundation of China (NSFC) (No. 62506305), Zhejiang Leading Innovative and Entrepreneur Team Introduc- tion Program (No. 2024R01007), Key Research and Development Program of Zhejiang Province (No. 2025C01026), Scientific Research Project of Westlake University (No. WU2025WF003), Chinese Association for Artificial Intelligence (CAAI) & Ant Group Research Fund AGI Track (No. 2025CAAI-ANT-13) is also supported by the research funds of the National Talent Program and Hangzhou Municipal Talent Program.
ETHICS STATEMENT
This work does involve human subjects or sensitive personal data. REASONMAP-PLE is constructed from publicly available transit maps with automatically generated question-answer pairs, ensuring no privacy or security concerns. The datasets are released exclusively for academic research under the Apache License 2.0 on HuggingFace, and all reported results are fully reproducible with released code and configurations.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we provide the evaluation setup details in Scction 5.1 and Appendix C, including hardware and implementation, which facilitates rapid replication.

---

Published as a conference paper at ICLR 2026
REFERENCES
Zhenxin Ai, Huilan Luo, and Jianqin Wang. A lightweight multistream framework for salient object detection in optical remote sensing. IEEE Transactions on Geoscience and Remote Sensing, 63: 1-15, 2025.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716- 23736, 2022.
Haolei Bai, Siyong Jian, Tho Liang, Yu Yin, and Huan Wang. ResSVD: Residual compensated SVD for large language model compression. arXiv preprint arXiv:2505.20112 2025a.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-Ve VL technical report. arXiv preprint arXiv: 13923, 2025b.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 2022.
Mahtab Bigverdi, Zelun Luo, Cheng-Y Yu Hsich, Ethan Shen, Dongping Chen, Linda G Shapiro, and Ranjay Krishna. Perception tokens enhance visual reasoning in multimodal language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3836 33355,845, 2025.
Meng Cao, Lei Shu, Lei Yu, Yun Zhu, Nevan Wichers, Yinxiao Liu, and Lei Meng. Beyond sparse rewards: Enhancing reinforcement learning with language model critique in text generation. arXiv preprint arXiv:2401.0777 00.38822.... 2024.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we the right way for evaluating large vision- -aiiuugee models? Advances in Neural Information Processing Systems, 37:27056-2708% 2024a.
Weizhe Chen, Sven Koenig, and Bistra Dilkina. Why solving multi-agent path finding with large language model has not succeeded yet. arXiv preprint arXiv:2401.03633 2024b.
Xinghao Chen, Anhao Zhao, Heming Xia, Xuan Lu, Hanlin Wang, Yanjun Chen, Wei Zhang, Jian Wang, Wenjie Li, and Xiaoyu Shen. Reasoning beyond language: A comprehensive survey latent chain-of thought reasoning. arXiv preprint arXiv:2505. 2025.
Christopher Cherry, Mark Hickman, and Anirudh Garg. Design of map-based transit itinerary planner. Journal of Public Transportation, 9(2):55-68 2006.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc Le, Sergey Levine, and Yi Ma. SFT memorizes, RL generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501.17161 2025.
Yong Deng, Yuxin Chen, Yajuan Zhang, and Sankaran Mahadevan. Fuzzy Dijkstra algorithm for shortest path problem under uncertain environment. Applied Soft Computing, 12(3):123-11237, 2012.
Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Winston Hu, Yongming Rao, and Ziwei Liu. Insight-V: V: Exploring long-chain visual reasoning with multimodal large language models. In IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 9062-9072, 2025.
Sicheng Feng, Keda Tao, and Huan Wang. Is oracle pruning the true oracle? arXiv preprint arXiv:2412.00143. 2024.
Sicheng Feng, Gongfan Fang. Xinyin Ma, and Xinchao Wang. Efficient reasoning models: A survey. arXiv preprint 2rXiv:2504. 10903, 2025a.
12

---

Published as a conference paper at ICLR 2026
Sicheng Feng, Song Wang, Shuyi Ouyang, Lingdong Kong, Zikai Song, Jianke Zhu, Huan Wang. and Xinchao Wang. Can MLLMs guide me home? a benchmark study on fine-grained visual reasoning from transit maps. arXiv preprint 2.Kiv:2505.8675, 2025b.
Sicheng Feng, Zigeng Chen, Xinyin Ma, Gongfan Fang, and Xinchao Wang. dvoting: Fast voting for dllms. arXiv preprini arXiv:2602. /2153, 2026.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. DeepSeek-RI: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprini arXiv:2501. /2948, 2025a.
Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang. Jianyu Jiang, Jiawei Wang, et al. Seedl. 55-VL technical report. arXiv preprint (tt::250500706... 2025b.
Jarvis Guo, Tuney Zheng, Yuelin Bai, Bo Li, Yubo Wang, King Zhu, Yizhi Li, Graham Neubig, Wenhu Chen, and Xiang Yue. MAmmoTH-VL: Eliciting multimodal reasoning with instruction tuning at scale. arXiv preprint arXiv:2412.05237, 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. Reinforce++: An efficient RLHF algorithm with robustness to both prompt and reward models, arXiv preprint arXiv:2501.0326. 2025a.
Tianshuai Hu, Xiaolu Liu, Song Wang. Yiyao Zhu, Ao Liang, Lingdong Kong, Guoyang Zhao, Zeying Gong, Jun Cen, Zhiyu Huang, Xiaoshuai Hao, Linfeng Li, Hang Song. Xiangtai Li, Jun Ma, Shaojie Shen, Jianke Zhu, Dacheng Tao, Ziwei Liu, and Junwei Liang. Vision-language-actin models for autonomous driving: Past, present, and future. arXiv preprint arXiv:2512.167 2025b.
Xin Jin, Siyuan Li, Siyong Jian, Kai Yu, and Huan Wang. MergeMix: A unified augmentation paradigm for visual and multi-modal understanding. arXiv preprint arXiv:2510.23479 2025.
Lingdong Kong, Dongyue Lu, Ao Liang, Rong Li, Yuhao Dong, Tianshuai Hu, Lai Xing Ng, Wei Tsang Ooi, and Benoit R. Cottereau. Talk2Event: Grounded understanding of dynamic scenes from event cameras. arXiv preprint arXiv:2507. 17664, 2025a.
Lingdong Kong, Wesley Yang, Jianbiao Mei, Youquan Liu, Ao Liang, Dekai Zhu, Dongyue Lu, Wei Yin, Xiaotao Hu, Mingkai Jia, Junyuan Deng, Kaiwen Zhang, Yang Wu, Tianyi Yan, Shenyuan Gao, Song Wang, Linfeng Li, Liang Pan, Yong Liu, Jianke Zhu, Wei Tsang Ooi, Steven C. H. Hoi, and Ziwei Liu. 3D and 4D world modeling: A survey. arXiv preprint arXiv:2509.07996 2025b.
Bohao Li, Yuying Ge. Yi Chen, Yixiao Ge, Ruimao Zhang, and Ying Shan. Seed-Bench-2-Plus: Benchmarking multimodal large language models with text-rich visual comprehension. arXiv preprint ar Xiv::24004 16790, 2024.
Bonan Li, Zicheng Zhang, Xingyi Yang, and Xinehao Wang- Coser: Towards consistent dense multiview text-to-image generator for 3d creation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 2880-2890, 2025a.
Qi Li and Xinchao Wang. Sponge tool attack: Stealthy demial-of-efficiene against tool tool-augmented agentic reasoning. arXiv preprint 2rXiv.2601.17566 2026.
Qi Li. Runpeng Yu, and Xinchao Wang. Vid-SME: Membership inference attacks against large video understanding models. arXiv preprint 2tt.:::25000031179. 2025b.
Rong Li, Yuhao Dong, Tianshuai Hu, Ao Liang, Youquan Liu, Dongyue Lu, Liang Pan, Lingdong Kong, Junwei Liang, and Ziwei Liu. 3EED: Ground everything everywhere in 3D. arXiv preprint urXiv:25/1.0/75 2025c.
Rong Li al. SeeGround: See and ground for zero-shot vpcabuacyyylary 3D visual grounding. In IEFF/VVF Conference Computer Vision and Pattern Recognition, pp. 3707-3777, 2025d.
13

---

Published as a conference paper at ICLR 2026
Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi- Quan Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. arXiv preprint arXiv:2310.105000 2023.
Ziqi Li, Tao Gao, Yisheng An, Ting Chen, Jing Zhang, Yuanbo Wen, Mengkun Liu, and Qianxi Zhang. Brain-inspired spiking neural networks for energy-efficient object detection. In IEEE/CVF Conference on Computer Vision and Patlern Recognition, 2025e.
Ao Liang, Lingdong Kong, Tianyi Yan, Hongsi Liu, Wesley Yang. Ziqi Huang, Wei Yin, Jialong Zuo, Yixuan Hu, Dekai Zhu, Dongyue Lu, Youquan Liu, Guangfeng Jiang, Linfeng Li, Xiangtai Li, Long Zhuo, Lai Xing Ng, Benoit R. Cottereau, Changxin Gao, Liang Pan, Wei Tsang Ooi, and Ziwei Liu. WorldLens: Full-spectrum evaluations of driving world models in real world. arXiv preprint arXiv:2512. 10958, 2025.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems, volume 36, pp. 34892-349166 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. LLaVA-NeXT: Improved reasoning, OCR, and world knowledge. arXiv preprint arXiv:2401 13601, 2024.
Yunze Man, De-An Huang, Guilin Liu, Shiwei Sheng, Shilong Liu, Liang- Yan Gui, Jan Kautz, Yu-Xiong Wang, and Zhiding Yu. Argus: Vision-centric reasoning with grounded chain-of-thought In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14268- 14280, 2025.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A bench- mark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.1 10244, 2022.
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographic VQA. In /EEE/CVF Winter Conference on Applications of Computer Vision, pp. 1697-1706, 2022.
Jamshed Memon, Maira Sami, Rizwan Ahmed Khan, and Mueen Uddin Handwritten optical character recognition (ocr): A comprehensive systematic literature review (slr). IEEE Access, 8: 142642-142668, 2020.
Masato Noto and Hiroaki Sato. A method for the shortest path search by extended Dijkstra algorithm. In IEEE International Conference Systems, Man and Cybemetics, volume 3, pp. 2316-2320, 2000.
OpenAl. Hello GPT4-o. https::/ http::///ppppaaaa.ccc//////eee/r 2024.
OpenAl. OpenAl o3 and 04-mini System Card.
cttpp//nniccccc///////epcp 0711c875-22d-47885 212-0--000000077552222222000-0-- pdf, 2025.
Shubham Parashar, Shurui Gui, Xiner Li, Hongyi Ling, Sushil Vemuri, Blake Olson, Eric Li, Yu Zhang, James Caverlee, Dileep Kalathil, et Curriculum reinforcement learning from easy to hard tasks improves LLM reasoning, arXiv preprint arXiv:2506.066 066.26 2025.
André Quadros, Cassio Silva, and Ronnie Alves. LLM-driven intrinsic motivation for sparse reward reinforcement learning. arXiv preprint 2xxii:2508./8422 2025.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In Advances in Neural Information Processing Systems, volume 36, pp. 53728-537411 2023.
Gabriel Sarch, Snigdha Saha, Naitik Khandelwal, Ayush Jain, Michael Tarr, Aviral Kumar, and Katerina Fragkiadaki. Grounded reinforcement learning for visual reasoning. arXiv preprint arXiv:2505.23678. 2025.
Kele Shao, Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. HoliTom: Holistic token merging for fast video large language models. arXiv preprint arXiv:2505.2133. 2/334, 2025a.
14

---

Published as a conference paper at ICLR 2026
Kele Shao, Keda Tao, Kejia Zhang, Sicheng Feng, Mu Cai, Yuzhang Shang, Haoxuan You, Can Qin, Yang Sui, and Huan Wang. When tokens talk too much: survey of multimodal long-context token compression across images, videos, and audios. arXiv preprint arXiv:2507.20199 2025b.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li. Y Wu, et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint wrXiv:2402.03300 2024.
Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. VLM-RI: A stable and generalizable RI-style large vision-language model. arXiv preprint arXiv:2504.076/5, 2025.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhí framework. arXiv preprint arXiv: 2409. /0../922 19256, 2024.
Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. Dycoke: Dynamic compression of tokens for fast video large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, 2025a.
Keda Tao, Kele Shao, Bohan Yu, Weiqiang Wang, Huan Wang, et al. Omnizip: Audio-guided dynamic token compression for fast omnimodal large language models. arXiv preprint 2..::22//11..88. 2025b.
Keda Tao, Haoxuan You, Yang Sui, Can Qin, and Huan Wang. Plug-and-play 1.x-Bit KV cache quantization for video large language models. arXiv preprint arXiv:2503. /6257, 2025c.
Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, et al. Kimi-VL technical report. arXiv preprint NKKV:22504.04491 2025.
Chenru Wang, Beier Zhu, and Chi Zhang. Free lunch for stabilizing rectified flow inversion. arXiv preprint arXiv:2602.//850 2026.
Jiawei Wang, Jiacai Liu, Yuqian Fu, Yingru Li, Xintao Wang. Yuan Lin, Yu Yue, Lin Zhang, Yang Wang, and Ke Wang. Harnessing uncertainty: mntropy-modulated policy gradients for long-horizon LLM agents. arXiv preprint arXiv:2509.0922 09265, 2025a.
Jiayu Wang, Yifei Ming. Zhenmei Shi, Vibhav Vineet, Xin Wang, Sharon Li, and Neel Joshi. Is a picture worth a thousand words? delving into spatial reasoning for vision language models. Advances in Neural Information Processing Systems, 37:75392 75421, 2024a.
Song Wang, Wentong Li, Wenyu Liu, Xiaolu Liu, and Jianke Zhu. LiDAR2Map: In defense of LiDAR- based semantic map construction using online camera distillation. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5186-5199 2023.
Song Wang, Gongfan Fang, Lingdong Kong, Xiangtai Li, Jianyun Xu, Sheng Yang, Qiang Li, Jianke Zhu, and Xinchao Wang. PixelThink: Towards efficient chain-of pixel reasoning. arXiv preprint arXiv:2505.23722 2025b.
Song Wang, Lingdong Kong, Xiaolu Liu, Hao Shi, Wentong Li, Jianke Zhu, and Steven C. H. Hoi. Forging spatial intelligence: A roadmap of multi-modal data pre- pre-training for autonomous systems. arXiv preprint arXiv:2512.24385 2025c.
Song Wang, Xiaolu Liu, Lingdong Kong, Jianyun Xu, Chunyong Hu, Gongfan Fang, Wentong Li, Jianke Zhu, and Xinchao Wang. PointLoRA: Low-rank adaptation with token selection for point cloud learning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 6605-6615, 2025d.
Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internv13.5: Advancing open speresource multimodal models in versatility, reasoning, and efficiency. arXiv preprint 2rXiv:2508. / 18265, 2025e.
15

---

Published as a conference paper at ICLR 2026
Wenbin Wang, Liang Ding, Minyan Zeng, Xiabin Zhou, Li Shen, Yong Luo, Wei Yu, and Dacheng Tao. Divide, conquer and combine: A training- free framework for high-resolution image perception in multimodal large language models. In AAAI Conference Artificial Intelligence, volume 39, pp. 7907-7915, 2025f.
Yaoting Wang. Shengqiong Wu, Yuecheng Zhang, Shuicheng Yan, Ziwei Liu, Jiebo Luo, and Hao Fei. Multimodal chain-of-thought reasoning: A comprehensive survey. arXiv preprint (rXiv:2503./2605, 2025g.
Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, al. Charxiv: Charting gaps in realistic chart understanding in multimodal L.Ms. In Advances in Neural Information Processing Systems, 2024b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. thain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022.
Lai Wei, Yuting Li, Kaipeng Zheng, Chen Wang, Yue Wang, Linghe Kong, Lichao Sun, and Weiran Huang. Advancing multimodal reasoning via reinforcement learning with cold start. arXiv preprint arXiv:2505.22333: 2025.
Hsiang-Yun Yun Wu, Benjamin Niedermann, Shigeo Takahashi, Maxwell J Roberts, and Martin Nöllenburg. A survey on transit map layout-from design, machine, and human perspectives. In Computer Graphics Forum, volume 39, pp. 619-646. Wiley Online Library, 2020.
Penghao Wu and Saining Xie. V*: Guided visual search as a core mechanism multimodal LLMs. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13084 13094, 2024.
Weijia Wu, Chen Gao, Joya Chen, Kevin Qinghong Lin, Qingwei Meng, Yiming Zhang, Yuke Qiu, Hong Zhou, and Mike Zheng Shou. Reinforcement learning in vision: A survey, arXiv preprint arXiv:2508.088 19, 2025a.
Xianzu Wu, Zhenxin Ai, Harry Yang, Sernam Lim, Jun Liu, and Huan Wang. Niagara: Normal- integrated geometric affine field for scene reconstruction from a single view. IEEE Transactions Circuits and Systems for Video Technology, 2025b.
Zeyu Xiao, Zhuoyuan Li, and Wei Jia. Doclusion-embeddde hybrid transformer for light field super-resolution. In AAAI Conference on Artificial Intelligence, 2025.
Shaoyuan Xie, Lingdong Kong, Yuhao Dong, Chonghao Sima, Wenwei Zhang, Qi Alfred Chen, Ziwei Liu, and Liang Pan. Are VLMs ready for autonomous driving? empirical study from the reliability, data, and metric perspectives. In IEEE/CVF International Conference on Computer Vision, pp. 6585-6597, 2025a.
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-RL: Unleashing LLM reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2502. 14768, 2025b.
Shuo Xing, Zezhou Sun, Shuangyu Xie, Kaiyuan Chen, Yanjia Huang, Yuping Wang. Jiachen Li, Dezhen Song, and Zhengzhong Tu. Can large vision language models read maps like a human? arXiv preprini 2503. /460000033.77..... 2025.
Guowei Xu, Peng Jin, Ziang Wu, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. LLaVA-CoT: Let vision language models reason step-by-step. arXiv preprini 2rXiv:2411.10444 2024.
Weiye Xu, Jiahao Wang. Weiyun Wang, Zhe Chen, Wengang Zhou, Aijun Yang. Lewei Lu, Houqiang Li, Xiaohua Wang. Xizhou Zhu, et al. Visulogic: A benchmark for evaluating visual reasoning in multi-modal large language models. arXiv preprint a5Xiv:2504. 15279, 2025.
Tianyi Yan, Tao Tang, Xingtai Gui, Yongkang Li. Jiasen Zhesng, Weiyao Huang, et al. AD-RI: Closed-loop reinforcement learning for end-to-end autonomous driving with impartial world models. arXiv preprint arXiv:2511.2032. 2025.
16

---

Published as a conference paper at ICLR 2026
Kaiyan Zhang, Yuxin Zuo, Bingxiang He, Youbang Sun, Runze Liu, Che Jiang, Yuchen Fan, Kai Tian, Guoli Jia, Pengfei Li. et al. A survey of reinforcement learning for large reasoning models. arXiv preprint wwXiv:2500008822 2025a.
Kejia Zhang, Keda Tao, Jiasheng Tang, and Huan Wang. Poison cure: Visual noise for mitigating object hallucinations in LVMs. In Advances in Neural Information Processing Systems, 2025b.
Xinglang Zhang, Yunyao Zhang, ZeLiang Chen, Junqing Yu, Wei Yang, and Zikai Song. Log- ical phase transitions: Understanding collapse in LLM logical reasoning. arXiv preprint arXiv:2601.02902, 2026.
Yi-Fan Zhang, Xingyu Lu, Shukang Yin, Chaoyou Fu, Wei Chen, Xiao Hu, Bin Wen, Kaiyu Jiang, Changyi Liu, Tianke Zhang, et al. Thyme: Think beyond images. arXiv preprint arXiv:2508. 1630, 2025c.
Yunyao Zhang, Xinglang Zhang. Junxi Sheng, Wenbing Li, Junqing Yu, Wei Yang, and Zikai Song. From ambiguity to verdict: A semiotic-grounder multi-perspectivg agent for LLM logical reasoning. arXiv preprint arXiv:2509.24765, 2025d.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00993, 2023.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhnn Yenanhn, and Zheyan Luo. LlamaFactory: Unified efficient fine-tuning of 100+ language models. In Annual Meeting of the Association for Computational Linguistics, pp. 400-410, 2024.
Junhan Zhu, Hesong Wang, Mingluo Su, Zefang Wang, and Huan Wang. OBS-Diff: Accurate pruning for diffusion models in one-shot. arXiv preprint 2rXiv:2510.0651,,0022 2025.
17

---

Published as a conference paper at ICLR 2026
APPENDIX
In Appendix A, we provide a statistical overview of REASONMAP- question templates, and fine- grained annotation. Appendix B outlines the computation pipeline of the detail reward. Appendix C describes evaluation settings, including difficulty-aware weighting, benchmark datasets, and result sources in Table 2. In Appendix D, provide more discuss future work in Appendix E. Finally, Appendix F presents the statement on LLM usage.
support claims in Section 5.3. We
A Dataset Construction Details
18
A.1 Statistical Overview
18
A.2 Question Template
18
A.3 Fine-Grained Annotation
19
B REWARDMAP Details
20
B.1 Computation Pipeline of Detail Reward
20
Evaluation Details
20
C.I Details of Difficulty-Awar Weighting
20
C.2 Details of Evaluation Datasets
20
C,3 Result Source Summary
Supplementary Results
D.I Comparison Cases
21
Future Work
22
Large Language Model Usage Statement
22
A DATASET CONSTRUCTION DETAILS
A.1 STATISTICAL OVERVIEW
REASONMAP-PLIC mirrors REASONMAP in image composition, comprising high-resolution transit maps from 30 cities. It contains 018 questions across five categories (e.g., Local Counting 1 1, 050; Local Counting - 970; Global Counting 30; True or False 1 1-,, 039; True or False 2 929), with difficulty distribution of 1, 259 easy, 1, 342 middle, and 1, 417 hard. To preserve difficulty balance and diversity, we adopt the same split as REASONMAP, using questions from 11 cities for the test (1, 448) and the remainder for the training set (2, 570).
A.2 QUESTION TEMPLATE
We present the question templates of REASONMAP- PLUS follows.
Local Counting
Please solve the multiple choice problem and put your answer (one of ABCD) in one boxed". According to the subway map, how many intermediate stops are there between stop 1 and stop 2 (except for this stops)? A) B) x C) D)
18

---

Published as a conference paper at ICLR 2026
Local Counting
Please solve the problem and put your answer one boxed". According to the subway map, how many lines through stop 1?
Global Counting
Please solve the problem and put your answer in one boxed", According to the subway map, how many subway (metro) lines are there in total?
True or False
Please solve the problem and put your answer (only answer yes no) in one boxed",, According to the subway map, true that stop the same line as stop 2?
True or False
Please solve the problem and put your answer (only answer yes or no) in one boxed". According to the subway map, is true that stop on the line x?
A.3 FINE-GRAINED ANNOTATION
We provide more details of the process-level annotation in this section. The construction involves two steps. First, we annotate the route-stop structure of each transit map, including the ordered stops for every line, interchange stations, and branching points. These annotations, listed in the Meta Data, are recorded in a JSON file. An excerpt for Singapore is shown below:
Meta Data
"Circle Line": ["HarbourFront (Transfer)", "Telok Blangah", "Labrador Park", ...) "Downtown Line": [..].
Based on the Meta Data, we generate process-level annotations with task-specific Python scripts. For example, in the Local Counting category of REASONMAP-PLUC where the question asks for the number of intermediate stops between two stations, the script identifies the two queried stops. extracts all intermediate stops between them, computes the count the final answer, and records the intermediate stops as the process-level annotation. Other task types, such as True/False or shortest-pan path queries, follow similar procedure.
19

---

Published as a conference paper at ICLR 2026
Example (Local Counting
"country": "singapore",
"city": "singapore".
"station_1": "Clarke Quay",
"station_2" "Farrer Park".
"figure": /maap//inngaprr//ssgapore
"question": "According the map, how many intermediate stops are there between Clarke Quay and Farrer Park (excluding the endpoints)?", "answer": "2",
"fine-grained answer": ("Dhoby Ghaut", "Little India"),
"type": "counting__"
"difficulty.cii city"; "hard",
"city Llnelllieeeeeeee www.lieeccoi... count": "6",
"city ttransfer_couutt:: "53".
"json": " /ssafoon/singgprrreigggppeee//ss
Example 2 (True False 1)
"country": "singapore", "city": "singapore",
"station_ "Somerset",
"station_2" 22" "Bras Basah",
"figure": /mmpa///nnaapnn//sigggorrrppppp
"question": "According to the map, is it true that Somerset is on the same line as Bras Basah?",
"answer"" "No",
"fine-grained answer":
"Somerset": "North South Line", "Bras Basah": "Circle Line",
"type" "torf_1", "difficulty-cii :itt":: "hard",
"city www.e "6", "city transfer count": "53".
"json": " "/statons/iinggporsssingpppepee
B REWARDMAP DETAILS
B. COMPUTATION PIPELINE OF DETAIL REWARD
We present the complete computation pipeline of Detail Reward in Algorithm
C EVALUATION DETAILS
C.1 DETAILS OF DIFFICULTY-AWAN WEIGHTING
For REASONMAP, we follow the weighting scheme described the original paper (see Appendix B.3 therein). For REASONMAP-PLE weights assigned solely based on map difficulty, with values of 1.0, 1.5, and 2.0 for easy, medium, and hard maps, respectively.
C.2 DETAILS OF EVALUATION DATASETS
We provide brief description the six evaluation benchmarks used in our paper:
SEED-Bench-2-PP (Map) (Li et al., 2024) describes three categories (Charts, Maps, Webs) with human-verified multiple-choice items, from which we use the Map slice.
20

---

Published as a conference paper at ICLR 2026
Algorithm 1: Detailed Reward for Planning Questions in REASONMAP
Initialize score <- O;
if route_data is empty format is wrong then
return score;
departure stop of first segment = stop_1 or arrival stop of last segment = stop_2 then
score score +2;
foreach segment s; in predicted route do
current..ransser...e............
> queetion.transfer.ccou then
score score -5;
current_trans imme = 0 and is.correct(route name) then score score +4;
if departure stop stations and arrival stop stations then segment 8; is not the last then
if arrival stop departure stop of next segment then score + score +1;
score min(score, 10);
return score;
2. SpatialEval (Wang al., 2024a) targets spatial intelligence across relationships, position, counting, and navigation.
3. ""Benchh (Wu & Xie, 2024) evaluates fine-grained attribute recognition and spatial relation- ships on high-resolution images.
4. HRBench (Wang et al., 2025f) evaluates MILMs on 4K/8K high-resolution images and introduces a training- free enhancement baseline.
5. ChartQA (Masry et al., 2022) benchmarks QA over charts with visual and logical reasoning.
6. MMStar (Chen et al., 2024a) offers 1,500 human-curated, vision-indissensable samples covering 6 core capabilities and 18 axes.
C.3 RESULT SOURCE SUMMARY
We present sources of results in Table 2: f. $,*$ correspond to the Seed1.5-VI technical re- port (Guo et al., 2025b), the Qwen2.5-Vi VL technical report (Bai et al., 2025b), the Intern technical report (Wang et al., 2025e), the Kimi-Vi VL technical report Team et al. (2025), and the official Hugging Face repository of Qwvan2.5-V1-32BB-nsttue
respectively.
D SUPPLEMENTARY RESULTS
D.1 COMPARISON CASES
We provide more comparison cases to support deeper analysis Section 5.3. In the following case, the prediction from Qwen2.5-VL-7B-- Instruct exhibits a visual confusion error at <wrong/> and hallucination <wrong2>, misidentifying three intermediate stops but selecting option A (6). In contrast, the RewardMap-trained model correctly identifies a single intermediate stop and selects the correct option C.
Question: How many intermediate stops are there between A Station and B Station?
(<budapest. .png> + Format Requirement (A: 6, 8, 1, D:
thttps://uugiinngfacccooo
Instruct.
21

---

Published as a conference paper at ICLR 2026
Answer (Qwen255VV--7--nsttuu Wrong).
To determine the number of intermediate stops between A Station and B Station, we need to look at the yellow line on the map, which represents Metro Line 1. 1. Identify the stations on the yellow line between A and B. 2. Count the number of stations, excluding A and B.
From the map: A
Széchenyi Thermal Bath
Bajza Street -B
There are 3 intermediate stops between A and B. <wrongl> Therefore, the correct answer <wrung2>
Answer (Qven2.5-VL-7B-Instru) + RewardMap, Correct).
To solve this problem, we need to count the intermediate stops between A Station and Station, excluding these two stops. Starting from A: - The next stop is Széchenyi Thermal Bath. Thus, the only intermediate stop between A and B is Széchenyi Thermal Bath. Therefore, the correct answer
E
FUTURE WORK
Our framework is not limited to transit maps and can be extended to other structured visual domains such as charts and diagrams. Our results support this direction: CharlQA (Masry et al., 2022) improves accuracy from 86.12% 87.24%, Charxiv (Wang al., 2024b) from 66.41% 68.90%, and InfoVQA (Mathew et al., 2022) from 82.22% + 82.71% These gains suggest that the detail rewards and multi-stage RL scheme generalize to structurally similar tasks. Future work will explore domain-agnostic rewards and broader applications across diverse structured visual reasoning settings.
Efficient methods can further improve the fine-tuned model by framework (Feng al., 2024; Bai et al., 2025a; Zhu et al., 2025; Shao et al., 2025a;b; Tao et al., 2025a;b;c). The idea detail reward can be extended other fields that require reasoning capability (Zhang et al., 2025d; 2026; Wang et al., 2025d;e; Ai et al., 2025; Wu et al., 2025b; Li et al., 2025e; Wang et al., 2023; Jin et al., 2025; Xiao et al., 2025; Li al., 2025a; Feng al., 2026; Wang et al., 2026). Leveraging reinforcement learning for safety alignment (Zhang et al., 2025b; Li & Wang, 2026; Li et al., 2025b) remains promising direction for further exploration.
F LARGE LANGUAGE MODEL USAGE STATEMENT
Large Language Models (LLMs) were used only for surface-level editing of the manuscript (e.g.. polishing grammar and style, rephrasing for clarity, and making minor ETEX adjustments). They were not involved in generating ideas, methods, algorithms, code, experiments, figures, tables, or citations. All research design, implementation, data processing, and analysis were performed by the authors. L.M use was limited to de-identified text snippets, with no proprietary data or unpublished results shared, and all outputs were manually reviewed and revised. This limited assistance does not affect reproducibility, as every reported result is fully reproducible from the released code and configurations.
22


Reviews:
[{"review_id": "fZJDJAEpjb", "reviewer": "Reviewer_3FQv", "summary": "This paper proposes REWARDMAP, a multi-stage reinforcement learning (RL) framework for improving fine-grained visual reasoning in multimodal large language models (MLLMs). It builds on the REASONMAP benchmark and introduces REASONMAP-PLUS, a companion dataset providing dense supervision via VQA-style tasks organized by difficulty. The approach combines a difficulty-aware reward design with curriculum-based RL under GRPO to mitigate sparse reward issues. Experiments show consistent but small gains (around 3.5%) on REASONMAP, REASONMAP-PLUS, and several external benchmarks. Ablations suggest that reward shaping and staged training both contribute to improvements.", "questions": "1. How sensitive are results to the hyperparameters \\( \\alpha \\) and difficulty weights (\\( \\gamma_e, \\gamma_m, \\gamma_h \\))?  \n2. Could the proposed curriculum RL pipeline be applied to other structured domains (e.g., diagrams, charts)?  \n3. How do you ensure no data leakage between REASONMAP-PLUS easy/hard tasks and REASONMAP test sets?  \n4. Why were RLHF or DPO-based baselines not compared for reward densification?  \n5. Can you provide error-type or reasoning-chain analyses to explain where REWARDMAP helps most?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Addresses a critical problem of sparse rewards in RL for multimodal reasoning with a practical and elegant solution.  \n- Well-engineered multi-stage RL approach combining dense and sparse supervision effectively.  \n- Comprehensive evaluation with ablations that isolate reward design and curriculum effects.  \n- Clear writing, reproducible setup, and solid empirical results across multiple benchmarks.", "weaknesses": "- Conceptual novelty is limited. The method is primarily a well-engineered combination of known ideas: reward shaping, difficulty weighting, and curriculum learning under GRPO. There is no new RL algorithm or theoretical contribution.  \n- Reported performance gains are numerically small relative to the additional dataset, compute, and engineering effort. It is unclear whether such modest improvements are statistically significant.  \n- The reward weighting scheme (α, γ_e/m/h, β_0/1) is ad-hoc and lacks sensitivity analysis. Without this, claims about “difficulty awareness” remain anecdotal.  \n- Evaluation is restricted to Qwen2.5-VL, raising concerns about generalization and overfitting to a single model architecture.  \n- REASONMAP-PLUS is narrowly scoped to transit maps; claims of general fine-grained reasoning ability are not convincingly supported.  \n- The paper lacks deeper insight or analysis into why multi-stage RL works, there is no visualization or error decomposition that would make the mechanism interpretable.  \n- The average improvement on external benchmarks (+3.47%) is within noise for modern MLLMs; effect size is weak.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LgUoq2JUjO", "reviewer": "Reviewer_GZ1U", "summary": "This paper proposes RewardMap, a multi-stage RL framework that mitigates sparse-reward problems in fine-grained visual reasoning for multimodal LLMs. Building on ReasonMap, the authors introduce ReasonMap-Plus, a dense and difficulty-graded dataset for visual question answering on transit maps. RewardMap integrates (1) a difficulty-aware reward function combining format, correctness, and detail rewards, and (2) a multi-stage GRPO training curriculum progressing from perception to reasoning tasks. Experiments on RewardMap, ReasonMap-Plus, and six external benchmarks show consistent gains and reduced visual confusion, demonstrating improved visual understanding and reasoning performance.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. New dataset and training paradigm: the paper introduces ReasonMap-Plus, which is a novel dataset that helps conducting RL using dense reward signals.\n2. Models being trained on the dataset seem to perform well on the proposed benchmarks", "weaknesses": "1. Lack of baseline VLMs on ReasonMap and ReasonMap-Plus. What would more recent and powerful VLMs like GPT-4o and GPT-5 perform on these benchmarks?\n2. Would be better to train other VLMs other than Qwen2.5-VL, such as InternVL models to see if the training and data could actually be generalized.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pPly4IedeC", "reviewer": "Reviewer_owjV", "summary": "This paper studys the problem of sparse reward in standard reinforcement learning for spatial reasoning in structured and information-rich setting.  To address the problem, the authors construct REASONMAP-PLUS, a dataset with dense reward signals and propose a multi-stage RL framework for improving the corresponding capabilities in current MLLM. Experiment results demonstrate its effectiveness across various reasoning tasks and general tasks.", "questions": "1. In the acquisition of the detailed reward, how is the correctness of each sub-part determined? Is an external LLM employed to extract the relevant content, or is the response constrained to a fixed format?\n2. Please refer to the weaknesses highlighted above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The research topic of this work is interesting, as sparse reward is indeed a significant challenge in current vision tasks, particularly in structured visual input tasks as highlighted in the paper.\n2. This work introduces the REASONMAP-PLUS dataset, which could be beneficial for future related research.\n3. The experimental results seem promising, demonstrating the effectiveness of the proposed method on transit map planning tasks and showing its transferability to other tasks.", "weaknesses": "1. The method parts of the paper presents certain ambiguities. For instance, how does the Difficulty-Aware Weighting mechanism operate? Are weights assigned to each sample within the groups of GRPO, or to each sample across the entire training dataset? Furthermore, what does Multi-stage RL training entail—does it refer solely to the sequencing of data based on difficulty, or does it also encompass different training phases?\n2. The novelty of the approach appears limited, as the core methodology remains largely aligned with that of GRPO. While the design of the detailed reward function is interesting, it only considers specific components of the response (e.g., destination stops, route names), thereby constraining its overall flexibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7femZ9v9tw", "reviewer": "Reviewer_ekjh", "summary": "This paper tackles the problem of improving reasoning of multi-modal large language models. They use the REASONMAP dataset and show that directly performing GRPO on the dataset is not very effective for multi-modal reasoning because of high difficulty of questions and sparse rewards. To alleviate this, they construct REASONMAP-PLUS, an augmented dataset which easier VQA style queries with denser reward. They also show improvements in training MLLMs using GRPO by adding a curriculum (easy->hard) questions and by adding partial rewards to alleviate the problem of sparse rewards. They name this new recipe as REWARDMAP which shows performance gains on the test set of REASONMAP and gains of around 3.5% on other reasoning benchmarks as well.", "questions": "Please refer to the weakness section for questions.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "1. The paper shows gains using REWARDMAP on both in-distribution and out of distribution test sets, suggesting that the curriculum learning + denser reward works well for multi-modal reasoning training \n2. They construct an augmented dataset REASONMAP-PLUS with around 4k VQA pairs across different types like local counting, global counting, and true/false\n3. The difficulty and dense rewards are assigned programatically without the need for any reward models or human labelling which makes the method cheap and efficient", "weaknesses": "1. Incremental novelty: both curriculum learning (https://arxiv.org/abs/2506.06632, https://arxiv.org/abs/2501.12599, https://arxiv.org/abs/2502.14768) and denser rewards to improve reasoning are approaches which have been tried in several existing works, while this paper shows a combination which works well, the approach as a whole feels incremental. \n2. Reward shaping not generalizable: The detail reward (credit for route names, transfers) is tightly linked to transit-map structure; it’s unclear how the same scheme would extend to charts, floor plans, or natural images without such information. Likewise, the difficulty weights rely on a dataset-specific notion of map/question difficulty (e.g., transfer counts). It would be good to show a more generalizable approach to reward design. \n3. Reward shaping not ablated well: The reward seems ad-hoc, there are no clear ablations for how the value of alpha was chosen, nor are there ablations on how W_diffculty is decided, or why it is multiplied to the other rewards. It would be good to provide some additional experimental results as to how this particular reward structure was chosen. \n4. Limited OOD improvements: While the authors show performance on other benchmarks like MM-Star as well, the performance on them shows negligible gains. It might be more informative if there is a task wise breakdown of the performance which might help analyze the exact tasks where performance improves vs where it stagnates.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)