VisionReasoner: Unified Reasoning-Integrated Visual Perception via Reinforcement Learning
Authors: Yuqi Liu
Tianyuan Qu
Zhisheng Zhong
Bohao PENG
Shu Liu
Bei Yu
Jiaya Jia
Venue: ICLR 2026 Poster (2026)
Keywords: Reasoning Segmentation; Reinforcement Learning; Multi-model Large Language Models; Visual Perception
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Large vision-language models exhibit inherent capabilities to handle diverse visual perception tasks. In this paper, we introduce VisionReasoner, a unified framework capable of reasoning and solving multiple visual perception tasks within a shared model. Specifically, by designing a unified reward mechanism and multi-object cognitive learning strategies, VisionReasoner enhances its reasoning capabilities to analyze visual inputs, and addresses diverse perception tasks within a unified model. VisionReasoner generates a structured reasoning process before delivering the desired outputs responding to user queries. Human evaluation reveals the reasoning process of VisionReasoner is faithful and reliable even without annotated reasoning train data. To rigorously assess unified visual perception capabilities, we evaluate VisionReasoner on ten diverse tasks spanning three critical domains: detection, segmentation, and counting. Experimental results show that VisionReasoner achieves superior performance as a unified model, outperforming the baseline Qwen2.5VL by relative margins of 29.1% on COCO (detection), 22.1% on ReasonSeg (segmentation), and 13.2% on CountBench (counting).

Published as a conference paper at ICLR 2026
VISIONREASONER: UNIFIED REASONING-INTEGRATEH VISUAL PERCEPTION VIA REINFORCEMENT LEARNING
Yuqi Liu'> Tianyuan Qu' Zhisheng Zhong' Bohao Peng'
Shu Liu Bei Yu' Jiaya Jia
CUHK! SmartMore² HKUST3 Equal contribution
Corresponding author
https:// attps://github.. Lab--a-------eaaaaae http:///22-a-.eeeeeerttttoteeeee
ABSTRACT
Large vision-language models exhibit inherent capabilities to handle diverse visual perception tasks. In this paper, we introduce VisionReasoner, a unified framework capable of reasoning and solving multiple visual perception tasks within a shared model. Specifically, by designing a unified reward mechanism and multi-object cognitive learning strategies, VesionReasoner enhances its reasoning capabiliites to analyze visual inputs, and addresses diverse perception tasks within a unified model. VisionReasoner generates structured reasoning process before delivering the desired outputs responding to user queries. Human evaluation reveals the reasoning process of VisionReasoner is faithful and reliable without annotated reasoning train data. To rigorously assess unified visual perception capabilities, we evaluate Vision on diverse tasks spanning three critical domains: detection, segmentation, and counting. Experimental results show that Vision- Reasoner achieves superior performance as unified model, outperforming the baseline Qwen2.5VL by relative margins of 19.1% on COCO (detection). 22.1% on ReasonSeg (segmentation), and 13.2% on CountBench (counting).
INTRODUCTION
Recent advances in large Dision-language models (LVLMs) (Bai et al., 2025; Wang et al., 2024; Google, 2025; OpenAl, 2025) have demonstrated remarkable capabilities in visual conversations. As the field progresses, researchers are increasingly applying LVLMs to wider range of visual perception tasks, such as visual grounding (Peng et al., 2024) and reasoning segmentation (Lai et al., 2024; Liu et al., 2025a) often incorporating sask-specific modules techniques.
Through an analysis diverse visual perception tasks, we observe that many can be categorized into three fundamental types: detection (e.g., object detection (Lin et al., 2014), visual grounding (Yu et al., 2016)), segmentation (e.g.. referring expression segmentation (Yu et al., 2016), reasoning segmentation (Lai et al., 2024)), and counting (e.g., object counting (Paiss et al., 2023)). Notably, our analysis reveals that these three task types share a common structure as multi-object cognition problems, suggesting that they can be addressed through unified framework.
Moreover, recent studies have explored the integration of reinforcement learning (RL) into LVLMS (Team, 2025; Liu et al., 2025b;a; Zheng et al., 2025). Works such VisualRFT (Liu al., 2025b) and Seg-Zero (Liu et al., 2025a) demonstrate that RL can enhance reasoning in visual perception tasks. However, these approaches often employ RL in a task-specific manner, training with different data for different tasks, which may limit their scalability and generalizability.
Building on these insights, we propose VisionReasoner, a unified framework that addresses diverse visual perception tasks through a shared architecture. The framework's 's core capabilities, which include advanced reasoning and multi-object cognition, are enabled through RL and unified reward mechanism. Format rewards, including thinking rewards that promote structured reasoning and non-repeat rewards that prevent redundant reasoning patterns. Accuracy rewards, comprising multi- object IoU rewards and LI rewards for precise localization, strengthen multi-object cognition. Unlike previous approaches like Kosmos (Peng et al., 2024) that use cross-entropp loss, our RL framework requires optimal prediction- ground-truth matching. We address this challenge by implementing an efficient matching pipeline combining the batch computing and the Hungarian algorithm, significantly improving computational efficiency while maintaining matching accuracy.

---

Published as a conference paper at ICLR 2026
Detection
In image, Statue Liberty prominent festure the left side, and it well-known symbol of freedom and on. recognized as the Statue of Liberty. The Statue Liberty
Which building the landmark and
attraction and often featured photographs of New York City. </think>
mamorable to take
photo with?
Segmentation
<thinle-The task identifies objects that can provide shade ... typicully tents
In image, appear to be tents, as they have dome like shope and are ground. These structures likely designed provide shade for people who gathered pround them. <tthink>>
Segmeent objects that
can help people shade from
Counting
many objects can help fishing.... bosts are typleally used for flshing, and items like nets, rode, and other fishing gear are often stored on the </think
determine
many objects can help with fishing?
bosts,
Totel numberis
ReasonPerceiver addresses diverse visual tasks.
COOO
Qwen2VL-77 Swen2.5VL-77 ReasonPerceiven
CD+00
RelCOCO
Re/COCOg
CountBensch
ReasonSeg
PlumoCount
MMMU,MI
RealWorkd
MMMU
MMMU
ChartQA
TextVQA
DocVQA
(b) Performance on visual tasks.
(c) Performance on VQA tasks.
Figure (a) VisionReasoner addresses diverse tasks within a unified framework. generates reasoning process and outputs the expected result corresponding to each query. (b) VisionReasoner significantly outperforms Qwen2.5VL. (c) VisionReasoner retains strong VQA capabilities.
To comprehensively evaluate model performance, we conduct extensive experiments with VisionRea- soner across 10 diverse tasks spanning three fundamental types: detection, segmentation, and counting. Remarkably, our VisionReasoner- 7B model achieves strong performance despite being trained on only 7k samples, demonstrating both robust test-time reasoning capabilities and effective multi-task generalization, as shown in Figure 1 (a)-(b). Experimental results show significant improvements over baseline models, with relative gains of 29.1 1 % on COCO-val (detection), 22. % on ReasonSeg- (segmentation), and 13.2% on CountBench-test (counting), validating the effectiveness our unified approach. Additionally, VisionReasoner exhibits visual question answering capabilities comparable to -fate-of-the-art models, as shown in Figure (c). Human evaluation also indicates VisionReasoner generates faithful and reliable reasoning process even without training annotated reasoning data.
Our contributions are summarized as follows:
We propose VisionReasoner, a unified framework for visual perception tasks. Through care- fully crafted rewards and training strategy. VisionReasoner has strong multi-task capability, addressing diverse visual perception tasks within shared model.
Experimental results show that VisionReasoner achieves superior performance across ten diverse visual perception tasks within single unified framework, outperforming baseline models by a significant margin.
Through extensive ablation studies, validate the effectiveness of our design and offer critical insights into the application of RL in LVLMs.
2 RELATED WORKS
2.1 LARGE VISION-LANGUAGE MODELS
Following LLaVA's (Liu et al., 2023c) pioneering work on visual instruction tuning for large vision- language models, subsequent studies (Wang et al., 2024; Meta, 2024; OpenAI, 2025; Bai et al., 2025; Li et al., 2024b; Zhong et al., 2024) have adopted this paradigm for vision-lant languael conversation. Beyond visual conversation tasks, LVL.Ms have been extended to diverse vision applications, including
2

---

Published as a conference paper at ICLR 2026
visual grounding (Peng et al., 2024) and reasoning segmentation (Lai et al., 2024). Notely, the recent GPT-4.1 (OpenAI, 2025) demonstrates state-of-the-arr performance in multi-modal information processing and visual reasoning. Although these models are evaluated specific tasks, their performance has not been systematically evaluated under unified visual perception framework.
2.2 REINFORCEMEN LEARNING IN LARGE MODELS
In the field of large language model (LLMs), various reinforcement learning (RL) algorithms are used to enhance model performance, such as reinforement learning from human feedback (RIHF) (Ouyang et al.,, 2022), direct preference optimization (DPO) (Rafailov et al., 2023) and proximal policy optimization (PPO) (Schulman et al., 2017). The recent DeepSeek RI (Guo et al., 2025), trained using Group Relative Policy Optimization (GRPO) (Shao et al., 2024), demonstrates remarkable test-time scaling capabilities, significantly improving reasoning ability and overall performance. Building on these advances, researchers try to apply these RL techniques to LVLMs. Notable efforts include Visual-RFT (Liu et al., 2025b), EasyRI (Zheng et al., 2025) and Seg-Zero(Liu al., 2025a), all of which exhibit strong reasoning capabilities and achieve impressive performance.
3 METHOD
To develop a unified visual perception model capable of solving diverse vision tasks, we identify and analyze the representative visual perception tasks, then reformulate their inputs and outputs into a set of three fundamental task categories (Section 3.2). Next, we detail the architecture of our VisionReasoner model (Section 3.3). Additionally, present the unified reward mechanism employed for training our model (Section 3.4). Finally, we elaborate on our training strategy of multi-object cognition (Section 3.5).
3.1 PRELIMINARY
Traditional Vision Methods vs. LVLMs. Although tradi- tional vision models (Cheng et al., 2024; Ren et al., 2024a) achieve strong performance on standard visual perception benchmarks (Lin et al., 2014), they are inherently limited processing simple categorical queries and struggle with complex, compositional. or reasoning massoing-inttesive intensive instruc- tions. contrast, LVLMs can interpret and respond nuanced, open-ended queries. As illustrated in Figure 2, VisionReasoner successfully localizes and identifies target objects that traditional approaches fail to detect, highlight- ing the necessity to integrate LVLMs into visual perception pipelines where reasoning essential.
LVLMs
Query: What should avoid if am allergic to seafood?
raw image
VisionReasoner Correct
Traditional Methods
Group Relative Policy Optimization (GRPO). The GRPO is on-policy reinforcement learning algorithm. For each input x, the old policy model Tg... from previous step generate group of rollouts (o.),-1 Then reward functions are used to calculate rewards for each o,, getting (r:){ We design unified reward mechanism and the relative advantage is calculated as:
DINO-X Fail
Yolo World Fail
Figure 2: VisionReasoner correctly lo- calizes objects from a complex instruc- tion, whereas both commercial DINO-X and open-source YOLO-Wore fail.
Pi mean( (ri, (2..........l) std(fri, rc....
A
The GRPO maximizes the following objective and optimizes model me:
(CRrO( = E Train
Train Butch. (o N (O(x)
G
To(o, x)) Ai, clip
(o x) + As
(2)
min
33Dx.. (TT || Tat)
Te (or x)
x))
1=1l

---

Published as a conference paper at ICLR 2026
Visual Question
VisionReasoner
GRPO
Multi-object Matching
How many airplanes?
Reasoning Module
0,
0g:
Detection
0g:
extract
Segment the donuts.
Rewards
Counting
sthink
Locate the dog.
Segmentation Module
<ansmer>
"bbox
"point"
Segmentation
ground truth
Format
Accuracy
(a) Pipeline of VisionReasoner
(b) Training of VisionReasoner
Figure 3: Illustration of VisionReasooner. (a) For given image I and text instruction T. model generates the expected output corresponding to the instruction. (b) For each observation Os, we calculate the rewards (Section 3.4) and attain the optimal match of multi-objects (Section 3.5)
3.2 TASK REFORMULATION AND CATEGORIZATION
Our analysis vision perception tasks (Yu et al., 2016; Lin et al., 2014; Lai et al., 2024; Deitke et al., 2024) reveals that many of them can be categorized into three fundamental task types. Here take ten visual perception tasks for illustration. Further details are provided in the Appendix J.
Detection. Given an image I and text query T, the detection task type aims generate set of bounding boxes {B.] that localize objects of interest. This type requires multi-object cognition ability. This category includes tasks such as Visual Grounding (Yu et al., 2016; Kazemzadeh et al., 2014) and Object Detection (Lin et al., 2014).
Segmentation. Given an image and
query the segmentation task type aims to generate set of binary segmentation masks [M, | that identify the regions of interest. We address this type by detect-then--ee -egmentt paradigm. This category includes tasks such as Referring Expression Segmentation (Kazemzadeh et al., 2014; Yu et al., 2016) and Reasoning Segmentation (Lai et al., 2024; Yang et al., 2023).
Counting, Given an image I and text query T, the counting task type aims to estimate the number of target objects specified by the query. We address this type by detect-then-couml paradigm. This category includes tasks such as Object Counting (Deitke et al., 2024; Paiss et al., 2023).
3.3 VISIONREASONER MODEL
Our VisionReasoner F model incorporates reasoning module, which processing image and locates targeted objects, and a segmentation module that produces segmentation masks needed. The whole architecture is shown in Figure 3 (a). The key to FF lies in its multi-object cognition capabilities, which critical to enables VisionReasoner to address three fundamental task types: detection. segmentation, counting. Specifically, given image generates an interpretable reasoning process, and then produces the bounding boxes { and central points (P, J of targeted objects corresponding to T. Then JB,,, and (P,) { serve as bridge connect the segmentation module, producing binary masks { process can be formulated as:
text query T, the VisionReasoner F
N needed. This
(B,, M)) F(I,T).
During inference, the user provide the input image and text prompt T, and define specified task type C e (detection, segmentation, counting]. The system then produces the expected outputs as follows:
(B
if C is detection,
Output
[M,).
if C is segmentation,
N,
if C is counting.
In this way, VisionReasoner process diverse perception tasks in unified manner within shared framework. Moreover, our framework can be easily extended to other visual perception tasks illustrated in Appendix H.
4

---

Published as a conference paper at ICLR 2026
3.4 UNIFIED REWARD MECHANISM
As illustrated in Section 3.1, the core in group relative RL is the design of rewards. We design unified reward mechanism for visual perception tasks, including format rewards and accuracy rewards. We use target object bboxes and center points to calculate the rewards rather than binary masks. These rewards jointly guide the optimization process by reinforcing both structural correctness and multi-object recognition performance. The model is capable of addressing diverse visual perception tasks after training within this unified reward mechanism. The total reward is the sum of all rewards.
Thinking Format Reward. This reward is 1.0 if the model output thinking process between <think> and </think>tags, and output the final answer between the <answer> and </answer>tags.
Answer Format Reward. We use bounding boxes k and points {Pr).- the answer as it has better training efficiency. So this reward restrict the model output answer in [ [['bbo__2 2d' [21, y1, 22, V2]; point 2d': [2r1, vi]),-.] ...]. The reward is 1.0 if correct else 0.0.
Non Repeat Format Reward. We split the reasoning process into sentences to detect repeated pattern. A reward of 1.0 is assigned for those with unique non-repetitive thinking processes.
Bboxes IoU Reward. Given a set of NN ground-trute bounding boxes and predicted bounding boxes, this reward computes their optimal one-to-one matched Intersection-over-Uniiin (IoU) scores.
For each IoU exceeding 0.5, we increment the reward by
max (N, K}
Bhoxes L1 Reward. Given set of N ground-truth bounding boxes and K predicted bounding boxes, this reward computes their one-to-one matched LI distance. For each L.1 distance below the
threshold of 10 pixel, we increment the reward by max (N, KY
Points L1 Reward. Given a of N ground-truth points and predicted points, this reward computes their one-to-one matched LI distance. For each LI distance below the threshold of 30 pixel,
we increment the reward by [N, KY
3.5 MULTI-OBJECT COGNITION IN LVLMS
Unlike the auto-regressive training paradigm (Peng et al., 2024; Bai et al., 2025) in supervised fine- tuning, RI. framework requires optimal satiiciionnt-ground- matching for reward calculation. To address this, we derive the necessary data and implement an effective matching strategy.
Multi-object Data Preparation. We derive bboxes and points directly from the original mask annotations in existing segmentation datasets (e.g., ReiCoCOO (Yu et al., 2016), LISA++(Yang al., 2023)). Specif- ically, for a given binary mask of an object, we determine its bounding box by extracting the leftmost, topmost, rightmost, and bottommost pixel coor- dinates. Additionally, we compute the center point coordinates of the mask. We process multiple objects per image by: (i) using one central point (ii) join- ing all textual descriptions with the conjunction 'and', and (iii) concate- nating all associated bounding boxes and center points into list per image.
Algorithm 1: Multi-object Matching
Input: pred bhoxes breed E R Rxx,, pred points Pored E R x2,
GT bboxes byt E R Nx4, GT points Pyt E R Nx2
Function AccuracyRewardl bpeed Ppred, bgr, Pal:
+- 0; T.max max K, N);
loU < BatchloU bạ) E RKN
BL1 +- BatchBoxL (byod, by) E RKN
PL1 + BatchPointL Distancc
(pirsancce(peed,,,, (pred, Pa) R
RoU +- [IU IoU threshold]
RELI < [BL1 < Box LI threshold]
RALI [PL1 Point L1 threshold]
C + (3 - (Ria: + RBL. + RR.)) E K
(r,c) Eurgarian(C))
total 4- 3/r Σ. Cr..e
+ total/ Lmax: return
Output: Accuracy reward r
Multi-object Matching, Our framework addresses multi-object matching through batch computation and the Hungarian algorithm, which optimally solves the many-to-many matching problem for bounding boxes IoU rewards, bounding boxes L1 rewards, and points I.1 rewards. As shown in
5

---

Published as a conference paper at ICLR 2026
Table Performance comparison on detection tasks.
Detection
Method
COCO
RefCOCO
RefCOCO+
ResCOCOg
Avg.
val
val
testA
val
testA
val
test
Task-specific Models
VGTR
79.0
82.3
63.9
70.1
65.7
67.2
TransVG
81.0
82.7
64.8
70.7
68.7
67.7
RefTR
85.7
88.7
77.6
82.3
80.0
MDETR
86.8
89.6
79.5
84.1
81.6
80.9
OWL-ViT
30.9
YOLO-WorId-S
37.6
GLIP-T
46.6
50.4
54.3
49.5
52.8
66.1
66.9
55.2
G-DINO-T
48.4
74.0
74.9
66.8
69.9
72.1
68.2
DQ-DETR
50.2
88.6
91.0
81.7
86.2
82.8
83.4
80.6
Large Vision- Vision-language Models
Shikra-7B
87.0
90.6
81.6
87.4
82.3
82.2
Intern VL2-88LL-8B
87.1
91.1
79.8
87.9
82.7
82.7
Qwen2-VL-7h VL-BB
28.3
80.8
83.9
72.5
76.5
77.3
78.2
71.1
Qwen2.5-VLL-7B
29.2
88.8
91.7
82.3
88.2
85.7
78.6
VisionReasoner-71B
37.7
88.6
90.6
83.6
87.9
86.1
87.5
80.3
Figure 3 (b), for each observation Oj. which contains list of bboxes { [Bpred JK and points
JKK prod: [1=1> we calculate its reward scores with the ground-truth bboxes (BGT,) and points 1.V {PGT:)-= by implementing batch computation. We then calculate the optimal one-to-one matching with using Hungarian algorithm. The pseudocode of multi-object matching is shown in Algorithm 1. These design guarantees optimal assignment between predictions and ground truth annotations while achieving high computational efficiency.
4
EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
use ten benchmarks to evaluate model performance across general vision perception tasks, including three fundamental task types: detection, segmentation and counting. Specifially, we employ COCO (Lin et al., 2014) and ReiCOCO(+/g) (Yu et al., 2016) for detection evaluation; RefCOCO(+/g) and ReasonSeg (Lai et al., 2024) for segmentation evaluation; PixMo- Count (Deitke et al., 2024) and CountBench (Paiss et al., 2023) for counting evaluation. Details of benchmarks and metrics can be found in Appendix B.
Evaluation Benchmark.
Training Data. The training data is sourced from the training splits of four datasets: LVIS (Gupta al., 2019), RefCOCOg (Yu et al., 201 6). gRefCOCO (Liu et al., 2023a), and LISA+ (Yang et al., 2023). We randomly collect approximately 7k training samples, with around 1,800 from each dataset.
Implementation Details. We initialize VisionReasoner with Qwen2.5-VL. and SAM2. We employ batch size of 16 and a learning rate of le-6. The training objective is Equation (2).
4.2 MAIN RESULTS
We compare the results with LVLMSs and task-specific models on each the three fundamental task types. It is worthy note that our VisionReasoner capable of handling different tasks within the same model and is evaluated in a zero-shot manner.
Detection. We compare VisionReasoner with several state-of-the-art LVLMs, including Shikra (Chen et al., 2023), InternVL2-82 (OpenGVLab, 2024), Qwen2-VL-77 VL-7B (Wang et al., 2024) and Qwen2.5VL. 7B (Bai et al., 2025). For task task-specific models, we evaluate against VGTR (Da et al., 2023), Trans VG (Deng et al., 2021), RefTR (Li Sigal, 2021), MDETR (Kamath et al., 202 OWL-ViT (Minderer al., 2022), YOLO-Worle (Cheng al., 2024), GroundingDINO (Liu al., 2024a), DQ-DETR (Liu

---

Published as a conference paper at ICLR 2026
Table 2: Performance comparison on segmentation tasks and counting tasks, We use SAM2 for vision-language models if necessary in segmentation tasks.
Segmentation
Counting
Avg.
Method
ReasonSeg
RCO
RCO+
RCOg
Pixmo
Count
Avg.
val
testA
testA
val
test
test
Task-specific Models
LAVT
75.8
68.4
62.1
ReLA
22.4
21.3
76.5
71.0
66.0
51.4
Large Vision-language Models
LISA-7B
44.4
36.8
76.5
67.4
68.5
58.7
LLaVA-OV-7B
55.8
53.7
78.8
62.8
GLaMM-7B
58.1
47.1
55.6
PixeIIM--B
76.5
71.7
70.5
Seg-Zero-7B
62.6
57.5
80.3
76.2
72.6
69.8
Owen2-VL-7B
VL-7B
44.5
38.7
68.7
65.7
63.5
56.2
61.6
56.3
80.4
66.1
Qwen2.5-VL-7B
56.9
52.1
79.9
76.8
72.8
67.7
58.1
53.1
78.8
63.6
VisionReassner-7B
66.3
63.6
78.9
74.9
71.3
71.0
70.1
70.7
89.2
76.7
Table Comparison multi- object matching. Our code achieves a speedup.
Table Comparison on the reasoning length.
Table 5: Comparison on differ- ent RL algorithm.
Data
Avg. Len (# words)
RL
ReasonSeg-val
Hungarian CatchComp/
Time
COCO
62
Baseline 56.9
x 10
RefCOCOg ReasonSeg 71
65
GRPO DAPO
61.9
5 x 10
61.7
al., 2023d), GLIP (Li al., 2022). Since LVLMs do not output confidence score, we approximate it using the ratio of the bounding box area to the total image area (bbox area / image_ area) to enable compatibility with COCOAPI (Team, 2014). However, this coarse approximation leads to underestimated AP scores. As shown in Table 1, VisionReasoner achieves superior performance among LVLMs. While our model shows a performance gap compared to some spec-fc baselines on COCO datasets, it maintains competitive advantages due its superior generalization capability,
Segmentation. evaluate VisionReasoner against state-of-the-art LVLMS,, including LISA (Lai et al., 2024), GLaMM (Rasheed et al., 2024), PixelLM (Ren et al., 2024b), Seg-Zero (I.iu et al., 2025a), Qwen2-VL. (Wang et al., 2024) and Qwen2.5VL (Bai et al., 2025). For these LVLMs, we first extract bounding box predictions and subsequently send them into SAM2(Ravi et al., 2024) to generate segmentation masks. We also compare task-specific models, including LAVT (Yang et al., 2022) and ReLA (Liu al., 2023b). For models that do not report gloU, report their cloU as an alternative. As shown in Table 2, VisionReasoner achieves state-of-the-ard performance, outperforming both general-purpose LVLMs and task-specific approaches.
Counting. We evaluate ResionReasoner against state-of-the-ard LVV.Ms, including LLaVA-OneVisiie (Li al., 2024a), Qwen2-VL-7B (Wang et al., 2024) and Qwen2.5VL-7B (Bai al., 2025). We evaluate these LVLMs in a wwweddeeet-the--ooue manner. As shown in Table 2, VisionReasoner achieves satt--ofthh-all performance.
4.3 ABLATION STUDY
We perform ablation studies assess the effecttivneess of our design and validate the optimal hyper- parameter selection and training recipe design for VisionReasonner. We also evaluate VisionReasoner VQA tasks.
Multi-object Matching. We quantitatively assess the efficiency of our two key design choices for multi-object matching: the Hungarian algorithm and batch computation. In scenario with 30 objects, Table 3 demonstrates that a non-batch matching require 2 x 10 seconds to complete, while our 4 speedup.
optimized approach achieves matching in just 5 10-1 seconds
7

---

Published as a conference paper at ICLR 2026
RefCOCOg ReasonSeg
Repeat
Response Length
300
Non-repeat
61.9
280
82.2
250
84.1
150
Step
(a) Comparison non-repeat reward
(b) Avg. response length during training
Figure 4: Ablation on non non-repeal reward. (a) Consistent performance gain across different datasets
using non-repeated reward. (b) Non-repeat rewards lead to shorter response lengths.
Table 6: Performance comparison on different training data.
Training Data
Det
Seg
Method
Avg.
RefCOCOg gRefCOCO LVIS LISA+
Re(COCOg-val ReasonSeg-val
84.1
61.9
73.0
85.8
63.8
74.8
VisionReassnne-7B
85.5
64.2
74.9
86.1
66.3
76.2
Table Performance comparison on VQA tasks.
Method
OCRBench RealworldQA MMMUPromisionn MMMUPro ChartQA Doc
Qwen2VL-7B
809
66.1
28.0
33.8
81.4
94.5
Owen2.5VL-7B
822
69.2
32.4
36.4
83.1
95.7
Vision 7B
825
69.5
32.6
37.4
84.9
96.0
Reasoning Length. As shown in Table 4, our analysis reveals that the model's reasoning length adapts dynamically to text query complexity. Specifically, for simple class names in COCO and short phrases in RefCOCOg, the reasoning process is relatively concise. In contrast, complex reasoning intensive queries in ReasonSeg require longer reasoning processes.
-easoning-intensivee
Non Repeat Reward. Figure 4 (a) presents the performance comparison with and without the non-repeat reward. Models are trained only on 2,000 samples from RefCOCOg. The model achieves better results when trained using the non-repeat reward. Additionally, model without non-repeat reward tends to generate longer reasoning processes, as shown in Figure 4 (b), and we observe repetitive reasoning patterns during inference.
Different RL Algorithm. We use different on-policy RL training algorithm: the GRPO (Shao et al., 2024) and DAPO (Yu et al., 2025). Models trained only on 2,000 samples from RefCOCOg. As shown in Table 5, performance consistently improves across both algorithms, demonstrating that our training framework is both stable and generalizable.
Different Training Data. We conduct an ablation study on different training datasets, with results presented in Table 6. The four datasets provide diverse text annotations: LVIS uses simple class names, RefCOCOg contains single-object referring expressions, gReíCOCO includes expressions that may refer to multiple objects, and LISA++ features texts requiring reasoning. Our experiments demonstrate that these datasets consistently improve model performance.
Visual QA Ability. We also compare VisionReasonne's VQA (Masry al., 2022; Mathew et al., 2021; Liu et al., 2024b; Yue al., 2024; xAI, 2024) ability with Qwen2VL (Wang et al., 2024) and the baseline model Awen2.55V(BB. et al., 2025). As shown Table 7, VisionReasoner achieves a slight performance gain even though we do not train VQA data.
Sampling Number. Figure 5 presents performance comparison with different sampling number. Models are trained using all 7k training samples. We observe initial performance gain followed by a notable decline with larger sampling number, suggesting that excessive sampling may induce overfitting to the training distribution and consequently degrade generalization capability.
8

---

Published as a conference paper at ICLR 2026
Reason Reason5eg...
63.5
ReasonSeg.es ReasonSeg...
58,9
66.3
60.1
60.8
66.3
baseline
62.9
52.1
reasoning
63.6
reasoning
61.1
Figure 5: Different sampling number.
Figure Reasoning vs. no reasoning
Reasoning, Figure 6 compares the performance of models with and without reasoning. We eliminate reasoning process by removing thinking instruction and thinking reward. Models are trained using all 7k training samples. Our results show that both approaches outperform the baseline. And the reasoning-enhanced model demonstrates significant gain on intricate reasoning segmentation data.
4.4 HUMAN EVALUATION ON REASONING PROCESS
We employ three experts to conduct human evaluations on ReasonSeg- val to assess both answer consistency and im- age consistency of the reasoning trace. Image consistency measures whether the reasoning trace accurately describes the visual content in the image. Answer consistency eval- uates whether the objects and information mentioned in the reasoning trace are consistent with the final predicted output. These evaluations help assess the faithfulness and reliability of the model's internal reasoning.
Table 8: Reasoning Process Analysis by IoU Range. IC: Image Consistency; AC: Answer Consistency.
IoU Range
Num
IC (%)
AC (%)
0-0.25
26
76.9
46.2
0,25-0.50
29
100.0
93.1
0.50-0.75
35
100.0
91.1
0.75-1.00
110
100.0
100.0
We evaluate the reasoning traces different levels based on IoU (between prediction and ground-truth truth)) values and results shown in Table We find that the overall image consistency accuracy and answer consistency accuracy reach 97.0% and 90.5%, respectively. The majority of problematic reasoning traces are concentrated in cases where the IoU is below 0.25.
ALL
200
97.0
90.5
These results demonstrate that the reasoning trace of VisionReasoner is accurate and well well-grounded, even though the model was trained without human-annotated reasoning data.
4.5 QUALITATIVE RESULTS
We visualize some results Figure 7. Notably, VisionReasoner addresses several visual percep- tion tasks within shared model. More results and the reasoning process are provided in Ap- pendix G.
Detection
Segmentation "Where the garbage should be put?
Counting How many nail polishes?
Man yellow
bike.
CONCLUSION
Qualitative results different tasks. Reasoning process and more results are provided in Appendix G.
Figure
We present VisionReasoner, unified vision- language framework for reasoning visual per- ception tasks, By introducing novel multi-object cognitive learning strategies and curated reward functions, VisionReasoneer demonstrates strong capabilities analyzing visual inputs, generating structured reasoning processes and delivering task-specific outputs. Experiments across ten diverse tasks, spanning detection, segmentation and counting, validates the robustness and versatility of our approach. Notably, VisionReasoner achieves significant improvements over bascline, with relative performance gains of 29.1% % on COCO (detec- tion), 22.1 1 % on ReasonSeg (segmentation), and 13.2% on CountBench (counting). Human evaluation gellndro and faithful.
further reveals the reasoning traces of VisionReasoner

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This work was supported in part by the Research Grants Council under the Areas of Excellence scheme grant AoE/E-601/22-R.
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13992. 2025. 2, 6, 7,8
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint arXiv:2306.15799 2023. 6
Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real-time open-vocabulary object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16901-16911, 2024. 3, 6, 17
Cheng Da, Chuwei Luo, Qi Zheng, and Cong Yao. Vision grid transformer for document layout analysis. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 19462-19472, 2023. 6
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang. Jac Sung Park, Moham- madreza Salchi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint 47914:240..17146. 2024. 4, 6, 14
Jiajun Deng, Zhengyuan Yang. Tianlang Chen, Wengang Zhou, and Houqiang Li. Transvg: End-to- end visual grounding with transformers. In Proceedings the IEEE/CVF International Conference on Computer Vision, pp. 1769-1779, 2021. 6
Google. Gemini pro 2.5. https: http:///awam...oooo//////ceoe/aa
2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12944 2025. 3, 15
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance
segmentation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 5356-5364, 2019. 6
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1780-1790, 2021. 6
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 787-798, 2014. 4
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9579-95988 2024. 3, 7, 14
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.0 03326. 2024a.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded lunguage-mmae pre- training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10965-10975, 2022.
10

---

Published as a conference paper at ICLR 2026
Muchen Li and Leonid Sigal. Referring transformer: A one-step approach to multi-task visual grounding. Advances in neural information processing systems, 34:19652-1996 2021. 6
Yanwei Li, Yuechen Zhang, Chengyao Wang. Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jía. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403. 18814, 2024b. 2
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitmick. Microsoft coco: Common objects in context. In Computer vision- ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, 4, proceedings, part 13, pp. 740-755. Springer, 2014. 1, 3, 4, 6, 14, 17
Chang Liu, Henghui Ding, and Xudong Jiang. Gres: Generalized referring expression segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
2023a.
Chang Liu, Henghui Ding, and Xudong Jiang. Gres: Generalized referring expression segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
23601, 2023b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in 2023c. 2
neural information processing systems, 36:34892-3444
Shilong Liu, Shijia Huang, Feng Li, Hao Zhang, Yaoyuan Liang, Hang Su, Jun Zhu, and Lei Zhang Dq-detr: Dual query detection transformer for phrase extraction and grounding. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 1728-1736 2023d.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision, pp. 8-55. Springer, 2024a. 6
Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67(12)220000, 2024b. 8
Yuqi Liu, Bohao Peng, Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, and Jiaya Jia. Seg- zero: Reasoning-chain guided segmentation via cognitive reinforcement. arXiv preprint arXiv:2503.06520, 2025a. 1, 3, 15
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.01785 2025b. 1,
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A bench- mark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 10244, 2022. 8
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2200-2209, 2021. 8
Meta.
Llama 3.2:
Revolutionizing
edge ai and vision with https: //ai meta. com/blog/ 2024. 2
open,
customizable models.
1lama - 3- cowooonntt-222---iiieee
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al. Simple open-vocabulary object detection. In European conference computer vision, pp. 728-755. Springer, 2022.
OpenAL. Chatgpt. https: https////hht..opa......m www.oppencc.ooo 2023. 15
OpenAl. Introducing gpt-4.1 in the api. https: /ttp:///ppeaaa..a com/ wwt.c//....innceeeeeeeeeeeeeeeee 1--pp---111 2025. 1, 3,3
11

---

Published as a conference paper at ICLR 2026
OpenGVLab. Introduction of internv12 series, 2024. URL htt.///tttt///ttttteereraaaadaaa io/ ht//////aattt/tteeeeeeeeeeeeeeee int.essttinneetniii1 //intoduuciionnnn.mmmmmomm html.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching clip to count to ten. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3170-3180, 2023. 1, 4,4, 6, 14
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, Qixiang Ye, and Furu Wei. Grounding multimodal large language models to the world. In The Twelfth International Conference Learning Representations, 2024. 3,
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-5336 -5341,, 2023. 3
Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing. Ming-Hsuan Yang, and Fahad Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13009-13011 2024.
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint 240.::20000000144.. 2024. 7
Tianhe Ren, Yihao Chen, Qing Jiang, Zhaoyang Zeng, Yuda Xiong, Wenlong Liu, Zhengyu Ma, Junyi Shen, Yuan Gao, Xiaoke Jiang, et al. Dino-x: unified vision model for open-world object detection and understanding. arXiv preprint arXiv:2411.1434.
2024a. 3
Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 26374-26383, 2024b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17X7v:1707.06347, 2017. 3
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024. 3, 8, 15
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409919226, 2024. 14
COCO Team. Microsoft COCO: Common objects in context.
https::/gittubbcoo com/
cocodataset/cocoaat 2014. 7, 14
RI-V
RI-V. readme- ov- file, 2025. 1
Team.
com/ http:s////tttaa.ccc/////////////
attps://gitte http://tttuuuuccooppeeepeepppp
Peng Wang. Shuai Bai. Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing lisiunllanguage model's perception of the world at any resolution. arXiv preprint arX9: 12/91, 2024. 1, 2, 6, 7,
XAI. 1r vision. httpp///...alnnensggokk-11. 2024. 8
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li. Dayiheng Liu, Fei Huang. Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 1515 2024. 15
12

---

Published as a conference paper at ICLR 2026
Senqiao Yang, Tianyuan Qu, Xin Lai, Zhuotao Tian, Bohao Peng, Shu Liu, and Jiaya Jia. Lisa++: An improved baseline for reasoning segmentation with large language model. arXiv preprint arXiv:2312.17240, 2023. 5,
Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr. Lavt: Language-aware vision transformer for referring image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18155-18165, 2022. 7
Licheng Yu, Patrick Poirson, Shan Yang. Alexander C Berg. and Tamara Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Purt 14, pp. 69-85. Springer, 2016. 1, 4,
6, 14
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 2025.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng. Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi multi-discipline multimodal under- standing and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 9536-9567, 2024. 8
Yaowei Zheng, Junting Lu, Shenzhi Wang, Zhangchi Feng, Dongdong Kuang, and Yuwen Xiong. Easyrl: An efficient, scalable, multi-modality training framework. https: htapp:///gthhan /ttp//ggtttt///t//////tt com/ hiyouga/Ease EasyRl, 2025. 1.3
Zhisheng Zhong, Chengyao Wang, Yuqi Liu, Senqiao Yang, Longxiang Tang, Yuechen Zhang, Jingyao Li, Tianyuan Qu, Yanwei Li, Yukang Chen, et al. Lyra: An efficient and speech-centric framework for omni-cognition. arXiv preprint 2rX0v:2412.09501,, 2024.
13

---

Published as a conference paper at ICLR 2026
Detection
Segmentation
Counting
"Car'
"Pole on a green umbrella
"How many light bulbs?"
"A van behind the yellow vehicle"
"eessn who is speaking currently."
"How many airplanes? ?"
Figure 8: Examples from evaluation benchmarks. Zoom in for better viewing,
A THE USE OF LARGE LANGUAGE MODELS (LLMs)
LLMs are used only polish writing in this paper.
B DETAILS OF EVALUATION BENCHMARKS
We use ten benchmarks to evaluate model per- formance across general vision perception tasks. Our evaluation includes three fundamental task types: detection, segmentation and counting. Specifially, we employ COCO (Lin et al., 2014) and RefCOCO(+/g) (Yu et al., 2016) for detec- tion evaluation; RefCOCO(+/g) and ReasonSeg (Lai et al., 2024) for segmentation evaluation; PixMo-Count (Deitke et al., 2024) and Count- Bench (Paiss et al., 2023) for counting.
Table Statistics of evaluation benchmarks. We report the number of instances for detection and segmentation tasks. The reported numbers com- bine validation and test splits where applicable.
Type
Data
# of samples
COCO
36.781
RefCOCO
5,786
Det
RefCOCO+
5,060
RefCOCOg
7,596
Annotation Preparation. To ensure consis- tency across all evaluation tasks, we standard- ize the evaluation data by converting all sam- ples into unified multi-modal conversation for- mat and removing potential information leakage, This preprocessing involves: converting numeric class labels to textual descriptions in COCO (Lin et al., 2014); removing explicit numerical ref- erences from text descriptions in CountBench (Paiss et al., 2023); applying consistent formatting across all datasets to maintain evaluation fairness.
RefCOCO
1.975
RefCOCO+
1,975
Seg
RefCOCOg
5,023
ReasonSeg
979
Pixmo- Coumt
1,064
Count
CountBench
504
SUM
66,023
Evaluation Metrics. For object detection COCO, adopt the standard AP metric computed using the COCO API (Team, 2014). For referring object grounding on RefCOCO(+/g). we use bbox AP, which measures detection accuracy at IoU threshold of 0.5. For object segmentation on RefCOCO(+/g) and ReasonSeg, we use gloU, computed as the mean IoU across all segmentation masks. For counting tasks, we use count accuracy as evaluation metric.
Statistics and Visualization. We show the statistic data in Table 9, For detection and segmentation tasks, we report the number of valid instances. For counting tasks, we provide the total number of test samples. Our evaluation comprises total of 66,023 test samples, covering three fundamental visual perception task types and 10 specific tasks. We visualize some examples in Figure
C MORE TRAINING DETAILS
The training is conducted on a single node with GPUs and the entire training process takes hours. The peak GPU memory usage is approximately 80 GB, though this can be adjusted through hyperparameters such as memory utilization in VeRL (Sheng et al., 2024). The reward converges at around 100 steps, and the best checkpoint is typically obtained at around 200 steps.
14

---

Published as a conference paper at ICLR 2026
D EXPERISSION LEVEL EVALUATION ON REFCOCO(+/()
Our primary evaluation, detailed in Appendix B, reports instance-level performance. However, since the RefCOCO(+/g) benchmarks provide multiple expressions per image, we additionally present expression-level results in Table 10.
Table 10: Performance comparison on expression-level RefCOCO((/g) tasks. Results with are cited from the Qwen2.5-VL report but not reproducible in our environment.
Detection
Method
RefCOCO
RefCOCO+
RefCOCOg
SUM
val
testA
val
testA
val
test
Wwen2.5-VL--BB
84.2
89.1
87.2
87.2
$30.2
Qwen2.5-VL-77 VBBBB
89.0
92,0
83.2
88.3
86.4
86.5
525.4
Vision
89.1
91.0
85.0
87.6
87.6
88.5
528.8
E TASK ROUTER
In order to identify users' instruction automatically during inference, we also train Touter The RoseR Frumer is pure language model that processes textual instructions. For any given instruction T. TaskRouter performs a semantic analysis and outputs task classification C into one of four predefined fundamental task calegories. This mapping can formally expressed as:
C = Frouter T
We train TaskRouter using the GRPO algorithm (Shao et al., 2024), providing reward signals exclusively upon correct task classification. We evaluate the effective of the TaskRouter and results are shown on Table 11. The task classification dataset is constructed from diverse visual perception datasets and Al-generated samples. For datasets that include textual instructions (e.g., RefCOCOg), retain their original instructions and corresponding fundamental task categories. Additionally, for each fundamental task type, employ ChatGPT (OpenAI, 2023) to generate instructions and target categories. The final dataset comprises 20,000 training samples and 4,000 test samples. Although the state-of-the-a Qwen2.5 (Yang et al., 2024) demonstrates strong performance in instruction following and zero-shot task classification, its accuracy drops below 50% in our complex scenario. contrast, our task router module, trained using reinforcement learning, achieves significantly better performance.
Table 11: Comparison on the task classification.
Model
Accuracy
Qwen2.5-1.5B
46.3
TaskRouter 1.5B
F USER PROMPT TEMPLATE
To guide the policy model toward generating desired outputs during exploration, we employ the user prompt template presented in Table 12. This prompt template is inspired by DeepSeek-R1-Zens (Guo et al., 2025) and Seg-Zero (Liu et al., 2025a).
G QUALITATIVE RESULTS
We visualize results on Figure 9. Our model generates comprehensive reasoning processes for all tasks while producing expected outputs. We find that VisionReasoner can effectively distinguish between similar objects, as shown in the visual grounding and referring expression segmentation. VisionReasooner also accurately localize multiple targets, shown in object detection and counting. We also observe that the length of the reasoning process adapts dynamically: more intricate image- query pairs elicit detailed rationales, while simpler inputs result in concise explanations.
15

---

Published as a conference paper at ICLR 2026
Table 12: User Prompt. " "[Questionf" is replaced by user questions during training and inference.
User Prompt
"Please find "/Question]" with bboxs and points."
"Compare the difference between object(s) and find the most closely matched object(s)."
"Output the thinking process in <think> </think> and final answer in <answer> </answer> tags." "Output the bbox(es) and point(s) inside interested object(s) in JSON format."
i.e. <think> thinking process here </think>
<answer> "bbox 2a": 110, 100, 2001, "point 2d" 130, 11011, ("bbox_2d"; [225, 296, 706, 786], "point_2d": [302, 4101 </an///nnwwr
COCD
<thinlo An oven is typically built-in appliance used for baking cooking food.
image, appllances that appear to be ovens, on the left and one on the right. The left one is standalone oven with visible door, while right integrated into the cabinetry and has a similar design. </think
Oven
control panel
RefCoCo
<thinlo yellow matorcycle clearly visible
of the image, with a rider wearing black and orange sult.
Man on yellow bike.
The rider on the yellow matorcycle is wearing helmet and
gloves, and the motorcycle has distinct yallow color with
some green and black accents. </thinks
R
<thinlo The task involves identifying the person holding the blue mug. need to look for the blue mug and then determine holdling The blue clearly visible on
Who is holding the blue mug?
the table, and can see hand holding it. The hand is identifying
wearing beige sweater,
person. </thinks
RefCOCOg
and blue, bath parked on grass. The green truck on the left side right. green truck is clearly visible and distinct in color, making it the query. </thinks
<think The image contains trucks,
the image, and the blue truck
green truck is gress.
object to
ReasonSeg
<thinkc
The task involves identiffyng where garbage should which typically refers designated trash bin receptacle, In the image, there is gray trush bin on the right likely place garbage to be put, as common location for such items.
Where the garbage should be put?
garage
</think>
Pixmo- Count
<think The task involves identifying and bounding the
people image, ... closely matched ones. people the image are clearly visible and can be identified by their features clothing.
People
people
main subjects of the image. </thinko
Total number is
CountBench
The task involves identifying the number of nail polishes the image and comparing the objects find the most closely matched ones..... I will count the nall polishes.
<think
How many nail
polishes are in the image?
Total number is..
Figure 9: Qualitative results on different tasks. Zoom in for better visualization.
H EXTENSION
More Applications. Our VisionReasoner can be easily extended to other visual perception tasks that need reasoning. We just need to add light-weight module for different output format. The intermediate output {B and [N,) serve as bridge to connect other modules. Figure shows extension to referring expression pose estimation and referring expression depth estimation.
16

---

Published as a conference paper at ICLR 2026
Pose Est.
<think> To estimate the pose of the person making phone identify individual holding phone the left holding phone to their car, indicating he making a phone call. The other individuals
Estimate the
their In the image, the
pose of people
who making
them holding
phone call.
phone to their ear. </thinks
Depth Est.
<think> question
Where can
of the depth of the object. The object in the image a red sultable place rest. estimate depth, need to consider the distance from the front edge of the armchair to the back edge. <dge.</think>>
armchair, which
estimate the depth.
Figure 10: Extending VisionReasoner to more visual perception tasks.
Hybrid Mode, We can also employ a hybrid mode, that is directly using traditional visual models (c.g. World (Cheng et al., 2024)) for simple categorical instruction (i.e. bird) and VisionReasoner for complex instructions (i.e. 'Where can have a rest?").
I CONCEPTS CLARIFICATION
We formally define the key terms used in this work. As illustrated in Table 13, our hierarchical task formulation adopts the COCO dataset (Lin al., 2014) representative example:
Table 13: Key Terminology and Definitions.
Concept
Definition
Fundamental Task Types
Reformulated task categorics (e.g.. detection)
Task Type
Task catcgory (e.g., object detection)
Task
Concrete benchmark (e.g., COCO object detection)
J DETAILS OF TASK REFORMULATION
Within our framework, we categorize task types illustrated in Table 14 and Table 15, These tables highlight our grouping of task types based on their similarities. important to note that although this taxonomy covers broad range of task types, the current implementation of VisionReasoner is evaluated on only 10 representative tasks, with comprehensive evaluation all task types reserved for future research.
Table 14: Fundamental Task Types: Counting and Visual Question Answering.
Counting
VQA
Object Counting
Visual Question Answering (VQA)
Crowd Counting
Classification
Density Estimation
Image Captioning
Podestrian Detection
Question Answering
Crowd Estimation in Dense Scenes
Visual Reasoning
Traffic Counting in Surveillance
Visual Question Answering
Relational Reasoning
17

---

Published as a conference paper at ICLR 2026
Table 15: Fundamental task types: Detection and Segmentation.
Detection
Segmentation
Visual Grounding
Semantic Segmentation
Object Detection
Instance Segmentation
2D Object Detection
Lane Detection
Small Object Detection
2D Semantic Segmentation
Defect Detection
Medical Image Segmentation
Face Detection
Human Part Segmentation
License Plate Detection
Action Segmentation
Anomaly Detection
Video Object Segmentation
Human Detection
Referring Expression Segmentation
Surgical Tool Detection
Saliency Detection
Dense Object Detection
Salient Object Detection
Open World Object Detection
Semantic Segmentation Remote Sensing Imagery
Zero-Shot Object Detection
Crack Segmentation
Animal Action Recognition
Action Unit Detection
Robotic Grasping
RGB Salient Object Detection
Object Localization
Boundary Detection
Hand Detection
Crack Segmentation for Infrastructure
Visual Relationship Detection
Surgical Tool Segmentation
Open Vocabulary Object Detection
Oriented Object Detection
Object Detection in Indoor Scenes
Object Detection in Aerial Images
Person Search
Object Recognition
18


Reviews:
[{"review_id": "a0nPAlaSO0", "reviewer": "Reviewer_ybFD", "summary": "The authors propose a framework wherein a base vision-language model (VLM) is RL-postrained for 3 text-guided perception tasks: object detection, segmentation and counting. They utilize the Qwen-2.5-VL-7B model as the base VLM and SAM2 for segmentation, while GRPO is utilized for RL training on 7000 training samples collected from LVIS, RefCOCOg, gRefCOCO and LISA++. Rewards for RL include thinking format, answer format, 'non repeat' format, bboxes IoU reward, bboxes L1 reward and points L1 reward. For matching rewards for multiple objects, the authors propose a hungarian matching algorithm.\n\nResults show the authors proposed 'VisionReasoner' model outperforms existing large VLMs and the base Qwen2.5-VL-7B model on multiple benchmarks. Further analysis and ablations are also provided such as impact of RL algorithm, human analysis of reasoning process comparison of response lengths across dataset and results on visual question answering (which was not RL trained for).", "questions": "Please refer to weaknesses section above. \n\nIn addition:\n1.  What are potential reasons for the RL-postrained result being less than the base model for Detection and segmentation on RefCOCO, RefCOCO+, RefCOCOg?\n2. Will code and models be open source for reproduction?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The proposed framework is straightforward and clear to understand -- usage of RL to postrain a VLM for 3 core perception tasks with multiple rewards to capture the perception tasks and reasoning lengths.\n2. The results show clear improvements of RL postraining (using GRPO objective) in improving results on appropriate benchmarks.\n3. Experiments include ablations and analysis to understand the method.", "weaknesses": "1. The authors state that previous methods employ RL in a task-specific manner and utilize distinct reward functions for different tasks. However, in my opinion, authors in their work also seem to employ task-specific rewards for detection and point matching in addition to format rewards. \n2. A zero-shot chain-of-thought prompted baseline should be present as without it, it is currently unclear whether just directly prompting model to think step-by-step or breakdown the prompt can also be sufficient to obtain a decent performance boost on the base model for considered perception tasks. This is an important missing baseline in my view.\n3. Authors mention performing human evaluation on reasoning process but do not provide details on how this is done (e.g.  how many participants, the task setup, etc.)\n\nRelatively minor: \n\n4. Experiment details can be more extensive: e.g. \n- How many GPUs are used for training and how many epochs/steps for RL convergence? \n- Will code and models be open source for reproduction?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UX1dK5Di6m", "reviewer": "Reviewer_9SJ9", "summary": "This paper presents VisionReasoner, a unified framework for compositional multi-hop reasoning across a wide range of vision-language tasks. Unlike traditional task-specific methods, VisionReasoner decomposes complex queries into structured reasoning steps using a Unified Intermediate Representation (UIR) and a task-agnostic planner-executor architecture. The planner produces reasoning programs from natural language questions, and the executor interprets them over visual observations. The authors train the model on diverse reasoning tasks and demonstrate strong zero-shot generalization, outperforming task-specific models on multiple benchmarks without fine-tuning.", "questions": "See weaknesses above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed architecture (planner + executor + UIR) offers a principled way to unify reasoning over different modalities and domains.\n2. The system generalizes across unseen reasoning tasks and domains with minimal or no task-specific supervision.", "weaknesses": "1. The strongest recent baselines use retrieval-augmented reasoning where structured planning is implicit. The authors only compare with older systems like RAML and ReGrouP, not modern VLMs fine-tuned with in-context reasoning.\n2. The planner is the backbone of VisionReasoner, yet the accuracy of program generation is not reported. The author shall consider to include planner-only accuracy (e.g., execution success rate, semantic match with gold reasoning paths).\n3. The UIR uses a limited set of programmatic operations that must be predefined. This suggests limited compositional expressivity, especially when handling spatial, temporal, or logical reasoning beyond object-centric understanding.\n4. The executor module is assumed to interpret UIR programs correctly. There is no ablation showing how executor errors affect end-to-end performance, nor how robust the system is to ambiguous or poorly planned programs. The authors shall consider to include experiments around the accumualted error issue.\n5. While modularity aids interpretability, separating planner and executor may hurt end-to-end task performance. There's no attempt at joint fine-tuning, which could close the gap with SOTA models on certain benchmarks. I think joint fine-tuning would interest more potential audiences in the area.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Oa9lbF4ZQF", "reviewer": "Reviewer_7Pxd", "summary": "The paper proposes VisionReasoner, a unified RL-based LVLM that handles detection, segmentation, and counting in a single model via a “locate first (boxes + points), then segment/count” paradigm, while producing interpretable reasoning traces. It uses GRPO with a task-agnostic reward scheme (thinking/format and non-repetition rewards, plus multi-object IoU and L1 accuracy rewards) and efficient multi-object matching via batched computation and the Hungarian algorithm. The authors report zero-shot performance on 10 benchmarks, claiming substantial gains with only ~7k training samples and no degradation to VQA.", "questions": "1. Insufficient Transparency in Training Data Distribution\nThe paper reports using approximately 7,000 training samples but does not provide details on their distribution across different task types. Since the allocation of training data can significantly affect performance balance in multi-task learning, it is recommended to specify the number and proportion of samples for each task type, along with the criteria for data partitioning, to enhance reproducibility and interpretability.\n\n2. Limited Coverage of Large Vision-Language Models\nThe current experiments mainly compare with models such as Shikra and Qwen2.5-VL. However, Shikra is an early work, and the overall coverage of more recent and powerful large vision-language models (LVLMs) remains limited. Including additional representative LVLMs—such as LLaVA, MiniGPT-4, InstructBLIP, or Qwen-VL-Max—would provide a more comprehensive and up-to-date evaluation, thereby better demonstrating VisionReasoner’s competitiveness within a unified multi-task framework.\n\n3. Ambiguity in Reward Function Weight Design\nThe paper introduces multiple reward functions (e.g., format reward, IoU reward, L1 reward) but does not clearly explain the weighting scheme or its underlying rationale. Furthermore, the paper lacks quantitative analysis of the relative importance of these rewards (e.g., through ablation studies). It is recommended to include further experiments or discussions to strengthen the justification and interpretability of the reward design.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Unified multi-task framework design. \nThe paper successfully constructs a unified framework capable of handling three major categories of visual perception tasks—detection, segmentation, and counting—simultaneously. This unified design offers several notable advantages.\n\n2. Outstanding data efficiency and scalability. \nWith only 7,000 training samples, the VisionReasoner-7B model achieves strong performance, demonstrating impressive data efficiency and generalization capability.", "weaknesses": "1. The experimental evaluation could benefit from a broader and more up-to-date set of baseline models. The paper mainly compares VisionReasoner with Shikra and Qwen2.5-VL; however, Shikra, as an early work from 2023, may not fully reflect the current progress of large vision-language models (LVLMs). Expanding the comparison to include more recent LVLMs could provide a fairer and more comprehensive assessment of VisionReasoner’s performance in the current landscape.\n\n2. Some implementation aspects could be described in greater detail. For instance, the distribution of training data across different task types is not fully specified, and the weighting or design rationale of the overall reward function is somewhat unclear. Providing additional clarification in these areas would enhance the work’s reproducibility and help readers better understand the factors contributing to the model’s performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "na8i7rMsPl", "reviewer": "Reviewer_H7fg", "summary": "The paper proposes a unified LVLM for different perception tasks such as detection, segmentation, counting, and more. Through jointly optimizing the reward formulated with different metrics for perception tasks using GRPO and DAPO, the model is able to outperform the base model significantly on various perception tasks. Meanwhile, the model also shows better performance on VQA tasks despite not being fine-tuned on those tasks.", "questions": "What is the numerical performance of $mmmu_{val}$ for VisionReasoner?\n\nOn which tasks does it perform well, and on which tasks does it underperform compared with the base model?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. A unified LVLM for different perception tasks is proposed, which could benefit the broader community.\n\n2. The proposed method is simple yet effective.\n\n3. The experiments and ablation studies are comprehensive.", "weaknesses": "1. The contribution seems somewhat limited, as the paper mainly focuses on designing effective rewards for various perception tasks.\n\n2. More comparisons between VisionReasoner and the base model on VQA tasks would be beneficial. It would also be interesting to understand whether there is any degradation in the model’s general capability since it is trained on perception-specific tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)