FineRS: Fine-grained Reasoning and Segmentation of Small Objects with Reinforcement Learning
Authors: Lu Zhang
Jiazuo Yu
Haomiao Xiong
Ping Hu
Yunzhi Zhuge
Huchuan Lu
You He
Venue: NeurIPS 2025 poster ()
Keywords: High-Resolution
Small Object Segmentation
MLLMs
Reinforcement Learning
Primary area: deep_learning

Abstract: Multi-modal Large Language Models (MLLMs) have shown remarkable capabilities across a wide range of vision-language tasks. However, due to the restricted input resolutions, MLLMs face significant challenges in precisely understanding and localizing visual details in high-resolution images---particularly when dealing with extra-small objects embedded in cluttered contexts. 
To address this issue, we propose FineRS, a two-stage MLLM-based reinforcement learning framework for jointly reasoning and segmenting extremely small objects within high-resolution scenes. FineRS adopts a coarse-to-fine pipeline comprising Global Semantic Exploration (GSE) and Localized Perceptual Refinement (LPR). Specifically, GSE performs instruction-guided reasoning to generate a textural response and a coarse target region, while LPR refines this region to produce an accurate bounding box and segmentation mask. To couple the two stages, we introduce a locate-informed retrospective reward, where LPR's outputs are used to optimize GSE for more robust coarse region exploration.  Additionally, we present FineRS-4k, a new dataset for evaluating MLLMs on attribute-level reasoning and pixel-level segmentation on subtle, small-scale targets in complex high-resolution scenes. Experimental results on FineRS-4k and public datasets demonstrate that our method consistently outperforms state-of-the-art MLLM-based approaches on both instruction-guided segmentation and visual reasoning tasks.

FINERS: Fine-grained Reasoning and Segmentation of Small Objects with Reinforcement Learning
Lu Zhang's Jiazuo Yu!, Haomiao Xiong', Ping Hu?, Yunzhi Zhuge'; Huchuaa Lu', You He
Dalian University of Technology. Dalian, China
2 University of Electronic Science and Technology of China, Chengdu, China
Tsinghua Shenzhen International Graduate School, Shenzhen, China
zhangluu@d edu. cn, yujiazzo@maiil.e.. www.ddl.ddd... .duitedddccom
attpss//iiat hiti//iiiaauuuzguluu get.gggtthit github..... 1o/FINERS/
Abstract
Multi-modal Large Language Models (MLLMs) have shown remarkable capabili- ties across a wide range of vision- Dision-language tasks. However, due to the restricted input resolutions, MLLMs face significant challenges in precisely understanding and localizing visual details in high-resolution umages-particullll particularly when dealing with extra-small objects embedded in cluttered contexts. To address this issue, we propose FINERS, a two-stage MLL.-bassd reinforcement learning framework for jointly reasoning and segmenting extremely small objects within high-resolution scenes. FINERS adopts a coarse-to- pipeline comprising Global Semantic Exploration (GSE) and Localized Perceptual Refinement (LPR). Specifically, GSE performs instruction-guide target region, while LPR refines this region to produce an accurate bounding box and segmentation mask. To couple the two stages, we introduce a pocate-informec retrospective reward, where LPR's s outputs are used optimize GSE for more ro- bust coarse region exploration. Additionally, we present FINERS-44 for evaluating MI.LMss on attribute-level reasoning and pixel-level segmentation subtle, small-scale targets in complex high-resolution scenes. Experimental results FINERS -4k and public datasets demonstrate that our method consistently outperforms state-of of-the-ar art MLLM-bas approaches on both instruction-guie guidedd segmentation and visual reasoning tasks.
reasoning to generate a textural response and a coarse
new dataset
Introduction
Recently, Multi-modal Large Language Models (MLLMs) D 2 3 4 have achieved remarkable success in a variety of vision-language tasks, such as visual question answering, referring expression comprehension, and instruction-guided segmentation. Among these tasks, a capacity of not only understanding what the
foundational challenge
instraction-guiided reasoning and segmentation user is asking, but also where the image the referred object appears at the pixel level. Some early attempts [3] integrate MLLMs 4
5 6 with foundational segmentation models 7 g enabling joint language generation and object segmentation for more interactive and interpretable visual understand- ing. However, these methods are tailored to standard-resolution images and large, prominent objects, where spatial structures are casily accessible the model's visual backbone. Their heavy reliance on global visual-semantic alignment becomes increasingly unreliable in scenes with dense layouts and small, low-saliency targets (see Fig. (a)).
To address this challenge, the research community has begun to explore the capacity of MLLMs to perceive fine-grained detailed objects within high-resolution images. To mitigate detail degradation
"Conesponding authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Segitero
93
the slide next court?
39
Yellow
Red
Open ended and Multiple- choice Instructions
Answer:
223
(A)
Input High-resokution Images and Ground Truth
Performance Comparison
FineRS 4k test
Incorrect Response and Segmentation
Incorrect Response Not Support Segmentation
(a)
(b) DC
of slide the tennis court is blue. The Final aption response is "B"
Sure, the result is Blue. [Seg]
(c) Segizero
Response and Segmentation
(d) Ours
Resporse and Segmentation
<think 1. Identify the tennis 6.
<think>> The slide next to the tennis court is yellow in color-.. distinet yellow hue.
Output the bounding box and the color the slide.</tink> Response: The slide next to the tennis yellow. Response: "A"
</think>
Figure 1: Given a user instruction and high-resolution image, existing MLLMs face limitations in effectively reasoning and segmenting small objects. (a) MLLMs (e.g.. LISA (31) designed for haandard-resolutio images fail to generalize to small objects. (b) MLLMs (e.g.. DC² (10) developed for high-resolution images only support optional question answering and lack localization ability. (c)
MLLMs (e.g., SegZero [16]) with RFT fail to produce explicit answers and accurate segmentation masks within a unified framework. (d) Our FINERS combines a coarse-to-fine perception pipeline with reinforcement learning, enabling unified, precise instruction-guided reasoning and segmentation of small objects in high-resolution images.
caused by image downsampling, existing methods I 9 10 mimic human visual perception by decomposing high-resolution images into smaller patches to achieve local vision-text alignment. However, due the scarcity of high-resolution data, these methods typically adopt training-free free pipeline, where the absence of supervised fine-tuning limits their perception accuracy in complex scenarios (as shown in Fig. (b)). More importantly, the lack of precise localization ability restricts their scalability and applicability to downstream tasks demanding pixel-level grounding and spatial reasoning.
Recent studies 12 13 have revealed that LLMs can generalize effectively to domain-specific tasks with only thousands of training samples. Morcover, incorporating "thinking" process prior to answering can significantly enhance their reasoning ability. The core technique behind this improvement is Reinforcement Fine- Tine-Tuning (RFT) 12 13, which enables LLMs to be emergently optimized for downstream tasks via data- fine-tuning. The success of RFT has driven the extension of vision RFT [4 including image classification [4] object detection [14], and reasoning segmentation [16]. However, due to input resolution limitations, these methods [16] still struggle to capture fine-grained details, and cannot simultaneously generate explicit answers and segmentation masks without unified
15] [6] empower MILMs across variety of dision-languaa tasks,
multi-task reward mechanism (see Fig. (c)).
To address the above issues, we propose FINERS, a two-stage MLLM framework for instruction- guided reasoning and segmentation of small objects in high-resolution images. FINERS is designed to jointly optimize semantic reasoning and spatial perception through coarse-to to- to-fine pipeline. Specifically, Global Semantic Exploration (GSE) module first performs instruction guided reasoning to produce both textual response and a coarse target region that contains the referred objects inside. The subsequent Localized Perceptual Refinement (LPR) module then refines this region by generating an accurate bounding box and the corresponding segmentation mask. To reduce reliance on heavy supervised fine-tuning we integrate vision RFT into our framework and design effective rewards to simultancously address object reasoning and segmentation. Specifically, in addition to

---

basic region/box regularization rewards, we introduce response reward to encourage the model to simultaneously generate both textual answers and object boxes. Furthermore, to effectively couple these two stages, we introduce a locate-informed retrospective reward, which uses LPR to guide and optimize the exploration behavior of GSE via reinforcement learning. The synergy between the coarse-to- fine framework and reinforcement learning not only enables fine-grained perception of extremely small objects but also allows for data-efficient multi-task training, resulting in a unified framework that consistently delivers high performance across both reasoning and segmentation tasks.
To enable comprehensive evaluation, we introduce FINERS -4k, a human-annotated high-quality dataset designed to benchmark model performance on Instruction Instruction-guided Segmentation (IS), Open- ended Visual Question Answering (OVQA), and Multiple-choice Visual Question Answering (MVQA). Compared to previous high-resolution benchmarks [0], FINERS-4K leverages UAV- captured imagery, providing large-scale, complex environments with extreme object size variability, scattered small-object distributions, and cluttered spatial contexts. Extensive experiments on FIN- ERS--4 and other public datasets [9 IQ] demonstrate that FINERS consistently outperforms existing MLLM-based methods in both answer accuracy and segmentation precision.
To summarize, our contributions are as follows:
We propose FINERS, a two-stage MILMM framework that jointly performs instruction-guidee reasoning and segmentation for small-object understanding in high-resolution images. To the best of our knowledge, FINERS is the first method to unify reasoning and fine-grained segmentation under a reinforcement learning paradigm.
We introduce FINERS-4K -4k, the first UAV-captured high-resolution dataset designed for instruction- guided reasoning and segmentation ultra-small objects, offering more challenging object distri- butions and spatial variability compared to previous datasets.
We conduct extensive experiments on FINERS-44 and other public datasets, demonstrating that the proposed FINERS consistently outperforms state-of-the-art MLLM-based approaches across instruction guided segmentation, open open-ended ended VQA, and multiple-choice VQA.
2 Related Works
MLLM-based Reasoning and Segmentation. The success of MLLMs has significantly advanced object detection and segmentation for more accurate open-world understanding. Pioneering efforts such as LISA 3 and LISA++ [17 introduce a <SEG> token to bridge MLLMs with segmentation models 77 To evaluate performance, they propose reasoning segmentation, extension of referring segmentation that enables simultaneous generation of text responses and segmentation masks. This design has inspired a series of follow-up works [18 19]. which further explore special-token-based interfaces to integrate vision-language reasoning with segmentation. Despite their promising results, these approaches still face notable limitations. First, most MLLM-based models restrict the input resolution to avoid off-oo-mymory risks, leading to severe downsampling that compromises fine- grained visual details and degrades performance on high-resolution imagery. Second, they rely heavily on supervised training with extensive public datasets [20] 3) which limits their adaptability and transferability to more challenging scenarios with limited training data availability. In this work, we aim to address the challenges of small- reasoning and segmentation within nigh-resolution images. We propose a two-stage MILM framework that combines global semantic exploration with localized perceptual refinement and applies reinforcement learning strategy to achieve data-efficient fine-tuning.
High-resolution Image Understanding and Reasoning, Recent studies 9 10 21
have revealed that MLLMs still face significant challenges in perceiving and reasoning over high high-resolution images, particularly for small and densely distributed objects. To overcome the resolution restrictions of MLLMs, fine-tuning-basee process them in parallel with visual encoders, enabling MLLMS to handle arbitrary-resllution inputs. Concurrently, SEAL (9] considers a more complex challenge of small object perception in high-resolution images. It introduces both an evaluation benchmark and a guided visual search mechanism that leverages LLM priors to selectively focus on important regions, effectively improving visual reasoning ability in complex and crowded high-resolution scenarios. Due to the scarcity of available training data, subsequent methods 110 21 have developed training-free pipelines that apply hierarchical image partitioning to form stepwise reasoning processes. Similarly, attention-baut based
methods 22 23 divide input images into uniform patches and
3

---

Table I: Comparisons of different benchmarks. The annotation type includes Question (Q), Answer (A), and object mask. The supported tasks are Multiple-choice Visual Question Answering (MVQA), Open-ended Visual Question Answering (OVQA), and Instruction-guide Segmentation (IS). For small object granularity. "Partial" means that the dataset contains some small objects but doesn't t provide explicit indication.
Dataset
HR Images
Annotation Type
Sample Num
Supported Tasks
Small Object Cranularity
V
Q&&A
191
MVQA
IRR Bench
10
/
Q&cA
200
MVQA
x
reCOCCOg
20
Qask
95,010
Partil Partill
ReasonSeg
X
QdcMask
1,218
Ours (F 48-4k)
/
OlcA&MMask
12,132
MYQALOVOA&IS
S/XS/XXS
about Color 185 about Othn
Quediton clock the about Pestion
Quntiinn parsn wearing black top Referneg Sey
oveN aboutt
Porkiom
OVER deoue
Qusitiors What shape a the wenting sign (2) Oclugon.
Quentoo Wirch country's slag evected
FINDES 4
Ancaer: Frszes
Guiellot colr the triysle wearnng (A) tlack (8) Gray Purple
Queeton What itap of the bilbard
2003
aboud Colar
OVEA
Option Sog about Color
MYOn about Postion
MMVQA Others
Ayounee Rectangle
(a) FiNmRt-44 Benchmark Category Distribution
(b) Visual Examples of Representive Task Types
Figure 2: Overview of our benchmark. (a) The innermost ring shows three instruction types. The middle ring presents the mask size distribution within each type. The outermost ring breaks each type down into four attribute categories: color, shape, position, and others. (b) Visualization of six representative examples, each illustrating a different attribute category (color, shape, position, others) across the three instruction types in the outermost ring.
visual intervention methods 11] enhance the perception of small visual details by interpreting and manipulating internal attention maps of MILMs. Despite these advancements, existing methods remain sensitive heuristic cropping algorithms and still fail to achieve precise object localization. In this paper, we introduce a large-scale dataset and propose a data-efficient fine-tuning pipeline that employs reinforcement learning adapt MILMs for small object reasoning and segmentation.
has become new-emerging technique for enhancing reasoning in large language models, as demonstrated by OpenAl's ol [1]] and DeepSeek R1-Zero [1]. Among them, a critic-fres free algorithm, Group Relative Policy Optimization (GRPO) 33] is designed eliminate Supervised Fine-Tuning (SFT) by directly comparing candidate responses in groups. Inspired by this, Visual-RFT [1] proposes an RL-based fine-tuning strategy for Large Vision-Language Models (LVLMss). improving performance on classification and detection tasks using GRPO-based rewards under limited supervision. However, is limited coarse-level tasks such as classification and detection, and does not support fine-grained segmentation. In contrast, Seg-Zero [16) leverages high-quality box-level rewards within an RL framework and feeds them into frozen SAM2 [8] segmentation model, enabling pixel-level visual perception and reasoning. While it performs well on challenges remain in small-object segmentation for high-resolution images. Moreover, Seg Zero adopts fixed reward paradigm. making it difficult to generalize to more diverse question-answert formats, such as open-ended and multiple-choice VQA. Inspired by the success of RFT in vision- language tasks, we apply RFT to our two-stage MILM framework, where both localization and VQA rewards are integrated to boost the unification of object segmentation and VQA. Besides, we introduce a retrospective reward between two stages for more consistent global-to-local perception.
Reinforcement Learning for MLLMs. Recently, Reinforcement Learning (RL) 24 25
and regular object scenarios,

---

3 FINERS-4K Benchmark
To comprehensively evaluate the capacities of MILMs for instructed-guided ultra-small object reasoning and segmentation, we construct new dataset FINERS-4K Unlike previous high-resolution benchmarks 10 that are primarily captured by handheld or ground-hased cameras under structured conditions, FINER S-kk comprises images captured by Unmanned Aerial Vehicles (UAVs). This enables a much wider field-of-view and introduces complex visual challenges such as dense layouts, extreme variations in object scale, and small-object sparsity.
We begin by collecting 4k-resolution drone videos (3840 2060) from YouTube and our own UAV footage. Volunteers are then tasked with filtering high-quality frames and annotating small objects with a triplet annotation consisting of question, answer, and mask. Considering the difficulty in annotating ultra-small objects in high resolution images, annotators were primarily instructed to identify a single, unambiguous small object of interest from the image and compose questions that uniquely specify it (e.g., referencing color, shape, position, or context). For several cases with multiple similar objects, they were required formulate disambiguating questions and to visually inspect the entire image to ensure no other object matched the same description. The annotation was completed by 14 volunteers, organized in pairs for mutual cross-checking. In addition, a team of 4 senior reviewers conducted final round of quality assurance to correct ambiguities and verify consistency. This multi-stage validation process was designed to maximize precision and minimize annotation bias. This process results in 8,411 annotated small entities across 4,563 high high-resolution images, yielding a total of 12, text-mask pairs. Specifically, we divide them into train set (8,956), validation set (749), and test set (2,427) The overall comparison of our FINERS 444 and other datasets is illustrated in Tab.
Fig. 2 illustrates detailed ynalysis of the distribution of task type and object sizes. As shown in the innermost ring of Fig. ( FINERS-4K provides instructions for three sub-tasks, including 1) Instruction-guided Segmentation (IS, 39%) that requires generating mask based on the instruction; 2) Multiple-choice VQA (MVQA, 30.5% that involves predicting both a segmentation mask and an option based on the option- given instruction; and 3) Open-ended VQA (OVQA, 30.5%) that requires producing both a mask along and a free-form answer. Each entity is bound to at least one instruction type. We further classify object size into three categories based on their proportion to the entire image area: small (S, >0.05%%) extra small (XS, 0.017%-0..055% ), and extra-extra small (XXS, 0.017%). The distribution of these object sizes across task types is shown in the second ring of F The outermost ring illustrates the types of attribute-specific instructions, including color, shape, position, and other distributions. Combining the three task types and four attribute types yields 12 distinct instruction-task combinations, with the numbers indicating the annotated instance count for each sub-task. Fig. (b) illustrates the visual examples of different tasks in FINERS-4K. More
detailed analysis about object size and spatial distribution can be found in Fig. Al
Methodology
4.1 Preliminary
Task Definition. Given an image I and a user instruction Q, MLLMS 42 are capable of jointly understanding visual and textual input to generate appropriate responses Apre, The objective of instruction-guiie guided segmentation [6] is to predict segmentation mask Mrre based on the image 7 and instruction Q. In contrast, our method unifies these two tasks into single framework, (I,Q) + (Apre, MPrr), enabling simultancous instruction-guidn guided segmentation, open-ended VQA, and multiple-choice VQA.
GRPO for Visual Perception. Visual-RFT [4] and Seg-Zero 16 extend the GRPO 13 framework to visual perception tasks by introducing task- specific rewards. Given an input image I and instruction Q. the model generates candidate predictions of the expected output, each of which is compared against ground-truth coordinates to compute individual rewards. GRPO then performs group-wist normalization over these rewards, guiding the model to favor perceptually accurate outputs, even in the absence of reasoning data during cold-start training. This approach enables more effective visual alignment and improves the generalization capability of MLLMs for object detection, classification, and referring segmentation.
5

---

Diffreet Coopped Syuatt with Randere Mtif
Paint Li
Fiack Format
Paint
Localized Parcephai Refinemmntt (LPR)
Joon Formet
Thisk Formet
Format
11
Raxdomly Crapped brutt
Refinement Sisae Marke
Conerato Portta
QSA Acc.
LPR Rloard
Box
Proven LPK
the parting lut
Think Format
Raglon
Irsl
Global Semartic
Jean-3 Format
Region
GGlahel far.
Orgiial lepus
Gonorate Caurse Region
Reglow
(e) Training Pipeline
Ragion
Bos
Eeploration (668))
Ieput
Localized Perceptual Frfermennt (1P9)
SAM
Gorowatte Caurse (Ragher
Refived BSg and Potets
Original HR Iepurt
Mork of Target Object
(b) Inference Pipeline
Reward Functions
Figure 3: Framework overview FINERS. (a) During training, we design specific reward functions to train GSE and LPR, where the LPR is optimized first and adopted to form retrospective reward to enhance the coarse region accuracy of GSE. (b) During inference, GSE takes a high-resolution image and user instructions input and produces
answer and coarse region containing referred objects. Then, LPR processes the instruction and coarse region to generate the object box and adopts SAM2 [8] to generate the final mask. (c) To unify VQA and segmentation into single MLLM, we design a multi-task reward pool and assign the items to supervise GSE and LPR.
4.2 Overall Framework of FINERS
The overall framework of FINERS is illustrated in Fig. 3 It consists of two-stage MLLM pipeline, comprising Global Semantic Exploration (GSE) and Localized Perceptual Refinement (LPR). Unlike prior high-resolution MILMs IO 21] that rely complex search strategies across multiple cropped image patches, our framework is designed to produce both textual responses and object segmentation masks through single feedforward pass. As shown in Fig. 3 (a). during training, GSE and L.R are optimized independently with specially designed rewards to facilitate perception at different levels of granularity. We introduce iocate-nformmed retrospective reward. which leverages LPR to select a robust coarse region, enhancing the global exploration precision of GSE. The detailed reward formulations for both modules are shown in Fig. F During inference, as shown in Fig. (b), only the original high-resolution image and user instruction are fed into GSE, which directly outputs the coarse box Br and the text response A The GSE stage can be formulated
(Brre APTC) = ( (T,0sa((,Q)) (T)Q)),
where represents post-processing function to extract the keywords of the long response. To constrain the search space of Bere, we set a fixed box size to optimize only the center offsets. The coarse box indicates an enlarged region around the referred object, which will subsequently be passed to LPR for object localization and mask generation. In the second stage, LPR first crops the original image based on Brre to obtain lower-resolution input Lc, and performs local reasoning to generate bounding boxes B and points pire, Ppre. The LPR stage can be formulated as:
(Bure, pare, Pp" = (fLPpa TT,, Q))
Finally, the generated boxes and points are passed to the frozen SAM2 mask Mre for the target object.
8 to produce the segmentation
4.3 Training Pipeline
The comparison in Tab. Jillustrats that the number of samples in existing high-resolution benchmarks substantially lower than that of standard-resolution datasets. To avoid data overload of supervised fine-tuning, we exploit recent vision reinforcement learning strategy to enhance the reasoning capacities of two-stage MLLMs. The training process is illustrated in Fig. (a). Specifically, we train our FINERS model with the GRPO algorithm in two stages. First, the coarse- -rrsineg LPR module is trained with LPR rewards to generate the object box from a local image crop. Then, the GSE module is optimized using combination of GSE rewards and locate-informel retrospective reward guided by the L.R module, addition, a standard KI. divergence penalty is applied between the policy and the reference model.

---

Refinement Stage for Masks. The supervised annotations for LPR training include bounding boxes 8st points Pt and P2 and text responses A all derived from the masks and answer annotations provided in training data. The <think> process originates from a prompt-based cold start and is supervised using the <think> format. Notably, since LPR is designed to focus on fine-grained perception within lower-resolution regions, is trained on image patches that are randomly cropped around the ground-truth bounding boxes.
Coarse Region Selection with LPR. Unlike LPR, the coarse region of GSE lacks explicit ground truth supervision for reward calculation. To address this, we design a locate-informed retrospective reward that uses the outputs of LPR provide robust coarse region supervision for GSE. Specifically, for each training sample, we first generate n randomly offset coarse regions that cover the GT bounding box 1 We then compute the IoU score between the LPR-predicted boxes Bere and the ground-truti boxes, selecting the region with the highest IoU as the GT regions / for training the GSE module.
@ Global Stage for Coarse Region. Due to the complex scenes in high-resolution images, which make it challenging for the model focus on small targets, the GSE model is designed generate approximate regions where small targets are likely to exist, based on the instruction context. Therefore, the training data for GSE consists of high-resolution images annotated with the optimal regions By selected by LPR and corresponding ground-truth answer labels A9t.
4.4 Reward Functions
Inspired by the reward functions of the GRPO strategy, design distinct reward functions at different levels of granularity for the LPR and GSE modules.
Rewards for LPR. The rewards for LPR include Point LI Rpoint, Box LI REEE, Box IoU Rytou, ISON-1 format Rformetl. Think format Rehink. and Q&A Accuracy Ryesponee. Among these functions, Rpoints RYE1, and RytOW are computed based on the predicted boxes Bure, predicted points pore, GT boxes Bdd and GT points Pst, to enforce spatial alignment between predictions and ground truth annotations. The JSON-1 format reward R formatt is only considered correct if the model outputs exact keywords (bbox, points I, points 2, response) ) in the required structure. The response reward Rresponse for the final response A is defined as:
{
1
if Apre is True, if Aure is False,
Rresponse
(3)
where the criteria for determining whether Arre is correct vary across task settings. For instruction- guided segmentation, the response considered correct it includes phrases like "is detected" or "is found". In the multiple-choice VQA setting, the response is correct if it exactly matches the ground-truth option. In the open-ended VQA setting, the response is deemed correct the fuzzy matching similarity to the ground-truth answer exceeds 0.8. The final reward of LPR is computed as:
RLPR- RRSOU RoLi + Rpoint Rfoomat1 + Rresponee + Rehink-
(4)
Rewards for GSE. Unlike LPR, the reward functions for GSE are designed to encourage alignment between the predicted coarse region Bere and the ground-truth region Bet selected by LPR. Specif- ically, we name this reward as locate-informed retrospective reward that consists of region IoU reward Rregionted and a region LI Ryeegiont1 between the predicted and GT regions Brre, D3 Since this stage focuses solely on contextual regions rather than reward is omitted, and the output JSON format is updated to a new template as (region, response). }. Additionally, to ensure that the predicted regions are compatible with the input size expected by the LPR module and sufficiently cover the target object, we introduce a region size Reize reward and a box-in-region Rooeer reward. They encourage the model to generate regions of appropriate size and position, aligned with those used during LPR training. The think format and Q&A accuracy rewards kept consistent with those used in LPR. The final reward of GSE is computed
localization, the point-level
RGSE Roegionnov + Bregion'1 + Rusze + Reover + Rfformat RRessouse + Rehink- (5)
7

---

Table 2: Performance comparison the test set of FINERS-4K, "t" indicates that the corresponding method is retrained with our dataset. We label the best results with a bold style.
ToU (gloUictoU) xx5
Acc. (MVQAVVVQA)
Method
AR
Color
Shope
Others
Position
AR
Training-free
LISA 7B X
19. 1/6.49 8.28/1120 20 4.19/0.34 34
9.002.33 38
0.006.111
0.000.000
0.0099337
0.00/16.7
0.00/5.31
LISA 13B
X
16.4/3.86 7,02/0.73 2.55/0.18 7.29/1142
0.00/6:46
0.0066.55
0.0099.37
0.00/5.55 0.0066.58
LISA+ 78
17
25. 25.9/12.3 13,5/2.90 3,700.70
12.35 20
5.909.779
0.82/1.66 63
35,7/6 24
26. 26.3/5.55
6.72/8. 19
PixeILM
13.6/6.70 3.30/1.10
0.500.10
4.40/2110
0.0044.56
0.82/0.09
3,57/9.31
0.09/5.55
0.27/4.05
SEAL g
7.46/3.14 8.26/0.03
7.14/15.6
5,26/16.7 7.53/3.49
DC [T]
39.0/19.5 36.3/9000
57.1/21.8
36.8/16.6
39.2/17.8
MI..MMs- Knoow
52.8/50.2 55.5/32.0
60.7/34,4
36.8/556
51.5145.4
MLLMs-Know 138
0
56. 1/540 36.4/32.0 67.9/37.5
2/,6/ 52.6448.8
MLLMs-Know 7B 0
LISA TB 3
21.2/19.2 15.07795
11.3/455
14.9/12.5
52.8/5.22 45. 5/..0 60,7/34.4
36,8/556 51.5/45.4
MLLMs Know 138
0
LISA 153
27.0/20.3 18.3/9.29 12.4/3.62 17.9/12.8
56.1/54.0 36.4/32.0
67.9/37.5
26.7/16.7
52.6/48.8
Zero 78
L+
55.9/18.6 34.7/4.94
16.5/0884
32.1/6.61
Training
LISA!
13.09.62 15.0/114
8.78/5.59 12.1/9.64
23.9/24.3
24.8/18.6
25.0662
21.100.0
23.9/22.0
PixelLM
1,27/1.02 0.52/0.35 0,08/0022 0.16/
0.0000
0.000
0.0.0.0
0,00000
0.0/0.0
MI..Mss-now 78 0
LISA! 78 B
1.32/1.16 2.08/1.90
277 2.85/3.15
52.8/50,2 45.5/32.0
60.7/34,4
36.8/5.56
51.5145,4
Seg-zenn ner- 7B T6
61.8/50.5
53.0/30.2 31,7/20.7 46.6/38.6
Ours (FINERS)
62.2/52.6
59,043311
47.2/27.5
55.1/46.5
85.8/60.5
76.0/49.2
78.6/444
63.2/27.8
83.3/56.7
5
Experiments
5.1
Experimental Settings
Implementation Details. Our two-stage MLLMs are built upon Qwen2.5-VL-7B 22. with input resolution of 1920 x 1080 for GSE and 512 x 512 for LPR. The output coarse region of GSE is 256 x 256, which will be 2x upsampled before being fed into LPR. The whole model is trained on a 4x A800 GPU (80G) setup using the Seg-Zero [16] and DeepSpeed [12] library. During training, the GSE module uses total batch size of 16 with samples per training step, while the LPR module uses total batch size of 32, also with 8 samples per step. For both stages, the initial learning rate is set to le-6 and the weight decay is 0.01. addition, we adopt SAM2 [8] for box-to-mask generation, which is kept frozen during training. The user prompts for GSE and LPR across three tasks are A4
presented in Fig. A3 and Fig.
Evaluation Metrics. Following previous works 226 27]. we calculate gloU and cloU for instruction- guided segmentation. The gloU is the average of all per-image Intersection Unions (IoUs), while the cloU calculates the cumulative intersection over the cumulative union. We evaluate cloU and gloU metrics across different object sizes. In addition, we calculate the accuracy of multiple- choice (MVQA) and open-ended (OVQA) visual question answering using option accuracy and the Suaffib,SequenceMaaccerr algorithm, with matching threshold set to 0.8.
5.2 Comparison with State-of-the-art Methods
We evaluate FINERS and other methods instruction-guided segmentation, open-ended VQA, and multiple-choice VQA.
FINERS dataset across three tasks, including
0 Comparison on FINERS-4K. The comparison results on the test and validation sets of our FINERS- 4k are illustrated Tab. 2 and Tab. AT respectively. We report both training-fres approaches and selected retrained methods in our dataset. As shown, our model consistently outperforms of the-art segmentation approaches and high- computed on all samples from the three instruction types, while QA accuracy is calculated only on the samples of MVQA and OVQA. The performance drop of LISA B and PixcILM 19] likely arises from the domain shift and task complexity of FINERS 4-k, which contains 4K UAV imagery with Altra-small, sparse objects. All baselines are fine-tuned on FINERS -4k under the same settings. In this low-data. high-resolution regime, fixed-architecture models tend to overfit underfit.
VQA methods. Note that the IoU scores are
We also conduct comparison with other approaches on public high-resolution VQA datasets, including V* O and HR-Bench [IO]. Compared with our benchmark, these datasets are captured in a general, UVA perception with optional VAQ annotations. As shown in Tab. 3 without an additional finetuning process, model achieves
Comparison on Other VQA Datasets.
8

---

Table 3: Performance comparison other high-resolution VQA benchmarks. "Att."" and "Spat." denote attribute and spatial, respectively. We label the best methods with a bold style.
V
IR Bench 4K
HR Bench SK
Method
Segmentation
Artr
Spot.
Overall
FSP
FCP
Avg.
FSP
FCP
Avg.
SEAL 9
X
74.8
76.3
75.4
47.0
29.3
38.1
42.5
28.8
35.6
Milmms-nnow
11
X
62.3
52.4
30.2
41.3
47.2
30.7
38.9
VILA-HD-L.5K
68.1
VILA-HD-44K 2X
x
71.2
DC (T)
X
57.3
53.0
47.0
50.0
37.2
44.2
40.8
Ours (FINERS)
76.5
79,0
77.5
66.4
61.2
63.8
60.2
55,9
58,1
What the shape the trash placed the overpass?
OVGA
<thine> 1.Identiffy the overpass the image. 2.Locate the trash can on the overpass.
67L
3. Determine the shape of the trash can, 4.Crop 258x256 bounding box around the cylinder. 6.Output the bounding shape. appears cylinder. </think>
<arswar>> ( "bbox"" |808,,55 1064,1023) "resporss": "ylindrriall"" </answer>>
What the person wearing who wearing
MVQA
White Blue Red (D) Green.
<think> identify person wearing
image. person
truck, they appear wearing white shirt. Check the color of shirt. The shirt is white. 3. Compare the shirt color with the given options: (A) Whits, (8) Blue, (C) Red, (D) Green. 4. The shirt color matches aption (A) White, The relevant region to focus on would be around the person near the track, specifically the upper part image the person standing. </think>
<anower> ('bbox": (880,384, "response': "A') c/onswers
The green sign side large indicator board.
othinès 1. Identify the large indicator board on the right side of the image. 2. Locate the green sign on the left side of the large Indicator board. Determine the reglon around green sign. Crop
bounding box around the green sign. bounding box. </think> <answer> (" "bbbx"" (1600,818,18888, 1858, "response": "The object is found." "j/aas))<//sswerr>>
Check object (grean sign) is found
Figure 4: Visual results on Open-ended VQA (OVQA), Multiple-choice VQA (MVQA), and Instruction-guide guided Segmentation (IS). The listed images are sampled from FINERS-44 test set.
significantly better accuracy in nvn-UVA scenarios from 4k 8k resolutions. Note that SEAL is the model proposed in the V* benchmark.
@ Visualization Results. Representative visual results for all three tasks are provided in Fig. As shown, the proposed two-stage framework is effective in locating tiny objects from a cluttered background and generating accurate open-ended or optional answers from the textual instructions.
5.3 Ablation Study
Tab. illlssrates the evaluation results by separately removing the key components from the whole framework. To validate the efficacy of retrospective reward, we replace the LPR-informed coarse regions with a random crop centered around the GT object to supervise the GSE for coarse box generation. The result of "woo Restrospective Reward" demonstrates the effectiveness of this design. To mitigate the sensitivity of LPR to input box variations, we apply a box augmentation strategy during LPR training. The improvement observed in "w/o Random Input Region in LPR" verifies the efficacy of this data augmentation strategy. In contrast to [16), our FINERS can generate exact text answers to user questions, which requires the incorporation of a QA accuracy reward during training. The performance drop in "w/o QA Acc. Reward" supports the importance of this component. More detailed analysis on this reward can be found Tab. A2 In GSE, we use extra box rewards to facilitate the optimization of coarse region generation. The ablation results of "w/o Box Size Reward" and "w/o Bex-in-region Reward" demonstrate the positive impact of these rewards on model performance.

---

Table 4: Ablation studies about the proposed components FINERS-45
Different Settings
Test sel
Val set
glot/
clol/
MVQ4
OVQ4
gloU;
cloUj
MVQ4
OV24
FINERS
55.1
46.3
83.1
56.7
49.9
39.4
87.2
60.0
Retrospective Reward
54.0
44.0
82.3
53.0
49.4
38.0
86.2
55.8
woo Random Ieput Region in LPR
53.9
46.7
83.7
61.9
48.7
39.4
84.8
59.4
QA Acc. Reward
33.7
w/o Box Size Reward
51.0
43.4
56.5
40.8
44.5
35.8
78.4
56.4
wio Box in region Reward
50.1
42.0
56.5
40.8
44.1
34,2
77.9
55.7
6 Discussion
In this paper, we aim to resolve the fine-grained reasoning and segmentation of ultra-small objects in high-resolution images. We introduce 1) FINERS, a two-stage MLLM-based reinforcement learning framework that combines global semantic exploration with localized perceptual refinement; and 2) FINERS-4K, a new dataset featuring challenging scenes annotated with text-mask pairs across three types of tasks. Extensive experiments on FINERS -4k and other public benchmarks demonstrate the superiority of the proposed method in both answering aceuracy and segmentation precision.
Although our method made an early attempt to apply reinforcement learning for joint object reasoning and segmentation, several limitations remain in the current architecture. First, the localization accuracy of LPR is highly dependent on the output coarse region of GSE. The text answer and object additional signal
box will be incorrect if the GT object exceeds the coarse region. Incorporating for object existence and enabling re-exploration could help mitigate such missed detection errors. Second, due memory constraints, the two-stage models are not jointly optimized during training. Designing more efficient architecture for jointly end-to-end training of two stages remains an important future direction.
Acknowledgment
This work was supported by 2024-0011 (ZX20240867).
References
[1] Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, and Filip llievski. Milms know where look: Training-fri free perception of small visual details with multimodal llms. arXiv preprint arXiv:2502.17422. 2025.
[2] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arXiv preprint arXiv:2412.1511 2024.
[3] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579-95889 2024.
[4] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec. Visual instruction tuning. Advonces in neural information processing systems, 36:34892 34916 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et Llaaa Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
[6] Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuaa. Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
[7] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages
2023.
[8] Nikhila Ravi, Valentin Gabeur, Yuan- Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma. Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofeer Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.000 007044 2024.
10

---

[9] Penghao Wu and Saining Xie. V*; Guided visual search as core mechanism in multimodal 1lms. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13084- 13094, 2024.
[10] Wenbin Wang, Liang Ding. Minyan Zeng. Xiabin Zhou, Li Shen, Yong Luo, Wei Yu, and Dacheng Tao. Divide, conquer and combine: A training-free free framework for high ressuresoottion image perception in multimodal large language models. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 39, pages 7907-7915, 2025.
[11] Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Hel- yar, Alcksander Madry, Alex Beutel, Alex Camey, et Openai system card. arXiv preprint arXiv:2412. 16720, 2024.
[12] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepscek-rl: Incentivizing reasoning capability in lims via reinforcement learning. arXiv preprint arXiv:2501. 12948 2025.
[13] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, al. Deepseckmath: Pushing the limits of mathematical reasoning in open language models, arXiv preprint arXiv:2402.03300 2024.
[14] Ziyu Liu, Zeyi Sun, Yuhang 7ang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.0/78. 2025.
[15] Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.000 06749,, 2025.
[16] Yuqi Liu, Bohao Peng, Zhisheng Zhong, Zihao Yue, Fanbin Laı Bei Yu, and Jiaya Jia. Seg-zero: Reasoning- chain guided segmentation via cognitive reinforcement. arXiv preprint arXiv:2503.06522 2025.
[17] Senqiao Yang, Tianyuan Qu, Xin Lai, Zhuotao Tian, Bohao Peng, Shu Liu, and Jiaya Jia. Lisa++: An improved baseline for reasoning segmentation with large language model. arXiv preprint arXiv:2312.172200 2023.
[18] Zechen Bai, Tong He, Haiyang Mei, Pichao Wang, Ziteng Gao, Joya Chen, Zheng Zhang, and Mike Zheng Shou. One token seg them all: Language instructed reasoning segmentation in videos. Advances Neural Information Processing Systems, 3775893-6859 2024.
[19] Zhongwei Ren, Zhicheng Huang. Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. In Proceedings of the IEEESCVF Conference on Computer Vision and Puttern Recognition, pages 26374-26383, 2024.
[20] Junhua Mao, Jonathan Huang, Alexander Toshey, Oana Camburu, Alan I. Yuille, and Kevin Murphy. Generation and comprchension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11 11-20, 2016.
[21] Haozhan Shen, Kangjia Zhao, Tiancheng Zhao, Ruochen Xu, Zilun Zhang, Mingwei Zhu, and Jianwei Yin. Zoomeye: Enhancing multimodal Ilms with human-like zooming capabilities through tree-based image exploration. arXiv preprint arXiv:241 1. 160044 2024.
[22] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lee. Improved baselines with visual instruction wuingg In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296-26306, 2024.
[23] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual- linguitic tasks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 24185 24198, 2024.
[24] Yuji Cao. Huan Zhao, Yuheng Cheng, Ting Shu, Yue Chen, Guolong Liu, Gaoqi Liang, Junhua Zhao, Jinyue Yan, and Yun Li. Survey on large language model-enhanced reinforcement learning: Concept. taxonomy, and methods. JEEE Transactions on Neural Networks and Learning Systems, 2024,
[25] Shangding Gu, Long Yang. Yali Du, Guang Chen, Florian Walter, Jun Wang. and Alois Knoll. A review of safe reinforcement learning: Methods, theories and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
11

---

[26] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th Luropean Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 14, pages 69-85. Springer, 2016.
[27] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring objects in photographs of natural scenes. Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787-798, 2014.
[28] Baifeng Shi, Boyi Han Cai, Yao Lu, Sifei Liu, Marco Pavone, Kautz, Song Han, Trevor Darrell, Pavlo Molchanov, et al. Scaling vision pre- pre-training to 4k resolution. arXiv preprint vrXvv:2503.19033 2025.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The motivations and contributions of this paper is highly summarized in our abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
We discussed the limitations in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
13

---

Justification:
This paper focuses on the application of MLLMs without theoretical compo-
nents.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have included the implementation details and will provide the code, models, and dataset once publication.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
14

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We promise to release the code and data after acceptance.
Guidelines:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
7 //nips. htcp//blcc/guddeeeeeeeeeeeeeeeee
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We clarify the experiment details in the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We have discussed the metrics in experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
15

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes)
Justification:
We have claimed the used computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have checked it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no socictal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [Yes]
Justification: Our data comes from Internet videos or videos shot by ourselves, and there is no copyright dispute. We mosaic the sensitive data to prevent privacy leakage.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The baseline code used are open-source from Inertnet.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.paaersstt
package should be provided. For popular datasets, paperswi thcode. cow/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have processed the data avoid this issue.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
18

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: Our method used MLLMs baseline for image reasoning and segmentation.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neur// htttg////berrreeccc////ffeererrs cc///oofeereencc//22///.... for what should or should not be described.
19

---

A
Appendix
In this section, we provide additional figures and tables of the analysis on the proposed method and benchmark.
A.1 More Analysis on FINERS-4k
In this paper, introduce new dataset, FINERS-4K which consists of high-resolution images containing ultra-small objects with diverse spatial distributions. Fig. [T illustrates the detailed distribution of object sizes and spatial locations across all samples in the training, validation, and test sets, As shown, our dataset exhibits more challenging scenarios with extra-small objects and sparse distributions.
198
Truat
1988
Midalle
Middle 49,4%
Middlk 52.1%
53.2%
Barder
Bartler
Center 28.0%
24.3% 23.6%
Rore
(0000,
A01)
OATST
8I1
10
Sample Distribution Across Mask Ratios
(b) Image region division
(0) Propertions mask locations region
Figure AI: (a) Distribution of mask sizes across three subsets, where the x-axis indicates the ratio of the mask area to the entire image. (b) Spatial division of the image into three regions center, middle, and border-based on the location of the mask. (c) Distribution mask locations across three subsets.
A.2 Rewards Definition Details
In Eq. and Eq. we introduce several rewards for LPR and GSE modules. Specifically, the detailed definitions of each rewards in Eq. 4 are:
Rpoint = LI distance between predicted point and GT point is less than 100 pixels.
RyEs if the LI distance between the predicted box and GT box less than 10 pixels.
RyJow = 1 if their IoU is greater than 0.5.
Rresponse: is set to 1 if the predicted answer is correct (exact match for multiple choice, fuzzy match for open-ended QA).
Rformat and Rehink binary rewards that verify whether the output adheres to the expected JSON and reasoning formats.
In Eq. 5
Rregion!1 1 if the 1.1 distance between the predicted coarse box Bere and GT region By is less than 10 pixels.
Rregion for if their IoU is greater than 0.5; Ruise is when the predicted coarse box is of size 512 x 512.
Reover is when the ground-trute object lies fully inside the predicted region.
We assign equal weights all binary terms, following standard GRPO practices 16
14
which
yielded stable performance without tuning.
A.3 More Comparison Results
Tab. AT illustrates the comparison results the validation set of FINERS-4k. While method performs slightly lower than SegZero on small-sized objects, it significantly outperforms SegZero on xs and xxs objects. More importantly, method supports VQA tasks and outperforms other approaches this setting. Additional qualitative results produced by our method are shown in Fig. A2
20

---

Table A1: Performance comparison the validation set of FINERS 4k. "I" indicates that the corresponding method is retrained with our dataset. We label the best results with a bold style.
IoU (gloU//llo) 1X5
QA Ace. (Oplion/Open)
Method
x
All
Color
Shape
Others
Position
All
Truining- frre
LISA vis
14.3/291 6.40L .6 3,54/0,36 6. 6.58/1.65 66
0.00/1511 0.0010.00 0.000000
0.00/16.7
0.0011 1.
LISA
12.1/2.42 5.10/.588
1.21/0.14 4.28/1.
0.00/19.4
0,002.222
0.000.00
0.00/16.7 0.00/15.2
LISA+ 78
25.077. 10 8 80/2110
2,3000600
8.91/3.22
4.86/16,7 4.86/0.00
4.0077800 8.33/16.7
5.99/13.2
PixelLM 7B
19
10.5/3.31 3,30/100
0.42/0.13 3.31/1600
0.00/3.78
0.00/2.22
0.000..00 0.00/0.00 0.00/3.21
SEAL 5
2.16/9.68
6,98A1.00
0.0000.00
8.33/33.3 3.207.80
DC [T0
34.6/21.0 25.6/6.67
40.00.00
33.3/16.7 33.2/17.2
KnoKKKKwWTTT 78
46.5/50.0 2/37/
41.7/50.0 47.2/47.2
MLLMs- 130
50.3/51.1
32.6/26.7
80.023.1
91 91.766.7 50,4/45,6
MLLMs-Koow 7B
LISA 7B X
17.1/112 8.94/464
8.982.78
10:47.07
46.5/50.0
44.2/37..8 80.038.5 41.7/50.0 47.2/47.2
MLLMs- 13B
LIIA 138
23.2/16.0 13,5/6.01 9.57/2.55
13.6/9.16
50.3/51.1 32.6/26.7
80,03311
91.7/66.7 50.4/4.66
Seg zero 78
16
56.5/24.6 28.0/3.75
13.8/1.41
27.077.9
Training
LISA!
14.0/10.8 9.92/7.70
7.27/4,52
9.50/8.6 62
4.80/16.7
2.32/0.00
99.99976688.333 39/.13/16.7
5.99/3.2
PixelLMM 78
1.27/1.02 0.52/0.35 0.08/0.02 0.16/0.13
0.0000
0.000.
0.0N.00
0.000.0
0.0/9.0
MI.LMs. Know 78
LISA!
1.10/0.22 1,44/140
1,77/1884
1.52/1 05
46.5/50.0 44.2/37.8
80.038.5
41.7/50.0 47.2/47.2
Seg-zero" 78 16
67,2/338 45.8/15.8 30.2/16.9 42.9/31.3
Ours (FINERS)
64.8/55.3 50.8/27.1
42.5/21.0 49.7/38.6
85,4/65,6 48467
91.7/66.7
N6.8/60.4
What is shape ornament on top domed building?
OVQA
<think> 1. Identify the domed building in the image. 2. Locete the ornament on top of the damed building. 3. Determine the shape the ornament. 4. Crop 256x256
region around ornament. Check ornament star Output region and the word "The ornament star." </thinks
canswer> ("bbox": [5040,,61,255), "response": "star." </answer>>
Trampoline the yard
I5
<thini 1. Identify the trampoline in the yard. 2. Deternine the bounding box around the trampoline. 3. Crop a 256x256 region around the trampoline. Check If the trampoline is present in the cropped region. Output the bounding box and the result. The trampoline is located in the yard of a house on the right side of the image. near the center-right of the image. </think>
<answar> ("bbox": (952,620,1225) 1216,877) "response" "The object found. ")</answerr>
What the shape the clock the building the left side of the picture? (A) Square, (B) Girde, (C) Triangle, Rectangle.
<think 1. Identify the building on the left side of the picture. It appears to be a clock tower: 2. Observe the shape of the clack on this building. The clock face circular. 3. Check the options provided: (A) Square, (B) Circle, (C) Triangle, (D) Rectangle. 4. Determine which shape best matches the clock face on the building. The clock face on the building clearly circular. </think>
<answer> ("bbox" (76,488,332,73888 "response": </anewer>
Figure A2: More visual results on Open- VQA (OVQA), Multiple-choice VQA (MVQA), and Instruction-guidee Segmentation (IS). The listed images sampled from FINERS-44 4k test set.
A.4 More Ablation Studies
QA Rewards in Different Stages. Tab. A2 illustrates the ablation results on the effectiveness of QA accuracy reward and the two-stage designs. To evaluate the efficacy of our two-stage design, we remove the LPR stage and modify the GSE to directly generate object bounding boxes (i.e., H in E This baseline, "One Stage _xx", can be seen as higher-resolution version of SegZero [16]. From the comparison results, we can observe that the proposed two-stage coarse-to-fine framework is significantly effective in improving the localization precision of small objects. We also verify the effect of QA accuracy reward GSE and LPR. The comparison results between "GSE w/o QA Acc. & LPR w/o QA Acc." and "FINERS" demonstrate that the incorporation of additional VQA ability can effectively enhance the segmentation accuracy on small objects. Besides, the comparison between "GSE w/ QA Acc. & LPR w/o QA Acc." and "GSE w/o QA Acc. & LPR w/ QA Acc."
21

---

demonstrates that the textual response on GSE shows better performance due to the exploration of global visual information.
Table A2: Ablation study about QA accuracy reward.
Test sel
Val set
Method
gloU
Option
Open
gloUj
Option
Open
One Stage w/o QA Acc.
46.6
38.6
41.6
31.3
One Stage with (AA Acc.
42.3
41.8
84.4
58.2
40.1
37.6
88.5
63.6
GSE w/o Acc. & LPR wlo Q4 Acc.
52.8
45.7
48.6
37.7
GSE w/ QA Acc. & LPR w/o Q4 Ace.
53.7
46.2
83.3
56.7
49.2
38.2
87.2
60.0
GSE w/o
& LPR QA
54.4
56.2
39.1
38.2
55.6
40.4
FINERS
55.1
46.5
83.3
56.7
49.9
39.4
87.2
60.0
Effects Hypeer-prameetes. In periments, all aaper-parameters are set to the default values in (16) without specific tuning. We conduct a sensitivity analysis on key sypeerparametest including group- ing n, KL weight, and seeds. Tab, A3 illus- trates the representative pyper-paramettr results on SINERS-44 test set, which are consistent with SegZero. For fair compar- ison, we did not perform expensive hyper- parameter tuning.
Table A3:
sensitivity analysis on FIN-
ERS 4k test set.
Hyper-parameters
gloUj
clo
Option
Open
Group 8
55.1
46.5
83,3
56,7
Group 6
54.3
43.7
82,6
53.5
Group
Se-3
55.1
46.5
83.3
56.7
KL 5o-2
54.4
46.1
82.9
55.8
Seed
55.1
46.5
83.3
56.7
Seed 48
53.9
45.4
82.0
56.2
55.7
Efficiency Comparison. We report the average wall-clock inference time for 4k-resolution inputs in Tab. A4 All models were evaluated on a single A100 GPU with consistent runtime environments. As shown, compared to SEAL [9] and DC2 (10), our method and Seg-Zero [16] exhibit higher inference latency due the use of CoT reasoning. Despite our two-stage framework requires extra inference time, this design is essential for achieving precise reasoning and segmentation of ultra-small objects in high- high-resolution scenes.
Table A4: Inference latency and performance of different methods.
FINERS 4k test set
IRR hench 4k
Method
glotikllot/rvQQAOOOQAA
(s/img)
Q4 Ace.
Time (s/ing)
SEAL 78
-1-77.3/3.49
1.21
38.1
1.15
DC
7B
M0
--/39.2/17.88
2.69
50.0
2.90
ZogZero
70 16
46.4/38.6/-/-
8.67
Ours (SINEKS) 7B
55. 35.1146.//3.33667
7.31 (GSE) 3.57 (LPR&SAME)
63.8
6.35 (GSEonly)
Domaín Generalization on ReasonSeg 33] We evaluate our model on ReasonSeg in zero-shot manner to verify its generalization on non-UVA scenarios. ReasonSeg contains ground-level daily scenes with moderate-tee tosttt--t--aaeee to-large object sizes, significantly different from aerial, ultra-high weae-hihh--esoous resolution imagery and ultra-small objects focus of FINERS-4K, -4k. We categorize test samples by mask-to-image area ratio into Large 50%), Middle (10-50%) ), and Small 10%). Notably, ReasonSeg's "Small" objects are still much larger than FineRS 4k'ss (<1%), introducing a challenging domain and scale gap. Despite this, as shown in Tab, A5 our two-stage model achieves better gIoU on Small objects, outperforming finetuned baselines like LISA and Seg-Zerof, This confirms the effectiveness of our coarse-to-Bi fine strategy in segmenting small targets, even under mismatched resolution and context. Moreover, we find that our LPR shows strong adaptability across all sizes and outperforms other methods even without domain-specific finetuning, highlighting its robustness.
Results on Other Baselines. We evaluate our two-stage framework based on Qwen-3b [2] and the results are reported in Tab, A6 As observed, employing a smaller backbone leads to noticeable performance degradation, particularly on small objects. Nevertheless, our method still surpasses SegZero [16] under the same backbone configuration, demonstrating its superior adaptability and robustness.
22

---

Table A5: Performance the ReasonSeg test set (loU: gloU / cloU). Results grouped by object size. Notely, "f" denotes the corresponding methods are finetuned with FineRS-4k without pretraining on large-scale referring segmentation datasets. "Resize" means directly resize ReasonSeg image to meet our model's require (1920 x 1080). "Padding" means that the low-resolution image are padded to meet our model's resolution.
Method
Large (316 samples)
Middle (388 samples)
Small (72 samples)
ALI.
Ours (PINERS) (Resine)
25.2/4.92
41.4/156
42.3/7.12
35.0/7.16
Ours (FINERS) 7B (Podling)
27.5 5.00
43.2 16.1
40.0/9.42
36.1 7.52
LPR only
SegZero
49.3/413
46.1 /3099
35.8/ 15.4
47.1/38.8
SegZoro 70 16
65.3/555. 55.2
53.5
39.0 16.0
57.5/
L.8A 78
0.44/0.34
4,08
8.14/1.12
2.98/0.55
LISA
55.3/561
34.2/26.9
20.3/ 24.8
48.7/48.8
Table A6: Performance comparison the test of FINERS-4k using Qwen2.5-VL. (3b). "1" indicates that the corresponding method is retrained with our dataset.
IoU (gloUicloU)
QA Acc.
(Option/Open)
xS
XXS
AN
Color
Shape
Others
Position
All
Seg-zero 38 16
$7.7/45.3 47.4/22.8 26.3.0.86 41.6/29.5
Ours (FINERS)
57.3/4544 53.8/23.3 43,0/9.86 50.4/29.9
65.6/50.5 68.5/41.0 60.7/21.8
52.6/22.2 65.644..0
A.5 User Prompt for FINERS
Fig. A3 and Fig. Allllustee the user prompts of Global Semantic Exploration (GSE) and Localized Proceptual Refinement (LPR) modules across three tasks, including Instruction-guiie guided Segmentation (IS), Open-ended VQA (OVQA), and Multiple-choice VQA (MVQA).
Prompt for IS
Bassd on the 'QQuestion",, identify 256-256 bounding box that best localizes the region most relevant to the query.
And respond with whether the object is found."
Compare the difference between regions and find the most closely matched one.
Output the thinking process <\think> and final answer in <\answer> <lanswer> tags.
Output 256*256 region bbox
response inside interested object in JSON format.
<think> thinking process here <\think>>
<answer> ('bbox': min.yy_inn,x.mm 256. min) response: 'The object here.' ><\answer>"
Prompt for OVQA
Based the 'Queestinn', identify 256*256 bounding box that best localizes the region relevant query.
And give me final response with word phrase.
Compare the difference between regions and find the most closely matched one.""
Output the thinking process in <\think> and final answer in <\answer> <\answer> tags."
Output the 256 region bbox and the final response inside the interested object in JSON format."
<think> thinking process here <\think> "
<answer> bbox': Dx min.y w.y.miin,x.mmmm min+256,yyn......mmm min+2m6], response: 'The cat is white. ><aanswe>>"
Prompt for MVQA
Based on the 'QQuestion', identify a 256-256 bounding box that best localizes the region most relevant to the query. option from (Options).
And give me
Compare the difference between regions and find the most closely matched one.
Output the thinking process in <\think> and final answer <\answer> <lanswer> tags."
Output the 256 region bbox and the final response inside the interested object JSON format."
ie., <think> thinking process here <\think>
<answer> 'bbox' [x min.y min,x min+ 256,+256,y.om min+25m] 256), response: 'A' ><lanswer>*
Figure A3: User prompt for GSE module.
23

---

Prompt for IS
Please find '(Question'' with bbox and points. And respond with whether the target is found."
Compare the difference between objects and find the most closely matched one.
Output the thinking process <\think> and final answer in <\answer> <\answer> tags.
Output the one bbox and center points of two largest inscribed circles inside the interested object JSON format.
iee. <think> thinking process here <\think>
<answer>("bbox' 210,100.200,210], 'points 1': (30,110), 'points 2': [35,180), response: 'The object here. ><lanswer>"
Prompt for OVQA
Please find ''Question'' with bbox and points. And give final response with word phrase."
Compare the difference between objects and find the most closely matched
Output the thinking process <\think> and final answer in <\answer> <\answer> tags."
Output the bbox center points
largest inscribed circles inside the interested object JSON format.
iee. <think> thinking process here <\think>
<answer> bbox': [10,100,200,210, 'points 1' [30,110), 'points 2': [35,180], response: 'The cat is white.' ><lanswer>"
Prompt for MVQA
Please find (Question) with bbox and points. And give
option from (options)."
Compare the difference between objects and find the most closely matched one."
Output the thinking process in <\think and final answer in <\answer> <\answer> tags."
Output the one bbox and center points of two largest inscribed circles inside the interested object in JSON format."
ie., <think> thinking process here <\think>
<answer>
bbox': boeswer>bbbxx'.
100, "points 1": (30,110), 'points (35,180], response: ><aanswer>"
Figure A4: User prompt for LPR module.
24


Reviews:
[{"review_id": "gACUVmNMwZ", "reviewer": "Reviewer_kUFY", "summary": "The paper proposes FINERS, a two-stage multimodal large-language-model (MLLM) framework for instruction-guided reasoning and position of small objects in 4K images.\n\nIn the stage 1(GSE),  the model reads the HR image and instruction, outputs a textual answer and a fixed-size 256 × 256 coarse region.\nIn the stage 2(LPR), the model crops that region, predicts the position information, a precise box + two points, and calls SAM2 to obtain a pixel mask.\n\nThe stages are coupled with a locate-informed retrospective reward so that LPR’s IoU/L1 feedback reinforces GSE during vision reinforcement fine-tuning with GRPO.\n\nThe authors introduce a dataset, FINERS-4k, for evaluation.", "questions": "Robustness to coarse-box errors: How does FINERS detect or recover when GSE misses the target? Please quantify failure rates where the target object lies partially or fully outside the coarse region and evaluate simple fall-back strategies.\n\nEfficiency: Please report wall-clock inference time for 4K inputs, vs VILA-HD, Seg-zero and DC^2. That would strengthen practicality.\n\nDomain generalization: Evaluate FINERS on at least one non-UAV dataset containing small indoor objects. Comparable accuracy would indicate broader utility.", "limitations": "No.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "[+] Coarse-to-fine RL design with retrospective reward is conceptually elegant and empirically helpful.\n\n[+]  FINERS-4k fills a data gap for UAV, tiny-object HR tasks and could benefit the community.\n\n[-]  Entire pipeline hinges on GSE: if the target object straddles the coarse box, both answer and mask fail.\n\n[-]  Two stages are trained separately; no end-to-end fine-tuning —may be sub-optimal.\n\n[-]  Hyper-parameter sensitivity (group size n, KL weight) and seed robustness are not analyzed; reproducibility risk.", "was_revised": true, "final_justification": "The authors have addressed my concerns. I increase my score", "rebuttal": "We thank the reviewer for the constructive comments on our paper. Regarding the concerns of Reviewer kUFY, we provide the following responses.\n\n---\n**W1: Entire pipeline hinges on GSE: if the target object straddles the coarse box, both answer and mask fail.**\n\nThank you for the thoughtful observation. At first, we would like to clarify that the QA output is not dependent on the GSE’ s coarse box, as answers are generated from the full-resolution image and instruction before region cropping. Thus, **QA accuracy remains stable** even when the coarse region fails to cover the target object. \n\nHowever, the downstream segmentation dose rely on GSE. This is consistent with our **coarse-to-fine design**, inspired by human vision---first identifying rough regions before refining focus. Like humans, our system may fail if the initial focus region omits key information, particularly under ultra-small object conditions. To address this, we introduce a retrospective reward that leverages LPR feedback to improve GSE training, effectively raising hit rate from 76% to 77.7% (see Table R2 below). \n\nNevertheless, we recognize this is not a complete solution. Unlike humans, our model lacks fast re-evaluation or error correction once the region is fixed. In future work, we plan to integrate dynamic re-focusing mechanisms, such as iterative refinement or region ensemble strategies, to address this limitation more effectively.\n\n---\n\n**W2: Two stages are trained separately; no end-to-end fine-tuning —may be sub-optimal.**\n\nThank you for the insightful comment. Our current design adopts a two-stage training strategy to balance optimization complexity and modular flexibility. While GSE and LPR are trained separately, they are coupled via a retrospective reward during GRPO fine-tuning, enabling LPR feedback to improve GSE performance (**+1.1/2.5** in gIoU/cIoU of test set). We agree that full end-to-end optimization could enhance stage coherence, but doing so at 4k resolution presents challenges in memory and reward propagation. In future work, we aim to explore unified architectures or lightweight joint tuning to better integrate both stages.\n\n---\n\n**W3: Hyper-parameter sensitivity (group size n, KL weight) and seed robustness are not analyzed; reproducibility risk.**\n\nIn our experiments, all hyper-parameters are set to the default values in Seg-Zero [16] without specific tuning. We conduct a sensitivity analysis on key hyper-parameters, including group n, KL weight, and seeds. **Table R1** below shows that the hyper-parameter sensitivity of our method is consistent with Seg-Zero. Due to the limited rebuttal period, we present representative cases here and will include a more comprehensive analysis in the final version.\n\n**Table R1: Hyper-parameter sensitivity analysis on FineRS-4k test set.**\n\n| Hyper-params| gIoU | cIoU | MVQA| OVQA|\n|------------------|------|------|------|------|\n| Group 8  | 55.1 | 46.5 | 83.3 | 56.7 |\n| Group 6    | 54.3 | 43.7 | 82.6 | 53.5 |\n| KL 5e-3    | 55.1 | 46.5 | 83.3 | 56.7 |\n| KL 5e-2    | 54.4 | 46.1 | 82.9 |  55.8|      \n| Seed 48    |  53.9 | 45.4 | 82.0 | 56.2| \n\n---\n\n**Q1: Robustness to coarse-box errors: How does FINERS detect or recover when GSE misses the target? Please quantify failure rates where the target object lies partially or fully outside the coarse region and evaluate simple fall-back strategies.**\n\nAs clarified in W1, **QA outputs are not dependent on GSE**. However, **segmentation performance is sensitive to coarse-box accuracy**. When GSE misses the target, LPR segmentation degrades. To quantify this, we report a **22.3% failure rate** where the GT mask lies partially or fully outside the GSE region. Even so, FINERS still outperforms Seg-Zero by **+10.5 gIoU / +6.9 cIoU**. Besides, we also list two baselines: “one-stage” (can be seen as Seg-Zero with QA) and “w/o retrospective reward”, which further demonstrate the effectiveness of our two-stage design. \n\nBesides, we implement a simple fallback strategy: expanding GSE’s output box to $800\\times800$ pixels, which boosts coarse-box hit rate from 77.7% to 82.6%. However, the altered crop size introduces more context to LPR, slightly compromising final performance. In Table R2, we additionally report **upper-bound performance** using GT-based coarse boxes. These results confirm that our **two-stage design with retrospective reward** is effective, and we will explore more robust fallback mechanisms in future versions.\n\n**Table R2: Analysis on failure rate mitigation strategies on FineRS-4k test set.**\n\n| Method | gIoU  | cIoU  | Box Acc (%) |\n|-------------------------------|-------|-------|------------|\n| FineRS (GSE + LPR)            | 55.1  | 46.5  | 77.7      |\n| Seg-Zero$^\\dagger$            | 44.6  | 38.6  | 77.7     |\n| One-stage                   | 42.3  | 41.8  | --       |\n| w/o Retrospective reward  | 54.0  | 44.0  | 76.0     |\n| GSE Box Expand 800×800        | 54.5  | 46.2  | 82.6   |\n| GT Coarse Box + LPR      | 70.6 | 68.2 | 100  |\n\n---\n\n**Q2: Efficiency: Please report wall-clock inference time for 4K inputs, vs VILA-HD, Seg-zero and DC$^2$. That would strengthen practicality.**\n\nThanks for the insightful comment. We report the average wall-clock inference time for 4K-resolution inputs in **Table R3**. All models were evaluated on a single A100 GPU with consistent runtime environments. As shown, compared to SEAL[9] and DC2[10], our method and Seg-Zero[16] exhibit higher inference latency due to the use of CoT reasoning. Despite our two-stage framework requires extra inference time, this design is essential for achieving precise reasoning and segmentation of ultra-small objects in high-resolution scenes. Notably, the VILA-HD[28]'s results in Tab.3 are copied from its technical report, since its inference code is released after our submission. Due to the limited rebuttal time, we fail to implement their testing code and don't report its time here. We promise to complete this experiment and update the complete results in the next version.\n\n**Table R2: Inference latency and performance of different methods on FineRS-4k test set (left two columns) and HR-bench-4k (right two columns).**\n\n| Method  | MVQA/OVQA/gIoU/cIoU   | Time (s/img) | QA ACC.  | Time (s/img)  |\n|-----------------------------|-----------------------------------------|---------------------------|-----------------------|-------------------|\n| SEAL-7b [9]  | 7.53 / 3.49 /  ----  / ----   | 1.21   | 38.1   | 1.15 |\n| Seg-Zero$^\\dagger$-7b [16]  | ---- / ---- / 46.6 / 38.6   | 8.67   | --  | --   |\n| DC$^2$-7b [10]    | 39.2 / 17.8 / ---- / ----    | 2.69   | 50.0    | 2.90   |\n| **Ours-7b**   | **83.3 / 56.7** / **55.1 / 46.5**  | 7.31 (GSE) + 5.57 (LPR+SAM2) | **63.8**  | 6.35 (GSE only) |\n\n---\n\n**Q3: Domain generalization: Evaluate FINERS on at least one non-UAV dataset containing small indoor objects. Comparable accuracy would indicate broader utility.**\n\nThank you for your valuable suggestion. We have carefully reviewed the existing referring segmentation datasets but did not find an exact dataset specifically designed for non-UAV, high-resolution, and small indoor objects. To address this, we conduct **zero-shot evaluations** on ReasonSeg (proposed by LISA [3]) test set, which includes **ground-level daily scenes with moderate-to-large object sizes**, significantly different from **aerial, ultra-high-resolution imagery and ultra-small object** focus of FineRS-4k. \n\nWe categorize test samples by mask-to-image area ratio into Large ($>$50%), Middle (10–50%), and Small ($<$10%). Notably, **ReasonSeg's \"Small\" objects are still much larger than FineRS-4k’s (<1\\%), introducing a challenging domain and scale gap.** Despite this, our two-stage model achieves better gIoU on Small objects, outperforming finetuned baselines like LISA$^\\dagger$ and Seg-Zero$^\\dagger$. This confirms the effectiveness of our coarse-to-fine strategy in segmenting small targets, even under mismatched resolution and context. Moreover, we find that our LPR shows strong adaptability across all sizes and outperforms other methods even without domain-specific finetuning, highlighting its robustness. \n\nThese results indicate that FINERS, while designed for ultra-small aerial targets, retains strong generalization capability to conventional domains, especially when object granularity matters. We will include these findings in the final version.\n\n**Table R4: Performance on the ReasonSeg test set (IoU: gIoU / cIoU). Results are grouped by object size.**\n\n| Method    | Large (316 samples) | Middle (388 samples) | Small (72 samples) | All  |\n|------------------------------|---------------------|-----------------------|--------------------|-------------|\n| FineRS 7B (resize)   | 25.2 / 4.92         | 41.4 / 15.6           | **42.3** / 7.12        | 35.0 / 7.16 |\n| FineRS 7B (padding)    | 27.5 / 5.02         | 43.2 / 16.1           | **40.0** / 9.42        | 36.1 / 7.52 |\n| LPR only    | 59.0 / 52.4         | **54.4** / **35.3**           | **45.7** / **24.6**    | 56.6 / 42.1 |\n| Seg-Zero† 7B [16]  | 49.3 / 41.7         | 46.1 / 30.9           | 35.8 / 15.4        | 47.1 / 38.8 |\n| Seg-Zero 7B [16]  | 65.3 / 55.2         | 53.5 / 31.4           | 39.0 / 16.0        | 57.5 / 52.0 |\n| LISA† 7B [3]   | 0.44 / 0.34         | 4.08 / 1.38           | 8.14 / 1.12        | 2.98 / 0.55 |\n| LISA 7B [3]    | 55.3 / 56.7         | 34.2 / 26.9           | 20.3 / 24.8        | 48.7 / 48.8 |\n\n- *Note:* “†” denotes the corresponding methods are finetuned with FineRS-4k without pretraining on large-scale referring segmentation datasets. \n- “resize” means we directly resize ReasonSeg image to meet our model’s require ($1920\\times1080$). “padding” means that the low-resolution image are padded to meet our model’s resolution. \n\n---\n\nWe hope these clarifications address your concerns. We are grateful for your thoughtful feedback and are committed to improving our work in future versions."}, {"review_id": "i67enhzP8Q", "reviewer": "Reviewer_gkEp", "summary": "This paper proposes FINERS, a novel multi-modal large language model (MLLM) framework for precise reasoning and segmentation of ultra-small objects in high-resolution images. The key innovation is a two-stage, coarse-to-fine pipeline: the first stage predicts a coarse region and textual answer from the full image and instruction, while the second stage refines this prediction into a precise bounding box and segmentation mask. The system is trained end-to-end with reinforcement learning. The authors also introduce FINERS-4k, a new benchmark dataset specifically designed for evaluating small-object reasoning and segmentation. Experimental results demonstrate strong performance compared to state-of-the-art baselines.", "questions": "Questions\n1.\tComparison to Agentic  Approaches:\nHow does FINERS compare, both in design and performance, to agent-based or active search methods like V*? What are the pros and cons of your end-to-end RL approach versus more modular or agentic frameworks?\n2.\tAblation on Pretrained Models:\nHow does the performance of FINERS change when initialized with different pretrained MLLMs? Is the framework robust to different base model choices?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Significance: The task of localizing and segmenting very small objects in large, cluttered scenes is both challenging and highly relevant for real-world vision-language applications.\n- Clarity: The paper is generally clear in its explanation of the methodology, and the introduction of a new dataset (FINERS-4k) is a valuable contribution to the community.\n\nMinor Weaknesses\n- Baselines: The experimental comparison is missing several important and recent baselines, especially methods focused on guided visual search for MLLMs (e.g., V*[1]). Including such comparisons would strengthen the claims.\n\n[1] Wu, Penghao, and Saining Xie. \"V?: Guided visual search as a core mechanism in multimodal llms.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "was_revised": true, "final_justification": "The rebuttal response clarified all of my prior concerns (missing baselines and the effect of the base model). Therefore, my opinion remains positive regarding this paper.", "rebuttal": "We thank the reviewer for the constructive comments and positive feedback on our paper. Regarding the concerns of Reviewer gkEp, we provide the following responses.\n\n---\n\n**W1: Baselines: The experimental comparison is missing several important and recent baselines, especially methods focused on guided visual search for MLLMs (e.g., V$*$[1]). Including such comparisons would strengthen the claims.**\n\nThank you for pointing out the importance of guided visual search baselines. We would like to clarify that V* has been evaluated in our experiments across three benchmarks, which can be found under the SEAL[9] entry in **Table 2** and **Table 3** of the main paper. Specifically, the QA accuracy of FineRS vs. V* is 83.3/56.7 vs. 7.53/3.49 on FineRS-4k, 77.5 vs. 75.4 on V* bench, 63.8 vs 38.1 on HR-Bench 4k, and 58.1 vs 25.6 on HR-Bench 8k, respectively. We will make this connection more explicit in the final version to avoid confusion.\n\n---\n\n**Q1: Comparison to Agentic Approaches: How does FINERS compare, both in design and performance, to agent-based or active search methods like V$*$? What are the pros and cons of your end-to-end RL approach versus more modular or agentic frameworks?**\n\nThank you for the insightful question. Below, we compare the two paradigms in terms of design and performance. \n\n- **Design Difference**: Agentic methods like V* adopt a modular visual planning pipeline, where a controller iteratively proposes image regions and queries a frozen MLLM to gradually refine attention and predictions. These methods typically operate in a training-free or shallowly tuned setting, relying on handcrafted rules or search heuristics. In contrast, **FineRS** introduces a two-stage, coarse-to-fine framework, which contains two end-to-end trainable MLLMs (GSE and LPR). We propose a retrospective reward mechanism that uses LPR’s fine-grained feedback to enhance GSE’s localization during GRPO reinforcement fine-tuning.\n\n- **Performance Comparison**: As shown in **Table 2** and **Table 3**, FineRS consistently outperforms state-of-the-art modular methods (e.g., V*[9], MLLM-know[1], DC$^2$[10]) across all benchmarks, with gains of **+30.7/7.9** on FineRS-4k, **+2.1** on V* bench, **+13.8** on HR-Bench 4k and **+17.3** on HR-Bench 8k.\n\nIn summary, unlike agentic approaches that rely on manually crafted policies, FineRS directly learns region selection and structured outputs from instruction-driven supervision, making it more adaptive and unified across tasks. However, the agentic methods are usually easier to implement, while our RL framework requires careful reward design and higher training cost. Nonetheless, the gains in spatial precision and reasoning integration justify the added complexity in high-resolution scenarios.\n\n---\n\n**Q2: Ablation on Pretrained Models: How does the performance of FINERS change when initialized with different pretrained MLLMs? Is the framework robust to different base model choices?**\n\nThank you for the constructive suggestion. Following previous RL-based approaches (e.g., Seg-Zero[16]), we implement our method on Qwen2.5-VL (7b) as the default backbone. Due to the limited rebuttal period, we re-implement our framework with the smaller Qwen2.5-VL (3b) model and compare it against the finetuned Seg-Zero (3b) baseline. The results in **Table R5** show that FineRS maintains promising performance even at smaller scales, highlighting its robustness to backbone size and model initialization.\n\n**Table R5: Performance comparison on the FineRS-4k test set using different pretrained MLLMs. “$\\dagger$” indicates retraining on FineRS-4k.  \n| Method    | S | xS| xxS | All | Color |Shape|Others|Pos.|All |\n|-------------------------------|----------------|----------------|----------------|----------------|----------------|---------------|----------------|-------------------|-------------------|\n| Seg-Zero-7B$^\\dagger$ [16]     | 61.8/50.5 | 53.0/30.2 | 31.7/20.7 | 46.6/38.6 |      | --       | --       | --         | --       | --          |\n| **Ours-7B**              | **62.2/52.6** | **59.0/43.1** | **47.2/27.5** | **55.1/46.5** | 85.8/60.5 | 76.0/49.2 | 78.6/34.4 | 63.2/27.8 | 83.3/56.7 |\n| Seg-Zero-3B$^\\dagger$ [16]  | **57.7**/45.3 | 47.4/22.8 | 26.3/0.86 | 41.6/29.5 | --       | --       | --         | --       | --          |\n| **Ours-3B**              | 57.4/**45.4** | **53.8/23.3** | **43.0/9.9**  | **50.4/29.9** | 65.6/50.5 | 68.6/41.0 | 60.7/21.8 | 52.6/22.2 | 65.6/47.0  |\n\n- **Note**: the columns from “s” to “All” are segmentation accuracy in gIoU/cIoU, and the columns from “color” to “All” represent QA accuracy in MVQA/OVQA.\n\n---\n\nThank you again for your valuable feedback. Should there be any additional concerns or suggestions, we are happy to address them."}, {"review_id": "v8YiJeXRWu", "reviewer": "Reviewer_Du15", "summary": "This paper presents a reinforcement learning framework based on MLLMs for joint reasoning and fine-grained segmentation of small objects in high-resolution scenes. It introduces a new dataset with annotations that support three types of tasks including Multiple-choice Visual Question Answering (MVQA), Open-ended Visual Question Answering (OVQA), and Instruction-guided Segmentation (IS). Extensive experiments were conducted, and experimental results show that the proposed method outperforms SOTA MLLM-based approaches on the newly introduced dataset as well as other public datasets.", "questions": "- Each reward function in (4) and (5) contain several reward terms. Are these terms all binary like the one in (3)? Is there any need to balance the trade-off between different reward terms?\n- In Table 2, the IoU metrics of some methods being compared become much worse after retraining with FINERS dataset, compared to their training-free counterparts. Any explanation for that?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- It introduces a large-scale dataset of UAV imagery to address the challenging problem of reasoning and segmentation of ultra-small objects in complex high-resolution scenes.\n- The newly introduced high-resolution benchmark is strategically designed to evaluate the capability of MLLMs to perform both attribute-level reasoning and pixel-level segmentation.\n- It proposes a coarse-to-fine pipeline for efficient fine-tuning with reinforcement learning. \n\nWeaknesses:\n- The performance of the proposed two-stage approach heavily relies on the output quality of the first stage. \n- Some details are lacking, especially the definition of reward terms in the reward functions.\n- Experimental results are not discussed thoroughly enough.", "was_revised": true, "final_justification": "The rebuttal clarified some of the concerns on performance sensitivity and provided additional discussion on experimental results. Overall, the FINERS-4k dataset is useful for evaluating model capability in fine-grained reasoning and segmentation of small objects. Therefore, I maintain my positive assessment of the paper.", "rebuttal": "We thank the reviewer for the constructive comments on our paper. Regarding the concerns of Reviewer Du15, we provide the following responses.\n\n---\n\n**W1: The performance of the proposed two-stage approach heavily relies on the output quality of the first stage.**\n\nThank you for the thoughtful observation. To begin with, we should clarify that the **QA output is not dependent on the GSE’s coarse box**, as the answer is generated from the full-resolution image and instruction before region cropping. Therefore, QA accuracy remains stable even if the coarse region partially or totally misses the target. \n\nHowever, we acknowledge that the downstream localization performance is sensitive to the GSE output. This design reflects our coarse-to-fine search strategy inspired by human vision, where attention is first directed to a broad region and then progressively refined. Like humans, our system may fail if the initial focus region omits key information, particularly under ultra-small object conditions. To improve robustness, we introduce a retrospective reward that leverages LPR feedback to improve GSE training, which achieves an improvement of **1.1/2.5** on gIoU/cIoU (see **Table 4** in the main paper). While this mitigates some failure cases, we acknowledge the lack of a re-evaluation or correction mechanism, which is a limitation compared to human vision. In future work, we plan to explore iterative refinement or ensemble-based re-focusing strategies to enhance this pipeline.\n\n---\n\n**W2&Q1: Some details are lacking, especially the definition of reward terms in the reward functions.**\n\nThank you for pointing this out. We apologize for the lack of clarity. Following Seg-Zero [16], all reward terms used in Eq.4 \\& Eq.5 are binary-valued. \n\n- In Eq.4:\n  1. $R_{point}=1$ if L1 distance between predicted point and GT point is less than 100 pixels. \n  2. $R_{b^{L1}}=1$ if the L1 distance between the predicted box and GT box is less than 10 pixels.\n  3. $R_{b^{IoU}}=1$ if their IoU is greater than 0.5.\n  4. $R_{response}$ is set to 1 if the predicted answer is correct (exact match for multiple choice, fuzzy match for open-ended  QA).\n  5. $R_{format}$ and $R_{think}$ are binary rewards that verify whether the output adheres to the expected JSON and reasoning formats.\n\n- In Eq.5：\n  1. $R_{{region}^{L1}}=1$ if the L1 distance between the predicted coarse box $B_r^{pre}$ and GT region $B_r^{gt}$ (see L215-221) is less than 10 pixels.\n  2. $R_{region^{IoU}}=1$ if their IoU is greater than 0.5; $R_{size}$ is 1 when the predicted coarse box is of size $512 \\times 512$.\n  3. $R_{cover}$ is 1 when the ground-truth object lies fully inside the predicted region.\n\nWe assign equal weights to all binary terms, following standard GRPO practices [14,16], which yielded stable performance without tuning. We will clarify these details explicitly in the final version.\n\n---\n\n**W3&Q2: Experimental results are not discussed thoroughly enough.**\n\nThank you for these valuable comments. We will improve the experiment discussion of both comparison and ablation studies in the final version. \n\nRegarding the observed performance drop of some baselines (e.g., LISA$^\\dagger$, PixelLM$^\\dagger$) after retraining, we believe this is primarily due to the significant domain shift and task complexity introduced by FineRS-4k. It consists of 4k-resolution UAV imagery with ultra-small, sparsely distributed objects, which differs drastically from the training distributions of most MLLM-based models (e.g., refcoco(+/g), ADE20k, etc.). To ensure fairness, all baselines are fine-tuned on the same FineRS-4k, starting from either LLaVA (LISA, PixelLM) or Qwen (Seg-Zero, ours). Under this low-data, high-resolution regime, fully supervised models with fixed architectures tend to overfit or underfit.\n\nA closer examination reveals different levels of robustness across baselines:\n\n- PixelLM [19] doesn't use pretrained SAM but adopts a lightweight decoder for model efficiency. However, the significant drop in training data from the original 200k images to 3626 images (8.9k samples) makes heavy LLM difficult to adapt well to this challenging task under limited training data.\n\n- LISA [3], in contrast, integrates SAM for more robust spatial grounding ability. We would also like to correct a reporting error in Table 2, where the gIoU (All) of LISA$^\\dagger$ was mistakenly reported as 1.21 and should be 12.1. After this correction, LISA$^\\dagger$ shows modest improvements on most metrics, demonstrating better adaptability compared to PixelLM. \n        \nThese observations highlight the importance of explicit spatial reasoning and modular design in adapting to high-resolution, small-object scenarios. Our FineRS adopts a coarse-to-fine reinforcement learning framework that is more data-efficient and robust to domain shifts, leading to stable performance across benchmarks and object sizes. \n\nWe will provide a more in-depth analysis of these observations and additional insights into cross-method performance trade-offs in the final version.\n\n---\n\n**Q1: Each reward function in (4) and (5) contain several reward terms. Are these terms all binary like the one in (3)? Is there any need to balance the trade-off between different reward terms?.**\n\nAddressed in W1.\n\n---\n\n**Q2: In Table 2, the IoU metrics of some methods being compared become much worse after retraining with FINERS dataset, compared to their training-free counterparts. Any explanation for that?**\n\nAddressed in W3.\n\n---\n\nWe thank the reviewer for the helpful suggestions. If you have any further questions or suggestions, we would be happy to provide additional clarifications."}, {"review_id": "PJci4NhYkc", "reviewer": "Reviewer_tyVp", "summary": "The paper addresses instruction-guided reasoning and segmentation for high-resolution images by introducing the FINERS framework. This framework integrates the reinforcement learning (RL)-based approach Seg-Zero [1] with two modules for region search: Global Semantic Exploration (GSE) and Localized Perceptual Refinement (LPR). A new UAV-based benchmark dataset, FINERS-4k, is curated for evaluation. The proposed framework demonstrates superior performance on the FINERS-4k dataset as well as two other high-resolution VQA benchmarks, V* and HR-Bench, outperforming existing methods.\n\n[1] Liu, Yuqi, et al. \"Seg-zero: Reasoning-chain guided segmentation via cognitive reinforcement.\" arXiv preprint arXiv:2503.06520 (2025).", "questions": "1. Please explicitly highlight the key technical contributions of the proposed method in comparison to the Seg-Zero framework. If possible, demonstrating that the proposed modules (GSE and LPR) are also effective when applied to other frameworks or tasks would significantly strengthen the paper.\n\n2. Please refer to Weakness 2 and address the concerns regarding the construction of the FINERS-4k dataset. Clarifying this aspect is crucial to ensure that no potential bias was introduced during the dataset collection process.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The curated FINERS-4k benchmark has the potential to be useful and valuable for various machine learning communities. It provides approximately 12k high-resolution (4K) images containing small objects, along with corresponding questions, answers, and mask annotations, supporting a wide range of tasks.\n\n2. FINERS demonstrates promising performance across all tasks (instruction-guided segmentation, multiple-choice VQA, and open-ended VQA) as well as across all benchmarks (FINERS-4k, V*, and HR-Bench), as presented in Tab.2 and 3.\n\n3. FINERS outperforms training-free methods such as [2] that utilize image patch partitioning, highlighting the effectiveness and necessity of the proposed region search mechanism.\n\n4. The effectiveness of each proposed component is well validated through a series of ablation studies (Tab. 4).\n\nWeaknesses:\n\n1. The proposed FINERS framework appears to be heavily based on Seg-Zero [1], sharing a similar pipeline and loss design. As such, FINERS functions more as a great extension of Seg-Zero, integrating the region search mechanism (GSE and LPR) and supporting reasoning tasks, rather than representing a significant scientific advancement. Moreover, the region search mechanism (based on RL) has been explored in other tasks [3], which may further diminish the paper’s technical novelty and contribution.\n\n2. Although the paper provides extensive details about the FINERS-4k dataset, such as its distribution and sample data, it does not clarify how volunteers were instructed to annotate small objects using triplet annotations (question, answer, and mask). For instance, how did the volunteers determine which object to select? How did they decide what question to ask and how to answer it? How did they ensure that no other objects also satisfied the given answer? Take Fig. 4 as an example: what if there is more than one person riding a motorcycle on the road, or more than one umbrella on the beach? This issue is particularly challenging in high-resolution images, where exhaustively checking for all possible matching objects is nearly impossible.\n\n[2] Wang, Wenbin, et al. \"Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 8. 2025.\n\n[3] Gao, Mingfei, et al. \"Dynamic zoom-in network for fast object detection in large images.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.", "was_revised": true, "final_justification": "While I still view FINERS as an engineering solution that heavily builds upon Seg-Zero, the proposed extensions (e.g., GSE and LPR) effectively broaden its applicability. Although some concerns about potential bias in the FINERS-4k dataset remain, the dataset is still valuable if its limitations are properly discussed. I have decided to maintain my positive assessment of the paper.", "rebuttal": "We sincerely thank Reviewer tyVp for the thoughtful and constructive feedback. Below, we address the main concerns in detail.\n\n---\n\n**W1: The proposed FINERS framework appears to be heavily based on Seg-Zero [1], sharing a similar pipeline and loss design. As such, FINERS functions more as a great extension of Seg-Zero, integrating the region search mechanism (GSE and LPR) and supporting reasoning tasks, rather than representing a significant scientific advancement. Moreover, the region search mechanism (based on RL) has been explored in other tasks [3], which may further diminish the paper’s technical novelty and contribution.**\n\nThank you for raising this important concern. \n\nWhile FineRS adopts the GRPO-based learning paradigm from Seg-Zero [16], it introduces several key innovations in task formulation, modular design, and training strategy. \n\n- **Task Formulation**: Unlike Seg-Zero [16] that solely addresses reason segmentation, our method focuses on **a boarder and harder task formulation** that addresses multi-output spatial reasoning, including textual answering, coarse region box, and accurate object box and points. Such unified task is conducted under a more challenging setting, ultra-small object localization in 4K UAV imagery, which significantly increases both semantic and spatial complexity.\n\n- **Architecture Design**: We propose a two-stage architecture that effectively incorporates **coarse semantic localization (GSE)** and **perception-level fine localization (LPR)**. Importantly, FINERS introduces a novel retrospective reward coupling, where LPR feedback is back-propagated to improve GSE via reinforcement learning. This closed-loop learning strategy improves both exploration efficiency and stability, and is not present in Seg-Zero or other prior works.\n\nWhile prior work like [3] (\"Dynamic zoom-in network\") also explores zoom-based strategies, their RL is applied to conventional CNN networks to navigate visual space for efficient detection. In contrast, FineRS operates in a language-conditioned setting, with region search fully guided by multi-modal reasoning in MLLMs. Moreover, [3] relies on standard detectors with fixed reward structures, whereas our method leverages instruction-driven feedback and formulates more complex multi-modal reward strategies.\n\nIn summary, these differences highlight that FINERS is not a simple extension of prior works, but a task-level and framework-level advancement aimed at enabling structured multi-modal reasoning for ultra-small object understanding in high-resolution visual environments.\n\n---\n\n**W2&Q2: Although the paper provides extensive details about the FINERS-4k dataset, such as its distribution and sample data, it does not clarify how volunteers were instructed to annotate small objects using triplet annotations (question, answer, and mask). For instance, how did the volunteers determine which object to select? How did they decide what question to ask and how to answer it? How did they ensure that no other objects also satisfied the given answer? Take Fig. 4 as an example: what if there is more than one person riding a motorcycle on the road, or more than one umbrella on the beach? This issue is particularly challenging in high-resolution images, where exhaustively checking for all possible matching objects is nearly impossible.**\n\nThank you for this valuable comment. During dataset construction, annotators were primarily instructed to identify a single, unambiguous small object of interest from the image and compose questions that uniquely specify it (e.g., referencing color, shape, position, or context). For several cases with multiple similar objects, they were required to formulate disambiguating questions and to visually inspect the entire image to ensure no other object matched the same description. \n\nThe annotation was completed by 14 volunteers, organized in pairs for mutual cross-checking. In addition, a team of 4 senior reviewers conducted a final round of quality assurance to correct ambiguities and verify consistency. This multi-stage validation process was designed to maximize precision and minimize annotation bias.\n\nWhile it remains difficult to guarantee absolute uniqueness in complex 4K imagery, our annotation pipeline is designed to ensure consistency and minimize overlap ambiguity. We will release detailed annotation guidelines and the full benchmark to support reproducibility and transparency.\n\n---\n\n**Q1: Please explicitly highlight the key technical contributions of the proposed method in comparison to the Seg-Zero framework. If possible, demonstrating that the proposed modules (GSE and LPR) are also effective when applied to other frameworks or tasks would significantly strengthen the paper.**\n\nThank you for this insightful suggestion. The contributions over Seg-Zero are discussed in **W1** above. Here, we mainly addressed the framework or task generalization. \n\n- **Module generalization**: The GSE and LPR can generalize across domains:\n\n  - **LPR**, when evaluated on the ReasonSeg dataset (See **Table R4** in Reviewer kUFY), shows promising localization performance even without any large-scale pretraining (like LISA[3]) or in-domain finetuning (like Seg-Zero[16]), demonstrating robustness to significant different domains, scales, and resolution changes.  \n\n  - **GSE** also generalizes well across domains, as evidenced by QA performance on external high-resolution benchmarks. As shown in **Table 3** of main paper, its QA accuracy outperforms state-of-the-art methods by **+2.1** on V* bench, **+13.8** on HR-Bench 4k and **+17.3** on HR-Bench 8k. \n\n- **Model scalability**: We re-implement FINERS using a smaller backbone (Qwen2.5-VL-3B) and observe consistent gains over Seg-Zero 3B (see **Table R5**, Reviewer gkEp), confirming that our framework maintains effectiveness under limited capacity and is not reliant on large-scale models.\n\nThese results validate the transferability and robustness of our modular design, suggesting that FineRS is broadly applicable beyond the original task.\n\n---\n\n**Q2: Please refer to Weakness 2 and address the concerns regarding the construction of the FINERS-4k dataset. Clarifying this aspect is crucial to ensure that no potential bias was introduced during the dataset collection process.**\n\nAddressed in W2. \n\n---\n\nThanks again for your valuable suggestions. If you have any additional questions or would like further clarifications, we would be happy to provide more details."}]

Author rebuttal:

Meta-review: This paper proposes a multimodal LLM with an RL-based reasoning approach for understanding and localization of very small objects in high resolution images, along with a new benchmark dataset for the task. Four reviewers provided final scores of 4 x borderline accept. The reviewers appreciated the work for its novel proposed approach, the significance of its task and its dataset. The reviewers' initial concerns were adequately addressed during the author-reviewer discussion phase and they all rated the work positively post-discussions. The AC concurs with the reviewers' consensus and recommends acceptance. Congratulations! The authors should make the changes that they have promised in the camera ready version of their paper.
Decision: Accept (poster)