Title: FineRS: Fine-grained Reasoning and Segmentation of Small Objects with Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: High-Resolution

Small Object Segmentation

MLLMs

Reinforcement Learning
Abstract: Multi-modal Large Language Models (MLLMs) have shown remarkable capabilities across a wide range of vision-language tasks. However, due to the restricted input resolutions, MLLMs face significant challenges in precisely understanding and localizing visual details in high-resolution images---particularly when dealing with extra-small objects embedded in cluttered contexts. 
To address this issue, we propose FineRS, a two-stage MLLM-based reinforcement learning framework for jointly reasoning and segmenting extremely small objects within high-resolution scenes. FineRS adopts a coarse-to-fine pipeline comprising Global Semantic Exploration (GSE) and Localized Perceptual Refinement (LPR). Specifically, GSE performs instruction-guided reasoning to generate a textural response and a coarse target region, while LPR refines this region to produce an accurate bounding box and segmentation mask. To couple the two stages, we introduce a locate-informed retrospective reward, where LPR's outputs are used to optimize GSE for more robust coarse region exploration.  Additionally, we present FineRS-4k, a new dataset for evaluating MLLMs on attribute-level reasoning and pixel-level segmentation on subtle, small-scale targets in complex high-resolution scenes. Experimental results on FineRS-4k and public datasets demonstrate that our method consistently outperforms state-of-the-art MLLM-based approaches on both instruction-guided segmentation and visual reasoning tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses instruction-guided reasoning and segmentation for high-resolution images by introducing the FINERS framework. This framework integrates the reinforcement learning (RL)-based approach Seg-Zero [1] with two modules for region search: Global Semantic Exploration (GSE) and Localized Perceptual Refinement (LPR). A new UAV-based benchmark dataset, FINERS-4k, is curated for evaluation. The proposed framework demonstrates superior performance on the FINERS-4k dataset as well as two other high-resolution VQA benchmarks, V* and HR-Bench, outperforming existing methods.\n\n[1] Liu, Yuqi, et al. \"Seg-zero: Reasoning-chain guided segmentation via cognitive reinforcement.\" arXiv preprint arXiv:2503.06520 (2025).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The curated FINERS-4k benchmark has the potential to be useful and valuable for various machine learning communities. It provides approximately 12k high-resolution (4K) images containing small objects, along with corresponding questions, answers, and mask annotations, supporting a wide range of tasks.\n\n2. FINERS demonstrates promising performance across all tasks (instruction-guided segmentation, multiple-choice VQA, and open-ended VQA) as well as across all benchmarks (FINERS-4k, V*, and HR-Bench), as presented in Tab.2 and 3.\n\n3. FINERS outperforms training-free methods such as [2] that utilize image patch partitioning, highlighting the effectiveness and necessity of the proposed region search mechanism.\n\n4. The effectiveness of each proposed component is well validated through a series of ablation studies (Tab. 4).\n\nWeaknesses:\n\n1. The proposed FINERS framework appears to be heavily based on Seg-Zero [1], sharing a similar pipeline and loss design. As such, FINERS functions more as a great extension of Seg-Zero, integrating the region search mechanism (GSE and LPR) and supporting reasoning tasks, rather than representing a significant scientific advancement. Moreover, the region search mechanism (based on RL) has been explored in other tasks [3], which may further diminish the paper’s technical novelty and contribution.\n\n2. Although the paper provides extensive details about the FINERS-4k dataset, such as its distribution and sample data, it does not clarify how volunteers were instructed to annotate small objects using triplet annotations (question, answer, and mask). For instance, how did the volunteers determine which object to select? How did they decide what question to ask and how to answer it? How did they ensure that no other objects also satisfied the given answer? Take Fig. 4 as an example: what if there is more than one person riding a motorcycle on the road, or more than one umbrella on the beach? This issue is particularly challenging in high-resolution images, where exhaustively checking for all possible matching objects is nearly impossible.\n\n[2] Wang, Wenbin, et al. \"Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 8. 2025.\n\n[3] Gao, Mingfei, et al. \"Dynamic zoom-in network for fast object detection in large images.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.\n\nQUESTIONS:\n1. Please explicitly highlight the key technical contributions of the proposed method in comparison to the Seg-Zero framework. If possible, demonstrating that the proposed modules (GSE and LPR) are also effective when applied to other frameworks or tasks would significantly strengthen the paper.\n\n2. Please refer to Weakness 2 and address the concerns regarding the construction of the FINERS-4k dataset. Clarifying this aspect is crucial to ensure that no potential bias was introduced during the dataset collection process.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper presents a reinforcement learning framework based on MLLMs for joint reasoning and fine-grained segmentation of small objects in high-resolution scenes. It introduces a new dataset with annotations that support three types of tasks including Multiple-choice Visual Question Answering (MVQA), Open-ended Visual Question Answering (OVQA), and Instruction-guided Segmentation (IS). Extensive experiments were conducted, and experimental results show that the proposed method outperforms SOTA MLLM-based approaches on the newly introduced dataset as well as other public datasets.

STRENGTHS AND WEAKNESSES:
Strengths:
- It introduces a large-scale dataset of UAV imagery to address the challenging problem of reasoning and segmentation of ultra-small objects in complex high-resolution scenes.
- The newly introduced high-resolution benchmark is strategically designed to evaluate the capability of MLLMs to perform both attribute-level reasoning and pixel-level segmentation.
- It proposes a coarse-to-fine pipeline for efficient fine-tuning with reinforcement learning. 

Weaknesses:
- The performance of the proposed two-stage approach heavily relies on the output quality of the first stage. 
- Some details are lacking, especially the definition of reward terms in the reward functions.
- Experimental results are not discussed thoroughly enough.

QUESTIONS:
- Each reward function in (4) and (5) contain several reward terms. Are these terms all binary like the one in (3)? Is there any need to balance the trade-off between different reward terms?
- In Table 2, the IoU metrics of some methods being compared become much worse after retraining with FINERS dataset, compared to their training-free counterparts. Any explanation for that?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes FINERS, a novel multi-modal large language model (MLLM) framework for precise reasoning and segmentation of ultra-small objects in high-resolution images. The key innovation is a two-stage, coarse-to-fine pipeline: the first stage predicts a coarse region and textual answer from the full image and instruction, while the second stage refines this prediction into a precise bounding box and segmentation mask. The system is trained end-to-end with reinforcement learning. The authors also introduce FINERS-4k, a new benchmark dataset specifically designed for evaluating small-object reasoning and segmentation. Experimental results demonstrate strong performance compared to state-of-the-art baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Significance: The task of localizing and segmenting very small objects in large, cluttered scenes is both challenging and highly relevant for real-world vision-language applications.\n- Clarity: The paper is generally clear in its explanation of the methodology, and the introduction of a new dataset (FINERS-4k) is a valuable contribution to the community.\n\nMinor Weaknesses\n- Baselines: The experimental comparison is missing several important and recent baselines, especially methods focused on guided visual search for MLLMs (e.g., V*[1]). Including such comparisons would strengthen the claims.\n\n[1] Wu, Penghao, and Saining Xie. \"V?: Guided visual search as a core mechanism in multimodal llms.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\nQUESTIONS:\nQuestions\n1.\tComparison to Agentic  Approaches:\nHow does FINERS compare, both in design and performance, to agent-based or active search methods like V*? What are the pros and cons of your end-to-end RL approach versus more modular or agentic frameworks?\n2.\tAblation on Pretrained Models:\nHow does the performance of FINERS change when initialized with different pretrained MLLMs? Is the framework robust to different base model choices?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes FINERS, a two-stage multimodal large-language-model (MLLM) framework for instruction-guided reasoning and position of small objects in 4K images.\n\nIn the stage 1(GSE),  the model reads the HR image and instruction, outputs a textual answer and a fixed-size 256 × 256 coarse region.\nIn the stage 2(LPR), the model crops that region, predicts the position information, a precise box + two points, and calls SAM2 to obtain a pixel mask.\n\nThe stages are coupled with a locate-informed retrospective reward so that LPR’s IoU/L1 feedback reinforces GSE during vision reinforcement fine-tuning with GRPO.\n\nThe authors introduce a dataset, FINERS-4k, for evaluation.\n\nSTRENGTHS AND WEAKNESSES:\n[+] Coarse-to-fine RL design with retrospective reward is conceptually elegant and empirically helpful.\n\n[+]  FINERS-4k fills a data gap for UAV, tiny-object HR tasks and could benefit the community.\n\n[-]  Entire pipeline hinges on GSE: if the target object straddles the coarse box, both answer and mask fail.\n\n[-]  Two stages are trained separately; no end-to-end fine-tuning —may be sub-optimal.\n\n[-]  Hyper-parameter sensitivity (group size n, KL weight) and seed robustness are not analyzed; reproducibility risk.\n\nQUESTIONS:\nRobustness to coarse-box errors: How does FINERS detect or recover when GSE misses the target? Please quantify failure rates where the target object lies partially or fully outside the coarse region and evaluate simple fall-back strategies.\n\nEfficiency: Please report wall-clock inference time for 4K inputs, vs VILA-HD, Seg-zero and DC^2. That would strengthen practicality.\n\nDomain generalization: Evaluate FINERS on at least one non-UAV dataset containing small indoor objects. Comparable accuracy would indicate broader utility.\n\nLIMITATIONS:\nNo.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}