Title: SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Learning

Multimodal Large Models

Image Segmentation
Abstract: Leveraging multimodal large models for image segmentation has become a prominent research direction. However, existing approaches typically rely heavily on manually annotated datasets that include explicit reasoning processes, which are costly and time-consuming to produce. Recent advances suggest that reinforcement learning (RL) can endow large models with reasoning capabilities without requiring such reasoning-annotated data. In this paper, we propose SAM-R1, a novel framework that enables multimodal large models to perform fine-grained reasoning in image understanding tasks. Our approach is the first to incorporate fine-grained segmentation settings during the training of multimodal reasoning models. By integrating task-specific, fine-grained rewards with a tailored optimization objective, we further enhance the model's reasoning and segmentation alignment. We also leverage the Segment Anything Model (SAM) as a strong and flexible reward provider to guide the learning process. With only 3k training samples, SAM-R1 achieves strong performance across multiple benchmarks, demonstrating the effectiveness of reinforcement learning in equipping multimodal models with segmentation-oriented reasoning capabilities.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper  propose SAM-R1,  a novel framework that enables multimodal large models to perform fine-grained reasoning in image understanding tasks. It incorporates fine-grained segmentation settings during the training of multimodal reasoning models. Extensive experiments demonstrate its effetiveness.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n1.\tThe paper is well-written and easy to follow.\n\n2.\tThe visualisation is straightforward, and easy to demonstrate the effectiveness of the proposed approach.\n\n3.\tThe topic the paper addressed is cut-edging and deserve to explore.\n\nWekaness:\n\n1. Missing related work. The current setting of task-generic, promptable segmentation also aims to infer image-specific answers given a generic prompt and perform corresponding segmentation. It would be beneficial to add a brief discussion of the literature on this setting [1][2][3]:\n\n[1] Hu, J., Lin, J., Gong, S., et al. Relax image-specific prompt requirement in SAM: A single generic prompt for segmenting camouflaged objects. Proceedings of the AAAI Conference on Artificial Intelligence, 2024, 38(11): 12511–12518.\n\n[2] Tang, L., Jiang, P. T., Shen, Z. H., et al. Chain of visual perception: Harnessing multimodal large language models for zero-shot camouflaged object detection. Proceedings of the 32nd ACM International Conference on Multimedia, 2024: 8805–8814.\n\n[3] Hu, J., Lin, J., Yan, J., et al. Leveraging hallucinations to reduce manual prompt dependency in promptable segmentation. The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.\n\n2. Insufficient discussion of innovation. The innovation of this work needs to be further clarified. The proposed method appears to directly apply GRPO to the reasoning segmentation task, so the key differences and improvements over standard GRPO must be more explicitly articulated.\n\nQUESTIONS:\n1.Design and sensitivity of the reward. The goal of SAM-R1 is to enable the segmentation model to leverage GRPO so that it can exhibit reasoning capabilities without requiring an explicit reasoning chain as ground truth. This makes the design of the reward particularly critical. Specifically, the reward in Equation (5) depends heavily on the chosen threshold. Could you explain how this threshold is determined? Why is the reward discretised into multiple levels? Moreover, would small variations in the threshold value lead to significant differences in the results?\n\n2. Clarification of improvements over GRPO. The method presented in this paper appears to directly apply GRPO to the reasoning segmentation task. What are the main technical contributions or modifications compared to standard GRPO? It would be helpful to highlight the key improvements that make the proposed approach better suited for reasoning segmentation.\n\nLIMITATIONS:\nPlease refers to weakness and questions sections", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an end-to-end multimodal segmentation framework, SAM-R1, which integrates the Segment Anything Model as a reward provider in reinforcement learning for reasoning-based segmentation tasks. The approach introduces fine-grained segmentation-aligned rewards and modifies the policy optimization objective to improve information density and training stability. Experimental results show that SAM-R1 achieves reasonable improvements in gIoU and cIoU compared to previous methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper is clearly written, with well-structured content and logical flow.\n\n2. The work introduces an interesting reinforcement learning approach for vision tasks and provides new insights into applying reward-based optimization in multimodal segmentation.\n\n3. The proposed method achieves reasonable empirical results.\n\nWeaknesses\n\n1. The technical novelty is limited, since the approach primarily leverages existing open-source models rather than introducing fundamentally new architectures or algorithms.\n\n2. A core limitation is the feasibility of reward computation in SAM-RL. The method requires extremely fine-grained segmentation annotations to compute meaningful rewards. For example, as illustrated in Figure 1, evaluating the model's ability to focus on a cat's eyes would require annotations for the cat eyes themselves, which are rarely available in standard segmentation datasets, where labels typically cover entire objects such as the whole cat. Even when using SAM's segment-everything mode to generate masks for parts like eyes, these regions lack semantic labels. Achieving high-precision fine-grained segmentation remains a challenging problem in the segmentation field. To fully leverage the RL mechanism proposed in this work, a large-scale dataset with detailed fine-grained segmentation annotations would be required, which inherently restricts the applicability of the method.\n\n3. There are potential concerns regarding the fairness of comparisons in Table 1. Most baselines, except for Seg-Zero, use weaker MLLM backbones compared to Qwen2.5VL-7B adopted in this paper, and it is unclear whether Seg-Zero's results correspond to the 3B or 7B model. Additionally, as noted in the paper, the Seg-Zero results are reproduced by the authors rather than directly from the original work. To ensure fair comparison, alternative training strategies and backbones should be tested under identical conditions.\n\nQUESTIONS:\nPlease refer to the weaknesses section for the rebuttal.\n\nLIMITATIONS:\nYes, this is discussed in the Broader Impact section of the supplementary material, though it is not explicitly marked with a heading.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the SAM-R1 framework, aiming to empower multimodal large models (MLLMs) with superior reasoning capabilities in image segmentation tasks through reinforcement learning (RL). Specifically, the authors calculate reward scores using masks generated by SAM and conduct RL training based on an improved GRPO algorithm to establish a dynamic feedback loop between reasoning and segmentation. The experimental section conducts comprehensive comparative validations on multiple benchmark datasets such as ReasonSeg and the RefCOCO series.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The proposed reinforcement learning training pipeline for segmentation in the paper is concise and highly applicable.\n2. The paper demonstrates strong experimental performance, achieving excellent results with a small number of reinforcement learning samples.\n3. The experiments are well-detailed and presented in a highly readable manner.\n\nWeaknesses:\n1. The overall approach of the paper is relatively similar to works like Seg-Zero, and the degree of innovation is somewhat limited.\n\nQUESTIONS:\n1. The authors of this paper believe that the difference between SAM-R1 and Seg-Zero lies in the integration of reasoning and segmentation capabilities, forming a closed loop of \"reasoning-segmentation-reward\". However, Seg-Zero also uses SAM as a reward generator. Is the only difference that the reward form determination of SAM-R1 is divided into multiple levels?\n2. What is the size of the VLM in Seg-Zero used in the experiment? Since the original paper mentions 3B, it is not fair to make a direct comparison.\n3. If instead of using SAM, we follow works like PixelLM and make the segmenter a trainable part of the model, will the reinforcement learning training be more effective?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SAM-R1, a novel framework for enhancing multimodal large language models' (MLLMs) reasoning segmentation capabilities via reinforcement learning (RL). The method uniquely incorporates the Segment Anything Model (SAM) as an active reward provider within an RL loop, offering fine-grained, segmentation-aware feedback. SAM-R1 is trained with only 3K samples, yet achieves state-of-the-art performance on ReasonSeg and competitive performance on referring expression segmentation tasks (RefCOCO, RefCOCO+, RefCOCOg).\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- Novel Integration of SAM as Reward Provider\n\nRather than passively using SAM for downstream inference, this work integrates SAM directly into the RL training loop, allowing reasoning quality to be directly shaped by segmentation accuracy.\n\n- Compact yet Effective Training\n\nAchieves strong zero-shot performance on complex reasoning-segmentation tasks with only 3,000 training samples.\n\n- Well-Designed Reward System\n\nThe tiered reward function based on IoU allows for dense and scalable feedback. Additional format and structure rewards (for reasoning and segmentation) further guide training.\n\n- Strong Experimental Results\n\nOutperforms or matches prior art like Seg-Zero across multiple benchmarks (e.g., +1.9 gIoU over Seg-Zero on ReasonSeg-zero-shot test). Ablations confirm the effectiveness of each design.\n\n- Clarity and Organization\n\nThe method is clearly motivated, the framework is well-illustrated (Figure 2), and the paper includes extensive implementation and training details.\n\n## Weaknesses\n- Lack of Direct Comparison with Alternative RL Strategies\n\nWhile GRPO is well explained and modified, it would be helpful to see direct comparisons with more traditional RL methods (e.g., PPO) under the same setting.\n\n- Missing User-Centric Evaluation\n\nNo human evaluation or qualitative user study is conducted to assess reasoning interpretability, mask quality, or real-world usability.\n\nQUESTIONS:\n1. Comparison with PPO\n\nCan the authors add or discuss results comparing GRPO-based SAM-R1 with standard PPO or RLHF strategies? This would better justify the algorithmic choice beyond citing past literature.\n\n2. Data Efficiency vs. Scale\n\nSAM-R1 performs well with 3K samples. Can the authors comment on how performance scales with larger datasets? Is the approach still competitive under full-data settings?\n\n3. Failure Case Analysis\n\nAre there common failure modes in either reasoning or segmentation (e.g., hallucinations, over-segmentation)? A brief discussion or visualization would strengthen trust in robustness.\n\nLIMITATIONS:\nFailure case analysis is not included in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}