SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learning
Authors: Jiaqi Huang
Zunnan Xu
Jun Zhou
Ting Liu
Yicheng Xiao
Mingwen Ou
Bowen Ji
Xiu Li
Kehong Yuan
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning
Multimodal Large Models
Image Segmentation
Primary area: deep_learning

Abstract: Leveraging multimodal large models for image segmentation has become a prominent research direction. However, existing approaches typically rely heavily on manually annotated datasets that include explicit reasoning processes, which are costly and time-consuming to produce. Recent advances suggest that reinforcement learning (RL) can endow large models with reasoning capabilities without requiring such reasoning-annotated data. In this paper, we propose SAM-R1, a novel framework that enables multimodal large models to perform fine-grained reasoning in image understanding tasks. Our approach is the first to incorporate fine-grained segmentation settings during the training of multimodal reasoning models. By integrating task-specific, fine-grained rewards with a tailored optimization objective, we further enhance the model's reasoning and segmentation alignment. We also leverage the Segment Anything Model (SAM) as a strong and flexible reward provider to guide the learning process. With only 3k training samples, SAM-R1 achieves strong performance across multiple benchmarks, demonstrating the effectiveness of reinforcement learning in equipping multimodal models with segmentation-oriented reasoning capabilities.

SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learning
Jiaqi Huang", Zunnan Xu', Jun Zhou', Ting Liu, Yicheng Xiao, Mingwen Ou, Bowen Ji, Xiu Li, Kehong Yuan Tsinghua University
Abstract
Leveraging multimodal large models for image segmentation has become a promi- nent research direction. However, existing approaches typically rely heavily manually annotated datasets that include explicit reasoning processes, which are costly and time-consuming to produce. Recent advances suggest that reinforce- ment learning (RL) can endow large models with reasoning capabilities without requiring such reasoning-annohie annotated data. In this paper, propose SAM-R1, novel framework that enables multimodal large models to perform fine-grained reasoning in image understanding tasks. Our approach is the first to incorporate fine-grained segmentation settings during the training of multimodal reasoning models. By integrating task-specific, fine-grained rewards with tailored opti- mization objective, we further enhance the model's reasoning and segmentation alignment. We also leverage the Segment Anything Model (SAM) as a strong and flexible reward provider to guide the learning process. With only 3k train- ing samples, SAM-RI achieves strong performance across multiple benchmarks, demonstrating the effectiveness of reinforcement learning in equipping multimodal models with eegmentation-orientee reasoning capabilities.
Introduction
Multimodal Large Language Models (MLLMs)
37
22
5
4
51 have achieved remarkable
12, with their capabilities extending to more complex and fine-grained perception tasks like multimodal segmentation [38 21 40]. Compared to conventional segmentation methods that rely simple categorical labels, the reasoning segmentation task [2] 16. 31 has garnered significant attention for its flexibility and practical applicability, but it also introduces substantially greater challenges. Specifically, it requires models not only to comprehend the intent behind user providdd textual queries accurately but also to perform strong logical reasoning generate high-quality, pixel-level segmentation outputs.
progress in the field of visual understanding [36 42 27
43
LISA was the first to introduce the integration of MILMs with segmentation models via spe- cialized tokens, demonstrating the feasibility of applying MLLMs to reasoning segmentation tasks. Building on this foundation, subsequent studies 225 2
41 have adopted similar strategies, lever- aging task-specific tokens generated by MLLMs to improve pixel-level segmentation performance. While these approaches are promising, they often rely heavily on large-scale annotated datasets to jointly fine-tune the language model and the segmentation decoder. This not only increases training costs but also raises the risk of catastrophic forgetting, where models perform well on in-domain data but fail generalize to out-of-domain scenarios [7]. Furthermore, the reasoning segmentation tasks frequently involve ambiguous and complex text queries from users, which demand strong reasoning
'qqual contribution Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Boss Model
SAM-RI Fromework
RC Model
bata propartion
Matti medi LLM
Back Propepptionn
0
1
anonerl
-trisioTh Image dhove a clee--pp of comano lost focuetss ring fouss reg typically y used teajjai foom of lose. the part tnat a phaniggag paggghhe weuld war gulate to oljui the the that wodk fermed change the fouss The ramber's cround the ring redcstee different foous lings, but timming
Queiitie:: phategr soher nerti fanue conaro lose, in the pictene sould they muniulate to
chavart
A (p49,
X
presunse of the lamb and the mumoers around focus
pamt the camaro munt (100.13
Vporrel "ijs w0e (400 410 atoUij--/anpee
User Frompt for SAM-RI
"images> Ploase the bounding (bhox) and points contoining "[Qquestion)"
identify the clotely matched target object within v/thinoo togs, the final within Output thinking that exactly contains torget
objects,
Output
and
better locate the object within the bounding
For within the bounding bocpoiets of the torget object marked with L.
Points withi the bounding box but
the The points represented [xl. volun Polue] cutput JSON format, for example, -thinio here is the thinking precess </thinio To generate the mosk of the torget object, conpurr (Answerry//oreeer>
marked with
Figure 1: SAM-RI generates a reasoning chain prior to producing the segmentation mask. It employs reinforcement learning (RL) strategy, learning the reasoning process from scratch. In comparison to supervised fine-tuning (SFT), the RL-enhanced model, which incorporates fine-grained rewards based on SAM, demonstrates superior performance on both in-domain and out-of-domain data.
capabilities from MILMs to accurately interpret intent and precisely localize the target segmentation regions.
Recent research has shown that reinforcement learning (RL) can significantly enhance the reasoning capabilities of large language models (LLMs) through reward-based feedback mechanisms [13 DeepSeek-RI [9] leverages rule-based rewards to further improve the model's ss capacity for complex reasoning. This method requires the model to undergo an extensive reasoning process before producing a final answer, with rewards assigned solely based on the correctness of the final response and its adherence to a predefined output format. Such rule-based reward designs align naturally with visual understanding tasks, which often come with accurate ground-truth (GT) annotations.
Inspired by this, numerous efforts [ 46 50 39] have applied Group Relative Policy Optimization (GRPO) 3 to vision-language models, incorporating task-specific reward signals. For example, VLM-R1 34 introduces both format and accuracy rewards for general vision-language tasks, and further incorporates customized rewards tailored to specific applications to mitigate reward hacking. Seg-Zero 23 expands this paradigm by designing a more comprehensive reward system, including maasoning-formall to stimulate robust reasoning in segmentation contexts. Although Beg-Zero demonstrates strong performance in emergent reasoning tasks, its complete decoupling of the reasoning model and segmentation decoder prevents access to pixel-level feedback from the segmentation results, thereby increasing the risk of reward hacking. To address this, involving the segmentation decoder directly in the reward loop, as a reward provider, not only ensures alignment between optimization objectives and task goals but also alleviates the need for extensive annotated reasoning data, enabling a more efficient and scalable learning paradigm.
regmentation-forman and accuracy rewards based on loU and L distance,
Building upon the insights presented above, we propose SAM-RI, an efficient end-to-end framework tailored for reasoning segmentation. SAM-RI utilizes reinforcement learning with reward-driven op- timization to enhance the reasoning capabilities of MI.MMs in complex scenarios. A key component of our framework is the design of task-speeific, fine- aine-graieed reward functions, particularly a seg- mentation accuracy reward derived directly from the output of the Segment Anything Model (SAM). This enables the model develop fine-grained perceptual reasoning in an end-to-end manner- aspect that has been largely overlooked in previous multimodal reasoning models for segmentation. Integrating powerful SAM [15] has become a prevalent strategy for achieving precise pixel-level segmentation. SAM's zero-shot segmentation capabilities, facilitated by flexible prompt- based in- puts, render it a highly adaptable component. While existing approaches often employ SAM as a downstream module to generate segmentation masks based on MI.MM outputs (23] our framework distinguishes itself by incorporating SAM directly into the reinforcement learning training loop as a reward signal generator. This integration enables the MLLM to receive direct, task-relevant feedback
rrsept-based

---

based on segmentation accuracy, thereby aligning model optimization with the final task objective in a principled and effective manner.
Moreover, we introduce a subtle modification to the clipped objective of PPO to fully utilize its potential in the reasoning segmentation task. First, we increase the upper clipping threshold to encourage updates from highly advantageous actions, thereby granting the model greater flexibility in optimizing the task-specific reasoning model. Second, we observe that GRPO may occasionally produce overly lengthy responses with limited informative content. During the GRPO optimization process, overly long responses will confuse the model and prevent it from obtaining higher reward signals, which can lead to reward hacking. Rather than constraining each token in single response, we treat all tokens within response group, encouraging the policy model to focus on generating responses with higher information density. By integrating lask-specific, fine- line-grained rewards with a tailored optimization objective, SAM-RI precisely interprets complex instructions and accurately localizes segmentation targets. Using only 3K training samples, our method surpasses the base model by 34. 1 on the challenging ReasonSeg benchmark in zero-shot setting. In conclusion, our contributions can be summarized as below:
We present a novel end-to-end framework for fine-graine ane-grained, reasoning segmentation that employs rule-based rewards enhance comprehension of complex instructions.
We devise task-specific, fine-grained reward functions that leverage SAM as an active reward provider, driving continuous self-improvement of the reasoning model.
We provide extensive empirical evidence demonstrating the effectiveness of SAM-RI and offer new insights into synergizing reinforcement learning with MLLMs.
2
Related Works
2.1 MLLMs for Vision and Reasoning Segmentation
Multimodal Large Language Models (MLLMs) have significantly advanced visual understanding, extending from foundational tasks like image captioning and visual question answering [ 26] 6 to more intricate, fine- -eeiigraidd perception challenges such as image segmentation. A notable direction
reasoning segmentation [45]] 2 31, which necessitates that models interpret implicit user queries and perform logical deduction to generate pixel-level masks. A relevant line of research [10 35 I
focuses on using a single generic prompt to perform segmentation, thereby reducing the reliance on manually provided, image-specific inputs. Seminal works like LISA [16] demonstrated the viability of MILMS for such tasks by interfacing them with segmentation models via specialized tokens. However, these initial approaches frequently depended on Supervised Fine-Tuning (SFT) using datasets with simple categorical labels or rudimentary descriptions (23). This reliance often curtailed out-of-domain generalization and lacked explicit, interpretable reasoning processes [23 34], thereby motivating the exploration of methods to instill more robust reasoning capabilities within MLLMS for segmentation.
2.2 RL for Enhanced Reasoning in Multimodal Tasks
Reinforcement Learning (RL) has emerged as a potent methodology for eliciting and augmenting the reasoning capacities of large models, circumventing the need for datasets with explicit reasoning annotations. Research indicates that reward-driven optimization can effectively activate emergent test-time reasoning. Algorithms such as Group Relative Policy Optimization (GRPO) 33 employed in models like DeepSeek-RR R1 for language tasks 19. Seg-Zero for reasoning segmentation [23] and
VLM-R1 34 for general vision-languse
tasks, have achieved considerable success in training models to generate reasoning chains and attain high performance with limited supervision. These RL-based strategies often exhibit superior generalization compared to SFT methods 77), which are prone to overfitting and catastrophic forgetting of general abilities. Our work leverages this paradigm by adapting an RI. training algorithm based on GRPO [33], specifically tailored the multimodal segmentation task, to cultivate fine-grained perceptual reasoning.
3

---

2.3 Segmentation Feedback with Task-Specific Rewards
The incorporation of powerful, pre-trained segmentation models like the Segment Anything Model (SAM) [15] has become a prevalent strategy for achieving precise pixel-level segmentations. SAM's zero-shot segmentation capabilities, prompted by diverse inputs, render it versatile component. While many frameworks employ SAM as a downstream module to produce segmentation masks from MLLM outputs [23], our approach uniquely integrates SAM as active element within the RL training loop, functioning as a reward provider. This allows the MLLM to receive direct feedback on the quality of its generated information, assessed by the final segmentation accuracy.
The design effective reward functions is paramount in RL. Related works often employ rule-based rewards, encompassing format rewards for structured outputs and accuracy rewards (e.g., Intersection over Union (loU) for bounding boxes or masks, L1 distance) to quantify the quality of spatial predictions. For instance, Seg-Zero utilizes reasoning-formen format, segmentation format, and aceuracy rewards based on IoU and L1 distance [23]. VLM-R1 also employs accuracy and format rewards for tasks such as referring expression comprehension and open-vocabulary object detection (34). Other works like RM-RI focus on correctness-based rewards for reward modeling itself [3]. and R1-Reward introduces consistency rewards alongside formatting and result rewards for training multimodal reward models [50]. Our SAM-RI framework is distinguished by its design of task- specific, fine-grained reward functions, notably a segmentation-aacuuaay reward that directly utilizes SAM's output. This enables the model to learn fine-grained reasoning for segmentation tasks in an end-to-end manner, an aspect largely overlooked in prior work on fine-grained segmentation settings within multimodal reasoning models.
3
Method
In this section, we elaborate on the architecture of our framework. In section 3.1 we explain how our framework enables multimodal large models achieve fine-grained perceptual reasoning capacities. The enhancements made to the reinforcement learning algorithm, which significantly enhance the model's 's multimodal reasoning performance, are detailed in section 3.2 Furthermore, in section 3.3 we offer detailed discussion of our approach to designing the reward function, with SAM integrated as strong and flexible reward provider.
3.1 SAM-R1
As depicted in Figure 2 our framework takes user-supplied questions and images as input. It performs reasoning and analysis to pinpoint the target object by synthesizing information from both modalities. Subsequently, the model generates intermediate reasoning outputs, which serve as inputs to the segmentation model for mask generation. During this process, the model has the flexibility produce outputs that enhance the segmentation model's performance. Our approach diverges from prior work (23), which centered training the multimodal large model alone. Instead, we incorporate the segmentation model as reward provider in the reinforcement learning phase. This integration enables the segmentation model's 's outputs to offer detailed feedback, thereby refining the training of the reasoning model.
3.2 RL Training Algorithm
Using reinforcement learning g to train large models and enhance their performance in specific domains, such as mathematics and programming, has proven effective. However, previous reinforce- ment learning methods often relied on pre-trained model, which led to significant increase in cost and complexity. At the same time, acquiring reasoning capabilities previously required carefully curated datasets that included explicit reasoning processes. Models needed to be trained on these reasoning reasoning-annotaann annotated datasets to achieve competitive performance.
Recent research 9 has shown that large models' reasoning abilities can merge even when trained on datasets without explicit reasoning rules, and the reward mechanism can be greatly simplified while still maintaining the model's strong performance.

---

3.2.1 DeepSeek R1-Zero and GRPO
The DeepSeek R1-Zero algorithm introduces novel training approach using Group Relative Policy Optimization (GRPO). This method trains the model to output both reasoning process and a final answer, while supervision is applied only to the answer. Despite this limited supervision, the model still achieves robust reasoning performance, In this framework, rule-based and accuracy-based reward functions are used to evaluate the model'ss responses, effectively preventing reward hacking and simplifying the overall reward mechanism.
Unlike previous reinforcement learning algorithms such as PPO [32. which require a separate critic model to evaluate performance, GRPO eliminates the need for an additional model by directly comparing all scores within group as baseline. Specifically, for each input question GRPO samples set of responses {01, 02, ... .G from the old policy Taw The reward advantage A; for the i-th response is then computed by normalizing the group of rewards (r1, P2.,2....,,
Mr
(I)
where 14, and or denote the mean and standard deviation of the rewards in the group, respectively. Similar to PPO, GRPO adopts clipped objective, together with directly imposed KL penalty term:
JGRPO (0) = Elg P(Q). (oi) Toae (0)z)]
od
T (oilg)) (0
To(0:19) (0.lg) - E, -,, A1) -BDKL (No (To((
min
A,, clip
G
O
(2)
where the KL divergence is defined as:
Tre9(0;14) (0(0i))) n (0:1))
Trer
DKL (Fol(
log
(3)
3.2.2 Our Training Algorithm
Similar to recent studies 449 24] we observe that the clipping term utilized in advantage estimation
beneficial for maintaining stability in policy updates. At the same time, the KL-divergence penalty already limits the distributional shift between successive policies and therefore also serves stabilizing factor. In our multimodal image-segmentation task, we aim to allow the large multimodal model greater freedom explore finer-grained interpretations while preserving training stability. Hence, we retain the KL. constraint but decouple the clipping mechanism: we replace the single
Trainable
Frozen
Masks
Reward functions
<thinis
Instructions
In nodern hespitals, advanced technology often used to monitor patientss vitol during surgery. What the picture
Multi Modal LLM
Segment Anything
6.20 5
+>
othorrate
Aasswerr
this purpose?
Tiered Reword Fusction
Think Format
Reasoning
<thinleThe image shows medical setting with a patient on opencting table and medical professional surgical attire. The presence monitors displaying
Answers
.bbbx....
that the image haspital setting. tonttors commonly image
of the spine
canawerof "bbbax [280.2 2400224440.3 310].
Outpur Format
Philips X-ray siges during surgery, inaging during surgery. The monitors object that commonly vital signs during surgery in this context-</tinio
"pintss: (132011001]] vansserr
monitoring patients'
penaity
Policy
Ref
Palicy
Figure 2: Our framework integrates the Segment Anything Model (SAM) as reward provider in the reinforcement learning training of a multimodal large model (MLLM). The two models jointly process user-input questions and images to identify target objects and generate masks. Specifically, the MLLM generates the reasoning process and answer, then passes them SAM. fine-grained reward based on Intersection over Union (IoU) is calculated to optimize the MLLM.
5

---

threshold E with asymmetric bounds Elow and Ehigh- We keep unchanged and slightly raise Ehigh encourage broader exploration.
We also observe that GRPO can sometimes yield very long yet low-information answers. Such responses waste tokens and increase the risk of hallucination, long and short answers incur the same total loss, thereby causing the per-token penalty for longer responses. To counter this, we rescale the loss so that every token receives the same loss, discouraging redundant and repetitive outputs. With these changes, our training objective becomes:
Jours (8)
Elq
P(Q), forle
T0 (O(4)
loal
To(o: q) T0 (0:]g)
min
Ai.
(4)
[0s
1==1
1=1
tm1
(o gg)
To
clip
Elow.
1 Ehigh
A, 5L (Te ||])
Te.. (o:lg)
These modifications allow the model to explore aggressively, achieve fine-grained understanding, and train stably without incurring the extra cost and complexity of an additional critic model.
3.3 Reward Functions
A reward model is crucial component of reinforcement learning (RL): combined with preference- alignment algorithms, steers the policy toward the desired objectives. Following earlier work (9], we likewise employ reward functions and adapt them to the multimodal segmentation setting through three task-speeific, rule-based rewards.
Tiered Segmentation-aceuree
Reward Function. Departing from carlier reward designs, we treat SAM (Segment Anything Model) as external reward provider. The target location predicted by the multimodal model is passed to SAM, which returns mask prediction. We compute the IoU between this mask and the ground-trun mask and assign piecewise rewards:
4, loU > 0.80,
3, 0.70 < loU < 0.80,
reward
(5)
0.50 < loU 0.70,
otherwise,
which provides robust positive feedback only when the predicted region closely aligns with the ground truth, guiding the model toward gradual improvement at lower loU levels.
formma reward. To encourage explicit reasoning, the model should enclose its chain-of- thought between "<think>" and "</think>" tags and place the final answer between "<answer>" and "</answer>" tags. Outputs that adhere this structure receive a positive reward, while malformed outputs incur a penalty.
Segmentationn-orras reward. To ensure the multimodal large model provides fine-grained cues to the downstream segmentation module, it must emit the detected bounding box, a reference point, and descriptive textual flag in preseribed JSON-like format. Compliance with the schema yields reward; deviations incur a penalty.
Experiment
4.1 Experimental Setup
We use Qwen2.5VL-78 I as our base model and SAM2-Large 29] as the segmentation model. All experiments are conducted on Ax GPUs. During training, we sample responses per question, set Ehigh 0.3, and use a learning rate of 1.0 10-6, 6 To ensure the model's robustness across different domains, we resize images to 840 before feeding them into the MLLM during both training and evaluation. We follow previous works [16 23 and use both cloU and gloU as evaluation metrics. gloU defined by the average of per-image intersection-over-uiiono while cloU is defined by the cumulative intersection over the cumulative union.

---

4.2 Datasets
48 which contains 104,560 referring expressions tied to 54,822 objects across 26,711 images. We use the official RefCOCOg test set as our in-domain evaluation set. To assess generalization across datasets, we use the testA subsets from RefCOCO and RefCOCO+ 14] as our out-of-distribution (OOD) evaluation sets. RefCOCO consists of 142,210 expressions for 50,000 objects across 19,994 images, while RefCOCO+ includes 141,564 expressions for 49,856 objects in 19,992 images, with both datasets providing predefined splits. RefCOCO+ is considered more challenging due to the exclusion of absolute location terms. In addition, we include ReasonSeg [16], a dataset that requires strong visual- linguistic reasoning, to further evaluate our model's ability to perform fineggriiedd segmentation under complex reasoning conditions.
For training, we randomly sample 3,000 instances from the training set RefCOCOg
4.3 Main Results
ReasonSeg. Table I shows the zero-shot performance of SAM-R1 the ReasonSeg benchmark. Our method achieves 60.2% gloU and 54.3% cloU on the test set, outperforming the previous best, Seg-Zero (58.3% gloU and 53.4% cloU). This improvement mainly due our fine-grained reward design, which integrates SAM into the RL loop to provide loU-based feedback during training, aligning reasoning with segmentation. Unlike Seg-Zero's 's decoupled design, our unified framework introduces finer- snerrggriied segmentation rewards, enabling stable optimization and better generalization with only 3k training samples. Additionally, our improved GRPO strategy wwith asymmetric clipping and token-level loss normalization enhances informativeness and robustness under domain shifts, supporting SAM-RI's strong zero-shot performance in complex reasoning segmentation. Seg-Zero- 7B* denotes performance based on provided model weights, as their reported results used different weights per metric and could not be reproduced.
Table 1: Comparison on ReasonSeg-zero-shot benchmark (val/test). The best results are in bold.
ReasonSeg-zero-shor
Method
val
test
gloU
cloU
gloU
cloU
OVSeg
18
28.5
18.6
26.1
20.8
ReLA
19
22.4
19.9
21.3
22.0
Grounded-SAr
30
26.0
14.5
21.3
16.4
LISA-BB-LLaVAII
[16
53.6
52.3
48.7
48.8
LISA-I3B-LLaVAI. [6]
57.7
60.3
53.8
50.8
SAM44LLMM 4
46.7
48.1
Seg-Zero-7B*
62.0
52.0
58.3
53.4
SAM-R1 (Ours)
64.0
55.8
60.2
54.3
Referring Expression Segmentation. Our evaluation results on the Referring Expression Segmen- lation datasets are shown in Table 2 We use the testA subsets of RefCOCO and ReiCOCO+ as OOD test sets, and the test set of RefCOCOg as the in-domain test set. It can be seen that our model, trained on only 3,000 samples, still achieves competitive performance compared to prior methods. Specifically, on the in-domain dataset RefCOCOg, our algorithm SAM-RI is only 0.2 points lower than Seg- Zero, despite using fewer style-consistent training samples. On the OOD datasets, our model performs comparably to Seg-Zero on RefCOCO, and improves the performance on RefCOCO+ from 73.9 to 74.7. This demonstrates the effectiveness of our approach SAM-RI. We attribute this improvement to the fine-grained reward mechanisms and the flexible exploration strategy, which allows our model to surpass previous out-of-domain performance with significantly less training data.
4.4 Visualization Analysis
As shown in Figure 3 we present some representative cases to analyze the reasoning and segmentation performance of our model in diverse scenarios.
Multiple Subjects with Fine-Grained Segmentation. In certain situations, it necessary to identify specific subject among multiple subjects. For example, identifying Santa Claus amidst little girl,
7

---

Table 2: Performance comparison on referring expression benchmarks using cloU,
Method
refCOCO refCOCO+
refCOCOg
LAVT
75.8
68.4
62.1
ReLA
19
76.5
71.0
66.0
LISA-7B
[6
76.5
67.4
68.5
PixelLM-7B
31
76.5
71.7
70.5
PerceptionGP11-7B
28
78.6
73.9
71.7
Seg-Zero-7B* 23
79.2
73.9
73.3
SAM-R1 (Ours)
79.2
74.7
73.1
chairs, Christmas trees, and various decorations, each of which is complex and numerous. The model utilizes cues, such as red clothing and the act of listening to wishes, to successfully identify and segment Santa Claus.
Global To Local Reasoning. scenes containing rich local details, identifying specific part from the overall structure is highly challenging. For example, in an image of an airplane composed of various components, our model accurately locates the engine by reasoning over the spatial relationship between the engine and the wings.
Challenging Environment With Distractors. In cluttered environments, such as an airport filled with various signs, identifying a specific sign, such as "Watch Your Step", poses significant challenges. Our model effectively distinguishes the target sign from visually similar ones by leveraging contextual reference objects and localizing the identification process step by step.
Complex Boundaries. For complex boundaries, such those found in gymnastics competitions, the model integrates textual and visual information to infer that gymnastics involves specific movements. This understanding suggests the of vaulting table, which in turn facilitates the generation of coherent segmentation masks.
4.5
Ablation Study
In this section, we validate the effectiveness of the proposed components. As shown in Table the tiered threshold strategy demonstrates superior performance compared to fixed thresholds across both in-domain and OOD benchmarks. While fixed thresholds of 0.5, 0.7, and 0.8 achieve 56.5-58. 6 gloU the ReasonSeg-zero-sho (test), the dynamic tiered approach significantly outperforms them with 60.2 gloU (+3.5% absolute improvement). This performance gap highlights the limitations of static thresholds in handling complex reasoning scenarios, where overly conservative predictions at high thresholds (e.g., 0.8) degrade cloU performance despite improved localization precision. The tiered mechanism's phased threshold adjustment seems more effectively balance precision-recat trade-offs, particularly enhancing OOD generalization, as demonstrated by its 75.4 gloU score on the refCOCOg test set, which 0.8% higher than the best fixed threshold.
Table 3: Ablation study different threshold strategies under ReasonSeg-zero--tee (test) and refCOCOg-test benchmarks.
Method
ReasonSeg (gloU)
ReasonSeg (cIoU)
refCOCOg (gloU)
refCOCOg (cloU)
0.5
56.5
51.9
74.7
72.8
0.7
56.2
51.6
74.9
72.6
0.8
58.6
50.8
74.6
71.9
Tiered
60.2
54.3
75.4
73.1
We further analyze the algorithmic components presented in Table validate the effectiveness of token-level constraints and the use of an asymmetric clipping strategy. The token-level reward mechanism yields consistent improvements across various metrics, enhancing ReasonSeg cloU by 0.5% (from 51.2% to 51.7%) and refCOCOg cloU by 0.6% (from 71.8% to 72.4%) through fine- grained output format regulation. Meanwhile, increasing the upper clipping threshold in our GRPO

---

User: During the he holiday scason, childrer
Thinking: The traditional figure assacicted with the holiday season, known
of hen have the apportuntt conit somsone lop share wishes. Who
for weering red suit and sitting on throne-like chair to listion to
wishes image there a perzon
suit traditionaly ly
white beard, sittingg
oligns with description
chair
placed the figure identified the forget dbject
Clous.
cornectly
to listenn children's withest
Thinking The person who is not liddl be player In this image, the person
on the for left is wecring a jocket and jeans, shich are not typical of affietic
User: The person who not likell to player
ottine, and is standing with o more formal posture, Siggosting they not
player. The other individuals
wearing athletic clething oppeor be
typical player bounding box drown indicate they
left. points
on person's
object.
Thinking: engines the aircreft, which are typioally locoted under the
wings, bounding box is drown encompazs onco
wings
User: The engines for this plans.
where ongines usuelly mounted, and points one placed on the bounding they
reprecient the engines. points the ongines if they
User:
When we are a coffee shop and want to enjoy our drink while walking arsund, what objest jest the picture beverage socuraly?
while walking abject quoction the oup held in person's hand.
Thinking:
The object in the image that used to hald a beverage securely
which designed
and cornidd comfortably. The bounding box contered the cup,
use
points are placed on cup to indicame
forget sbjuct.
Thinking sign that specifically seys Witthh Step't omong the the the and distinct lscoted placement. The bounding box drewn around this and points placed on the indicate presence.
User: The zign that reminds people to watch their
image. sige
User: gymnestics competitions, otiletes perform voriety acrebatic movements
The apporctus used gynnestics for perforning flips voutss The image shows gymnest mid-air, suggesting the of voulting tob which is common picce of equipment gymmastics for such movemants The bounting bax is centered on the apporatus, us and the points are placed en the surface of the table which the area used for vaulting.
Thinking
on different apporatus. What the picture
could be used for othletes nss to perform
imreessiv and challenging novements such
as flps and voults?
Thinking:
The two blue motorcycles that are tied together with choin. The bounding box drown to encompass both netorcycles, ensurtng that the thain connecting then is included. Points are placed on the motorcycies to indioste ther presence, with volux for paints the menorcycles points the surrouncinn onea.
User: The notorcycles fied together.
User: In formal attire, people often wear block accesstry around ther neck add Touch elegonce. Whet object in picture
Thinking The black accessory typically worn around the neck: for elegance. which i the In the imgge the dog weoring o red bow the which is type of placement cround nack points placed farget object.
the closest match
in terma
aooeswery?
indicate
Figure 3: Qualitative results on ReasonSeg [16] demonstrate that SAM-RI exhibits robust zero-shot performance, further enhanced by the chain-of-though approach with improved reasoning capacity.
variant provides more flexibility in updating highly advantageous actions, which proves especially beneficial in OOD reasoning tasks. This adjustment improves ReasonSeg gloU by 1.3%, compared to a 0.8% gain on refCOCOg, suggesting that such flexibility more impactful in addressing complex reasoning challenges. Notably, combining both techniques yields synergistic effect, raising ReasonSeg cloU to 54.3%, a 3.1%% % improvement over the GRPO baseline. The full method achieves peak gloU scores of 75.4 on the refCOCOg test set and 60.2 on ReasonSeg, demonstrating the effectiveness of jointly enforcing fine-grained output structure and reward-sensitive policy adaptation.
Table 4: Ablation study of algorithmic components based on the GRPO baseline on ReasonSeg-zero- shot and refCOCOg-tess
Method
Token-level
CLIP higher
gloU (RS)
cloU (RS)
gloU (Reg)
cloU (Reg)
GRPO
x x
57.8
51.2
74.1
71.8
Token-level
x
58.0
51.7
74.5
72.4
Clip higher
x
V
59.1
52.8
74.9
72.5
Ours
v
60.2
54.3
75.4
73.1
4.6 Generalization to REC task
Although our model is not trained on any Referring Expression Comprehension (REC) datasets, we observe strong performance on REC task, thanks to the model's enhanced reasoning ability and fine-grained perceptual capabilities. As shown in Table our method, SAM-R1, achieves state-of- the-art performance on the ASS--rGonn benchmark with 63.8, significantly surpassing previous methods such GroundedSAM (26.2), OV-Seg (28.4), X-Decoder (28.5), and Visual-RFT (43.9).

---

Model
LISA-Grounding
GroundedSAM
26.2
OV-Seg
28.4
X-Decoder
28.5
Visual-RFT
43.9
SAM-RI(Ours)
63.8
Table 5: Performance comparison the LISA- AroA-Gruun benchmark. Our method significantly outperforms prior vve--oocayllary and vision-language segmentation approaches, demonstrating strong generalization ability on reasoning-intenss REC tasks.
This substantial improvement demonstrates the effectiveness our reinforcement learning- baaeed reasoning framework in complex visual grounding tasks. Unlike prior approaches, which often rely on large-scale supervised training or handcrafted prompt engineering, our method leverages task-aligned rewards and structured reasoning supervision to enable fine-grained object understanding and robust generalization in sessoning-intessvee scenarios. These results demonstrate the generality and adaptability of our method beyond segmentation, highlighting its strong alignment capabilities and transferability to challenging REC scenarios.
4.7 Broader Impact and Discussion
Our work shows that reinforcement learning, guided by a segmentation model, can effectively cultivate reasoning multimodal models. The strong performance of SAM-RI with only 3,000 training samples highlights promising path toward data efficiency. By using standard segmentation masks as the supervisory signal, our approach bypasses the need for costly and potentially biased, manually annotated reasoning chains, thus enhancing scalability. More broadly, this study supports a paradigm where models learn complex reasoning from task-aligned rewards rather than explicit instructions. This shift toward learning from weaker, accessible supervision is particularly impactful for domains with scarce reasoning data, such as robotic perception and medical image analysis.
We recognize several limitations for future work. First, SAM's parameters remain frozen, creating a one-way information flow that prevents it from adapting to the reasoning model. Jointly optimizing both models is a compelling next step. Though computationally demanding. this could foster a synergistic alignment where the models co-adapt. Second, our model struggles generate meaningful negative reference points, key capability for robust discriminative reasoning. Our RL framework failed to encourage this, suggesting a foundational limitation that may require new architectural or algorithmic solutions to improve robustness in complex visual scenes.
5 Conclusion
In this paper, we present SAM-RI, an innovative framework that leverages reinforcement learning to enhance the reasoning capabilities of multimodal large models for image segmentation. Our method introduces fine-grained segmentation settings into the training process, enabling more precise and task-relevant reasoning. Furthermore, propose task-specifie, fine-grained reward design that incorporates the Segment Anything Model (SAM) as a flexible and reliable reward provider. By integrating these components with tailored optimization objective, SAM-R1 achieves strong performance using only 3,000 training samples, demonstrating the practicality and effectiveness of reinforcement learning in this domain. This work not only contributes advancing multimodal image segmentation but also highlights the potential of reward-guided learning for developing more efficient and adaptable multimodal large models.
Acknowledgements
This work was supported by the STI 2030-Major Projects under Grant 202 1D00014444
10

---

References
[1] Bai, S., Chen, K., Liu, X., Wang, J., Ge, W. Song. S., Dang. K., Wang, P., Wang. S., Tang, J., et al.: Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923 (2025)
[2] Bai, Z., He, Mei, Wang, P., Gao, Z., Chen, Zhang, Z., Shou, M.Z.: One token seg them
all: Language instructed reasoning segmentation in videos. Advances in Neural Information Processing Systems 37, 6833-6859 (2024)
[3] Chen, X., Li, G., Wang, Z., Jin, B., Qian, C., Wang. Y., Wang, H. Zhang, Y., Zhang, D., Zhang, T., et al.: Rm-rl: Reward modeling reasoning. arXiv preprint arXiv:2505, 023887 (2025)
[4] Chen, Y.C., Li, W.H., Sun, C., Wang, Y.C.F., Chen, C.S.: Sam4mllm: Enhance multi modal large language model for referring expression segmentation. In: European Conference Computer Vision. pp. 323 340. Springer (2024)
Chen, Z., Wang, W., Cao, Y., Liu, Y., Gao, Z., Cui, E., Zhu, J., Yc, S., Tian, Liu, Z., al.: Expanding performance boundaries open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271 (2024)
[6] Chen, Z., Wang, W. Tian, H., Ye, S., Gao, Z., Cui, E., Tong, W., Hu, K.,, Luo, Ma. Z., et al.: How far are to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences 67(1)). 220101 (2024)
[7] Chu, T., Zhai, Y., Yang, J. Tong, S., Xie, S., Schuurmans, D., Le, Q.V., Levine, S., Ma. Y:: Stt memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501.17161 (2025)
[8] Deng. Y., Bansal, H., Yin, F., Peng, N., Wang, W., Chang, W.: Openvithinker: An early exploration to complex vision-language reasoning via iterative sell-improvement, arXiv preprint arXiv:2503.1735 17352 (2025)
[9] Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q. Ma, S., Wang, P., Bi, X., et al.: Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025)
[10] Hu, J.. Lin, Gong. S., Cai, W.: Relax image-specific prompt requirement in sam: A single generic prompt segmenting camouflaged objects. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 12511-1251 1-12518 (2024)
[11] Hu, J., Lin, J.. Yan, J.. Gong, S.: Leveraging hallucinations to reduce manual prompt dependency in promptable segmentation. arXiv preprint arXiv:2408. 1.0.212088.. 15205 (2024)
[12] Huang, J., Xu, Z., Liu, T., Liu, Y., Han, H., Yuan, K., Li, X.: Densely connected parameter-efficient tuning for referring image segmentation. arXiv preprint arXiv:2501 .08580 (2025)
[13] Jaech, A.. Kalai, A.. Lerer. A., Richardson, A., Ei-Kks,,y, A., Low, A., Helyar, A., Madry. A., Beutel, A.. Carney, A., et al.: Openai ol system card. arXiv preprint arXiv:2412.167700 (2024)
[14] Kazemzadch, S., Ordonez, V., Matten, M., Berg, T.: Referitgame: Referring to objects in photographs of natural scenes. Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 787-798 (2014)
[15] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L.,, Xiao, T., Whitehead, S., Berg, A.C., Lo. W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4015-4026 (2023)
[16] Lai, X., Tian, Z., Chen, Y., Li, Y., Yuan, Y., Liu, S., Jia, J.: Lisa: Reasoning segmentation via large language model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9579 9589 (2024)
[17] Li, J., Li, D., Savarese, Hoi, S.: Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In: International conference on machine learning. pp. 19730-19742. PMLR (2023)
[18] Liang, F., Wu, B.. Dai. X., Li, K., Zhao, Y., Zhang. H., Zhang. P., Vajda, P., Marculescu, D.: Open- vocabulary semantic segmentation with mask-adapted clip. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 7061 7070 (2023)

---

[19] Liu, C., Ding, H., Jiang, X.: GRES: Generalized referring expression segmentation. In: CVPR (2023)
[20] Liu, H., Li, C., Wu, Q. Lee, YJ.:: Visual instruction tuning. Advances in neural information processing (2023)
systems 36,
[21] Liu, T., Liu, X., Huang, S., Shi, L.. Xu, Z., Xin, Y., Yin, Q. Liu, X.: Sparse-tuning: Adapting vision transformers with efficient fine-tuning and inference. arXiv preprint arXiv:2405. 14700 (2024)
[22] Liu, T., Xu, Z., Hu, Y., Shi, L.. Wang, Z., Yin, Q.: Mapper: Multimodal prior-guided parameter efficient tuning for referring expression comprehension. arXiv preprint arXiv:2409.1 13609 (2024)
[23] Liu, Y., Peng, B., Zhong, Z., Yue, Z., Lu, F., Yu, B., Jia, J:: Seg-zero: Reasoning-chain guided segmentation via cognitive reinforcement. arXiv preprint mtXiv:2503.0655 (2025)
[24] Liu, Z., Chen, C., Li, W. Qi, P., Pang, T., Du, C., Lee, W.s., Lin, M.: Understanding rl rl-zero-like training:
A critical perspective. arXiv preprint arXiv:2503.2072 20783 (2025)
[25] Luo, Z., Xiao, Y.. Liu, Y., Li, S., Wang. Y., Tang, Y., Li, X., Yang. Y:: SOC: semantic-assisted object cluster for referring video object segmentation. In: NeurlPS (2023)
[26] Ma, Y., Wang. Y., Wu, Y., Lyu, Z., Chen, S., Li, X., Qiao, Y.: Visual knowledge graph for human action reasoning in videos. In: Proceedings of the 30th ACM International Conference on Multimedia. pp. 4132 4141 (2022)
(27] Niu, Y., Tang, K., Zhang, H., Lu, Z., Hua, X.S., Wen, J.R.: Counterfactual vqa: A cause-effect look at language bias. In: Proceedings of the IEEE/CVF conference computer vision and pattern recognition. pp. 12700-12710 (2021)
[28] Pi, R., Yao, I.., Gao, J., Zhang, J., Zhang, T.: Perceptiongpt: Effectively fusing visual perception into Ilm (2023)
[29] Ravi, N., Gabeur, V., Hu, Y.T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., Mintun, E., Pan, J., Alwala, K.V., Carion, N., Wu, C.Y., Girshick, R., Dollár, P., Feichtenhofer, C.: Sam 2: Segment anything images and videos, arXiv preprint arXiv:2408.0077 007004 (2024), lttps //axxiv. .rrivi.orr//ao ab2// 2000000000000
[30] Ren, T., Liu, S., Zeng, A., Lin, J., Li, K., Cao, H., Chen, J., Huang, X., Chen, Y., Yan, F., Zeng. Z., Zhang, H., Li, F., Yang. J., Li, H., Jiang, Q., Zhang, L.: Grounded sam: Assembling open-world models for diverse visual tasks (2024)
[31] Ren, Z., Huang. Z., Wei, Y., Zhao, Y., Fu, D., Feng, J., Jin, X.: Pixellm: Pixel reasoning with large multimodal model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 26374-26383 (2024)
[32] Schulman, J., Wolski, F. Dhariwal. P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv preprint 1IX.:i100.006444 (2017)
[33] Shao, Z., Wang, P., Zhu, Q. Xu, R., Song, J. Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al.: Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 (2024)
[34] Shen, H., Liu, P. J.,
J., Fang, C., Ma, Y., Liao, J., Shen, Q., Zhang, 7., Zhao, K., Zhang, Q., et al.: Vlm-rl: A stable and generalizable rl-stycc large vision-language model. arXiv preprint arXiv:250M.07615 (2025)
[35] Tang, L., Jiang, P.T., Shen, Z.H., Zhang. H., Chen, J.W., Li, B.: Chain of visual perception: Harnessing multimodal large language models for zero-shot camouflaged object detection. In: Proceedings of the 32nd ACM international conference on multimedia. pp. 8805-8814 (2024)
[36] Wada, Y., Kaneda, K., Saito, D.,. Sugiura, K.: Polos: Multimodal metric learning from human feedback for image captioning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13559-13568 (2024)
[37] Wang, P. Bai. S., Tan, S. Wang, S., Fan, Z. Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al.: Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191 (2024)
[38] Xiao, Y., Luo, Z., Liu, Y. Ma, Y., Bian, H., Ji. Y., Yang. Y., Li, X.: Bridging the gap: A unified video comprehension framework for moment retrieval and highlight detection. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18709-1888 18719 (2024)
12

---

[39] Xiao, Y., Song. L., Chen, Y., Luo, Y., Chen, Y., Gan, Y.. Huang. W. Li, X., Qi. X., Shan, Y: Mindommi: Unleashing reasoning generation in vision language models with rgpo. arXiv preprint Stdi::2505.1303 (2025)
[40] Xiao, Y., Song. L.. Wang. J.. Song. S., Ge, Y.. Li. X., Shan, Y., al.: Mambatree: Tree topology is all you need in state space model. Advances in Neural Information Processing Systems 37, 75329-75354 (2024)
[41] Xiao, Y., Song, L.,, Yang, R., Cheng, C.,, Ge, Y., Li, X., Shan, Y.: Lora-gen: Specializing large language model via online Iora generation. arXiv preprint arXiv:2506.1 1163066.16668 (2025)
[42] Xu, Z... Chen, Z., Zhang. Y., Song. Y., Wan, X.,, Li, G.: Bridging vision and language encoders: Parameter- efficient tuning for referring image segmentation. In: Proceedings the IEEE/CVF internntiii conference on computer vision. pp. 17503-175 (2023)
[43] Xu, Z., Huang. J., Liu, T., Liu, Y., Han, H., Yuan, K., X.: Enhancing fine- Line-grained multi modal alignment via adapters: a paraaeteer-fffcciet training framework for referring image segmentation. In: 2nd Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ ICMI. 2024) (2024)
[44] Yang, R., Song, L., Xiao, Y., Huang, R., Ge, Y., Shan, Y. Zhao, H.: Haplovl: A single-transforerr baseline for multi-modas understanding. arXiv preprint arXiv:2503.1460 (2025)
[45] Yang, S., Qu. T.. Lai, X., Tian, Z., Peng. B., Liu, S., Jia, J.: Lisa+ An improved baseline for reasoning segmentation with large language model. arXiv preprint arXiv:2312. (2023)
[46] Yang. Y., He, X., Pan, H., Jiang, X.. Deng, Y.. Yang. X., Lu, H., Yin. D., Rao, F., Zhu, M.. et al.: RI- onevision: Advancing generalized multimodal reasoning through cross- modal formalization. arXiv preprint arXiv:2503. 10615 (2025)
[47] Yang, Z., Wang, J., Tang, Y., Chen, K., Zhao, H., Torr, P.H.: Lavt: Language-aware vision transformer for referring image segmentation. In: CVPR. pp. 18155 18165 (2022)
[48] Yu, L., Poirson, P., Yang. S., Berg, A.C., Berg. T.L.: Modeling context in referring expressions. In: Computer Vision-ECCV 2016: European Conference, Amsterdam, The Netherlands, October 11-14. 2016, Proceedings, Part II 14. pp. 69-85. Springer (2016)
[49] Yu. Q.. Zhang, Z., Zhu, R., Yuan, Y., Zuo, X., Yue, Fan. T., Liu, Liu, L., Liu, X., al.: Dapo: open-source IIm reinforcement learning system at scale, arXiv preprint arXiv:2503. 2500..143366 (2025)
[50] Zhang, Y.F., Lu, X., Hu, X., Fu, C., Wen, B., Zhang, T., Liu, C., Jiang, K., Chen, K., Tang, K., et al.: R1-reward: Training multimodal reward model through stable reinforcement learning. arXiv preprint arXiv:2505 028835 (2025)
[51] Zhou, J., Li, J.. Xu, Z., Li. H., Cheng, Y., Hong, FT. Lin, Q., Lu, Q., Liang, X.: Fireedit: Fine- Grainer instruction- based image editing via region-aware vision language model. arXiv preprint ar arXiv:2503. 19839 (2025)
13

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]. .
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Overall, the abstract and introduction provide a concise yet comprehensive sum- mary of the paper's objectives, methods, and findings, accurately reflecting its contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
14

---

Justification:
Yes, we discuss the limitations in the Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: By adhering to the principles mentioned in the Guidelines, we ensure that each theoretical result underpinned by full set of assumptions and complete, correct proofs, thus reinforcing the credibility and reliability of the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: In the experiments section in the main text, we report all the experiment settings, implementation details, and metrics, which disclose all the information needed to reproduce the main experimental results.
15

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel urchitecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification:
The code will be made available after being accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
www.obiiiiiiioooooioiiooo for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
16

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In the experiments section in the main text, we report all the experiment settings, implementation details, and metrics, which disclose all the information needed to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: N/A.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the computer resources in the implementation details of the experi- ment section.
17

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttps://neuripssttt httpp:///aaaaps..cpppoooccccccaa
Answer: [Yes]
Justification: Yes, the research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: Yes, discuss both potential positive societal impacts and negative societal impacts of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
18

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly cite the original owners of code, data and models.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pwperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
19

---

14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper has no crowdsourcing experiments and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: N/A.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The LLM is used only for formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
20

---

A Technical Appendices
(a) Training collapse without KI. (b) Negative points placed outside constraint.
(c) All points clustered on the object when restricted to box.
bounding boxes.
Figure 4: Ablation study failures: (a) Removing the KL. constraint leads to training instability and collapse. (b) Encouraging both positive and negative point generation causes negatives appear outside target areas. (c) Forcing all points into the bounding box eliminates useful contrast, reducing performance.
A.1 Ablation Failure: Removing the KL Constraint
During the development of our method, we explored various strategies to encourage broader explo- ration by the model. One such attempt involved removing the KL divergence constraint, which is commonly used to regularize policy updates and limit deviation from the reference distribution.
However, empirical results showed that eliminating the KL term led to significant instability during training. As illustrated in Figure 4a the model initially exhibited effective learning behavior with a strong exploratory signal. Yet, after approximately 100 training steps, observed sharp fluctuations in performance, eventually leading to complete collapse of the training process.
This outcome indicates that the KI. constraint plays crucial role in maintaining training stability, especially in our multimodal reasoning setting. Consequently, we decided to retain the KL divergence term in our final framework, despite its potential to limit aggressive exploration.
A.2 Ablation Failure: Encouraging Negative Reference Points
In designing the reward function, we initially allowed the multimodal large model to freely determine the value of the reference point-positive (1) or negative (0)-without explicit supervision. However, we observed that the model strongly preferred generating only positive points, rarely including any negatives. We hypothesized that incorporating both positive and negative points could provide richer target information and improve segmentation performance.
To encourage this behavior, awarded point when both 0 and 1 values appeared in the output. As shown in Figure this led the model include both types points. While the positive points remained well-aligned with the target object, the negative points were typically placed at the image boundaries, far outside the bounding box, offering no useful contrast for object discrimination.
introduced a format-based reward component, point value, which
We then modified the rule to grant the reward only when both positive and negative points were located within the bounding box. As shown in Figure 4c| this adjustment led to all roints-regardless of label-being clustered directly on the target object, effectively eliminating the intended contrast and introducing noise instead.
These results suggest that, despite reward incentives, the multimodal large model lacks the inherent ability identify meaningful negative examples in visual space. Therefore, we decided not to enforce negative point generation in our final design.
A.3 Failure Analysis
As illustrated in Figure 5] we present several typical failure cases, which mainly highlight two issues: incomplete segmentation and over-segmentation.
21

---

Prompt
Image
Ground Truth
Thinking
Predict
Thinking ctya, fechury or motir padinnet of coube deas are them a decaratis henghs from the ooher
grond building cireiica
User Samething thet prevers
building
isto the bu liinng the mend fencs in fromt the entrance by
Fes foree directly from the enriecce, which ic
tng fover market some posnts wftee entare etject carrectly Marr
nocoles reage shows nod oxtrogashers
User: In code a file, ihie
indistnnl
bodigrouss Fire
compoother are speatically des gree arons ease
saficty what
abberrecee iagh, cling
jajest picture
teut synos than conforee pockyoent is ert and cafeny eqe perett and sen cicsely
destned aters
and releads fire extrggaderg
notched tarqet objects the ctune.
Taiskig mape showr black sports can wits open comparteant inta compeeaete
con est cisse to the duployed Fectures, what part viewere the ingina comportnant?
car sheechs los vioturt ln let
practice
picure
bec recegreand. and the open head cleary vortle, ind roliating that within the boanding bax are marned to nighighh the open hood shich rigget aojer iage.
Figure 5: Visualization of some failure cases for our SAM-RI method on the ReasonSeg- val dataset, which shows that our approach still has some limitations.
notable observation is that our SAM-RI, through its Thinking process (fourth column), successfully comprehends the prompt and correctly localizes the target object(s). However, this correct semantic understanding does not always translate perfectly into the final segmentation mask. For example, in cases of incomplete segmentation: In the first row, the model correctly identifies the fence, but the Predict mask only covers a small portion of the target. In the second row, the model recognizes The fire extinguishers but incorrectly segments only one of the two instances. Conversely, over- segmentation is demonstrated in the third row: The prompt asks for part of the car (the hood), and the Thinking process also pinpoints the open hood. However, the model incorrectly segments the entire vehicle instead of just the specified part.
These examples indicate that, while our model performs well in high-level semantic reasoning, limitations still exist in its ability to precisely map this understanding to pixel-level masks, particularly concerning fine-grained segmentation and instance completeness. This remains key area for future improvement.
A.4 Data Efficiency and Scalability Analysis
To investigate the scalability and data efficiency of SAM-R1, we conducted additional experiments by increasing the size of the training data from 3k to 10k. The results clearly show that our method is highly data-efficient, with performance saturating just 3k samples.
We present the direct comparison in Table As shown, increasing the data to 10k results in negligible fluctuations in ReasonSeg: the cloU shifts slightly from 55.8 to 55.5 on the val split and from 54.3 to 53.9 on the test split. Similarly, on the RefCOCO benchmarks, we observe only marginal gains, which strongly indicates that performance has already plateaued.
Table 6: Data efficiency analysis with 3k vs. 10k training samples.
ReasonSeg
RefCOCO Benchmarks
Method
Val
Test
refCOCO
refCOCO+ refCOCOg
SAM-RI (Ours, 3k)
55.8
54.3
79.2
74.7
73.1
SAM-RI (Ours, 10k)
55.5
53.9
79.9
75.3
73.5
Gain
-0.3
-0.4
+0.7
+0.6
+0.4
From these results, it evident that our method's core performance saturates at 3k samples. Given the substantial increase in training cost versus the minimal performance returns, we deliberately chose 3k samples as the optimal trade-off point for demonstrating our method's capabilities.
22


Reviews:
[{"review_id": "hbLIPpacVx", "reviewer": "Reviewer_jxxo", "summary": "This paper introduces SAM-R1, a novel framework for enhancing multimodal large language models' (MLLMs) reasoning segmentation capabilities via reinforcement learning (RL). The method uniquely incorporates the Segment Anything Model (SAM) as an active reward provider within an RL loop, offering fine-grained, segmentation-aware feedback. SAM-R1 is trained with only 3K samples, yet achieves state-of-the-art performance on ReasonSeg and competitive performance on referring expression segmentation tasks (RefCOCO, RefCOCO+, RefCOCOg).", "questions": "1. Comparison with PPO\n\nCan the authors add or discuss results comparing GRPO-based SAM-R1 with standard PPO or RLHF strategies? This would better justify the algorithmic choice beyond citing past literature.\n\n2. Data Efficiency vs. Scale\n\nSAM-R1 performs well with 3K samples. Can the authors comment on how performance scales with larger datasets? Is the approach still competitive under full-data settings?\n\n3. Failure Case Analysis\n\nAre there common failure modes in either reasoning or segmentation (e.g., hallucinations, over-segmentation)? A brief discussion or visualization would strengthen trust in robustness.", "limitations": "Failure case analysis is not included in the paper.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- Novel Integration of SAM as Reward Provider\n\nRather than passively using SAM for downstream inference, this work integrates SAM directly into the RL training loop, allowing reasoning quality to be directly shaped by segmentation accuracy.\n\n- Compact yet Effective Training\n\nAchieves strong zero-shot performance on complex reasoning-segmentation tasks with only 3,000 training samples.\n\n- Well-Designed Reward System\n\nThe tiered reward function based on IoU allows for dense and scalable feedback. Additional format and structure rewards (for reasoning and segmentation) further guide training.\n\n- Strong Experimental Results\n\nOutperforms or matches prior art like Seg-Zero across multiple benchmarks (e.g., +1.9 gIoU over Seg-Zero on ReasonSeg-zero-shot test). Ablations confirm the effectiveness of each design.\n\n- Clarity and Organization\n\nThe method is clearly motivated, the framework is well-illustrated (Figure 2), and the paper includes extensive implementation and training details.\n\n## Weaknesses\n- Lack of Direct Comparison with Alternative RL Strategies\n\nWhile GRPO is well explained and modified, it would be helpful to see direct comparisons with more traditional RL methods (e.g., PPO) under the same setting.\n\n- Missing User-Centric Evaluation\n\nNo human evaluation or qualitative user study is conducted to assess reasoning interpretability, mask quality, or real-world usability.", "was_revised": true, "final_justification": "borderline accept", "rebuttal": "1. Regarding the comparison with other RL strategies\n\nThank you for this insightful question. Our decision was based on a principled evaluation of efficiency and stability, particularly within the resource-intensive context of VLM fine-tuning. In this setting, GRPO presents clear advantages over standard approaches like PPO:\n\n●Computational and Resource Overhead: PPO and its mainstream implementations typically rely on a separate value network to estimate state values and use Generalized Advantage Estimation (GAE) to reduce variance. This not only increases the number of parameters but also leads to significant computational and memory overhead. In contrast, GRPO cleverly omits the value network, estimating advantages solely through the normalization of group-wise responses. This allows it to achieve high sample efficiency with lower resource consumption, making it more suitable for fine-tuning large models like VLMs.\n\n●Stability and Hyperparameter Tuning: To prevent excessively large policy updates that can lead to training collapse, PPO introduces complex mechanisms like Clipped Probability Ratios and/or KL divergence penalties. The design of GRPO's loss function is more elegant, as it naturally integrates these stabilization mechanisms into a single objective function. This not only simplifies hyperparameter tuning but also further enhances overall training stability.\n\nWe will add a detailed discussion in the Methods section to articulate these points, thereby strengthening the justification for our deliberate algorithmic choice.\n\n2. Regarding data efficiency and scale\n\nThank you for your excellent question concerning the model's scalability. To investigate this, we conducted additional experiments by increasing the training data size from 3k to 10k. The results clearly show that our method is highly data-efficient, with performance saturating at 3k samples. We are pleased to present the direct comparison below.\nOn the ReasonSeg dataset, performance remains stable with no significant changes:\n\n| Method                | Split | gIoU | cIoU |\n| :-------------------- | :---- | :--: | :--: |\n| SAM-R1 (Ours, **3k**) | val   | 64.0 | 55.8 |\n| SAM-R1 (Ours, **3k**) | test  | 60.2 | 54.3 |\n| SAM-R1 (Ours, **10k**) | val   | 64.2 | 55.5 |\n| SAM-R1 (Ours, **10k**) | test  | 60.8 | 53.9 |\n\nAs shown in the table, increasing the data to 10k results in negligible fluctuations on ReasonSeg. The test gIoU slightly moves to 60.8 (+0.6) while the cIoU shifts to 53.9 (-0.4), strongly indicating that performance has already plateaued.\nOn the RefCOCO benchmarks, we observe only marginal gains, all under 1 percentage point:\n\n| Method                | refCOCO | refCOCO+ | refCOCOg |\n| :-------------------- | :-----: | :------: | :------: |\n| SAM-R1 (Ours, **3k**) |  79.2   |   74.7   |   73.1   |\n| SAM-R1 (Ours, **10k**) |  **79.9**   |   **75.3**   |   **73.5**   |\n| *Gain*                |  *+0.7* |  *+0.6*  |  *+0.4*  |\n\nThese results lead to a clear conclusion: our method's core performance saturates at 3k samples. Given the substantial increase in training cost versus the minimal performance returns, we deliberately chose 3k samples as the optimal trade-off point for demonstrating our method's capabilities. This finding is a strong testament to the excellent data efficiency and cost-effectiveness of our SAM-R1 framework, which we consider a core advantage of our work.\n\n3. On failure case analysis\n\nThank you for this important suggestion, which greatly enhances the rigor of our paper. To provide a comprehensive view of our method's capabilities and limitations, we have already included an analysis of \"Ablation Failures\" in the submitted appendix. For instance, we show that:\n\n●Removing the KL constraint leads to training instability.\n\n●An attempt to encourage the model to generate \"negative reference points\" was not effective. This existing analysis validates the rationale behind our current framework design.\n\nAs we cannot include new figures during the rebuttal phase, we have completed a qualitative analysis and commit to adding a dedicated \"Failure Case Analysis\" section in the final paper. We will focus on typical cases such as:\n\nReasoning‑Execution Disconnect: For instance, given the instruction “segment the apple closest to the camera,” the model’s reasoning chain correctly identifies that there are three apples and that the largest bottom one is nearest, yet the final segmentation mask erroneously includes an adjacent apple. This highlights remaining challenges in precise spatial localization and the conversion of textual reasoning into geometric masks.\n\nBy integrating our ablation findings with these new output failure analyses, we will offer readers a more honest and comprehensive evaluation of SAM‑R1. Thank you again for this valuable recommendation."}, {"review_id": "2rgOIZS1hY", "reviewer": "Reviewer_WuMV", "summary": "This paper proposes the SAM-R1 framework, aiming to empower multimodal large models (MLLMs) with superior reasoning capabilities in image segmentation tasks through reinforcement learning (RL). Specifically, the authors calculate reward scores using masks generated by SAM and conduct RL training based on an improved GRPO algorithm to establish a dynamic feedback loop between reasoning and segmentation. The experimental section conducts comprehensive comparative validations on multiple benchmark datasets such as ReasonSeg and the RefCOCO series.", "questions": "1. The authors of this paper believe that the difference between SAM-R1 and Seg-Zero lies in the integration of reasoning and segmentation capabilities, forming a closed loop of \"reasoning-segmentation-reward\". However, Seg-Zero also uses SAM as a reward generator. Is the only difference that the reward form determination of SAM-R1 is divided into multiple levels?\n2. What is the size of the VLM in Seg-Zero used in the experiment? Since the original paper mentions 3B, it is not fair to make a direct comparison.\n3. If instead of using SAM, we follow works like PixelLM and make the segmenter a trainable part of the model, will the reinforcement learning training be more effective?", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The proposed reinforcement learning training pipeline for segmentation in the paper is concise and highly applicable.\n2. The paper demonstrates strong experimental performance, achieving excellent results with a small number of reinforcement learning samples.\n3. The experiments are well-detailed and presented in a highly readable manner.\n\nWeaknesses:\n1. The overall approach of the paper is relatively similar to works like Seg-Zero, and the degree of innovation is somewhat limited.", "was_revised": true, "final_justification": "The authors' response has addressed the questions I raised. I will keep my rating.", "rebuttal": "We thank the reviewer for the valuable feedback. Below, we address each point in detail:\n\n1. On novelty and distinction from Seg‑Zero\n\n Thank you for highlighting the perceived similarity to Seg‑Zero. We clarify that SAM‑R1 differs fundamentally from Seg‑Zero in its training paradigm. Although both methods leverage the SAM model, Seg‑Zero does not integrate SAM or any segmentation feedback during training. Instead, it preprocesses the segmentation task into an object detection problem and trains its vision–language model accordingly; the segmentation module itself is never part of a closed‑loop optimization. Consequently, Seg‑Zero cannot benefit from pixel‑level feedback, limiting its ability to learn fine‑grained visual detail.\n\nIn contrast, SAM‑R1 introduces a true reasoning–segmentation–reward loop during training. Our key innovations include:\n\n●SAM as an RL reward provider: We retain the SAM module in the RL loop and compute rewards based on the IoU between SAM’s output mask and the ground‑truth mask. This preserves spatial detail and lets the model learn directly from pixel‑level supervision without converting to a detection formulation.\n\n●GRPO objective modifications for fine‑grained multimodal tasks: We adapt the GRPO reward structure to include multi‑level, token‑wise feedback, increasing exploration and providing richer signals for reasoning chains.\n\n●Closed‑loop training for joint generalization and expressivity: Our reward design not only assesses segmentation quality but also serves as a semantic alignment signal for the reasoning path. This encourages the model to abstract the referred object from complex instructions and precisely localize it, driving co‑evolution of language understanding and spatial perception.\n\nThus, while both methods utilize SAM, SAM‑R1 is the only approach that jointly optimizes reasoning and segmentation in a fully closed loop during training, representing a fundamental shift in paradigm.\n\n2. On backbone model size fairness\n\n Thank you for this important question. We want to assure the reviewer that ensuring a rigorous and fair comparison was our top priority.\n\nAll comparisons between SAM-R1 and Seg-Zero in our paper use the identical backbone: Qwen2.5-VL 7B.\n\nWe took the step of running their officially released model ourselves for a crucial reason: during our initial review, we observed that the results reported in their paper were generated using different model weights for different metrics. This prevented a direct and consistent reproduction across all benchmarks from a single model.\n\nTherefore, to establish a truly controlled and fair comparison, we used their public weights to generate all Seg-Zero results with the same backbone and under the same evaluation protocol. This approach ensures that the performance differences we report are attributable to the methods themselves, not variations in experimental setup. We will emphasize this specific motivation in the revised manuscript to ensure full transparency.\n\n3. On making the segmenter trainable \n\nWe did consider integrating a trainable segmenter for end‑to‑end learning, as in PixelLM. However, we opted against this for two main reasons:Scalability and modularity: SAM is a powerful, promptable segmentation model validated across many tasks. By keeping it as a separate module and driving it with a reasoning-based prompt, we maintain flexibility to adapt to diverse downstream tasks without the overhead of fine-tuning the entire segmenter.\n\nTraining stability and efficiency: Credit assignment in RL is already challenging. Incorporating the segmenter into the RL loop would introduce additional noise from unstable segmentation outputs, demanding far more data and hyperparameter tuning to converge effectively. By focusing RL optimization on the VLM alone, we achieve more stable training and faster convergence while substantially improving reasoning and expressive capabilities."}, {"review_id": "6jEH39EVuF", "reviewer": "Reviewer_Cvwx", "summary": "This paper proposes an end-to-end multimodal segmentation framework, SAM-R1, which integrates the Segment Anything Model as a reward provider in reinforcement learning for reasoning-based segmentation tasks. The approach introduces fine-grained segmentation-aligned rewards and modifies the policy optimization objective to improve information density and training stability. Experimental results show that SAM-R1 achieves reasonable improvements in gIoU and cIoU compared to previous methods.", "questions": "Please refer to the weaknesses section for the rebuttal.", "limitations": "Yes, this is discussed in the Broader Impact section of the supplementary material, though it is not explicitly marked with a heading.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The paper is clearly written, with well-structured content and logical flow.\n\n2. The work introduces an interesting reinforcement learning approach for vision tasks and provides new insights into applying reward-based optimization in multimodal segmentation.\n\n3. The proposed method achieves reasonable empirical results.\n\nWeaknesses\n\n1. The technical novelty is limited, since the approach primarily leverages existing open-source models rather than introducing fundamentally new architectures or algorithms.\n\n2. A core limitation is the feasibility of reward computation in SAM-RL. The method requires extremely fine-grained segmentation annotations to compute meaningful rewards. For example, as illustrated in Figure 1, evaluating the model's ability to focus on a cat's eyes would require annotations for the cat eyes themselves, which are rarely available in standard segmentation datasets, where labels typically cover entire objects such as the whole cat. Even when using SAM's segment-everything mode to generate masks for parts like eyes, these regions lack semantic labels. Achieving high-precision fine-grained segmentation remains a challenging problem in the segmentation field. To fully leverage the RL mechanism proposed in this work, a large-scale dataset with detailed fine-grained segmentation annotations would be required, which inherently restricts the applicability of the method.\n\n3. There are potential concerns regarding the fairness of comparisons in Table 1. Most baselines, except for Seg-Zero, use weaker MLLM backbones compared to Qwen2.5VL-7B adopted in this paper, and it is unclear whether Seg-Zero's results correspond to the 3B or 7B model. Additionally, as noted in the paper, the Seg-Zero results are reproduced by the authors rather than directly from the original work. To ensure fair comparison, alternative training strategies and backbones should be tested under identical conditions.", "was_revised": true, "final_justification": "The authors' rebuttal successfully addressed my initial concerns, I am raising my score.", "rebuttal": "We thank the reviewer for the thoughtful and constructive feedback. We address the raised concerns as follows:\n\n1. On technical novelty and architectural contributions\n\n We acknowledge that our work leverages open-source components such as Qwen2.5-VL and SAM. However, we emphasize that the novelty of SAM-R1 lies not in introducing entirely new architectures, but in designing a novel reinforcement learning framework that enables these powerful models to collaborate effectively on fine-grained reasoning and segmentation tasks.\n\nWhile prior works have mainly focused on model architecture or segmentation backbones, we propose a unified system-level approach that includes:\n\n●Directly using SAM as a reward provider, introducing pixel-level supervision into RL without requiring any special preprocessing or conversion of segmentation data;\n\n●Modifying the GRPO reward objective to suit multimodal fine-grained reasoning tasks, allowing for interpretable and stable optimization signals;\n\n●Designing an end-to-end training pipeline that jointly optimizes reasoning and segmentation under natural language instructions.\n\nThis represents a contribution in system design rather than architecture alone, and we believe it offers more practical value for advancing instruction-based fine-grained vision understanding.\n\n2. On the feasibility of reward computation and dependence on fine-grained annotations\n\n We appreciate the reviewer’s concern regarding the reliance on fine-grained annotations. We would like to clarify that our method does not require manually annotated fine-grained segmentation masks.\n\nThe primary goal of the RL stage in SAM-R1 is to help the VLM understand the target referred to in the instruction via reasoning and accurately pass that semantic information to SAM for segmentation. The model's fine-grained understanding emerges from internal reasoning rather than from external labels.\n\nOur reward function provides feedback based on the IoU between the predicted segmentation mask and the ground-truth object-level mask. While standard datasets typically label whole objects, this still provides meaningful supervision — the goal is not part-level semantic correctness but rather alignment between the referred object and the predicted mask. Even in the absence of detailed part annotations, IoU-based feedback is sufficient to drive learning of semantic localization and perceptual alignment.\nFurthermore, we believe SAM-R1’s design actively mitigates the challenge of lacking fine-grained annotations. By delegating reasoning to the VLM and segmentation to SAM, the model learns to bridge language and vision without requiring costly, dense labels. Our results on standard datasets like RefCOCOg and ReasonSeg demonstrate that this design is both effective and practical in real-world settings.\n\n3. On fairness of comparison in Table 1\n\nWe sincerely thank the reviewer for raising these important concerns regarding experimental fairness. Addressing this point allows us to clarify the rigor of our evaluation. \n\nWe want to state unequivocally that the comparison between our SAM-R1 and Seg-Zero is entirely fair and controlled. Both models are built upon the exact same backbone, Qwen2.5-VL 7B.\n\nFurthermore, you correctly note that we reproduced the Seg-Zero results ourselves. This was a deliberate and necessary step to ensure a true comparison. We found that the results reported in their original paper were generated using different model weights for different metrics, which prevented a reproducible comparison from a single, unified model. To address this, we used their official, publicly released weights to generate all Seg-Zero results under an identical evaluation protocol.\n\nThis meticulous approach ensures that the observed performance gains are directly attributable to our proposed training paradigm—the RL-based reasoning and segmentation framework—rather than any differences in backbone or evaluation setup.\n\nWe agree with the reviewer that other baselines listed in Table 1 utilize different, and often weaker, MLLM backbones. Our primary goal with this table was to position our work within the broader landscape of existing methods.\n\nTo provide full transparency and address your concern directly, we will update Table 1 in the revised manuscript to include a dedicated column specifying the backbone model used by each method."}, {"review_id": "Tu1ZycM6pS", "reviewer": "Reviewer_KmuG", "summary": "This paper  propose SAM-R1,  a novel framework that enables multimodal large models to perform fine-grained reasoning in image understanding tasks. It incorporates fine-grained segmentation settings during the training of multimodal reasoning models. Extensive experiments demonstrate its effetiveness.", "questions": "1.Design and sensitivity of the reward. The goal of SAM-R1 is to enable the segmentation model to leverage GRPO so that it can exhibit reasoning capabilities without requiring an explicit reasoning chain as ground truth. This makes the design of the reward particularly critical. Specifically, the reward in Equation (5) depends heavily on the chosen threshold. Could you explain how this threshold is determined? Why is the reward discretised into multiple levels? Moreover, would small variations in the threshold value lead to significant differences in the results?\n\n2. Clarification of improvements over GRPO. The method presented in this paper appears to directly apply GRPO to the reasoning segmentation task. What are the main technical contributions or modifications compared to standard GRPO? It would be helpful to highlight the key improvements that make the proposed approach better suited for reasoning segmentation.", "limitations": "Please refers to weakness and questions sections", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength:\n\n1.\tThe paper is well-written and easy to follow.\n\n2.\tThe visualisation is straightforward, and easy to demonstrate the effectiveness of the proposed approach.\n\n3.\tThe topic the paper addressed is cut-edging and deserve to explore.\n\nWekaness:\n\n1. Missing related work. The current setting of task-generic, promptable segmentation also aims to infer image-specific answers given a generic prompt and perform corresponding segmentation. It would be beneficial to add a brief discussion of the literature on this setting [1][2][3]:\n\n[1] Hu, J., Lin, J., Gong, S., et al. Relax image-specific prompt requirement in SAM: A single generic prompt for segmenting camouflaged objects. Proceedings of the AAAI Conference on Artificial Intelligence, 2024, 38(11): 12511–12518.\n\n[2] Tang, L., Jiang, P. T., Shen, Z. H., et al. Chain of visual perception: Harnessing multimodal large language models for zero-shot camouflaged object detection. Proceedings of the 32nd ACM International Conference on Multimedia, 2024: 8805–8814.\n\n[3] Hu, J., Lin, J., Yan, J., et al. Leveraging hallucinations to reduce manual prompt dependency in promptable segmentation. The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.\n\n2. Insufficient discussion of innovation. The innovation of this work needs to be further clarified. The proposed method appears to directly apply GRPO to the reasoning segmentation task, so the key differences and improvements over standard GRPO must be more explicitly articulated.", "was_revised": true, "final_justification": "I have read the authors' rebuttal, I believe my concern have been addressed mostly.", "rebuttal": "We sincerely thank the reviewer for the insightful and constructive comments. We have carefully addressed each point as follows:\n\n1. Missing related work discussion\n\nThank you for your valuable suggestion. In the revised version, we will include a dedicated discussion on the task-generic, promptable segmentation setting, particularly focusing on methods that aim to infer image-specific answers based on generic prompts and produce corresponding segmentations. Specifically, we will incorporate the three cited papers into this discussion to better contextualize recent advances in this area. \n\n\n2. Insufficient clarification of innovation\n\n Leveraging multimodal large models for image segmentation has become a prominent research direction. However, existing approaches typically rely heavily on manually annotated datasets with explicit reasoning chains, which are costly and time-consuming to construct. Some recent methods have explored reinforcement learning for segmentation, such as Seg-Zero, but they often pre-process segmentation tasks into object detection form and fail to leverage task-specific, fine-grained reward signals.\n\n In contrast, we propose a novel formulation where SAM is directly employed as a reward provider in the reinforcement learning loop, requiring no special pre-processing of the segmentation dataset. This design enables us to use the predicted segmentation masks as supervision, providing direct, pixel-level feedback to guide reasoning. To complement this, we modify the GRPO optimization objective to improve exploration ability and interpretability, allowing the model to follow complex multimodal instructions and accurately localize the segmentation target. These innovations collectively allow SAM-R1 to achieve fine-grained perceptual reasoning with a simplified, end-to-end training pipeline.\n\n3. Reward threshold design and sensitivity\n\n Thank you for these detailed questions regarding our reward function. \n\nThe core motivation for our tiered reward design is to provide a stable and informative learning curriculum. A single, fixed threshold creates a challenging dilemma in RL-based training:\n\n●A high threshold makes rewards too sparse, especially in early training. Most predictions receive no reward, hindering efficient learning and exploration.\n\n●A low threshold leads to quick reward saturation. Once the model can consistently achieve a coarse segmentation, it receives no further incentive for fine-grained refinement.\n\nOur tiered reward strategy (as defined in Equation 5) elegantly resolves this trade-off. By discretizing IoU into multiple levels, it creates a hierarchical learning path, guiding the model progressively from coarse localization (earning a small reward) to precise segmentation (earning the maximum reward).\n\nThe effectiveness of this tiered approach is empirically validated by our ablation study, which we include here for your convenience:\n\n| Method | ReasonSeg (gIoU) | ReasonSeg (cIoU) | refCOCOg (gIoU) | refCOCOg (cIoU) |\n| :--- | :---: | :---: | :---: | :---: |\n| Fixed Threshold: 0.5 | 56.5 | 51.9 | 74.7 | 72.8 |\n| Fixed Threshold: 0.7 | 56.2 | 51.6 | 74.9 | 72.6 |\n| Fixed Threshold: 0.8 | 58.6 | 50.8 | 74.6 | 71.9 |\n| **Tiered (Ours)** | **60.2** | **54.3** | **75.4** | **73.1** |\n\nAs the results clearly demonstrate, the tiered strategy consistently and significantly outperforms all fixed-threshold alternatives. On the challenging ReasonSeg dataset, it yields a substantial +1.6 to +4.0 point gain in gIoU and a +2.4 to +3.5 point gain in cIoU compared to the single-threshold methods. This provides clear evidence that the hierarchical reward signal is crucial for achieving superior performance.\n\n4. Clarification of improvements over standard GRPO\n\nWhile GRPO provides a solid foundation, our method is not a direct application. We introduced significant, task-specific adaptations that are essential for enabling a model to perform complex reasoning for segmentation.\n\nOur main technical contributions can be summarized in three key areas:\n\n●Task-Specific Reward Engineering: Standard GRPO is designed for general text generation. We engineered a completely new reward function tailored for vision-language grounding. This includes:\n\nA multi-level, IoU-based reward (Eq. 5) to provide a granular learning signal for segmentation accuracy.\n\nA supplementary format-based reward to enforce correct structural output (i.e., reasoning text followed by segmentation coordinates).\n\n●Core Algorithmic Enhancements: We modified the GRPO optimization objective itself to improve training stability and performance on this unique task. Our key changes are token-level advantage normalization and an asymmetric clipping mechanism (Clip higher). These are not part of the standard GRPO implementation and are vital for balancing the needs of generating fluent reasoning with precise, numerical coordinate outputs.\n\n●Unified End-to-End Training: Unlike prior works that separate reasoning and segmentation modules, we train our system jointly using RL. This allows the model to learn a direct mapping from visual input and natural language instructions to segmentation masks, using pixel-level feedback to refine its reasoning process.\n\nThe necessity and impact of our algorithmic enhancements are empirically validated in our ablation study. We present the key results from Table 4 of our paper below:\n\n| Method | Token-level | CLIP higher | gIoU (RS) | cIoU (RS) | gIoU (Rcg) | cIoU (Rcg) |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: |\n| GRPO (baseline) | ✘ | ✘ | 57.8 | 51.2 | 74.1 | 71.8 |\n| Token-level | ✔ | ✘ | 58.0 | 51.7 | 74.5 | 72.4 |\n| Clip higher | ✘ | ✔ | 59.1 | 52.8 | 74.9 | 72.5 |\n| **Ours (Full)** | **✔**| **✔** | **60.2** | **54.3** | **75.4** | **73.1** |\n\nAs the table clearly shows, the standard GRPO baseline yields the lowest performance. Adding our proposed token-level normalization or asymmetric clipping individually provides noticeable gains. When combined, our full model (Ours) achieves a substantial +3.1 cIoU gain on ReasonSeg over the baseline.\n\nThis result directly demonstrates that our modifications are not minor tweaks but rather crucial components that collectively enable the success of our approach. We will ensure this clarification is prominently featured in the revised manuscript."}]

Author rebuttal:

Meta-review: This paper presents a novel framework named SAM-R1, which integrates SAM into a reinforcement learning loop to improve multimodal reasoning-segmentation. All reviewers actively engaged in the discussion, and most major concerns on novelty, fairness of comparisons, and reward design were resolved during rebuttal. While some questions remain e.g. the scalability to highly granular tasks, these do not outweigh the contributions and empirical strength of the work. The authors are encouraged to incorporate reviewer suggestions into the final version.
Decision: Accept (poster)